School Districts Getting Free Evaluation Tools Developed by UW-Madison Education Researchers
Team from WEC develops ‘toolkit’ to help improve required academic and career plans for students
March 12, 2019 | By Karen Rivedal
The process of evaluation includes the key task of figuring out what worked and what didn't work in a program.
School districts across Wisconsin can get expert help making sure their academic and career planning for students is working well using a new set of tools developed by evaluation professionals at the Wisconsin Center for Education Research (WCER) in UW-Madison’s School of Education.
The free guidance, now live online at a state Department of Public Instruction website, is aimed at helping district and building leaders systematically assess the effectiveness of their ACP plans, which are required to be in place for every student in grades 6-12 under a state law that took effect in fall 2017.
Any school or district ACP program is intended to get students thinking sooner about what they’re good at and what they might want to do after high school or college, using a variety of delivery methods such as course content, job shadowing, counseling, internships and interest surveys.
But while the state provided a wealth of resources to help districts create and implement their ACP plans – and a two-year, ramping-up period before the requirement took effect – little attention was paid to encouraging districts to regularly examine whether those plans result in expected student outcomes.
So additional guidance on evaluation was needed, says researcher Robin Worth, who led the team at WCER’s Wisconsin Evaluation Collaborative (WEC) that suggested and then created the ACP Program Evaluation Toolkit at DPI’s behest in about four months.
“Knowing how you’re doing helps you do better,” Worth, a professional evaluator, notes. “Many districts, I would say, can’t afford an outside evaluation (to assess their plans). But that doesn’t mean they can’t get quality data and decent feedback if they do it in a thoughtful, systematic way.”
The toolkit includes a sample evaluation plan, 10 pages of instructions, and five question banks with suggested inquiries to be used in surveys and for stakeholder focus groups to collect feedback. The stakeholder groups cover the full range of those with a hand in or affected by the plans: students, teachers, families, district leaders and building leaders, such as principals or ACP coordinators.
The question banks revolve around a variety of concepts such as program awareness, implementation, perceptions of value, student experiences, family involvement and equity designed to get at fundamental issues of whether and how the ACP is working and what could be different. Questions are offered in different formats as well, including yes/no, along a scale, multiple choice and open-ended.
WEC Qualitative Researcher Robin Worth led the team that developed the ACP Toolkit.
But Worth describes the 10-page guidance document as perhaps the toolkit’s most essential piece, as it offers a way to highlight the most important questions that would-be assessors should be asking themselves, before they even begin trying to evaluate their ACP’s value and effectiveness.
“What do you want to know and why?” Worth says. “There are all these things to think about before you just dive in and do a survey. How are you going to go about it? Do you have the capacity for this? Plan it first. Focus down. Don’t collect data you’re not going to use.”
“It’s sort of a how-to,” Worth adds about the guidance document. “Because it’s one thing to have (answers to) a bunch of survey questions, but it’s another thing to know what to do with all this stuff.”
Milton School District School-to-Work Coordinator Amy Kenyon agrees it’s important to evaluate ACP programs, noting Milton High School requires a portfolio presentation from all graduating seniors about their college/career plans that helps officials see if their extensive ACP programming is working, along with staff and program volunteer feedback. Kenyon says she looks forward to exploring the new ACP Toolkit on the DPI website, while the district’s own evaluation process continues to evolve, including the addition of surveys for former students one year and eventually five years after graduation.
“We would like to see if (the plan) was doing what it’s intended to do,” Kenyon says. “Did it help them? Are we seeing more that are continuing on in their field of choice or do we need changes?”
WEC’s idea for the ACP Toolkit grew out of its contracted work over the past three years evaluating DPI’s rollout and implementation of the ACP program. The self-evaluation plans envisioned for individual school districts or schools through the kit are like WEC’s work for DPI on a miniature, more basic level.
Statewide surveys and pilot programs at about 20 school districts done as part of the ACP assessment for DPI also made clear that districts would welcome the tool kits. Some participants received free “mini-evaluations” from WEC and were “so grateful and found them so useful,” Worth says.
“We realized schools can really benefit and are really interested in evaluation on their local level,” Worth says. “They want to know how they are doing with their own design in their own context. And we realized we could support schools and districts in doing their own evaluations (with the ACP Toolkit.)”
One thing schools and districts can do for WEC, on the other hand, is share information on their experiences using the toolkit, Worth says. Users can link to a brief online survey from the guidance document and question banks to leave feedback that WEC will use to revise the kit for fall 2019.
Robin Kroyer-Kubicek, DPI's ACP Implementation co-lead, says WEC has been a great partner and continues to support DPI through this tool and through the annual ACP evaluation to better define which practices and components of ACP lead to the best outcomes for schools overall.
Created in 2016, WEC is a network of experienced evaluators from across WCER and the School of Education that does program evaluations within the preK-12 education system for school districts, professional associations, state agencies and education-based community organizations. Its evaluators bring a diversity of content and methods expertise, with services offered including initial consultations, feedback on plan designs, full evaluations and strategic planning for logic modeling, crafting research questions, identifying potential data sources and designing continuous improvement processes.
WEC’s work for DPI on the ACP assessment toolkit was paid for through a grant from the State Longitudinal Data System (SLDS). The SLDS is an ongoing program created by the U.S. Department of Education to develop tools and data systems to help school districts make data-driven decisions aimed at increasing student performance and closing academic achievement gaps.
