New Approaches to Performance Management and Value-Added in Urban Schools

June 10, 2011

More schools and districts around the country are using data at the school level to inform instructional decisions and support student learning. That’s a promising trend.

But research on using data to make instructional decisions doesn’t yet provide conclusive evidence of “what works” to improve student achievement.

Sara Kraemer, Elisabeth Geraghty, Deborah Lindsey, and Cynthia Raven (2010) argue that the performance management movement addresses this concern. Researchers at WCER’s Value Added Research Center, Kraemer and Geraghty analyzed human factors and organizational factors that affect performance management, school improvement, and data use. Working with Lindsey and Raven , Department of Research and Assessment, Milwaukee Public Schools, they found that the design and implementation of data-centric improvement efforts are enhanced by using what’s called a macroergonomic approach.

In the context of education, performance management refers to improving the effectiveness of schools and districts using rigorous data, regularly assessing school and district performance, and holding managers accountable. Every level of a school district can use performance management principles of quality and systems management and the associated tools and processes.

Macroergonomics is a subdiscipline of ergonomics. It focuses on the design of an overall work system. A work system can include personnel, technology, organization and management, and the external environment.

Kraemer and colleagues analyzed eight public schools in Milwaukee, Wisconsin, from October 2008 through February 2009. They measured performance at the school level via value-added analysis. Value-added analysis measures school productivity and the contribution of schooling to growth in student achievement. It uses statistical techniques to separate the effects of schooling from other non-school factors that may influence student achievement, including students’ prior academic achievement, family mobility, race, and socioeconomic status.

The Milwaukee Public Schools system designates individual schools as high- or low-performing, based on student attainment scores on Wisconsin Knowledge and Concepts Exam (WKCE) . However, simple measures of student attainment do not filter out students’ prior academic achievement, family mobility, race, or socioeconomic status.

To assess school performance taking these factors into account, Kraemer and colleagues sampled eight schools that varied in performance by comparing their value-added measures and student attainment measures on the WKCE.  They established four performance levels, each including two dimensions:

• At High VA/high attainment schools, students tested high on attainment and their academic performance grew faster than the district average.
• At High VA/low attainment schools, students’ academic performance grew faster than the district average, even though these students tested low on the state test and they did not achieve the mandated proficiency goal.
• At Low VA/high attainment schools, students tested very high, but relative to the district average, their academic performance grew slower from year to year.
• At Low VA/low attainment schools, students tested low on the state test and student achievement grew slower than the district average.

Schools’ Reactions to Their Scores

The high VA/high attainment schools tended to view value-added measurement as a validation because it demonstrated that they produce high-performing students. The high scores were not simply the result of having “good” students. The high VA/low attainment schools also felt validated. Their student growth rates were high, even though their attainment scores didn’t meet district standards.

Low VA/high attainment schools tended to view their VA scores with some disbelief. Even though their students tested high, these schools were not contributing to growth in student achievement. So, in fact, they were not high-performing schools. They did, however, want to learn about growth strategies for producing high-attaining students. Low VA/low attainment schools believed that students were the problem. They did not acknowledge value-added scores as a valid measure of student learning.

How Schools Used Data

The eight schools varied in the sophistication of their data use. The high VA/high attainment schools were able to identify mismatches between content areas of the state test and their current curriculum. These high-performing schools articulated a “culture of data use and mindset of student growth.” By contrast, low-performing schools focused on students’ behavior rather than their academic growth.

School learning teams also differed in performance. High-performing schools emphasized collaboration among school leaders and teachers. Teams met regularly to discuss and plan specific goals. Low VA/low attainment schools did not demonstrate this level of team cohesion or focus. Some meetings did not include an agenda or lacked discussion of data or improvement planning.

Kraemer and Geraghty found that a school’s perception of its productivity shapes how it plans (or doesn’t plan) for performance management. High VA/high attaining schools demonstrate that it’s possible to grow high-attaining students every year. Conversely, high VA/low attaining schools serve as examples that all students can learn, regardless of their starting point.

The Evolution of Performance Management in MPS

This study demonstrated that human factors and ergonomics may be a viable approach to the evolution of performance management at the school level.

First, schools should adapt or develop team-based organizational models for a learning-team approach to performance management. Those models should account for the realities and constraints of teachers’ workload, their task/teaching composition, and training development. Developing organizational and job design methods for teachers and school leaders can help reconcile mismatches between teacher job design and collaborative, team-based approaches for school improvement.

Second, in some circumstances, school leadership practices may benefit from adopting a top-down, macroergonomic approach and systematizing school functions across the organization.

Third, a human factors approach to communicating productivity metrics, such as value added, may assist in the accurate recognition of school performance. For example, parents and administrators can make comparisons across schools in networks with similar student characteristics, but which differ on productivity scores. The value added/attainment comparison metric can help differentiate school performance and provide support to schools based on their performance needs.

Reference

Kraemer, S., Geraghty, E., Lindsey, D., & Raven, C. (2010). School leadership view of human and organizational factors in performance management: A comparative analysis of high- and low-performing schools. In Human Factors and Ergonomics Society (Ed.),Proceedings of the Human Factors and Ergonomics Society Annual Meeting (pp. 1287-1291). San Francisco, CA: Human Factors and Ergonomics Society.

Acknowledgements

This research was funded by the U.S. Department of Education - Institute of Education Sciences (Grant # R305A080038, PI: Robert H. Meyer) and the Joyce Foundation. The authors would like to thank Deborah Lindsey and Cynthia Raven, Department of Research and Assessment, Milwaukee Public Schools, for their collaboration and support in this study.