The Brown Center on Education Policy at the Brookings Institution released a report yesterday on the impact of superintendents on student achievement.
The results are challenging; and, at first look, lead one to conclude that the role of the superintendent is superfluous to improving student achievement. The results are based on analysis of longitudinal student performance data in North Carolina and Florida. The statistical impact on student achievement attributed to superintendents is quite low, i.e. explaining only 0.3% of the variance in 4th and 5th grade test scores in the state of North Carolina. Student data from Florida was not used in the student achievement analysis. (See page 5 of the Brown study for details on the data set.) The data in the study suggests that teachers, classrooms, schools, principals, and the school district as a whole have statistically higher impacts on student achievement than does the superintendent. The study attributes characteristics of the district, such as, civic commitment to schools, school board leadership, press scrutiny, economic development, family social services, and so on, as having an impact on student performance in ways that are independent of the superintendency, i.e., they would happen regardless of who is in the superintendent’s role. However, no specific data analysis of individual superintendent characteristics is provided, and there is no recognition that the superintendent is a substantial contributor to the factors that the authors identify. The primary measure in the report is an examination of student performance data in school districts in the two states following the replacement of one superintendent with another and/or acceleration of student performance growth over the tenure of the superintendent. The study found that superintendents, in general, in these two states had an average of only three to four years in the role. To oversimplify, they found no difference in student performance when superintendents change. This is a fairly direct challenge to the concept of superintendent leadership and bears careful examination.
It is always a challenge to precisely identify the consequences of leadership behavior. This is true in any complex organization where the direct relationship between a CEO and the performance of the organization’s employees or the accomplishment of its goals is a hard connection to make. The study of leadership behavior has become a lucrative field for authors to present their particular views on how the leadership of the CEO can transform the organization and its culture. The focus is on identifying behaviors, both motivational and managerial, that contribute to an organization’s accomplishment of its goals. Authors Chingos, Whitehurst and Lindquist raise a good question: what is the evidence that the superintendent impacts student achievement?
Does a CEO matter to an organization? Is a CEO actually worth the substantial salaries they receive? Answering this question is normally based on some understanding of what a CEO actually does, i.e., what actions, goal setting functions, policies, budget appropriations, personnel practices, management strategies, Board relations, and so on are attributable to the CEO? The question for analysis is not whether there is a CEO and how long that person has been in place. The question is what actions are taken to support the overall success of the organization? In the case of the Brown Center report the authors do not attempt to identify specific leadership behaviors. Instead, they look only at length of service of the superintendent, glossing over the fact that both elected and appointed superintendents are in the same sample, i.e., politicians vs. professional educators; and they only looked at 4th and 5th grade student achievement data in one state as their measure of impact on student achievement. No analysis of differences in leadership behavior is provided.
This is methodologically weak in that no attempt is made to identify specific actions of superintendents that did or did not contribute to the positive effects of the teachers, principals, schools and districts that the report highlights. By not asking the right questions, the study draws a conclusion, i.e., superintendents don’t matter, without actually collecting and reporting information on any actions taken by superintendents. The authors looked at superintendents from 67 districts in Florida and 115 districts in North Carolina and the time period of the analysis captured a total of 434 individual superintendents. Student achievement data analysis was restricted to 4th and 5th grade results in North Carolina only. Narrowing effectiveness on student achievement to only 4th and 5th grade data in one state inappropriately narrows the definition of leadership impact.
In addition, unmeasured differences in achievement are attributed in the Brown Center report as “Student” differences without definition. “Controls” are measured differences, such as, race/ethnicity. In Figure 4 (Page 10) of the Brown Report the authors claim that 90.8% of the variance in 4th and 5th grade math achievement in North Carolina is attributed to student variables (52% “Student” and 38.8% “Control”). The authors describe the “Students” variable, accounting for 52% of the variance in the data, as “all unexplained variance including measurement error.” (Page 10) By calling this “Student” variance and not “unexplained” variance, a reader can inaccurately conclude that the combination of student level variables explains most of the variance. This attempt to define school outcomes as the result of differences in students is inconsistent with the significant amount of research on quality schools and districts that demonstrates the characteristics of high performing schools that successfully address student level variables. The Brown Center data attributes only 4% of the explained variance to teacher differences which is a significant departure from other research. They attribute only 0.3% of the explained variance in test data to differences in the superintendent. However, the majority of the variance is being attributed to unmeasured variables assuming that the unmeasured variables have nothing to do with the superintendent. Is it not possible that the Brown Center report simply did not ask the right questions? For example, a superintendent may exert leadership around issues of a race-based achievement gap and you would not, necessarily, see that influence attributed to the superintendent in this type of data collection. If parents become more engaged, if local community-based organizations become more collaborative partners, if teachers begin to focus on racial disparities in more effective ways, etc., this study would not attribute those differences to the superintendent because it did not ask that question. Attributing unmeasured variance under a “Student” category is misleading and likely inaccurate. At a minimum it reflects that many questions were left unasked in this analysis.
In contrast, consider the 2006 study by Waters and Marzano of the Mid-Continent Educational Laboratory. It is interesting to note that this well-known study was not even mentioned in the Brown report. The Waters and Marzano work is a meta-analysis of 27 studies of superintendent leadership behaviors and their impact on student achievement. The analysis looks at studies that involved 2,817 school districts and the achievement scores of 3.4 million students. Fourteen of the 27 studies had specific student achievement data for analysis of district leadership impact on student data. The study found a .24 positive correlation between district leadership and student achievement (p<.05). This explains as much as a 9.5 percentile point growth in student achievement attributed to leadership characteristics of superintendents. The analysis looks specifically into the types of leadership behaviors that contribute to positive student outcomes and identifies statistically significant predictors. Waters and Marzano identified five statistically significant factors:
- The goal setting process (Average r = .24, p<.05)
- Non-negotiable goals for achievement and instruction (Average r = .33, p<.05)
- Board alignment with and support of district goals. (Average r = .29, p<.05)
- Monitoring the goals for achievement and instruction (Average r = .27, p<.05),
- Use of resources to support the goals for achievement and instruction. (Average r = .26, p<.05)
Particularly relevant to understanding the Brown Center report is a sixth finding in the Waters and Marzano study that found that the degree of autonomy provided to principals to lead their schools within the context of clear district goals, expectations and supports for teachers was positively correlated to student achievement. (r=.28, p<.05) In other words, there is a positive impact on student achievement when superintendents focus on hiring high quality principals and teachers, when they empower them to focus on the mission of the district at the local school level; when there is support for innovation and provision of district supports for professional development and innovative practices; and where high levels of achievement by all students is a clear, measured, district-wide expectation. Therefore, the evidence in the Brown Center report showing the positive role of teachers, principals and school level factors may, in fact, also be seen as evidence of district level leadership that has created the conditions for this level of impact to occur, i.e., clear policy, high expectations, personnel practices, budget alignment. This “tight-loose” management style may be very effective as a district strategy, but the Brown Center report would see these as effects attributed to teachers, principals, and buildings independent of the superintendent.
By not asking the question of specific leadership behavior, the Brown Center report concludes:
In the end, it is the system that promotes or hinders student achievement. Superintendents are largely indistinguishable creatures of that system. (Page 14)
This is a misleading and inaccurate representation of the role of district leadership in creating conditions for success. The issue is not whether the superintendent is treated as an independent variable from teacher, principal and school effects. On the contrary, the issue is how the superintendent influences, supports and contributes to the effectiveness of these variables. If teachers are effective, what leadership behavior supports that effectiveness? If principals are effective leaders, what steps did the superintendent take to support those administrators? If the Board is demonstrating effective leadership, what steps did the superintendent take to foster and collaborate with that vision? If the system has invested in effective strategies and practices, what budgetary practices were used to support those elements? The Brown Center report does a disservice to understanding the impact of district leadership by not looking at specific leadership behavior in the successful districts they identified.
Pat Burk, Ph.D.
Department of Educational Leadership and Policy
 Waters, J. T., & Marzano, R. J. (2006). School district leadership that works: The effect of superintendent leadership on student achievement. Denver, CO: Mid-continent Research for Education and Learning. – See more at: http://www.mcrel.org/products-and-services/products/product-listing/01_99/product-90#sthash.wYWLQAAj.dpuf