Research Rewind: Back to Basics, Quality Matters

It’s no secret that the performance of charter schools varies quite a bit. With over 5,000 charter schools and wide variety in mission, vision and instructional focus, examining charter schools as a whole will lead to a mixed bag of results. Research is moving in the direction of identifying the educational conditions that may lead to differences in results, but there is still much to learn about effective practices.

As the year comes to a close, let’s step back and ask a very basic school choice question: what happens when students choose high quality schools?

A series of studies on a district-wide open enrollment school choice policy in Charlotte-Mecklenburg, North Carolina provide some answers. The studies are sophisticated by design, but offer straightforward results:

• Families who selected schools based on academic quality (higher school performance), rather than by the proximity of schools or the student demographic make-up of schools, experienced significant gains in test scores as a result of attending the higher quality school.
• Students zoned to low-quality neighborhood schools who won lotteries to attend higher quality high schools were more likely than students who lost the lotteries and attended lower quality neighborhood schools to graduate from high school, attend a four-year college, and earn a bachelor’s degree. Moreover, these students were twice as likely to attend an elite university.
• Families who received direct and easy-to-read information on school test scores selected higher-scoring schools for their children. And again, the researchers found that attending higher performing schools led to better academic outcomes for the students.

These studies should motivate the charter school sector. The studies do not support school choice for school choice’s sake. And the studies don’t suggest that school choice is very effective when families choose schools for non-academic reasons. Rather, the studies provide concrete evidence that when families use school choice to select higher performing schools, their children perform better.

The charter sector has to use evidence like this to ensure quality and uphold the charter school bargain—autonomy in exchange for accountability. Consistently low performing charter schools should not be allowed to stay open. Instead, the sector should be working to provide more high quality options to families. If the charter sector can define what academic quality looks like and guarantee that every charter school is a very good option for students, widespread results will follow.

Charters can ‘TUDA’ their own horns

The recently released NAEP Trial Urban District Assessment (TUDA) includes breakouts for charter public schools and traditional public schools in six out of 21 districts assessed. The table below shows that charter schools outperformed traditional public schools in the majority of districts in both subject areas and grade levels. In all of the other instances, charter schools performed as well as traditional public schools (e.g., the differences in scores were not statistically significant). In both the 2009 and 2011 NAEP TUDA, charter schools never underperformed against traditional public schools.

Others have pointed to the limited number of districts in NAEP with both charter and traditional public school data to compare. A larger sampling of charter schools in big districts like DC, Detroit, Los Angeles, New York City, Philadelphia, and San Diego could allow for further analyses of NAEP performance trends at the district level.

The NAEP assessment does have limitations, as we have discussed in previous blogs. Moreover, comparing a snapshot of student performance between charters and traditional public schools does not control for the potential bias of students selecting to attend charter schools. However, the large differences below deserve attention and further analysis.

2011 NAEP, Trial Urban District Assessment (TUDA)  for Charter and Traditional Public Schools

Data compiled by NAPCS from: http://nces.ed.gov/nationsreportcard/naepdata/dataset.aspx
Note: CPS – Charter Public School; TPS – Traditional Public School; N/A – insufficient data to compare CPS and TPS; ↔ – no statistically significant difference in the scores

New Study on Unionized Charter Schools

Two Million Charter Students!

Details from the Dashboard

Charters Lagging on NAEP? NOPE!

There has been a fair amount written about the recently released NAEP results for 4th and 8th graders in math and reading, and most of the discussion has focused on the minimal overall gains from 2009 to 2011. But a closer look at charter schools compared with traditional public schools demonstrates that students in charter schools made some very large gains (see table below).

In nearly every category that we examined, students in charter schools made larger gains than students in traditional public schools. And these gains are substantial. At the overall level, charter school gains decreased the gap between students in charters and traditional public schools. In many of the breakouts by student and school categories, charter schools exceeded traditional public schools and the gains charters made are quite large.

Take a look at the gains ELL students attending charter schools made—a 15 point gain in 4th grade reading, a 10 point gain in 8th grade reading, and a 7 point gain in 4th grade math. In every assessment, ELL students who attended charter schools scored higher than ELL students in traditional public schools. Charter schools still have work to do to close achievement gaps between subgroups, but with these kind of gains charter schools are making good progress.

NAEP certainly has its limitations—the assessment is administered every two years to different students, and only to a sample of schools nationwide. However, NAEP has value as a national indicator of achievement trends. And the trend for charter schools appears to be on the rise. These NAEP results are consistent with the findings from the UCSD meta-analysis that showed positive results for elementary and middle school students who attend charter schools.

NAEP Gains between 2009 & 2011, Charter and Traditional Public Schools

Data compiled by NAPCS from: http://nces.ed.gov/nationsreportcard/naepdata/dataset.aspx
Note: 2011 NAEP Average Scores in parenthesis; CPS - Charter Public School; TPS -Traditional Public School; FRL - Students Eligible for Free or Reduced Price Lunch; ELL - English Language Learners

Taking Excellence to Scale

The good news to come out of the new study of Charter Management Organizations (CMOs) is that there is strong evidence that high quality public charter schools can be taken to scale.  Several of these CMOs produce outcomes equivalent to three years of learning gains in just two academic years. The even better news is that the apparent drivers of these impressive performance gains are educational practices that the entire charter sector (and probably all schools)  can put into practice.  

There are two promising practices identified in the study:  intense instructional coaching, particularly with new teachers, and the implementation of a culture of high expectations through comprehensive behavior policies.  Broad application of these approaches could lead to more widespread academic quality, a true scaling of excellence.  Importantly, these practices can be adopted without diminishing individual charter autonomy over curricular content or instructional strategies.  In short, the practices associated with high performance in large CMOs don’t require a central office or multiple locations. 

Let’s take a closer look at the study, by Mathematica and the University of Washington’s Center for Reinventing Public Education (CRPE). It examines nonprofit CMOs that had a minimum of four schools open in 2007, direct control over the decision to hire and fire school leaders (which excludes most KIPP schools), and served students who were not primarily dropouts or similar special populations. Forty CMOs with 292 schools met these criteria. For the achievement impact analysis, the sample was further decreased to 22 CMOs based on having schools with middle school grades and data to compare with traditional public schools.

Some interesting findings:

• The overall impact of CMOs on student achievement is positive, but not statistically significant.
• On average, large CMOs made larger gains. The magnitude of the positive gains experienced by highest performing CMOs are great enough to overcome black-white and Hispanic-white NAEP achievement gaps.
• The study found no evidence that CMOs focus on one subject area at the expense of another. Rather, CMOs that perform well in math also perform well in reading.
• There is a positive relationship between math performance and a larger percentage of CMO teachers  who are from Teach for America (TFA) or other teacher fellowship programs.
• Compared with schools in traditional school districts, CMOs are less prescriptive in determining curriculum and instructional materials, log more instructional time (mostly through longer school days and more time on task), are more likely to hire teachers based on sample lessons and teacher commitment to the mission of the school, have larger applicant pools of teachers for each open position (63 vs. 20 applicants), are more likely to employ performance-based compensation, observe teachers and provide feedback more frequently, more often require teachers submit lesson plans for review, and use coaching and monitoring more often than “in-service” or workshop days for professional development.
• States with higher scores on the components that lead to more autonomy in NAPCS’ model law have more CMOs.

This report provides lots to dig through, and there will certainly be a lot of discussion around this report. But the big takeaways are that a good number of CMOs are taking high quality education to scale, and there are common, high quality practices that can be implemented in more charter schools.

Practice Makes…Effective

Charter management organizations (CMOs) are in the news. And we are going to hear a lot more about the impact of CMOs on student performance after tomorrow’s  release of Mathematica/CRPE’s report that represents the most comprehensive look at  CMO effectiveness to date.

But before the release of tomorrow’s report, let’s revisit the first report, which detailed the practices of CMOs, drawn from observations, interviews, surveys, and reviews of financial documents. The first report is packed with information about the national CMO landscape, CMO characteristics, how CMOs operate, and plans for growth. The report notes that, despite the relative newness of the CMO landscape, they are already a significant presence in the charter school movement. The report provides insight into the commonalities across CMOs—such as being highly concentrated in certain urban areas and having a mission to serve disadvantaged  youth and reform large school systems, as well as hurdles many CMOs face—such as extending program design into new grade configurations, increasing the pool of talented teachers and school leaders, reducing staff burnout, and watching out for bureaucratic creep.

The first report is definitely worth a read as it gives context for the upcoming performance evaluation. Like the charter sector as a whole, there is a good deal of variation in terms of how CMOs operate. It will be interesting to see how the variation in practices translates into student performance. And while you’re waiting for tomorrow’s release, you can also get lots of facts about CMOs and EMOs here.

Great and Still Striving

The Results Are In…