Measuring the Impact of School Closures on Student Performance

The public charter school sector is coalescing around school quality. The goal is to support the growth of new, high-quality charter schools and close low-performing charter schools (see NAPCS’ response to NACSA’s One Million Lives Campaign).

What is striking is that the public charter sector is reaching a consensus around the need for strategic charter school closures. Conversely, you don’t often hear clamoring for closure of low-performing traditional public schools from district school supporters—whether it’s community nonprofits or teachers unions. Closures of traditional public schools have typically been proposed by local school districts or called for by the U.S. Department of Education, and public responses verge on outrage (see examples from Philadelphia, D.C., Chicago, and NYC).

School closures are upsetting for students, parents, teachers, school staff, and the community. However, recent research on the DC Closure Initiative, which closed or consolidated 32 elementary and middle traditional public schools  the summer before the 2008-2009 school year, may provide some comfort to policymakers and educators contemplating school closures. The researchers found that student performance dipped in the first year after school closures, but rebounded by the second year. Moreover, students affected by school closures did not demonstrate higher rates of subsequent mobility. The study builds on previous research on the impact of school closures by RAND and the Consortium on Chicago School Research.

The findings from the study support the notion that making tough but necessary decisions to close chronically low-performing schools will ultimately ensure student access to better learning environments.

Lifting the Public Charter Cap in Massachusetts: Will It Reduce Achievement Gaps? Research Says Yes.

In January 2010, the Massachusetts legislature raised the cap on public charter schools in the state’s lowest-performing school districts. This opened the way for 20 new charter schools state-wide, with 11 slotted for Boston. As the Massachusetts Public Charter School Association (MPCSA) points out, the state legislature recognized the tremendous impact public charter schools have on student performance, and eased the cap with the specific goal of using high-quality charter schools to tackle persistent racial and socioeconomic achievement gaps.

Public charter schools in Boston have demonstrated large increases in achievement (see the 2009 Harvard/MIT study). Specifically, the study found very large gains in both math and reading for middle and high school students attending charter schools in Boston. Further, MPCSA reports that “the five highest performing public high schools are all charters and seven of the eight highest performing public middle schools are charters (based on 2010 MCAS scores comparing open admission public schools).”

Any curious person would ask: Did the policymakers’ decision to lift the cap on the number of charter schools in Massachusetts actually lead to better student outcomes?

As luck would have it, a new working paper by a researcher at MIT takes a very thorough and innovative look at the projected impact of opening new charter schools in Boston and reports: “[S]imulations show that Boston's proposed expansion, which raises the share of middle schoolers attending charters from 9 percent to 15 percent, is expected to reduce the gap in math scores between Boston and the rest of Massachusetts by 10 percent, and reduce citywide achievement gaps by roughly 5 percent.”

Take a look at two of the graphics presented in the paper (see below). Figure 8 shows the projected average test scores of all students in Boston as the number of charter schools increases.  The vertical black dotted line indicates the number of charter middle schools as of the 2010-11 school year, and the red dotted line indicates the number of charter middle schools after the proposed number of new schools open. Figure 9 shows the projected achievement gap among all students in Boston as the number of charter schools increases.

 
 

Source: Walters, C. (2012). “Predicting the effects of charter school expansion.” MIT working paper, http://economics.mit.edu/files/8138.

The big take away: The presence of more high-quality charter schools leads to higher citywide test scores and smaller achievement gaps throughout the city.

The simulations in the study indicate that there may be a limit in terms of demand to enroll in charter schools at about 24 percent of district market share. However, with 20,000 students on waiting lists to attend the roughly 7,000 seats currently available in charter schools in Boston, and a total of roughly 60,000 students in the district, it is feasible that charter schools could enroll between 40-50 percent of all public school students before demand would taper off. Moreover, if public charter schools produce the type of performance increases (and reductions in the achievement gap) that the simulations project, Boston could see parental demand for access to high-quality charter schools increase as the public sees significant results.

What do Special Education Enrollment Figures Really Tell Us?

• The statewide comparison of the difference in charter and district enrollment is too simplistic—charter schools on average serve a smaller share of special education students than New York’s district-run schools, but the distribution and range of enrollment are not that different from the district-run schools’ composition
• Charter middle and high school special education enrollments are indistinguishable from district enrollments, while charter elementary schools show underenrollment of students with disabilities.
• There is variation among charter authorizers—some oversee schools with special education enrollments that closely track those of nearby district-run schools; others do not.

Given the variation of special education enrollment across charter and district schools, the report calls for nuanced policies. Rather than using sweeping measures such as enrollment targets, policymakers and authorizers should conduct further research to identify where special education underenrollment exists in charter schools and examine possible explanations. Then work should be done with the charter school community to develop innovative strategies to address specific problems.

The charter community is taking this work very seriously. Last summer, the U.S. Government Accountability Office released a report that found that charter schools, on average, serve a smaller proportion of students with disabilities than district-run public schools. As a response to these concerns and to better serve their students and community, public charter schools, advocates in districts, states, and courts across the country have sought to improve access. The new analysis by CRPE helps the public charter community understand the problem and create appropriate responses.

Q&A with Naomi Chudowsky, Co-Author of New NAEP Report on Public Charter Schools

The National Assessment Governing Board (NAGB) recently commissioned a new study of public charter schools using National Assessment for Educational Progress (NAEP) data. In this Q & A, NAPCS Research Director Anna Nicotera talks with Naomi Chudowsky, co-author of the study.

AN: What has previous research using NAEP data found regarding the comparison of traditional public schools and public charter schools?

NC: The main U.S. Department of Education study was released in 2005 using 2003 NAEP data. The report examined 4th grade reading and math. For the first time in the 2003 NAEP administration, charter schools were oversampled to make sure there were enough students to compare with students in regular public schools (roughly 3,300 charter students and 190,000 students in regular public schools). The 2005 study found that charter school performance lagged behind that of regular schools in 4th grade math, and was about the same in 4th grade reading. As one of the first national studies comparing performance differences between charter and regular public schools, the study received a lot of attention.

AN: Why did NAGB commission a new analysis of charter schools using NAEP data?

NC: It had been a number of years since the first study using 2003 NAEP data, and given the increase in charter school enrollment, NAGB considered it worthwhile to look at the charter school performance data once again. Beginning in 2005, NAEP included enough charter school students to examine 8th grade in addition to 4th grade. New research could use 2003 as a baseline for 4th grade and 2005 as a baseline for 8th grade performance data.

There has also been more interest among NAGB members to examine the background information collected by NAEP, including information on student background variables, what teachers have been doing in class, school characteristics, amount of time spent in different subjects, etc. NAGB wants to publish more studies using background data and support researchers using the data. The background data elements are available on the NAEP Data Explorer and are publicly available to anyone interested in the data for research.

AN: What did you find in your study of public charter schools on NAEP?

NC: We looked at two major areas: charter school enrollment and student achievement. In terms of enrollment, some interesting subgroup data pieces emerged. For example, between the 2003 and 2011 NAEP administrations, enrollment of Black students in charter schools in urban areas jumped from four to 12 percent. The data suggest that charter schools are playing a larger role in the education of some subgroups of students.

The main story that came out of our analyses for performance was that when you look at the national level, NAEP scores for regular public schools look higher than for charter schools. However, when you focus in on cities and subgroups, you start to see that charter schools demonstrate higher performance.

Nationally, we found that NAEP scores for regular public schools were higher than for charter schools with statistical significance. But the category of regular public school includes all students who attend regular public schools across the country in all types of communities. When we focus in on urban areas where most charter schools are located, we no longer found an advantage for regular public schools.

Moreover, some student subgroups performed significantly higher in charter schools. For example, Black and Hispanic students attending charter schools performed higher than similar students in regular public schools, with statistical significance.

The most interesting analysis, in a way, was looking at four urban districts (DC, Atlanta, Chicago, and Milwaukee). The students in charter schools performed significantly better than students in regular public schools in many subject areas and grades analyzed. There were no instances of students in regular public schools performing significantly better.

It should be noted that because the charter school sample is small in NAEP, there is more room for statistical error and the performance differences have to be quite large to find the statistically significant differences we found between charter and regular public schools.

AN: Given the limitations of comparing public charter schools and traditional public schools with NAEP data, what can the findings tell us about charter school performance?

NC: There is a lot that we don’t know and can’t know from analyses using NAEP data. NAGB and NCES staff are very careful to explain that NAEP data cannot be used for causal inferences. We still don’t know that what is going on in the charter schools is the reason for the difference in performance. It could be that the kids in these schools are different based on background characteristics, and that it is these differences that cause differences in performance.

However, the results from our analyses point to some interesting descriptive findings. Researchers should build on these data and look at why students in charter schools are performing better. Is it because of the school? Or is it because of student background or because certain students are selecting charter schools? The findings are not conclusive, but suggest that there is something good happening in charter schools that we should find out more about. If the results are because of particular teaching methods, the organization of charter schools, or the extent to which charter schools involve parents, can these practices be replicated in other schools?

When studies using different data and different methodologies converge, the findings become compelling. And that is what appears to be happening with charter school research, particularly in certain cities. When research on charters started, the findings were all over the place, often finding exactly opposite results. But more recent research is showing that charter schools often outperform regular public schools when like groups of students, attending schools in similar locations, are compared.

AN: What additional research on charter schools do you believe can be done with NAEP data?

NC: My co-author, Alan Ginsburg conducted a study using the NAEP background data to study trends in time on learning. He looked at time spent on various subjects, absenteeism, and homework. But he wasn’t able to focus on charter schools because there was so much data. It would be interesting to compare time on learning between charter and regular schools. The NAEP background data also include a lot of information about instructional content and methods. It would be interesting to explore, for example, whether teachers in charter schools are more frequently engaging students in hands-on activities in science, or role-playing in social studies, or manipulatives in math.

NAEP would be good for getting a broad look into the “black box” of what’s going on in charter schools, and then further research could examine instructional methods in more detail using more rigorous research designs or case studies.

 

Naomi Chudowsky is a consultant with Caldera Research LLC, which does research on federal education policy for state and national clients. She was previously a senior program officer at the National Research Council, where she worked on a variety of studies related to education and testing. Before that she worked on test development for President Clinton’s Voluntary National Testing Initiative at the U.S. Department of Education and as the coordinator of Connecticut’s statewide high school testing program. She has a Ph.D. in educational psychology from Stanford University.

NAGB Again Pursuing Analysis of Public Charter Schools on NAEP

For almost a decade, there’s been a heated debate over what the National Assessment of Educational Progress (NAEP), commonly referred to as the nation’s report card, tells us about student achievement in public charter schools.

A couple of items of consensus emerged from the debate:

• NAEP data should be used and interpreted with caution. NAEP data present snapshot results of different students at different points in time. Moreover, comparing a snapshot of student performance between charters and traditional public schools does not control for the potential bias of students selecting to attend charter schools. (Check out this transcript of a 2004 PBS NewsHour interview with our President and CEO Nina Rees, who was then Assistant Deputy Secretary for the Office of Innovation and Improvement at the U.S. Department of Education, commenting on the release of 2003 NAEP data.)
• NAEP data may be used for descriptive analyses that can generate hypotheses for further study.

With that background, the National Assessment Governing Board (NAGB), the governing board of NAEP, commissioned a new study of NAEP data for charter schools this past summer. At their November board meeting, exploratory results were presented. The study addresses the following questions:

• Who attends charter schools?
• Does student achievement differ between charter schools and regular public school students?
• How do student subgroups perform in charter schools?
• How do charter schools compare with regular public schools in urban school systems?

While the study authors, Naomi Chudowsky and Alan Ginsburg, use caution about comparing public charter schools and traditional public schools with NAEP data, they report the following:

• “There is a consistent pattern of higher average NAEP scores for regular public schools than for charter schools when we look at the nation as a whole. But public school advantage has diminished.”
• “However, the closer we focus in on large cities, where most charter schools are located, the more the picture changes in favor of charter schools. In all large cities combined, student achievement is roughly even overall, but the black and Hispanic subgroups show higher scores in charter schools.”
• “When we examined four urban areas specifically (DC, Atlanta, Chicago, Milwaukee), students in charter schools significantly outperformed their peers in regular public schools in almost all of the subjects/grades analyzed.”
• “Charter school class sizes are smaller, and there is some evidence that charter school students receive more instructional time in some subjects.”

Last year we wrote about the most recent 2011 NAEP data (see here and here), pointing out the rise in NAEP test scores for public charter schools, especially among several subgroups and on the Trial Urban District Assessment (TUDA).

So what can NAEP data tell us about the performance of charter schools? In the best case scenario, the uptick in NAEP scores reflects the overall upward trend in charter student performance that more rigorous studies have been identifying over the past decade (see our write-up here). NAEP data on charter schools should be used in context and with caution. The higher NAEP test scores for charter schools may suggest that charter schools are getting better in comparison with traditional public schools, but the data cannot say this definitively. Fortunately, longitudinal studies using student-level data (and a meta-analysis of these studies) are pointing in the direction of increasing student performance of charter schools.

Public Charters Lead the Way in Extended Learning

Over the past 20 years, the public charter school movement has been on the cutting-edge of key school reforms: setting high expectation for all students, offering an education that prepares students for success in college and careers, and identifying highly effective teachers.

A new report documents just how far public charters are leading in another area: expanded learning time.

The National Center on Time & Learning (NCTL) examined all of the schools in its database of schools with school days longer than the tradition 6½ hours. They found that 60 percent of them are public charters. What’s really noteworthy about that figure—only 6 percent of all public schools are charter schools.

Here’s more from the report:

• Expanded-time (ET) charter schools typically offer longer school days than traditional district expanded-time schools. Just over half the traditional district ET schools provide a school day longer than 7.5 hours. In contrast, three-quarters of all public charter ET schools operate a school day that is longer than 7.5 hours, and nearly half of all charter ET schools operate a school day that is longer than 8 hours. This difference is particularly apparent at the elementary and middle school levels, where charter ET schools are much more likely than traditional district ET schools to offer a school day longer than 8 hours . Charter ET schools are also more likely than traditional district ET schools to offer a longer school year. While nearly one-third of charter ET schools operate with yearly schedules that are at least 10 days longer than surrounding schools, only 10 percent of traditional district ET schools fall in this category.
• Charter ET schools offer more total time in school per year that traditional district ET schools. Charter ET schools in the NCTL Database offer on average 100 more hours per year than traditional district ET schools in the database. Charter ET schools are also more likely to offer both a longer school day (8+ hours) and a longer school year (>187 days) than traditional district ET schools. Over 41 percent of charter schools fall in both these categories, while only 17 percent of traditional district schools do. By contrast, 81 percent of traditional district ET schools have both the shortest school year (<181 days) and the shortest school day (< 7.5 hours).

Images via the National Center on Time & Learning. See the full infographic here.

Are Newer Studies Finding Greater Charter School Performance Effects?

The recent CREDO study of New Jersey charter schools showed positive results for charter schools across the state, and really quite remarkable results for charter schools in Newark (read more about the study here, here, and here). The findings come on the heels of a couple of additional reports from CREDO since the large study in 2009 that showed positive results for charter schools in Indiana/Indianapolis and New York City.

Here’s our prediction: studies that show positive results for charter schools may be foreshadowing positive results to come. And we make this prediction based on evidence from studies over the past decade. Take a look at the figure below.

The figure plots out the impact from charter school studies released, by year, that qualified for the Betts & Tang meta-analysis, with the addition of significant studies released in 2011 and 2012. All of the studies used student-level, longitudinal data to compare the performance of students enrolled in charter schools with their counterparts in traditional public schools. The large circles indicate results from studies that used randomized field trial lottery designs and the smaller circles are results from studies that used other quasi-experimental research designs (e.g., student fixed-effects, student matching). The triangles represent results that were not statistically significant, but are included to demonstrate the direction of results. The dotted lines represent the overall effect sizes (ES) from the meta-analysis, by grade level and subject area.

The distribution of effect sizes indicates that findings from high-quality studies on charter school performance are trending positive, and getting bigger in more recent years. The dotted lines, which show the overall effect sizes from the meta-analysis, confirm this trend. In general, the meta-analysis found positive and statistically significant results from charter school studies for elementary and middle school grades. So it would not be a surprise if future studies show a similar pattern.

What could explain the upward trajectory in charter school performance effects in newer studies? There are a lot of possible explanations—here are just a few:

• The charter school sector is maturing. While making sure that low-performing charter schools are closed is a priority (see NACSA’s One Million Lives Campaign), authorizers, charter support organizations, up-and-coming school leaders, and charter schools that want to replicate are getting better at identifying the charter schools that will result in high quality learning environments. In other words, the charter sector is learning how to replicate success and take it to scale.

Other explanations are aligned with the research itself:

• Newer studies are using better data. The data in the more recent studies include longer spans of academic years, as well as data from more recent years. And the studies include data from more states. Up to early 2009 when NAPCS released this charter school research synthesis, the years covered in studies only went through 2006-07 and 14 states out of 40 with charter schools had been examined. Now, the newer studies are examining charter schools through the most recent year of available data (2010-11) and including more states (up to 23 states at last count). The possible implication of this is twofold: 1) studies that use more years of data are able to capture charter schools operating longer, and as charter schools are around longer, they get better; 2) the inclusion of more states (with newer data) means that the early results from just a handful of states don’t outweigh the impact of charter schools in more locations.
• Studies have become more sophisticated, using research designs that control for selection bias and student demographics.