The National Assessment Governing Board (NAGB) recently commissioned a new study of public charter schools using National Assessment for Educational Progress (NAEP) data. In this Q & A, NAPCS Research Director Anna Nicotera talks with Naomi Chudowsky, co-author of the study.
AN: What has previous research using NAEP data found regarding the comparison of traditional public schools and public charter schools?
NC: The main U.S. Department of Education study was released in 2005 using 2003 NAEP data. The report examined 4th grade reading and math. For the first time in the 2003 NAEP administration, charter schools were oversampled to make sure there were enough students to compare with students in regular public schools (roughly 3,300 charter students and 190,000 students in regular public schools). The 2005 study found that charter school performance lagged behind that of regular schools in 4th grade math, and was about the same in 4th grade reading. As one of the first national studies comparing performance differences between charter and regular public schools, the study received a lot of attention.
AN: Why did NAGB commission a new analysis of charter schools using NAEP data?
NC: It had been a number of years since the first study using 2003 NAEP data, and given the increase in charter school enrollment, NAGB considered it worthwhile to look at the charter school performance data once again. Beginning in 2005, NAEP included enough charter school students to examine 8th grade in addition to 4th grade. New research could use 2003 as a baseline for 4th grade and 2005 as a baseline for 8th grade performance data.
There has also been more interest among NAGB members to examine the background information collected by NAEP, including information on student background variables, what teachers have been doing in class, school characteristics, amount of time spent in different subjects, etc. NAGB wants to publish more studies using background data and support researchers using the data. The background data elements are available on the NAEP Data Explorer and are publicly available to anyone interested in the data for research.
AN: What did you find in your study of public charter schools on NAEP?
NC: We looked at two major areas: charter school enrollment and student achievement. In terms of enrollment, some interesting subgroup data pieces emerged. For example, between the 2003 and 2011 NAEP administrations, enrollment of Black students in charter schools in urban areas jumped from four to 12 percent. The data suggest that charter schools are playing a larger role in the education of some subgroups of students.
The main story that came out of our analyses for performance was that when you look at the national level, NAEP scores for regular public schools look higher than for charter schools. However, when you focus in on cities and subgroups, you start to see that charter schools demonstrate higher performance.
Nationally, we found that NAEP scores for regular public schools were higher than for charter schools with statistical significance. But the category of regular public school includes all students who attend regular public schools across the country in all types of communities. When we focus in on urban areas where most charter schools are located, we no longer found an advantage for regular public schools.
Moreover, some student subgroups performed significantly higher in charter schools. For example, Black and Hispanic students attending charter schools performed higher than similar students in regular public schools, with statistical significance.
The most interesting analysis, in a way, was looking at four urban districts (DC, Atlanta, Chicago, and Milwaukee). The students in charter schools performed significantly better than students in regular public schools in many subject areas and grades analyzed. There were no instances of students in regular public schools performing significantly better.
It should be noted that because the charter school sample is small in NAEP, there is more room for statistical error and the performance differences have to be quite large to find the statistically significant differences we found between charter and regular public schools.
AN: Given the limitations of comparing public charter schools and traditional public schools with NAEP data, what can the findings tell us about charter school performance?
NC: There is a lot that we don’t know and can’t know from analyses using NAEP data. NAGB and NCES staff are very careful to explain that NAEP data cannot be used for causal inferences. We still don’t know that what is going on in the charter schools is the reason for the difference in performance. It could be that the kids in these schools are different based on background characteristics, and that it is these differences that cause differences in performance.
However, the results from our analyses point to some interesting descriptive findings. Researchers should build on these data and look at why students in charter schools are performing better. Is it because of the school? Or is it because of student background or because certain students are selecting charter schools? The findings are not conclusive, but suggest that there is something good happening in charter schools that we should find out more about. If the results are because of particular teaching methods, the organization of charter schools, or the extent to which charter schools involve parents, can these practices be replicated in other schools?
When studies using different data and different methodologies converge, the findings become compelling. And that is what appears to be happening with charter school research, particularly in certain cities. When research on charters started, the findings were all over the place, often finding exactly opposite results. But more recent research is showing that charter schools often outperform regular public schools when like groups of students, attending schools in similar locations, are compared.
AN: What additional research on charter schools do you believe can be done with NAEP data?
NC: My co-author, Alan Ginsburg conducted a study using the NAEP background data to study trends in time on learning. He looked at time spent on various subjects, absenteeism, and homework. But he wasn’t able to focus on charter schools because there was so much data. It would be interesting to compare time on learning between charter and regular schools. The NAEP background data also include a lot of information about instructional content and methods. It would be interesting to explore, for example, whether teachers in charter schools are more frequently engaging students in hands-on activities in science, or role-playing in social studies, or manipulatives in math.
NAEP would be good for getting a broad look into the “black box” of what’s going on in charter schools, and then further research could examine instructional methods in more detail using more rigorous research designs or case studies.
Naomi Chudowsky is a consultant with Caldera Research LLC, which does research on federal education policy for state and national clients. She was previously a senior program officer at the National Research Council, where she worked on a variety of studies related to education and testing. Before that she worked on test development for President Clinton’s Voluntary National Testing Initiative at the U.S. Department of Education and as the coordinator of Connecticut’s statewide high school testing program. She has a Ph.D. in educational psychology from Stanford University.