The exuberant new Hoxby study of NYC charters, reviewed here yesterday, came bundled with a surprise: a stinging “note” criticizing the methodology of CREDO’s downbeat 16-state charter study released in June – and affirming concerns the Alliance raised at that time.
A central issue was that CREDO’s student matching technique failed to adequately mitigate the effect of self-selection bias on the achievement outcomes. Self-selection bias is a fairly complicated statistical problem, but it can be explained in layman’s terms.
Parents voluntarily choose to enroll their children in charter schools. Parents choose charter schools for a variety of reasons. Self-selection bias arises because the motivation for selecting charter schools makes the students different than students who remain in traditional public schools, in ways that may impact student achievement. To attribute achievement differences to the time spent in charter schools rather than the motivation to attend charter schools, researchers must control for self-selection bias.
The most striking evidence that something was amiss with the CREDO study was that half of the charter school sample was matched after the decision to attend a charter school had been made. Specifically, half of the charter school students did not have test scores in traditional public schools so CREDO did not actually know which school the students would have attended if they had not attended the charter school.
Consequently, charter school students were matched with “virtual twins,” groups of traditional public school students who may or may not have had the same opportunity or motivation to choose a public charter school. The self-selection bias was not controlled for in the study.
Hoxby’s response on this point is technical but concise. She adds other criticisms on the assumptions necessary for matching that the CREDO study violates – but saves her strongest language for the sheer lack of analysis, making replication of the CREDO study impossible.
We look forward to CREDO’s response to Hoxby’s critique. Maybe CREDO will use this opportunity to present detailed analytic models, robustness tests, and descriptive data that were all missing from the June study.
|