Making sense of the evidence on charter school test scores

Imagine a police officer pulls you over and tickets you for speeding. She tells you she measured you going 50.5 MPH in a 50 MPH zone. No, you reply, my speedometer shows that I was going exactly 49.5. The entire discussion would be absurd, since neither your speedometer nor the officer’s radar gun is sufficiently accurate to support the opposing claims, and a 0.5 MPH difference is not practically meaningful.

Welcome to the world of debates about the effects of American charter schools on children’s test scores. This was all underscored last year when advocates of charter schools – which are publicly funded but independent – reacted to a high-profile study released by CREDO, a charter-friendly research unit connected to Stanford University. The study showed great variation between different charter schools and between states.

The CREDO study also offered an overall national estimate showing students in charter schools scoring approximately 0.01 standard deviations higher on reading tests and 0.005 standard deviations lower on math tests than their peers in conventional public schools (the former being statistically significant; the latter not). These findings were, in fact, highly consistent with an overall body of researchconcluding that the test-score outcomes of charter schools and public schools are almost identical.

Notwithstanding these pedestrian results as well as some limitations and problems with the study itself, the results were seized upon by advocates and resulted in headlines such as, Stanford Study Says Charter School Children Outperform and Study: Minority, poor students gain from charters.

Bangs for the buck

Even more recently, the staunchly pro-charter Department of Education Reform at the University of Arkansas attempted to shift the discussion to issues of productivity, arguing that charter outcomes look more impressive when we take inputs such as resources into account. In a report called The Productivity of Public Charter Schools they put forward the contention that charter funding per child is relatively low compared to regular public schools, so the charter sector is more productive.

This report generated headlines such as: A bigger bang for school bucks and Charter schools: More bang for the buck. But huge problems undermined the Arkansas report’s figures for both financial inputs into schools and outputs such as test scores.

This brings us right back to the speeding ticket. The lack of controlled laboratory conditions for this kind of research overwhelms attempts to isolate small differences that might be inherent in charter versus non-charter status. Since many school districts, for example, are called upon to provide services such as transportation to charter schools, any attempt to measure the resources put into the charter system in those jurisdictions should assign a cost to the school districts and a benefit to the charter schools. Yet the Arkansas report does the opposite.

Sensible estimates on school outputs, such as analyses of test scores, would attempt to compare like with like – comparing schools with similar demographics like poverty (measured by free lunch eligibility) or special-needs status – but the Arkansas report fails to do that too.

Even analyses that do try to control for such variables come up short, given that part of the variance is undoubtedly due to non-random but unmeasured factors – such as parental differences linked to motivation for seeking out a charter school.

More variation between charters

Some researchers have attempted to get around these bias issues by studying charter schools that are over-enrolled and that use lotteries to select which students to admit. But these studies suffer from at least two limitations.

First, they only allow for studies of over-enrolled, popular charter schools, thus undermining the ability of the research findings to be generalized to the larger population that includes less popular charter schools.

Second, parents tend to wreak havoc on the control groups in such studies by continuing to seek out other options, whether they be other charters, selective public schools, or even private schools. In other words, it’s difficult to get a clear picture.

In any case, while researchers aspire to make sweeping conclusions that compare the charter sector to the traditional public school sector, the variation among schools of the same type swamps any differences between charter and public schools that we might be able to tease out.

This shouldn’t come as a surprise. Students learn when they have opportunities to learn, and the quality and quantity of those opportunities vary greatly among different charter schools and among different traditional public schools. Our own research at the National Education Policy Center suggests very large resource differences among charter schools. This is in accord with longstanding research showing such differences among traditional public schools.

Feeding research into policy

None of this means research cannot help us understand charter schools. The fact that the studies that do exist find little overall difference between charters and non-charters is itself important. Even with the limitations of any given study, this finding has consistently held up. The entirely predictable finding of variations within the charter sector also helpfully points us to more useful areas of inquiry. For example, how do well-resourced charters use those extra resources to extend or enrich learning time?

Advocates use research as a tool, and charter school research is inextricably tied to charter school policy. For this reason and others, we can expect that the heated debate will continue even while more research continues. In the meantime, those in charge of education policy need to make decisions about whether to encourage the expansion (or contraction) of charter schools, whether they should be funded at higher (or lower) levels, and whether they should be subjected to regulation (or further deregulation). Keep a close eye on the speedometer.

The National Education Policy Center (NEPC), which Kevin Welner directs, has received a grant from The Atlantic Philanthropies to study issues of access to charter schools. NEPC funding from the Ford Foundation and the Great Lakes Center for Education Research and Practice has also at times been used to commission work related to charter schools.