NZBR Blog

I miss Roger Kerr's blog.

The New Zealand Business Roundtable has resumed blogging, though, at Policy Matters.

Roger Partridge points to some of the NZBR's resources on school choice and to a 2006 report by Caroline Hoxby for the New Zealand Education Forum.

Great to see the NZBR back to blogging; I've added Policy Matters to my RSS feed.

School quality and crime

David Deming argues in the Quarterly Journal of Economics (HT: @Crime_Economist) that high risk kids experienced lower likelihood of later criminal activity if they won random lotteries allowing them to attend their first choice public school.

How does it work? High-risk youths spread out over more schools rather than being concentrated in a few bad schools. If peer effects work with thresholds or if there are increasingly negative peer effects as concentrations of high-risk kids increase, this makes a lot of sense.

Lessons for us in New Zealand?

• The lottery design in North Carolina resulted in higher-risk youths being more, not less, likely to flip into competitive schools. This cuts against arguments that school choice and charters necessarily result in pools of worse kids left behind. 
• The gains from moving to charters here might be rather more extensive than just educational outcomes. This is also consistent with evidence from things like the Perry Preschool project where educational outcomes washed out over time but social outcomes, like crime reduction, persisted.

Here's an ungated working paper version.

For and against charters

It being useful to develop the best possible argument against a position you support, here's my best case against New Zealand moving towards the use of charter schools. I'm not convinced by this case. Or, at least I don't think I am. Let's see if I've convinced myself by the end of the argument.

- Begin Advocatus Diaboli -

First, we have to recognize that the New Zealand school system is, on the whole, pretty good as compared to the American school system. Not always and everywhere, but on average. That isn't to say that we can't have improvements. But we have to keep in mind that a lot of the gains found in charter schools in the U.S. are measured against a counterfactual of abysmal urban public schools that compare poorly with some of Dante's circles of hell. Here's a fine description of one such nightmare case. That the enthusiastic young teacher there was able to flip over to a charter school, along with some of the students, produces real gains, both for the students who flip, and for the average quality of those engaged in teaching. The worst US public schools drive the best teachers out of teaching; the Charters give them an option where those who'd rock the boat against horrible school principals and stultifying union rules can flourish.

But are any New Zealand public schools really that bad? If the worst New Zealand public schools don't come within cooie of those American schools, we'd be overestimating the potential gains here from adopting charters. Sure, we can find a few isolated cases where the school principal condoned or covered up horrible bullying. But the worst here just doesn't match the worst there (though I invite readers to show me otherwise).

Further, it's really hard to pin down the gains from American charter schools. Sure, those quasi-experimental studies I cited yesterday are nice. But there's a meta-selection issue: the only schools that use lottery for entry are the really successful charter schools that are oversubscribed. You can identify the gains to the students who make it from the public schools into those schools by that method, but you shouldn't extrapolate from those gains to the average potential gains from widespread use of charters. Then, you have to go to some of the estimates of the overall effects of charter schools, and the empirics there are, frankly, a mess. Selection issues are all over the place. Where do new charters open? Where the worst public schools are as that's where parents want to get their kids out. Which kids flip? Those whose parents care more. Taking average cohort performance across public and charter schools doesn't work if charters disproportionately pick up the better kids from the worst areas, but it's not immediately clear whether that helps or hurts charters' performance stats. And so we find all kinds of studies showing little effect in the aggregate, or strong heterogeneity in results across different charters.

Finally, many of the gains from charters are already available in New Zealand in the form of Integrated Schools - state-funded schools established for a special purpose or with a special mission, but that are restricted to cover the national curriculum. Those aren't impossible to open. What's then gained by allowing entry to those who would perhaps eschew the national curriculum in favour of weird stuff? Yes, it's possible to improve on NCEA, and the best charters will. And, sure, there are already religious integrated schools using public funds to provide a religious message. Brian Tamaki's Destiny Church even runs one. But do we really want Tamaki released from requirements to nominally adhere to the national curriculum?

In the best case, charter schools open in neighbourhoods where zoning keeps poor kids stuck in underperforming schools, parents send their kids there, and public schools are forced to lift their game or suffer enrolment drops, funding drops, and downsizing or closure. And so everything's great. But the more likely case, depending entirely on how the regulations are set up, is that those charters pull the better students from the worst schools. Those better students do fare better in the charter schools. But those left behind fare worse as the public schools remain the residual claimant on the worst cases - the disruptive students that the public school system, for better or worse, cannot help but keep integrated into mainstream classrooms where they ruin outcomes for others. Those students become a greater proportion of those left in the public schools, further depressing outcomes for decent students whose parents just don't care enough to send them over to the charter school; cohort effects matter. Those best students would have made it through regardless, but we worsen outcomes for the middling students of indifferent parents.

- End Advocatus Diaboli -

What do I take from this? Upside gains here, relative to those in the States, are likely to be limited. And, as Seamus noted, it can be tough to extrapolate from US studies to here. But particularly bad outcomes seem unlikely unless we make strong complementary assumptions about parental indifference / parental stupidity and about public school inflexibility in response to the competitive threat.

There will be here, as there are in the States, some charter schools that fail. But even the failures let us learn. Roland Fryer dug into the correlates of charter school success (Freakonomics summary here) and found results a bit divergent from usual teachers' union requests. Input-based solutions like smaller class sizes and higher proportions of staff with teaching certification don't do much. Instead, "frequent teacher feedback, data driven instruction, high-dosage tutoring, increased instructional time, and a relentless focus on academic achievement" explains most differences in outcomes across charter schools. Perhaps it's possible to mandate that state-sector schools here simply adopt best practice. But I worry that that's best-case thinking; the government barely seems able to compel its public schools to disclose student achievement statistics. The threat of student exit seems the best long-term discipline for poorly performing schools. And, as Seamus noted, there's a lot of value in the experimentation to see what methods work here. I find it very plausible that charter schools drawing on tikanga will find more appropriate instruction methods for lower decile Maori groups, and that other approaches will be successful in poorer Pacific Island communities. But we won't know until we try.

More on Charter Schools

I have no doubt that John Pagani is wrong about the charter schools proposal. My fear is that Banks and Key are making the same mistake.

Pagani starts off with similar criticisms to Dim-Post linked to by Eric yesterday: Charter schools will cherry pick the best students and then use raw achievement results to trumpet their success; they will charge top-up fees to provide be able to afford better resourced schools, etc. He may be right, he may not be; the devil is in the details that have not been worked out yet. I hope he is wrong, but I will avoid cheering the policy before the details are known. But this is not why I say that Pagani is clearly wrong. It is this comment that misses the point:

The problem is this: If there is something that a charter school can do better than a state school, why don't we make the state school do it?

This is where I fear that Key and Banks are making the same mistake. The point is not that there is clear evidence that there is an alternative way of doing things that are better and that charter schools will deliver on this. Eric has pointed to useful U.S. studies, but we don't know anything in a New Zealand context. We simply don't know what will work, and experimenting with different approaches will give us useful information. So here is my plea to all the parties involved:

1. Please, all read Tim Harford's Adapt.
2. Involve competent social scientists in the design of the policy, with a view to making sure that useful data is available by which different schools and the system as a whole can be judged. This will involve both making sure that there is some random assignment of students to schools, and also making sure that similar data is collected in areas where the system isn't trialed to tease out whether any differences in outcome between charter and state schools is because of an effect on both with no change in the aggregate. 
3. To the Government: make it your stated policy that this is a trial and that you are not committed to the model if it doesn't seem to work;
4. To the Opposition: make it your stated policy that you accept that this is a trial and that, while you have doubts, you are interested in investigating anything that might help our education system, and will keep an open mind while waiting to see outcome data.

Charter Schools

The coalition agreement between ACT and National has provision for trial charter schools in South Auckland and Christchurch:

With respect to education, the parties have, in particular, agreed to implement a system, enabled under either sections 155 (Kura Kaupapa Maori) or 156 (Designated character schools), or another section if appropriate, of the Education Act, whereby school charters can be allocated in areas where educational underachievement is most entrenched. A series of charters would initially be allocated in areas such as South Auckland and Christchurch. Iwi, private and community (including Pacific Island) groups and existing educational providers would compete to operate a local school or start up a new one. Schools would be externally accountable and have a clearly-defined, ambitious mission. Public funding would continue to be on a per-child basis. (Details are included in the attached Annex)

So, what evidence do we have on the performance of charter schools in the States?

First, a note of caution in all the American studies - charter schools operate under regulatory constraints and vehement opposition from American teachers unions, both of which may affect observed results. Further, suppose a charter school opens in a neighbourhood and you find higher test scores among the students who go there compared to those who stay at the public school. Are the results due to differences in school quality, or selection effects? Two kinds of selection effects will be relevant - schools may try to get the best students, and high quality parents may be more likely to flip their kids out of failing schools. DimPost seems to think everything's due to selection; it would have been more helpful had he hit Google Scholar.

There are ways of controlling for selection effects. Here's Josh Angrist and coauthors from 2009, HT: Marginal Revolution. They used a lottery as random assortment mechanism: Boston charter schools were oversubscribed, so a lottery was used to determine which kids got in. So Angrist can compare those who won lotto with those who didn't. Remember, this is Josh-freaking-Angrist. The econometrics here aren't going to be wrong.

Charter schools are publicly funded but operate outside the regulatory framework and collective bargaining agreements characteristic of traditional public schools. In return for this freedom, charter schools are subject to heightened accountability. This paper estimates the impact of charter school attendance on student achievement using data from Boston, where charter schools enroll a growing share of students. We also evaluate an alternative to the charter model, Boston's pilot schools. These schools have some of the independence of charter schools, but operate within the school district, face little risk of closure, and are covered by many of same collective bargaining provisions as traditional public schools. Estimates using student assignment lotteries show large and significant test score gains for charter lottery winners in middle and high school. In contrast, lottery-based estimates for pilot schools are small and mostly insignificant. The large positive lottery-based estimates for charter schools are similar to estimates constructed using statistical controls in the same sample, but larger than those using statistical controls in a wider sample of schools. The latter are still substantial, however. The estimates for pilot schools are smaller and more variable than those for charters, with some significant negative effects.

Upshot: charter schools do better, and because they're outside of collective bargaining and because they face the risk of shutting down if parents don't choose to send their kids there.

Other evidence?

• In another randomized trial, Caroline Hoxby finds substantial gains for students winning entry to charter schools in New York. Here's an ungated version of the paper. Here's the report.
• New Orleans moved heavily towards charter schools after Katrina; outcomes improved.

Here's Caroline Hoxby explaining her work.

Caroline Hoxby: The Promise and Performance of Charter Schools from The Hoover Institution on FORA.tv

I'd not be surprised if charter schools weren't already in National's plans, but it's still a nice win for ACT.

And it's great fun to watch all those who rallied for MMP now whining about post-election coalition deals. You guys should have ticked the box for FPP.

When you go out to do your own literature search, be sure to upweight results from studies that control for selection effects either by this kind of randomized lottery treatment, or by instrumental variable approaches (like this one). Put less weight on studies that just compare charter and regular schools without addressing selection issues.