May 14, 2010

The Thursday's-child research design

This morning's St. Petersburg Times is reporting on a working paper on the Florida class size initiative by Harvard researcher Matthew Chingos. This paper is now available on the PEPG website, and the Times asked me to read it and comment, and there are a few things I can say:

  • It's stronger technically than a previous paper by Chingos and West that I commented on recently. As a former editor, I'd guess it's one thorough scrub away from being publishable at a reasonably selective journal.
  • It's based on a clever concept, comparing districts and schools that had "far to go" on class size with districts and schools that were very close to the starting mandates and framing the comparison as one of "if you give districts and schools more money, do you get more bang from your buck by mandating that they use that money to lower class size?" That's not the question on the ballot this fall, but it's a reasonable slice given the difficulties of disentangling policy effects. And if you know the "Monday's Child" rhyme, you'll understand why I'm calling this a Thursday's-child research design.
  • The paper focuses on grades 4-8, which is an important contribution to the literature because of the focus of much class-size research on primary grades. But that focus comes because grades 4-8 was the policy interval with the most variation in starting points among districts, and so the paper can't tell us about class-size effects at other grades.
  • Chingos addressed a number of potential weaknesses in reasonable fashion. There are two weaknesses remaining that struck me on first reading the paper -- the failure to address student migration (both interstate and intrastate) and the failure to consider "contamination" of the design by student experiences with primary-grade class-size reduction. The first can be addressed by some alternate samples, or even identification of students who moved from "control" districts to Thursday's-child districts. But the second is tough to address: while the majority of districts didn't have far to go on grades 4-8 starting in 2004, most of the "control" (or not-far-to-go) districts for grades 4-8 were still "far to go" districts as far as primary grades were concerned, and for intermediate-grade students in the third and later years, their test scores would have reflected experiences with class-size reduction in primary grades.
  • There are some interesting variations in tables and effects by specific grade.

As I told reporter Ron Matus yesterday, I'm sure this will be spun out of all proportion to the study because there's little solid research on Florida's class-size mandate, the stakes with the ballot measure to change the mandate are high, and the proponents of changing the mandate are evidently starting well behind the eight-ball with public opinion. So the temptation to exaggerate the findings is high. Jeff Henig would not be surprised.. and speaking of which, congrats to Henig for the AERA award for Spin Cycle! It's highly deserved.

Listen to this article
Posted in Education policy on May 14, 2010 8:47 AM |