January 14, 2007

Figlio and Kenny paper on merit pay

David Figlio and Lawrence Kenny's paper Individual Teacher Incentives and Student Performance (NBER, restricted) has been reported by Ed Week and discussed by Joanne Jacobs and at NCLBlog. Howard at AFT (F. Howard Nelson?) lists a number of caveats.  Some more information (thanks to David Figlio for permission to crib from our e-mail correspondence):

  • Because the measure is from 12th grade, a logical question is about selection effects from differential dropping out. Figlio and Kenny did check for sensitivity to graduation.
  • The questions on merit pay were worded carefully to exclude bonuses for AP teachers, etc. (something that would otherwise make merit pay measures a proxy for the proportion of teachers who teach AP or do other service).
  • Something I brought to David Figlio's attention was something they hadn't considered: distribution of teachers within a school. Some administrators spend some effort to recruit teachers who work well with 9th grades. Others distribute teachers' assignments evenly. Others assign the teachers perceived to be most skilled in their content areas to 12th grade (and advanced) classes. The result is that we don't know if 12th grade teachers are selected for certain qualities, and those qualities (especially if they're seen as high-academic content specialists) may make them more responsive to individual pay incentives. (Howard at AFT noted that the measure may be a long-term indicator, but it may also reflect short-term changes. We just don't know.)

Because David Figlio is on sabbatical in England, he didn't have immediate access to the data to answer my most important question: what's the distribution of schools' mean test scores? Without that information (which isn't in the NBER working-paper version), we don't know the effect size.

I'll repeat what I've said before: discussions of compensation systems often (mis)portray the issue as a dichotomy: either incentives or no incentives. That's a oversimplification, and despite the data limitations that led to Figlio and Kenny's approach, it's a serious problem with generalizing from the research (which I'm sure people will do in abusing the Figlio and Kenny article). Again, I'm not an industrial-organizational psychologist, and if key people who are talking about merit pay know less than I do about the relevant research, there's a problem.

Listen to this article
Posted in Education policy on January 14, 2007 10:40 AM |