October 17, 2008

Magister economicus?

A few months ago, I became a ringer in an August 19 Ed Week chat with David Figlio and Jennifer Jennings. I've known economist David Figlio for about a decade, I've respected his work on Florida and accountability, and I've wanted to see how he'd respond to an argument from the young-Turk subfield of behavioral economics. (For one taste of this approach, see Dan Ariely's comment on market fundamentalism this week.) While Figlio is very clever in thinking up eyecatching projects as well as solid substantive work, it's from a fairly standard microeconomic perspective. So I sent in a question before the chat, and it was the first one out of the chute.

Let's see how he responded:

Q: The general theory of action for NCLB and other high-stakes accountability systems appears to assume the existence of magister economicus, the theoretically rational school employee. On the other hand, critics of NCLB, Florida's systems, and others are concerned with the potential harms of irrational responses, unintended consequences such as narrowing the curriculum or teaching to the test. The critics seems closer to the mindset of behavioral economists. Is there any research currently going on to determine if teachers are magisters economici, irrational actors, or a mix (and what type of mix)?

A: I think that the evidence is becoming clearer that many of the hopes of high-stakes accountability advocates and many of the fears of high-stakes accountability critics are correct -- school administrators and teachers can and do respond to accountability pressures, at least at the margins.

A number of recent studies have shown that schools subject to greater accountability pressure tend to improve student test performance in reading and mathematics to a meaningful degree—my recent study of Florida with Cecilia Rouse, Jane Hannaway and Dan Goldhaber (working paper on the website of the National Center for the Analysis of Longitudinal Data in Education Research), for instance, suggests test score gains of one-tenth of a standard deviation in reading and math associated with a school getting an "F" grade relative to a "D" grade. We find that these test score gains persist for several years after the student leaves the affected school. Jonah Rockoff of Columbia University has a new working paper studying New York City's rollout of school grades that suggests that responses to grading pressure seem to happen immediately—grades released in November were mainfested in test score changes in the same winter/spring.

In the case of my study with Rouse, Hannaway and Goldhaber, we try to look inside the "black box" by studying a wide variety of potentially productive school responses, and it appears that Florida schools responded to accountability pressures by changing some of their instructional policies and practices, rather than "gaming the system."

The rapid and apparently productive response of school personnel to school accountability pressure suggests that educators are, at least to some degree "magisters economici," responding to the incentives associated with the system. And this makes getting the system right so important, because if schools and teachers respond quickly to incentives, the incentives had better be what society/policymakers want.

Many people raise concerns about teaching to the test, and there is certainly evidence of this—consistently, estimated effects of accountability on high-stakes tests are larger than those on low-stakes tests—though the low-stakes test results tend to be meaningful still, especially with respect to math. Harder to get a handle on is the narrowing of the curriculum to concentrate on the measured subjects; there is a lot of suggestive evidence that this is taking place to a small degree at the elementary level, though studies of the effects of accountability on performance on low-stakes subjects typically don't find that performance on these subjects suffers—but of course, those subjects are still being measured with tests. Still there is certainly the incentive to reduce focus on "low-stakes" subjects. One possible solution for those concerned about low-stakes subjects being given short shrift would be to impose requirements such as minimum time spent of instruction or portfolio reviews.

There is a lot of evidence that accountability systems can have unintended consequences that are predicted by the magister economicus model. Derek Neal and Diane Whitmore Schanzenbach at the University of Chicago note that accountability systems based on getting students above a given performance threshold tend to induce schools to focus on the kids on the "bubble." I've found that that type of system may lead schools to employ selective discipline in an apparent attempt to shape the testing pool, or even to utilize the school meals program to artificially boost student test performance by "carbo-loading" students for peak short-term brain activity. These types of unintended consequences are much more likely in accountability systems based on the "status" model of getting students above a proficiency threshold, rather than the "gains" model of evaluating schools based on how much these students gain.

But there's a tradeoff here. The more we evaluate schools based on test score gains, where gaming incentives are lower, the more the focus is taken off of poorly-performing students whom society/policymakers would like to see attain proficiency. How the system is designed is crucially important.

I was hoping that Jennings (known then only as Eduwonkette) would respond, in part because I suspected she was a sociologist (she is, an ABD at Columbia University) and because there are some very interesting critiques of the homo economicus assumption from sociology, most notably Viviana Zelizer's work on the nonfungible, social meaning of money. But it looked like the chat had a structure that didn't allow a back-and-forth discussion between Figlio and Jennings, instead being a two-person panel, with questions alternatively answered by each.

But back to the central question: to what extent are teachers and administrators people who respond to financial incentives? Figlio argues that they are, though we have to be wary of the consequences of a poorly-designed incentive system. I am not entirely convinced; while I agree that people respond to incentives, they don't necessarily do so in the way Figlio assumes (i.e., to maximize their gain). First, there is the phenomenon Zelizer noted, which is the social meaning of money. For a number of teachers in Florida, bonuses tied to the state system of assigning grade labels to schools is dirty money. That doesn't mean that teachers won't respond to the system in Florida (Cecilia Rouse, and Jane Hannaway, Dan Goldhaber, and Figlio make a pretty good argument that they do respond in ways that raise test scores). But that changed behavior may be tied to the reputational threat/promise of school grades rather than the bonuses. (Also see the Damian Betebenner review of their paper.)

Even if money doesn't mean different things to people depending on the context, there is the broader question of money in the context of other motivations. Here, behavioral economics is the tip of the iceberg; there are plenty of other nonfinancial reasons that drive people's behavior. That doesn't mean money is entirely unimportant but that it is one of many motivations. I suspect Figlio et al. would agree with me but point out that their analysis concerns the marginal effect of a change in incentive—that is, people's behavior can be driven by relatively small economic motives when that is the possible change they will attend to.

But in reality, you can't hold everything else constant. Given the resource and time constraints in the real world, you have to choose whether to try financial motivation for behavior or something else. What most arguments ignore if they favor public policies emphasizing financial assumptions is a fundamental economic concept: opportunity cost. What is the opportunity cost of trying to drive teacher (or student!) behavior by offering financial incentives? That is more than a thought experiment, and the cost is not just in tangible ways. But that discussion is for another evening.

See comments: Jennifer Imazeki takes me to task for viewing economists' work too narrowly, with some justification.

Listen to this article
Posted in Education policy on October 17, 2008 10:35 PM |