December 21, 2006

Theodore Porter, Trust in Numbers, and picking the right fights

On the way to and from my mother-in-law's house today, I finished Theodore Porter's Trust in Numbers (1995). (I should say that I finished it while my spouse was driving!) While I was distraught this morning at Porter's style, I slogged through, a matter which I knew was important. And the book has plenty of food for thought. But the (dis)organization remained problematic, and not surprisingly, the book reviews varied fairly dramatically in terms of how they read the main argument. In particular, the reviews in the Economic History Review and Technology and Culture read Porter's book as less deterministic than I thought he was in the end.


That determinism is a critical question. Is autonomy such a driving force that weak disciplines and administrative apparatuses under political threat will resort to statistics as a buffering mechanism to protect autonomy, even while higher-status disciplines or bureaucracies can still turn to networks of trust and rely on elite status? If so, then test-score accountability was inevitable, as Stephen Turner suggests. But I think the details in Porter's book belie that argument of virtual inevitability (which Porter makes clear, I think, in the second-to-last chapter). As Porter notes, weights and measures have historically been more negotiable than we assume today, and his description of the origins of the Chicago futures market is a fascinating tale of contingent events. There was nothing inevitable in it.

We don't have to look at NCLB and debates over NAEP to see how flexible truth is and the porous factual claims that permeate education. Evidence of how negotiable education "facts" are lies in the current debate over measuring graduation—or, as is more common, mismeasuring graduation. There is no agreement on how to measure graduation, the sides are frequently identified as biased in terms of other issues (support of public schools v. vouchers), and even the terms of the debate are vigorously argued, an argument that suggests that education facts are not completely behind the boundaries of expertise.

The debatability of education facts suggests another way of looking at accountability: given the fact that accountability systems will produce arguments, maybe one way of thinking about them is to structure the system so you get the argument that you want. If proponents of high-stakes accountability are sick of educators responding to accountability by blaming parents, maybe they should look in the mirror: didn't the system predictably set up that argument?  And if so, what's the argument that you want to have? 

Maybe it's because my father grew up on Flatbush Avenue, but I don't think there's anything wrong with a good argument, as long as it's about the right things. Do we really want to keep arguing about whether the scores mean something or who's responsible? I can predict continuing arguments precisely on these issues for as long as accountability is based entirely on test scores. I know of one commendable accountability mechanism—Rhode Island's site-visit system—that produces enormous discomfort in schools that are judged wanting and some arguments, but I think they're arguments worth having, about the nature of the school, what isn't happening, and what could be happening. Those arguments can only happen if you get beyond test scores.

Listen to this article
Posted in Accountability Frankenstein on December 21, 2006 10:53 PM |