July 22, 2006

Cut scores and democracy

This evening, I've been trying to boil down the controversy over cut scores (jump-started by Gene Glass in 1978, in the first wave of minimum-competency tests), and I've been trying to pay close attention to the counter-arguments. The one that pulls at me most is one advanced separately by Michael Scriven and James Popham in the same issue of the Journal of Educational Measurement where Glass lambasted the techniques used for standard-setting: the use of arbitrary cut-scores is justified not by their technical merits but by their use to improve education.

I'm not sure that's exactly fair as a characterization of their common position—Popham was saying that the possibility of a defensible cut-score definition should be magnified by the potential loss of educational benefits from going without (in the case of minimum competency tests), and Scriven was doing his Scriven-ish (or maybe Scriven-er) thing of discussing alternatives in the case of admissions tests (i.e., that the predictive validity of the underlying measure was minimally sufficient to make decisions).

On the other hand, there was a common assumption behind both remarks—the case for the use of cut scores depends on consequences. If the use of a cut score improves education, then the process used to set the cut score does not need to meet high professional standards. I have heard the same argument from many others. Why quibble with test scores, if it gives a kick in the pants to schools? These are advocates of a scarlet-letter policy, to shame schools into improvement.

There are a number of problems with this kick-in-the-pants approach to accountability and the glossing over technical problems. Fundamentally, it sees the ranking as the purpose of testing, completely unconnected to information about student performance. Using the ends to justify standardized tests and cut scores for high-stakes purposes displays a profound distrust of democratic processes. If this argument is correct—that the pressure applied to schools is more important than the basis for such pressure—then we can only reform a critical government institution through deception, deception practiced by an expert profession.

This is a chilling thought.

Listen to this article
Posted in Accountability Frankenstein on July 22, 2006 12:29 AM |