March 5, 2009

Sherman Dorn is called a dirty name: behaviorist

Not really, but in this week's Education Sector forum on technology and assessment, Scott Marion of the National Center for the Improvement in Educational Assessment comes close. In reading Bill Tucker's report on the subject, I had previously been concerned that it was too close to boosterism for bells-and-whistles approaches, broad claims that we can address concerns about the quality of testing if we can just get stuff online. Yes and no, I thought (as I explained February 20). So when the online discussion this week mentioned one alternative, the quick-and-dirty approach of curriculum-based measurement (CBM, also called progress monitoring), I asked a question:

This type of formative assessment is not only too-often ignored by general-ed researchers, it's also being ignored by the think-tank community. If you read the NCIEA's The Role of Interim Assessments in a Comprehensive Assessment System: A Policy Brief (recently published), you'll see that the report entirely skips between a definition of formative assessment that is entirely informal and casual to the type of periodic/benchmark assessments that are much more complicated than CBM/progress monitoring. What's necessary to turn technology-based assessment away from the bells-and-whistles assumption and devote enough attention to the "here's what we can do now that has documented research support?"

Marion responded in part as follows

Sherman sounds like a supporter of CBM, so he should probably be happy that we stayed away from doing a critical analysis of this enterprise. I have only begun to look into CBM and what I've seen makes me very nervous. These multiple "measures" are treated as if the inferences drawn from each of the measures are based upon some sort of valid equating. Everything I've seen thus far-and admittedly I need to look into this much more--suggests that inferences made about student growth and not supported by a psychometric foundation.

Hmmn... there's a broad base of research on curriculum-based measurement, and Marion blithely skips by that. Essentially, CBM suggests a regime of regular testing that samples the entire year's curriculum. Is it possible to construct alternative forms that aren't precisely equated? Absolutely. But the same standard should be applied to so-called benchmark/periodic assessments, and as far as I'm aware, the NCIEA brief referred to above doesn't raise that issue at all. The deeper technical question is whether score movement reflects real achievement change and whether lack of score movement indicates real achievement stagnation, and that's a thorny issue for almost any system of measurement you can shake a stick at. I'll let Stan Deno, Lynn Fuchs, and other researchers in the area defend themselves, but there was something in Marion's answer that just didn't sit right. (Among other things, someone who claims to be an expert on assessment who has only "just" started looking at the CBM literature? It's been around for more than 20 years.)

Then there's the label slapped on at the end:

Finally, and related to Bill's report, CBM fits smack within a fairly outdated behaviorist conception of student learning.

I've apparently become a behaviorist in my free time. Quick: someone find me a deprogrammer! If you're curious, step back and think about the decisions a teacher has to make about classroom time and his or her energies. Might it be useful to give a teacher or a principal a tool to gather information efficiently, with little time stolen from classroom instruction? At least theoretically, a consistent sampling frame for CBM could include free-response items with interesting cognitive demands. Not a problem! Well, until you get to the nuts-and-bolts of practice, when you need to write and decide on items. Then it gets interesting. There is nothing wrong with quantitatively-scored items used for screening/informational purposes as long as those tests are not taken as the only source of legitimate information on children (and I don't believe I've seen that in the CBM literature). If Debbie Meier thinks it's valuable for third-grade students to learn their times tables, I think we can safely assume that some mix of information is fine, and the behaviorist label is pretty silly.

Besides, I think we experienced silly arguments about two decades ago around the construct of "authentic" assessment, and at least a few states spent millions and millions of dollars on performance exams that were beautiful in theory but problematic in different ways from multiple-choice exams. If I remember correctly, at least one or two studies from the U. Minnesota National Center on Educational Outcomes concluded that performance exams gave children with disabilities at least as many problems as multiple-choice exams.

So let's not reinvent the wheel and cast aspersions on our least favorite formats. It's the use and not the format that matters. I don't want to see a few billion dollars spent entirely on blue-sky test development projects when a few million dollars could develop something that is practical across millions of classrooms. Let's ensure that at least some of the current money is spent on stuff that's useful today and also invest in long-term development projects.

Incidentally, I should have given credit to Charles Barone for his suggestion of open-source testing. I knew I had seen it somewhere, but the only stuff I could find for the February 20 entry was something else.

Listen to this article
Tags: assessment, curriculum-based measurement, Education Sector
Posted in Education policy on March 5, 2009 12:20 PM |