March 22, 2009

Grokking social-science statistics

Several comments in the past few weeks have expressed some wonder that I use statistics when I am publicly skeptical of several policy-related uses of education statistics. I am a little confused by the comments (and implicit accusation of inconsistency), since many of the most articulate critics of high-stakes testing are assessment experts, but for the record, here are a few of my personal stances towards social-science statistics:

  • If for no other purpose than to engage in political debates in a conscientious and credible fashion, adults need to have some rudimentary knowledge of statistics and probability and also be able to listen to and discuss essential concepts without doing enormous violence to them. This is on the same order as needing to have some rudimentary knowledge of Newtonian motion, thermodynamics, electricity, algebra, natural selection, etc., to engage in public policy debates in a constructive fashion. Know why perpetual-motion machine patents require extraordinary (and highly improbable) evidence; know why regression to the mean invalidates many change-over-time claims when the baseline comes from a sample of outliers. 
  • If you're tempted to be proud that you don't know statistics, see what happens to the following sentence if you replace "in French" with "using statistics" and "French history" with your current interest: "Yes, I'm writing about French history; what do you mean, I need to read stuff that's written in French?"
  • One of the reasons why one needs that basic knowledge is to know the limits of statistics and be able to ask probing questions of the claims that are made in public debates. Probing questions are not of the formalist type that could be applied to any claim, "You can say what you want by picking a statistic" or "It's unethical to use statistics without talking about the metause of statistics." Probing questions engage the specific claims made in debate: "Politician Yodel says we saw a 102% increase in the incidence of Echoing Disease last year, but I want to know what the incidence was the year before so I know if this is a serious problem."
  • Though social-science statistics are inherently constructed objects, they can nonetheless be enormously useful. For a thoughtful and useful discussion of social-constructionist arguments, see Ian Hacking's The Social Construction of What? (1999). (Michael Berube and I both very much like Hacking's discussion of dolomite, though I suspect I am closer to Hacking's end view than is the Paterno Family Professor of American Airspace and Dangeral Studies.)
  • To work with social-science statistics, at least I find it tough to simultaneously criticize every character that I type in a statistics program and also work the darned program and think about what I'm doing. So I engage in a form of suspension of disbelief, work the statistics, pause and think about the larger meaning and doubts, work again, doubt, work, doubt, etc. I know I'm embedded in the statistical machinery when I hear, "Sherman, are you going to get any sleep tonight?" And I know when I've doubted enough when I realize I forget the syntax for calling up multiple regression.

And tomorrow morning, because of the idiosyncrasies of the USF IRB-02 records, I need to write and print an IRB protocol so I can finish a long-delayed project ... assuming I can climb the learning curve for the R-Project language.

Listen to this article
Tags: social construction, statistics
Posted in Research on March 22, 2009 9:38 PM |