August 24, 2008

On sampling error and troubling year-to-year jumps in the data

Phi Delta Kappan's Bill Bushaw responded to my entry Thursday on the sample distribution of the Kappan poll:

For clarification, the Gallup Organization ensures that the poll sample is identified through a truly random process. This means it's possible to oversample one portion of the population. In order to correct for this, the responses are matched and balanced against the U.S. Census population parameters. That balancing process ensures that the sample reflects the U.S. population. Of course, in all polling, there remains a sampling error, in our case, +/-3%, standard for a national sample.

Bushaw is partly correct: The last page of the report describes in general the reweighting of the final interviews. But there's still a lot missing and some troubling data in the results. The specifics of weighting matter, and they are not described on the last page. In addition, the oversampling of parents of school-age children is not described (how many of the 1002 interviews were with parents of public-school students? how many of those public-school-parents sample were men?), and depending on the size of individual cells, the underrepresentation in the sample may skew the results.

What worried me when skimming the questions was a set of large jumps between 2006 and 2007 and then back again in 2008, moves greater than the margin of error for either the national or parent sample. For example, in Table 10, PDK reports that in 2006, 21% of public-school parents thought that funding was the worst problem for local publi schools. In 2007, PDK reported a jump up to 26% reporting that funding was the worst problem (a change barely within the margin of error), and this year it's down to 19%. In Table 13, PDK reported that in 2006, 21% of parents thought that they'd give the nation's schools an A or B and 51% would give the nation's schools a C. In 2007, PDK reported what looks like a big shift: 16% for A or B and 57% for C (changes at the edge of the MoE). Comes 2008 and here's another shift, reversed: 22% for A and 44% for C.

That up-and-down quality for other questions (32, 41-44) made me wonder about the sample, and as followers of national political polls are aware, the population models used for weighting can skew results one way or another. With political polls, we KNOW that pollsters make certain assumptions about the proportion of the population that belongs to different parties. With the PDK poll, the weights/models are not explicit, and there's something about the jumps in the results that raise red flags for me. Maybe public and public-school parent opinion changed between the summers of 2006 and 2007 and back again in 2008. But color me skeptical, and I think PDK isn't being entirely fair with its readership in focusing on the point estimates from the poll.

Then again, I have the same complaint when there is an obsession with the point estimates from test score data. So does Eduwonkette (on test scores).

Listen to this article
Posted in Education policy on August 24, 2008 8:12 AM |