November 6, 2006

On first reactions to anonymous diploma registration proposal

I'm in the Minneapolis-St. Paul airport right now, waiting for my flight.  (Pay no attention to the date and time this gets posted.  I really am writing this entry in the airport and just posting it later, since I'm not willing to pay $8 for internet access for an hour.) Responses to the anonymous diploma registration proposal in the full entry...


Online responses

The immediate reactions to my anonymous diploma registration proposal (or the higher-ed version of it) are available at the bottom of my Inside Higher Ed column.

Existing databases?

One of note comes from Daniel R. Boehmer, president of the National Student Clearinghouse. He wrote:

There already exists a database of college enrollment and degrees—the National Student Clearinghouse. The Clearinghouse has enrollment data on 91% of the nation's students and 80% of all degrees granted in the nation. This database is open to researchers wishing to know the subsequent enrollment and degree attainment of their alumni, "drop-outs", and admits that declined to enroll. College Registrars voluntarily populate the database by providing frequent data updates.

I had thought of the clearinghouse as a diploma verification service that institutions can use for individual students, but I'm delighted that Boehmer says researchers can use it. I'll have to look into that. If it checks out, the higher-ed part of what I propose may not be necessary, in terms of a federal (as opposed to national) database.

Financial aid

The other reactions I saw before my internet access was cut off, from Jonathan Dresner and Tod Massa, was about my argument that a diploma registration system shouldn't have financial-aid data.  Dresner writes:

[T]he pressure to create the "unit-records" database is... coming from politicians and administrators who want precisely the information that Mr. Dorn doesn't want to share—financial aid and grades. It's concern about grade inflation and the possible abuse of financial aid—both of which are real, though desparately overhyped, issues—that make graduation rates relevant.

I'm glad Dresner and Massa raise this point, because this is precisely the type of question that an anonymous registration system (or the National Student Clearinghouse, if it's open to researchers) would be able to answer in general but not about specific students. Suppose you wanted to know in general about the relationship between different types of financial aid structures in community colleges and B.A. completion rates.  One could probably negotiate a research project with several community colleges to get the distribution of financial aid among students and then, after classifying the financial aid and agreeing on a matching procedure (propensity-score would probably be the easiest), get birthdates of students in the various categories of financial aid. With that batch of birthdates from each institution, it's a relatively simple matter to estimate the number of students who eventually graduated.

If the point is to identify fraud, that's already illegal and I'm sure there are structures in place to look at potential patterns of abuse. I doubt that a unit-records database is superior to the financial-aid databases that already exist. And if the point is just to snoop on private institutions' patterns of aid, it seems absolutely ludicrous to build a multi-billion-dollar system just to be petty about a handful of private colleges and universities.

And about grade inflation... doesn't it seem silly to invade student privacy for this purpose, when national longitudinal panels can answer the question much better? For individual institutions, state legislators mandate all sorts of reports from public institutions, and relevant questions such as this one can be easily answered for institutions that cover the vast majority of students.

In general, the legitimate question driving this is to track the completion of students who move among different institutions, and I suppose one could treat the reaction to this proposal as a Rorschach test. I suppose some advocates of the unit-records database will not be satisfied until they can go on fishing expeditions for embarrassing information about private colleges and universities If this proposal sniffs out the illegitimate intentions for a unit-records database (all of which constitute a waste of taxpayer's money), I won't mind.

"You College of Ed Folks are Incompetent"

No, I don't think (or can't quite be sure) that's the point Glen McGhee of the Florida Higher Education Accountability Project made when he wrote,

If, as Sherman contends, "Florida and some other states have extensive experience with unit records..." where are the mountains of research and studies and discoveries that this should be producing?
Rather than blame this lack on the fact that "very few researchers use the data [and] data sets are complicated [and] the expertise needed to understand and work with the structures is specialized," shouldn't we be pointing the finger at the CoEs? Why aren't these brain-trusts being represented at Florida's Articulation Coordinating Committee meetings, and giving their input?

I'm not sure whether CoEs refers to Colleges of Education and perhaps me personally. Apart from the fact that I've never been invited to these meetings...? More seriously, there can be more research conducted, but it requires collaboration between Institutional Research staff who have expertise in working with the data structures and faculty who have specific attainment issues in their research agenda (usually established in graduate school, when almost all of them were outside Florida). From talking a few times with IR folks, I suspect most would welcome such collaborative efforts as long as the resources/time existed. I don't know if there are any faculty with that interest in Florida, but there are nationally. Often, these networking connections are idiosyncratic, with researchers not knowing about the availability of data. (My personal research agenda at the moment is not about higher ed attainment specifically, incidentally. But I think those of us working mostly on K-12 research and mostly on higher ed research should be able to talk to each other!) McGhee also wrote,

There is, I believe, as massive disconnect between the researchers on the ground, and the policy makers at the very top.

Agreed.  I think that's true no matter where you define researcher. But I know there is plenty of work ongoing with IR staff, so I include them in the term researcher. Those of us who are faculty do have an obligation to try to reach those policymakers.  Maybe writing that column and writing more in this blog (and elsewhere) can be part of that connection effort...

Responses in the seminar

I was a bit surprised that I only had a few questions and comments in the seminar itself.  That may have been my fault, since I spoke for about 45-50 minutes when I had intended to only go 35-40.  Or maybe I spoke too quickly for attendees to follow. Or maybe what I said was just Too Darned Obvious.  (That last possibility is not bad, in itself. If my various audiences think of what I say as obvious, but no one else has said it before, I've still taught something.) Here are the themes I remember right now...

"How do I calculate a graduation rate?"

One attendee was a school official in the St. Paul schools, and another was from Hennepin County schools (or at least the county name was written on her lanyard). In both cases, they've been struggling with measurements of graduation rates, and I hope I made it clear that the fault lay not in themselves but in the institutional basis for the various rates that there isn't a simple method. In St. Paul, for example, a number of recent immigrants enter schools as teenagers but a grade or more behind their same-age peers, and they graduate a little later. Or a lot later: St. Paul divides students into graduated, left school, and continuing in school. The staff member wondered about how to measure graduation under those circumstances, and my brief answer was that it's easier to look at things by age and adjust for a few things, and that transfers/migration is the tougher issue. (If you're the official from St. Paul, could you please e-mail me? I had wanted to talk with you after the seminar about how to use the age-based data, but you left before I could get to you!) Maybe I should offer a seminar at the 2008 AERA on calculating graduation rates using age-based data. First, I'd need to work with some local school officials to make sure my ideas are practical on the local level.

"Americans are too hung up on privacy"

I'm not the only person who stayed on in Minneapolis after the end of the Social Science History Association meeting. The members of the coordinating group for the North Atlantic Population Project are conducting meetings early this week, and I had a very nice dinner with them and a few Minnesota Population Center staff members last night. A few stayed through their lunch to listen to me, and one came up to me and said he was astounded that we think it's possible to have a modern society without individual tracking.

His point was more subtle than the heading I've put up there. He pointed out that our Social Security numbers are regularly abused, and that all sorts of marketing databases have information about us that are bought, sold, folded, spindled, and mutilated. "Bad data is a worse invasion of privacy than good data," he said, and he may have a point. Ever been a victim of identity theft?

I don't think that we can assume, however, that the creation of accurate individual-level databases will mean the end of inaccurate invasions of privacy. The marketing-database clock cannot run backwards (or the genie is out of the bottle: pick your metaphor). And I suspect that many of those around the world (not just in our idiosyncratic republic) would disagree with my thoughtful colleague about modernity as an inevitable cause of encroachments on privacy.

Limitations on research using an anonymous database

A few comments suggested that what I proposed will not get at individual experiences or at more complex questions, and I absolutely agreed. An anonymous diploma registration system is not meant for individually-based, multi-level analysis. But the systems proposed for longitudinal analysis aren't great for that, either, or at least there are substantial tradeoffs.

Listen to this article
Posted in Education policy on November 6, 2006 11:28 PM |