April 27, 2008

My graduation-rate regulatory comments

I finally had some time tonight to read the proposed regulatory changes for NCLB and focus on the graduation-rate piece. I decided to comment, and here's the substance of my comments on the graduation-rate definition (below the cut):

This comment focuses entirely on aspects of 34 CFR 200.19 concerning graduation rates (esp. 200.19(a)). There are several strengths of the proposed definition and two weaknesses in the proposed regulations (one in the definition of a permanent standard rate and one in the proposed transition option of the Averaged Freshmen Graduation Rate).

In addition, the proposed regulations would require additional discussion around minimum cohort and subgroup sizes (and alternatives for very small cohorts/subgroups), they also would require technical assistance around documentation of transfers, and there should be a follow-up technical study on differences in measures depending on whether a cohort is grade- or age-based.

Strengths of proposed changes to 34 CFR 200.19:

Strength 1) The move to a longitudinal rate based on ninth-grade entering cohorts uses the most accessible and publicly understandable option of several valid ways to measure graduation. (An alternative would be to use an age marker, such as students' 14th birthdays--for several reasons, using age cohorts would be superior to a grade cohort, but the technical improvement probably does not justify changing the regulations, at least until we see what differences arise.)

Strength 2) The definition of a cohort eliminates several loopholes that states have been documented to use (e.g., students who drop out to join a GED program are removed from a cohort, and students who are retained in grade are moved to a later cohort--both of which are events that would not trigger removals from the cohort under the new 200.19(a) definition). Of particular note is the definition of a confirmed transfer as requiring official documentation that the student has moved to a program that ends in a standard academic diploma.

Strength 3) The disaggregation of graduation rates (in 200.19(e)) is absolutely appropriate.

Weaknesses of proposed changes to 34 CFR 200.19:

Weakness 1) The definition of a four-year graduation measure is intended as a proxy of an "on-time" graduation rate. It is not entirely clear what the difference between graduation rates means, with a single indicator: does the fact that one school has a 70% on-time graduation rate and another school has a 73% on-time graduation rate means that the second school has higher overall graduation or that its students tend to graduate earlier... or even that they are more likely to be retained in 8th grade (something not accounted for in the draft regulation's definition)?

While there is no formal analysis I am aware of on this point, I strongly suspect that an on-time rate will be more sensitive to on-time/lateness issues than the overall level of graduation. Large, 20-30 point differences with large cohorts are going to be the result of substantial differences in the overall level of graduation, but smaller differences with large cohorts, or larger differences with smaller cohorts, might well reflect a slight delay in graduation rather than differences in overall graduation rates if one were able to look at completed-cohort experiences.

For policy reasons, I strongly advise against using such a restrictive definition with subgroups and smaller cohorts. In a cohort of 30, if all graduate but 10 graduate in their fifth year of high school, the four-year graduation rate will be 67%. Is that an accurate picture of the cohort experience if the rest graduate in the next year? Moreover, a four-year-or-bust measure gives schools no incentive under NCLB to keep students in school past the fourth year, and our national experience of the past 5 years has shown that many schools respond to the mechanical parts of NCLB in perverse ways.

A better regulation would require reporting of the four-year graduation rate but the calculation of three different measures and the ability of the state to craft an index that combines the three different measures:

a) A four-year cohort graduation rate
b) A five-year cohort graduation rate
c) A longer-term cohort graduation rate

If the regulation were to define a legitimate graduation index as "a weighted combination of the three cohort graduation rates where the four-year cohort graduation rate has no less than a 70% weight," that would serve the public interest in emphasizing on-time graduation without unduly penalizing schools that graduate high proportions of students, if some of them are not "on time." (While there are ways to model such measures with two years' worth of data using synthetic cohorts, there are a variety of ways of constructing the longer-term rates.)

Weakness 2) The Averaged Freshman Graduation Rate is an inadequate substitute for a true longitudinal rate. It is NOT true that "It has been shown to be a reliable, accurate estimate of the high school graduation rate" (Federal Register, p. 22025). Seastrom et al. 2006 (the reference used for AFGR) was written and submitted to the internal USDOE review process before the publication of John Robert Warren's 2005 article in Education Policy Analysis [Archives], which demonstrated several technical flaws in AFGR and other commonly-used graduation measures. (A note in Seastrom et al. acknowledges that it does not include or respond to Warren's analysis.)

There are two primary weaknesses in AFGR, in practicality. One is the tendency for any measure relying on aggregate data to be vulnerable to unmeasured migration/transfers. Without auditing, this first weakness is very difficult to remedy. The second is the lack of any explicit modeling to connect the average of 8th, 9th, and 10th grade enrollment in successive years to first-time 9th grade enrollment in the middle year. For a variety of reasons detailed in Warren 2005, the better alternative is to use the prior year's 8th grade enrollment as a proxy for the next year's first-time 9th grade enrollment. This does not eliminate the problem of unmeasured migration/transfers, but it is a more sound measure from a modeling perspective, and its elements are as readily available to calculate as AFGR.

Additional considerations:

1) Minimum cohort and subgroup sizes (and alternatives for very small cohorts/subgroups). In the Federal Register notice, there is no discussion of the numbers in a cohort that is a minimum to be reported for graduation rates (either as a subgroup or as a cohort size). Because graduation rates are highly vulnerable to misspecified transfers/migrations, it is especially important that there be alternative measures available for small group sizes and graduation rates. The Federal Register notice does not indicate whether the graduation rate changes come under the same averaging rule as other measures for AYP. If so, it may be necessary to issue nonregulatory documents on whether such averaging (or smoothing) has to be either weighted or unweighted. I would recommend that such changes be left open to states, since it is unlikely that either weighted or unweighted smoothing would clearly give an advantage to states in "gaming" the system. (I would recommend some simulations, though, to check.)

2) The USDOE will need to provide technical assistance around documentation of transfers. SIFA and the CCSSO EIMAC projects are currently in (what I think are) the formative stages of providing useful technical guidelines, and the requirement to confirm transfers for cohort-adjustment purposes should push those types of projects to the front burner on both funding and also USDOE support to resolve potential political issues among states and between different sectors of education (public and private).

3) The need for a follow-up technical study on differences in measures depending on whether a cohort is grade- or age-based. The proposed graduation rate definition for 200.19(a) is based on first-time-in-ninth-grade cohorts. This comports with common understandings of a high school graduation rate and with several existing measures. It is not entirely clear, however, that this is appropriate, both because of the numbers of students who are well over the age 14 when entering high school (because of grade retention below high school) and also because of the ninth-grade "bump" that has led Warren and others to recommend 8th-grade enrollment as the base number for cohorts in aggregate-data estimates.

One alternative would be to use an age rather than a grade as the starting point -- thus, one could use birth years as the natural cohort, with the students in a school/district as of their 14th birthdays as the start of the cohort (with adjustments after age 14). This age-based cohort measure would fit better with standard demographic analytical models. But it is not clear how much difference the two measures would make. For this reason, I recommend that USDOE or NIST call a workgroup together no later than 2010 or 2011 to examine the practical and theoretical differences between age- and grade-based cohorts.
For those who care about such things, the comment tracking number is 80537727. Listen to this article
Posted in on April 27, 2008 10:40 PM |