As a product of viewing information from Ohio’s Department of Education, while also reviewing its just released “draft rankings of Ohio school districts,” another standardized testing enigma was highlighted.
As Dr. Diane Ravitch, referenced repetitively in SQUINTS, has been the most professional and eloquent source of critique of the shortfalls of NCLB’s standardized testing, the observations prompted the following post to Dr. Ravitch. Appended in its entirety, the letter explains in detail the standardized test dysfunction uncovered.
Ravitch Letter
TO: Dr. Diane Ravitch
DATE: November 27, 2011
SUBJ: Standardized Test Dysfunction
Dr. Ravitch,
You may wish to add this tale (still short an ending) to your repertoire of K-12 standardized test debacles.
The tale starts with an upside, the appointment of a new state superintendent -- Stan Heffner -- in Ohio. In the months after his change in status from acting to appointed, and through the early months of 2011, the Department issued a series of policy papers on K-12 reform that set for Ohio a new record for education sanity and common sense, departing the trajectory only where standardized testing is involved. One proposal is linked here.
However, Ohio's Republican Legislature created and passed Ohio FY2011 H.B. 153, mandating that the Ohio Department of Education (ODE) produce and publish for each Ohio school district a Performance Index (PI), ranking all school districts on the basis of a weighted composite of the Ohio standardized (NCLB) achievement tests and Ohio's graduation test (OGT). The OGT is roughly equivalent in form and rigor to the referenced achievement tests.
Parenthetically, the bill also required that ODE post along with the PI an expenditure number, the total school expenditures per pupil. Notably, no discrimination was requested for how those dollars were allocated, between those going specifically to the classroom versus administration and other non-instruction functions. As myopic as this latter specification, it becomes almost a minor issue.
The draft rankings of Ohio's schools were just published, raising some eyebrows because of the rankings cited. More interesting, however, is a statement that was part of the earlier and above referenced policy publications:
"In the current system, a school can be recognized as Excellent with Distinction while having nearly one in five students fail. Ninety excellent rated districts had ACT scores below the state average. One excellent rated district had a college remediation rate of 81%. Sixty-five excellent rated school districts had negative value added scores. Clearly, excellence doesn’t mean high student results in Ohio."
Perplexing, the proposal for a remedy to district letter grades or a qualitative descriptor as above, both based on standardized tests, was to rank all Ohio systems, but using -- hold your applause and grab the arm of your chair -- the same standardized test and OGT scores that determined the former assessments.
The translation of the ACT issue is that approximately 27 percent of the districts with an "excellent" rating or better -- systems in the 65th to 100th percentile based on the PI calculation -- obviously didn't exceed the State's ACT average or district averages' 50th percentile if mean and median were comparable. It is unknown whether the logical sequel was checked, calculating the proportion of systems in the 65th to 100th percentile of ACT scores that were in the comparable percentile range based on PI scores; it was never reported.
If the scores are reasonably normally distributed, the implication of the report is that as much as two-thirds of the districts in the 65th to 100th percentile on PI scores could have ACT scores below the 65th percentile of the ACT distribution, challenging the credibility of any Ohio assessment based on standardized test scores.
There are five hypotheses that might explain discrepant Ohio district PI ratings versus ACT average scores: One, that the ACT is a flawed test -- given its service over decades, reliance on it by higher education, national representation, and prior oversight, not highly likely; two, because there is a negative correlation between ACT average scores and ACT participation, the discrepancy is an artifact of high ACT participation -- possible, but needs to be tested because it seems unlikely to account for the magnitude of the discrepancy; three, that pedagogy and learning in grades 1-8 are distorted by Ohio standardized testing, subsequently impacting 9-12 performance unless offset by different 9-12 pedagogy; four, that the grade 1-8 negative effects of hypothesis three attenuate with time, but grades 9-12 are subject to misdirected administration and/or academically unprepared teachers -- plausible given that the conditions are actually visible in area schools; and five; that hypotheses two, three and four are jointly causal variables -- also plausible.
Given that Ohio's ODE web site, ODE’s representations to government and the public, and even material rewards or penalties for districts are so heavily invested in and impacted by standardized test results, it would seem logical that an agency would wish to explain the noted discrepancy? Counterpoint, doing so could embarrass the Legislature, or even mark it educationally uninformed, as well as undermining ODE's credibility. In other venues, those implications have produced both high level cheating and attempted cover-ups.
Related, Ohio's ODE web site does not currently report the district ACT averages and participation rates. But, materially, I was provided in a previous request comparable ACT data as public records under the prior Ohio and ODE administrations.
Invoking Ohio's open records act, I requested from ODE the most recent ACT scores; the intent to do the analysis of concomitance between the PI rankings and ACT rankings by district. Because participation rate is not uniform, the more complete analysis might be a partial regression of ACT scores on PI scores with the effect of participation held constant. Not the usual form of testing rigor for such data, but straightforward and de rigueur in virtually all multivariate analyses of any consequence.
To date ODE has neither provided the ACT data nor indicated a willingness to follow Ohio law. One inference is that the analyses described could be an embarrassment to the Ohio Legislature and/or ODE, non-response interpreted as a cover-up by ODE of public information that might undermine the validity of the standardized testing being massively employed in its tactics and reporting.
A serious negative impact: It is self-evident that Ohio districts that achieve some public approbation because of rankings based on the PI and its standardized test scores will likely resist changing any strategy or tactic, no matter how illogical or harmful to actual learning as advocated by you and other nationally recognized educators, in favor of flogging standardized test scores with the known tactics for cheating.
The jury is still out, but I suspect that unless I am willing to go to the mat -- meaning petitioning a superior or Ohio's Supreme Court for a writ of mandamus -- the ACT data by district will not materialize. As an educator for decades, it is painful to experience what can only be described as an egregious failure of Ohio ethics and concern for real learning in a profession and national endeavor that should reflect their highest achievement.
Regards,
Ron Willett
Dr. Ronald Willett, P. O. Box 81, New Bremen, OH 45869
Cell: 419-202-2044 Office: 419-977-2103
Postscript
Subject to some attempt to explain the issues defined above, Ohio’s education credibility at the State level is in question. Clearly, it is hard to assert to Ohio’s individual districts that K-12 reform is of paramount importance when the questionable logic described above is employed as the basis for judging a system’s attempts to improve learning outcomes.
Ohio’s first K-12 reform challenge should be to clean up its own education act.
No comments:
Post a Comment