Change in scoring system perplexes schools

Educators marshaled evidence Tuesday that a grading scale change -- and not a dip in achievement -- caused reading scores to tank for juniors statewide last year.

And, they charged, the grading scale punished even students with excellent ACT scores.

The number of 11th-graders passing the reading portion of the Prairie State Achievement Exam dropped in 2007 by more than 4 percentage points to 54 percent.

The results have major implications for suburban high schools, where lagging scores have led to financial and administrative penalties under the federal No Child Left Behind law.

The Illinois test for high school juniors includes two assessments, the ACT college admissions test and the ACT Work Keys, which is designed to measure workplace skills.

Suburban school officials have speculated for months that the grading of the Work Keys exam was flawed, and state officials have hired an independent third party to investigate the matter.

The results of the third-party audit may be available as early as this month, state Superintendent Christopher Koch said.

On Tuesday, a delegation from Palatine-Schaumburg District 211 presented its case to the Illinois State Board of Education.

According to a District 211 analysis, students who answered the same number of questions correctly could receive a lower score in 2007 than in 2006.

The grading shift, according to District 211 officials, explains why significantly fewer students passed the two-part exam, even though average ACT scores were nearly identical in 2006 and 2007.

ACT officials, according to Koch, say their internal analysis shows the 2006 and 2007 versions of the Illinois test were comparable.

ACT officials do acknowledge that the number of correct answers needed for a particular score on the Work Keys can change year to year.

The number is adjusted based on the difficulty of the test questions, according to ACT.

But District 211 officials charge ACT curved the 2007 test -- deciding after the tests were administered and graded just how many correct answers are needed for a particular score.

The number of questions students need to pass should be predictable and transparent, they say.

"Why would it not be possible, even more so desirable, for ACT to create an exam where a student who marked 27 items correct in 2007 would receive the same score on Work Keys as a student who marked 27 items correct in 2006?" Thornton wrote in a letter to Koch.

A curved test would, by definition, be an inadequate assessment under the federal accountability system, which requires schools show year-to-year improvement in the number of students who pass state exams.

ACT officials did not return calls for comment for this report and have declined to release information about the grading of the Illinois exam, citing a confidentiality agreement between the state and ACT.

School officials say the lack of transparency -- particularly after numerous appeals to ACT -- is troubling, and prevents districts from using the test results to guide instruction.

"We need to resolve these problems because these tests have such high stakes," said Steve Cordogan, director of research and evaluation at Northwest Suburban High School District 214. "We can't afford to have problems in alignment from year to year, and we do need to know exactly how things are being computed."

Suburban schools have undertaken their own analyses to determine how ACT grades the Prairie State Exam.

The analyses show ACT grades the Illinois test on a sliding scale. Students who achieve high Work Keys scores can earn lower ACT scores and still pass the Illinois test. Conversely, students can post a high ACT score but still fail the Illinois test due to a low Work Keys score.

To complicate matters, Elgin Area School District U-46 data consultant Ed DeYoung said his data shows a student who fails and a student who passes the Illinois test can receive the exact same ACT score and the exact same Work Keys score.

And, year to year, the relationship between the two tests has changed, DeYoung said.

Too often, a poor Work Keys score trumps an excellent ACT score, District 211 principals told the state board.

"I don't understand how a student can score a 24 on the ACT and a 4 on Work Keys and fail, when another can score 16 on ACT and a 5 on Work Keys and meet standards," Palatine High School Principal Gary Steiger said.

The question is more than academic, the principals said, as teachers are working to prepare this year's junior class to take the same tests this spring.

"We have a very small window of time," said Hoffman Estates High School Principal Theresa Busch. "It's urgent. I need to be working wisely and efficiently to be that effective with our kids."

Superintendent Koch said Tuesday that the state continues to review the test, to determine the proper relationship between Work Keys and ACT. A federal review has indicated the state may need to adjust the test or the grading system to strike a better balance between ACT and Work Keys.