On May 10, 2016, several members of TASA (Texas Association of School Administrators) sent a letter to Commissioner of Education Mike Morath relating the many concerns they have over several problems experienced across the state in all areas of testing. They sent the letter after Morath responded to news reports about the testing irregularities saying that this was “unacceptable.” Before ETS (Educational Testing Services), the New Jersey-based nonprofit, received the $280 million contract for STAAR, testing in Texas had been handled by Pearson Education. Pearson, ETS, and TEA worked together this year as the testing administration moved to the new vendor.
“All three parties agreed to the plan and reviewed milestones together as it was carried out. Pearson met every commitment laid out in this plan,” Pearson spokeswoman Laura Howe said.
“The high-stakes nature of standardized testing requires that the state ensure, at a minimum, that assessments are valid and reliable, that appropriate testing procedures are in place and testing materials are secure, that those grading the assessments are qualified and available, and that student data is secure,” states the TASA letter.
Some of the problems addressed in the letter concerning data validity and security include:
- Some districts received test results for students not enrolled in their districts and did not receive results for students in their own districts.
- There were too many scores of “0” for short answer questions, and the justifications for those scores were not acceptable.
- ETS staff was unaware that some districts received extra answer booklets — an indication that appropriate inventory control measures were lacking.
- Demographic information and names were incorrect on the pre-coded answer documents.
- Students’ STAAR-A Algebra I end-of-course (EOC) exams were scored as STAAR EOCs.
- Scores had not been corrected weeks later.
- Student results were missing from the December and March campus reports. Many of the students with missing results from the December administration had to retest in March as it was still unknown if they had passed the previous test.
- District personnel were told by ETS to plan on retesting when grades 5 and 8 results were missing and ETS could not guarantee the student results would be found before the retest.
- Students were concerned with online testing glitches so they took extra time to complete their answers. Students marked answers and wrote essays and then had to resubmit lost work.
- Responses show concern that the data might not be valid. For example, did the system record the first essay a student wrote that was lost? Or, was the second essay the student wrote the essay of record? Neither ETS nor TEA could confirm.
- Student data files and scores were missing or incorrect. Some districts received multiple scoring documents for the same student with different scores.
Other problems were outlined in the areas of online testing, communications, and shipping of materials. Some of the problems were:
- STAAR-A, STAAR-L, and STAAR student responses were lost, erased, or disappeared.
- Often the system indicated that a student was still “actively” taking the test even after the student had completed the test and submitted his/her responses. Students were logged out of the test and unable to regain access.
- Individual student test scores were missing altogether from roster.
- STAAR-A online accommodations did not work well or did not work at all (e.g., text-to-speech tool, pencil tool, highlighter tool, and graphing tool). In addition, the oral administration feature was “too fast, would fade in and out, mispronounced words, and had speech boxes that disappeared.” Or, as another response indicated, “the computerized voice reads the wrong words (e.g., will substitute a or the etc.).”
- Districts received conflicting information. ETS would advise one way, and TEA would advise another.
- Tier 1 responders were unable to answer many questions after district staff was placed on hold for extended time periods. Then they would be forwarded to Tier 2 and sometimes Tier 3 responders.
- ETS would say they would call back with answers and never did.
- Test booklets were shipped to the incorrect address, and in many cases, to the wrong school district or campus. Campus testing materials in some cases were mixed with materials for another campus, and materials were not properly labeled for campus distribution.
- Grave concerns were expressed that high school students who are dependent on the test results to graduate were completely let down by the system due to scoring errors. Some were failed that actually passed, and the May results will not arrive until after graduation.
The concerns outlined in the letter are added to the recent move by the Texas Education Agency (TEA) to contract with Human Resources Research Organization (HumRRO) to provide an independent evaluation of the validity and reliability of the State of Texas Assessments of Academic Readiness (STAAR) scores, including grades 3-8 reading and mathematics, grades 4 and 7 writing, grades 5 and 8 science, and grade 8 social studies. The independent evaluation is required as per HB 743, which states that before an assessment may be administered, “the assessment instrument must, on the basis of empirical evidence, be determined to be valid and reliable by an entity that is independent of the agency and of any other entity that developed the assessment instrument.”
The HumRRO report concluded that their “independent evaluation finds support for the validity and reliability of the 2016 STAAR scores.” Following are the findings as to the percent of test items that are fully aligned to the expectations for the grade and subject listed:
- Grade 5 Mathematics: 98.5%
- Grade 8 Mathematics: 97.8%
- Grade 5 Reading: 88.6%
- Grade 8 Reading: 96.6%
- Grade 4 Writing: 93.4%
- Grade 7 Writing: 88.7%