RSS
Home
Standardized tests: We know who takes them and what they count for. But who MAKES and SCORES them?
Print
Email
RSS
Subscribe
The Florida Comprehensive Assessment Test (FCAT), the bane of the existence of schoolchildren, teachers, administrators and parents around the state, has recently been de-emphasized in assigning grades to schools (now taking into account, for high schools, other factors such as participation and success in Advanced Placement and dual-enrollment courses, and graduation rates) and signal a shift away from using a single standardized test to determine everything from whether or not a child can graduate from high school, teacher bonuses and the funding schools receive. Governor Charlie Crist promised the de-emphasis of the FCAT, the centerpiece of former Governor Jeb Bush's education policy, with full support from Democratic legislators.
Yet, even in the midst of steps to remove some of the importance of the FCAT, even as schools are no longer to devote resources to special FCAT workbooks, FCAT coaches or pre-FCAT "rallies," some counties seem to be lagging behind the curve. In Palm Beach County, the current school year marks the inauguration of a new system of "standardization" of curriculum, which includes departmentalization of elementary schools (i.e., elementary children now change classes much like middle school or high school children, and elementary teachers now teach one or two subjects all day long to different groups of children), mandatory "embedded assessments" given once every three weeks to ensure that benchmarks are being met, and an imposed scope-and-sequence, complete with ready-made lesson plans, that teachers are expected to follow to the day. These changes are being applied across the board, whether the school in question has a record of earning "A"s or "D"s.
The purported idea is to make sure that Palm Beach County students are meeting high standards. In reality, the new system has many bright youngsters confused and struggling when they never were before, due to the illogical sequence of instruction imposed on their educators, has teachers frustrated and angry that they can no longer adjust their teaching to meet the needs of their students, and has parents up in arms--so much so that the city of Boca Raton is talking about taking control of its schools from the Palm Beach County school district.
Essentially, the focus of the new curriculum is on these so-called "embedded assessments." In Miami-Dade County, we have similar tests (which went from being "interim assessments," administered about once per nine weeks, to being "biweekly assessments," to "triweekly assessments," and back to "interim assessments," apparently largely based on funds available to support these extremely expensive tests), but they currently carry less weight than those being applied in Palm Beach County.
Perhaps School Board members, and more importantly, the higher-ups in the Florida Department of Education should read Todd Farley's new book, Making the Grades.
Mr. Farley had a fifteen-year career in the standardized testing industry, working for NCS (now known as NCS-Pearson), the largest test-scoring corporation in the country, responsible for scoring high-stakes standardized tests in at least 15 states, including Florida, Texas and New York). He started out in 1994 as the lowest-level employee in the company--a temporary scorer, paid $7.75 an hour to, as he says, "slap scores on student responses as quickly as possible." Gradually he worked his way up through the ranks to work as a test writer and administrator at ETS (Educational Testing Service). He left ETS after a couple of years, mostly out of boredom, and became a private consultant, offering his "expertise" in test-writing, range-finding, scorer-training and scoring to various testing companies and departments of education for exorbitant sums. Despite having enjoyed a lucrative career in the standardized testing industry--which he is quick to point out on multiple occasions throughout the book is just that, an industry, where the first and foremost goal is to turn a profit--he wrote this book largely as a demonstration to anyone who will listen why standardized tests are anything but standardized, and why, in his experience, the scores mean next to nothing.
He gives numerous reasons for why standardized tests scores come far from proving anything about the intelligence, knowledge or even comprehension of the test-takers; in fact, that is the basis of the entire book. Those hired to score short and extended responses to standardized tests (such as can be found on the FCAT reading tests) and standardized writing tests (such as FCAT Writing) are, according to Farley, generally "unhirable" people: overworked, underpaid, most with absolutely no experience in education in any form, and most with no qualifications at all other than a college degree (the only prerequisite for employment). Some even have limited English, and misscored responses because they did not understand the words the student used in the response. The scorers--all hired as temps, with relatively low hourly pay and no benefits--are trained to score responses by the provided rubrics with "Anchor Papers," or student responses selected because they conform neatly to the outlines established in the rubrics. They then must pass a "qualifying test," in which they score 70% of the training papers correctly (that is, in agreement with the official scores). However, Farley said that that 70% qualifying mark was more often than not ignored or waived, due to the need of scorers.
"Quality assurance" means that a certain number of responses are scored by two different individuals, and if their scores do not agree, they must be scored by a third individual. In reality, according to Farley, supervisors (of which he was one for a period) simply change the troublesome scores (without even reading the actual student response) to make their "reliability statistics" come out correct. They are operating with huge numbers of responses (often tens of thousands) and very limited time. In fact, he says that they would often score 250 responses an hour, and after a while would not even really read the responses; rather, they would scan them for certain words or phrases, and base the score on that.
Besides the sheer ineptitude of a great many of the scorers, Farley highlights the vagueness of rubrics used, how open to interpretation they are, and how they fail to predict the wide variety of responses they may elicit. Sometimes, halfway through a project, a supervisor would decide to give credit for something he had previously told his scorers to deny credit for, so that the second half of the students scored would get the benefit of credit where others putting the same answer had not gotten credit.
The writing tests were even worse. He was forced to give low scores to excellent samples of student writing, simply because they did not conform to the five-paragraph format established in the rubric, while giving passing scores to substandard, formulaic essays--simply because they were in the preferred five-paragraph format. (FCAT Writing scores are not required for graduation, but they do factor into the grades assigned to schools by the state, which can consequently affect school funding and even forced reassignment of teachers and principals.)
Farley expresses his disgust with standardized testing thus:
From my first day in standardized testing until my last day, I have worked in a business seemingly more concerned with getting scores put on student responses than getting meaningful scores put on them, a reality that can't be too surprising given the massive scope of the assessment industry and the limited time available to score those tests. Consider if there are 20 short-answer/essay questions on each of the 60 million tests mentioned earlier. That means there would be 1.2 billion student responses that would need to be read and scored within the same two- or three-month time frame. (...)
I don't believe the results of standardized testing because most of the major players in the industry are for-profit enterprises that--even if they do have the word education in their names--are pretty clearly in the business as much to make big bucks as to make good tests. (...) Because the testing company was a for-profit business, I wasn't surprised they wanted to recycle the questions already in their item bank [despite Department of Education complaints about the quality of the questions] instead of paying someone to write new ones, as I was never surprised during my time in testing when any company opted for expediency and profit over the quality of the work. (...)
I don't believe in the results of standardized testing because "intelligence" and "teaching experience" turned out not to be attributes one would want in a professional scorer but rather faults in them that needed to be overcome. The job of test scorers really only involves learning and following the simplistic rules someone else has established. I always hoped my teams would consist of people who were smart enough to understand the rules but not so smart they'd be inclined to disagree with them. (...) I didn't need scorers spending five minutes looking for the hidden truth in every response; I needed them to look for key words and slap down a quick score. People who were too smart or cared too much were certainly not the people who helped most in that regard, and hence I found myself hoping for a team of scorers that was neither too brainy nor too invested in the state of American education. (...)
I don't believe in the results of standardized testing because it is the most inexact of sciences. At some point when scoring an item saying the football player Dallas was "shy" might not be worth any points, but later--after further consideration of the third paragraph of the second page of the story--that word might not seem so unreasonable and could be awarded credit after all. That means that some student responses would earn points for the word shy but earlier responses would not have been given the same credit, an inconsistency that was no more than an unfortunate reality of the standardized testing business. The thought process changed, so the scores changed--c'est la vie. (...)
[U]ncertainty was nearly always evident when committees of teachers came together, whether it was a development meeting when those educators were writing test questions or a range-finding meeting where they were trying to establish or approve scoring systems. Differing opinions were always prevalent. In my time in testing, I consistently worked with committees that disagreed with former committees, committees that disagreed with each other within a committee, and committees that often ended up even disagreeing with themselves. (...) Meanwhile, amid all the differing opinions, and amid all the score changes and rules changes, the assessment industry was ostensibly doing the work of "standardized" testing. (...)
Fifteen years of scoring standardized tests has completely convinced me that the business I've worked in is less a precise tool to assess students' exact abilities than just a lucrative means to make indefinite and indistinct generalizations about them. The idea standardized testing can make any sort of fine distinction about students--a very particular and specific commentary on their individual skills and abilities that a classroom teacher was unable to make--seems like folderol to me, absolute folderol.
What does it really mean to entrust decisions about this country's students, teachers, and schools to the massive standardized testing industry? In my opinion, it means trusting an industry that is unashamedly in the business of making money instead of listening to the many people who went into education for the more altruistic desire to do good. It means giving credence to the thoughts of the mobs of temporary employees who only dabble in assessment while ignoring the opinions of the men and women who dedicate themselves daily to the world of teaching and learning. It means saying you're not interested in what Mrs. White or Mr. Reyes who stands in front of a classroom of children every day might think about their students' progress, but you're absolutely enthralled to hear the thoughts on that same subject of a dopey Hank, a non-English-speaking Michi, a senile Alice, or an uninterested Todd. It means ignoring the conclusions about student abilities of this country's teacher--the people who instruct and nurture this country's children every single day--to instead heed the snap judgments of bored temps giving fleeting glances to student work.
As a final footnote, I can attest that, as a teacher who has attended quite a few in-service workshops dealing with (essentially) how to help students succeed on standardized testing, not only do I agree with Farley's distaste for the so-called "standardized" subjectivity of the scoring of writing and essay question responses, but I do not even share his view that multiple-choice questions are more cut-and-dry, and have only one obviously right answer. While this is certainly true a great deal of the time, even many multiple choice questions--especially when it comes to reading passages and questions dealing with ideas in the readings--are ambiguous, vaguely worded, and their answer choices often contain more than one answer that could be justified as correct. At one FCAT Reading workshop I attended, we--a room full of about thirty English and reading teachers--debated at least fifteen minutes over the correct response to one multiple choice question. I pointed out the ludicrousness of the situation: that a room full of English teachers could not agree on the right answer to this multiple-choice question, yet it could be this very question that would determine whether or not a student would graduate from high school. These questions were targeted at tenth grade students, and college graduate English teachers could not agree on the answer. How on earth could we expect a tenth grader to get the answer right--and have so much depend on that?
There are many problems in America's education system. We are behind most other industrialized nations. Our children do graduate at an alarmingly lower level of skill and knowledge than those of other nations. But life is not a series of standardized tests. Part of getting through to kids and making them take responsibility for what they are learning is giving them a meaningful purpose for what they are learning. If children could somehow understand the importance of their rights and civic duties, perhaps they would retain more of what they are taught about civics and history. But they are overwhelmingly convinced that what happened on television last night, and what football team won last night's game, is more important. Those sensibilities do not exist in a vacuum. Most adults in the U.S. share the same priorities. That is precisely why the American underclass continues to work harder and harder for less and less, allowing themselves to be taken advantage of by corporations and those on Wall Street. The corporate world, with its infinite and ubiquitous advertising campaigns, has quite effectively convinced the working classes and middle class that they are free and happy--or, at least, that they will be happy once they have the newest iPhone with the newest apps.
If our children struggle to find a main idea when they read a text, is it simply because American teachers have failed them, or is it because they have grown up in families where the television is always on, where the computer is a greater lure than a book, and where they rarely (if ever) see their own parents pick up a newspaper, book or magazine (unless it's about sports, or the latest electronic toy)?
There are many mediocre and even completely incompetent teachers in American schools right now; and the poorer the neighborhood, the more of them there are. We have explored the reasons for the achievement gap before. Many of them are linked to their situation at home; many others are linked to the quality of education they receive within their schools--which is highly unequal, compared to the experience of middle-class white students.
But how is the answer to that problem simply turning over the fate of America's students to this massive standardized testing industry, where scores are doled out, hundreds by the hour, by scorers ranging from incompetent to totally uninterested?
Students need to understand that what they are learning has some kind of value in the real world--something more important than passing a test.
Many state universities no longer require SAT or ACT scores because studies have proven that a student's grades in high school are a far better indicator of how students will perform in college than standardized test scores.
In Florida, many students are allowed to graduate from high school, despite never passing the FCAT, because of their scores on the SAT or ACT. What sense does it make, that students can perform better on the SAT or ACT--college entry exams--than on a tenth-grade exam (the FCAT) that is meant to ensure a bare minimum level of skills?
And what on earth does that say about an entire year's curriculum revolving around standardized tests?
Do not be fooled. The people creating and scoring those tests are not some educational gods in the sky, omniscient and dedicated to your child's education. Omniscient, no. Omnipotent? Perhaps.
They can decide whether or not your child is held back in elementary school or middle school, or whether or not he/she graduates from high school. They can decide which schools receive what funding. My own school risks losing its administrators after this year if it does not bring its grade back up to a C from its current D--despite the fact that we had enough points to earn a C this year, but a caveat in the grading system prevented us from actually being given that C. Our administrators are dedicated, smart, hardworking and caring. I consider myself extremely lucky to work in a school with such good administrators; I understand that it is not necessarily a common occurrence. Yet they could all be involuntarily reassigned--along with many teachers--if the standardized testing industry decides (with the arbitrariness described in Farley's book) that it is so.
Enough is enough. If there are problems with teacher quality, those problems must be addressed; and we already know a few solutions. Better training and support, especially in early years; better pay, so that the career attracts those qualified college graduates with too much student loan debt to consider going into a field whose salary prospects pale in comparison with those in the corporate world; community schools in poor neighborhoods; de-emphasizing sports and focusing on hiring teachers rather than coaches, just for starters. The American education system as a whole could also be improved by lengthening school days and the school year, and by ensuring true integration of schools across racial and economic lines, to the benefit of all children.
Standardized testing is sucking millions of dollars out of already very needy Florida schools. The cost of creating and administering the tests, plus the cost of having them scored; schools also hire "reading coaches" and "math coaches" whose primary function seems to be scanning tests and compiling pages of statistics for the district. Wouldn't this money be better spent on having smaller, better-equipped classrooms with qualified (and satisfied) teachers?
Enough is enough.
Related Articles:
Comments
