Another Edison Schools achievement claim falls apart under scrutiny

July 29, 12:59 PMSF Education ExaminerCaroline Grannan

Previous Next

4 comments Print Email RSS Subscribe

Subscribe

Get alerts when there is a new article from the SF Education Examiner. Read Examiner.com's terms of use.

Email Address   Include other special offers from Examiner.com Terms of Use

Illuminating another questionable claim

The minor flurry of

news

about the failed for-profit school manager Edison Schools – and an indignant post here by an Edison supporter or employee – spurred a longtime Edison critic who is an indefatigable number-cruncher to take a close look at Edison’s newest claims about achievement in its Philadelphia schools. Even though that’s a long way from here, it’s still a useful lesson in how a cagey spinner can twist student achievement figures (as Edison was wont to do about its San Francisco school back in its heyday). This is  admittedly a post for serious wonks, so I’ll give the short version in four words: The claims are bogus. Here’s her report:

In a press release issued recently by EdisonLearning (formely Edison Schools), and obviously meant to do damage control on the news that the Philadelphia School District was taking back four of the 20 schools Edison had been given to manage in 2002, some claims were made for Edison’s achievement which sound almost unbelievable. Maybe that’s because they are unbelievable.

The release addresses percentages of students making progress at Edison’s schools as compared to those making progress at Philadelphia schools districtwide. Edison honcho John Chubb is quoted: “The state’s results clearly confirm that the outside providers are meeting and exceeding expectations in Philadelphia, and they validate the work and highlight the success of all the private providers in Philadelphia….More importantly, this is a pure and objective measurement of student performance in the Philadelphia’s schools – the Pennsylvania System of School Assessment.”

This test, the Pennsylvania System of School Assessment, or PSSA, is indeed considered the gold standard within the state, and its results are used to track whether schools are making appropriate progress under No Child Left Behind. Some Pennsylvania school districts may use other tests as well, and Edison is widely known to do a lot of in-house testing of students, but it is the PSSA results that are the official measure of how students in the state are doing.

So, when the press release compares test results from the most recent school year (07-08) to results from several years ago, when Edison first took over those 20 schools, it would be reasonable to assume that it is the PSSA results (that “pure and objective measurement of student performance”) that are being presented, although the release does not specifically state this.

To quote:

“When EdisonLearning began its partnership with Philadelphia in 2002, 860 students out of roughly 13,000 in the 20 schools they were assigned were performing at grade level in math. Today, there are nearly 3,500 performing at grade level and above. In reading, there were 1,400 students at grade level; today, more than 3,000.”

Wow – what spectacular results those are! Just 860 of 13,000 students were at grade level in math in 2002, but now 3500 are! And in reading, just 1,400 students were at grade level, but now more than 3,000 are! It’s a miracle!

But wait….what do those numbers represent? The “roughly 13,000” number represents the total enrollment in Edison Schools in 2002, and I guess the reader is supposed to assume that all 13,000 of them were tested with the PSSA, with the dismal result that only 860 were found to be at grade level. But back in 2002, the PSSA was given only in grades 5, 8, and 11 (Edison operates only schools with grades between K-8.) So clearly, with 13,000 students spread out over nine grades (an average of about 1444 student per grade) and only two of those nine grades taking the PSSA, it was not 860 of 13,000 students who were found to be “performing at grade level in math”, but rather 860 of a much smaller number – that is, students in grades 5 and 8, a number likely to be closer to 3,000 than 13,000.

Well, but how about those 3500 student who are now testing at grade level or above in math – surely the jump from 860 to 3500 is miraculous, even if it took six years to happen? Perhaps it would be miraculous if the PSSA continued to be given only in grades 5 and 8, but (unmentioned in the press release), in reality, since 2005-06, the PSSA has been given in grades 3-8. In other words, the 860 students at grade level in math in 2002 were from just two tested grades, but the 3500 at grade level in 2008, to which that number is compared, were from six tested grades. Is it any wonder that more students tested at grade level in 2008? More students were tested; many, many more! The reading results are similarly questionable.

The release goes on to gush:

“In 2002, more than 10,000 students out of 13,000 were ‘below basic’ in math in the EdisonLearning partnership schools. Now that number has been reduced by more than half to 4,800. In 2002, nearly 9,000 students were below basic in reading; that number has been reduced to 5,800.”

It is unknown how Edison determined that “more than 10,000 students out of 13,000 were “below basic” in math” in 2002, since only grades 5 and 8 took the PSSA that year; the actual number tested was closer to 3,000 than 13,000. So how did Edison determine that “more than 10,000” of their students were below basic in math? Clearly, it was not by using the PSSA, that “pure and objective measurement of student performance.” Maybe they used an in-house diagnostic tests, you know the ones which may not be so “pure and objective.” We’ll never know, because Edison does not say which test results they are talking about here.

At least we can all agree that reducing that number (however it was derived) from 10,000 below basic in math in 2002, to 4,800 below basic in math in 2008, is a great leap forward? Maybe – if it was 4,800 of 13,000 students, just as the 2002 figure was 10,000 of 13,000 students. But Edison schools have been losing enrollment; in fact one Philadelphia Edison school, Sulzberger, is being closed by the school district in June 2009 because of low enrollment, according to a Philadelphia Inquirer report.

The state has not yet posted PSSA test results for 2007-08 and won’t until late August, although Edison does have those results now. Until the results are made public, it is impossible to tell how many Edison students were tested in 2007-08 (certainly not 13,000), but it is likely to be closer to the number tested in 2006-07, which was 6975 tested in math and 6932 in reading. So 4,800 “below basic”, of about 7,000 tested students, tells a whole other story than 4,800 “below basic”, of about 13,000 students, doesn’t it? Just giving the number of students who are still testing “below basic” only tells half the story; the other half is, how many took the test? The Edison press release leaves out this key detail. In fact, 4,800 of approximately 7,000 tested students yields a result of 68.5% of students still testing below basic in math. The 5,800 students reported to be still testing “below basic” in reading equates to nearly 83% testing “below basic” in reading.

One more key point not mentioned in the press release: the PSSA divides students into four possible levels, of which “below basic” is the lowest. However, the next level up, called “basic”, is not the same as “grade level”; it is only when students make it to the third level up, called ‘proficient”, that they are considered to be at “grade level.” So the 4,800 Edison students who are still testing at “below basic” in math, and the 5,800 who are ‘below basic” in reading, represent only some of those who are still performing “below grade level” in Edison schools. The total numbers, including all of the students who tested only at “basic,” are far higher, but Edison is not sharing those numbers with us.

Some miracle.