CRU raw data loss not an accident

Climatic Research Unit (Photo courtesy CRU)

With the recent release of "correspondence, code and documents" from the Climatic Research Unit, the earlier admitted "loss" by CRU of key raw data begins to look deliberate.

As previously mentioned, Steven McIntyre had sought release of CRU's data under the UK Freedom of Information Act. At first, Jones simply refused this and several similar requests from other parties. Then on July 27, 2009, CRU erased three key files from its public database, as Mr. McIntyre can prove easily because he has before-and-after screenshots. CRU followed up, in short order, with what McIntyre and some of his readers called an "unpredecented" "purge" of its public data directory. McIntyre's screenshots tell a breathtaking story of wholesale removal of files previously made available to the public, including, at one point, the deletion of every single listing in Phil Jones' public directory. Anthony Watts summed up the situation in one word: "panic."

Patrick J. Michaels reported on the story in his column at National Review online. In his report, dated September 23, 2009, Michaels quoted this statement by Jones to Roger Pielke, in response to Pielke's own request to Jones for raw data:

Since the 1980s, we have merged the data we have received into existing series or begun new ones, so it is impossible to say if all stations within a particular country or if all of an individual record should be freely available. Data storage availability in the 1980s meant that we were not able to keep the multiple sources for some sites, only the station series after adjustment for homogeneity issues. We, therefore, do not hold the original raw data but only the value-added (i.e., quality controlled and homogenized) data.

As Michaels himself pointed out, the above statement is highly dubious. The CRU was established in the mid-1980s (with at least partial funding by the US Environmental Protection Agency) specifically to build a "comprehensive history of surface temperature" from weather stations. At the time of its founding, the CRU would have had magnetic-tape storage techniques that would have been more than adequate to the task. (The legacy of the use of magnetic tape remains today in one of the most common archiving commands used in UNIX or GNU/Linux systems, "tar", for Tape ARchive.)

Then on October 6, 2009, the Competitive Enterprise Institute filed a petition with the EPA to reopen for public comment their proposal to regulate carbon dioxide as a pollutant. In support of that petition, CEI's general counsel, Sam Katzman, said this:

EPA is resting its case on international studies that in turn relied on CRU data.  But CRU’s suspicious destruction of its original data, disclosed at this late date, makes that information totally unreliable.  If EPA doesn’t reexamine the implications of this, it’s stumbling blindly into the most important regulatory issue we face.

In his initial post describing the loss of the first three files, Steve McIntyre cited the relevant British law:

Under U.K. Freedom of Information Act, once FOI requests have been made for information, public authorities are not permitted to "alter, deface, block, erase, destroy or conceal any record held by the public authority, with the intention of preventing the disclosure by that authority of all, or any part, of the information to the communication of which the applicant would have been entitled." As [one of McIntyre's commenters] observed the other day, the Computer Misuse Act also prescribes various offences, one of which is the unauthorized modification of the contents of a computer with the intent "to prevent or hinder access to any program or data held in any computer".

In this light, these e-mails, that bear directly on the issue of the data destruction, are especially embarrassing.

File No. 1212063122 (Michael Mann to Phil Jones, May 2008) indicates that the attempt to scrub the e-mail system of embarrassing correspondence began more than a year before the problem with the public servers became apparent. In this e-mail, Mann quotes an earlier e-mail from Jones to Mann (in File No. 1212073451.txt):

Mike,

Can you delete any emails you may have had with Keith re AR4? Keith will do likewise. He's not in at the moment - minor family crisis.

Can you also email Gene and get him to do the same?  I don't have his new email address.

We will be getting Caspar to do likewise.

These deletions were clearly targeted, and went far beyond routine e-mail system storage management. And Michael Mann's quoting of this message back to Jones indicates that he was well aware of Jones' plan of action, and his failure to protest that plan suggests agreement with it. Worse yet, Jones asks Mann to contact a colleague and pass along his "instructions" to him, and Mann explicitly agrees so to act.

File No. 1228258714.txt (Gavin Schmidt to Ben Santer, December 2008) demonstrates the typical attitude, and hence motive, for CRU's actions:

The contrarians have found that there is actually no limit to what you can ask people for (raw data, intermediate steps, additional calculations, residuals, sensitivity calculations, all the code, a workable version of the code on any platform etc.), and like Somali pirates they have found that once someone has paid up, they can always shake them down again.

File no. 1228330629.txt (Phil Jones to Ben Santer and Tom Wigley, December 2008) illustrates Phil Jones' successful persuasion of a compliance officer to support him in non-compliance:

When the FOI requests began here, the FOI person said we had to abide by the requests. It took a couple of half hour sessions - one at a screen, to convince them otherwise showing them what CA was all about. Once they became aware of the types of people we were dealing with, everyone at UEA (in the registry and in the Environmental Sciences school - the head of school and a few others) became very supportive. I've got to know the FOI person quite well and the Chief Librarian - who deals with appeals. The VC is also aware of what is going on - at least for one of the requests, but probably doesn't know the number we're dealing with. We are in double figures.

And in addition:

The inadvertent email I sent last month has led to a Data Protection Act request sent by a certain Canadian [presumably McIntyre], saying  that the email maligned his scientific credibility with his peers! If he pays 10 pounds (which he hasn't yet) I am supposed to go through my emails and he can get anything I've written about him. About 2 months ago I deleted loads of emails, so have very little - if anything at all. This legislation is different from the FOI - it is supposed to be used to find put why you might have a poor credit rating!

The trouble being that Phil Jones could not then, and cannot now, pretend to have deleted files inadvertently, because his earlier exhortation to others to begin targeted deletion predates this message by at least seven months.

Thus Phil Jones is now on record willfully destroying correspondence relevant to a Freedom of Information request, and suborning others to act likewise. But in addition to that, File No. 1255095172.txt illustrates the cavalier attitude on the part of Ben Santer and other members of Phil Jones' professional circle toward scientific data retention and recordkeeping:

The CEI and Michaels are applying impossible legal standards to science. They are essentially claiming that if we do not retain - and make available to self-appointed auditors - every piece of information about every scientific paper we have ever published, we are perpetrating some vast deception on the American public. I think most ordinary citizens understand that few among us have preserved every bank statement and every utility bill we've received in the last 20 years.

Mr. Santer knows better than that. He is not "an ordinary citizen" being called on the carpet for not "preserv[ing] every bank statement and every utility bill [he has] received in the last 20 years." He is a senior scientist at the Lawrence Livermore National Laboratory. How he would react if the Jet Propulsion Laboratory abruptly lost all the raw data returned by, say, the Voyager 1 and 2 probes (which launched in September and August 1977, respectively) is difficult to predict, according to his currently expressed logic.

If this is the attitude taken by a scientist to actions by a colleague that he knows, or ought to know, are unlawful, then the American public do indeed have a right to wonder whether said scientist, and others of his colleagues on American payrolls, have indeed been "perpetrating some vast deception on the American public."

Like this article? Want to be notified of more? Click Subscribe, above.