Skip to main content

See also:

Web crawling software eased Edward Snowden's collection of NSA documents

Edward Snowden
Wikimedia Commons

How exactly did former NSA contractor Edward Snowden grab all the classified documents that he eventually leaked to the media? Investigators, trying to piece together his actions, have found Snowden managed to automate the process, using inexpensive and widely available software to scrape the data from NSA servers, the New York Times reported Saturday.

Snowden used web crawler software, akin to the software that Google uses to index web pages. A senior intelligence official said that Snowden “scraped data out of our systems” while he went about his daily assignments. “We do not believe this was an individual sitting at a machine and downloading this much material in sequence,” the official said. The process, as well, was “quite automated.”

The discoveries about Snowden's activities bring up more questions: Why didn't the web crawler software raise an alert? In fact, it appears that if Snowden had been working in a different NSA office, they might have caught him easily.

Snowden worked an agency outpost that had not yet been upgraded with modern security measures, including those that could be seen at NSA headquarters at Fort Meade, Md. That site was equipped with monitors designed to detect when a huge volume of data was being accessed and downloaded.

One official familiar with Mr. Snowden’s activities, said that “Some place had to be last” in getting the security upgrades. He added, though, that Snowden’s actions had been “challenged a few times.”

That brings up the question of how Snowden got out of those tough spots. As a systems administrator, Snowden was responsible for conducting routine network maintenance, which could include the backing up the computer systems and servers, and moving information to local servers, giving him cover.

It's unknown if Snowden "chose" the location he ended up with so that he would be better able to scrape data, or if it was just a coincidence. Hopwever, in terms of agency insecurity, Snowden learned all he needed to know in his early days as an NSA insider. While the agency had massive security walls to scale for those trying to hack in from the outside, it was very vulnerable to insiders.

Richard Bejtlich, the chief security strategist for FireEye, a Silicon Valley computer security firm, and a senior fellow at the Brookings Institution said:

Once you are inside the assumption is that you are supposed to be there, like in most organizations. But that doesn’t explain why they weren’t more vigilant about excessive activity in the system.

As the investigation continues, it searches for more than "how" Snowden did what he did. It also searches for "what he took." The head of the Defense Intelligence Agency, Lt. Gen. Michael T. Flynn, told lawmakers last week that there was a great deal of uncertainty about what Snowden did and did not take.

Everything that he touched, we assume that he took,” said General Flynn. The word "assume" means the government remains in the dark as to the exact content. Still, Flynn added, “We assume the worst case.”