Skip to main content
  1. Tech
  2. Gadgets & Tech
  3. Tech Gear

Web crawling software eased Edward Snowden's collection of NSA documents

See also

How exactly did former NSA contractor Edward Snowden grab all the classified documents that he eventually leaked to the media? Investigators, trying to piece together his actions, have found Snowden managed to automate the process, using inexpensive and widely available software to scrape the data from NSA servers, the New York Times reported Saturday.

Snowden used web crawler software, akin to the software that Google uses to index web pages. A senior intelligence official said that Snowden “scraped data out of our systems” while he went about his daily assignments. “We do not believe this was an individual sitting at a machine and downloading this much material in sequence,” the official said. The process, as well, was “quite automated.”

The discoveries about Snowden's activities bring up more questions: Why didn't the web crawler software raise an alert? In fact, it appears that if Snowden had been working in a different NSA office, they might have caught him easily.

Snowden worked an agency outpost that had not yet been upgraded with modern security measures, including those that could be seen at NSA headquarters at Fort Meade, Md. That site was equipped with monitors designed to detect when a huge volume of data was being accessed and downloaded.

One official familiar with Mr. Snowden’s activities, said that “Some place had to be last” in getting the security upgrades. He added, though, that Snowden’s actions had been “challenged a few times.”

That brings up the question of how Snowden got out of those tough spots. As a systems administrator, Snowden was responsible for conducting routine network maintenance, which could include the backing up the computer systems and servers, and moving information to local servers, giving him cover.

It's unknown if Snowden "chose" the location he ended up with so that he would be better able to scrape data, or if it was just a coincidence. Hopwever, in terms of agency insecurity, Snowden learned all he needed to know in his early days as an NSA insider. While the agency had massive security walls to scale for those trying to hack in from the outside, it was very vulnerable to insiders.

Richard Bejtlich, the chief security strategist for FireEye, a Silicon Valley computer security firm, and a senior fellow at the Brookings Institution said:

Once you are inside the assumption is that you are supposed to be there, like in most organizations. But that doesn’t explain why they weren’t more vigilant about excessive activity in the system.

As the investigation continues, it searches for more than "how" Snowden did what he did. It also searches for "what he took." The head of the Defense Intelligence Agency, Lt. Gen. Michael T. Flynn, told lawmakers last week that there was a great deal of uncertainty about what Snowden did and did not take.

Everything that he touched, we assume that he took,” said General Flynn. The word "assume" means the government remains in the dark as to the exact content. Still, Flynn added, “We assume the worst case.”


Don't Miss

  • Massive
    Ubisoft Massive exclusive: 'The Division', PS4 & Xbox One, Activision to Ubisoft & more
    Games Exclusive
  • iPhone
    Get your wallet ready: The next iPhone could cost $100 more than your last one
    Tech Buzz
  • Civ
    Need to catch up on 'Sid Meier's Civilization'? Here is everything you need to know
    Games Feature
  • Google Glass
    See how Google Glass is letting sick kids go to the zoo without leaving the hospital
    Tech News
  • Upcoming
    These are 2014's biggest PS4, Xbox One and Wii U games
    Games Feature
  • Google
    Google has filed for a patent to develop contact lenses capable of taking photos

User login

Log in
Sign in with your email and password. Or reset your password.
Write for us
Interested in becoming an Examiner and sharing your experience and passion? We're always looking for quality writers. Find out more about and apply today!