Edward Snowden Used a Simple Web Crawler To Gather NSA Data

 Pages 1 2 NEXT
 

Edward Snowden Used a Simple Web Crawler To Gather NSA Data

Snowden

Snowden used a tool similar to the one Google uses to index websites for its search engine to gather his controversial NSA data.

It would be fun to imagine Edward Snowden speed-hacking his way into the NSA servers like Hugh Jackman in Swordfish, or physically stealing a briefcase full of hard-disks and making a daring escape from NSA headquarters, but the truth is much simpler. Speaking to the New York Times, a senior intelligence official said that Snowden used nothing more than a simple web crawler to gather his controversial data.

Using the web crawler, Snowden "scraped data out of our systems" while he went about his day job, said the official. "We do not believe this was an individual sitting at a machine and downloading this much material in sequence," he said, adding that the process was "quite automated." To automatically collect the info he wanted, Snowden only needed the right logins to bypass what internal defenses were in place.

What makes this data so damning is that the NSA's mission statement is to "protect the nation's most sensitive military and intelligence computer systems from cyberattacks," which is quite embarrassing considering the simplicity of Snowden's technique - Investigators found that Snowden's attacks were hardly sophisticated and should have been easily detected.

Agency officials insist that if Snowden was working at NSA's headquarters at Fort Meade, he would have been caught, but the Hawaii branch that he was employed at lacked the activity monitors that would have found his bot.

Web crawlers are commonly used by search engines like Google to index websites.

Regardless of whether or not you think Snowden was "right" or "wrong," you have to admit the NSA is partly to blame for not protecting itself properly.

Source: New York Times via Engadget

Permalink

Now I know this whole business has been around for a while now but what EXACTLY did he do?

I have inklings here and there but I would like a full unbiased story of exactly what he did.

Quiet Stranger:
Now I know this whole business has been around for a while now but what EXACTLY did he do?

I have inklings here and there but I would like a full unbiased story of exactly what he did.

he basically blew open the extent of what the NSA actually gets up to and how much data they collect on every day americans.

There shouldn't be any controversy, the man was not only right, he's a patriot in the truest sense. It's the people clamoring for his head as a traitor who are the traitors, to the very ideals of democracy.

Quiet Stranger:
Now I know this whole business has been around for a while now but what EXACTLY did he do?

I have inklings here and there but I would like a full unbiased story of exactly what he did.

The short version is that he gathered a bunch of information regarding some rather severely illegal activity performed by the US government, then released it to the public at large. He's the primary reason people are actively aware of the NSA's ongoing efforts to actively spy on the American populace, rampant violations of the spirit, if not the letter, of nearly every privacy law in existence, and general anti-democratic behavior. It caused something of a kerfluffle over here a bit under a year ago.

Naturally, the US government charged him with treason and he fled the country rather than be executed. Last I heard, he's now living in Russia, and is under their protection.

As far as I'm concerned, the man did the right thing and the feds need to eat a nuke already, but I'm a raging libertarian, so I can't pretend I'm not biased.

Owyn_Merrilin:
There shouldn't be any controversy, the man was not only right, he's a patriot in the truest sense. It's the people clamoring for his head as a traitor who are the traitors, to the very ideals of democracy.

Sadly the people arguing in favour of lynching Snowden are woefully unawares as to the concept of the chilling effect.

"In United States and Canadian law, the term chilling effects refers to the stifling effect that vague or excessively broad laws may have on legitimate speech activity."

If out of the blue you are a whistle blower sending emails to your publisher and you get a knock on your door from government agents. Imagine the paranoia,how that could uniformly cripple freedom of speech in a nation.

Owyn_Merrilin:
There shouldn't be any controversy, the man was not only right, he's a patriot in the truest sense. It's the people clamoring for his head as a traitor who are the traitors, to the very ideals of democracy.

I have to agree with this. I am one of those who wholeheartedly support the notion that he should get the Nobel Peace Prize for what he did. We gave it to Obama for far less and we have given it to people for showing they oppose violations of human rights not that long ago. I don't think it will happen for political reasons though. Giving him the Nobel Peace Prize would screw up the relationship with USA so they'll probably give it to someone planting trees again.

wombat_of_war:

Quiet Stranger:
Now I know this whole business has been around for a while now but what EXACTLY did he do?

I have inklings here and there but I would like a full unbiased story of exactly what he did.

he basically blew open the extent of what the NSA actually gets up to and how much data they collect on every day americans.

Not just americans...
We're talking about spying on a global scale here.

I just hope that this controversy won't stop until something happens. As long as he has media coverage, he's safe...

Well it's not that surprising, yes they no doubt have huge protection protocols from outside access to their internal system, probably even complete physical separation on most ends.

But when it comes to the internal system things don't get monitored that much because everyone who has clearance is already expected to search and cross reference information in their daily operation with automated software, they could go and interview everyone why they needed every particular piece of info but that would take several hours per employee each day... no one is willing to put that much effort into it, at least not for stuff that wasn't that critical to them.

TheSniperFan:

wombat_of_war:

Quiet Stranger:
Now I know this whole business has been around for a while now but what EXACTLY did he do?

I have inklings here and there but I would like a full unbiased story of exactly what he did.

he basically blew open the extent of what the NSA actually gets up to and how much data they collect on every day americans.

Not just americans...
We're talking about spying on a global scale here.

I just hope that this controversy won't stop until something happens. As long as he has media coverage, he's safe...

To be fair, there's no protections in the US constitution against spying on non-citizens in foreign countries. US Citizens, however, are supposed to be protected, and that's why the PATRIOT act, which is the legal basis for pretty much all of this crap, is so horrible.

Owyn_Merrilin:

TheSniperFan:

wombat_of_war:

he basically blew open the extent of what the NSA actually gets up to and how much data they collect on every day americans.

Not just americans...
We're talking about spying on a global scale here.

I just hope that this controversy won't stop until something happens. As long as he has media coverage, he's safe...

To be fair, there's no protections in the US constitution against spying on non-citizens in foreign countries. US Citizens, however, are supposed to be protected, and that's why the PATRIOT act, which is the legal basis for pretty much all of this crap, is so horrible.

To be fair, the vast majority of the people doesn't give two shits about the US and its constitution.
The thing is that currently the US is just showing every country its middle fingers and our governments (the German government in my case), don't do anything about it.
Instead of protecting us, they just bury their heads in the sand and wait for people to forget about it. Occasionally saying something among the lines of "It's all good. They're our friends." and "Hereby I declare the NSA-affair for finished."
The sad thing for us German citizens is that they want to (and most likely will) spy on us ourselves (keyword: VDS).

I'm with you on the patriot act thing though.
Whenever I hear someone talk about how free 'Murica is, I think about it. A country with such a law in place has no business in calling itself "the land of the free".

Owyn_Merrilin:
To be fair, there's no protections in the US constitution against spying on non-citizens in foreign countries. US Citizens, however, are supposed to be protected, and that's why the PATRIOT act, which is the legal basis for pretty much all of this crap, is so horrible.

He hasn't just shone a light on US spying of the rest of the world though. The stuff he's revealed about the NSA has people looking more closely at other countries spy agencies as well. Turns out that CSEC here in Canada isn't a whole lot better.

Quiet Stranger:
Now I know this whole business has been around for a while now but what EXACTLY did he do?

I have inklings here and there but I would like a full unbiased story of exactly what he did.

The president of the united states and the nsa decided that they were going to violate the law, the bill of rights, and the constitution, break the laws of the patriot act, there is total and complete oversite built into tpa, they went around all that stated mass spying on people.

Mr snowden a rather low level nsa guy sees stuff going on he finds objectionable presumably, see above at all the stuff above on doing this and he was the only person in washington that had concerns about what the hell they were doing exactly.

He steals a bunch of files flees the nation, releases the files on rededit, the whole thing blows wide open, the nation learns what is going on the congress learns of what is going on and they sit with their thumbs well and truely up their asses congress and the president wants snowden dead, the press and public instead of focusing on what is important here the ILLEGAL mass spying of this white house and the nsa on INNOCENT americans and political enemies.

The usg at first denied most of it, then they switch to we were doing it for your protection, and we stil in this strange twilight zone nation were we have a administration that give not one single solitary piss about the laws of this nation or the oath he took to uphold the constitution.

Thats about as unbiased as i can put it. It's a fucking crime.

Vivi22:

He hasn't just shone a light on US spying of the rest of the world though. The stuff he's revealed about the NSA has people looking more closely at other countries spy agencies as well. Turns out that CSEC here in Canada isn't a whole lot better.

Off topic but lol Canada has a security agency named C-sec

Vakarian approves

Steven Bogos:
To automatically collect the info he wanted, Snowden only needed the right logins to bypass what internal defenses were in place.

What makes this data so damning is that the NSA's mission statement is to "protect the nation's most sensitive military and intelligence computer systems from cyberattacks," which is quite embarrassing considering the simplicity of Snowden's technique - Investigators found that Snowden's attacks were hardly sophisticated and should have been easily detected.

To be fair. The technique could be simple... because he had all the security logins, the actual hard bit of acquiring that information. Its kind of like giving the keys to your house over and being confused why they got into your computer so easily. He had all the hard work done, so it necessarily didn't require much effort to gain what he needed.

When he started working there he never had to sign a confidence agreement or anything?

Nimcha:
When he started working there he never had to sign a confidence agreement or anything?

Yes mr snowden violated the law also, he just violated it less than your president and congress and nsa. Half of them swore an oath on a bible to uphold the laws and constitution of this nation. mr snowden didnt he signed some papers, what should carry more weight?

cerebus23:

Nimcha:
When he started working there he never had to sign a confidence agreement or anything?

Yes mr snowden violated the law also, he just violated it less than your president and congress and nsa. Half of them swore an oath on a bible to uphold the laws and constitution of this nation. mr snowden didnt he signed some papers, what should carry more weight?

Don't you know that pointing out the illegal things your government does is illegal.

I would hope it would educate people some on the hyprocracy when these politicians stand up and call snowden a traitor.

So many people seem so wholly unaware of what their government does or cares one bit its near frightening. do we teach the bill of rights, the constitution anymore at all?

This is also a government supposedly made of we the people, that work for our interests, big interests and big money have made that impossible for a long ass time, and now we just on cruise control to the end.

Governments are supposed to be working for us not against us and we better get this ship sorted asap and people better start to wake up, this ship is sailing and its going straight to the bottom of the ocean without some real quick and drastic changes.

Owyn_Merrilin:
There shouldn't be any controversy, the man was not only right, he's a patriot in the truest sense. It's the people clamoring for his head as a traitor who are the traitors, to the very ideals of democracy.

My feelings is that Snowden should be punished for breaking the law, as should those indicted by the information he revealed. In the eye of the law all is equal and there to be no exceptions. You break the law you need to face the prescribed punishment whether it is fines, jail time, or death.

TheSniperFan:

Owyn_Merrilin:

TheSniperFan:
Not just americans...
We're talking about spying on a global scale here.

I just hope that this controversy won't stop until something happens. As long as he has media coverage, he's safe...

To be fair, there's no protections in the US constitution against spying on non-citizens in foreign countries. US Citizens, however, are supposed to be protected, and that's why the PATRIOT act, which is the legal basis for pretty much all of this crap, is so horrible.

To be fair, the vast majority of the people doesn't give two shits about the US and its constitution.
The thing is that currently the US is just showing every country its middle fingers and our governments (the German government in my case), don't do anything about it.
Instead of protecting us, they just bury their heads in the sand and wait for people to forget about it. Occasionally saying something among the lines of "It's all good. They're our friends." and "Hereby I declare the NSA-affair for finished."
The sad thing for us German citizens is that they want to (and most likely will) spy on us ourselves (keyword: VDS).

I'm with you on the patriot act thing though.
Whenever I hear someone talk about how free 'Murica is, I think about it. A country with such a law in place has no business in calling itself "the land of the free".

The even sadder thing for us Germans is, that Snowden also revealed the NSA engages in industrial espionage against its allies, meaning us. A few years ago the Spiegel found out that about 48% of German companies notice some leaking of know-how. Guess where that's going.

kiri2tsubasa:

Owyn_Merrilin:
There shouldn't be any controversy, the man was not only right, he's a patriot in the truest sense. It's the people clamoring for his head as a traitor who are the traitors, to the very ideals of democracy.

My feelings is that Snowden should be punished for breaking the law, as should those indicted by the information he revealed. In the eye of the law all is equal and there to be no exceptions. You break the law you need to face the prescribed punishment whether it is fines, jail time, or death.

Except the law he violated violated the constitution, and so did the people following it. The constitution is the highest law in the US, anything that violates it is superceded by it. Snowden is a hero.

Besides, if you think there's no reason to ever break the law, you've got a lot to learn about the law. There's this myth that the rule of law is some magical thing that keeps us separate from countries ruled by the whims of their leaders. That myth ignores who writes the laws.

kiri2tsubasa:

Owyn_Merrilin:
There shouldn't be any controversy, the man was not only right, he's a patriot in the truest sense. It's the people clamoring for his head as a traitor who are the traitors, to the very ideals of democracy.

My feelings is that Snowden should be punished for breaking the law, as should those indicted by the information he revealed. In the eye of the law all is equal and there to be no exceptions. You break the law you need to face the prescribed punishment whether it is fines, jail time, or death.

"beep boop, I'm a robot that decides what right or wrong based on the latest law; morality or ethics be damned, if it's in da rulez it goes"

That's what you sound like right now.

I think Snowden should have leaked all of the data on the internet, that way the entire situation would most likely have caused massive uprisings aswell as people demanding that their governments resign or shut down certain branches. I don't care about the information regarding foreign citizens, but I do care about anything that involves the danish government.

Government employees work for and exist to serve the citizens of their country.. not the country itself.

Is there some kind of American lobby which takes this seriously and works for better privacy protection? Or couldn't all you Americans organize yourselves against this? Start a revolution or something?

Web crawler? He hired Spiderman to do it? Good lord.

*ahem*

Jokes aside, just goes to show that information inherently wants to be free, and makes itself available to anyone who does some digging.

Also, at this point, it doesn't actually matter what the US government does with and/or to Snowden. The information is free.

Snowden was far smarter than them. He knew what they were monitoring internally and what they weren't.

cerebus23:

Nimcha:
When he started working there he never had to sign a confidence agreement or anything?

Yes mr snowden violated the law also, he just violated it less than your president and congress and nsa. Half of them swore an oath on a bible to uphold the laws and constitution of this nation. mr snowden didnt he signed some papers, what should carry more weight?

I don't know about the USA, but in the UK every citizen is held under the Official Secrets Act. If a citizen knows a national secret, or even international secret, and knowingly spreads it to others not in the know they will be prosecuted...

It doesn't matter whether you have signed the official secrets act or swore an oath.

cerebus23:

Nimcha:
When he started working there he never had to sign a confidence agreement or anything?

Yes mr snowden violated the law also, he just violated it less than your president and congress and nsa. Half of them swore an oath on a bible to uphold the laws and constitution of this nation. mr snowden didnt he signed some papers, what should carry more weight?

signing a legal document carries more weight than swering on a bible. First is legally binding, second one is just some religiuos ceremony.

Sofus:
I think Snowden should have leaked all of the data on the internet, that way the entire situation would most likely have caused massive uprisings aswell as people demanding that their governments resign or shut down certain branches.

he DID leak it on the internet. Reddit to be specific.

Senare:
Is there some kind of American lobby which takes this seriously and works for better privacy protection? Or couldn't all you Americans organize yourselves against this? Start a revolution or something?

In US in order to lobby you need rich corporation backing. since no corporation wants privacy this is not going to happen.
And you really asking majority to organize about anything? have you noticed that revolutions only ever start when people got nothing to loose and are starving to death?

Yes mr snowden violated the law also, he just violated it less than your president and congress and nsa. Half of them swore an oath on a bible to uphold the laws and constitution of this nation. mr snowden didnt he signed some papers, what should carry more weight?[/quote]
signing a legal document carries more weight than swering on a bible. First is legally binding, second one is just some religiuos ceremony.

I would hazard legally they are both equally binding and if your "god" fearing person than arguably the bible oath would be the one that would hold your immortal soul in peril to violate.......

And in either case when you agree to swear an oath you agree to the terms of the oath, weither or not you personally belive in the invisible old man in the sky or not does not matter.

But your attitude completely explains why mr obama, and so many other sentaors and house members wipes their ass with the bill of rights and constitution after swearing to uphold protect and defend those documents.

To get this thread away from fierce politics arguments for just a sec: It's funny how many people think a lot of stuff in the real world they haven't seen happens just like in those Hollywood movies when just about the only thing those guys get right is people need to breath. And sometimes Hollywood screws that up, too.

I'd bet Snowden got his tools from something like sourceforge and the combined time of downloading it to a flash drive he could hide up his bum(citation needed) at home, plug it into his pc at work and copy all those files would be around 5-10 minutes and take like a dozen clicks of his mouse. Many sensitive installations like nuclear power plants, military bases, and super secret corporate research networks don't allow Internets or even usb port access to anyone but IT and high level personel. (For having a genius intellect, Q was a fucking idiot for plugging in the laptop of a known hacker wiz into M6's LAN.) Knowing the NSA and the government in general, they probably relaxed on way to many things during the recent budget cuts. He probably didn't really need to hide it all that well. The Fort Meade officials might be bluffing a little, too. Unless a guard is staring at a screen, or waiting for a sound to let him know something fishy is going on in the network, I'd say Snowden could have had enough time to get his files and run out saying my girlfriend was in an accident or something before they could look up her credit cards and say she just bought a latte 2 minutes ago. With the cuts of government employees and their hours all over while congress still gets full pay to sit on their ass and argue where to cut next, I'd say it could have happened at any of their locations, until he alerted them it was possible.

Owyn_Merrilin:
There shouldn't be any controversy, the man was not only right, he's a patriot in the truest sense. It's the people clamoring for his head as a traitor who are the traitors, to the very ideals of democracy.

Couldn't have said it better myself. It's not a coincidence that among the -actual- traitors calling him the traitor are such "sterling" figures as former US President Bush and then... Basically the entire right-wing side of the political system.

Also if a simple web-crawler can bypass the NSA cyber-security, we've got a digital armageddon just waiting to happen.

Agayek:

Naturally, the US government charged him with treason and he fled the country rather than be executed. Last I heard, he's now living in Russia, and is under their protection.

That's... not at all true. Nothing you've said there is true except "he's now living in Russia."

Nobody's charged Snowden with treason. He's been charged with theft of government property, unauthorized communication of national defense information, and willful communication of classified communications information. None of these crimes carries the death penalty.

Also, he left the country after obtaining but before leaking the documents. He left to Hong Kong, where he stayed at a hotel. He took a flight to South America by means of Moscow, but his passport was pulled while he was in Moscow, so he was stuck in Moscow airport for several months. Putin eventually granted him asylum in Russia.

Strazdas:

he DID leak it on the internet. Reddit to be specific.

No, he didn't. He hasn't leaked anything at all to reddit. He leaked the documents to three reporters (Glenn Greenwald, Laura Poitras, and Barton Gellman), who he charged with carefully selecting from these only those which would expose government wrongdoing and would not put any people in harm's way. I don't know where you're getting this whole 'he leaked to reddit' thing from.

Strazdas:

Senare:
Is there some kind of American lobby which takes this seriously and works for better privacy protection? Or couldn't all you Americans organize yourselves against this? Start a revolution or something?

In US in order to lobby you need rich corporation backing. since no corporation wants privacy this is not going to happen.
And you really asking majority to organize about anything? have you noticed that revolutions only ever start when people got nothing to loose and are starving to death?

Um, ahem:

https://www.aclu.org/

https://www.eff.org/

Two of the higher profile organizations dedicated to this sort of thing.

Also, you don't need rich corporate backing in order to lobby in the US. It just helps if you want to get unscrupulous politicians to listen, because the laws are set up so you can butter them up with campaign donations, and until about 10 years ago, when a law was passed banning it, they'd go further with gifts and fancy dinners -- bribes that didn't involve directly giving someone cash, in other words. Lobbying in itself is just the act of talking to a politician personally and letting him know what you as a constituent want him to do. It's not a bad thing. It becomes a bad thing when bribery is a part of it.

It bugs me that the article didn't say the name of the tool used.

It was nothing uncommon. It is a tool that comes standard with basically any Linux distribution. For those curious:

https://en.wikipedia.org/wiki/Wget

The usual use-case is to download a single file, like the source code for some server application. It shows up in a bunch of tutorials.

 Pages 1 2 NEXT

Reply to Thread

Log in or Register to Comment
Have an account? Login below:
With Facebook:Login With Facebook
or
Username:  
Password:  
  
Not registered? To sign up for an account with The Escapist:
Register With Facebook
Register With Facebook
or
Register for a free account here