Tech Guys Parse the NSA's Data "Firehose"

Over at Defense Tech, the tech guys have added their input on the information trickling out about what the NSA is doing to spy on Americans. Following up on an NYTimes editorial, they've added some insight into the way that the program can digest all that data and spit out information.

Once the data mining programs get going this voluminous data can then be further analyzed and used for further research...

The tech guys seems somewhat in awe of the program's breadth and length. They especially note how impressive it is that US companies opened up their communication lines to the NSA like a lazy whore spreading her legs to a rich john.

Commenting on how the NSA is sifting through this haystack of data to find that proverbial needle, they note:

"The NSA is intercepting huge streams of communications, taking in 2 million pieces of communications an hour," James Bamford, the author of two books on the NSA, told the Boston Globe on Friday.

"They have a capacity to listen to every overseas phone call," added Tom Blanton, director of the National Security Archive at George Washington University."

But the question has been: how do you turn all that data into something useful? You've got to find a realtively simple way to get rid of 99.99999% of the calls and e-mails quickly. Otherwise, it's like drinking from a firehose.

But as link analysis and data mining programs have become more sophisticated, that sifting process has gotten easier. And, I'll bet, it is simpler still when the telecom companies are playing ball.

All the cliches about Brave New World, Big Brother and 1984 seem almost trite here. Those stories still had the veneer of simple dystopic nightmares, a liberal's worst dream that conservatives could pooh-pooh and jeer at.

As usual, the reality is much stranger than fiction. Another cliche, but one whose reality those insiders in power will justify by ends-justify-the-means logic or that a fearful public who either do not care simply will accept as an inevitable stage in securing safety and health in a time age of wars on terror.