The NSA can reportedly collect 100 percent of an unnamed country's phone calls

[aGameOfThrones]

The NSA is capable of recording 100 percent of a country's telephone calls, according to sources who spoke with The Washington Post. Adding to the intrigue, the Post says it's "withholding details that could be used to identify the country where the system is being employed," implying that this surveillance method is used outside the US. It's also unclear whether the NSA's so-called MYSTIC voice-interception program (yep, that's the emblem above) is actually recording 100 percent of this country's calls, or whether it simply has the ability.

According to the story, the NSA also uses a tool called RETRO to retrieve audio that wasn't deemed suspicious at the time of its original recording. Apparently, RETRO is only used in one foreign country, but documents suggest it could soon be deployed in five more. With the tool, billions of conversations are stored with a 30-day buffer for clearing out the oldest calls as new ones come in.

It's not necessarily surprising that the agency would be reaching into the past as part of its data-collecting efforts, but it's certainly news that it can record and store every single telephone conversation. While we've heard about the NSA collecting call logs in bulk, recording the content of all voice calls is very different than stockpiling metadata.

http://www.engadget.com/2014/03/18/nsa-collecting-100-percent-countrys-calls/

[Philhelm]

The NSA is capable of recording 100 percent of a country's telephone calls, according to sources who spoke with The Washington Post. Adding to the intrigue, the Post says it's "withholding details that could be used to identify the country where the system is being employed," implying that this surveillance method is used outside the US.

Bullshit. It's the U.S.

[Zippyjuan]

Maybe the ability but:

It's also unclear whether the NSA's so-called MYSTIC voice-interception program (yep, that's the emblem above) is actually recording 100 percent of this country's calls, or whether it simply has the ability.

Lichtenstein would be an easy target.

[HOLLYWOOD]

Change thread title word "can" to "have"

[CPUd]

They wouldn't be able to store US calls in their own system, and if the day ever happens when they can, it won't be on anything that involves moving parts.

[KingNothing]

Is the unnamed country called Everycountryonearth?

[KingNothing]

Bullshit. It's the U.S.

I'm certain that you are correct. When has government ever had a fun tool it could use in secret, and not used it?

[FindLiberty]

GREAT, They can record and scan
everything for keywords, metadata;
almost have total surveillance, needs
just a little more...

Yes! No, we need more government
for more protection, just a little more
to protect the nation from turrrists,

...just a little more,

Yes! No, we need more government
for more protection, just a little more
to protect the nation from turrrists,

...just a little more,

Yes! No, we need more government
for more protection, just a little more
to protect the nation from turrrists,
just a little more...

[Mini-Me]

They wouldn't be able to store US calls in their own system, and if the day ever happens when they can, it won't be on anything that involves moving parts.

Oh, they could. Consider an extreme overestimate: There are 300 million people in the US, and every single person (including every infant) spends 24 hours a day on the phone. They're all speaking with someone internationally, not each other, so that's a full 7.2 billion hours of audio per day, or 2.628 trillion hours of audio per year. Let's say they're using the Opus codec at 16kbps, which is more than enough by a good bit. 2.628 trillion hours = 9.4608 quadrillion seconds of audio per year = 18.9216 quadrillion KB/year. If we convert upwards from KB to base-10 SI units (which hard drives are measured in basically), that's 18.9216 exabytes/year.

Conservative estimates of the new NSA data center's storage capacity are now at about 12 exabytes (although they were previously throwing around numbers in the yottabytes, which will take longer to achieve). So, if the NSA only stored audio data from phone calls, they would be restricted to storing 24-hour calls from 300 million Americans for only 7.61 months or so (on a continual replacement cycle). However, the according to a telephia survey (http://www.accuconference.com/blog/Cell-Phone-Statistics.aspx), the average American (probably meaning the average American of survey-taking age, so it's an overestimate) spends about 13 hours per month, or about 1.78% of the amount of time I figured in the extreme overestimate above. Based on this figure, if the NSA only stored audio data from phone calls, a conservative estimate of their new data center's capacity (12 exabytes) indicates they can store about 427.6 months (35.6 years) of all US calls concurrently in 16kbps Opus format.

I wasn't really tracking significant figures, and we should probably drop it to two sig-figs (because of the 12 exabytes and 13 hours/month figures), or even lower given the one sig-fig on the 300 million Americans figure, but...you get the idea.

Even before unified solid state memory, their capacity is going to get a lot bigger very quickly after the hard drive manufacturers master HAMR technology (which enables 60TB hard drives).

[ghengis86]

Oh, they could. Consider an extreme overestimate: There are 300 million people in the US, and every single person (including every infant) spends 24 hours a day on the phone. They're all speaking with someone internationally, not each other, so that's a full 7.2 billion hours of audio per day, or 2.628 trillion hours of audio per year. Let's say they're using the Opus codec at 16kbps, which is more than enough by a good bit. 2.628 trillion hours = 9.4608 quadrillion seconds of audio per year = 18.9216 quadrillion KB/year. If we convert upwards from KB to base-10 SI units (which hard drives are measured in basically), that's 18.9216 exabytes/year.

Conservative estimates of the new NSA data center's storage capacity are now at about 12 exabytes (although they were previously throwing around numbers in the yottabytes, which will take longer to achieve). So, if the NSA only stored audio data from phone calls, they would be restricted to storing 24-hour calls from 300 million Americans for only 7.61 months or so (on a continual replacement cycle). However, the according to a telephia survey (http://www.accuconference.com/blog/Cell-Phone-Statistics.aspx), the average American (probably meaning the average American of survey-taking age, so it's an overestimate) spends about 13 hours per month, or about 1.78% of the amount of time I figured in the extreme overestimate above. Based on this figure, if the NSA only stored audio data from phone calls, a conservative estimate of their new data center's capacity (12 exabytes) indicates they can store about 427.6 months (35.6 years) of all US calls concurrently in 16kbps Opus format.

I wasn't really tracking significant figures, and we should probably drop it to two sig-figs (because of the 12 exabytes and 13 hours/month figures), or even lower given the one sig-fig on the 300 million Americans figure, but...you get the idea.

Even before unified solid state memory, their capacity is going to get a lot bigger very quickly after the hard drive manufacturers master HAMR technology (which enables 60TB hard drives).

Isn't math fun!?!?

[liberty2897]

This one sounds pretty good.. even though I'm sure they could compress it even more if they don't need to retain the original speakers vocal characteristics.

http://www.rowetel.com/blog/?page_id=452


Codec2 is an open source low bit rate speech codec designed for communications quality speech at 2400 bit/s and below. Applications include low bandwidth HF/VHF digital radio and VOIP trunking. Codec 2 operating at 2400 bit/s can send 26 phone calls using the bandwidth required for one 64 kbit/s uncompressed phone call. It fills a gap in open source, free-as-in-speech voice codecs beneath 5000 bit/s and is released under the GNU Lesser General Public License (LGPL).

Also, Utah is just one new location..

[Mini-Me]

Isn't math fun!?!?
A little... ;)
Speaking of which, the US population is more like 317 million than 300 million, so we're looking at 404.7 months of concurrent US call storage rather than 427.6, but...yeah, the point it really could be the US.

[CPUd]

Oh, they could. Consider an extreme overestimate: There are 300 million people in the US, and every single person (including every infant) spends 24 hours a day on the phone. They're all speaking with someone internationally, not each other, so that's a full 7.2 billion hours of audio per day, or 2.628 trillion hours of audio per year. Let's say they're using the Opus codec at 16kbps, which is more than enough by a good bit. 2.628 trillion hours = 9.4608 quadrillion seconds of audio per year = 18.9216 quadrillion KB/year. If we convert upwards from KB to base-10 SI units (which hard drives are measured in basically), that's 18.9216 exabytes/year.

Conservative estimates of the new NSA data center's storage capacity are now at about 12 exabytes (although they were previously throwing around numbers in the yottabytes, which will take longer to achieve). So, if the NSA only stored audio data from phone calls, they would be restricted to storing 24-hour calls from 300 million Americans for only 7.61 months or so (on a continual replacement cycle). However, the according to a telephia survey (http://www.accuconference.com/blog/Cell-Phone-Statistics.aspx), the average American (probably meaning the average American of survey-taking age, so it's an overestimate) spends about 13 hours per month, or about 1.78% of the amount of time I figured in the extreme overestimate above. Based on this figure, if the NSA only stored audio data from phone calls, a conservative estimate of their new data center's capacity (12 exabytes) indicates they can store about 427.6 months (35.6 years) of all US calls concurrently in 16kbps Opus format.

I wasn't really tracking significant figures, and we should probably drop it to two sig-figs (because of the 12 exabytes and 13 hours/month figures), or even lower given the one sig-fig on the 300 million Americans figure, but...you get the idea.

Even before unified solid state memory, their capacity is going to get a lot bigger very quickly after the hard drive manufacturers master HAMR technology (which enables 60TB hard drives).

It looks good on paper, and a well-written grant proposal and a couple powerpoints could probably get this funded, but the logistics of implementing it pose a few challenges:

- power consumption and heat dissipation for a lot of hardware crammed into a relatively small space- probably the easiest one to deal with.

- drive failure rate: 12 exabytes would require several million drives, and with an average 1-3% failure rate per year (assume they initially install 1 exabyte per month), they could be replacing several hundred drives per day.

- encoding and indexing: I'm not entirely convinced this can be done in real time, depends on what they are using to do it. Specialized hardware can probably do it, but it would then be restricted by the speed of the bus they are connected to. This is a problem I know they are actively trying to solve.


Then, there is the issue of being able to actually use the data. I know what they would like to do, but they are not there yet. If private industry would take a real stand against them, it would slow them down. The bad press I think is also hurting their ability to attract talent, but if those guys want you bad enough, they won't come asking.

[liberty2897]

Of course NSA would probably just do an ASIC, but off-the-shelf hardware is getting pretty fast these days...


http://www.xilinx.com/support/documentation/data_sheets/ds180_7Series_Overview.pdf

Built on a state-of-the-art, high-performance, low-power (HPL), 28 nm, high-k metal gate (HKMG) process technology, 7 series FPGAs enable an
unparalleled increase in system performance with 2.9 Tb/s of I/O bandwidth, 2 million logic cell capacity, and 5.3 TMAC/s DSP, while consuming 50% less
power than previous generation devices to offer a fully programmable alternative to ASSPs and ASICs.

Wouldn't want to be the one laying the board out or soldering (1925-pin BGA)... one mistake could cost a lot of paychecks worth..

[Mini-Me]

It looks good on paper, and a well-written grant proposal and a couple powerpoints could probably get this funded, but the logistics of implementing it pose a few challenges:

- power consumption and heat dissipation for a lot of hardware crammed into a relatively small space- probably the easiest one to deal with.

- drive failure rate: 12 exabytes would require several million drives, and with an average 1-3% failure rate per year (assume they initially install 1 exabyte per month), they could be replacing several hundred drives per day.

- encoding and indexing: I'm not entirely convinced this can be done in real time, depends on what they are using to do it. Specialized hardware can probably do it, but it would then be restricted by the speed of the bus they are connected to. This is a problem I know they are actively trying to solve.


Then, there is the issue of being able to actually use the data. I know what they would like to do, but they are not there yet. If private industry would take a real stand against them, it would slow them down. The bad press I think is also hurting their ability to attract talent, but if those guys want you bad enough, they won't come asking.

This is a few days old, but...
Hasn't the new NSA data center in Colorado already been built and put to use? That's the one I'm talking about with the conservative 12 exabyte figure, not some hypothetical future data center. It surely requires a lot of drive replacements, but that's something a few well-trained employees can handle (and with their budget...).

Also, real-time voice encoding is perfectly possible with a CPU-based software implementation: In fact, the Opus codec (and the underlying SILK codec for low-bitrate voice) was specifically designed for low-latency real-time streaming. Even the earlier Speex codec worked for real-time streaming with software encoding. Moreover, the encoding doesn't have to be done at the data center itself: It can be done at the tapping points as the calls are being routed through the system, and they'd install sufficient processing capacity to handle the full throughput of the cell tower, etc. Real-time reencoding probably would require ASIC's given the high volume, but that wouldn't be a problem for the NSA. Then again, it's not like cell towers (etc.) transmit uncompressed signals or anything either nowadays, do they? The NSA may just copy an already-encoded stream as-is.

[tangent4ronpaul]

Oh, they could. Consider an extreme overestimate: There are 300 million people in the US, and every single person (including every infant) spends 24 hours a day on the phone. They're all speaking with someone internationally, not each other, so that's a full 7.2 billion hours of audio per day, or 2.628 trillion hours of audio per year. Let's say they're using the Opus codec at 16kbps, which is more than enough by a good bit. 2.628 trillion hours = 9.4608 quadrillion seconds of audio per year = 18.9216 quadrillion KB/year. If we convert upwards from KB to base-10 SI units (which hard drives are measured in basically), that's 18.9216 exabytes/year.

Conservative estimates of the new NSA data center's storage capacity are now at about 12 exabytes (although they were previously throwing around numbers in the yottabytes, which will take longer to achieve). So, if the NSA only stored audio data from phone calls, they would be restricted to storing 24-hour calls from 300 million Americans for only 7.61 months or so (on a continual replacement cycle). However, the according to a telephia survey (http://www.accuconference.com/blog/Cell-Phone-Statistics.aspx), the average American (probably meaning the average American of survey-taking age, so it's an overestimate) spends about 13 hours per month, or about 1.78% of the amount of time I figured in the extreme overestimate above. Based on this figure, if the NSA only stored audio data from phone calls, a conservative estimate of their new data center's capacity (12 exabytes) indicates they can store about 427.6 months (35.6 years) of all US calls concurrently in 16kbps Opus format.

I wasn't really tracking significant figures, and we should probably drop it to two sig-figs (because of the 12 exabytes and 13 hours/month figures), or even lower given the one sig-fig on the 300 million Americans figure, but...you get the idea.

Even before unified solid state memory, their capacity is going to get a lot bigger very quickly after the hard drive manufacturers master HAMR technology (which enables 60TB hard drives).

We are creating 21 zetabytes of data a year (2011) and have 2.1 billion people on the Internet (2011). The GCHQ doc said they kept audio of calls for 30 days.

How Much Data Is Created Every Minute? [INFOGRAPHIC]
http://rack.1.mshcdn.com/media/ZgkyMDEyLzA2LzIyLzA2XzU0XzE0XzYzM19maWxlCnAJdGh1bW IJMTIwMHg5NjAwPg/1c7d4fe7



- drive failure rate: 12 exabytes would require several million drives, and with an average 1-3% failure rate per year (assume they initially install 1 exabyte per month), they could be replacing several hundred drives per day.

Was watching a talk about the development of MapReduce and the guy got into the early days at google. caseless computers on the cheapest shelving they could get, i got the impression they were on beer flats, but not sure, however the hd's were velcro'd to the system. They wouldn't change drives as needed, but rather just take a machine off the active list if it didn't send results in a specified time. so 2-3 times a day they go out and service the misbehaving list. efficient.

a couple of other huge speedups were not waiting the usual time to try and re-read a sector it didn't pick up as a normal drive does and disabling the last accessed timestamp write.

-t

[CPUd]

We are creating 21 zetabytes of data a year (2011) and have 2.1 billion people on the Internet (2011). The GCHQ doc said they kept audio of calls for 30 days.

How Much Data Is Created Every Minute? [INFOGRAPHIC]
http://rack.1.mshcdn.com/media/ZgkyMDEyLzA2LzIyLzA2XzU0XzE0XzYzM19maWxlCnAJdGh1bW IJMTIwMHg5NjAwPg/1c7d4fe7



Was watching a talk about the development of MapReduce and the guy got into the early days at google. caseless computers on the cheapest shelving they could get, i got the impression they were on beer flats, but not sure, however the hd's were velcro'd to the system. They wouldn't change drives as needed, but rather just take a machine off the active list if it didn't send results in a specified time. so 2-3 times a day they go out and service the misbehaving list. efficient.

a couple of other huge speedups were not waiting the usual time to try and re-read a sector it didn't pick up as a normal drive does and disabling the last accessed timestamp write.

-t

I used to use these to mount my systems:

http://i.imgur.com/vwqxN3i.jpg

[tangent4ronpaul]

I really screwed up the numbers. probably confused a couple of diff ones. it's 1.8 zettabytes (2011) however the data in the world doubles every year, so it should be around 7 zettabytes today.

-t

[aGameOfThrones]

We are creating 21 zetabytes of data a year (2011) and have 2.1 billion people on the Internet (2011). The GCHQ doc said they kept audio of calls for 30 days.

How Much Data Is Created Every Minute? [INFOGRAPHIC]
http://rack.1.mshcdn.com/media/ZgkyMDEyLzA2LzIyLzA2XzU0XzE0XzYzM19maWxlCnAJdGh1bW IJMTIwMHg5NjAwPg/1c7d4fe7



Was watching a talk about the development of MapReduce and the guy got into the early days at google. caseless computers on the cheapest shelving they could get, i got the impression they were on beer flats, but not sure, however the hd's were velcro'd to the system. They wouldn't change drives as needed, but rather just take a machine off the active list if it didn't send results in a specified time. so 2-3 times a day they go out and service the misbehaving list. efficient.

a couple of other huge speedups were not waiting the usual time to try and re-read a sector it didn't pick up as a normal drive does and disabling the last accessed timestamp write.

-t

Where is the porn data on that pic?

[Peace Piper]

Kingston DataTraveler HyperX Predator 1TB USB 3.0 Flash Drive

http://ecx.images-amazon.com/images/I/81cZ%2B1fyHpL._SL1500_.jpg

Product Dimensions: 2.8 x 1.1 x 0.8 inches ; 1.6 ounces

No moving parts

http://ecx.images-amazon.com/images/I/71lN319K9%2BL._SL1500_.jpg

Price: $1,233.36

[tangent4ronpaul]

Dang! You could put a small data center in a suite case with those!

I think I'll wait till the price comes down though... ;)

-t

[milgram]

So here's Friday's Democracy Now interview with Ashkan Soltani, the WaPo researcher


https://www.youtube.com/watch?v=ru8-xs70IhI


ASHKAN SOLTANI: And an often kind of—or a common theme is that the NSA hits up on technology barriers, so limits to storage, limits to bandwidth, limits to processing power. And that’s when we see the information in the slides that we have.

And that’s really telling, since it highlights a kind of a larger problem, which is that the limitations are not legal or policy restrictions, right? The limitations are technical restrictions. And as technical capacity goes up, I think the NSA will be kind of growing these programs.

...

AMY GOODMAN: "At least one country," explain that.

ASHKAN SOLTANI: So, we—so, the documents we had were from last year, and they indicated a system up and running. And they would kind of go from, you know, earlier coverage to 100 percent, once—when we saw the documents. But they hinted at the expansion of this program in at least one other place by October 2013, October of last year. And there was kind of hints in the documents and in the budget documents that this capacity would be growing to other places as soon as the technical capacity was there.

[tangent4ronpaul]

Where is the porn data on that pic?

no one really agrees, but 10-30% are common claims. about 20% of Skype traffic is cybering/sexting.

this analysis says 30%
http://www.extremetech.com/computing/123929-just-how-big-are-porn-sites

the largest porn site, xvideo gets twice the traffic of redit and transfers 50GB of data a second.

oh - and the average fap? 5 1/2 mins

-t

[CPUd]

http://i.imgur.com/FBcBdQH.png