I'm still quite astonished that (according to some of the comments) it's fairly standard that traffic between datacenters is not encrypted.
Granted we're talking about extremely large throughput, but I thought Google of all companies would have invested in routers capable of doing this or at least gone to the trouble of designing their own hardware that does it; since Google is no stranger to this already.
Why would routers do this? IPSec didn't exactly take off (this weeks news reminded us why), so end-to-end encryption wouldn't really happen on their level.
That said, I certainly expected and assumed Google would already encrypt traffic between data centers. Whats the point of forcing HTTPS on gmail when you constantly backup my complete email repository across the world over unencrypted connections?
In this new context, the statements on Prism ("we do not give them direct access!") certainly seem misleading. Right, you do not give them direct access, you just sync your databases over fibers that you know they have access to, without encrypting the data.
> Whats the point of forcing HTTPS on gmail when you constantly backup my complete email repository across the world over unencrypted connections?
To protect users on questionable public WiFi, or traveling through a country known for spying on its telecommunications, or who might be using an ISP with untrustworthy employees, or working for a company with a nosy IT department.
Until recently, the USA wasn't generally considered part of that second category, and private leased lines were generally considered secure -- at Google and elsewhere -- for the same reason a LAN transmission within a datacenter is / was generally considered secure.
Speaking of untrustworthy employees, and HUMINT, the NSA/CIA could just have agents infiltrated at Google, to get access to a lot of that data.
This is what's so striking about this. I thought Google already encrypted all data, and only a few people had access to it. Didn't they say this many years ago?
"Encrypt all data" is a vague term. For example, would that include encrypting it while a CPU is processing it? What about when the CPU is writing it to RAM? What if the RAM is on some other computer in the rack? What if it's in some computer on the other side of the world?
I'm not sure what Google's public statements on this have been, but if you want to research it, be sure to distinguish statements regarding user data at rest from those regarding user data in transit.
"the NSA/CIA could just have agents infiltrated at Google, to get access to a lot of that data" -- this would be really difficult, because no single person at Google can just sudo in some system and feast over sensitive data. Any access to user data or other highly sensitive data needs to go through a stack of authorizations, the granted access will be scoped, and we have obsessive amounts of logging in place even for trivial stuff let alone sensitive systems and data. One thing that surprised me with the Snowden leaks is that the NSA's "checks and balances" seems to be orders of magnitude inferior to what we have at Google; something like that is unimaginable here (esp. if you can believe the NSA in that they don't even know which documents were accessed). Even a conspiracy involving several infiltrated googlers at the right places couldn't do it without anyone noticing, because they'd need to sabotage code and configurations that live in repositories that thousands of engineers have access to (and have enforced code-review workflows); the internal openness of our systems is in fact among our best defenses against sabotage.
IPsec (s is in lowercase) is the standard to securing L2 connectivity and it has been ubiquitously used for site-to-site and client-to-site connectivity for ages. In addition to several mature FOSS implementations, every network equipment vendor ships one. There is also a ton of client software - Windows supported it since Windows 2000, the SSH company (you know, the ssh creators) has been selling an IPsec Toolkit since as early as 1999, Cisco has a VPN client that is de-facto software in Cisco-based shops for remote workers, etc.
> > IPSec didn't exactly take off
> IPsec (s is in lowercase) is the standard to securing L2 connectivity and it has been ubiquitously used for site-to-site and client-to-site connectivity for ages
The question wasn't whether the standard exists but whether it's widely used. Your claims don't sound plausible to me as I've yet to work on a network (corporate, .edu, .gov) where IPSec is used – everyone focused on protocol level security like SSH or SSL instead.
What raidi and other said. But also, do not assume that we didn't encrypt any of the backend backbone transfers, or that some specific kind of information was exposed before -- for one thing, when we make inter-DC backups/replication of data that's already stored in encrypted form (you gmail folders and such), it's likely that we just ship this data around without bothering to decrypt and re-encrypt it, which would be wasteful and pointless. (I'm not a intimate with netops here, just making educated guesses like anybody could do.) Also there's some significant data that needs no encryption, e.g. the gobs of public youtube content that we have to mirror and cache in a thousand places.
We have tons of our own stuff moving through the same pipes (proprietary source code, all files in our corp network filesystems...), so it's our best interest to protect these. I suspect most of the unencrypted traffic is actually what the NSA would call "metadata" -- if valuable information can be mined from simple metadata like phone calls, I guess even better stuff can be derived from extremely rich RPCs/protobufs even if the core information was already in encrypted fields. Anyway the more comprehensive encryption support should turn our backbone into a wasteland for spooks.
In this context eavesdropping is probably far more likely than MITM.
Quite apart from MITM being an active attack only needed (ish) for manipulation of the transferred data, at the throughputs likely involved here and the fact that leased lines are used, MITM would probably provide far too much hassle for its worth.
Agreed, I first thought the title said that Google was decreasing the key size of the crypto to speed up the traffic, and that the story was a negative one.
I work for TeliaSonera and I can tell you that it's very standard for us to use VPN connections between worldwide datacenters.
If Google comes up with a much more sane version of or alternative to IPsec, deployed on all their boxes, it would be an amazing improvement for the world.
Especially if they integrated it with Chrome (a la VPN over SSL). If I have to pick between Google having all my traffic and the NSA, its an easy pick.
Self hosting is somewhat better, if nothing else because it presents a much less tempting target, and it is a privilege currently only available to the technical.
Well implemented strong encryption is another avenue.
Especially if you design everything about individual components being potentially either backdoored or just buggy, and make it hard for failures to lead to successful outside penetration or inside data exfiltration. It doesn't help if everything is compromised fully, but if some things are not, they can usually block or at least detect problems. Defense in depth, and all.
Yep, for better or worse it's a digital life now. Good luck opting out of the cloud for your bank transactions, your health insurance claims, Amazon orders, etc. Our emails, chats and cat videos are but a small part of sensitive data that we have stored in somebody else's data centers; cynicism and Luddite rejection of the few items we can "take back" like email servers will not help much. The only solution is to fix the system. At Google, and other conscious internet services, we're doing everything we can. But users and voters have to help too. For example, yesterday it was revealed that the NSA has been able to successfully MITM users connecting to SSL-protected Google services. Now I don't want to use this to plug Google Chrome, but if you're using any browser without all bleeding-edge security features like cert pinning and PFS, then you deserve to be 0wned by various high-profile hackers.
Lack of opportunistic encryption (in the style of Hugh Daniels Free S/WAN).
General complexity and baroqueness.
IKE and various implementations -- keying in general. True, keying is a hard problem, but I'd prefer a simple default which was secure against passive eavesdroppers, and then progressively better systems.
No, since they don't fall under CALEA. It's possible Google Voice specific stuff would need CALEA access, but I'm sure they handle that at a higher level in the application.
Google needs this kind of stuff not to defend against just NSA but also every other intelligence agency out there. For 0.1% of the IC's annual budget, I could give a third-tier country (Belgium? Nigeria?) about 5% of NSA's capability. That, IMO, is the true risk here.
This is _not_ end-to-end crypto, that would require users storing the keys to their own emails on their own systems. This is basically TLS on a giant scale but does not prevent email from being intercepted while "at rest" on a gmail server.
I'm still quite astonished that (according to some of the comments) it's fairly standard that traffic between datacenters is not encrypted.
Granted we're talking about extremely large throughput, but I thought Google of all companies would have invested in routers capable of doing this or at least gone to the trouble of designing their own hardware that does it; since Google is no stranger to this already.
Why would routers do this? IPSec didn't exactly take off (this weeks news reminded us why), so end-to-end encryption wouldn't really happen on their level.
That said, I certainly expected and assumed Google would already encrypt traffic between data centers. Whats the point of forcing HTTPS on gmail when you constantly backup my complete email repository across the world over unencrypted connections? In this new context, the statements on Prism ("we do not give them direct access!") certainly seem misleading. Right, you do not give them direct access, you just sync your databases over fibers that you know they have access to, without encrypting the data.
> Whats the point of forcing HTTPS on gmail when you constantly backup my complete email repository across the world over unencrypted connections?
To protect users on questionable public WiFi, or traveling through a country known for spying on its telecommunications, or who might be using an ISP with untrustworthy employees, or working for a company with a nosy IT department.
Until recently, the USA wasn't generally considered part of that second category, and private leased lines were generally considered secure -- at Google and elsewhere -- for the same reason a LAN transmission within a datacenter is / was generally considered secure.
Speaking of untrustworthy employees, and HUMINT, the NSA/CIA could just have agents infiltrated at Google, to get access to a lot of that data.
This is what's so striking about this. I thought Google already encrypted all data, and only a few people had access to it. Didn't they say this many years ago?
"Encrypt all data" is a vague term. For example, would that include encrypting it while a CPU is processing it? What about when the CPU is writing it to RAM? What if the RAM is on some other computer in the rack? What if it's in some computer on the other side of the world?
I'm not sure what Google's public statements on this have been, but if you want to research it, be sure to distinguish statements regarding user data at rest from those regarding user data in transit.
"the NSA/CIA could just have agents infiltrated at Google, to get access to a lot of that data" -- this would be really difficult, because no single person at Google can just sudo in some system and feast over sensitive data. Any access to user data or other highly sensitive data needs to go through a stack of authorizations, the granted access will be scoped, and we have obsessive amounts of logging in place even for trivial stuff let alone sensitive systems and data. One thing that surprised me with the Snowden leaks is that the NSA's "checks and balances" seems to be orders of magnitude inferior to what we have at Google; something like that is unimaginable here (esp. if you can believe the NSA in that they don't even know which documents were accessed). Even a conspiracy involving several infiltrated googlers at the right places couldn't do it without anyone noticing, because they'd need to sabotage code and configurations that live in repositories that thousands of engineers have access to (and have enforced code-review workflows); the internal openness of our systems is in fact among our best defenses against sabotage.
> IPSec didn't exactly take off
Oh, jezus. Did you read it on the Internets?
IPsec (s is in lowercase) is the standard to securing L2 connectivity and it has been ubiquitously used for site-to-site and client-to-site connectivity for ages. In addition to several mature FOSS implementations, every network equipment vendor ships one. There is also a ton of client software - Windows supported it since Windows 2000, the SSH company (you know, the ssh creators) has been selling an IPsec Toolkit since as early as 1999, Cisco has a VPN client that is de-facto software in Cisco-based shops for remote workers, etc.
IPsec, which runs at layer 3, secures IP (layer 3) traffic.
To protect L2 connectivity with IPsec, you'll need to tunnel the l2 frames inside IP.
You're right. And he's right too.
Standalone IPsec is supposedly nearly non-existent, but there's a plenty of L2TP/IPsec traffic out there.
> > IPSec didn't exactly take off > IPsec (s is in lowercase) is the standard to securing L2 connectivity and it has been ubiquitously used for site-to-site and client-to-site connectivity for ages
The question wasn't whether the standard exists but whether it's widely used. Your claims don't sound plausible to me as I've yet to work on a network (corporate, .edu, .gov) where IPSec is used – everyone focused on protocol level security like SSH or SSL instead.
Mail messages may go through multiple relays. Even if they are encrypted between relays, all routing mailers see the message unencrypted.
Your only option to secure your mail messages is end-to-end encryption. Do you really trust Google or any other company more than you trust the NSA?
SMTP-TLS should fix that, but then the problem is that many email providers don't support that.
What raidi and other said. But also, do not assume that we didn't encrypt any of the backend backbone transfers, or that some specific kind of information was exposed before -- for one thing, when we make inter-DC backups/replication of data that's already stored in encrypted form (you gmail folders and such), it's likely that we just ship this data around without bothering to decrypt and re-encrypt it, which would be wasteful and pointless. (I'm not a intimate with netops here, just making educated guesses like anybody could do.) Also there's some significant data that needs no encryption, e.g. the gobs of public youtube content that we have to mirror and cache in a thousand places.
We have tons of our own stuff moving through the same pipes (proprietary source code, all files in our corp network filesystems...), so it's our best interest to protect these. I suspect most of the unencrypted traffic is actually what the NSA would call "metadata" -- if valuable information can be mined from simple metadata like phone calls, I guess even better stuff can be derived from extremely rich RPCs/protobufs even if the core information was already in encrypted fields. Anyway the more comprehensive encryption support should turn our backbone into a wasteland for spooks.
Yes, this seems like a huge man-in-the-middle vulnerability.
In this context eavesdropping is probably far more likely than MITM.
Quite apart from MITM being an active attack only needed (ish) for manipulation of the transferred data, at the throughputs likely involved here and the fact that leased lines are used, MITM would probably provide far too much hassle for its worth.
These are private circuits between datacentres, and as such have always been in the "assumed secure" category.
Agreed, I first thought the title said that Google was decreasing the key size of the crypto to speed up the traffic, and that the story was a negative one.
I work for TeliaSonera and I can tell you that it's very standard for us to use VPN connections between worldwide datacenters.
Anyone who wants their mail to remain encrypted until it's read by the recipient should use GPG.
The PGP encrypted emails I sent were very secure - no one ever read them!
If Google comes up with a much more sane version of or alternative to IPsec, deployed on all their boxes, it would be an amazing improvement for the world.
Especially if they integrated it with Chrome (a la VPN over SSL). If I have to pick between Google having all my traffic and the NSA, its an easy pick.
Except of course, as we've learnt over the last few weeks, they're essentially one and the same thing.
You can't trust a third party with your data if you want it kept secure. Period.
You can't trust a third party with your data if you want it kept secure. Period.
It's impossible to do otherwise, though.
That is the rub of course.
Self hosting is somewhat better, if nothing else because it presents a much less tempting target, and it is a privilege currently only available to the technical.
Well implemented strong encryption is another avenue.
Even with self-hosting, there's a lot of third-party hardware and software that you have to trust.
Absolutely, but it changes the economics substantially in your favour.
Especially if you design everything about individual components being potentially either backdoored or just buggy, and make it hard for failures to lead to successful outside penetration or inside data exfiltration. It doesn't help if everything is compromised fully, but if some things are not, they can usually block or at least detect problems. Defense in depth, and all.
Yep, for better or worse it's a digital life now. Good luck opting out of the cloud for your bank transactions, your health insurance claims, Amazon orders, etc. Our emails, chats and cat videos are but a small part of sensitive data that we have stored in somebody else's data centers; cynicism and Luddite rejection of the few items we can "take back" like email servers will not help much. The only solution is to fix the system. At Google, and other conscious internet services, we're doing everything we can. But users and voters have to help too. For example, yesterday it was revealed that the NSA has been able to successfully MITM users connecting to SSL-protected Google services. Now I don't want to use this to plug Google Chrome, but if you're using any browser without all bleeding-edge security features like cert pinning and PFS, then you deserve to be 0wned by various high-profile hackers.
What exactly don't you like about IPsec?
Lack of opportunistic encryption (in the style of Hugh Daniels Free S/WAN).
General complexity and baroqueness.
IKE and various implementations -- keying in general. True, keying is a hard problem, but I'd prefer a simple default which was secure against passive eavesdroppers, and then progressively better systems.
At this point everything they could come up with, would have the feature to be accessible by the NSA, don't you think?
No?
No, since they don't fall under CALEA. It's possible Google Voice specific stuff would need CALEA access, but I'm sure they handle that at a higher level in the application.
Google needs this kind of stuff not to defend against just NSA but also every other intelligence agency out there. For 0.1% of the IC's annual budget, I could give a third-tier country (Belgium? Nigeria?) about 5% of NSA's capability. That, IMO, is the true risk here.
This is _not_ end-to-end crypto, that would require users storing the keys to their own emails on their own systems. This is basically TLS on a giant scale but does not prevent email from being intercepted while "at rest" on a gmail server.
let's not pretend these emails are sitting in plaintext on google's servers though. they are "fragmented and obfuscated" at rest.
The timing suggests this is supposed to be a PR move. And a cheap/ridiculous one at that.
Don't know why you are downvoted, but it's clearly a damage control move.
Damage control yes, but not necessarily external.
Google probably knows even though they're mostly in good terms, this may change.
I guess they got uncomfortable with people knowing too much already and don't trust too much
If you were to actually read the article, you'll note that the project began _before_ the Snowden revelations started coming out.
It's all about protecting the data from non-US interests, Google has probably already given the keys to the NSA.