Major Cloudflare bug leaked sensitive data from customers websites

Cloudflare also posted about this, look here.

Quote:Cloudflare revealed a serious bug in its software today that caused sensitive data like passwords, cookies, authentication tokens to spill in plaintext from its customers websites. The announcement is a major blow for the content delivery network, which offers enhanced security and performance for more than 5 million websites.

This could have allowed anyone who noticed the error to collect a variety of very personal information that is typically encrypted or obscured.

Remediation was complicated by an additional wrinkle. Some of that data was automatically cached by search engines, making it particularly difficult to clean up the aftermath as Cloudflare had to approach Google, Bing, Yahoo and other search engines and ask them to manually scrub the data.

The leak may have been active as early as Sept. 22, 2016, almost five months before a security researcher at Googles Project Zero discovered it and reported it to Cloudflare.

However, the most severe leakage occurred between Feb. 13 and Feb. 18, when around 1 in every 3,300,000 HTTP requests to Cloudflare sites would have caused data to be exposed. Attackers could have accessed the data in real-time, or later through search engine caches.

Cloudflare notes in its announcement of the issue that even at its peak, data only leaked in about 0.00003% of requests. It doesnt sound like much, but Cloudflares massive customer base includes categories like dating websites and password managers, which host particularly sensitive data.

?At the peak, we were doing 120,000 leakages of a piece of information, for one request, per day, Cloudflare chief technology officer John Graham-Cumming told TechCrunch. He emphasized that not all of those leakages would have contained secret information. ?Its random stuff in there because its random memory, he said.

The bug occurred in an HTML parser that Cloudflare uses to increase website performance it preps sites for distribution in Googles publishing platform AMP and upgrades HTTP links to HTTPS. Three of Cloudflares features (email obfuscation, Server-side Excludes and Automatic HTTPS Rewrites) were not properly implemented with the parser, causing random chunks of data to become exposed.

Ultimately, even Cloudflare itself was affected by the bug. ?One obvious piece of information that had leaked was a private key used to secure connections between Cloudflare machines, Graham-Cumming wrote in Cloudflares announcement. The encryption key allowed the companys own machines to communicate with each other securely, and was implemented in 2013 in response to concerns about government surveillance.

Graham-Cumming emphasized that Cloudflare discovered no evidence that hackers had discovered or exploited the bug, noting that Cloudflare would have seen unusual activity on their network if an attacker were trying to access data from particular websites.

?It was a bug in the thing that understands HTML, Graham-Cumming explained. ?We understand the modifications to web pages on the fly and they pass through us. In order to do that, we have the web pages in memory on the computer. It was possible to keep going past the end of the web page into memory you shouldnt be looking at.

Cloudflares teams in San Francisco and London handed off shifts to one another, working around the clock to fix the bug once it was reported. They had stopped the most severe issue within seven hours. It took six days for the company to completely repair the bug and to work with search engines to scrub the data.

Tavis Ormandy, an engineer at Google, first noticed the bug, which he jokingly called ?Cloudbleed in reference to the Heartbleed vulnerability. He said in a blog post that he encountered unexpected data during a project and wondered at first if there was a bug in his own code. Upon further testing, he realized the leak was coming from Cloudflare.

?We fetched a few live samples, and we observed encryption keys, cookies, passwords, chunks of POST data and even HTTPS requests for other major Cloudflare-hosted sites from other users, Ormandy wrote. ?This situation was unusual, [personally-identifiable information] was actively being downloaded by crawlers and users during normal usage, they just didnt understand what they were seeing. Ormandy added that he later destroyed the samples because of the sensitive information they contained, but he posted redacted screenshots of some of the information leaked from Uber, Fitbit and OkCupid.

Beyond the samples Ormandy collected, its not clear what other information may have leaked. ?Its very hard to say, because this information is transient, Graham-Cumming said. But Ormandy says his samples revealed highly sensitive data.

?We keep finding more sensitive data that we need to cleanup. I didnt realize how much of the internet was sitting behind a Cloudflare CDN until this incident, Ormandy wrote. ?Im finding private messages from major dating sites, full messages from a well-known chat service, online password manager data, frames from adult video sites, hotel bookings. Were talking full HTTPS requests, client IP addresses, full responses, cookies, passwords, keys, data, everything.

Although Cloudflare worked with Ormandy to address the issue, he contends that the companys final blog post on the matter ?severely downplays the risk to customers. Ormandy also expressed frustration that Cloudflare didnt move faster in the remediation process.

But Graham-Cumming says it wouldnt have been possible for Cloudflare to work any more quickly than it did. Graham-Cumming also says that Ormandy called Cloudflares disclosure ?completely acceptable when he reviewed a copy.

?This is subject to a 90 day disclosure. We were disclosing after six days, Graham-Cumming said. ?Hes saying hes frustrated but Im a little bemused at why hes frustrated with six days rather than 90. We would have disclosed even earlier, but because some of this info had been cached, we thought we had a duty to clean that up before it became public. There was a danger that info would persist in search engines like Google.

Graham-Cumming said that Cloudflare customers like Uber and OkCupid werent directly notified of the data leaks because of the security risks involved in the situation. ?There was no backdoor communication outside of Cloudflare only with Google and other search engines, he said.

Source

In addition to this, here's a comprehensive list of sites that use Cloudflare: https://github.com/pirate/sites-using-cloudflare
If you see a site that you have an account in on that last, best change your passwords and enable 2FA.

Here are some sites that use Cloudflarethat are most likely relevant to the NBB community:

On a good note, NBB wasn't impacted :cool:

How much of a risk do we have if we DON'T change our passwords?

24-02-2017, 05:48 PM

Nyan Wrote: How much of a risk do we have if we DON'T change our passwords?

Very tiny. Only 770 leaks were found that were cached by search engines(not too sure about cache part).
When those caches expire and no one has a copy of them you're technically safe.

24-02-2017, 05:56 PM

Faded Wrote: Very tiny. Only 770 leaks were found that were cached by search engines(not too sure about cache part).
When those caches expire and no one has a copy of them you're technically safe.

Sweet, there were a lot of sites that I have accounts on on the list...
I'm just going to wait. :-)

I read that when i woke up, they fucked up

One of the members on the forums I generally lurk on posted this same information there. I use a pretty shit password for a lot of the random sites I am on and the ones like netflix, paypal, etc. all have different randomly generated passwords. Ive found this is the best way to stay secure since this kind of stuff keeps happening.

25-02-2017, 09:55 AM

Protodarthbot Wrote: One of the members on the forums I generally lurk on posted this same information there. I use a pretty shit password for a lot of the random sites I am on and the ones like netflix, paypal, etc. all have different randomly generated passwords. Ive found this is the best way to stay secure since this kind of stuff keeps happening.

But problems arise when many of those sites share a single service that had a major security flaw. Case in point.
But you do follow good practice with different passwords. Use 2FA if you can

This is crazy, but its a common thing now a days. That's why I use different passwords for every site.