mark nottingham

How (Not) to Control Your CDN

Wednesday, 7 June 2017

HTTP Caching HTTP

In February, Omer Gil described the Web Cache Deception Attack.

In a nutshell, it goes like this: when your CDN (or reverse proxy) applies caching policy to responses based upon the URL, there’s an opportunity to get something unintentionally cached – and thereby made public – if the server’s understanding of the responses it’s serving is even slightly different.

For example, if your CDN is configured to cache everything ending in .css and your server treats /user/personal-info/foo.css the same as /user/personal-info, you’ve got a problem. Or, if you serve “default content” for non-existant pages to be “helpful”, you’ve got a problem. Don’t laugh – Omer found this on Paypal.

Joshua Liebow-Feeser at Cloudflare wrote about how to protect yourself from this:

The best way to defend against this attack is to ensure that your website isn’t so permissive, and never treats requests to nonexistent paths (say, /x/y/z) as equivalent to requests to valid parent paths (say, /x). In the example above, that would mean that requests to /newsfeed/foo or /newsfeed/foo.jpg wouldn’t be treated as equivalent to requests to /newsfeed, but would instead result in some kind of error or a redirect to a legitimate page.

To me, however, that feels like a band-aid. Although the attack is described as being about caching, it’s actually more general; any time you have out-of-band policy applied rather than having the policy actually contained in the response itself, there’s a chance for it to be out of sync, causing misconfiguration and potential vulnerabilities.

So while Omer’s example is getting something cached when it wasn’t intended to be, it could equally be something like injecting processing or transformation such as ESI, adding new response headers, and so on. Considering how security sensitive that can be (see: CSP), site operators really need to think creatively here; the attackers surely will be.

What’s Up With All The Control Panels?

There’s some context that’s important to understand at this point.

It’s become popular for CDNs to offer “control panels” where administrators can add lots of fancy processing to their sites. They generally work by keeping some sort of separate configuration file that gets applied to requests and responses as them come in. Full disclosure: I’ve contributed to at least one of these systems.

Depending on how that file is managed, there might be a fair amount of lag between when a change is made and it’s applied to messages on the wire, leading to the problems outlined above.

But that’s not the worst thing. What’s worse is that for most site, the person who has administrative access to the configuration for the site and the person actually creating the content and assigning it URLs are in completely different parts of the company. I’ve personally experienced this, to the point where we couldn’t even find the person who was responsible for configuring our company’s CDN.

As a result, keeping the site’s CDN configuration in sync with the actual content being served takes a super-human effort of discipline and coordination. Even in the best case where the developer/content creator has access to the control panel, they have to get the mapping right in every case (including the subtleties that Omer points out). Given the power of most modern intermediation platforms, this is a pretty tall order.

However, CDNs still keep on creating control panels – I suspect mostly because administrators are used to having control panels, and it’s become a checkbox feature.

If that’s the case, we need to think more carefully about how we use CDN control panels, and what kinds of power they expose. I’m hoping the dangers that Omer illustrates so graphicallyq motivates that change in thinking.

A Better Way

One of the axiomatic things in HTTP is that messages are self-describing. Any time you are consulting an external authority, there is an opportunity for it to be out of sync with what actually happens on the wire.

So, respect this. Where your CDN allows it, use in-band control mechanisms – usually HTTP response headers – to control response handling like caching policy, content transformation, header additions and deletions, and so forth.

Unfortunately, there isn’t a uniform standard for such a header (Surrogate-Control is out there, but it’s not universal, and it’s pretty horrible; mea culpa). It’d be good if the CDNs could get together here. For caching, you might consider using plain old Cache-Control.

Of course, it’s not practical to convey policy for a request in-band, so you’ll still need to use the control panel for request-side things like routing, finding the origin, and fallback behaviours when a cached response is not available. That’s OK, I think; the impact of bad request routing is much more obvious to the developer than some of the subtle conditions we’re talking about here, and it’s simpler (often, it’s just mapping one URL space to another).

If you can discipline yourself to do this, your policy will be co-located with the content or code generating it, thereby making your site more secure and less error-prone.

On their side, CDNs could offer a “safe mode” for configuration where their control panel will only affect request-side behaviour. They could also defer to in-response policy when it’s available, and assure that the out-of-band control mechanism be as available to the actual developer/content creator as possible; I’d suggest a well-known URL that the CDN fetches.