mark nottingham

The Resource Expert Droid

Thursday, 25 June 2009

HTTP Caching HTTP

A (very) long time ago, I wrote the Cacheability Engine to help people figure out how a Web cache would treat their sites. It has a few bugs, but is generally useful for that purpose.

However, as I’ve got more involved in using HTTP for non-browser things, it’s become apparent that more than caching is important when you’re examining a resource to see how it behaves; things like partial content, syntax checking and other esoteric but important details. Very often, I’d find myself manually debugging a RESTful Web service with telnet — and as they say, that doesn’t scale.

Looking back at that decade-old code, I decided that rather than fixing it up (“lipstick” and “pig” are two words that come to mind), I’d rewrite. The result, after quite a few evenings and weekends, is the Resource Expert Droid.

In a nutshell, RED is a framework for testing HTTP resources; it fetches responses, analyses them, and then based upon the responses it may interact with the resource more to see how it behaves. In this manner, it’s very purposefully encouraging RESTfulness.

Note that I say “resources”, not “servers.” Since a single server can serve you content in a number of different ways (think plain files vs. CGI vs. mod_autoindex), you need to test on a per-resource granularity when you have problems. Of course, this observation isn’t new.

A few examples for fun (please take it easy on these!):

Make sure you hover over the messages in the list for a full explanation of each.

REDbot is Open Source, and hosted at Github. It’s nowhere near finished yet, there’s still lots more to do (see the issues list), but contributions and suggestions are more than welcome.


15 Comments

jgraham said:

The first time I clicked on the apache.org link I got a UnicodeDecodeError (ascii codec cannot… and so on; the usual thing). I can’t reproduce it though.

That said, it looks like a useful tool, although I’m not sure about the reveal-stuff-on-hover UI; it is rather distracting to have the side text changing as you mouse over different parts of the page.

Thursday, June 25 2009 at 7:53 AM

Philip Taylor said:

“The server’s clock is 5 sec behind.” - oops, yes, looks like I forgot to install ntpd. Thanks for the reminder!

The problem I have with the hover UI is that I don’t have much vertical screen space (1280x800, minus toolbars etc), and if I scroll to the bottom message then the response headers are off the top of the screen. Maybe it would be better if it was possible to ‘lock’ the mouseover selection by clicking on an item, so it wouldn’t change when you move the mouse away to scroll.

Thursday, June 25 2009 at 8:53 AM

Steve Clay said:

This will be very helpful. Only glaring bug I see is it never seems to recognize “Vary: Accept-Encoding”.

Friday, June 26 2009 at 1:53 AM

Carl Meyer said:

Sweet tool, thanks for creating it.

@Steve Clay – it recognized Vary: Accept-Encoding fine for me, once I actually made sure it was being served. Ubuntu 8.04 nginx 0.5 didn’t add that header, I had to upgrade to nginx 0.6 from backports and add “gzip_vary on;” to my config (apparently nginx doesn’t turn this on by default due to possible problems with IE <= 6).

Friday, June 26 2009 at 3:16 AM

Xiven said:

Very nice tool. It doesn’t seem to like my “Vary: Accept,Accept-Encoding” header though: “This response was negotiated for gzip content encoding, so the Vary header needs to contain Accept-Encoding, the request header used.”

Friday, June 26 2009 at 6:23 AM

MikeD said:

Very very cool Many thanks!

Friday, June 26 2009 at 7:00 AM

Kevin Reid said:

It seems to incorrectly think 301 responses shouldn’t have a body; RFC 2616 says “Unless the request method was HEAD, the entity of the response SHOULD contain a short hypertext note with a hyperlink to the new URI(s).” which implies the request has a body but:

http://redbot.org/?uri=http%3A%2F%2Fwaterpoint.org%3A8080%2Focean_maps

“HTTP/1.0 301 Moved Permanently

Connection This response is not allowed to have a body.”

Saturday, June 27 2009 at 4:12 AM

Chris Ferris said:

These are not the droids you are looking for?

Seriously, this is a very useful tool.

Saturday, June 27 2009 at 8:47 AM

Mark said:

I’ve been testing on diveintopython3.org, which is nothing but static HTML served by a (relatively) stock Apache server. I’ve gotten everything to go green except this: “The ETag doesn’t change between representations.” I understand the explanation, but how do I fix it? I have full .htaccess rights on the server, but no combination of FileETag options seems to help.

Wednesday, July 1 2009 at 2:18 AM

Mark said:

Another question: how do I remove the “Cache-Control: public” header? I’m using the following .htaccess in my images directory:

ExpiresActive On ExpiresDefault “access plus 1 year”

That seems to include the “public” keyword in the Cache-Control header, which I suppose makes sense as a default (since, according to your explanation, there are some cases where it does something useful). But if I know that I’ll never hit the cases that you mention, how can I get Apache to remove it?

I tried adding the following .htaccess directives before and after the Expires* directives listed above:

Header unset Cache-Control Header set Cache-Control “max-age=31536000”

Neither seems to have any effect, presumably because Apache is adding the Expires and Cache-Control directives after .htaccess is processed.

Wednesday, July 1 2009 at 2:28 AM