Web Server Capabilities

Mark Nottingham
mnot@pobox.com

Products which use HTTP/1.1 have been demonstrated to run significantly faster than those which do not. I urge everyone to check for HTTP/1.1 compliance when choosing software.”
- Tim Berners-Lee, upon the approval of HTTP/1.1 as a Draft Standard on http://www.w3.org/

Although most popular Web servers now claim HTTP/1.1 compliance, it is unclear to what extent they support the enhanced features of the protocol, or how consistently they do so.

Specifically, I’m interested in the consistency of support for protocol features with generated content— that is, content that the server must process before serving, either through an external or internal process.

Anecdotal evidence suggests that such content doesn’t enjoy full protocol support; this is borne out by Wills’ study, which found that roughly a third of HTML representations examined did not have a validator associated with them, even though they did not change.

Understanding these issues is vital to making the Web work efficiently, as more Web sites use such non-traditional means of managing content.

To further explore the level of support offered, I undertook a survey of current Web servers[pdf], to test for protocol compliance, with a focus on caching and other performance-enhancing mechanisms.

Scope

Because a server can source content by any of several methods (such as CGI, SSI, ASP, JSP and Cold Fusion, to name a few), I decided to work at a higher granularity than by-server. A few widely-deployed servers were chosen and divided into the various content delivery mechanisms that they implement.

The primary protocol features tested for were:

• Setting of response metadata by publishers and administrators - such as Expires times, Cache-Control headers and arbitrary HTTP response headers.
• Validation - by Last-Modified/If-Modified-Since and/or ETag/If-None-Match.
• Persistent Connection support - either by use of a Content-Length header or chunked encoding, for HTTP/1.1 clients.
• Partial Content - through Range requests.
• Transfer Encoding - for compression.

Methodology

During testing, all requests were HTTP/1.1, except for Content-Length based persistent connections.

• Setting of metadata by publishers and administrators - I searched both the publisher and administrator interfaces (detailed below) for evidence of a means of setting these parameters.
• Validation - each validation method was tried; if a content generation system did not supply a validator to use, I attempted to add my own by appending the appropriate header.
• Persistent Connections - by Content-Length for HTTP/1.0 clients (with Connection: keep-alive), and chunked encoding for HTTP/1.1 clients. If a Content-Length header was not generated, I attempted to append one.
• Partial Content - with a Range: bytes=x-y request header.
• Transfer Encoding - with a TE: gzip, deflate, compress request header.

Notes

In the publisher and administrator control categories, partial support in the “Expires” or “Cache-Control” fields indicatest that support for that function is through the “arbitrary headers” function; in other words, the header must be entered manually.

The interfaces cited for setting these parameters are described below.

• FrontPage - Microsoft’s page layout and site management tool.
• browser - any Web browser-based management tool.
• NetShare - Netscape’s Java applet that allows publishers to upload and manipulate resources.
• AolPress - AOL’s page layout and site management tool.
• mgmt console - Microsoft Management Console.
• .htaccess - Apache configuration files in the target directory, which can contain one of a number of directives.
• config - Apache server configuration files.

Other notes about support and individual behaviours are in the “Notes” column.

Observations

The survey results clearly illustrate that support for “core” protocol features such as validation, persistent connections and partial content is available in most servers, but is not consistent across all requests served by them. Additionally, publishers are not offered a consistent, easy-to-use method of assigning metadata to resources.

Particularly, although static content is fairly well covered, support for persistent connections, validation and partial responses falls off dramatically for generated content, even when using a dedicated server (such as Domino or Zope).