what is cgi_buffer?

cgi_buffer is a group of libraries (for Perl, Python and PHP) that automatically handle performance-improving HTTP features that most Web servers don't implement (at least for scripts).

Currently, these include:

Best of all, cgi_buffer is easy to use - it only requires a one-line include in your script. The Python version can also be used as a wrapper around any other CGI program, enabling the benefits for other languages as well.

See the code examples for an idea of what using cgi_buffer involves.

using cgi_buffer

Ideal conditions for cgi_buffer include pages that have a lot of included objects (which will benefit from persistent connections), large HTML and text pages (which will benefit from compression), and objects which don't change every time they are generated (which will benefit from ETag validation, although this isn't yet supported by many clients).

In particular, cgi_buffer tends to help with SSL sites, bad connectivity (for instance, if you have a lot of international users) and highly interactive sites.

To install and use cgi_buffer, see the instructions in the appropriate language's subdirectory; there may be some requirements for supporting libraries. Using cgi_buffer in your script is as easy as import cgi_buffer, or the equivalent.

The Python library can also be used as a wrapper around other CGI programs, so that you can cgi_buffer C programs (for example). See the Python library's documentation for more information.

obtaining cgi_buffer

cgi_buffer is freely available under the same terms as the Python license.

Documentation is also available online;


Why doesn't cgi_buffer do Last-Modified/If-None-Match validation?
This would require specific information (i.e., the last modified time) from your application for every response; cgi_buffer focuses on automatically handling HTTP features with no changes to the code. This may be supported in a future version, however.
How do I include HTTP headers in my response?
Just as you would normally; as long as cgi_buffer is included before you write the HTTP headers, it will automatically collect them.
How do I control the features (e.g., ETag generation) of cgi_buffer?
Each library has its own means of controlling individual features; see the README documentation specific to your language.
My Web page changes every time it's generated. Do ETags cause problems with caching because of this?
No. Unlike Last-Modified, ETag doesn't allow a cache to replay a response based on time since it was changed; instead, the cache will go forward to the origin server for every request to validate. If your page really changes every time it's accessed, no matter what, you might want to turn off ETags to be friendly to caches (since they'll still be stored).
My application streams a response to the client gradually, so that it can update as it goes. Can I still use cgi_buffer?
No. cgi_buffer stores the entire response to enable its features; this interferes with these (very rare) applications.
Persistent connections don't seem to be working; is cgi_buffer broken?
Probably not; cgi_buffer just inserts the Content-Length header; it's up to your Web server to use it to effect a persistent connection. In particular, IIS 4.0 is known not to do this.
Does Gzip compression cause extra load on my server?
Some, but probably not as much as you think; gzip is a very efficient algorithm, and it's being used at a fairly low compression setting to reduce overhead.
How do I tell if cgi_buffer is working?
Any tool that shows you HTTP headers can be used; the cacheability engine also shows where persistent connections, and future versions will also check ETag validation and compression support.