Re: [syndication] robots.txt and rss

["Mark Nottingham" <mnot@mnot.net>]

> As you're no doubt aware, the level of developer understanding about
caching and
> proxing is very low.  It's unfortunate.  But a bit of digging reveals a
TON of
> work has already been done in these areas.  I'd strongly suggest others
read up
> on it.

If that's an aspect to this problem, certainly. Might I suggest:
  http://www.web-caching.com/
as a starting point. Brian does a good job of collecting information in
that industry. A more academic, but still incredibly useful, resource is
Craig Wills' Webbib;
  http://www.cs.wpi.edu/~webbib/