Re: [syndication] Re: robots.txt and rss

[Ian Davis <iand@internetalchemy.org>]

On Friday, 08 November 2002 at 16:51, Ben Hammersley wrote:

> Why would you have an rss feed, and an robots.txt that restricts using 
> it? Just don't publish an RSS feed.
You may want to deny access to the feed to certain aggregators,
perhaps ones that aggressively fetch it every 10 minutes.

Although, since robots.txt is just a convention, if the aggressive
aggregator wants your feed it'll just take it anyway.

I'm inclined to say that robots.txt is intended for use by robots that
are spidering for new links. The typical aggregator just fetches the
files it has been told to fetch and doesn't go poking around other
parts of the site so shouldn't need to respect robots.txt. After all
there aren't any web browsers that support robots.txt, and a rss
reader is just a specialised form of web browser.

Ian