Re: [syndication] robots.txt and rss

["Bill Kearney" <wkearney99@hotmail.com>]

> opinion:  RSS aggregators probably *should* respect robots.txt.

How so?  How is the robots.txt file germaine to a reader's behavior?

http://www.robotstxt.org/wc/faq.html#what

The only way a robots.txt file is going to have any relevance here is that the
robots.txt file could indicate that a particular user-agent should NOT load from
within a given part of the hierarchy.  This would be equivalent to not having
the feed available.

If you're interested in blocking the hammering of a feed then you'd need to use
other means to do so.  Ban the IP address of the offending client machine.  Or
use the server mechanisms to detect the user-agent and block it that way.

But you're on a slipperly slope here is you speak of banning user-agents in a
wholesale manner and that's all robots.txt would allow.

-Bill Kearney