Re: [syndication] Re: robots.txt and rss

["Mark Nottingham" <mnot@mnot.net>]

> my point was simply that encouraging robots.txt support won't do
> anything to solve this class of problem users, which dan seemed to be
> suggesting. i'm still skeptical it will solve any class of problem
> users, so only see the (albeit marginal) downside of having popular
> aggregation tools checking my robots.txt file on the very slim chance
> that someday some actual problem user might implement robots.txt
> handling correctly.

I tend to agree; robots.txt is best suited for stopping agents from
accessing a resource before they access it, and for making statements
about large numbers of resources, for efficiency (rather than putting a
statement in each one). For stopping problems with specific feeds
in-flight (i.e., something is already accessing them), it's not very
useful.