Re: [syndication] robots.txt and rss

["Mark Nottingham" <mnot@mnot.net>]

> > maybe i want to block out specific software users or types of users,
and
> > allow individuals to keep doing what they're doing.
>
> And robots.txt has NO mechanism for doing that.  It's an all user-agent
or
> nothing approach.

Not true; the whole point of robots.txt is that I can say "if you identify
yourself as the U-A Foo, don't go in this directory."

The problem is that robots.txt isn't any more refined than U-A and some
primitive URI matching. What is needed is a way to introduce new criteria
for matching, like time of day, IP address, and so forth, so that you can
describe how you want the robot to behave, rather than just outright ban
it.

This is one of the use cases I had in mind for URISpace [1]. I doubt
whether any robots.txt replacement will get much traction, however,
because it's so widespread.

1. http://www.w3.org/TR/urispace