Re: [syndication] Re: robots.txt and rss

["Mark Nottingham" <mnot@mnot.net>]

My .02 -

Aggregators probably should respect robots.txt, for the reason that Dan
points out. Robots.txt is a mechanism for *all* agents/robots/whatever,
not just spiders - the criteria is whether the request is a direct result
of a human action.

That having been said, robots.txt is a woefully inadequate mechanism, so
it's of limited use. Still, probably enough - there are quite a few
libraries out there to make it easy.


----- Original Message -----
From: "Dan Brickley" <daniel.brickley@bristol.ac.uk>
To: <syndication@yahoogroups.com>
Sent: Friday, November 08, 2002 8:58 AM
Subject: Re: [syndication] Re: robots.txt and rss


> On Fri, 8 Nov 2002, Ben Hammersley wrote:
>
> > On Friday, November 8, 2002, at 04:02  pm, Dave Winer wrote:
> >
> > >> opinion:  RSS aggregators probably *should* respect robots.txt.
> > >
> > > I agree. If someone were to post a BDG, like the one [1] Simon Fell
> > > posted for Etags, I would put together support for it in Radio
> > > (assuming it was easy, which I assume it is).
> > >
> >
> > Why would you have an rss feed, and an robots.txt that restricts using
> > it? Just don't publish an RSS feed.
>
> You might want to restrict certain (disfunctional, annoying etc)
> user-agents, eg. if they poll impolitely often.
>
> Dan
>
>
>
>
> Your use of Yahoo! Groups is subject to
http://docs.yahoo.com/info/terms/
>
>