Re: [syndication] Re: robots.txt and rss

["Bill Kearney" <wkearney99@hotmail.com>]

And what would you expect the aggregator to do?  There's nothing in a robots.txt
file that would 'help' an aggregator manage it's use of a resource.

The FIRST thing that would help is for a reader program like Radio to start
being more respectful of resources and allow per-feed scheduling.  Right now
Radio just hammers the feeds every hour on the hour.  And when RSS-1.0 feeds
take the time to include syn module scheduling Radio just ignores them.  It only
recently started supporting header checking but it's still doing it every hour
regardless of the feed's requested schedule.

So how about fixing Radio before you run off suggesting something like
robots.txt support?

-Bill Kearney

----- Original Message -----
From: "Dave Winer" <dave@userland.com>
To: <syndication@yahoogroups.com>
Sent: Friday, November 08, 2002 11:02 AM
Subject: [syndication] Re: robots.txt and rss


> > opinion:  RSS aggregators probably *should* respect robots.txt.
>
> I agree. If someone were to post a BDG, like the one [1] Simon Fell
> posted for Etags, I would put together support for it in Radio
> (assuming it was easy, which I assume it is).
>
> Dave
>
> [1]
> http://www.pocketsoap.com/weblog/stories/2002/05/19/bdgToEtags.html
>
>
>
>
> Your use of Yahoo! Groups is subject to http://docs.yahoo.com/info/terms/
>