Re: [syndication] Re: robots.txt and rss

[mof-syndication@mfd-consult.dk]

On Sat, 9 Nov 2002 01:14:59 -0800, "Mark Nottingham" <mnot@mnot.net> wrote:
>That having been said, robots.txt is a woefully inadequate mechanism, so
>it's of limited use. Still, probably enough - there are quite a few
>libraries out there to make it easy.
Indeed.

My homegrown aggregator makes use of the Perl LWP::RobotUA module, and
then I don't even have to think about it - if a request is not
allowed, it won't even try it.


Regards,

Morten Frederiksen
--- 
!07/11 PDP a ni deppart m'I  !pleH
-- 
<URL: http://www.mfd-consult.dk/ >