[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

robots.txt and rss

To: syndication@yahoogroups.com
Subject: robots.txt and rss
From: Alex McLintock <alex@arcfan.demon.co.uk>
Date: Fri, 08 Nov 2002 12:28:17 +0000

Hi folks,

I've taken to using a robots.txt file to ban (polite) web crawlers from mysite because I've got memory problems and can't handle them thrashing my box.

Does this mean that RSS fetchers will no longer be fetching my rss files?

Should I explicitly allow my RSS files in my robots file?


Alex

Follow-Ups:
- Re: [syndication] robots.txt and rss
  - From: "Bill Kearney" <wkearney99@hotmail.com>

Prev by Date: Re: [syndication] Preserving white space in elements?
Next by Date: Re: [syndication] robots.txt and rss
Previous by thread: Re: [syndication] Preserving white space in elements?
Next by thread: Re: [syndication] robots.txt and rss
Index(es):
- Date
- Thread