Re: [syndication] Contacting Aggregators

[Aaron Swartz <aswartz@swartzfam.com>]

Steven Roussey <sroussey@network54.com> wrote:

> The thing is, I'd like to warn our aggregators first, but I don't know who
> they are. Is there already a mechanism to for aggregators to register that I
> have yet to set up? Or should aggregators send a courtesy email to
> webmaster@.. when they pull more than X number of feeds from a single site?

Perhaps aggregators should add a header in the HTTP request, identifying
themselves? Or maybe a special User-Agent field? Something of the form:

Aggregator: name/version (http://www.site.for.more.info/)

would work well. Also, is anyone running a registry of aggregators? It's
getting a little hard to keep track of them all. I know that XMLtree has a
listing of some of them, perhaps we could set something up on dmoz.org. If
we had a listing like the Robots list,
(http://info.webcrawler.com/mak/projects/robots/ IIRC) which is a listing of
User-Agent strings and email addresses for various web spiders, that would
be another way of dealing with the issue.

If there's interest, I'll set up a site where people can register their bot.

-- 
        Aaron Swartz         |"This information is top security.
<http://swartzfam.com/aaron/>|     When you have read it, destroy yourself."
  <http://www.theinfo.org/>  |             - Marshall McLuhan