Re: [syndication] Contacting Aggregators

["james@xmlTree.com" <james@xmltree.com>]

Steven

> I think our forums have been picked up by some aggregators as I have seen
> our XML Exports grow greatly as of late. We are used to maybe a couple
> thousand a day but that has grown to over 200,000 a day this week.

If the XML is generated dynamically, this starts to amount to significant overhead.  I
have the same problem with some of my OCS files of available RSS channels.  Some
aggregators are insensitive and set up bots which collect RSS content every half hour,
without checking the HTTP expiry header or the <updateFrequency> element in the
corresponding OCS document.

> The thing is, I'd like to warn our aggregators first, but I don't know who
> they are. Is there already a mechanism to for aggregators to register that I
> have yet to set up? Or should aggregators send a courtesy email to
> webmaster@.. when they pull more than X number of feeds from a single site?

In my experience, these insensitive aggregators do not identify themselves through a
descriptive user agent string.  Of course, you could set up an identification scheme where
agents requesting RSS also registered first and provided an aggregatorID, but this is
bordering on the functionality that ICE sought to provide.  And what is an aggregator?
Could every user of Carmen's Headline Viewer be expected to register?  And since CHV is
built around the MSXML component, it can't provide a custom user agent string easily.

At xmlTree, we would love to understand the flows of content.  Who is syndicating from
who?  But in our opinion the premise of RSS is that content consumers should be able to
syndicate from content providers without intermediate gateways (beyond initial discovery).

My view is that if they have no user agent, and don't email you, you shouldn't concern
yourself too much in migrating from RSS 0.9 to 0.91.  Many aggregators have written code
that will work with both anyway.

Best regards,
James Carlyle

Calaba Limited
UK +44 (0)7720 468 986
http://www.xmltree.com - directory of XML content