RE: [syndication] Headline Viewers

["Carmen" <chv@vertexdev.com>]

Our Headline Viewer (www.headlineviewer.com) does asynchronous
fetching of XML from multiple feeds in parallel. It has a
regulation mechanism to control the number of outstanding
fetches. This is not currently domain-based. It will not hit 
the same URL more than once per hour per user.
 
There is an interesting tension here -- the robot exclusion
model is great for batch-oriented applications that can gather
data over a "long" (in Internet-time, e.g. an hour or two)
period of time. It is potentially not so great for a program
such as Headline Viewer.

People have talked about some kind of caching or distribution
model for feeds. We'd be interested in using such a model, and
perhaps in helping to architect it.

Carmen

Try Headline Viewer at http://www.headlineviewer.com

-----Original Message-----
From: Gleb Dolgich [mailto:glebd@kaunas.omnitel.net]
Sent: Friday, August 25, 2000 12:39 AM
To: syndication@egroups.com
Subject: Re: [syndication] Headline Viewers


Hello,

Novobot (http://www.proggle.com/novobot) gets XML feeds sequentially, but
may do it several times for one provider in succession. I would be glad to
hear your suggestions on how to make it more provider-friendly.

Thanks.
Gleb Dolgich (glebd@proggle.com)
http://www.proggle.com
The Web is News with Novobot!

----- Original Message -----
From: "Steven Roussey" <sroussey@network54.com>
To: <syndication@egroups.com>
Sent: Friday, August 25, 2000 00:38
Subject: [syndication] Headline Viewers


> Hi all!
<...>
> Soooo, there must be headline viewers out there that go and get all the
> feeds in rapid succession, even if they are from the same site (in which
> case they are ignoring the common robot protocol for spacing out
requests).
<...>
> Does anyone have a list of Headline Viewers? The one that really hurts is
> Java based and is likely European or at least not English based. The IPs
are
> generally in Europe and some in South America.
<...>
> Sincerely,
>
> Steven Roussey
> Network54.com
> http://network54.com/?pp=e
>