Re: [syndication] RSS vs. HTML Bandwidth and "Scalability"...

[Julian Bond <julian_bond@voidstar.com>]

How much of a problem is this really? Consider a few scenarios:-

1) A public aggregator, like my.userland.com, meerkat, newsisfree or
drupal collects a feed once an hour, stores it locally and then serves
it up to it's readers. A source colleted by all of them is going to get
hit maybe 10-15 times an hour as there just aren't that many of them.
Anyone building something like this will be taking care not to be a
nuisance to avoid having the feeds pulled. We should all really put a
bit more effort in to doing minimal collections, but these systems are
not really the problem.

2) Private aggregators and readers like Radio, Amphetadesk, Headline
viewer. Each instance of the reader will hit a feed when it starts up
and once an hour or so while it's running. If these hit the big time and
got 100,000 downloads each, then the most popular sources could get hit
quite badly. There's a strong case here for the authors of the packages
to do everything they can to avoid hammering the feed sources. They
certainly should be doing lots of local cacheing and respecting last
modified information. Let's say there are 1000 copies of Amphetadesk,
all of which collect news from slashdot. Slashdot is going to get an
additional ~100 hits per hour on average, peaking at 1000. Even if we go
up two orders of magnitude, it's still not that bad. If slashdot make
sure that the file is static and proxy cacheable, they'll hardly notice. 

3) Sidebar systems. The sort of javascript thing that displays a feeds
headlines in another site's side column. If this does no local cacheing,
which it probably doesn't, then every hit on the display site will be a
hit on the feed. If X sites get Y hits, the feed source gets hit X*Y
times. And if Site A gets popular and has to upgrade the hardware to
cope, then Feed F gets the same problem. This is exactly why Moreover
spent so much money on scalability and hardware. But actually this is no
different from the common practice of putting a logo gif up on your site
and encouraging people to point to it. If we all forced Scripting news
to host the XML gif and just put a pointer to it on our incredibly
popular sites instead of taking a copy and serving it ourselves, then
Userland would get justifiably upset. But even in this case, we're
talking about 1-10Kb of Gif or rss.xml, the same order of magnitude. How
bad can it get, really.

-- 
Julian Bond    email: julian_bond@voidstar.com
CV/Resume:         http://www.voidstar.com/cv/
WebLog:               http://www.voidstar.com/
HomeURL:      http://www.shockwav.demon.co.uk/ 
M: +44 (0)77 5907 2173  T: +44 (0)192 0412 433
ICQ:33679668 tag:So many words, so little time