[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

RSS vs. HTML Bandwidth and "Scalability"...

From w3future.com/weblog/:

>In July my RSS file has been downloaded 15741 times. That's 134Mbyte, 55%
>percent of my total traffic. This is way to much, if you compare that to the
>2617 times my html weblog has been downloaded last month. This looks like a
>scalability problem. But I have a monthly traffic limit of 1500MByte, so I
>don't worry.
Initially, I was "hey! what's the problem? people care more about your content than the pretty design! be happy!". But as I started to write exactly that, I shifted quickly.
He makes a bit of an interesting point when you think. Beside search engine spiders and proxies, I can think of no magical programs that hit a website time and time again to get updates, when there aren't updates to be had.
On the other hand, most "constant on" RSS aggregators hit websites every hour to get the latest updates. However, I'm not really sure how Meerkat or NewsIsFree.com handles it (or for that matter, xmltree.com). 

So, what's the solution to this pointless waste of possible bandwidth?

 a) Embed the time limit in the RSS file. This has been
    allowed in the old MS CDF format, as well as in
    scriptingNews (i think). The big problems is that
    aggregators don't listen to them, since there's a
    stunning lack of adoption. Sadly, I fall into this
    group with my AmphetaDesk too. I know Jeff Barr's
    Headline Viewer has an internal option on when
    to update; however, I don't know the default.

 b) Check the HTTP headers from the server. This would
    only work if the content wasn't dynamic, which is
    rare nowadays. For a while now, I've been thinking
    of checking content-length's / filesizes and
    comparing for newness.

 c) Implement server control - block repetitive ip's
    on a cron'd schedule and allow them back in when
    the going gets happy. This shifts the "blame"
    onto the server people though, and we really shouldn't
    be making RSS maintenance any harder than it is.

What are your thoughts? Any additions to the above?


--
Morbus Iff ( i am your scary godmother )
http://www.disobey.com/ && http://www.gamegrene.com/
please me: http://www.amazon.com/exec/obidos/wishlist/25USVJDH68554
icq: 2927491 / aim: akaMorbus / yahoo: morbus_iff / jabber.org: morbus