Re: [syndication] General purpose scrapers and bandwidth

[Ian Davis <iand@internetalchemy.org>]

On Thursday, 22 May 2003 at 16:03, Morbus Iff wrote:
>  >>RSSify is PHP? Got bandwidth/usage stats? Can I see the code?

> Julian - any words on the bandwidth/usage stats?

If you're interested in general usage of scrapers here are the
last 30 days stats for myRSS. Currently it's scraping 5649 sites into
various forms of RSS although that number will decrease as I've just
implemented an expiry procedure for old sites.

Analysed requests from Tue-22-Apr-2003 00:01 to Thu-22-May-2003 00:01
(30.00 days).

Successful requests: 3,400,253
Average successful requests per day: 113,341
Successful requests for pages: 251,502
Average successful requests for pages per day: 8,383
Distinct files requested: 43,957
Distinct hosts served: 55,253
Data transferred: 9.127 gigabytes
Average data transferred per day: 311.566 megabytes


Most referers are from netnewswire, followed by radio and then google.



- Ian <iand@internetalchemy.org>
"Imagination is more important than knowledge."