On Tue, Mar 30, 2004 at 12:36:10PM -0500, Dave Winer wrote:
Isn't it a bit ironic that Google News, which is itself a scraped service,
is protecting itself from scrapers?
It's a little bit of a gray area. Historically it's been easy for web
sites to stop google from spidering them by using robots.txt. Obviously
they wouldn't want to do that, and google is leveraging that to do its
news scraping. It is somewhat hypocritical for google to now send email
to people asking them to stop doing the same. I guess if the NY times
sends a cease and desist letter to google, they'll stop scraping.