Re: Robot Discovery ) OT Banning Aggregators

["Bill Kearney" <wkearney99@hotmail.com>]

A valid point.  Determining what's coming into a site as the client 
is often difficult to perform.  It's certainly something that anyone 
engaging in 'unwanted' behavior can get around with very little 
effort.

While I share your concern about the big-brother is watching 
situation, there's no way to avoid it.  If the material is on the web 
and freely available, you can't rely on obscurity to hide you.

What troubles me more is if a site provides content and expects to 
reap some benefit from people actually visiting the site.  Using an 
aggregator has the potential to completely sidestep visits to the 
site.  Worse yet, it has the potential to misrepresent the data's 
authorship.  I'm sure this is going to bother a potential content 
provider a lot more than being ultra-indexed by an external service.

When I spoke of a service being able to ask the content source for 
the data in an XML stream, I was considering these situations.  It 
might be early to talk about this but such an interface is bound to 
need user authentication or other policies for certain data.  Using 
this interface as a means to just suck out the content is something 
to be concerned about.

-Bill Kearney