Re: [syndication] Re: robots.txt and rss

["Bill Kearney" <wkearney99@hotmail.com>]

> eh, but having more than one way to do the same kind of thing has never
> hurt *anyone*.  :)

Indeed, that was my tack on the in-band redirection arguments.  That's /still/
mired in useless arguing.

> not everyone gets to use apache, even when they want to.  nor do they
> necessarily have access to that much of its configuration, when they *do*
> get to use it.  robots.txt can be handy because all of the work happens on
> the client-side, rather than depending on the ability of the content
> provider to gain access to their server's configs... (think of all of the
> people with blogs running in "shared hosting" scenarios....)

Again, I'm right there with you.  This is my perspective on dealing with server
config issues.

Look, I don't disgree with you.  What I'm saying is that using robots.txt for
this purpose misuses it.  It "doesn't" hurt but it certainly doesn't help unless
the software takes it upon itself to handle it.  This is extremely unlikely.

Here's why.  If a program has been cobbled together such that it barely works
it's unlikely the developer is going to add /new/ functionality to it.
Especially since robots.txt is a text file with a whole range of formatting
possiblities.  So here you'd be asking a program geared to process XML to add
new code that would also handle text parsing.  Having written code I can tell
you it's a non-starter asking that to happen.

And there's no consistently magic layer or proxies inbetween that will help you
either.

Yes, something IS worth doing.  What's worth doing is seeking out the defective
code and getting it fixed or replaced.  Education of the developers seems like
it will go a lot further than attempts to work around dumb code.

-Bill Kearney