Spies and XML

[Steven Roussey <sroussey@network54.com>]

I have recently begun an investigation into our XML syndication patterns,
and have come to a terrible conclusion.

Over 90% of our XML exports are going to a handful of IP numbers. I have
found that they go to either large multinationals (a foreign bank is the
biggest culprit) or inside government domains.

Since we are essentially an open discussion server, this gives them an
efficient mechanism to spy on users topics of conversation. And its causing
a heavy load on our servers.

They do not follow the general rule of one hit every six seconds like a
search engine. They want up-to-date info and hammer until they get it.

I realize that this is not unique to XML syndication, and that we ought to
watch IP numbers for heavy usage across the board. But providing information
in XML has made it easier for these people.

I have changed our Help documents on XML at our site and specify that any
domain that requests more than 100 XML docs a day will get banned unless
they send an email to support providing an explanation.

Does this seem reasonable? What are everyone else's experience on this?

Sincerely,

Steven Roussey
Network54.com
http://network54.com/?pp=e