Making Syndication Enterprise-Grade
Saturday, 16 July 2005
After more than five years, syndication is maturing rapidly. It’s being used for more than blogging — whether it be stock quotes, system logs, or order lists — and even blogging will change in nature as it gets more popular; people will be using blogs to fundamentally change the way they do business, inside and outside the firewall.
In the enterprise, people are going to look at syndication as a pub/sub replacement. While feeds over HTTP won’t be able to replace pub/sub systems in low-latency scenarios, they will be able to when a lag of a few tens of minutes (or less, depending) doesn’t matter.
In fact, syndication has a number of benefits over pub/sub, just as it does over e-mail. Mostly, this is because syndication doesn’t require close coordination between the publisher and the subscriber; in fact, unless authentication is used, the publisher doesn’t know who the subscriber is.
This means that the server doesn’t have to keep a list of subscribers, with all of the management and robustness concerns that entails. The subscribers poll when they want information, and stop polling when they’re done.
It allows the client greater mobility around the network, without worrying about updating their address with the server. It completely avoids the conundrum of server retry and back-off algorithms.
It means that clients are easy to write in any language, and that scalability means deploying simple, commodity Web caches, rather than duplicating expensive, proprietary servers.
In short, it means making pub/sub Web-like.
Secure and Reliable
However, there’s a catch; enterprise users are generally (although not always) more demanding than a fourteen-year-old with a crush on a boy band. They want security and reliability.
While Atom defines security mechanisms, and there are some interesting efforts to retrofit it onto RSS, reliability is a bit tricker. Right now, when you poll a feed, you have no idea if there were any missed entries between the last time you looked.
For the business, this is a serious failing. There’s no way you’re going to use a technology to carry business-critical information if, as a matter of course, it can “forget” some.
I think this is the most critical thing that we need to make syndication Enterprise-ready, and it’s why I’ve proposed the Feed History mechanism.
In a nutshell, it allows you to reconstruct the entire feed by “paging” back through the archives. It works in both RSS and Atom, and I’ve implemented it in this feed using plain, vanilla Moveable Type template tags. Now, I’m looking for other implementation experiences; if you put it in your blog, or your aggregator software, please comment below, or send me an e-mail.
6 Comments
Seth Russell said:
Tuesday, July 19 2005 at 3:29 AM
Henry Story said:
Tuesday, July 19 2005 at 10:23 AM
Meryn said:
Wednesday, July 20 2005 at 1:27 AM
Roger Benningfield said:
Thursday, July 21 2005 at 3:53 AM
Mark Nottingham said:
Saturday, July 23 2005 at 9:16 AM
William Henry said:
Friday, January 6 2006 at 10:56 AM