mark nottingham

Making Syndication Enterprise-Grade

Saturday, 16 July 2005

Web Feeds

After more than five years, syndication is maturing rapidly. It’s being used for more than blogging — whether it be stock quotes, system logs, or order lists — and even blogging will change in nature as it gets more popular; people will be using blogs to fundamentally change the way they do business, inside and outside the firewall.

In the enterprise, people are going to look at syndication as a pub/sub replacement. While feeds over HTTP won’t be able to replace pub/sub systems in low-latency scenarios, they will be able to when a lag of a few tens of minutes (or less, depending) doesn’t matter.

In fact, syndication has a number of benefits over pub/sub, just as it does over e-mail. Mostly, this is because syndication doesn’t require close coordination between the publisher and the subscriber; in fact, unless authentication is used, the publisher doesn’t know who the subscriber is.

This means that the server doesn’t have to keep a list of subscribers, with all of the management and robustness concerns that entails. The subscribers poll when they want information, and stop polling when they’re done.

It allows the client greater mobility around the network, without worrying about updating their address with the server. It completely avoids the conundrum of server retry and back-off algorithms.

It means that clients are easy to write in any language, and that scalability means deploying simple, commodity Web caches, rather than duplicating expensive, proprietary servers.

In short, it means making pub/sub Web-like.

Secure and Reliable

However, there’s a catch; enterprise users are generally (although not always) more demanding than a fourteen-year-old with a crush on a boy band. They want security and reliability.

While Atom defines security mechanisms, and there are some interesting efforts to retrofit it onto RSS, reliability is a bit tricker. Right now, when you poll a feed, you have no idea if there were any missed entries between the last time you looked.

For the business, this is a serious failing. There’s no way you’re going to use a technology to carry business-critical information if, as a matter of course, it can “forget” some.

I think this is the most critical thing that we need to make syndication Enterprise-ready, and it’s why I’ve proposed the Feed History mechanism.

In a nutshell, it allows you to reconstruct the entire feed by “paging” back through the archives. It works in both RSS and Atom, and I’ve implemented it in this feed using plain, vanilla Moveable Type template tags. Now, I’m looking for other implementation experiences; if you put it in your blog, or your aggregator software, please comment below, or send me an e-mail.