mnot’s blog

Design depends largely on constraints.” — Charles Eames

Saturday, 16 July 2005

Making Syndication Enterprise-Grade

After more than five years, syndication is maturing rapidly. It’s being used for more than blogging — whether it be stock quotes, system logs, or order lists — and even blogging will change in nature as it gets more popular; people will be using blogs to fundamentally change the way they do business, inside and outside the firewall.

In the enterprise, people are going to look at syndication as a pub/sub replacement. While feeds over HTTP won’t be able to replace pub/sub systems in low-latency scenarios, they will be able to when a lag of a few tens of minutes (or less, depending) doesn’t matter.

In fact, syndication has a number of benefits over pub/sub, just as it does over e-mail. Mostly, this is because syndication doesn’t require close coordination between the publisher and the subscriber; in fact, unless authentication is used, the publisher doesn’t know who the subscriber is.

This means that the server doesn’t have to keep a list of subscribers, with all of the management and robustness concerns that entails. The subscribers poll when they want information, and stop polling when they’re done.

It allows the client greater mobility around the network, without worrying about updating their address with the server. It completely avoids the conundrum of server retry and back-off algorithms.

It means that clients are easy to write in any language, and that scalability means deploying simple, commodity Web caches, rather than duplicating expensive, proprietary servers.

In short, it means making pub/sub Web-like.

Secure and Reliable

However, there’s a catch; enterprise users are generally (although not always) more demanding than a fourteen-year-old with a crush on a boy band. They want security and reliability.

While Atom defines security mechanisms, and there are some interesting efforts to retrofit it onto RSS, reliability is a bit tricker. Right now, when you poll a feed, you have no idea if there were any missed entries between the last time you looked.

For the business, this is a serious failing. There’s no way you’re going to use a technology to carry business-critical information if, as a matter of course, it can “forget” some.

I think this is the most critical thing that we need to make syndication Enterprise-ready, and it’s why I’ve proposed the Feed History mechanism.

In a nutshell, it allows you to reconstruct the entire feed by “paging” back through the archives. It works in both RSS and Atom, and I’ve implemented it in this feed using plain, vanilla Moveable Type template tags. Now, I’m looking for other implementation experiences; if you put it in your blog, or your aggregator software, please comment below, or send me an e-mail.


Filed under: Syndication Web

6 Comments

Henry Story said:

Yep.

Feed history is really useful for other reasons too. For BlogEd, a client side editor, a full history of a feed is also very useful way to get different instances of BlogEd running on different machines to synchronise their state with that on the server. Imagine that you go on holidays and want to write up a Blog in an java enabled computer in an internet cafe, you would just need to point BlogEd to the feed and it could download as much of the feed history as it wanted to. When you then get back to your office after travelling, your local BlogEd editor could also synchronise with the changes you made during your holidays.

Henry

Tuesday, July 19 2005 at 10:23 AM +10:00

Seth Russell said:

I would like to implement this but you have not given me enough information. How does the RSS feed change. My site is at www.speaktomecatalog.com, it is a store, each catagory has it's own feed. What changes do i make?

Tuesday, July 19 2005 at 3:29 PM +10:00

Meryn said:

Pagination by providing previous/next links is good. Actually I'm surprised that neither Atom or RSS support this. Maybe you're supposed to wrap the feed element in a surrounding structure to implement the pagination?

Maybe you could consider Xlink for linking to previous and next pages. See http://www.w3.org/TR/xlink/ .

Wednesday, July 20 2005 at 1:27 AM +10:00

Roger Benningfield said:

Mark: Okay, I've added history to my entertainment blog's feed:

http://admin.entertainment.journurl.com/?template=rss-atom

...but I have a question. When I crawl back to your final (or is that first?) archive document, you've dropped both the stateful and prev elements... in mine, I only drop prev.

So is one approach better than the other? The spec seems to make room for both, ao I'm just wondering if there are consequences either way.

Thursday, July 21 2005 at 3:53 PM +10:00

Mark Nottingham said:

The current draft says that stateful shouldn't be used in archives; only in the subscription feed.

Cheers,

Saturday, July 23 2005 at 9:16 AM +10:00

William Henry said:

Hi Mark,

Steve Vinoski (http://www.iona.com/blogs/vinoski/) pointed me to this post based on my post today on IP Babble:

http://www.ipbabble.com/2006/01/rss_for_soa_based_services_reg.html

Thought you might be interested in my experiment in using RSS as a Web services discovery mechanism.

Friday, January 6 2006 at 10:56 PM +10:00

Creative Commons