Re: [syndication] Syndication Subscription

[Aaron Swartz <aswartz@swartzfam.com>]

Gabe Beged-Dov <begeddov@jfinity.com> wrote:

> I'll ignore your age if you'll ignore my e-mail blooper :-( I thought
> I was sending to rael based on a conversation we had had about the
> usual fun game of trying to guess what people's physical presence is
> like based on thier net presence.

Ahh, I see. That makes a bit more sense now. Well, information wants to be
free, you know? It certainly is an interesting subject, but we're far enough
off-topic as it is.

> As to the topic of this thread, what are you thinking a publicationID
> is? Is it just another URL? Is it in fact the URL that you would have
> dereferenced in order to get the storylist, AKA RSS file?

That's certainly a possibility. I must admit I haven't really given the
issue much thought -- that's why I've posted it, to get discussion going. It
really only has to be unique to the specific server, but I guess if it was a
URL it would allow mirror servers to serve publications that don't actually
come from them. So yes, I think a URL would be perfect for a publication ID.

Then the question is what does it point to. I would suggest it point to an
address that provides the equivalent of a getStoriesSince request for the
past day/week/month or so. (Note I don't say RSS, see below for more.)

> Another question is whether you are talking about getting stories or
> getting headlines (or metadata about the stories). This is a key
> distinction that underlies alot of the tension between approaches like
> RSS and something like ICE. Rael has also made the point that
> ScriptingNews provides good support for syndicating stories rather
> than headlines.

Yes, I made this point too, back in "RSS in Two Directions". The plan would
be to have this protocol independent of any actual format -- it would only
specify the meta-data such as publicationIDs and storyIDs that are needed by
the protocol (to prevent duplication, etc.). This way, the format could be
used to distribute anything -- anywhere from headlines, personal
notifications, news stories, perhaps even mailing-list-like messages.

<http://www.egroups.com/message/syndication/159>

> Then there is the question of how much commonality you want to have
> between the push approach and the pull approach. I would hope that
> they would have alot in common and that I could get notification on
> almost anything that is addressable by a URI rather than having a
> separate naming convention for the notifiable things.

I agree -- perhaps the getStoriesSince and other pull functions would be
better replaced as normal URIs rather than XML-RPC applications. This would
allow people to more easily generate them and link to them. While we're on
the subject of pull, we'd also want a getLastNStories(n) and a
getStory(storyID). These should also be normal URIs.

> The trick to doing notification on any
> URI is efficiency of the change detection and being able to tell what
> a significant change is (like ignoring the dynamically generated date
> when doing a diff). This isn't as big a problem for syndication URL as
> it is for URL that are intended for direct human consumption. See how
> many false positives you get from services like spyonit.com :-(

That's my next task after I cover syndication. (I don't tackle small
problems. :->) It seems the easiest way to detect notifications is to do
regexps for the content. SiteScooper (http://sitescooper.cx) already does
this, but it's designed for Palm consumption rather than notification
purposes. 

The idea for this protocol is that it allows websites to provide
notification for changed items, but doesn't get involved with how those
items are specified. That way it can be used for RSS, scriptingNews, or just
a normal web page change. This cuts down on unnecessary server burden, but
it does put a bit more work on the publisher side of things. But in most
cases that makes sense -- the publisher _wants_ people to read their stuff.
A smart publisher would do as much as they could to get people to come back
to the site. This is a way of specifying that.

For those of you who are wondering what my secret plans for all this are,
here's a short explanation: I'm building a web site that will allow people
to notify it (hopefully through this protocol) with updated web pages.
Because it's done in an XML format, the site can keep track of which web
pages you've read, and which you haven't and only notify you once. It can
also sort based on importance and relevance, etc. And it will be an open
protocol so if you don't like my site you can go right ahead and build your
own. Initially the site will get it's data from RSS, but eventually it would
be nice if it could be notified through a push mechanism (so users would be
more up-to-date) and if it could receive whole articles, rather than just
URLs.

Well, I've gone on too long. Let's keep the discussion rolling,

-- 
        Aaron Swartz         |"This information is top security.
<http://swartzfam.com/aaron/>|     When you have read it, destroy yourself."
  <http://www.theinfo.org/>  |             - Marshall McLuhan