HTTP APIs
The orange site is currently discussing an article about Server-Sent Events, especially as compared with WebSockets (and the emerging WebTransport). Both the article and discussion are well-informed, but I think they miss out on one aspect that has fairly deep implications.
One of the concerns that often comes up when someone creates a new HTTP header is how much “bloat” it will add on the network. This is especially relevant in requests, when a little bit of extra data can introduce a lot of latency when repeated on every request.
There’s more than a little confusion and angst out there about HTTP status codes. I’ve received
more than a few e-mails (and IMs, and DMs) over the years from stressed-out developers (once at
2am, their time!) asking something like this:
There’s been a lot of interest in and effort expended upon “hypermedia APIs” recently. However, I see a fair amount of resistance to it from developers and ops folks, because the pragmatic benefits aren’t often clear. This is as it should be, IMO; if you’re not able to describe concrete benefits without hand-waving about the “massive scale of the Web.”
A common part of HTTP-based APIs is telling the client that something has gone wrong. Most APIs do this in some fashion, whether they call it a “Fault” (very SOAP-y), “Error” or whatever.
One of the most vexing problems that still seems to be facing people when I talk to them about HTTP APIs is how to handle versioning and extensibility – i.e., how they evolve.
Once in a while, people ask me whether they should use the OPTIONS HTTP method, and whether we should try to define formats for discovering resource capabilities with it.
A common problem for APIs is partial update; when the client wants to change just one part of a resource’s state. For example, imagine that you’ve got a JSON representation of your widget resource that looks like:
One thing I didn’t cover in my previous rant on HTTP API versioning is an anti-pattern that I’m seeing a disturbing number of APIs adopt; using a HTTP header to indicate the overall version of the API in use. Examples include CIMI, CDMI, GData and I’m sure many more.
@dret: if your scenario is homogeneous and models are harmonized across participants, #REST is of limited utility for you.
Erik Wilde - otherwise known as dret - has published an Internet-Draft for a “profile” link relation type:
When you’re designing HTTP APIs, you need to keep a lot of concerns in mind. Stealing a page from XP, let’s look at some possible personas and their user stories for HTTP-based APIs:
To be a full-fledged format on the Web, you need to support links – something sorely missing in JSON, which many have noticed lately.
A lot of bits have been used over on the OpenStack list recently about versioning the HTTP APIs they provide.
Since joining Rackspace to help out with OpenStack, one of the hot topics of conversation I’ve been involved in has been extensibility and versioning.
Now, you can test any URL to instantly determine if it’s RESTful.
One of the nagging theoretical problems in the Web architecture has been finding so-called “site-wide metadata”; i.e., finding something out about a Web site before you access it. We wrestled with this in P3P way back when, and the TAG took it up after that.
If you haven’t seen it already, check out the Call for Papers for the First International Workshop on RESTful Design (WS-REST 2010), where I’m on the program committee, along with many of the usual suspects.
UPDATE: RFC6648 is now the official word on this topic.
Metadata discovery is a nagging problem that’s been hanging around the Web for a while. There have been a few stabs at this problem (including at least one by yours truly), but no real progress.
Having complained before about the sad state of HTTP APIs, I’m somewhat happy to say that people seem to be getting it, producing more capable server-side and client-side tools for exposing the full range of the protocol; some frameworks are even starting to align object models with resource models, where HTTP methods map to method calls on things with identity. Good stuff.
I’m following the discussion of RESTful Web description in general, and WADL in particular, with both difficulty and interest (see Patrick and Joe’s thoughts for a nice contrast).
I’ve updated the WADL documentation stylesheet, primarily to:
I’ve been hoping to avoid this, but ETags seem to be popping up more and more often recently. For whatever reason, people latch onto them as a litmus test for RESTfulness, as the defining factor of HTTP’s caching model, and much more.
URI Templates -01 is now an Internet-Draft.
The QCon presentation ( slides) was ostensibly about how we use HTTP for services within Yahoo’s Media Group. When I started thinking about the talk, however, I quickly concluded that everyone’s heard enough about the high-level benefits of HTTP and not nearly enough details of what it does on the ground. So, I decided to concentrate on one aspect of the value that we get from using HTTP for services; intermediation, as an example.
I think that most of the debate about REST focuses on the wrong things, leading developers down the garden path at the expense of their productivity and the success of their projects. Time and time again, I’ve seen folks who are new to REST get caught up on small stuff like this;
Yahoo! (finally!) released Pipes as a beta today; congrats to the very talented team that put this together.
One of the perceived deficiencies of JSON is that it doesn’t have a schema language. I say “perceived” because the problems that a schema language brings often outweigh the benefits; after all, look at the mess that XML Schema is in.
My team at Yahoo! is looking for a mid-level developer (5-10 years experience) to help build our HTTP/REST toolkit, among other things.
Recently, there’s been a resurgence for the Link element in HTML; everything from Microformats to Atom autodiscovery is using it. This isn’t surprising; as machines start processing Web documents more, it’s necessary to use hyperlinks — the foundation of the Web — to tie resources together, without getting in users’ faces.
A friend (who shall remain anonymous) pointed me to Microsoft’s announcement today regarding their foray into robotics, of all things. My eyes glazed over until they rested upon the Microsoft Robotics Application Model;
It’s become axiomatic in some circles — especially in WS-* land, as well as in many other uses of XML — that the preferred (or only) means of offering extensibility is through URI-based namespaces, along with a flag to tell consumers when an extension needs to be understood (a.k.a. mustUnderstand).
Microsoft and friends (of the keep your enemy closer variety, I suspect) have submitted WS-Transfer to the W3C. I found the Team comment interesting; e.g.,
More and more people are getting turned on to the advantages of using REST as a higher-level abstraction for networked applications, often comparing it favourably to SOAP and Web services.
Why is it that Web browsers — Amaya excluded — don’t support PUT and DELETE? After all, if there are enough VCs foolish enough to part with their money for something like Flock, surely we could at least support all of HTTP’s methods.
You can describe just about anything with sufficient precision in plain English, given enough words. In practice, this doesn’t happen; specialised fields — whether science, finance or art — develop specialised jargon as a shorthand for concepts that are well-understood in that field. It gives greater precision, easier flow of ideas, and yes, it raises the bar to entry for newcomers.
The W3C has just started a mailing list for discussion of Web description formats;
Marc Hadley has released WADL in the wild, and I’m intrigued; based on a first look, I’d say it’s the most promising Web (as opposed to Web Services) description language yet.
A while back, I published a series of entries ( 1, 2, 3, 4) about would-be Web Description Formats, with the intent of figuring out which (if any) is suitable, or whether a new one is required.
A while back, I wrote up a description of a pattern for avoiding messages like “ click submit only once.” I didn’t do much after that, because I’ve been a bit busy, and because I wanted to do some implementation of a more general HTTP framework before I wrote a more formal document.
Werner makes an excellent point;
In a recent post, Don gave his take on the enlightening nature of WS-Transfer:
A few days ago I blogged a straw-man API for client-side HTTP based on dictionaries. This turns out to be well-aligned with a project I’ve had on the back burner for a while; coming up with some Python APIs for HTTP that are usable, encourage good practice, and well-aligned with the specifications.
From the Daily Python URL comes another noteworthy API for XML; XMLFragment. I haven’t tried it yet (it doesn’t appear to be separately available, hint, hint), but I like the look of it.
One thing about Web description formats that hasn’t seen much discussion yet is how people intend to use them.
In the past, I’ve talked about reusing WSDL as a format for describing Web resources, as well as coming up with a bespoke format.
Sean McGrath always has carefully considered positions, and he hits it out of the ballpark with this one. A few thoughts;
One of the things that people find compelling about Web services is its promise of asynchrony. “HTTP is only request/response, and therefore synchronous; it’s terrible for long-lived business processes, where the server needs to contact the client at some arbitrary time in the future” they say.
Aaron Swartz has started to document the iTunes Music Store; this is a good example of a non-browser, cross-platform application reusing HTTP. It would be interesting to see the interface documented on a per-URI basis.
Jon Udell is thinking about the benefits of data being globally available, rather than localised to a machine. I’m in complete agreement; in the last two years, I’ve used Linux, Windows and Mac OSX on the desktop, leading me to be ruthless about data portability.
Just got some mail regarding the Cacheability Engine which led me to NetKernel;
Wouldn’t it be great if, whenever a business, government organization or just the guy down the block came up with a new format for their documents, they could easily get a media type, so that the format would be a first-class citizen on the Web?
HTTP provides considerable benefits to Web applications that take advantage of it; everything from scalability (through caching), client-integrated authentication, automated redirection, multiple format support and lots more.
Adam asks if there’s a description format for REST. I don’t know of any that have wide acceptance (and I think the hard-core RESTafarians will answer “REST is self-describing, that’s the point” ;) but I have been noodling on something for my own purposes.
Mark Baker says that REST is SOA + late binding. While I see the truth in this, I think it’s pretty orthogonal, and it’s not that compelling for most SOAish folks.
I shudder when I see these words. Everyone I’ve asked has, at least once, gotten two orders of something online (personally, I’ve had the SonyEricsson store ship three duplicate orders); “Click Submit Only Once” is intended to stop that. The problem is, it puts me and every other shopper between a rock and a hard place.
I just found a draft finding that the W3C TAG published about a month ago, regarding the use of metadata in URIs. This is very cool, and I especially like the emphasis on authorities’ ability to embed metadata in URIs.
The W3C Semantic Web wiki has an entry called ‘BeesAndAnts’ that very effectively conveys something that I’ve been trying to articulate for a while (and, as usual, failing). It’s not about the Semantic Web in my mind, so much as it’s about REST and Web Services (which means that there’s something to this Web architecture stuff yet, I think).
Jonathan Rosenberg published a new Internet-Draft, XCAP, to the SIMPLE Working Group in the IETF. Here’s the skinny:
