Protocol Design Entries
Wednesday, 15 May 2013
A common part of HTTP-based APIs is telling the client that something has gone wrong. Most APIs do this in some fashion, whether they call it a “Fault” (very SOAP-y), “Error” or whatever. Most of them define a new format for just this purpose; for examples, see Amazon’s, OpenStack’s, Twitter’s, Facebook’s, and SalesForce’s. Twitter's is fairly representative: {"errors":[{"message":"Sorry, that page does not exist","code":34}]} Here, they associate a human-readable message and an error code with the error. That’s a good start, but...
Friday, 4 January 2013
One of the major mechanisms proposed by SPDY for use in HTTP/2.0 is header compression. This is motivated by a number of things, but heavy in the mix is the combination of having more and more requests in a page, and the increasing use of mobile, where every packet is, well, precious. Compressing headers (separately from message bodies) both reduces the overhead of additional requests and of introducing new headers. To illustrate this, Patrick put together a synthetic test that...
Tuesday, 18 December 2012
A proposal by John Graham-Cumming is currently doing the rounds: HMURR (pronounced ‘hammer’) introduces a new pipelining mechanism with explicit identifiers used to match requests and responses sent on the same TCP connection so that out-of-order responses are possible. The current HTTP 1.1 pipelining mechanism requires that responses be returned in the same order as requests are made (FIFO) which itself introduces a head-of-line blocking problem. This seems attractive at first glance; rather than starting a whole new protocol, why not just incrementally improve an...
Tuesday, 4 December 2012
One of the most vexing problems that still seems to be facing people when I talk to them about HTTP APIs is how to handle versioning and extensibility — i.e., how they evolve. I tend to think about this a lot and talk to quite a few people about it, since I’m intimately familiar with the approaches to versioning that the HTTP protocol itself takes, and with the general attitude taken to it in the broader Internet architecture by the...
Monday, 29 October 2012
Once in a while, people ask me whether they should use the OPTIONS HTTP method, and whether we should try to define formats for discovering resource capabilities with it. I usually say “no.” This is a personal position, nothing “official”. That said, the conversation usually goes like this: What’s Wrong with OPTIONS? A few things. First and foremost, it can’t be cached. Usually, people want to use OPTIONS to discover some metadata about the server (the “OPTIONS *” case) or...
Wednesday, 5 September 2012
A common problem for APIs is partial update; when the client wants to change just one part of a resource’s state. For example, imagine that you’ve got a JSON representation of your widget resource that looks like: { "name": "abc123", "colour": "blue", "count": 4 } and you want to update the “count” member’s value to “5”. Now, you could just PUT the entire thing back with the updated value, but that requires a recent GET of its...
Wednesday, 11 July 2012
One thing I didn't cover in my previous rant on HTTP API versioning is an anti-pattern that I'm seeing a disturbing number of APIs adopt; using a HTTP header to indicate the overall version of the API in use. Examples include CIMI, CDMI, GData and I'm sure many more. These APIs are designed so that the client can send a request header advertising what version of the protocol they support: GET /foo HTTP/1.1 Host: api.example.org BadApiVersion: 1.2 and the server will...
Monday, 25 June 2012
@dret: if your scenario is homogeneous and models are harmonized across participants, #REST is of limited utility for you. Erik’s tweet just now reminded me of something I’d been wanting to write about for a while; complexity in HTTP APIs, and their effect on how you use the protocol. One Client, One Server The simplest HTTP API is one where you have a known client and server talking to each other; they may be managed by separate people or the...
Tuesday, 17 April 2012
Erik Wilde - otherwise known as dret - has published an Internet-Draft for a “profile” link relation type: This specification defines the ‘profile’ link relation type that allows resource representations to indicate that they are following one or more profiles. Why am I excited? Three words: Media Type Proliferation For better or worse, everyone and his dog is minting “RESTful” APIs. One byproduct of this is the need to identify formats, so they’re going off and creating new media types....
Tuesday, 25 October 2011
A lot of bits have been used over on the OpenStack list recently about versioning the HTTP APIs they provide. This over-long and rambling post summarises my current thoughts on the topic, both as background for that discussion, as well as for review in the wider community. The Warm-up: Software vs. Web Versioning Developers are used to software versioning; e.g., for every release, you bump an identifier. There are usually major versions, minor versions, and sometimes things like package identifiers....
Wednesday, 12 October 2011
Since joining Rackspace to help out with OpenStack, one of the hot topics of conversation I’ve been involved in has been extensibility and versioning. I think most of my readers (yes, all six of you) are fairly familiar with, if not tired of (hi, Dave!) the various arguments and counter-arguments in this space. However, there is one new-ish bit; how to do distributed extensibility in JSON. That’s because OpenStack’s API allows vendors to add extensions in various ways, in an...
Wednesday, 24 August 2011
It used to be that when you registered a media type, a URI scheme, a HTTP header or another protocol element on the Internet, it was an opaque string that was a unique identifier, nothing more. Sure, there are some substructures (e.g., vnd. and prs. in media types) to aid in avoiding collisions, but they don’t actually do anything. And even so, they need to be used judiciously (e.g., the problems inherent in x-). However, it’s now becoming fashionable to...
Friday, 23 July 2010
Since SPDY has surfaced, one of the oft-repeated topics has been its use of TLS; namely that the SPDY guys have said that they’ll require all traffic to go over it. Mike Belshe dives into all of the details in a new blog entry, but his summary is simple: “users want it.” I don’t think it’s that simple. Trust I trust my ISP, to a point; I have a business relationship with them, so I don’t worry too much...
Wednesday, 5 May 2010
Steve Souders and others have been working for a while on HAR, a HTTP Archive format. I love the idea behind HAR, but as I expressed on the mailing list (so far no response), I have a hard time believing that different implementations will parse and process HTTP and then serialise it into HAR in exactly the same way. My use case is incorporating HAR support into RED, both for import and export. However, RED needs byte-for-byte access to...
Thursday, 18 February 2010
Resource Packages is an interesting proposal from Mozilla folks for binding together bunches of related data (e.g., CSS files, JavaScript and images) and sending it in one HTTP response, rather than many, as browsers typically do. Intuitively, this seems to make sense; less HTTP requests is good, right? Maybe, maybe not. AFAICT, there aren’t any metrics comparing RP vs. traditional sites (has anyone done this?). In any case, a few concerns come to mind about this approach to making...
Friday, 15 January 2010
If you haven’t seen it already, check out the Call for Papers for the First International Workshop on RESTful Design (WS-REST 2010), where I’m on the program committee, along with many of the usual suspects. Submissions due February 8, 2010, 23.59 Hawaii time. If only I were there to receive them…...
Friday, 13 November 2009
A couple of nights ago, I had a casual chat with Google’s Mike Belshe, who gave me a preview of how their “Let’s make the Web faster” effort looks at HTTP itself. SPDY (nee FLIP) is an alternate application protocol that’s in Chromium, but buried so deeply that you have to enable it with a command-line option (—use-flip). AFAICT there aren’t even any public servers that support it yet, but it’s still a very exciting development. Why? In a...
Wednesday, 18 February 2009
Sometimes, it seems like every time somebody has a great idea for a new HTTP header, media type, or pretty much any other protocol element, they do the same thing. Rather than trying to figure out how to fit into how the rest of the world operates, getting adequate review and socialising their proposal, they just stick a bloody X- on the front and ship it. The IETF has no-one to blame but itself for this situation, of course....
Thursday, 20 March 2008
Having complained before about the sad state of HTTP APIs, I’m somewhat happy to say that people seem to be getting it, producing more capable server-side and client-side tools for exposing the full range of the protocol; some frameworks are even starting to align object models with resource models, where HTTP methods map to method calls on things with identity. Good stuff. However, something’s been bugging me for a long time about this. While there’s a nice internal logic...
Monday, 3 March 2008
Not many people that I know outside of IETF circles realise that a new *DAV effort has started up; CardDAV. An address book access protocol leveraging the vCard data format. The Internet-draft draft-daboo-carddav will be the starting point. The WG is explicitly cautioned to keep the base specification feature set small with an adequate extension mechanism, as failure to do so was a problem for previous PAB efforts (ACAP). Draft-daboo looks like it’s taken a page out of CalDAV...
Sunday, 17 February 2008
It’s 7am, I’m sitting in the Auckland Koru Club on my way home and reading the minor kerfuffle regarding PATCH with interest. For me, the critical difference between PATCH and POST is generality; PATCH is a generic method (as all good HTTP methods should be), while POST is not (the exception that proves the rule). As such, it should be possible to take a PATCH request and a current representation of the resource it’s being applied to and —...
Wednesday, 6 February 2008
Here’s one that I’ve been wondering about for a while, for the LazyWeb (HTTP Geek Edition); PUTs and POSTs can result in the creation of new resources, or changes to the state of existing ones. The response to both can contain one of a status message about what happened, or a representation of the resource (or most important one, if more than one was created/modified); often signalled by the Content-Location header, or nothing (i.e., just the status line and...
Monday, 21 January 2008
I’m following the discussion of RESTful Web description in general, and WADL in particular, with both difficulty and interest (see Dare, Patrick and Joe’s thoughts for a nice contrast). Difficulty because there’s so much of it, and it’s hard to give each piece the attention it deserves. Interest because it goes to the heart of the harder parts of REST — e.g., “hypertext as the engine of application state.” On top of that, I was one of the people...
Wednesday, 12 December 2007
We use caching extensively inside Yahoo! to improve scalability, latency and availability for back-end HTTP services, as I’ve discussed before. However, there are a few situations where the plain vanilla HTTP caching model doesn’t quite do the trick. Rather than come up with one-off solultions to our problems, we tried going in the other direction; finding the most general solution that still met our needs, in the hopes of meeting others’ as well. Here are two of them (with...
Tuesday, 7 August 2007
I’ve been hoping to avoid this, but ETags seem to be popping up more and more often recently. For whatever reason, people latch onto them as a litmus test for RESTfulness, as the defining factor of HTTP’s caching model, and much more. So, let me counter: they’re not all that. In fact, there are a number of pitfalls you need to be wary of if you use them. First, depending on how they’re generated, you might find different boxes...
Saturday, 28 July 2007
URI Templates -01 is now an Internet-Draft. After sitting on the spec for a while and trying to figure out an elegant solution to the encoding problem, we decided to take the simple route and see how it sticks. The only encoding that the template spec itself does is for characters that are outside the range of those allowed in URIs. Everything else is application-specifc. So, for example, this variable; foo = "/thing?@ <--stüff-->" with this template: http://www.example.com/{foo} will...
Thursday, 10 May 2007
For a while, I’ve had the fairly well-known Charles Eames quote “Design depends largely on constraints” as the tagline on my blog (if you read this in a feed aggregator, you’ll have to go to one of the HTML pages to see it). In putting together one of her classes, Anitra came across the full interview from which it was sourced. The relevant section says it well, whether the topic is furniture design, graphic design or designing the architecture...
Thursday, 20 April 2006
Back at the W3C Technical Plenary, I argued that Working Groups need to concentrate on making more Web-friendly specifications. Here’s an example of one such lapse causing security problems on today’s Web. Safety in HTTP HTTP methods have a property known as safety; RFC2616 describes it in section 9.1.1; Implementors should be aware that the software represents the user in their interactions over the Internet, and should be careful to allow the user to be aware of any actions...
Friday, 7 April 2006
It’s become axiomatic in some circles — especially in WS-* land, as well as in many other uses of XML — that the preferred (or only) means of offering extensibility is through URI-based namespaces, along with a flag to tell consumers when an extension needs to be understood (a.k.a. mustUnderstand). The reasoning is that extensibility should be as easy as possible. By leveraging one registry — DNS — you can use URIs to allow anyone to create your own...
Monday, 7 November 2005
More and more people are getting turned on to the advantages of using REST as a higher-level abstraction for networked applications, often comparing it favourably to SOAP and Web services. However, as many have pointed out, this is a bad comparison to make; REST is an architectural style, while SOAP is a protocol. So, what should be compared? When this question comes up, I’ve found it helpful to draw a table with different architectural styles in the columns, and...
Saturday, 22 October 2005
Why is it that Web browsers — Amaya excluded — don’t support PUT and DELETE? After all, if there are enough VCs foolish enough to part with their money for something like Flock, surely we could at least support all of HTTP’s methods. It’s not just browsers, of course; heck, even Python’s new, mega-complete urllib2 library restricts itself to GET and POST. Other languages are similarly limited. So, why does the implemented Web restrict itself to half of CRUD? It’s...
Friday, 8 July 2005
You can describe just about anything with sufficient precision in plain English, given enough words. In practice, this doesn’t happen; specialised fields — whether science, finance or art — develop specialised jargon as a shorthand for concepts that are well-understood in that field. It gives greater precision, easier flow of ideas, and yes, it raises the bar to entry for newcomers. The trade-off is worth it, usually; although it would be genuinely useful if a layman could understand the...
Sunday, 22 May 2005
There’s been quite a kerfuffle over Google’s Web Accelerator, because it prefetches Web content. It’s amusing to see these issues recycle over time; in the late nineties, prefetching was one of the biggest areas of research in Web caching. There were lots of papers written (search for “prefetch”), and even a commercial implementation; CacheFlow was the prefetching cache (they called it “active content”), and it was pretty controversial in the industry. After a long period of back-and-forth and considerable...
Monday, 21 March 2005
A while back, I wrote up a description of a pattern for avoiding messages like “click submit only once.” I didn’t do much after that, because I’ve been a bit busy, and because I wanted to do some implementation of a more general HTTP framework before I wrote a more formal document. One delay led to another, and to make a long story short, I never got around to writing that specification… until now. Why now? The short answer...
Wednesday, 2 March 2005
So, you’ve got some data that you need to give to somebody else, and you want to use XML to do it; good for you, you’ve seen the light / hopped on the bandwagon / drunk the Kool-Aid. At first glance, this seems like a pretty straightforward task; after all, it’s just angle brackets, right? Not so fast. If you’re the only person who every has to look at the XML or write software to work with it, you’re...
Sunday, 10 October 2004
In a recent post, Don gave his take on the enlightening nature of WS-Transfer; Honestly, WS-Transfer has been in the oven for quite a while. It’s been interesting to see people’s reaction to it. Stage 1. What good is this? Stage 2. Ahh, the idea of uniform application protocols seems pretty useful. Stage 3. Wow, I can model my entire universe using less verbs/operations than I have fingers on one hand. Why would I ever author another WSDL portType?...
Thursday, 19 August 2004
Alfred Marshall, who is credited with turning economics from a sideline to a proper discipline of its own, had this to say: (1) Use mathematics as a shorthand language, rather than as an engine of inquiry. (2) Keep to them till you have done.... (4) Then illustrate by examples that are important in real life.
Thursday, 5 August 2004
(Another instalment in “XML Heresies.”) One of the foundations of most vendors’ approach to Web services is called document-oriented messaging. This is the notion that interoperability is improved by describing a protocol in terms of the artefacts that are exchanged on the wire, rather than how the code that handles them is written. As far as it goes, that’s good advice. Implementation-specific specifications lead to brittleness, because you can’t swap out the implementation; the message is too tightly coupled to...
Wednesday, 30 June 2004
Way back when the XML Protocol Working Group started kicking around, Henrik and I had a long-running, low-level “discusssion” about whether SOAP was a protocol or a format. Henrik won, and SOAP is known as a protocol* today (despite the fact that the ‘P’ no longer stands for anything). I’m wishing I’d fought just a bit harder to keep the distinction. What Makes SOAP A Protocol? Although people call SOAP a protocol, it’s really more of a protocol construction toolkit;...
Monday, 14 June 2004
One thing about Web description formats that hasn’t seen much discussion yet is how people intend to use them. The WSDL Working Group has a Usage Scenarios document and a Requirements document, but unfortunately they only talk about the kinds of Web services they want to describe, not how the descriptions themselves are to be used. So, here are the use cases for Web description formats that I’m aware of, along with examples and some of my conclusions (skip ahead...
Friday, 28 May 2004
I’ve been talking with a few people about my previous assertion that the Infoset is a bad abstraction for data modelling, and my subsequent post about the informational properties of the Infoset. The feedback has been positive, especially regarding the notion that the Infoset offers great tools for document markup, but presents more problems than solutions when directly used in non-markup applications; i.e., those that are data-oriented. The best examples of the kind of unneeded complexity I’m talking about are...
Wednesday, 12 May 2004
Recently, I’ve been thinking about the influences that using the Infoset has on the information you place in it. To put it another way: if you work with XML at the Infoset level, what tools are you given to express information with? As an informational channel, the structures that XML gives you can express pretty much anything, of course, but they lend themselves to some things better than others. As such, using the Infoset encourages data to be moulded to...
Wednesday, 5 May 2004
Without pointing fingers, some people have a bee in their collective bonnet about the dangers of allowing binary content to be represented in XML, care of XOP. Others are up in arms about re-inventing HTTP in SOAP, courtesy of the Representation Header. Both of these are products of the XML Protocol WG, of which I’m a member, so I’d like to share my viewpoint (which is not that of either my employer nor the working group, etc., ad nauseam). XOP...
Saturday, 1 May 2004
I’m watching a company called Riverbed with interest, because they just released a new product, “Steelhead”. In a nutshell, it’s IP datagram compression done with a shared, dynamic dictionary. That’s right, it has a disk on each end of the wire, and they keep copies of what goes by locally, so that they can send a hash or other nonce of it if they have a hit. They also do some optimisation of crappy protocols, but that’s the easy part....
Tuesday, 27 April 2004
Way back when in the XML Protocol Working Group, one of the concerns that came up was the processing model for SOAP headers. In particular, while SOAP 1.2 does a good job of specifying how that model operates, a key peice of information is missing; how to order the steps in processing a message. In other words, if you get a SOAP message that has headers to, say, encrypt a message, and also to apply an XSLT stylesheet, you have...
Tuesday, 27 April 2004
In the past, I’ve talked about reusing WSDL as a format for describing Web resources, as well as coming up with a bespoke format. One path that I’ve overlooked so far is reusing WebDAV to describe Web resources. The WebDAV protocol defines and allows you to describe the properties of a Web resource; e.g., the last-modified time is such a property, and so is the fact that it requires authentication. WebDAV also defines a way to get a representation of...
Tuesday, 20 April 2004
Sean McGrath always has carefully considered positions, and he hits it out of the ballpark with this one. A few thoughts; Eventually though, to fully realise RESTian SOA we need to get linguistic determinism working for us, not against us. To do that, we need to bake key SOA distributed computing concepts right into the language. Surely you mean that we should bake in REST, no? In the case of the all important concept of dynamic typing, this means we...
Monday, 19 April 2004
One of the things that people find compelling about Web services is its promise of asynchrony. “HTTP is only request/response, and therefore synchronous; it’s terrible for long-lived business processes, where the server needs to contact the client at some arbitrary time in the future” they say. A single HTTP request/response is indeed synchronous, so that that level this argument holds. In the common case, one party (the server) can’t send messages to the other (the client) without receiving a corresponding...
Friday, 16 April 2004
To use WSDL to describe RESTful interactions, you need some way of accommodating generative resource identifiers. In a nutshell, this means some part of the URI is dynamic. For example, with HTTP I might describe an address book where someone named “Jones” has a corresponding entry URI; http://www.example.org/people/Jones When describing the overall interface on offer, it’s not that interesting to talk about this particular resource; rather, you want to describe the interface provided for all people. To do this, you...
Thursday, 15 April 2004
Lots of papers come and go over the years; take a look at any tech conference, online bibliographies (even subject-specific ones; Webbib is a favourite), and you’ll be inundated. However, a few rise above the rest (no pun intended) and have real staying power; invariably, they’re about good, principled design, usually with the benefit of hard experience. I admit a bias towards those about the Web and HTTP, but considering its success, I think it’s something other protocols could emulate....
Wednesday, 14 April 2004
I’ve talked before about describing RESTful Web resources, going as far as prototyping a new format. That work was predicated on the assumption that WSDL wasn’t adequate. However, Dave Orchard has been looking at this in the WSDL WG, and recently challenged me to think about it again. I’m becoming convinced that it would be workable. To give an example (using my usual yardstick, an online address book); <portType name="AddressBookEntry"> <operation name="getHTML" web:Method="GET"> <output message="tns:AddressBookEntryDocument" wsa:Action="urn:ietf:params:mediatype:text/html"/> </operation> <operation name="getVCard" web:Method="GET">...
Saturday, 7 February 2004
Jon Udell is thinking about the benefits of data being globally available, rather than localised to a machine. I’m in complete agreement; in the last two years, I’ve used Linux, Windows and Mac OSX on the desktop, leading me to be ruthless about data portability. However, I’m not sure how this leads him to this conclusion; A general solution would require OSs that work like Groove, and applications that send messages rather than write files. Why do messages — which...
Monday, 12 January 2004
Wouldn’t it be great if, whenever a business, government organization or just the guy down the block came up with a new format for their documents, they could easily get a media type, so that the format would be a first-class citizen on the Web? Of course, they already can, but the hassles that you have to go through to get into the centralised registry dissuade most people from doing so. It used to be that this high bar was...
Saturday, 3 January 2004
In his blog, Sean McGrath wonders about two potentially competing faces of standards; extensibility and interoperability. If “compliance” to X is open-ended via an extensiblity mechanism, then “X-compliant” means very little when it comes to interoperability. This is the constant struggle of interop, how not to stifle innovation and yet avoid babelization? Good question. Interop is what everyone looks for in a standard; it’s the whole reason that companies spend so much money and time talking to each other in...
Friday, 3 October 2003
Mark Baker says that REST is SOA + late binding. While I see the truth in this, I think it's pretty orthogonal, and it's not that compelling for most SOAish folks. This is because their use cases are machines talking to machines, not people talking to machines. In Web browsing, this is great, because people are smart enough to look at the late-bound semantics in a representation and figure out what they want to do; machines aren't, yet. That means...
Saturday, 13 September 2003
I shudder when I see these words. Everyone I’ve asked has, at least once, gotten two orders of something online (personally, I’ve had the SonyEricsson store ship *three* duplicate orders); “Click Submit Only Once” is intended to stop that. The problem is, it puts me and every other shopper between a rock and a hard place. That’s because if there’s any problem with my browser, my computer’s network connection, the Internet itself or the server, I don’t know the status...
Tuesday, 24 June 2003
The W3C Semantic Web wiki has an entry called 'BeesAndAnts' that very effectively conveys something that I've been trying to articulate for a while (and, as usual, failing). It's not about the Semantic Web in my mind, so much as it's about REST and Web Services (which means that there's something to this Web architecture stuff yet, I think). Many people are approaching Web services as bees; you expose an API, do some dancing and you get interoperability. In my...
Tuesday, 20 August 2002
Don talks about the evils of tolerance in receiving implementations, and I say Amen, brother! Preach! The classic approach works when there are relatively few implementators; however, when the whole world implements a protocol (whether it's SOAP or HTML or whatever), you're asking for trouble if you allow too generously....
