mnot’s blog

Design depends largely on constraints.” — Charles Eames

Wednesday, 5 September 2012

Why PATCH is Good for Your HTTP API

A common problem for APIs is partial update; when the client wants to change just one part of a resource’s state. For example, imagine that you’ve got a JSON representation of your widget resource that looks like:

{
  "name": "abc123",
  "colour": "blue",
  "count": 4
}

and you want to update the “count” member’s value to “5”.

Now, you could just PUT the entire thing back with the updated value, but that requires a recent GET of its state, can get heavyweight (especially for mobile clients), and requires use of pre conditional requests to avoid “lost updates.”

For these and other reasons, many APIs define a convention for POSTing to resources that allows partial updates. E.g.,

POST /widgets/abc123?action=incrementCount

It seems simple, at first. However, this is a POST, so it doesn’t have any generic semantics; the server and client side developers have to write application-specific code to support it, then do QA on it, debug the corner cases, and eventually rewrite the API to fix the problems they inevitably find (partial updates can get subtle). Once you get a lot of these hanging around, it’s a pain.

Some folks go down the path of writing a more generic query parameter conventions for talking about the structure of a document in an API; this can be seen in the CIMI specification’s “$select” mechanism, as well as XCAP:

Assume that this document has a document URI of
"http://xcap.example.com/test/users/sip:joe@example.com/index",
where "test" is the application usage. This application
usage defines a default document namespace of
"urn:test:default-namespace". The XCAP URI:

http://xcap.example.com/test/users/sip:joe@example.com/index/
~~/foo/a:bar/b:baz?xmlns(a=urn:test:namespace1-uri)
xmlns(b=urn:test:namespace1-uri)

will select the first <baz> child element of the
<bar> element in the document.

These approaches still bring a lot of complexity to the API; what used to be a fairly simple URI space is now made much deeper, more complex, and brittle. There’s a lot more application-specific code to write, document, debug, understand and use. Blech.

Enter PATCH

A better way is to use the HTTP PATCH method. Rather than putting the semantics into the URI, or into an application-specific POST, we put them into a generic* request format — you can think of this as a diff — which is activated by the PATCH method.

This has a bunch of advantages. Since it’s a generic format, you can write server- and client-side code once, and share it among a number of applications. You can do QA once, and make sure you get it right. Furthermore, your API will become less complex, because it has less URI conventions, leading to more flexibility and making it easier to approach for new developers. 

Using this approach will also make caches operate more correctly; since modifications to a resource will “travel” through its URL, rather than some other one, the right stored responses will get invalidated.

One final benefit; PATCH is atomic, which means that if you need to do some complex changes to the state of a resource, you can do it with confidence. It also means that you can create synthetic resources and PATCH them if you need to orchestrate changes to the state of several resources.

So, what does PATCH look like?

PATCH with JSON

We’ve been working on a PATCH format for JSON in the IETF APPSAWG. Originally proposed by Paul Bryan, I’ve been editing it so that we can get it out the door. A request to patch a JSON document looks like this:

PATCH /widgets/abc123 HTTP/1.1
Host: api.example.com
Content-Length: ...
Content-Type: application/json-patch

[
  {"replace": "/count", "value": 5}
]

Easy to understand, and even write by hand. If it succeeds, the response can be as simple as:

HTTP/1.1 200 OK
Content-Type: text/plain
Connection: close

Your patch succeeded. Yay! 

If you want to get fancy, you can also use James Snell's Prefer mechanism to tell the server your preferences for the response, along with content negotiation for its format, of course.

PATCH with XML

The closest thing we have to json-patch for XML is RFC5261, but the media type it defines is application specific, and it’s somewhat complex to boot. I’m not aware of any Open Source implementations, so I’m not sure if it’d be best to start here, or just come up with something new. Thoughts? 

Or, just don’t use XML

Supporting PATCH in Frameworks

JSON Patch is almost ready for Working Group Last Call, which means it’ll be an RFC sometime soon. I want to get some implementation experience with it before then (there are already a few people dipping their toes in), and I think both client-side and server-side frameworks can make PATCH even easier. 

For example, on the client side, it should be easy to “record” modifications to a JSON object and then serialise them as a JSON Patch document. On the server side, the persistence layer can do lots to make PATCH automatic, as well as enabling things like enforcing access rights, triggering side effects, etc. 

If you’re aware of any JSON Patch implementation, have ideas, or want to help (we need to build a test suite), please comment below, or contact me.

* Some people have misinterpreted the PATCH specification as saying that any media type (e.g., application/json) can be used as a PATCH format. This is missing the point; if you do that, you’re not getting any benefit from shared code, etc. and you might as well use POST.


Filed under: HTTP Protocol Design Web

30 Comments

Jirka Kosek said:

Why you are not using XQuery Update (http://www.w3.org/TR/xquery-update-10/) for specifying XML patches? It's already standardized syntax and there are several implementations including open-source ones.

Wednesday, September 5 2012 at 10:05 PM +10:00

Mike Amundsen said:

I have been using a simple [patch][add /][replace /][remove /][/patch] format for a couple years (no support for move, copy, or test). I have been standing down while the JSON effort completes.

While non-trivial, I think it reasonable to pick up the work on an XML Patch diff format once JSON-Patch goes RFC.

Thursday, September 6 2012 at 12:42 AM +10:00

thomas.koch.ro said:

You might be interested in the small section 7.5.2 "patching resources" of my bachelor thesis. I saw three problems with PATCH:

  • No mediatypes for patch - Thank you for working on this!
  • No means to indicate the representation of a resource against which the patch should be applied
  • More philosophical: Is patch still rest since no full representation is submitted?

Example for the second point: A contact manager serving the portable contacts json format and a future vcard+json format. On receiving application/json-patch, which one should be the target of the patch?

I'm looking forward to your comments!

Thursday, September 6 2012 at 12:52 AM +10:00

rektide de la fey said:

JSON-LD at least allowed for a JSON all the way down idea, updates not an issue of API but merely the cognizance to address the proper ReSTful resource. Let me expand on your example:

{
  "name": {"@id": "/widgets/abc123/name", "@value": "abc123"}
  "colour": {"@id": "/widgets/abc123/color",
    "r": {"@id": "/widgets/abc123/color/r, "@value": 0},
    "g": {"@id": "/widgets/abc123/color/g, "@value": 0},
    "b": {"@id": "/widgets/abc123/color/b, "@value": 255},
  "count": {"@id": "/widgets/abc123/count", "@value": 4}
}

To use:

PUT /widgets/abc123/color/g
255

PUT /widgets/abc123/count
5

Here JSON declares addresses for sub-resources; no new APIs need to be concocted to update a sub-resource, we're free from /widgets/abc123?action=incrementCount. Rely upon ReSTful objects that weave the web of data about them.

I wish I could say I like JSON-LD, because with regards to creating a web of data I am zealously in favor of the plan. And JSON-LD gets most things right. With lamentation, I find the text of the Syntax specification and the entirety of the API specification to be regrettably if not criminally pedantic arm-twisting and hideous: I am in their choir, I already believe, and it's all I can do to make out what they're saying, the signal noise is overwhelming, originating from discussion of legacy topics like RDF. In spite of finding the specifications largely incomprehensible, the example I jotted down above demonstrates something rather pure, models what resourceful ought to mean on the web, and I wish JSON-LD the best.


W/r/t JSON Patch;

My primary interest in JSON Patch is atomicity: I can throw 300 different resource PUT requests at a server, but there are not great tools for coordinating those requests, nor for making sure my requests dont start interleaving reqeusts with someone else's ongoing updates. JSON Patch can return a nice easy identifier for an entire slew of changes, which is fantastic.

Glad to see Rob Sayre's old post continuing to pick up steam; thanks James Clark for helping take up the cause.

Easy to get me going on these topics, hope that's ok. Looking forwards to the day when I can make the time to get my streaming-data Arclamp project back under way, JSON-Patch was one of the biggies I wanted to tack to. Thanks mnot.

Thursday, September 6 2012 at 8:13 AM +10:00

rektide de la fey said:

JSON-Ref is more about pulling in other external data-elements than it is defining resourceful addressing, but it too could serve as a lighter-weight version of JSON-LD's "@id", and not take up so much semantic baggage. The unifying theme of making sub-resources their own first class resource is capital.

Thursday, September 6 2012 at 10:11 AM +10:00

markus-lanthaler.com said:

Hi Mark,

I've already tried to post that to ietf-types but apparently the list was renamed to media-types and I haven't been signed up yet.

So I'll ask it again here:

Why did you choose application/json-patch as media type instead application/patch+json?

Saturday, September 8 2012 at 2:01 AM +10:00

Johan Sundström said:

How would this be able to replace your incrementCount example, in a trivial case like two browsers visiting the same resource, each merrily clicking a "Like" button? Say, with an original state of "0 likes", two clicks in browser A (making it "2 likes") followed by one click in browser B (making it "1 like", not the "3 likes" we want).

As I read it, PATCH would only be useful for specific cases where the object being edited has no internal consistency constraints (i e where it's for some reason known that blindly overwriting one property always is a legal and safe operation, no matter what state that resource was in), or where you for some reason are certain that your current request has a server-global lock on the resource and that the client's idea of the object state is identical to the server's.

Am I missing something?

Saturday, September 8 2012 at 3:09 AM +10:00

Jon Moore said:

Regarding atomicity, one option would be to make conditional requests (If-Match). In the text world, the unified diff format carries optional context with it so that it's easier to understand where/if the patch can be applied. What might the json-patch equivalent to context look like?

Saturday, September 8 2012 at 2:01 PM +10:00

jaaju said:

Mark,

I find partial updates (and batched requests) are a common problem in HTTP that lead most RESTful designs to end up as RPCs! So, thank you for working on this.

I see that your model can be easily used to patch other mediatypes too, using json for the patch format. E.g., allowing xpath selectors in place of the json-pointer and accepting json-encoded xml fragments for the values should allow updates to xml mediatype resources. You may have to extend your model a bit to address attributes and namespaces.

In general, you should be able to define a generic patch format that can be used to partially update resources in any mediatype so long as the resource mediatype defines a mechanism to identify fragments. In practice it may turn out that these two are often the same or very similar, but they need not be. And, there can be cases where they can not be the same, e.g. text/plain.

As an example, given the document

foo bar baz

the patch instruction

[ { "add": "char=4", "value": "qux " } ]

changes the document to

foo qux bar baz

You need to provide a mechanism to choose a specific representation of the target resource (also pointed out by Thomas above), and that may be used to interpret the semantics of the fragment identifier.

Friday, September 14 2012 at 3:13 PM +10:00

Andrei Neculau said:

Out of curiosity, what do you make of this syntax https://github.com/algesten/jsondiff vs JSON PATCH?

Sunday, September 23 2012 at 8:06 AM +10:00

Jon Moore said:

Could you use XSLT as a media type for PATCHing an XML representation?

Tuesday, October 9 2012 at 11:15 PM +10:00

eugene-beresovksy.myopenid.com said:

The counterpart of PATCH would be a DIFF method. In my use case I have a long list of thousands of (JSON) objects that users need to download completely (I'm talking MBs of data here). Only a few objects change every day. Always having to download the whole list on every change (GET + If-None-Match) is slow and wasteful.

That's where DIFF (and some new header like 'Base-ETag' or so, specifying the ETag the diff is based upon) would come in handy. Obviously the server might not want to keep the whole history, in cases where the base version is not available any more (or if the given ETag never existed) I guess it would be a '412 Precondition Failed' and the client would have fall back to GET + If-None-Match.

Although PATCH is mentioned in a number of RFCs, DIFF seems to be of much less interest. I wonder why that is... Has anyone of you guys that looked into PATCH also contemplated the DIFF counterpart?

Wednesday, October 10 2012 at 2:44 PM +10:00

Mike Schinkel said:

Hi Mark,

How fully supported is PATCH on existing web servers, routers and other middleware? I'm concerned the web might need to be upgraded globally for this to work reliably in all cases? But I guess I can see it's value in supporting multiple operations per HTTP request.

Anyway, it's a shame that the following PUT:

	PUT /order/12345/status
	
	shipped
Doesn't get reflected in the cached representation of the following:
	GET /order/12345
Do you know if anyone considered as an option/addition to PATCH to add a PATCH-{op} header to specify the parent URL and the child path, i.e.
	PUT /order/12345/status
	PATCH-REPLACE /order/12345, /status
	
	shipped

Would that work? Pros, cons?

-Mike

Friday, December 14 2012 at 12:21 PM +10:00

http://openid.elfisk.dk/jornwildt said:

Has any work been done to support patching with binary data? I do not mean patching parts of a binary document, but rather being able to replace a binary document as part of a patch document?

Example:

[
{ op: "replace", path: "/Document", value: ... binary data ... }
]

This may though be more of a JSON than PATCH question, but perhaps some thought has been given to this already?

We have been considering two different solutions: 1) encode binary data as BASE64 and send it as a string, or 2) expose the binary document data as a separate resource which can be updated separately.

The BASE64 solution carries some overhead in the payload size, and the separate resource solution makes it impossible to do atomic/transactional updates of both the binary document data and related meta data in one single request. So neither are perfect.

Thanks, Jørn

Wednesday, January 2 2013 at 6:27 PM +10:00

Tim Stokes said:

I'm a bit surprised that the proposal is to bind the concept of partial updates to another HTTP method. None of the other methods dictate the style of the message to this degree. Couldn't this be just be part of future media types? I realize that there is value in establishing a widely adopted convention but it just seems too specific to have it's own method when POST seems adequate.

It's one thing to think about how a general transformation could be applied on the server but how do you communicate to the client the affordances for PATCH? The templates for what can be changed and what can't? Is this not specific to the domain.

Also if we are really doing Information Technology then while the consumable representation may appear as if state has been removed, replaces etc... in many cases we, at least should be, processing these types of messages in an event sourcing way. i.e. we continuously ADD revisions to a resource in which case the concept aligns perfectly with POST.

Tim Stokes

Friday, March 8 2013 at 1:13 PM +10:00

Tim Stokes said:

What I mean is that it's too representation orriented to have a general way to transform PATCH messages while I don't want to loose the resource orriented context of the HTTP methods. I POST state to my "resource" not a representation because the representation is decided by content negotiation. In fact my core business as a service is choosing how the posted state changes variouse representations of my resource. I don't see the value in a generic transformation model.

PATCH seems like its focusing on the representation rather than the abstract concept of the resource which is very important to me.

Friday, March 8 2013 at 10:10 PM +10:00

Tim Stokes said:

REST components communicate by transferring a representation of a resource in a format matching one of an evolving set of standard data types, selected dynamically based on the capabilities or desires of the recipient and the nature of the resource. Whether the representation is in the same format as the raw source, or is derived from the source, remains hidden behind the interface.

Tuesday, March 12 2013 at 3:58 AM +10:00

Joachim Wester said:

If anybody is interested, I made a new Javascript implementation (based on DRAFT 10) that focuses on footprint and performance (1.6 KB). It support applying patches to object trees.

I am currently adding a patch generator for outgoing patches (using ES6 Object.observe() or dirty checking A observe() is currently only in Chrome 27)

Sunday, March 17 2013 at 12:18 AM +10:00

https://me.yahoo.com/nickshanks#e0fbf said:

Hi Mark. I thought I ought to point out to your readers that this format may be overkill for many simple resources. Service designers should be kept in mind that a server can always accept x-www-form-urlencoded as a media type too!

The example you give would then just be:

PATCH /widgets/abc123 HTTP/1.1
Host: api.example.com
Content-Length: 7
Content-Type: application/x-www-form-urlencoded

count=5


By using this, it is easy to make the server compatible with POSTs from HTML forms using a method override parameter:

POST /widgets/abc123 HTTP/1.1
Host: api.example.com
Content-Length: 21
Content-Type: application/x-www-form-urlencoded

count=5&_METHOD=PATCH


My own intranet application uses this principle to support AJAX where available and fall back to UA default implementations otherwise. This way we get the semantics of HTTP as best supported by the client, and it's easy to have a single implementation at the back end that routes POSTs with an _METHOD parameter to the handler for the method specified.

Thursday, August 8 2013 at 6:21 PM +10:00

Leave a comment


Creative Commons