The Whole Web in a Python Dictionary

Saturday, 31 July 2004

A few days ago I blogged a straw-man API for client-side HTTP based on dictionaries. This turns out to be well-aligned with a project I’ve had on the back burner for a while; coming up with some Python APIs for HTTP that are usable, encourage good practice, and well-aligned with the specifications.

So, a first prototype is now available, with three modules defined; http.message, which defines common constructs for HTTP messages and their payloads, http.status, which enumerates the defined HTTP response status codes as exceptions to be raised during processing, and http.client, which currently contains a dictionary-like class along the lines discussed before.

For example, the following GETs the Apple home page, PUTs its representation to a fictional site, and then DELETEs it. When we try to access the new location, an exception corresponding to the HTTP error code is raised.

>>> from http.client import Dict
>>> web = Dict()
>>> apple = web['http://www.apple.com/']
>>> apple. __class__
<class http.message.Representation at 0x676c0>
>>> web['http://www.example.org/apple.html'] = apple
>>> del web['http://www.example.org/apple.html']
>>> new_apple = web['http://www.example.org/apple.html']
Traceback (most recent call last):
  File "", line 1, in ?
  File "http/client.py", line 50, in __getitem__
    response = dereference(request)
  File "http/client.py", line 97, in dereference
    raise response
http.status.NotFound
>>>

The one difference to my previous straw-man is in POST, which now goes like this:

>>> from http import message
>>> thing = message.Representation()
>>> thing.body = "hello. this is the POST body."
>>> thing.headers['content-type'] = "text/plain"
>>> thing_creation_result = web('http://www.example.org/thingCreator', thing)

This library is barely usable and I have a lot of plans for it, as can be seen in its TODO list. First and foremost is to support HTTP authentication, so that the PUT and DELETE functionality are more practically usable.

Comments and suggestions appreciated.

8 Comments

l.m.orchard said:

Excelent.. rubs-hands-together Something like this would work nicely for the REST-API’ed feed aggregation package I’m poking around with.

Monday, August 2 2004 at 9:24 AM

Ian Bicking said:

Have you looked at the libraries here: http://wwwsearch.sourceforge.net/

John Lee has done a lot of work on HTTP clients there.

Friday, August 13 2004 at 12:45 PM

Lion Kimbro said:

WOW! NEAT!

You may also be interested in something similar but different: nLSD graphs. Its a way of making networked native data structures.

You could combine the two ideas, so that you could segment and write out native data structures to different parts of the web.

Regardless- independent of all that-

WOW!

I’d want to make a webapp on top of that, that makes it so you can read, edit, and submit pages. And then a versioning system, on top of that, right? right?

Awesome!

(nLSD graphs: http://onebigsoup.wiki.taoriver.net/moin.cgi/nLSDgraphs )

Monday, August 16 2004 at 7:53 AM

Mark Nottingham said:

Props to AaronSW for the revised entry title.

Also, apologies to those who couldn’t access the .py’s over the weekend; I’ve just changed hosting arrangements, and there were a few glitches. Should be fixed now.

Tuesday, August 17 2004 at 3:01 AM

Mike D said:

Very cool. Something like this was outlined in the REST Wiki a while back. Except it used javascript syntax.

Is there a way for the members of ‘headers’ to surface directly in the language? For example thing.accept rather than thing.headers[‘accept’]

Also, the Representation class might benefit from having a property called ‘content’. This property gives you room to support a message body that is multi-part. It also lets you hang metadata about the content directly from that property (like length, type, encoding, etc)

Tuesday, August 17 2004 at 9:50 AM

Mike D said:

Forgot to add the link to the REST wiki

http://rest.blueoxen.net/cgi-bin/wiki.pl?PseudoCodeForRest

Tuesday, August 17 2004 at 9:55 AM

Mark Nottingham said:

Hi Mike,

WRT headers: That’s actually the part I’ve been working on for the longest; stay tuned.

WRT content: interesting; will think on it.

Thanks!

Tuesday, August 17 2004 at 10:20 AM

anthony baxter said:

Hm. I’d have thought passing a dictionary to a POST would be more obvious. That’s certainly what I’d expect to see.

Friday, August 27 2004 at 10:52 AM

mark nottingham

other HTTP APIs posts