Testing RESTful APIs the hard way

RESTful APIs tend to be written for use by other programs, but sometimes you just want to do some testing from the command line. This has a surprising number of gotchas; using curl or wget is harder than it should be.

Wget

Wget is the old standby, right? Does everything you’d ever want it to. Bit of minor tweaking to get it not to blab on stdout about what it’s resolving (-q) and meanwhile telling it to just print the entity retrieved to stdout rather than saving it to a file (-O -) is easy enough. Finally, I generally like to see the response headers from the server I’m talking to (-S) so as to check that caching and entity tags are being set correctly:

$ wget -S -q -O - http://server.example.com/resource/1
  HTTP/1.1 200 OK
  Transfer-Encoding: chunked
  Date: Fri, 27 Jul 2012 04:49:17 GMT
  Content-Type: text/plain
Hello world

So far so good.

The thing is, when doing RESTful work all you’re really doing is just exercising the HTTP spec, admittedly somewhat adroitly. So you need to be able to indicate things like the media type you’re looking for. Strangely, there’s no command line option offered by Wget for that; you have to specify the header manually:

$ wget -S -q --header "Accept: application/json" -O - http://server.example.com/resource/1
  HTTP/1.1 200 OK
  Date: Fri, 27 Jul 2012 04:55:50 GMT
  Cache-Control: max-age=42
  Content-Type: application/json
  Content-Length: 27
{
    "text": "Hello world"
}

Cumbersome, but that’s what we wanted. Great.

Now to update. This web service wants you to use HTTP PUT to change an existing resource. So we’ll just figure out how to do that. Reading the man page. Hm, nope; Wget is a downloader. Ok, that’s what it said it was, but I’d really come to think of it as a general purpose tool; it does support sending form data up in a POST request with it’s --post-file option. I figured PUT would just be lurking in a hidden corner. Silly me.

Curl

Ok, howabout Curl? Doing a GET is dead easy. Turn on response headers for diagnostic purposes (-i), but Curl writes to stdout by default, so:

$ curl -i http://server.example.com/resource/1
HTTP/1.1 200 OK
Transfer-Encoding: chunked
Date: Fri, 27 Jul 2012 05:11:00 GMT
Content-Type: text/plain

Hello world

but yup, we’ve still got to mess around manually supplying the MIME type we want; at least the option (-H) is a bit tighter:

$ curl -i -H "Accept: application/json" http://server.example.com/resource/678
HTTP/1.1 200 OK
Date: Fri, 27 Jul 2012 05:12:32 GMT
Cache-Control: max-age=42
Content-Type: application/json
Content-Length: 27

{
    "text": "Hello world"
}

Good start. Ok, what about our update? It’s not obvious from the curl man page, but to PUT data with curl, you have to manually specify the HTTP method to be used with (-X) and then (it turns out) you use the same -d parameter as you would if you were transmitting with POST:

$ curl -X PUT -d name=value http://server.example.com/resource/1
$

That’s nice, except that when you’re PUTting you generally are not sending “application/x-www-form-urlencoded” name/value pairs; you’re sending actual content. You can tell Curl to pull from a file:

$ curl -X PUT -d @filename.data http://server.example.com/resource/1

or, (at last), from stdin like you’d actually expect of a proper command line program:

$ curl -X PUT -d @- http://server.example.com/resource/1
You are here.
And then you aren't.
^D
$

That was great, except that I found all my newlines getting stripped! I looked in in the database, and the content was:

You are here. And then you aren't.

Bah.

After writing some tests server-side to make sure it wasn’t my code or the datastore at fault, along with finally resorting to hexdump -C to find out what was going on, I finally discovered that my trusty \n weren’t being stripped, they were being converted to \r. Yes, that’s right, mind-numbingly, Curl performs newline conversion by default. Why oh why would it do that?

Anyway, it turns out that -d is short for --data-ascii; the workaround is to use --data-binary:

$ curl -X PUT --data-binary @- http://server.example.com/resource/1

“oh,” he says, underwhelmed. But it gets better; for reasons I don’t yet understand, Curl gets confused by EOF )as indicated by typing Ctrl+D in the terminal). Not sure what’s up with that, but trusty 40 year-old cat knows what to do, so use it as a front end:

$ cat | curl -X PUT --data-binary @- http://server.example.com/resource/1
Goodbye
^D
$

The other thing missing is the MIME type you’re sending; if for example you’re sending a representation in JSON, you’ll need a header saying so:

$ cat filename.json |
    curl -X PUT --data-binary @-
    -H "Content-Type: application/json" http://server.example.com/resource/1
{
    "text": "Goodbye cruel world"
}
^D
$

which is all a bit tedious. Needless to say I’ve stuck that in a shell script called (with utmost respect to libwww-perl) PUT, taking content type as an argument:

$ ./PUT "application/json" http://localhost:8000/resource/1 < filename.json
HTTP/1.1 204 Updated
Server: Snap/0.9.1
Date: Fri, 27 Jul 2012 04:53:53 GMT
Cache-Control: no-cache
Content-Length: 0
$

Ah, that’s more like it.

AfC