Testing RESTful APIs the not-quite-as-hard way

Last week I wrote briefly about using wget and curl to test RESTful interfaces. A couple people at CERN wrote in to suggest I look at a tool they were quite happy with, called httpie.

I’m impressed. It seems to strike a lovely balance between expressiveness and simplicity. What’s especially brilliant is that it’s written for the common case of needing to customize headers and set specific parameters; you can do it straight off the command line. For what I was doing last week:

$ http GET http://localhost:8000/muppet/6 Accept:application/json
...

sets the Accept header in your request; sending data is unbelieveably easy. Want to post to a form? -f gets you url encoding, and meanwhile you just set parameters on the command line:

$ http -f POST http://localhost:8000/ name="Kermit" title="le Frog"
POST / HTTP/1.1
Accept: */*
Accept-Encoding: gzip
Content-Type: application/x-www-form-urlencoded; charset=utf-8
Host: localhost:8000
User-Agent: HTTPie/0.2.7

name=Kermit&title=le+Frog
...

Nice.

If you’re sending JSON it does things like set the Content-Type and Accept headers to what they should be by simply specifying -j (which sensibly is the default if you POST or PUT and have name=value pairs). And, -v gets you both request and response headers; if you’re testing at this level you usally want to see both. Good show.

$ http -v -j GET http://localhost:8000/muppet/6
GET /muppet/6 HTTP/1.1
Accept: application/json
Accept-Encoding: gzip
Host: localhost:8000
User-Agent: HTTPie/0.2.7


HTTP/1.1 200 OK
Cache-Control: max-age=42
Content-Encoding: gzip
Content-Type: application/json
Date: Thu, 09 Aug 2012 03:52:27 GMT
Server: Snap/0.9.1
Transfer-Encoding: chunked
Vary: Accept-Encoding

{
    "name": "Fozzie"
    "title": "Bear"
}
$

Speaking of bears, I’m afraid to say it turned out to be quite the bear getting httpie installed on Ubuntu. I had to backport pygments, requests, python-oathlib, and pycrypto from Quintal to Precise, and meanwhile the httpie package in Quintal was only 0.1.6; upstream is at 0.2.7 and moving at a rapid clip. I finally managed to get through dependency hell; if you want to try httpie you can add my network tools PPA as ppa:afcowie/network. I had to make one change to httpie: the default compression header in python-requests is

Accept-Encoding: identity, deflate, compress, gzip

which is a bit silly; for one thing if the server isn’t willing to use any of the encodings then it’ll just respond a normal uncompressed entity, so you don’t need identity. More importantly, listing deflate and compress before gzip is inadvisable; some servers interpret the order encodings are specified as an order of preference, and lord knows the intersecting set of servers and clients that actually get defalate right is vanishingly small. So,

Accept-Encoding: gzip

Seems more than sufficient as a default; you can always change it on the command line if you have to for testing. Full documentation at github; that said, once it’s installed, http --help will tell you everything you’d like to know.

AfC

Testing RESTful APIs the hard way

RESTful APIs tend to be written for use by other programs, but sometimes you just want to do some testing from the command line. This has a surprising number of gotchas; using curl or wget is harder than it should be.

Wget

Wget is the old standby, right? Does everything you’d ever want it to. Bit of minor tweaking to get it not to blab on stdout about what it’s resolving (-q) and meanwhile telling it to just print the entity retrieved to stdout rather than saving it to a file (-O -) is easy enough. Finally, I generally like to see the response headers from the server I’m talking to (-S) so as to check that caching and entity tags are being set correctly:

$ wget -S -q -O - http://server.example.com/resource/1
  HTTP/1.1 200 OK
  Transfer-Encoding: chunked
  Date: Fri, 27 Jul 2012 04:49:17 GMT
  Content-Type: text/plain
Hello world

So far so good.

The thing is, when doing RESTful work all you’re really doing is just exercising the HTTP spec, admittedly somewhat adroitly. So you need to be able to indicate things like the media type you’re looking for. Strangely, there’s no command line option offered by Wget for that; you have to specify the header manually:

$ wget -S -q --header "Accept: application/json" -O - http://server.example.com/resource/1
  HTTP/1.1 200 OK
  Date: Fri, 27 Jul 2012 04:55:50 GMT
  Cache-Control: max-age=42
  Content-Type: application/json
  Content-Length: 27
{
    "text": "Hello world"
}

Cumbersome, but that’s what we wanted. Great.

Now to update. This web service wants you to use HTTP PUT to change an existing resource. So we’ll just figure out how to do that. Reading the man page. Hm, nope; Wget is a downloader. Ok, that’s what it said it was, but I’d really come to think of it as a general purpose tool; it does support sending form data up in a POST request with it’s --post-file option. I figured PUT would just be lurking in a hidden corner. Silly me.

Curl

Ok, howabout Curl? Doing a GET is dead easy. Turn on response headers for diagnostic purposes (-i), but Curl writes to stdout by default, so:

$ curl -i http://server.example.com/resource/1
HTTP/1.1 200 OK
Transfer-Encoding: chunked
Date: Fri, 27 Jul 2012 05:11:00 GMT
Content-Type: text/plain

Hello world

but yup, we’ve still got to mess around manually supplying the MIME type we want; at least the option (-H) is a bit tighter:

$ curl -i -H "Accept: application/json" http://server.example.com/resource/678
HTTP/1.1 200 OK
Date: Fri, 27 Jul 2012 05:12:32 GMT
Cache-Control: max-age=42
Content-Type: application/json
Content-Length: 27

{
    "text": "Hello world"
}

Good start. Ok, what about our update? It’s not obvious from the curl man page, but to PUT data with curl, you have to manually specify the HTTP method to be used with (-X) and then (it turns out) you use the same -d parameter as you would if you were transmitting with POST:

$ curl -X PUT -d name=value http://server.example.com/resource/1
$

That’s nice, except that when you’re PUTting you generally are not sending “application/x-www-form-urlencoded” name/value pairs; you’re sending actual content. You can tell Curl to pull from a file:

$ curl -X PUT -d @filename.data http://server.example.com/resource/1

or, (at last), from stdin like you’d actually expect of a proper command line program:

$ curl -X PUT -d @- http://server.example.com/resource/1
You are here.
And then you aren't.
^D
$

That was great, except that I found all my newlines getting stripped! I looked in in the database, and the content was:

You are here. And then you aren't.

Bah.

After writing some tests server-side to make sure it wasn’t my code or the datastore at fault, along with finally resorting to hexdump -C to find out what was going on, I finally discovered that my trusty \n weren’t being stripped, they were being converted to \r. Yes, that’s right, mind-numbingly, Curl performs newline conversion by default. Why oh why would it do that?

Anyway, it turns out that -d is short for --data-ascii; the workaround is to use --data-binary:

$ curl -X PUT --data-binary @- http://server.example.com/resource/1

“oh,” he says, underwhelmed. But it gets better; for reasons I don’t yet understand, Curl gets confused by EOF )as indicated by typing Ctrl+D in the terminal). Not sure what’s up with that, but trusty 40 year-old cat knows what to do, so use it as a front end:

$ cat | curl -X PUT --data-binary @- http://server.example.com/resource/1
Goodbye
^D
$

The other thing missing is the MIME type you’re sending; if for example you’re sending a representation in JSON, you’ll need a header saying so:

$ cat filename.json |
    curl -X PUT --data-binary @-
    -H "Content-Type: application/json" http://server.example.com/resource/1
{
    "text": "Goodbye cruel world"
}
^D
$

which is all a bit tedious. Needless to say I’ve stuck that in a shell script called (with utmost respect to libwww-perl) PUT, taking content type as an argument:

$ ./PUT "application/json" http://localhost:8000/resource/1 < filename.json
HTTP/1.1 204 Updated
Server: Snap/0.9.1
Date: Fri, 27 Jul 2012 04:53:53 GMT
Cache-Control: no-cache
Content-Length: 0
$

Ah, that’s more like it.

AfC

Mounting a kvm image on a host system

Needed to mount a KVM disk image on a host system without booting the guest. I’ve used kvm-nbd in the past, but that’s more for exporting a filesystem from a remote server (and, in any event, we now use Sheepdog for distributing VM images around).

Found a number of references to using losetup but wanted something simpler; doing the loop part automatically has been a “take for granted” thing for years.

It turns out that if your image is in kvm-img‘s “raw” format already then you can pretty much access it directly. We found this article, dated 2009, which shows you can do it pretty quickly; assuming that the partition is the first (or only one) in the disk image:

# mount -o ro,loop,offset=32256 dwarf.raw /mnt
#

which does work!

I’m a bit unclear about the offset number; where did that come from?

An earlier post mentioned something called kpartx. Given a disk, it will identify partitions and make them available as devices in via the device mapper. Neat. Hadn’t run into that one before.

This comment on Linux Questions suggested using kpartx directly as follows:

# kpartx -a dwarf.raw 
# mount -o ro /dev/mapper/loop0p1 /mnt
#

Nice.

Incidentally, later in that thread is mention of how to calculate offsets using sfdisk -l, however that doesn’t help if you don’t already have the disk available in /dev. But you can use the very same kpartx to get the number of cylinders:

# kpartx -l dwarf.raw 
loop0p1 : 0 16777089 /dev/loop0 63
#

Ah ha; 63 sectors × 512 byte block size is 32256. So now we know where they got the number from; adjust your own mounts accordingly. :)

AfC

Comments:

  1. Pádraig Brady wrote in, saying “I find partx is overkill. I’ve been using the following script for years; lomount.sh dwarf.raw 1 /mnt is how you might use it.” Interesting use of fdisk, there.

Using tinc VPN

We’ve been doing some work where we really needed “direct” machine to machine access between an number of staff and their local file servers. The obvious way to approach this sort of thing is to use a Virtual Private Network technology, but which one?

There are a lot of VPN solutions out there. Quite a number of proprietary ones, and of course the usual contingent of “it’s-free-except-that-then-you-have-to-pay-for-it”. In both cases, why anyone would trust the integrity of code they can’t review is quite beyond me.

We’ve used OpenVPN for some of our enterprise clients, and it’s quite robust. Its model excels at giving remote users access to resources on the corporate network. Technically it is implemented by each user getting a point-to-point connection on an internal network (something along the lines of a 10.0.1.0/30) between the user’s remote machine and a gateway server, and then adding routes to the client’s system to the corporate IP range (ie good old 192.168.1.0/24). That’s fine so long as the assumption is that all the servers on the corporate network have the gateway as their default route, then reply packets to 10.0.1.2 or whatever will just go do default and be sent back down the rabbit hole. Gets messy with things like Postgres if your remote developers need access to the databases; in the configs you do need to add eg 10.0.1.0/24 to the list of networks that the database will accept connections from.

Anyway, that’s all fairly reasonable, and you can set up the client side from NetworkManager (install Debian package network-manager-openvpn-gnome) which is really important too. Makes a good remote access solution.

Peer to Peer

But for our current work, we needed something less centralized. We’re not trying to grant connectivity to a remote corporate network; we’re trying to set up a private network in the old-fashioned frame-relay sense of the word — actually join several remote networks together.

Traditional VPN solutions route all the traffic through the secure central node. If you’ve got one system in NSW and another in Victoria, but the remote access gateway is in California, then despite the fact that the two edges are likely less than 50 ms away direct path, all your traffic is going across the Pacific and back. That’s stupid.

A major complication for all of us was that everyone is (of course) stuck behind NAT. Lots of developers, all working remotely, really don’t need to send all their screen casts, voice conferences, and file transfer traffic into the central corporate network just to come all the way out again.

The 1990s approach to NAT implies a central point that everyone converges to as a means of getting their packets across the port address translation boundary. Things have come a long way since then; the rise of peer-to-peer file sharing and dealing with the challenges of internet telephony has also helped a great deal. Firewalls are more supportive and protocols have evolved in the ongoing attempt to deal with the problem.

Meet tinc

So the landscape is different today, and tinc takes advantage of this. According to their goals page, tinc is a “secure, scalable, stable and reliable, easy to configure, and flexible” peer-to-peer VPN. Uh huh. Because of its peer-to-peer nature, once two edges become aware of each other and have exchanged credentials, they can start sending traffic directly to each other rather than through the intermediary.

$ ping 172.16.50.2
PING 172.16.50.2 (172.16.50.2) 56(84) bytes of data.
64 bytes from 172.16.50.2: icmp_req=1 ttl=64 time=374 ms
64 bytes from 172.16.50.2: icmp_req=2 ttl=64 time=179 ms
64 bytes from 172.16.50.2: icmp_req=3 ttl=64 time=202 ms
64 bytes from 172.16.50.2: icmp_req=4 ttl=64 time=41.6 ms
64 bytes from 172.16.50.2: icmp_req=5 ttl=64 time=45.4 ms
64 bytes from 172.16.50.2: icmp_req=6 ttl=64 time=51.3 ms
64 bytes from 172.16.50.2: icmp_req=7 ttl=64 time=43.3 ms
64 bytes from 172.16.50.2: icmp_req=8 ttl=64 time=42.3 ms
64 bytes from 172.16.50.2: icmp_req=9 ttl=64 time=44.2 ms
...
$

This is with the tincd daemons freshly restarted on each endpoint. The first packet clearly initiates edge discovery, key exchange, and setup of the tunnels. It, and the next two packets, are passed across the Pacific to the central node. Ok, fine. But after that, the tunnel setup completes, and both edge nodes have been informed of the peer’s network addresses and start communicating directly. Nice.

See for yourself

Watching the logs under the hood confirms this. If you run tincd in the foreground then you can specify a debug level on the command line; I find “3″ a good setting for testing:

# tincd -n private -D -d3
tincd 1.0.13 (May 16 2010 21:09:47) starting, debug level 3
/dev/net/tun is a Linux tun/tap device (tun mode)
Executing script tinc-up
Listening on 0.0.0.0 port 655
Ready
Trying to connect to einstein (1.2.3.4 port 655)
Trying to connect to newton (5.6.7.8 port 655)
...

If you give it SIGINT by pressing Ctrl+C then it’ll switch itself up to the exceedingly verbose debug level 5, which is rather cool. SIGQUIT terminates, which you can send with Ctrl+. If you’re not running in the foreground (which of course you’d only be doing in testing),

# tincd -n private -kINT

does the trick. Quite handy, actually.

Performance is respectable indeed; copying a 2.8 MB file across the Pacific,

$ scp video.mpeg joe@einstein.sfo.example.com:/var/tmp

gave an average of 31.625 seconds over a number of runs. Doing the same copy but sending it over the secure tunnel by addressing the remote machine by its private address,

$ scp video.mpeg joe@172.16.50.1:/var/tmp

came in at an average of 32.525 seconds. Call it 3% overhead; that’s certainly tolerable.

Setup

Despite my talking above about joining remote networks, an important and common subcase is merely joining various remote machines especially when those machines are both behind NAT boundaries. That’s our in-house use case.

The tinc documentation is fairly comprehensive, and there are a few HOWTOs out there. There are a few gotchas, though, so without a whole lot of elaboration I wanted to post some sample config files to make it easier for you to bootstrap if you’re interested in trying this (install Debian package tinc).

tinc has a notion of network names; you can (and should) organize your files under one such. For this post I’ve labelled it the incredibly original “private“. Note that when you specify host names here they are not DNS hostnames; they are just symbolic names for use in control signalling between the tinc deaemons. Flexibility = Complexity. What else is new. Obviously you’d probably use hostnames anyway but administration of the tinc network doesn’t need to be co-ordinated with people naming their laptop my-fluffy-bunny or some damn thing. Anyway, on system labelled hawking I have:

hawking:/etc/tinc/private/tinc.conf

    Name = hawking
    AddressFamily = ipv4
    ConnectTo = einstein
    ConnectTo = newton
    Interface = tun0

Note that I’ve got an Interface statement there, not a Device one. That’s a bit add odds with what the documentation said but what I needed to make it all work. Only one ConnectTo is actually necessary, but I’ve got one server in California that is reliably up and one in Victoria that is not so I just threw both in there. That’s what your tincd is going to (compulsively) try to establish tunnels to.

hawking:/etc/tinc/private/hosts/hawking

    Subnet = 172.16.50.31/32

Somewhat confusingly, you need a “hosts” entry for yourself. Above is what you start with. Each host also needs a keypair which you can generate with:

# tincd -n private -K4096

with /etc/tinc/private/rsa_key.priv getting the private key and the public key being appended to the hosts/hawking file:

    Subnet = 172.16.50.31/32

    -----BEGIN RSA PUBLIC KEY-----
    MIICCgKCAgEAzSd5V91X6r3NB3Syh2FV8/JC2M7cx6o2OKbVzP6X5SFPI1lEH1AD
    7SfIlQF4TE++X8RcpJaBi4KjMS/Ul36Tuk75eKA18aNTBoVqH/ytY0BipQvJ6TUd
    BEkCjYrOUHFYOQn8MxQzziG6nk9tvhTWS0yKCNbd68e5i9uyKOem3R/pJsd/Kh9V
    wdVB51Wxs1Sv07OYmGYyRmGWh450wBNEmQfPHmM60Yh6uoQNJ0Ef41k1ZcswWcfO
    0jp9EOvbW/ZCdBW6teIYZ3GMuMB/cFj0Dw2fx6dHNHZVZrPcivt0cuOG8L4jNoHj
    HQUGuzMrpDN8N1ymM/eDlx+kBFYreKiEYGoWWqlZPNoY+bCekMrNf6Sr9bBwbj23
    xmY1jf6v1LkxGtOi4wWJfbU4xaMnquIRQe6FtB4LHp29l2SYWcpZnjuLcZ4ZoZLQ
    WK4bb0bUCAI/eYb19JRnfKEwS9MhYaQhZLWAJ3xyOt9u/Kk9KV7vWApxR1f5e2KT
    77A446eQU5aedm8nBDbd+WHqTdklAQ7SdRyYmbD8PoXBd3DGP6dFiURVTy8Wn4gz
    Bn7PMI3zmhfCMtwq/3A/xfyjQY3qesGCmKUwTno3fhv1DScS0rS9TRxZfyxlaOB1
    qjtlU79VhI0UKlha2Fv4XLshQ5dYEutpatpij0NzPYlwiQFphFQKStsCAwEAAQ==
    -----END RSA PUBLIC KEY-----

These are the public identifiers of your system and indeed the remote system in your ConnectTo statement must have a copy of this in its hosts/ directory. For nets of servers we maintain them in Bazaar and share them around using Puppet. Central distribution brings its own vulnerabilities and hassles; for very small team nets we just share around a tarball :).

You don’t need the /32, it turns out, but I left it in here to show you that tincd is effectively trading around network route advertisements, not host address.

hawking:/etc/tinc/private/tinc-up

    #!/bin/sh
    ifconfig $INTERFACE 172.16.50.31 netmask 255.255.255.0

This gets run when the net comes up. You can do all kinds of interesting things here, but the really magic part is assigning a broader /24 network mask than that given the interface in the hosts/hawking file. That means this interface is the route to the network as a whole (not just to a single-attached host on the other side of a point-to-point tunnel, which is what OpenVPN does, leaving the default gateway to sort it all out). Lots of other ways to wire it of course, but one /24 in RFC 1918 land is more than enough. I’ve even heard of some people using Avahi link-local networking to do the addressing.

I could have hard coded tun0 there, I suppose, but they supply some environment variables. Much better.

Now for the California node:

einstein:/etc/tinc/private/tinc.conf

    Name = einstein
    AddressFamily = ipv4
    Interface = tun0
    Device = /dev/net/tun

That one I did need a Device entry. Not sure what’s up there; it’s a server running Stable, so could just be older kernel interfaces. Doesn’t matter.

Note again though that the tinc.conf file doesn’t have a public IP in it or anything. Bit unexpected, but hey. It turns up in the hosts files:

einstein:/etc/tinc/private/hosts/einstein

    Address = 1.2.3.4
    Subnet = 172.16.50.1/32

    -----BEGIN RSA PUBLIC KEY-----
    MIICCgKCAgEAqh/4Pmxy5fXZh/O7NkvebFK0OP+YD8Ph7JvK8RsUn75FY3DXjCCg
    VNRR+kRhnVoKVJcIAuvW7Tbs4fovWELOJbbUbKea8G+HANCgOY5F0rkJVtIAcTCL
    Jg1OelAfhF6yHV4vVgcawafWiMF2CtprveHomCnOwCbGuTDwTBqaUBZ9IOLzU2bx
    ArVA2No9Ks+xaaeSHejYoii3+WT58HUccntmIYkcdBa0uKZSis1XLUwdT7Evr1Ew
    K54RyMMEPC0MUziYZhAA0Qqpz79EzLXAGgQeuFxLjPoW/NbAD0PEBmsdmI5odprp
    t9Tx11v/UuhK2fszYKjM+DF2pYxxrKlOyus58zx5KKJQjjrzazrru5Ny0DNf/E6Y
    uB2kUtt7TCmoZg2CLAbIkyGJEiK+Wy2x2mabGDgicIs422XVslz2EODSI3qqF+f6
    gu+h/vYvjZxglYrL0SxTRV7wkUc+o9OVXMMYPazgPIkwnBeLrEhGL8GS4wDIYu4G
    E89m9UBE0fhVPJyw4QSfdeJZ4PgpJk6SG/7koVsJqr9EZOLp53K7ipnPylUKaRLD
    mcarvoDO6ybCuHUVUsLuzZZStSG8JEEe/8jb/Ex7UNBzJ14Nglqtu0aUZ/tzkrdS
    nPFFhdIwlUctM7sWKVfBugEkWjs3sR+XRVsCjxMrpZX0lXzcw9vhu60CAwEAAQ==
    -----END RSA PUBLIC KEY-----

This file must be on every system in the net (that has a ConnectTo it) — it’s how the edges know where to call. So the same file is copied to hawking:

hawking:/etc/tinc/private/hosts/einstein

    Address = 1.2.3.4
    Subnet = 172.16.50.1/32

    -----BEGIN RSA PUBLIC KEY-----
    MIICCgKCAgEAqh/4Pmxy5fXZh/O7NkvebFK0OP+YD8Ph7JvK8RsUn75FY3DXjCCg
    VNRR+kRhnVoKVJcIAuvW7Tbs4fovWELOJbbUbKea8G+HANCgOY5F0rkJVtIAcTCL
    Jg1OelAfhF6yHV4vVgcawafWiMF2CtprveHomCnOwCbGuTDwTBqaUBZ9IOLzU2bx
    ArVA2No9Ks+xaaeSHejYoii3+WT58HUccntmIYkcdBa0uKZSis1XLUwdT7Evr1Ew
    K54RyMMEPC0MUziYZhAA0Qqpz79EzLXAGgQeuFxLjPoW/NbAD0PEBmsdmI5odprp
    t9Tx11v/UuhK2fszYKjM+DF2pYxxrKlOyus58zx5KKJQjjrzazrru5Ny0DNf/E6Y
    uB2kUtt7TCmoZg2CLAbIkyGJEiK+Wy2x2mabGDgicIs422XVslz2EODSI3qqF+f6
    gu+h/vYvjZxglYrL0SxTRV7wkUc+o9OVXMMYPazgPIkwnBeLrEhGL8GS4wDIYu4G
    E89m9UBE0fhVPJyw4QSfdeJZ4PgpJk6SG/7koVsJqr9EZOLp53K7ipnPylUKaRLD
    mcarvoDO6ybCuHUVUsLuzZZStSG8JEEe/8jb/Ex7UNBzJ14Nglqtu0aUZ/tzkrdS
    nPFFhdIwlUctM7sWKVfBugEkWjs3sR+XRVsCjxMrpZX0lXzcw9vhu60CAwEAAQ==
    -----END RSA PUBLIC KEY-----

Ok, you get the idea with the public keys, but I wanted to emphasize the point it’s the same file. This is what you need to share around to establish the trust relationship and to tell E.T. where to phone home.

The Address entry in the hosts/einstein files spread around is what tells edge nodes which have been configured to ConnectTo to einstein where the real public IP address is. You can use DNS names here, and could play dynamic DNS games if you have to (sure, further decentralizing, but). If you have a few machines capable of being full time central supernodes then you’ll have much better resiliency.

You do not, however, need to share a hosts/ file for every other node on the net! If laptop penrose is already connected in to einstein and has been assigned 172.16.50.142 say, and hawking joins einstein and tries to ping .142, the central node einstein will facilitate a key exchange even though neither hawking nor penrose have each others’ keys, and then get out of the way. Awesome.

And finally, this all works over further distributed topologies. When new nodes join the new edges and their subnets are advertised around to the rest of the net. So if central nodes einstein and curie are already talking, and sakharov joins currie, then traffic from our hawking will reach sakharov via eintstein and currie, and in fairly short order they will have handled key exchange, step out of the way, and hawking will be communicating with sakharov direct peer to peer. Brilliant.

Nothing stopping you from share around (or centrally managing out-of-band) the hosts/ files with the Subnet declarations and the public keys, of course; it’ll save a few round trips during initial key exchange. Up to you how you manage the trust relationships and initial key distribution.

For completeness,

einstein:/etc/tinc/private/tinc-up

    #!/bin/sh
    ifconfig $INTERFACE 172.16.50.1 netmask 255.255.255.0

No surprises there.

Applications

Using tinc to cross arbitrary NAT boundaries has turned out to be supremely useful. I have successfully used this from within my office, over 3G UTMS mobile broadband, at internet cafes around Australia, in airport lounges in the States, and even from beach-side resorts in Thailand. In all cases I was able to join the private network topology. In fact, I now just leave tincd running as a system daemon on my laptop. When I need to talk to one of the file servers, I ping, and it’s there.

One surprising benefit was in getting voice-over-Jabber running again. We had some horrible regressions with audio quality during the Maverick release series of Ubuntu Linux. At one point in our diagnostics we found that the STUN algorithms for local and remote candidate IP detection were preferentially choosing localhost virtual bridges with lower route metrics than the default gateway resulting in routing loops. We brought up tinc and since both parties were on 172.16.50.x, Empathy and Jingle chose those as the “best” network choice. Packet loss problems vanished and the audio quality really improved (it didn’t finally get resolved until we got a Natty base system, tore out the Unity stuff, and got GNOME 3 and Empathy 3.0 on board via ppa:gnome3-team/gnome3 but that’s a separate issue). And as a side-effect we’ve got some ice on our voice channel. Excellent.

I’ve since read about a number of other interesting applications. A frequent use case is not needing encryption. While most people would interpret the “private” in virtual private network as meaning “secure”, in the old days it just meant a custom routing and network topology layered over whatever the underlying physical transport was. One crew running a large farm of servers on cloud provided infrastructure struggled to enable their various distributed nodes to find and talk to each other. So they disabled the encryption layer but used tinc as a means to facilitate do IP-over-IP tunnelling and giving their sys admins a stable set of (private) addresses with which to talk to the machines. They gave a talk at FOSDEM [their slides here] about it.

Also at FOSDEM was a talk by the “Fair VPN” effort, who are looking at improving efficiency of the network when the number of nodes scales into the thousands. Some nodes are “closer” than others so presumably they should be used preferentially; you don’t really need to discover information about every other node in the network on joining, and so on. The fact that they were able to use tinc as a research platform for this is fascinating and a nice kudo.

Next steps

So I’m pretty pleased with tinc, obviously. We’ve had a very positive experience, and I wanted to put a word in. If you’re involved in network engineering or security hardening, then I’m sure they’d welcome your interest.

It would be outstandingly cool if we could work out a NetworkManager plugin to set this up on demand, but that can wait for tinc version 1.1 or 2.0. I gather they’re working on making the key exchange and configuration easier; what I showed above is obviously well thought out and flexible, but there’s no denying it’s a bit cumbersome; there are a fair number of little knobs that need to be just right. A fire-and-forget daemon cross-product with some form of automatic addressing would be brilliant. But on the other hand, when you put network and security in the same sentence there’s a limit to how much you want to happen without any direct influence over the process. As it stands now tinc strikes a good balance there, and is entirely suitable for an environment managed by properly competent sysadmins.

AfC

Updates

  1. Turns out I was wrong about needing the Interface statement. After Dan’s post I tried it without one and tincd worked fine. Then I remembered why I’d done it that way — without an Interface statement the network interface was named for the tinc net label, private in this case. Preferring tun0, I went back to manually forcing it for my own personal aesthetic benefit.

Get Some

The other day I was wading through Ulrich Drepper’s paper about how memory works these days and what programmers should do as a result.

My wife asked me what I was reading. I showed her the title and when she saw it concerned computer memory, she looked at me with a totally straight face and summed up the 114 page paper by answering “Yes, you should get some”.

Like most of Ulrich’s writing, this paper is immense: long on detail and well presented. I must admit that I agreed with many others that after having read the work I was left wondering what action, in practical terms, I should be taking as a result. Clearly most of this knowledge is very important for Kernel hackers and for those doing low level systems programming, but since I do higher level work (ie advising people with large enterprise systems), it is hard to asses the choices I might be making in the terms of what Ulrich describes. I can only hope that the people who work on the parts of the low level libraries that run next to the metal (such as drivers, GLib, and especially the VMs like the Java and Python runtimes) are aware of these issues and making design choices based on these insights.

AfC