Sometimes things crash. This is the normal order of things, even if we like to pretend that Linux is so much better than its proprietary competitors. When a native library crashes underneath a java-gnome program, however, this isn’t so much fun, because the actual process which crashed is a Java Virtual Machine.

Usually I see with crashes because of something I’ve done wrong in binding an underlying GNOME library from java-gnome. So I bisect & printf() my way down until I can find the thing that causes it, read the docs, and hopefully figure it out.

Recently, however, I’ve been getting crashes somewhere deep in libpangoft when my the app is first loading or worse just sitting there and I’m not doing anything more onerous than moving the cursor around a TextView. This is still likely due to something I’ve done wrong in my code or in the bindings layer, but when it’s not happening deterministically or on demand it’s hard to even begin to analyze the problem.

The OpenJDK HotSpot VM (formerly the Sun HotSpot VM) has a pretty good SIGSEGV handler; it does its best to show you what the C library call was that died, and what the Java and C call stacks were leading up to the crash. You may have seen them around as hs_err_pid10733.log and such [I wish it would just spew that out to stderr instead of troubling to write a file, but anyway].

Strangely, the only line I’m getting when reading the crash report is:

C  [libpangoft2-1.0.so.0+0x17687]

no other stack frames. Which is a bit strange, and decidedly unhelpful. So I’m going to need to try a bit harder to get a backtrace, it seems.

Getting a native C stack trace of a Java program with GDB

Much as I otherwise feel GDB is the most horrendous user interface in the history of civilization (ok, maybe second only to GPG’s command line interface), it does do one thing extraordinarily well and that’s stack backtraces of crashed programs. These days one normally runs one’s program in gdb, induces it to crash, and then runs bt:

$ gdb ./program
(gdb) run
SIGSEGV caught
(gdb) bt

And you get your stack trace.

That’s a bit of a pain with a Java “program” because as mentioned the process running is a Java Virtual Machine (and because invoking java is … almost as bad as GPG’s command line interface):

$ java ... gobbledygook ... package.Class arguments

Usually people put their invocation line in a shell script along with various environment setup and so on:¹

$ ./script arguments

and off they go. The complication is that’s not really easy to use with GDB, since you need to invoke gdb on the binary executable (the JVM) and then, once GDB is finally up, tell it to “run” that executable with a bunch of arguments. Which means you’re back to the gory mess:

$ gdb java
(gdb) run ... gobbledygook ... package.Class arguments
SIGSEGV caught
(gdb) thread apply all bt

which is incredibly tedious for casual use.

But the old fashioned 1980s way of using a debugger is to get a “core dump” of memory into a file called core and to run GDB on that. Just set your shell to core dump, then go back to running your program as normal:

$ ulimit -c unlimited
$ ./script arguments
Aborted (core dumped)
$ gdb java core
(gdb) thread apply all bt

and our stack traces will spill out in great gory detail. Hooray!

Incidentally, if you want to play with GDB and see what a HotSpot JVM is up to, then you need to induce it to crash; one way to do this is to send it a signal, say SIGSEGV or SIGBUS:²

$ kill -11 10733

Yeay, Open

The real point here is that with Sun having open sourced Java and it’s HotSpot VM implementation, we can now build Java ourselves and include debugging symbols [on Debian Linux, for example, install package openjdk-6-dbg along with the symbols for the various libraries in the GNOME stack, libgtk2.0-0-dbg and so on]. This means, at long last, we can actually run Java under GDB — something we weren’t able to do when Java was proprietary — and get lovely backtraces when it thunders in.

Yeay for crashes.

AfC

¹ Which is why people put this invocation into a shell script, which makes it even harder to debug because you’ve got to run ps axww or whatever to try and get the full command line used to run the program. {sigh}, but fixing this will have to wait for another day.

² Bernd Eckenfels suggests avoiding SIGSEGV as apparently he believes this is caught in some places and rethrown as NullPointerException. I’ve never observed that, but I thought I’d mention his advice to use SIGBUS instead.

Comments

• Josh Triplett wrote mentioning that he likes to use SIGQUIT to get his C applications to core dump, since you can trivially generate that signal from the console with Ctrl+. What he didn’t know was that Sun’s Java VM has always had a handler for SIGQUIT which prints a stack trace for each currently running Java thread (which is useful when trying to debug deadlock issues, but it’s the Java-side call stack only, not the native frames).

• Mark Wielaard mentioned a nice trick to attach to a crashing hotspot JVM to work around any core file limitations:

  java -XX:OnError="gdb - %p" <arguments>

• He and James Henstridge also note that having the “live, but almost dead program” around sometimes makes things a little easier on the debugger (as opposed to relying on a core dump). James suggests computers are fast, and just running all your [problem child] programs in gdb all the time. Fair enough, although in my case with such a hard to reproduce crash, I think I’ll wait on a core dump.

• Mark let me know that on Fedora you can get debug symbols for any package you are trying to inspect. If you do debuginfo-install java-1.6.0-openjdk (in this case) it will pull in every dependent debuginfo package also! He also notes that this crash in question might actually be a Pango problem, and cites this Fedora bug.

For a while now I’d been looking for something to just run up an HTTP server to answer simple queries.

The Java Servlet APIs are comprehensive, and there are a number of very good implementations (we use Resin in highly loaded environments and it scales like a dream; we recommend it heartily to our clients). Nevertheless, sometimes you’re working on something small or ad-hoc, and you just want a simple web server that doesn’t involve application descriptors, XML, configuring an entire service instance, and everything else. Somewhat to everyone’s long standing annoyance (not to mention the derision of people who work in other languages), there is nothing in the Java standard library to just get on with this.

In their release of Java 1.6, however, Sun quietly included such a server. Apparently it had been around for some time powering internals of various other projects, but in 1.6 they exposed it as a public library, in package com.sun.net.httpserver, and it ships alongside the rest of their Java VM and standard library.

It’s got a rather nice API. You fire up a server and add one or more handlers:

        addr  = new InetSocketAddress("localhost", 8000);
        server = HttpServer.create(addr, 10);
        server.createContext("/eg", new ExampleHandler());
        server.start();

the “contexts” are paths from root that are handled by different HttpHandler instances; implementing that interface is simply:

    public void handle(HttpExchange t) throws IOException {
        ...
    }

Once there, you’ve got a number of really straight forward methods on HttpHandler to get done what needs doing:

        t.getProtocol();
        t.getRequestMethod();
        t.getRequestURI();
        t.getRequestHeaders();
        t.sendResponseHeaders(code, length);

for the metadata, and an InputStream and an OutputStream from:

        t.getRequestBody();            
        t.getResponseBody();

respectively. You write the response to the later. Too easy. I did up a modified version of the example they have in the package description file as something that echos back the HTTP request as the response and it was quite straight forward. Java programmers can glance at EchoServer.java if they’re interested.

The catch

Nothing is ever simple. The bytecode for the com.sun.net.httpserver package is bundled in with the rt.jar that comprises the bulk of the standard library, but for whatever reason they didn’t include the JavaDoc for these classes in the same tree as the rest of the API documentation.¹ This is a huge pain, because once you’ve told Eclipse where to look for the docs for a given blob (in this case, for rt.jar) you can’t then tell it to look somewhere else for a sub part. And if you’re trying to develop something in a beautiful IDE like Eclipse, the idea of not having inline popup documentation and completion working properly is just a non-starter.

You can point Eclipse at either a JavaDoc tree, or at the raw sources and it’ll just extract the documentation on the fly. But since the src.zip shipping with Sun’s proprietary Java 1.6 didn’t include the classes for this web server code, that didn’t work either.

So that would have been the writing on the wall for Sun’s httpserver.

Open Java to the rescue

After chatting with some people in #gentoo-java and #classpath, we suddenly realized that the solution was at hand. Source code to Open Java is available.² And even though it’s still not yet a fully Libre replacement for Sun’s proprietary Java 1.6 (the size of the binary proprietary plugs you have to drop into it to make it work is staggering), it does contain most of the sources to the Java standard library. So we grabbed a copy of the openjdk 1.6 b08 code drop, and just pointed Eclipse at it as the sources for rt.jar for cases where it couldn’t find the JavaDoc when it was looking something up. And ta-da!

So, amidst considerable astonishment but great pleasure, here was an example of something someone had Open Sourced working in a synergistic way but in a way that they hadn’t quite expected. This wouldn’t have worked if Sun hadn’t Open Sourced (most of) Java; but because they have we were able to use those sources to solve a problem. Thanks to the heroes at Sun Microsystems who made this possible!

Eclipse users will have to adjust their filters so that com.sun.* isn’t excluded from Type completions and what not; While we’ve all loved our com.sun.* excludes, it blots out com.sun.net.httpserver. If you plan on using it, then I suggest switching to excluding com.sun.org.* and com.sun.xml.*, etc. A pain, but that’s what we get for Sun not having put this into the standard library.

Speaking of which, I doubt we’ll ever see this as a part of the “Java” (wouldn’t java.net.httpserver just do the trick?) because the marketing side of Sun has a fairly strong fixation on “Servlets are what you use for everything”. Oh well. It would be a really pleasant surprise, though. Here’s hoping.

But not really

We would never use this in something we were distributing to anyone outside; after all, they might be using some other VM and com.sun.net.httpserver isn’t available as a standalone library. And that’s not to mention scalability; if you need something bulletproof then use the Resin servlet runner. But if you need a quick-and-dirty but easy-to-use HTTP server and you happen to have Sun Java or the future tense Open Java available, then this should do the trick for you. And if you’re feeling like whipping up something of your own, this really does have a nice API (somewhat a rarity in the Java landscape) and so this might be a good starting point for you. Either way, have a look.

AfC

¹ As mentioned, the API docs for com.net.sun.httpserver are not in the usual location at docs/api/ along with the rest of the standard library but at docs/jre/api/net/httpserver/spec/ of all places. Who knows what they were smoking that day.

² I am very tired of apologizing to people who have just installed “Java” on their Linux boxes that they have to download something else. I’ve given up trying to explain the difference between a JRE and a JDK to people; even after I do people respond with “who cares; I just want a working Java that I can use to build your software.” Sun having attempted to brand their Libre implementation of Java as “Open JDK” just perpetuates this nonsense. Not including the compiler classes in something that is already 18 MB big is stupid, and it’s not like the desktop VM is what you use on a resource limited embedded device. Open Java works just fine as a name, thank you very much, and it is (almost) a Free Java. Almost.

A long-time critic of things Open Source showed up on the Classpath project’s mailing list and asked some rather provoking questions in a thread titled “Quality control and FOSS rant”. He at least ended with: “I suppose this is more of a troll than a criticism, sorry about that.”

Despite the flame bait, the thread contained some surprisingly insightful replies. It’s always great to hear some of the top software developers in the world noting their motivations and why they believe what they do works.

From Roman Kennke:

Both approaches (closed and open) apparently tend to produce relatively high quality code (or really crappy code, happens in both camps), where with the closed approach the developers (or vendors) have to take over 100% responsibility (because the end user has no way to interact with the development), which usually makes things very formal and slow, where the open approach relies very much on the end users reporting problems. In most active projects these are fixed really quickly, giving both the developers and the end users a warm fuzzy feeling ;-)

From Andrew John Hughes

There’s a lot to be said for feedback and interaction with your users that’s often overlooked. All the ideas of complicated quality control processes in the world is not going to make a user feel as loved as seeing someone responding quickly to their bug and fixing it in a short space of time.

From Mark Wielaard, a remark on the complex administrative process used by the project to review contributed code:

We do have a flow chart that people have to follow when contributing… It is all very formal really: http://gnu.wildebeest.org/~mark/patch.png

and from Archie Cobbs, a reminder about the track record of a certain formerly proprietary process on solving bug desperately desired by their user community:

The number #1 voted bug in their bug database has been unfixed for over 5 YEARS!

The comments on that bug make for hilarious reading, but the bigger point is this: the identity of the people making the decisions about the relevance of the issue are hidden. That sort of thing doesn’t inspire much hope for people on the outside. It’s not like we’re talking about national security or the future of western democracy; it’s a bug report that turned into a feature request for a piece of software that many, many people depend on! No one likes to be fed the line that their problem is so Top Secret that they won’t be told when (or even if) the problem will be addressed. The cloak of anonymity strikes again.

Fascinating thread.

AfC

I was just about to post an essay when one of my reviewers asked me:

“Hey, are you sure we did that? I can’t find it.”

“Sure,” I replied. “I put your project’s file name in our source code for reference. Let me pull it up.”

Tap tap tapity tap.

“Hm. Wait a minute. That’s not the right file after all.”

“Cause it sure seems strange that we would have done that, and I can’t find it there. You sure it wasn’t an older version?”

“No, I don’t think so. I only pulled the code out of VCS the other day. I’m sure that’s what I was looking at. In fact, I wouldn’t have noticed it except for the comment that explained what your code was doing which shocked me into reading it more closely. It was so unconventional, so amazing, so inspiring! I’m sure I saw it in here!”

Still looking.

It’s hard to credit someone for a good idea when you’re sure you got it from them but no one can find it anymore. And I’m an engineer. I know about writing things down. {sigh}

AfC

Not four hours after Sun released Java under the GPL + Classpath Exception, Edwin Steiner, one of the hackers on the Cacao Java VM, gave the HotSpot VM a spin and posted some analysis.

He notes that he did hit one stumbling block:

“Building is very easy, but I found out that the disassembler does not seem to be supplied.”

But check this out:

“So I quickly hacked in the cacao disassembler, and it works! :)”

Awesome! And the cool part is, as Mark Wielaard observed, since both are GPL, the disassembler hack is just fine to distribute.

Freedom to do your own thing. That, ladies and gentlemen, is the freedom in Software Libre.

AfC

Well it’s official. Sun is releasing Java under the GPL + Exception, just like Classpath is licenced!

The initial release includes the HotSpot VM, their Java compiler, and a few other tidbits. And get this: even the image of Duke, the Java mascot, has been released under an Open Source licence! These people really get it!

The reference implementation of the libraries aren’t out yet, though, so naturally what everyone wants to know is what the encumbrance situation is with the various classes in that library. Based on comments from various Sun people in #open-source-java on OFTC (and they’ve promised a full list shortly), it would seem the only major problems are in the areas of colour, font rasterization and graphics rasterization.

These were characterized as fairly major issues; it was funny to see the Classpath community respond with a unanimous “not for long they won’t be!”

Java Libre!

AfC

Well it’s all over the net now. Sun just might actually release Java under the GPL.

Before we get all excited, I would observe that this article, from Doctor Dobb’s Journal (ordinarily quite a respected publication) cites “several industry sources”. That’s not exactly a convincing reference. And even if it is hard news, my guess is that this is just an official trial balloon to see what industry reaction will be. Fair enough.

Now that I’ve been sufficiently pessimistic, listen to this:

The company is very close to announcing that it will put the mobile (ME) and standard (SE) editions of the Java platform into the GNU General Public License (GPL), with the Java Enterprise Edition and GlassFish reference implementation to follow

They then quote a developer as saying “I think Sun realizes that their other pseudo-open source efforts don’t work” — which is very true. Community reaction to CDDL has always been weak. It’s just too bloody complicated.

The article goes on to say:

Offering Java only under the GPL would have a cataclysmic effect on the software industry, forcing Java platform developers to freely release their contributions if they continue developing around the platform’s GPL code. … A more likely scenario is that Sun would offer dual licensing for Java

According to a Sun executive … dual-licensing is the plan. Sun will have open-source and commercial licenses, differentiated by their IP indemnifications

Awesome.

I noted in an article about Open Source Java a few months ago that Sun had indeed made it quite clear that this time they were listening and that they had made a real effort to get connected with the Free Java community. So if this is a trial balloon, then everyone who cares about software freedom needs to express their support and to continue encouraging Sun to choose this course. And if this is for real, then wow, and congratulations to Sun!

AfC

There is a project out there called the Byte Code Engineering Library (BCEL), these days under the Apache Jakarta umbrella. It has been around for quite a long while and is both widely used and well regarded. They have a documentation page which has some surprisingly informative background material which caught my eye.

Long as I’ve been working with Java, I’ve never bothered much learning about the internals of the language that Java Virtual Machines interpret or of how class files are laid out. The overview in the BCEL manual of the .class file format and how Java VMs work on bytecode is excellent — far more informative than the introductory material in the more usual official references. In particular, [section 2](http://jakarta.apache.org/bcel/manual.html#2 Java Virtual Machine), describing the general design of the Java Virtual Machine, including a section on the [Java class file format](http://jakarta.apache.org/bcel/manual.html#2.1 Java class file format), [bytecode instruction set](http://jakarta.apache.org/bcel/manual.html#2.2 Byte code instruction set) will be a fascinating read for anyone interested in language design issues.

In case you’re wondering why this came up, I’ve been working on a design review for how we might go about re-engineering java-gnome_, and have been doing one last analysis of the more outlandish outlying possibilities (ie unlikely to be chosen, but still worth a bit more investigation). Quite a number of problems would go away if we could do runtime class generation. It might be possible through BCEL._

As it happens I am extremely reluctant to follow this path largely because it’s a hell of a lot easier to express one’s ideas in Java and let a compiler generate the bytecode than attempting to muck about with bytecode oneself. Nevertheless, it may offer a possibility in terms of things like GObject properties which are largely only available via introspection at run time.

Naahhhh. :)

AfC

It’s been an interesting week in Free Java land. The buzz, of course, is about that fact that Sun may finally open source Java after all. All still rumour and speculation (you can easily find comments ranging from “why it will never happen” to “It’s already open source, didn’t you know?” [and no, while Sun finally relenting and allowing the community distros to actually redistribute their JDK binaries is a nice step in the right direction, it's still non-free, and that's that]. Still and all, the man has said that Java will be free. It might take them a while, but there’s really no reason to doubt that it will happen.

Which leads people in the Free Java communities to suddenly ponder their own raison d’étre.

It takes a lot of guts to ask yourself uncomfortable questions. “Has what I’ve been working on for the last many years suddenly become irrelevant?” has gotta be pretty high up on the discomfort list. What takes real courage, however, is to pose such rhetorical questions in the open, in a public forum, exposed by the harsh glare of the internet to the scrutiny, criticism and scorn of the world.

Regarding the possible freedom of Sun’s proprietary Java VM and class libraries, Tom Tromey, one of the hackers working on gcj and Classpath recently wrote:

“This makes me feel very weird. I assume for a moment that it is true and that it happens under acceptable conditions: it comes pretty soon, it is complete, it is under a non-crazy license. On the one hand, hallelujah! This is what we’ve wanted these 10 years.

On the other hand… I wonder what I’ll do with myself. I suppose there are plenty of interesting things to work on. Even the Sun JDK I suppose. But the dislocation goes far beyond my future to-do list. What does this mean about all the work I’ve done? Is it a waste?

I do have my own answers for those questions. Everything is born, lives for a while, and dies; our programs are no different. That they die early or late doesn’t render them meaningless — only dead. And meaning itself is something we bring, in interpretation; it isn’t an intrinsic quality. Of course it is one thing to think that and another to know.”

The commentary around Java of late is, in the main, entirely predictable. Some ask if Java is at some perilous crossroads. Others cry out that somehow Java will become irrelevant if it becomes free software. I suspect, however that most prognosticators who write for a living have missed a bet.

More than anything, the one thing that has kept Free and Open Source Software thriving as a phenomenon are its ability to embrace competing views. Not always well, not always politely, but taking in the phenomenon as a whole, competition is at its heart. Sometimes one approach wins out over the others to such a significant degree that the competitors fade away. Sometimes new developments occur in fields long thought static which suddenly shake the industry to its foundations and start entirely new fields off at a run. But quite often, several projects poodle along in the same space quite happily. The constant lament from you read in the trade press is “there’s too much choice” which really makes me laugh. In all the business textbooks on my wall, I can’t find one reference to a company that was dismayed that amongst its suppliers there was competition.

That means that existence of Free Java will continue to have great merit. The Classpath project, along with the libré VMs that use it provide an active and viable alternative to [at the moment] proprietary Java. One factor is the sheer number of platforms that these VMs run on, many which would never have been supported by the dominant player in the industry. Free Java has likewise been the catalyst for radical innovations (witness the marvel that is gcj). But most of all, because it’s competition for the incumbent, people on all sides keep their creative juices flowing.

Per Bothner raised this very constructively in a recent post to the gcj mailing list:

What with the Harmony project, and possible sometime-in-the-future open-sourcing of JDK, we need to think about how can gcj “compete”. I think we need to start focusing more on performance (both speed and footprint), and otherwise “leverage” of the advantages of ahead-of-time-compiling.

which is indeed an astute conversation for them to have — and this from a project that already has a strongly performing platform.

The one thing Sun has done really good job of (at least from the corporate perspective that I certainly have) is preserving the Java specification. As we go forward into somewhat uncertain waters, the one thing above all that will remain critical is the ability of someone to tell whether a given implementation is compatible — and that’s why it’s so important that the compatibility, compliance and testing kits be made libré available. With that in hand, then the fragmentation of the Java platform that the pundits are proclaiming won’t ever be a problem.

Even without it, people have done a pretty good job. The 99.06% API coverage that Classpath has reached is pretty damn good.

AfC