Whoopsie Ate My Server RAM?!? (An Ubuntu Tweak)

Whoopsie is Ubuntu’s automated error tracker submission daemon, it’s Wiki page says:

Ubuntu’s error tracker explains crashes, hangs, and other severe errors to end users; lets them report an error with the click of a button;

(emphasis, mine)

Which, certainly sounds like a good idea — with two minor drawbacks:

a) Ubuntu Server has no “buttons” to click.

b) Reporting “end-user formatted errors” on a Server is probably not useful — now, it might just be me, but seeing:

Apache has crashed

versus:

child pid 7716 exit signal Segmentation fault (11)

the latter is probably more useful to diagnosing the problem that caused the signal 11 in the first place.

Now to be fair to Ubuntu, the “Apache has crashed” error doesn’t really exist as a current error exhibited from the Whoopsie daemon, but that’s not the point of this post — which is actually the large amounts of system memory that it seems to use by default.

In addition, i’d suggest that on a desktop installation the Whoopsie daemon is actually quite useful, especially for non-technical end-users and first time Linux users (there are exceptions, such as the consistant crash that occurs on the Unity desktop environment on Ubuntu 12.04 or 12.10).

However, we’re talking about servers in this instance, where the same rules do not generally apply.

So, moving right along…

Compare the following two screenshots, which are examples from one of our running machines, the first with Whoopsie running on an installation running nginx and varnish — but it’s otherwise a barebones installation, just web software, no PHP, no SQL, no DNS.

htop, showing our pre-removal position.

Using htop, this is our install pre-removal.

The second, after purging the whoopsie program from our Ubuntu installation.

htop, showing our post-removal position.

Again, using htop, this is our install post-removal.

183M of Virtual RAM (vmsize) for a daemon we’ll never use, seems excessive.

Of course, using the incredibly useful smaps.pl script by Ben Maurer, we can see a much more detailed breakdown of the memory used by Whoopsie — both those that are shared with other processes on the system, as well as those that are exclusive to the particular program we’re looking into.

To run Smaps.pl

If you’re interesting in duplicating the tests described in this post, you’ll need to do the following:

a) the actual script, which doesn’t seem to exist on Ben’s original page, but you can find with a little searching through the interschnizel.

b) and, if you don’t already have it make :

# apt-get install make

c) then, simply install the required perl modules using CPAN, like:

# cpan App::cpanminus
# cpan Class::Member
# cpan Linux::Smaps

Once we’ve done all that, simply run:

# perl smem.pl <process ID>

(Obviously, if you’re not actually building binaries on your server, you should either remove the make software when you’ve completed your tests, or better, install a copy of your installation under virtualisation and run your tests there.)

So, if we now go ahead and analyse the amount of memory that the whoopsie daemon is actually using a total of 4.3M of RAM and we also find libgobject, x509, heimdal, SQLite and a number of other libraries that are, frankly un-needed on a server that is only serving static pages from a lightweight HTTP server.

smaps.pl showing a breakdown of the Whoopsie process.

Smaps.pl’s view of the Whoopsie process (trimmed to the first 25 lines).

It looks even worse, especially when the static webserver has a total vmsize of 31M and is only using 996kb of RAM of it’s own accord.

smaps.pl showing a breakdown of the NGiNX process.

Smaps.pl’s view of the NGiNX process.

So, in order to save yourself a little RAM on a server installation of Ubuntu — especially if you are running a low-memory system, are experiencing swappiness issues, or just want to have a leaner system.

# apt-get purge whoopsie libwhoopsie0

Razer Megalodon USB Headsets and Fedora (or Ubuntu)

Over my break, I picked up a bunch of new Razer peripherals for my machines — they make remarkably good wired mice and the odd piece of keyboard brilliance, so while I was at it, I thought about getting some headsets.

I settled on the Razer Megalodon after finding the other candidates had smaller inner ear cups (and as someone with boney ears, wearing over-ear cans that don’t fit for more than a few hours tends to hurt a bit) — the fact it had it’s own DSP was nice and the microphone did a good job at filtering the noise in the store where I bought it, so $more-than-I-wanted-to-spend later, I unboxed it and threw it at my two primary workstations.

Now, we’ve not had a lot of luck with headsets around here — i’m usually the guy that buys them, breaks them in the first Jabber discussion and then either a) returns them, for another type or b) hacks on various settings, throws things in a PPA or generally gives people enough fodder to say “Try another one”.

Having tried 3 types of Turtle Beach as well as an Astro A40 and having them all returned for one reason or another before christmas, the first outing started no better for the Razer’s.

First off, my Ubuntu 12.10 installation, as well as 12.04, as well as Fedora 18, loads usb-storage (as well as usb-audio) for them, which hampers the ability for the Sound Control on my GNOME 3.x based desktops to see them as an Output device (and there’s nothing worse than getting the input to work through the microphone, but nothing coming out the ears).

Often, we could fix this by doing a “pulseaudio -k” from a terminal, but only having “modprobe -f usb-storage” first (which is dangerous, especially for those of you with other devices plugged in when you’re trying to use the headset).

Second, all of them provide this error, streamed through dmesg(8),

cannot submit datapipe for urb 0, error -28: not enough bandwidth

Regardless of if the headset is plugged into a USB2 or USB3 port, even if it’s the only device on the bus.

Thirdly, The output is always muted on first plug-in, if I unmute it once (from alsamixer or from the PulseAudio control panel) it stays unmuted until I either power the machine off, or I plug it into a different port.

… and finally, the sound crackles once it does work, no worse than any of the other models we tried, but quite badly, especially if the DSP functions on the control configuration aren’t turned up to “just the right amount”

[Frustration!]

After a little Googling, turns out 3) can be fixed a multitude of ways, the easiest fix was here

Luckily, 1) & 2) have also been handled rather ingeniously by a couple of clever people with USB headsets, involving relaunching pulseaudio from udev — which only require little modifications to get working with the Megalodon.

The original post is here and there’s a guy who’s tweaked it further to run from a single file here

If you’re using the latter of the two versions with the Megalodon, you’ll want to make the following changes:

First, save a copy of the script, or ‘git clone’ it somewhere.

Second, change line 10 and 11 to the Input and Output sources of the Megalodon, which are:

OUTPUT="alsa_output.usb-Razer_Razer_Megalodon-00-Megalodon.analog-surround-71"
INPUT="alsa_input.usb-Razer_Razer_Megalodon-00-Megalodon.analog-mono"

Lastly, change the udev line on line 48 to the following, in order to make the control of the script more fine-grained, from:

#KERNEL=="card?", SUBSYSTEM=="sound", ATTR{id}=="Everyman", ACTION=="add", 
RUN+="/path/to/this/script/usb-headset --fork"

to:

#KERNEL=="card?", SUBSYSTEM=="usb", ATTR{idVendor}=="1532", ATTR{idProduct}=="000e", 
ATTR{product}=="Razer Megalodon", ACTION=="add", 
RUN+="/path/to/this/script/usb-headset --fork"

Now save the script somewhere, change it’s permissions to 0750 and follow the instructions accordingly — at this point you’ll have working Razer Megalodon hardware, but the sound will be a bit choppy and you may experience hangs on suspend, or audio dropouts when making calls with Empathy, which are evidenced by the URB error we saw above.

These problems are fixed by tweaking the way PulseAudio works with USB hardware and, if you have Intel HD Audio hardware, the power-saving functions on the codec and controller, which appear to be a little too overzealous in Fedora 18 and Ubuntu 12.04.1 or 12.10.

To fix them, as root, first make a file in /etc/modprobe.d called “usb-audio.conf” and add the following line:

options snd-hda-intel power_save=0 power_save_controller=N

(Yes, that’s a capital N at the end of the line.)

Then, we need to change the scheduling used by PulseAudio from the “new” style back to the older timer-based style, which seems to fix the bandwidth problems caused by the DSP in this particular card — to do this, we open /etc/pulse/system.pa as root and change line:

load-module module-udev-detect

to:

load-module module-udev-detect tsched=0

Save the file and either do a pulseaudio -k as your user or reboot the system to make the entire set of changes take effect.

… and finally, after much tweaking — you should have a working Razer Megalodon card on your Linux installation that runs just as well as it does under other operating systems out-of-the-box.

note: This post also fixes Ubuntu bug 658915 — which is not necessarily fixed, but is certainly not completely broken anymore.

WINE, i915 or i965 & intel_do_flush_locked

Recently, while debugging a WINE based application for a friend, who is by far not a Linux neophyte — I started seeing:

"intel_do_flush_locked failed: Input/output error"

on the CLI, as well as:

[ 366.415824] [drm:i915_hangcheck_elapsed] *ERROR* Hangcheck timer elapsed... GPU hung
[ 366.415847] [drm:i915_wait_request] *ERROR* i915_wait_request returns -11 (awaiting 77445 at 77442, next 77446)
[ 366.564710] [drm:i915_hangcheck_elapsed] *ERROR* Hangcheck timer elapsed... GPU hung
[ 366.564711] [drm:i915_wait_request] *ERROR* i915_wait_request returns -11 (awaiting 77471 at 77467, next 77472)

In the /var/log/Xorg.0.log file.

The only new thing was an upgrade from Ubuntu 10.04LTS to 12.04LTS a few months earlier, which was completely stock, aside from the WINE PPA being used, which was more a leftover from one the original machine had been installed some years ago with 8.04LTS.

… and this was the first time this particular issue had occurred.

They were using the WINE PPA already, so the copy of WINE was already up-to-date, although the program had no issues running between WINE 1.0 and 1.4, so I wasn’t terribly concerned that upgrading to 1.5.x would have anything to do with it.

Having tried numerous i965 related tweaks on the web, then finding that I was not alone in the issue — I investigated the alternative ways of fixing the problem.

The xorg-edgers PPA made little difference, with an updated driver and MESA installation — there was a longer period before the application crashed, but it still crashed after a few minutes, better than the standard 12.04 repository (as of 2012-11-28) where the application would crash immediately though.

The x-swat/x-updates PPA (the “more stable” version of the PPA above) was slightly better, WINE at least created the virtual desktop window before the messages started appearing, we called that “progress” and went looking for things specific to the code available in that repository.

Using i965 Frame Buffer Compression in the GRUB configuration made no difference, but this time other things on the desktop (GNOME Shell) started restarting intermittently before we’d even got to the terminal window in order to start WINE.

According to the PowerSavingTweaks page, Frame Buffer Compression (FBC) does:

Framebuffer compression reduces the memory bandwidth on screen refereshes and depending on the image in the framebuffer can reduce power consumption. … Some users report this breaks when using Unity 3D.

I thought we may be having issues because it was an older motherboard, a 1.3Ghz Celeron CPU board with 4GiB of RAM — possibly a Power Management thing too, Indeed, reverting the i915_enable_rc6 fix by changing the /etc/default/grub file to:

i915.i915_enable_rc6=0

“fixed” the problem even further, with the bug taking 5-6x longer to manifest (a period of minutes) itself.

Next, purging the xorg-edgers PPA and adding the Backported X / Kernel code from the q-lts-quantal PPA seemed to start off well, but within minutes offered a slightly different error:

"intel_do_flush_locked failed: No space left on device" 

( Which also appeared when we simply tried booting the machine from a 12.10 installer image and testing the issue there. )

Still annoying and quite unusable for day-to-day operations, so we purged that with the ppa-purge utility and re-re-replaced the xorg-edgers PPA but we found Oibaf’s “Updated and Optimized Graphics Drivers” which had an even newer (and arguably, more well tested, if these forums are to be believed) MESA and llvmpipe code, which, was about the last-gasp of a solution, the application in question was, after all a 2D educational program from the late 1990′s, running in a Win95-based WINE instance with no prior tweaking or modifification, so I thought: “*why would it need an updated 3D library?” … but, having found the mailing list and with the PPA page suggesting:

To force the use of gallium llvmpipe software render:
$ LIBGL_ALWAYS_SOFTWARE=1 glxgears -info

I installed the libgl1-mesa-dri-experimental package, then ran the obligitory “sudo apt-get update && apt-get dist-upgrade which pulled in MESA and llvmpipe and a bunch of other updates, rebooted and ran:

$ LIBGL_ALWAYS_SOFTWARE=1 wine /path/to/executable

… and up came the application, which happily wombled along for three hours while we drank scotch and celebrated.

In the end, the best working combination, for those having issues with WINE applications in Ubuntu 12.04 or 12.10 with these hangcheck bugs, was to use the software renderer available from the Oibaf PPA repository and turning OFF the rc6 power-saving fix in Ubuntu, in combination with the code in the xorg-edgers PPA (as of 2012-11-28).

So, the rub (if your Google-fu takes you here directly) to get this to work (on Ubuntu 12.04 and 12.10):

# add-apt-repository ppa:ubuntu-x-swat/x-updates
# add-apt-repository ppa:oibaf/graphics-drivers
# apt-get update && apt-get dist-upgrade
# sed -i -e 's/"quiet splash"/"quiet splash i915.i915_enable_rc6=0"/g' /etc/default/grub
# update-grub

( of course, add a “cp /etc/default/grub /etc/default/grub.backup if you’d like to backup your GRUB configuration :) )

… reboot, and then try your application – Milage May Vary(tm) but it works really well on older i915 hardware here.

If of course, you’d like to revert this, ppa-purge the two repositories and restore your GRUB configuration and you’re back to scratch.

Not Unifed In 12.10: The Quickening

Before I begin: if you would just like to remove “as much of Unity as possible” without reading my stance on Unity, or you’d like to see where this post began — please skip to the end :)

Personally, I like the basic idea of Ubuntu, I like the fact it mostly works as a Desktop, I like the PPAs and I like the fact there’s continuous improvement between releases.

It’s no secret though, some people, myself included — dislike the Ubuntu Unity interface.

Now, unlike my troubles with 12.04 LTS, Unity’s quirks in 12.10 were restricted to:

  1. Erratic Mouse Cursor Movement — basically, the touchpad was erratic and moving to what seemed like “the hot corner” whenever I removed a finger from the pad. Now, I always tune my touchpad to be slightly higher sensititivity than normal anyway, then using the synclient client to tune the areas of my touchpad — however, while that works in GNOME’s fallback desktop & GNOME 3.x, it now completely trips out Unity.

  2. Graphical Artifacts on-screen — usually behind windows, or when searching for something from the dash, they used to be largely inconsequental, but in Unity 6.x, they’re all over the place and solid colours, making a completely unusable desktop after 3-4 windows have been opened.

  3. Missing / Incomplete / Disappearing Shortcuts — I can click on a shortcut on the dock and it “crashes” into the icon that is above or below it, especially annoying when it crashes into the “Removable Drive” icon, which, combined with problem 1) tends to unmount things erratically, making me want to hurt something.

Problems One & Two turned out to be the easiest to fix, I have no Intel-Only (Embedded / Onboard) machines, so it was a simple case of using the ATi / nVidia binary blobs rather than relying on the discrete graphics that are fired up by default, then Unity’s interface at least played ball.

The Third issue seemed to occur regardless of things being on a binary driver or the F/OSS one.

So, what to do?

I thought, “Once again, I know GNOME Shell works on this setup — i’ll use that instead.”

# add-apt-repository ppa:gnome3-team/gnome3 && apt-get -f update
# apt-get -f install gdm gnome-shell gnome-tweak-tool gnome-session-fallback 

The former installs gnome-shell (and GDM, which is required by GNOME 3.6) & gnome-tweak-tool for customising your setup, the latter installs Ubuntu’s “classic GNOME desktop” — just in case.

Now, I almost always select the GNOME Fallback (No Effects) session the first time I want to strip Unity from an install, it’s a personal preference, so my desktop looked like:

The GNOME Fallback (GNOME 2.x-look) Default Desktop

The default desktop UI if you have selected “GNOME Fallback (No Effects)” from the Session List.

If you know your desktop can handle GNOME 3.x out-of-the-box, feel free to omit “gnome-session-fallback” from the apt-get line above.

note: GDM will ask you which “Default display manager” you wish to use, make sure you select gdm rather than lightdm– otherwise, X will fail to start once you’ve finished stripping out the Unity packages (but more on that later).

The Display Manager Selector

You should select the ‘GDM’ option here, if you wish to install the GNOME 3.x desktop in Ubuntu 12.xx

After you’ve done that, you should reboot your system for the changes to take effect, which will then display GDM at boot time like:

Looks rather old-school(tm) doesn’t it :)

So, having rebooted, selected GNOME as my default setup from the greeter and was ready to use my machine — I thought:

Some six months later, if i’m still never going to use Unity, because it doesn’t actually appear to work, why don’t I get rid of it?

Turns out, it’s a more invasive task than it was in 12.04LTS — there’s numerous examples of ways to do it on the interschnitzel, but other than my previous article on the subject, the majority of them ended up either:

  1. Breaking the Desktop (either by claiming X was broken and things were running in low-resolution mode, or by experiencing crashes in software that was linked to the Unity libraries themselves).

  2. Removing Applications you might actually need (including, for example, Nautilus and GNOME Shell itself, as well as Brasero, Rhythmbox, Totem & other GNOME-y related goodness) because they’ve been linked with Unity libraries at either the packaging level, buildtime, or runtime.

In addition, a number of the sites I found recommend getting rid of the indicator-applets packages — which, breaks the fallback session quite severely too and you might want / need to keep that, so we don’t want to do that either.

Unfortunately, there’s a bunch of new pieces in 12.10, relating the “webapps”, “web based services” and the infamous “Amazon Sales” issue — I mentioned on Twitter that i’m almost certain that Unity is becoming more like a tenticled beast that’ll get harder before it gets easier, but for now, we can still uncouple things to make a perfectly usable desktop for the masses.

Oh, that’s an important distinction — while i’m actually a fan of Ubuntu’s ideas for the future (TV’s, Phones, Tablets, Whatever) for now, i’d just like a DESKTOP that works — and chances are, if you’re reading this, you would too.

The Juice Is Loose (again)

So, without further delay (aside from saying “running these commands, as they are written here, in the order I have them listed, all work for me and have been well tested on my 12.10 installs, but if they hose your desktop, delete your information, or eat your dog — I take no responsibility for any of it” — operator beware and all that.):

Log Out of Unity, ALT-Fx to your favourite TTY, login, sudo to root and run:

# apt-get --yes purge unity 
# apt-get --yes purge unity-asset-pool unity-services 
# apt-get --yes purge unity-lens-* unity-scope-* ubuntu-webapps-*
# apt-get --yes purge unity-common
# apt-get --yes purge libunity-misc4 libunity-core-6* libunity-webapps*
# apt-get --yes purge gir1.2-unity-*
# apt-get --yes purge liboverlay-scrollbar* overlay-scrollbar*
# apt-get --yes purge appmenu-gtk appmenu-gtk3 appmenu-qt
# apt-get --yes purge libufe-xidgetter0
# apt-get --yes purge xul-ext-unity xul-ext-websites-integration
# apt-get --yes purge firefox-globalmenu thunderbird-globalmenu

To be fair, the commands above don’t remove all of Unity and friends, there’s parts you need for various things:

If you remove:

  • unity-greeter — If you have not rebooted after firstly installing your alternative desktop, X will fail to start — claiming you are running in Low Resolution mode and telling you to fix your configuration, troubleshoot or login to a console instead — this would appear to be an issue with the way the Classic (Fallback) GNOME Desktop is configured in Ubuntu, however investigation into this is still on-going.

    update 24-10-2012: if you reboot your system after installing gdm rather than simply logging out (as part of installing your alternative desktop — see above) then purge Unity from your system, your X session will start normally and GNOME will work normally too.

  • libunity9 — Various “core” GNOME applications will be uninstalled, such as Nautilus (File Manager), Brasero (CD Burning) & Rhythmbox (Music Player) & Totem (Movie Player) which you might actually need in order to retain a working desktop.

  • indicator-* — The Classic GNOME Desktop will break — the indicator applets on the top right-hand-side will be removed, but clicking on the locations they should have been will cause your session to crash.

(This list will be updated with any others I find, or people e-mail/tweet me about, but as of 24-10-2012 is correct.)

When you return, no Unity options should be present in your Session options — and once you’ve chosen your new environment of choice, you will be Unity-less and all of the GNOME based applications on your desktop should continue to operate correctly too.

In addition, you will also have cleaned out the Unity / Global Menu related components from Firefox / Thunderbird / LibreOffice, making it leaner, faster and less privacy-concerning than before too.

update 14-11-2012: added overlay-scrollbar packages to the list of packages to purge due to this bug which causes pieces of the default GTK+ theme to be either completely black or to disppear intermittently.

overlay-scrollbar should be fixed, but the bug is labelled as WONTFIX which is disappointing, given it seems to be an Ubuntu-specific issue and nothing at all to do with GTK+

Enjoy, until the hacksaws come out for the 13.04 release :)

Restarting GNOME Shell From the CLI

Earlier this evening, while attempting to use remote desktop to fix a system, I had the shell brick on me — no external mouse (USB), no touchpad and no window controls.

However, with a few booted VMs open, 200 or so browser tabs and several in-use terminals in use, not being able to switch between apps, or close them reliably might have caused me a problem, if, that is, the way we used to fix this kind of thing in the 1980′s didn’t still work in 2013.

So, Control-Alt-Fx’ing and logging into a terminal it is — and making sure my traditional ps axuw | grep gnome-shell returned nothing gnome-shell was running, I did:

export DISPLAY="0:0"

Which, obviously would be different if you have more than one X session running, or are working from a tunnel — but for end-users with a single desktop display, 0:0 will almost certainly work.

Then, it was a simple case of restarting the shell:

gnome-shell --replace

and switching back to my GUI with Control-Alt-F7 — to find a restarted shell complete with window controls — problem solved :)

Case For More Accurate Reporting In Open Source?

Recently, a popular pro-ubuntu website, posted a poll regarding the changes in the GNOME 3.x ‘Files’ Application (or as I like to call it “The Artist Formerly Known As Nautilus”)

The article referenced the amount of cleanups / modifications being made to the code to make it simpler for end-users to navigate — many of which, I will add, having been converting “end-users” to Linux for close to ten years — I agree with wholeheartedly, however, the article stated:

Amongst the features you will find missing in the version of Nautilus (now renamed ‘Files’) are ones you may have used on a daily basis, including:
  • Compact View *removed*
  • ‘Type Ahead Find’ *removed*
  • ‘New file’ templates *removed*
  • Application Menu *removed*
  • ‘Go’ menu *removed*
  • F3 split screen *removed*
  • ‘tree’ view *removed*
  • Bookmark menu items *removed*
  • Backspace shortcut to return to parent folder *removed*

Of those, the thought of the `New file’ templates removal scared me a bit, as I tend to manually add templates to end-user installations for Documents, Spreadsheets, Presentations, Images (PNG), E-Mails & Text Files — at least that way, users who are worried about the differences between Explorer and an Ubuntu or Red Hat installation can at least find their way to create the basic filetypes the same way.

So, I went to check the relevant changelog in git, which is here — and I find the feature is, by far not removed.

Infact, a better term for it would have been “adjusted” — the code has been altered in order to only show the menu if there are templates installed. Even Jon McCann, the developer of said “removal” had used the following description for the patch in question:

Hide the New Document menu if no templates are installed.

Given Ubuntu still don’t ship default Templates with their distribution (there’s an 8 year old bug about it here), I suppose yes, it will mean that menu will disappear from the default Ubuntu configuration — but that’s still not removed not by a long way.

Indeed, there’s even talk about it on various mailing lists, such as this one (on that, the template system is actually quite powerful if you take the few hours to configure it properly and it seems to make a world of difference for new users coming from Windows space, especially when they’re not immediately recognising the icon for a particular application.) — Though, I would hardly call it a “worrying development” that downstream ignored such an obvious request for 8 years, therefore a particular “feature” is effectively rendered “useless”.

Still, the issue i’m getting at here is not the feature itself, it’s the fact a widely read Ubuntu fansite can use the word “Removed” when the feature was “Modified” — especially when that particular article will be indexed by many search engines as a popular article, so it’s highly likely “mainstream media” may find it a lot easier than say this, or Jon McCann’s article on the most recent developments to Files.

I wonder (and it’s the reason I sit here at 3am local time writing this up, having had this thought for a few weeks now): Is this kind of reporting behaviour actually hurting GNOME’s image going forward, especially as we move into the newer development realm of 3.x — I mean, GNOME 3.x is a lot different to anything we’ve seen before from a shell point of view, but in terms of the underlying applications, the evolution is really no different to the changes between GNOME 0.x and 2.x — or Windows 3.1 and Windows 8 — a case of trying to make it easier for the people using the software, while embracing and supporting new technologies.

Now my thought turns to “how best to do that?

Ubuntu 12.04 LTS v. the Acer Aspire 57xx Optimus Technology

The first two candidates for upgrading to Ubuntu 12.04 were my Acer Aspire Laptops, both previously running a quite heavily customised version of 10.04.4, with numerous PPAs (ubuntu-audio-dev and ubuntu-x-swat being the main ones relating to hardware).

People often ask me at trade shows & conferences, “Why do you use 10.04.x, why not 11.10, or [insert-release-here-of-something-else]” — and usually got the response “because I need them to work”, which, is not a criticism of the quality of more recent releases, more a “that’s what LTS’s are for.” kind of response.

So, 12.04 being the +1 for LTS’s, I decided to upgrade — i’d been itching to try out GNOME 3.4 on more modern hardware — and the improvements to Unity (Ubuntu’s own Desktop Environment) sounded quite promising and upbeat.

Which wasn’t as difficult as it previous had been, i’m happy to point out — except, both my 5750G and 5742G have nVidia “Optimus” technology powering their graphics cards, which meant I was presented with the following screen following my install:

It's a blank screen, really.

Yes, it's a black screen -- in real life, it has a flashing cursor :)

So, an ALT-F’x’ — a login, a sudo and a couple of add-apt-repository commands later and:

The Standard Desktop after the Bumblebee code had been added to Precise

Um, that looks a lot better -- to change the resolution, optirun & nvidia-settings work a treat.

and, to make sure the nVidia GPU actually does something:

#  optirun nvidia-settings -c :8
Terminal Window running optirun on :8 Console

Using the nvidia-settings via Optirun allows you to alter Power Management and Display Size settings

So, the magic to make this work, is:

# add-apt-repository ppa:bumblebee/stable
# apt-get -f install bumblebee bumblebee-nvidia

Now, if you’re using the standard Ubuntu packages, you should be able to restart your machine and you’ll be back up and running — however, my 5750g and Precise’s nVidia packages didn’t play nicely — so i’d upgraded to the nvidia-current-updates package and rebooted, then everything ran according to plan.

For some reason, maybe because there was some 200 updates to do on the 5742, the post-installation of the package didn’t add my user to the bumblebee group, so I had to then do:

# usermod -a -G bumblebee *paul*

( where paul obviously is whatever your username is :) )

… then logged-out/logged-in, then the optirun based commands worked perfectly.

Not Unified: Removing Unity from Ubuntu 12.04 LTS

Before I begin: if you would just like to remove “as much of Unity as possible” without reading my personal upgrade woes — please skip to the end :)

So, i’ve almost succeeded in upgrading most of my personal boxes from Ubuntu 11.10 and 10.04.4 to the new Ubuntu release.

Oh, by almost and upgrading — I mean “back up to external drive, re-format, re-install, set up new configuration via a script, then copying everything back from the external drive” — but that’s just me, having used Linux in one way or another since 1993 and having seen ‘upgrades‘ fail in a wide variety of funny (or not so funny) ways.

I should point out, I tried upgrading this time around, one server and one desktop (from 9 machines and several VMs on those machines) — just to see what would happen & well, let’s just say Andrew covered things a lot better than I did.

Now, unlike Andrew — who seems to enjoy building things from bottom-up, I prefer to install whatever the distribution gives me then tweak it to do what I need to do in order to get things to work, which meant I was presented with a brand new Unity desktop.

Well, I wasn’t at first — that’s actually covered in another article — as the two machines I did first both have hybrid graphics systems by nVidia referred to as “Optimus”, which has one nVidia GPU and one discrete Intel one — so what I actually got, was, um, not much actually — but i’ll save you the read here and move on with the purpose of this post …

Skipping forward 9 or so hours:

I didn’t like what I saw.

To be fair, i’m not a one-desktop-to-rule-them-all purist either — as a firm believer of “just use the best thing for the job” — i’ve used, developed, packaged, advocated and supported desktops all over in my F/OSS life, CDE, KDE (1, 3), GNOME (0.x -> 3.0), Openbox, GNUStep, I used to tweak FVWM & TWM in the real early days — and before TWM, it was OS/2 and doing cool stuff with it.

But I digress…

I tried using it, for three whole days I tried to use it as my default desktop, which — comprises of approximately a VM, 12 or so Terminal windows, a web browser and a music player — plus or minus a jabber client.

I added “lenses“, had crashes.

I removed items, then added new ones to the dock, had crashes.

I fired up the first set of Terminals I usually used, then used the Window switcher to move to another virtual desktop, had crashes.

I tried Firefox and couldn’t see a global menu as suggested, instead, I saw the outside of the menu (aka. a 1px border) with speckled contents — then found errors in dmesg about DRM & hangcheck errors from the Intel based GPU in the machine.

Slept on it:

Tried using LibreOffice to write up a document and examine a spreadsheet and found everything too cramped — then used the interschnitzel to discover LibreOffice doesn’t actually support the global menu out of the box … did the requisite:

# apt-get install lo-menubar

and tried again, only to discover the global menu had wandered off like it had in Firefox and the messages in dmesg had returned.

Tried Ubuntu-2D, which didn’t start — it instead popped up a box that said my X configuration was broken.

"Broken X Configuration"

"This is what it sounds like, when Paul cries." -- but in all seriousness, what does an "everyday Ubuntu user" do when presented with this?

Needless to say, I clicked the Reconfigure Graphics option, which presented me with a black screen (ie. X crashed back to TTY1) — a reboot later, I tried the Run in low-graphics mode for just one session which, also presented me with a black, blank screen, so, being somewhat used to X crashes because of broken setups in days gone by, I went looking for my X configuration — then any reference to xorg.conf — then any help on the subject — then proceeded to swear, quite a bit.

… moving along …

I thought, “i’m over this, I wonder if GNOME Shell works on this setup?

# add-apt-repository ppa:gnome3-team/gnome3 && apt-get -f update
# apt-get -f install gnome-shell gnome-tweak-tool gnome-session-fallback 

The former installs gnome-shell & gnome-tweak-tool for customising your setup, the latter installs Ubuntu’s “classic GNOME desktop” — just in case.

Minutes later, I was in GNOME 3.4, in my browser, answering my mail.

Moments later, hangcheck bit me again, forcing a reboot, but as I said earlier, in my case, that’s a different issue.

So, having rebooted, selected GNOME as my default setup from the greeter and was ready to use my machine — I thought:

if i’m never going to use Unity, because it doesn’t actually appear to work, why don’t I get rid of it?

Turns out, that’s a little harder than it looks — there’s numerous examples of ways to do it on the interschnitzel, but all of them ended up either:

  1. Breaking the Desktop (either by claiming X was broken and things were running in low-resolution mode, or by experiencing crashes in software that was linked to the Unity libraries themselves).

  2. Removing Applications you might actually need (including, for example, Nautilus and GNOME Shell itself, as well as Brasero, Rhythmbox, Totem & other GNOME-y related goodness) because they’ve been linked with Unity libraries at either the packaging level, buildtime, or runtime.

(yes, 2 == 1, when you think about it).

In addition, a number of the sites I found recommend getting rid of the indicator-applets packages — which, breaks the fallback session quite severely too, so we don’t want to do that either.

The Juice Is Loose

So, without further delay (aside from saying “running these commands, as they are written here, in the order I have them listed, all work for me and have been well tested on my 12.04 LTS installs, but if they hose your desktop, delete your information, or eat your dog — I take no responsibility for any of it” — operator beware and all that.):

Log Out of Unity, ALT-Fx to your favourite TTY, login, sudo to root and run:

# apt-get --yes purge unity unity-2d unity-2d-places unity-2d-panel unity-2d-spread 
# apt-get --yes purge unity-asset-pool unity-services unity-lens-* unity-scope-*
# apt-get --yes purge liboverlay-scrollbar*
# apt-get --yes purge appmenu-gtk appmenu-gtk3 appmenu-qt
# apt-get --yes purge firefox-globalmenu thunderbird-globalmenu
# apt-get --yes purge unity-2d-common unity-common
# apt-get --yes purge libunity-misc4 libunity-core-5*

To be fair, the commands above don’t remove all of Unity and friends, there’s parts you need for various things:

If you remove:

  • unity-greeter — X will fail to start, claiming you are running in Low Resolution mode and telling you to fix your configuration, troubleshoot or login to a console instead — I thought this might be because of the Optimus setup on my primary machine, but it can be re-produced in VirtualBox and on another physical desktop where 3D actually works — investigation into this is on-going.

  • indicator-* — The Classic GNOME Desktop will break — the indicator applets on the top right-hand-side will be removed, but clicking on the locations they should have been will cause your session to crash.

(This list will be updated with any others I find, or people e-mail/tweet me about, but as of 21-05-2012 is correct.)

When you return, no Unity options should be present in your Session options — and once you’ve chosen your new environment of choice, you will be Unity-less and all of the GNOME based applications on your desktop should continue to operate correctly too.

Evolution, Databases, Grief.

Recently, Evolution on my Ubuntu Oneiric Desktop popped up with a dialogue stating:

Database Disk Image Is Malformed

Which caused it to not index anything in any of the folders I had listed in my IMAP setup — restarting, using evolution –force-shutdown and various other solutions found on the interschnitzel had no effect, however — a slightly modified version of this page worked a treat.

Slightly modified, as Evolution 3.x and beyond on Ubuntu use ~/.local/share/evolution/mail for their mail storage — so the correct sequence of events to fix this problem became:

sudo apt-get -f install sqlite3

Then:

cd ~/.local/share/evolution/mail
for i in `find . -name folders.db`; do 
echo "Rebuilding Table $i"; 
sqlite3 $i "pragma integrity_check;"; 
done

Which turned:

Rebuilding Table ./imap/paul@recovered-mail/folders.db

*** in database main ***

On tree page 11 cell 0: 2nd reference to page 173

On tree page 11 cell 1: 2nd reference to page 174

On tree page 11 cell 2: 2nd reference to page 450

On tree page 11 cell 3: 2nd reference to page 711

On tree page 11 cell 4: 2nd reference to page 924

On tree page 1060 cell 0: 2nd reference to page 805

On tree page 1060 cell 1: 2nd reference to page 849

On tree page 1060 cell 2: 2nd reference to page 921

On tree page 1060 cell 3: 2nd reference to page 851

On tree page 1060 cell 4: 2nd reference to page 911

On tree page 1060 cell 5: 2nd reference to page 850

On tree page 1060 cell 6: 2nd reference to page 848

Page 1067: btreeInitPage() returns error code 7

Page 1069: btreeInitPage() returns error code 11

Error: database disk image is malformed

Into:

Rebuilding Table ./imap/paul@recovered-mail/folders.db

ok

Of course, one needs to make sure the databases aren’t being used at the time — and, at least under Oneiric, evolution –force-shutdown tends to be a bit strange, so you might need to manually kill processes such as the evolution-alarm-notifier before starting this process.

Upstream, Downstream and … What is it exactly?

Talking with Peter the other evening about kernel development teams (if you’ve been following along here throughout October, you’ll see that’s been the bulk of my month.) — we wondered:

“What is it called when you’re doing your [kernel] development outside of any sanctioned tree, but other developers with the same vein/idea are *also* taking ideas / code from your tree?”

In the Linux world, that’s not mainline — because that’s Linus’s domain — and, in the Google Android case, Android != Mainline, so it’s not Android / Google either.

Indeed, it’s not “upstream” either — as we’ve seen, Google often does not have changes that are done in external trees — there’s a reason there’s a Qualcomm tree exists that specific vendors pull chipset changes from, for example.

It’s not “downstream” in that same vein — as individual products commonly have either differing hardware, or differing versions of the same code on a device-by-device basis.

So, the two of us got onto something else — and I suddenly thought:

Sidestream

and Peter & I mulled it over for a few sentences and thought, yes, that’s more like it — after all, Sidestream infers:

Development done in parallel with versions upstream (Android Versions, in this case), but not included in upstream, not cherry-picked by upstream.

but also

Development used by downstream (Mobile Vendors, in this case) to provide updates and fixes to individual products — but changes to those files (by the vendors) are not necessarily sent back to either upstream or sidestream repositories.

Taking the Qualcomm example, changes are taken from there and cherry-picked into the ARM “mainline”, but new developments are used and tested there for a suitable amount of time before this happens.

Depending on when and if new Android kernel releases are frozen, the “upstream” code may not include these changes (for obvious reasons).

Vendors who require fixes for the frozen drivers in their “upstream” code, can then cherry-pick or take verbatim changes from the Qualcomm “sidestream” tree when required.

Thoughts? Could it catch on as a new buzzword for external kernel development? ;)