Planet Debian

Mini-sendmail... in bash

2012-02-13T22:19:22+00:00

I recently faced an environment where there is no MTA.

WTF? The reason is that people who work there get security audits on a regular basis, and the security people are usually mo...deratly skilled guys that blindly run a set of scripts, e.g. by ordering to disable Apache modules that "where seen enabled in /etc/apache2/mods-available/"...

To avoid spending days arguing with them and nitpicking with non-technical managers, the system is trimmed to the minimum - and there no MTA. No MTA, so no cron output, so difficulty to understand why last night's cron job failed miserably.

Since it was not my role to reshape the whole business unit, I decided to hack a super-light, but functional way to get my cron output:

cat <<'EOF' > /usr/sbin/sendmail
#!/bin/bash
(
    echo "From me  $(LANG=C date)"
    cat
    echo
) >> /var/mail/all
chmod 755 /usr/sbin/sendmail

It works!

There is a companion logrotate script, to avoid filling the file system:

cat <<'EOF' > /etc/logrotate.d/mail-all
/var/mail/all {
  daily
  rotate 10
  compress
  delaycompress
  notifempty
  create 722 root mail
}
EOF

Bootstrap with:

touch /var/mail/all
logrotate -f /var/mail/all

You now can check your sys-mails with:

mutt -f /var/mail/all

Quasi-Private Resources

2012-02-13T17:37:32+00:00

Public Resource republishes many court documents. Although these documents are all part of the public record and PR will not take them down because someone finds their publication uncomfortable, PR will evaluate and honor some requests to remove documents from search engine results. Public Resources does so using a robots.txt file or "robot exclusion protocol" which websites use to, among other things, tell search engine's web crawling "robots" which pages they do not want to be indexed and included in search results. Originally, the files were mostly used to keep robots from abusing server resources by walking through infinite lists of automatically generated pages or to block search engines from including user-contributed content that might include spam.

The result for Public Resource, however, is that PR is now publishing, in the form of its robots.txt, a list of all of the cases that people have successfully requested to be made less visible!

In Public Resource's case, this is is the result of a careful decision; PR makes the arrangement clear in on their website. The robots.txt home page also explains the situation saying, "the /robots.txt file is a publicly available file. Anyone can see what sections of your server you don't want robots to use,", and "don't try to use /robots.txt to hide information."

That said, I've looked at a bunch of robots.txt files on websites I have visited recently and, sadly, I've found many sites that use robots.txt as a form of weak security. This is very dangerous.

Some poorly designed robots simply ignore the robots.txt file. But one can also imagine an evil search engine that uses a web-crawler that does the opposite of what it's told and only indexes these "hidden" pages. This evil crawler might look for particular keywords or use existing search engine data to check for incoming links in order to construct a list of pages whose existence is only made public through a file meant to keep people away.

Check your own robots.txt and ask yourself what it might reveal. By advertising the existence and locations of your secrets, the act of "hiding" might make your data even less private.

eSpeak GUI 0.4 is out!

2012-02-13T15:29:56+00:00

A few of you may remember espeak-gui, a graphical user interface for eSpeak I did release two years ago. While I haven’t been dedicating much time to it, it’s still alive and I’m happy to announce its fourth release!

This new release introduced automatic language guessing, a feature I’ve wanted for a while, as well as spell checking (contributed by Joe Burmeister).

Version 0.3, released a year ago and which I didn’t announce here, did also introduce Undo/Redo, pausing, file history logging, a bunch of translations and some voice options. It’s sister project, python-espeak, did also get some bug fixes since its first release.

As usual, packages for both espeak-gui and python-espeak are available in my PPA, and since last month in Debian unstable. Make sure you also install libtextcat0 (version 2.2-8 or newer) to get the language identification support.

espeak-gui 0.2 Yesterday I did a first release of espeak-gui, so while...
Introducing espeak-gui I’m joining the hype of presenting little new projects there...
Here is Zeitgeist 0.2.1! One month after the first Zeitgeist release (0.2), here is...

phpMyAdmin translations status

2012-02-13T11:00:00+00:00

As phpMyAdmin is approaching to 3.5 release, it has come time to share about translations status. Here comes third round of translation status update.

Since last update we have one new translation at 100%, but one has dropped from the list:

Czech
English (Great Britain)
Russian
Slovenian
Spanish
Swedish
Turkish

There are few which are really close to 100% and I hope they will get there soon:

French
Danish
Japanese
Polish

As you can see, traveling around FOSDEM was really useful for Czech translation, which I've managed to complete during that weekend (+ Monday when I traveled to Nuremberg).

There was great amount of work done on Polish translation, so hopefully it will be on 100% in next report. Also Rouslan is progressing quite well on Italian, but there is still fair amount of work to be done.

Was your language not mentioned? It's now perfect time to contribute to it.

Filed under: English Phpmyadmin | 0 comments | Flattr this!

Debian Wheezy: US$19 Billion. Your price… FREE!

2012-02-13T06:27:50+00:00

As many would know, Debian GNU/Linux is one of the oldest, and the largest Linux distributions that is available for free. Since it was first released in 1993, several people have analysed the size and produced cost estimates for the project.

In 2001, Jesús M. González-Barahona et al produced an article entitled “Counting Potatoes“, an analysis of Debian 2.2 (code named Potato). When Potato was released in June 2003, it contained 2,800 source packages of software, totalling around 55 million lines of source code. When using David A. Wheeler’s sloccount tool to apply the COCOMO model of development, and an average developer salary of US$56,000, the projected development cost that González-Barahona calculated to start-from-scratch and build Debian 2.2 in 2003 was US$1.9 billion.

In 2007 an analysis entitled ‘Macro-level software evolution: a case study of a large software compilation‘ by Jesús M. González-Barahona, Gregorio Robles, Martin Michlmayr, Juan José Amor and Daniel M. German was released. It found that Debian 4.0 (codename Etch released April 2007) had just over 10,000 source packages of software and 288 million lines of source code. This analysis also delved into the dependencies of software packages, and the update flow between Debian release (not all packages are updated with each release).

Today (February 2012) the current development version of Debian, codenamed Wheezy, contains some 17,141 source packages of software, but as it’s still in development this number may change over the coming months.

I analysied the source code in Wheezy, looking at the content from the “original” software that Debian distributes from its upstream authors without including the additional patches that Debian Developers apply to this software, or the package management scripts (used to install, configure and de-install packages). One might argue that these patches and configuration scripts are the added value of Debian, however the in my analysis I only examined the ‘pristine’ upstream source code.

By using David A Wheeler’s sloccount tool and average wage of a developer of US$72,533 (using median estimates from Salary.com and PayScale.com for 2011) I summed the individual results to find a total of 419,776,604 source lines of code for the ‘pristine’ upstream sources, in 31 programming languages — including 429 lines of Cobol and 1933 lines of Modula3!

In my analysis the projected cost of producing Debian Wheezy in February 2012 is US$19,070,177,727 (AU$17.7B, EUR€14.4B, GBP£12.11B), making each package’s upstream source code wrth an average of US$1,112,547.56 (AU$837K) to produce. Impressively, this is all free (of cost).

Zooming in on the Linux “Kernel”

In 2004 David A. Wheeler did a cost analysis of the Linux Kernel project by itself. He found 4,000,000 source lines of code (SLOC), and a projected cost between US$175M and US$611M depending on the complexity rating of the software. Within my analysis above, I used the ‘standard’ (default) complexity with the adjusted salary for 2011 (US$72K), and deducted that Kernel version 3.1.8 with almost 10,000,000 lines of source code would be worth US$540M at standard complexity, or US$1,877M when rated as ‘complex’.

Another Kernel Costing in 2011 put this figure at US$3 billion, so perhaps there’s some more variance in here to play with.

Individual Projects

Other highlights by project included:

Project	Version	Thousands of SLOC	Projected cost at US$72,533/developer/year
Samba	3.6.1	2,000	US$101 (AU$93M)
Apache	2.2.9	693	US$33.5M (AU$31M)
MySQL	5.5.17	1,200	US$64.2M (AU$59.7M)
Perl	5.14.2	669	US$32.3M (AU$30M)
PHP	5.3.9	693	US$33.5M (AU$31.1M)
Bind	9.7.3	319	US$14.8M (AU$13.8M)
Moodle	1.9.9	396	US$18.6M (AU$17.3M)
Dasher	4.11	109	US$4.8M (AU$4.4M)
DVSwitch	0.8.3.6	6	US$250K (AU$232K)

Debian Wheezy by Programming Language

The upstream code that Debian distributes is written in many different languages. ANSI C with 168,536,758 is the dominant language (40% of all lines), followed by C++ at 83,187,329 (20%) and Java with 34,698,990 (8%).

Break down of Wheezy by Language

If you are intersted in finding the line count and cost projections for any of the 17,000+ projects, you will find them in the raw data CSV.

Other Tools and Comparisons

Ohcount is another source code cost analysis tool. In March 2011 Ohcount was run across Debian Sid: its results are here. In comparison, its results appear much lower than the sloccount tool. There’s also the Ohloh.net Debian Estimate which only finds 55 Million source lines of code and a projected cost of US$1B. However Ohloh uses Ohcount for its estimates, and seems to be to be around 370 million SLOC missing compared to my recent analysis.

Summary

Over the last 10 years the cost to develop Debian has increased ten-fold. It’s intersting to know that US$19 billion of software is available to use, review, extend, and share, for the bargain price of $0. If we were to add in Debian patches and install scripts then this projected figure would increase. If only more organisations would realise the potential they have before them.

Need help with Linux (including Debian), Perl, or AWS? See www.jamesbromberger.com.

Mexico City Metro project

2012-02-13T05:13:39+00:00

Some days ago, reading my local Couchsurfing groups, I stumbled across an announcement by Australian Peter Davies to go to each of the 148 stations in the Mexico City Metro system, take some photos of the environment, and document on his impressions.

I have followed and enjoyed the Mexico City Metro blog since I learnt of it, and have grown used to looking forward to the daily post-or-two. Peter writes each of his entries both in English and Spanish (you can tell it's not a native Spanish, but it's a good effort). He has been doing the stations in a very well distributed order (I cannot say it's completely random, but it's surely not lineal or methodical).

I connected wiht his project as I love discovering the city more or less the same way, but with a different system: I try to have at least one long bike ride every two weeks (being "long" something over 40Km). I usually go either to the North or to downtown and to the East by the good old route I always take, and on my way back, at some point I decide just to turn right or left and discover yet another village slurped by the city. I don't usually take pictures, as I'm too much into the cycling thrill, left-right-left-right... But cycling has led me to appropriate my city (I don't know if that's proper English), to make my city really mine, to get to know parts of it I'd never otherwise go to.

Anyway, Peter's is a great way to document urban life. I'm in love with my city, and with expressions of urban appropriation. I loved his project, and if you are interested by what I say, go take a look at his wanderings in the city. I have suggested him two bits to check, but the work is very much an artist's — He accepts my input, but quite probably he will do whatever he pleases ;-) In case any of you is interested in contacting him, I can tell you for a fact he replies :-)

[*] And what is CouchSurfing? Oh, a great community where you can offer a space to crash at your house for unknown people from all around the world. I have never requested a couch, as the Free Software community is much more tightly knit, but I have offered it to several interesting people.

Cooling a Thinkpad

2012-02-13T04:14:06+00:00

Late last year I wrote about the way that modern laptops suck [1]. One of the problems that inspired that post was the excessive heat generated by my Thinkpad T61.

There is a partial solution to this, Fool Control explains how the kernel option pcie_aspm=force can be used on kernels from 2.6.38 onwards to solve a heat regression problem [2]. I applied this to my Thinkpad T61 and the result was that on a cool evening (ambient temperature about 24C) the temperature changed from 85C to 66C on the NVidia video card, and for the “virtual devices” it changed from 80C and 78C to 60C and 61C. I’m not sure exactly what each of those measurements refers to, but it seems that the change was somewhere between 17C and 20C.

This changes the system from being almost unbearable to use to being merely annoyingly warm.

I’m not going to make my laptop be my primary computing device again though, the combination of a desktop system with a 27″ monitor and an Android phone is working quite well for me [3]. But I haven’t yet got version control systems working for all my software. Also Wouter suggested using NBD which is something I haven’t got working yet and probably won’t until I can swap on it and therefore have a diskless workstation. Finally I still haven’t got the “Chrome to Phone” browser extension working such that a page I’m viewing at home can be loaded on my phone.

Taking my Thinkpad Apart and Cooling Problems I’ve been having some cooling problems with my Thinkpad recently....
thinkpad back from repair On Tuesday my Thinkpad was taken for service to fix...
I Just Bought a new Thinkpad and the Lenovo Web Site Sucks I’ve just bought a Thinkpad T61 at auction for $AU796....

Dear fellows of the SJCARS

2012-02-13T01:16:08+00:00

I’ve got a project I could use some help with. If you’ve got some tuits to spare, I’d love to talk with you about the plan. Perhaps I’ll post something to the blog…

[edit 20120212T2025]
blogged. http://sjcars.org/blog/2012/02/12/hardware-what-wants-deploying

[Edit 20120213T0026]

Branching FreeGLUT SVN with Git

2012-02-12T22:43:13+00:00

I'm working on porting FreeGLUT to Android!

Since I don't have access so the repository (I'm not a regular contributor), and since it will be a BIG patch, I thought I'd try a git-svn branch. So this won't be a post on OpenGL for once

I'm not an expert at managing Git branch but here's how I set it up.

In this case, let's forget about importing tags: they don't have the same directory branch point in the hierarchy (sometimes the freeglut/ module, sometimes the parent directory), and git-svn only correctly supports set of sane tags that track the same initial directory.

SVN is SLOW, especially remotely. I suggest you make a rsync copy of the repository first, it will greatly speed up the import :

rsync -avHSP freeglut.svn.sourceforge.net::svn/freeglut/ freeglut.svn/

Then start the import with git-svn :

git svn clone --trunk=trunk/freeglut/freeglut file://$(pwd)/freeglut.svn/ freeglut-git/

And edit .git/config to point to https://freeglut.svn.sourceforge.net/svnroot/freeglut :

[svn-remote "svn"]
    url = https://freeglut.svn.sourceforge.net/svnroot/freeglut
    fetch = trunk/freeglut/freeglut:refs/remotes/trunk

I work with 2 branches:

master : the SVN on-going sync - may be rebased to push with git svn dcommit
android : my feature branch - not rebased and hence trackable by contributors

Here's how I synchronize the upstream repository and merge it regularly with my feature branch:

git svn fetch
git checkout master
git merge remotes/trunk

git checkout android  # my feature branch
git merge master
# if conflict
git add .../sthing.c
git commit -m "Merge master"
git push

Any better way?

The ongoing work is available at https://gitorious.org/freeglut/android

Playing with the Trim-Slice

2012-02-12T20:43:00+00:00

There seems to be a lot of buzz around the Raspberry Pi these days, and I can sort of understand it—a cheap, pretty hackable machine with HDMI out and 1080p video playback for $25. (Somehow I doubt BBC BASIC is what makes people enthusiastic.)

What I find especially interesting about it is that together with Windows on ARM, we are starting to get into a situation where desktop computers again run on multiple architectures; x86 remains dominant, but this time, ARM is what makes its entrance on the desktop/HTPC stage. (Unfortunately, Windows on ARM is shaping up to be as locked as the iPad, but that's a different story.)

The device hasn't started shipping yet, so it's hard to know what to expect, but the last few days, I've had access to a Trim-Slice for reasons of some ARM optimization, so I've also spent the opportunity to evaluate it briefly as a replacement for my aging first-generation ION system (single core Atom); although it's not the same device, it's probably indicative of what you can expect from the Pi. It's by no means a thorough test, so take it more a personal opinion.

(Why do I want to replace my ION? There are two simple reasons: First, if I can have something that takes 5 W instead of 40 W or whatever the ION takes, I can go fanless, and the the CPU fan is actually a bit annoynig, especially with the ball bearings having problems. The second is that I'd love some more CPU power when compiling things and so on. Having a faster GPU wouldn't really hurt either.)

The Trim-Slice is more intended as end-user machine than the Raspberry Pi is; for one, it comes in a pretty sleek case, has more storage, and costs ten times as much. However, CompuLab (the company that manufactures the Trim-Slice) is pretty clear that it's not ready for anyone but developers yet, and this is my summary too: It's too young, but it's not the CPU's fault. The difference is, though, whether I think the current model will ever be ready.

Let's start with the CPU: The Trim-Slice comes with a dual-core Cortex-A9 clocked at 1GHz. ARM has this confusing thing about naming (like, ARM11 is based on the ARMv6 architecture), but the general summary is that it's pretty much the fastest ARM you can get out there in an end-user package right now, but you can get quad-core versions. (Cortex-A15 systems will be arriving in 6–12 months or so, I've been told.) The Cortex-A9 has an FPU, but unfortunately the model included does not have NEON (roughly ARM's equivalent to Intel's SSE). Software availability is generally not an issue at all unless you desperately need Flash; the machine ships with an ARM port of Ubuntu, and since volunteers have spent thousands of hours throughout the history of Debian/Ubuntu to make things cross-compile, the experience of using one is pretty much identical to using an x86.

The Cortex-A9 is an out-of-order-core, compared to the in-order Atom. It appears to be at least roughly clock-for-clock as fast as the Atom, maybe faster (but the Atom has more memory bandwidth, being clocked higher). It certainly does not feel sluggish in normal daily use. (Also, it has 1 GB of RAM, which is not half-bad for a low-end system like this.) Note that the Raspberry Pi will have a much weaker CPU, a single ARM11 at 700 Mhz.

What does feel sluggish, however, is the graphics. The Trim-Slice is based around the Tegra2 design, which is one of nVidia's standard tablet designs, and it's really sized for driving a 720p screen, tops, not 1080p. The drivers are not the same as nVidia's usual Windows drivers; even out-of-the-box, writing stuff in an xterm gives you weak display corruption, menus in GNOME need time to open, etc.. Also, the fill rate of the GPU is something like 1/10th of even the quite low-end GPU in my ION box, which becomes a problem when things like XBMC likes to draw its menu graphics using flashy shaders.

The GPU supports GLES, not regular OpenGL, but XBMC supports that; it doesn't support the GLES texture compression format, though, which probably doesn't help the fill rate. The drivers are sort of ad-hoc, and GLES development doesn't really work (a lot of people have problems with it simply dropping to software mode). CompuLab has promised this will get better real soon (there's a software update coming in a few days), but fundamentally they're really at nVidia's mercy here, and it's unknown to me to what degree nVidia will care about Tegra2 now that Tegra3 (with quad-core Cortex-A9 and a somewhat stronger GPU) is out.

There are a lot of people who think this will be better with Raspberry Pi, but note that the only real source saying that the Pi's GPU will be awesome is a Broadcom exec praising his own product's ability to win synthetic benchmarks, and there's no real indication that it will be the best thing since sliced bread; it's probably faster than the one in Tegra2, but not by a mile.

Another interesting point is the video acceleration; there's no way this thing is going to decode 1080p in software. The Tegra2's chip supports hardware 1080p decode, but it doesn't support the VDPAU/VAAPI APIs that are now customary on Linux—instead, it supports a GLES standard called OpenMAX, which is also relatively common on Android today. (The Tegra2 is pretty much geared as a chipset for Android tablets, so this is not a big surprise.) Unfortunately, OpenMAX is not very standardized with regards to header placement and so on, and there are lots of implementation-specific quirks. Even worse, the Tegra2 can only do H.264 Main Profile, and almost all the HD material you'll find out there uses H.264 High Profile. Oops. This basically renders the Trim-Slice useless as a HTPC.

There are other quirks as well; for instance, I went with the model that has a 32GB SSD from SanDisk (the other alternatives are a 4GB memory card, and a 2,5" SATA drive of your choice), and it's hooked up internally by USB. Unfortunately the included kernel has some USB issues, causing slow transfers and frequent bus resets, which makes I/O in general pretty painful. (This is another thing that will be fixed pretty soon, as I understand it.) Fortunately, if you're willing to cobble together patches from support forums and the likes, it doesn't seem impossible to compile your own kernel; the custom kernel has a git repository (and mainline has basic support for the machine from 2.6.39), and the Trim-Slice wiki has good information on how to cross-compile and install new kernels. (Even though the Cortex-A9 is prett usable, compiling on my 8-way Xeon machine is, strangely enough, a lot more comfortable still :-) ) I needed to do this because, incredibly enough, mutexes were broken, causing XBMC to segfault.

Again, it's one of these things that you need to expect when buying something as young as this, but maybe the Raspberry Pi community will be big enough that most of these kinks will be ironed out. The list of small things goes on: Power saving doesn't appear to be fully implemented yet, so the device gets really hot (even at 5 W max!). It doesn't show anything on boot before X starts, except through the included serial port. The DVI port has problems with high resolutions (the HDMI port works fine, and in recent versions also with audio and DTS passthrough and all, although the ALSA device names confuse XBMC). The Wi-Fi driver is unstable. And so on. All of these are things I can live with, really.

So, the conclusion? Really, if it had more workable graphics support (faster GPU, less problematic display drivers, H.264 High Profile support), I'd buy one as my HTPC; despite its shortcomings, it's a pretty neat thing, and at $325 for the SSD model, it's not all that expensive either. (It's sort of ironic that a device so heavily based on something from nVidia, a GPU company, would have graphics as its weakest point.) But no, I won't buy one, and I probably won't buy a Raspberry Pi—but in maybe a year, there will be Cortex-A15 designs out, and hopefully one with better graphics, and then I'll seriously consider buying one. If I can't find a next-generation fanless ION before that, that is. :-)

Some things you may have heard about Secure Boot which aren't entirely true

2012-02-12T19:55:01+00:00

Talking about Secure Boot again, I'm afraid. One of the things that's made discussion of this difficult is that, while the specification isn't overly complicated, some of the outcomes aren't obvious at all until you spend a long time thinking about it. So here's some clarification on a few points.

Secure Boot provides no additional security

Untrue. Attacks against the pre-boot environment are real and increasingly common - this is a recent proof of concept, and this has been seen in the wild. Once something has got into the MBR, all bets are off. It can modify your bootloader or kernel, inserting its own code to return valid results whenever any kind of malware checker scans for it. The only way to reliably identify it is to either move the disk to another system or to cold boot off verified media. By restricting pre-OS code to something that's been signed, Secure Boot does provide enhanced security.

Secure Boot is just another name for Trusted Boot

Untrue. Trusted Boot requires the ability to measure the entire boot process, which gives the OS the ability to figure out everything that's been run before OS startup. The root of trust is in the hardware and a TPM is required. Secure Boot is simply a way to limit the applications that are run before the OS. Once booted, there is no way for the OS to identify what was previously booted, or even if the system was booted securely.

Microsoft are just requiring that vendors implement the specification

Untrue. Quoting from the Windows 8 Hardware Certification Requirements:

MANDATORY: No in-line mechanism is provided whereby a user can bypass Secure Boot failures and boot anyway Signature verification override during boot when Secure Boot is enabled is not allowed. A physically present user override is not permitted for UEFI images that fail signature verification during boot. If a user wants to boot an image that does not pass signature verification, they must explicitly disable Secure Boot on the target system.

Section 27.7.3.3 of version 2.3.1A of the UEFI spec explicitly permits implementations to provide a physically present user override. Whether this is a good thing or not is obviously open to argument, but the fact remains that Microsoft forbid behaviour that the specification permits.

Secure Boot can be used to implement DRM

Untrue. The argument here is that Secure Boot can be used to restrict the software that a machine can run, and so can limit a system to running code that implements effective copy protection mechanisms. This isn't the case. For that to be true, there would need to be a mechanism for the OS to identify which code had been run in the pre-OS environment. There isn't. The only communication channel between the firmware and the OS is via a single UEFI variable called "SecureBoot", which may be either "1" or "0". Additionally, the firmware may provide a table to the OS containing a list of UEFI executables that failed to authenticate. It is not required to provide any information about the executables that authenticated correctly.

In both these cases, the OS is required to trust the firmware. If the firmware has been compromised in any way (such as the user turning off Secure Boot), the data provided by the firmware can be trivially modified and the OS told that everything is fine. As long as machines exist where users are permitted to disable Secure Boot, Secure Boot is not any kind of DRM scheme.

Secure Boot provides physical security

Untrue. Secure Boot does not in any way claim to improve security against attackers who have physical access, for the same reasons as the DRM case. The OS has no way to determine that the firmware's behaviour has been modified. A physically-present attacker can simply disable Secure Boot and install a piece of malware that lies to the OS about platform security. The "Evil Maid" attack still exists.

Secure Boot only defines the interaction between the firmware and the bootloader. It doesn't specify any higher policy

Misleading. It's true that Secure Boot only specifies the authentication of pre-OS code. However, if you implement Secure Boot it's because you want to ensure that only authenticated code has run before your OS. If there is any way for unauthenticated code to touch the hardware before your OS starts, you can't ensure that. If an authenticated Linux kernel is booted and then loads an unsigned driver, that unsigned driver can fabricate a fake UEFI environment and then launch Windows from it. Windows would falsely believe that it has booted securely. If that authenticated Linux kernel were widely distributed, attackers could use it as an attack vector for Windows. The logical response from Microsoft would be to blacklist that kernel.

The inevitable outcome of implementing Secure Boot is that every component that can touch hardware must be signed. Anyone who implements Secure Boot without requiring that is adding no additional security whatsoever.

Only machines that want to boot Windows need to carry Microsoft's keys

Again, misleading. Microsoft only require one signing key to be installed, and the Windows bootloader will be signed with a key that chains back to this one. However, the bootloader is not the only component that must be signed. Any drivers that are carried on ROMs on plug-in cards must also be signed. One approach here would have been for all hardware vendors to have their own keys. This would have been unscalable - any shipped machine would have to carry keys for every vendor who produces PCI cards. If a machine carried an nvidia key but not an AMD one, swapping a geforce for a radeon would have resulted in the firmware graphics driver failing to load. Instead, Microsoft are providing a signing service. Vendors will be able to sign up for WHQL membership and have their UEFI drivers signed by Microsoft.

This leads to the problem. The Authenticode format used for signing UEFI objects only allows for a single signature. If a driver is signed by Microsoft, it can't be signed by anybody else. Therefore, if a system vendor wants to support off-the-shelf PCI devices with Microsoft-signed drivers, the system must carry Microsoft's key. If the same key is used as the root of trust for the driver signing and for the bootloader signing, that also means that the system will boot Windows.

This is only a problem for clients, not servers

Not strictly true. While Microsoft's current requirements don't mandate the presence of Secure Boot on server hardware, if present it must be enabled and locked down as it is for clients. It's not valid for servers to ship with disabled Secure Boot support, or for it to be shipped in a mode allowing users to make automated policy modifications at OS install time.

Secure Boot is required by NIST

This is one that I've heard from multiple people working on Secure Boot. It's amazingly untrue. The document they're referring to is NIST SP800-147, which is a document that describes guidelines for firmware security - that is, what has to be done to ensure that the firmware itself is secure. This includes making sure that the OS can't overwrite the firmware and that firmware updates must be signed. It says absolutely nothing about the firmware only running signed software. Secure Boot depends upon the firmware being trusted, so these guidelines are effectively a required part of Secure Boot. But Secure Boot is not within the scope of SP800-147 at all.

It's easy for Linux to implement Secure Boot

Misleading. From a technical perspective, sure. From a practical perspective, not at all. I wrote about the details here.

It's only a problem for hobbyist Linux, not the real Linux market

Untrue. It's unclear whether even the significant Linux vendors can implement Secure Boot in a way that meets the needs of their customers and still allows them to boot on commodity hardware. A naive implementation removes many of the benefits of Linux for enterprise customers, such as the ability to use local modifications to micro-optimise systems for specific workloads. One of the key selling points of Linux is the ability to make use of local expertise when adapting the product for your needs. Secure Boot makes that more difficult.

Conclusion

Much reporting on the issues surrounding Secure Boot so far has been inaccurate, leading to misunderstandings about the (genuine) benefits and the (genuine) drawbacks. Any useful discussion must be based around an accurate understanding of the specification rather than statements from analysts with no real understanding of the Linux market or security.

comments

I’ve been away from Windows for more than a decade but …

2012-02-12T18:30:16+00:00

… but I’ve already forseen the unwanted meet for some time .

My elder daughter does need to run educational software from school. And my wife wants to run education software for our younger son. And making all that running under different emulation environments does eat my time, and result is not always good. And also children start to want games…

Issue 1. Need hardware to run more or less modern windows on.

For last 6 years, my home network consists of a more or less powerful headless server, and 3 terminals to access that. Terminals are definitely too weak for windows. And server is needed in 24/7 mode so sorry no windows there.

Likely solution for this is – one of 3 terminals is actually dying (it is doing so already for months but still works, just hangs sometimes) and needs a replacement, so I can just replace it now and use more powerful hardware.

Issue 2. Windows host or windows VM?

While windows VM feels preferable for a such a deep-linux person as myself, I tend to think that dual-boot host is better here. If the very reason of allowing windows into this house is – make it easy to install and run random windows software, then doing so in VM environment is explicitly against the goal.

Issue 3. Use unlicensed windows or buy a license?

Although absolute majority of windows users here is Russia use unlicensed windows, it is somewhat unfair to promote free software and use unlicensed windows at the same time. And windows home basic license is not that expensive after all.

The setup that makes me less uncomfortable than others is:
- obtain powerful enough hardware, and install that instead of failing terminal,
- obtain windows 7 home basic license
- install dual boot linux + windows, and under linux install windows VM
- when needed, try windows VM first, and in case of any issues, immediately reboot into windows host.

What I can’t find out for some time already is – does microsoft retail license allow to have both host and VM installation on the same computer? These will be physically different installations, but on the same computer, and will never run at the same time. License wording does not give exact answer on this.

I wanted to ask on Microsoft’s support forum but failed to register there. After asking me quite a few things it claimed that there is some temporary problem and suggested to retry later. I retried later and got the same.

I then asked here but looks like nobody knows.

At least I want to know, what will happen if I will try to activate windows in VM after I’ve activated windows (with the same key) on host. Will this render both installations “non-genuine”?

Also I’d appreciate any comments on the whole situation

perf stats for doing nothing

2012-02-12T16:28:30+00:00

Perf stats for "doing nothing"

I've recently discovered the perf Linux tool. I heard that oprofile was deprecated and that there is a new tool, and I noted down to try it sometime.

Updated: more languages, fixed typos, more details, some graphs. Apologies if this shows twice in your feed.

The problem with perf stats is that I hate bloat, or even perceived bloat. Even when it doesn't affect me in any way, the concept of wasted cycles makes me really sad.

You probably can guess where this is going… I said, well, let's see what perf says about a simple "null" program. Surely doing nothing should be just a small number of instructions, right?

Note: I think that perf also records kernel-side code, because the lowest I could get was about ~50K instructions for starting a null program in assembler that doesn't use libc and just executes the syscall asm instruction. However, these ~50K instructions are noise the moment you start to use more high-level languages. Yes, this is expected, but the I was still shocked. And there's lots of delta between languages I'd expected to behave somewhat identical.

Again, this is not important in the real world. At all. They are just numbers, and probably the noise (due to short runtime) has lots of influence on the resulting numbers. And I might have screwed up the measurements somehow.

Test setup

Each program was the equivalent of 'exit 0' in the appropriate form for the language. During the measurements, the machine was as much as possible idle (single-user mode, measurements run at real-time priority, etc.). For compiled languages, -O2 was used. For scripts, a simple #!/path/to/interpreter (without options, except in the case of Python, see below) was used. Each program/script was run 500 times (perf's -r 500) and I've checked that the variations were small (±0.80% on the metrics I used).

You can find all the programs I've used at http://git.k1024.org/perf-null.git/, the current tests are for the tag version perf-null-0.1.

The raw data for the below tables/graphs is at log-4.

Results

Compiled languages

Language	Cycles	Instructions
asm	63K	51K
c-dietlibc	74K	57K
c-libc-static	177K	107K
c-libc-shared	506K	300K
c++-static	178K	107K
c++-dynamic	1,750K	1,675K
haskell-single	2,229K	1,338K
haskell-threaded	2,629K	1,522K
ocaml-bytecode	3,271K	2,741K
ocaml-native	1,042K	666K

Going from dietlibc to glibc doubles the number of instructions, and for libc going from static to dynamic linking again roughly doubles it. I didn't manage to compile a program dynamically-linked against dietlibc.

C++ is interesting. Linked statically, it is in the same ballpark as C, but when linked dynamically, it executes an order of magnitude more instructions. I would guess that the initialisation of the standard C++ library is complex?

Haskell, which has a GC and quite a complex runtime, executes slightly less instructions than C++, but uses more cycles. Not bad, given the capabilities of the runtime. The two versions of the Haskell program are with the single-threaded runtime and with the multi-threaded one; not much difference. A fully statically-linked Haskell binary (not recommended usually) goes below 1M instructions, but not by much.

OCaml is a very nice surprise. The bytecode runtime is a bit slow to startup, but the (native) compiled version is quite fast to start: only 2× number of instructions and cycles compared to C, for an advanced language. And twice as fast as Haskell ☺. Nice!

Shells

Language	Cycles	Instructions
dash	766K	469K
bash	1,680K	1,044K
mksh	1,258K	942K
mksh-static	504K	322K

So, dash takes ~470K instructions to start, which is way below the C++ count and a bit higher than the C one. Hence, I'd guess that dash is implemented in C ☺.

Next, bash is indeed slower on startup than dash, and by slightly more than 2× (both instructions and cycles). So yes, switching /bin/sh from bash to dash makes sense.

I wasn't aware of mksh, so thanks for the comments. It is, in the static variant, more efficient that dash, by about 1.5×. However, the dynamically linked version doesn't look too great (dash is also dynamically linked; I would guess a statically-linked dash "beats" mksh-static).

Text processing

I've added perl here (even though it's a 'full' language) just for comparison; it's also in the next section.

Language	Cycles	Instructions
mawk	849K	514K
gawk	1,363K	980K
perl	2,946K	2,213K

A normal spread. I knew the reason why mawk is Priority: required is that it's faster than gawk, but I wouldn't have guessed it's almost twice as fast.

Interpreted languages

Here is where the fun starts…

Language	Cycles	Instructions
lua 5.1	1,947K	1,485K
lua 5.2	1,724K	1,335K
lua jit	1,209K	803K
perl	2,946K	2,213K
tcl 8.4	5,011K	4,552K
tcl 8.5	6,888K	6,022K
tcl 8.6	8,196K	7,236K
ruby 1.8	7,013K	6,128K
ruby 1.9.3	35,870K	35,022K
python 2.6 -S	11,752K	10,247K
python 2.7 -S	11,438K	10,198K
python 3.2 -S	29,003K	27,409K
pypy -S	21,106K	10,036K
python 2.6	25,143K	21,989K
python 2.7	47,325K	50,217K
python 2.7 -O	47,341K	50,185K
python 3.2	113,567K	124,133K
python 3.2 -O	113,424K	124,133K
pypy	90,779K	68,455K

The numbers here are not quite what I expected. There's a huge delta between the fastest (hi Lua!) and the slowest (bye Python!).

I wasn't familiar with Lua, so I tested it thanks to the comments. It is, I think, the only language which actually improves from one version to the next (bonus points), and where the JIT version also make is faster. In context, lua jit starts faster than C++.

Perl is the one that goes above C++'s instructions count, but not by much. From the point of view of the system, a Perl 'hello world' is only about 1.3×-1.6x slower than a C++ one. Not bad, not bad.

Next category is composed of TCL and Ruby, both of which had older versions 2-3× slower than Perl, but whose most recent versions are even more slower. TCL has an almost constant slowdown across versions (5M, 6.9M, 8.2M cycles), but Ruby seems to have taken a significant step backwards: 1.9.3 is 5× slower than 1.8. I wonder why? As for TCL, I didn't expect it to be slower to startup than Perl; good to know.

Last category is Python. Oh my. If you run perf stat python -c 'pass' you get some unbelievable numbers, like 50M instructions to do, well, nothing. Yes, it has a GC, yes, it does import modules at runtime, but still… On closer investigation, the site module and the imports it does do eat a lot of time. Running a simpler python -S brings it back to a more reasonable 10M instructions, which is in-line with the other interpreted languages.

However, even with the -S taken into account, Python also slows down across versions: a tiny improvement from 2.6 to 2.7, but (like Ruby) a 3× slowdown from 2.7 to 3.2. Trying the “optimised” version (-O) doesn't help at all. Trying pypy, which was based on Python 2.7, makes it around 2× slower to startup (both with and without -S).

So in the interpreted languages, it seems only Lua is trying to improve, the rest of the languages are piling up bloat with every version. Note: I should have tried multiple perl versions too.

Java

Java is in its own category; you guess why ☺, right?

GCJ was version 4.6, whereas by java below I mean OpenJDK Runtime Environment (IcedTea6 1.11) (6b24-1.11-4).

Language	Cycles	Instructions
null-gcj	97,156K	74,576K
java -jamvm	85,535K	80,102K
java -server	147,174K	136,803K
java -zero	132,967K	124,977K
java -cacao	229,799K	205,312K

Using gcj to compile to “native code” (not sure whether that's native-native or something else) results in a binary that uses less than 100M cycles to start, but the jamvm VM is faster than that (85M cycles). Not bad for java! Python 3.2 is slower to startup—yes, I think the world has gone crazy.

However, the other VMs are a few times slower: server (the default one) is ~150M cycles, and cacao is ~230M cycles. Wow.

The other thing about java is that it was the only one that couldn't be put nicely in a file that you just ‘exec’ (there is binfmt_misc indeed, but that doesn't allow different Java classes to use different Java VMs, so I don't count this), as opposed to every single other thing I tested here. Someone didn't grow on Unix?

Comparative analysis

Since there are almost 4 orders of magnitude difference between all the things tested here, a graph of cycles or instructions is not really useful. However, cycles/instruction, branches percentage and branches miss-predicted percentage can be. Hence first the cycles/instructions:

Pypy is jumping out of the graph here, with the top value of over 2 cycles/instruction. Lua JIT is also bigger than Lua non-JIT, so maybe there's something to this (mostly joking, two data points don't make a series). On the other hand, Python wins as best cycles/instruction (0.91). Lots of ILP, to get below 1?

Java gets, irrespective of VM, consistently near 1.0-1.1. C++ gets very different numbers between static linking (1.666) and dynamic linking (1.045), whereas C has basically identical numbers. mksh also has a difference between dynamic and static linking. Hmm…

Ruby, TCL and Python have consistent values across versions.

And that's about what I can see from that graph. Next up, percentage of branches out of total instructions and percentage of branches missed:

Note that the two lines shouldn't really be on the same graph; for the branch %, the 100% is the total instructions count, but for the branch miss %, the 100% is the total branch count. Anyway.

There are two low-value outliers:

dynamically-linked C++ has a low branch percentage (17.46%) and a very low branch miss percentage (only 4.32%)
gcj-compiled java has a very low branch miss percentage (only 2.82%!!!), even though is has a “regular” branch percentage (20.85%)

So it seems the gcj libraries are well optimised? I'm not familiar enough with this topic, but on the graph it does indeed stand out.

On the other end, mksh-static has a high branch miss percentage: 11.60%, which jumps clearly ahead of all the others; this might be why it has a high cycles/instruction count, due to all the stalls in misprediction; one has to wonder why it confuses the branch predictor?

I find it interesting that the overall branch count is very similar across languages, both when most of the cost is in the kernel (e.g. asm) and when the user-space cost heavily over-weighs the kernel (e.g. Java). The average is 20.85%, minimum is 17.46%, max 22.93%, standard deviation (if I used gnumeric correctly) is just 0.01. This seems a bit suspicious to me ☺. On the other hand, the mispredicted branches percentage varies much more: from a measly 2.82% to 11.60% (5x difference).

Summary

So to recap, counting just instructions:

going from dietlibc to glibc: 2× increase
going from statically-linked libc to dynamically-linked libc: doubles it again
going from C to C++: 5× increase
C++ to Perl: 1.3×
Perl to Ruby: 3×
Ruby to Python (-S): 1.6x
Python -S to regular Python: 5x
Python to Java: 1×-2×, depending on version/runtime
branch percentage (per total instructions) is quite consistent across all of the programs

Overall, you get roughly three orders of magnitude slower startup between a plain C program using dietlibc and Python. And all, to do basically nothing.

On the other hand, I learned some interesting things while doing it, so it wasn't quite for nothing ☺.

RC bugs 2012/05-06

2012-02-12T14:47:40+00:00

I was at FOSDEM over the last weekend, so here's my report for two weeks now:

~~#445842~~ – rmagic: "rmagic: fails with current libgd"
add patch from Andreas Beckmann, upload to DELAYED/2
~~#610332~~ – aiccu: "/etc/pm/sleep.d/60aiccu hook may cause unacceptable resume delays"
improve pm-utils hook script, upload to DELAYED/2, then moved to 0-day on maintainer's request
~~#628305~~ – src:gpa: "gpa: FTBFS: checking LIBASSUAN API version... does not match. want=1 got=2."
apply upstream patch, upload to DELAYED/2
~~#629693~~ – src:spim: "spim: FTBFS: ../CPU/spim-utils.c:117:7: error: invalid use of void expression"
add patch from Ubuntu / Matthias Klose, upload to DELAYED/2
~~#629747~~ – src:fso-abyss: "fso-abyss: FTBFS: build-dependency not installable: libvala-dev (>= 0.8.1)"
switch to vala-0.10, inspired by the patch from Ubuntu / Barry Warsaw, upload to DELAYED/2
~~#629782~~ – src:fso-misc-vapi: "fso-misc-vapi: FTBFS: build-dependency not installable: libvala-dev"
switch to vala-0.10, inspired by the patch from Ubuntu / Barry Warsaw, upload to DELAYED/2
~~#635550~~ – pmw: "pmw: piuparts: fails to install (update-gsfontmap again)"
send patch to the BTS, fixed by maintainer later
#642375 – cardstories: "cardstories: ftbfs on i386 due to test value > sys.maxint"
add patch from Ubuntu / Barry Warsaw, upload to DELAYED/3
~~#642566~~ – python-saga: "python-saga: doesn't depend on python"
apply missing part of Jakub Wilk's patch, upload to DELAYED/2, later integrated in maintainer upload
~~#650800~~ – src:hiredis: "hiredis: FTBFS on mipsel: rm: cannot remove `/tmp/redis.sock': No such file or directory"
sponsor maintainer upload
#652182 – src:cardstories: "cardstories: FTBFS: TypeError: 'NoneType' object has no attribute '__getitem__'"
drop "X-Python-Version: all", upload to DELAYED/3
~~#652231~~ – src:brickos: "brickos: FTBFS: semop(2): encountered an error: Invalid argument"
close as unreproducible
#656153 – jspwiki: "jspwiki: postinst failure: chown: invalid user: `tomcat6'"
add thoughts in the BTS
~~#656178~~ – mandos: "mandos FTBFS on buildds"
add build dependency and set locale for manpage building, upload to DELAYED/2
#657478 – biomaj-watcher: "biomaj-watcher: doesn't use invoke-rc.d"
use invoke-rc.d in postinst (and quote variable in config), upload to DELAYED/2, later cancelled on maintainer's request
~~#657746~~ – prolix: "prolix: under- and wrongly-documented, not working interactively at all"
upload new upstream release (pkg-perl)
~~#658394~~ – src:libio-socket-socks-perl: "libio-socket-socks-perl: FTBFS - test failures without network (was: [Bug 924169] [NEW] libio-socket-socks-perl fails to build without a network connection)"
upload package prepared by Jonathan Yu (pkg-perl)

Multi-Arch progress

2012-02-12T13:14:29+00:00

With dpkg from experimental and the new zlib upload in unstable, I've now got a partial Multi-Arch install. There are more packages necessary, particularly related to how -dev packages can exist and how a cross compiler gets built/installed in Multi-Arch world. One bug #659588 in libglib2.0-0 so far but that's quite good seeing as it's been all but impossible to test the Multi-Arch changes in existing packages until now.

$ dpkg --print-foreign-architectures

armel

i386

$ dpkg -l | cut -c -80 | grep armel|grep -v cross

ii  gcc-4.6-base:armel                   4.6.2-14                     

ii  libc6:armel                          2.13-26                    

ii  libdatrie1:armel                     0.2.5-3                

ii  libffi5:armel                        3.0.10-3                 

ii  libgcc1:armel                        1:4.6.2-14                 

ii  libgmp10:armel                       2:5.0.4+dfsg-1         

ii  libgmpxx4ldbl:armel                  2:5.0.4+dfsg-1        

ii  libgomp1:armel                       4.6.2-14         

ii  libmpc-dev:armel                     0.9-4              

ii  libmpc2:armel                        0.9-4          

ii  libmpfr-dev:armel                    3.1.0-3         

ii  libmpfr4:armel                       3.1.0-3       

ii  libpcre3:armel                       8.12-4                

ii  libpixman-1-0:armel                  0.24.4-1    

ii  libpng12-0:armel                     1.2.46-4     

ii  libpopt0:armel                       1.16-3        

ii  libpwl5:armel                        0.11.2-6    

ii  libselinux1:armel                    2.1.0-4.1      

ii  libstdc++6:armel                     4.6.2-14       

ii  libthai0:armel                       0.1.16-3     

ii  linux-libc-dev:armel                 3.2.4-1         

ii  zlib1g:armel                         1:1.2.6.dfsg-1

There are more packages which can be installed on amd64 for i386 as #659588 only affects Multi-Arch versions which cannot execute compiled binaries from the foreign architecture.

Further progress, inside a test chroot, involves using gcc-4.7 from experimental but even then, libc6-dev is not installable as a Multi-Arch package. That's the current blocker for toolchain stuff.



ii  gcc-4.7-base:i386               4.7-20120210-1

ii  libgcc1:i386                    1:4.7-20120210-1

Once we have libc6-dev:armel installable on i386 and amd64, work on cross-building Emdebian can be considered again. It's been a long time.

FACTA

2012-02-12T11:49:49+00:00

You know ACTA and related topics have reached the mainstream when the German news show devotes a segment of several minutes(!) to ACTA, complete with background information about the sleazy backroom dealings, and the various political sunday talk shows debate controversially about it, as well. In a round of five 60-plus aged people, only one defenced ACTA; the rest was sceptic or dead against it.

The several tens of thousands of people who walked the streets of Germany yesterday (I couldn't; I am ill) really managed to stir up Germany's political landscape. This. is. awesome.

Now if only the population at large would understand the inherent differences between physical counterfeiting, physical theft, and copyright infrigement.

That's where we all come in: Inform people about the background and actual issues. Now, more than ever, there's a chance that they will be willing to listen.

2012 update 9 for Debian Installer localization

2012-02-12T06:00:00+00:00

Indonesian completes level 1 and 3 and is now 100% everywhere
Turkish completes level 1 and is now 100% everywhere

Status for D-I level 1 (core D-I files):

34 languages 100%: ar ast bg bs cs de el eo es fa fr gu hi id it ja kk km kn ko mr nb nl pl pt ru sk sr sv ta th tr uk zh_CN
2 languages 99%: si te
1 language 97%: eu
2 languages 96%: be he
5 languages 95%: bn dz et ro zh_TW
2 languages 94%: da ga
3 languages 93%: hu is lo
1 language 92%: sl
2 languages 91%: pa vi
others are 90% or below

Status for D-I level 2 (packages that have localized material that may appear during default installs, such as iso-codes, tasksel, etc.):

23 languages 100%: bg cs da de eo es fa fr he id is it ja kk nl pl pt ru si sk tr uk zh_CN
4 languages 99%: be sl sv th
6 languages 98%: ast dz ca eu pt_BR ro
1 language 96%: nb
2 langauges 95%: el fi
5 languages 94%: ar gl hr vi zh_TW
6 languages 92%: bn bs hu ko ne sr
9 languages 91%: ga gu ka km lt mk mr ta te
others are 90% or below

Status for D-I level 3 (packages that have localized material that may appear during non-default installs, such as win32-loader)

31 languages 100%: be bg bs ca cs de el eo es fa fi fr ga gl he id is it ja kk nb nl pl pt ru sk sr sv th tr zh_CN
2 languages 98%: hu uk
others are 90% or below

Full 100% completeness (hall of fame) for 18 languages: Bulgarian, Czech, German, Esperanto, Spanish, Persian, French, Indonesian, Italian, Japanese, Kazakh, Dutch, Polish, Portuguese, Russian, Slovak, Turkish Simplified Chinese

XMPP for Children

2012-02-12T05:02:43+00:00

When Jacob was just born, I wondered how I might introduce them to computing. I thought over various things, but that wasn’t really the most pressing thing right then.

I don’t suppose that I could have predicted installing an XMPP IM server (Prosody) for the boys. And I certainly couldn’t have predicted creating accounts named: jacob, oliver, butterfly, bear. Because, as Jacob pointed out to me, if (Jacob’s favorite toy) butterfly is typing with his wings, then he shouldn’t be logged in as Jacob. I admire my 5-year-old’s security consciousness…

Anyhow, as I mentioned yesterday, Jacob and Oliver enjoy “their” computer, which I recently put on the LAN. The firewall does not pass any of its traffic to the Internet, though, with very limited exceptions.

Jacob can read, and is starting to enjoy typing as well. So I thought he would enjoy sending IMs to me. As his computer has no GUI, I needed a text-mode client. Something with an IRC-like interface that could be scripted to open up a window with me directly sounded perfect. Initially I tried irssi’s XMPP plugin, but it proved to be too buggy (wanting to always latch on to a particular resource on the remote end, not having very predictable window behavior, etc.) So I switched to mcabber. With a couple of quick configuration bits to get him automatically logged in, remove superflous windows, and connect him directly to a chat with me, it was set. And well-loved. He sent me a mix of real words and random things he created by replacing letters in “Jacob” or by holding down keys.

In the mcabberrc, besides the obvious setting of username and password, there is:

set log_win_height = 1 set hook-post-connect = source ~/.mcabber/post-connect.rc

The hook is simply:

roster search Dad roster hide

After awhile, Jacob wanted to switch computers. He wanted to use my laptop, and me use his computer. He refused to switch back. I asked him why. “Because on your computer, my name is red.” I should have known. I set it to bright white on his computer, but I think tomorrow we may need to upgrade him to the color monitor I’ve been saving for just such an occasion… It will be a whole new set of discoveries, I’m sure.

Update: I also tried out freetalk, which looked like it would meet my goals nicely. The problem was it didn’t have a dedicated “everything typed goes to this person” mode. It did have a mode where it put the person’s JID on the command line by default, but excessive use of backspace key by a 5-year-old could wipe that out and leave it in a state where he’d be confused.

Express Sessions with connect-redis + Socket.IO in CoffeeScript

2012-02-12T02:14:01+00:00

If you are working with Socket.IO and Express Framework while developing your applications you must know they work together so Socket.IO will listen on the same port that your HTTP/S server and will serve the requests by that way, but Socket.IO doesn’t works on the same paradigm where you pass a middleware to the route and you could get authentication on this manner. Instead, you must configure Socket.IO with a global authorization method or a handshake authorization manner.

Since in my applications I serve cookies already where I save session info and this is already stored in Redis with help of the module connect-redis then I should be able to get this info onto Socket.IO authorization method and with this allow only to authorized users to get info from my websockets.

I found how to do it with the default session-store in-memory from Express, and adapted that so it could work with Redis.

Here’s the snippet:

Hope anyone can found it useful!

On arranging meetings

2012-02-11T23:44:31+00:00

I've been spending a lot more time recently in meetings. Mostly things I should actually be at. And in general if it's something I think is reasonable I'll try to be there. In an effort to help with this I actually keep my work calendar up to date. Given that I'm running Linux on my laptop and the corporate standard is Exchange this requires a little bit of effort on my part (the Thunderbird Provider for Microsoft Exchange and Android support for talking to Exchange are helpful with this).

Sometimes it seems like I shouldn't bother. I spent this week at a conference, and marked my calendar to indicate I was out of the office. I think I had at least 3 meeting requests, all for things that would actually have been appropriate for me to go to. Last week I managed to be booked for 7 hours of meetings from 7am until noon. That included a 30 minute window where I was triple booked.

The thing is, I'm really not that busy in terms of meetings - you can usually find a spot when I'm free on any given day unless I'm actually not in the office. If you bother to check my calendar, that is.

Another problem I have is the times people like to book meetings at. Booking a technical meeting at 9am isn't going to get me at my best. Equally doing so at 5pm is likely to have me clock watching to make sure I don't miss my bus and/or train. Also I seem to work with far too many people who don't eat lunch and book hour long meetings at midday or 1pm.

I understand sometimes that's the only time you can get everyone into a room together, but at least bloody ask and explain the need rather than just sending out a meeting invite.

Finally, book meetings of a realistic length. There are some people who invite me to things and cause me to add another 30 minutes on the end, because I know it always overruns.

It's not all bad. I have a VP who always runs a meeting to time, and never seems to call one for spurious reasons. I've also worked with a program manager who will organize the meeting so that if you're only there for one point on the agenda that'll get dealt with near the beginning so it doesn't take up more of your time that it needs to. Funnily enough I'm much more likely to go to things both of these people arrange.

Disclaimer: In the unlikely event anyone I work with who invites me to meetings is reading this, I might be talking about you, but everything I mention has been done by more than one person, so I'm not thinking about anyone in particular for each point.

multiarch (and) hacks

2012-02-11T16:19:02+00:00

I experienced situations of writing substantial amounts of code (feature branches) for hours, days and even weeks only to find later that the written code can be thrown to the bin, either because of hidden design problem(s) or too much negative implications outweighting the positive implications of the change.

After reading some recent multiarch threads on debian-devel@ and seeing what hacks are seriously being proposed to implement only to keep the thing going, I now think that the low-level part of the Debian multiarch implementation proposal is no less broken than its high-level part, and the whole proposal is a one big hack which requires and will require more subhacks. Some will benefit from the added functionality, but all, both maintainers and users will suffer from drawbacks.

How two paragraphs above are related? We still have the time to revert the changes and say "sorry, it was a nice idea but the software world isn't ready".

mutt-notmuch 0.2

2012-02-11T14:15:20+00:00

My mutt-notmuch hack seems to be a quite popular way to integrate Mutt with notmuch. As a nice consequence, my (indexed!) inbox attracts patches from mutt-notmuch users eager to improve it. Collecting some of them, I've just tagged mutt-notmuch 0.2 with the following changes:

readline support and query history
support for spaces in mailboxes (use case: gmail+offlineimap)
a new "tag" action
normalization of =names to support mutt macros that pass folder names
do not treat +opt as a valid cmdline option (to ease tagging)

Many thanks to Scott Barker, Christine Spang, David Newgas, and Ryan Kavanagh for the above patches.

While I was at it, I've also moved mutt-notmuch repository to Gitorious. Git self-hosting is nice, but either you move to something like gitolite (which I didn't have time to setup and tune ATM) or you're stuck without merge requests which are quite nice. (Why not Github? Because.)

If you're using mutt-notmuch you might also be interested in the discussion of libnotmuch support in mutt. I'd love to see that landing in mutt and be able to throw away mutt-notmuch entirely, but that seems a bit premature as of yet.

It hurts…

2012-02-11T08:58:19+00:00

…when you find caution *after* being sufferer of some known problem.

Caution marked red at the beginning of this article is the way I bricked my laptop.

Maybe they should change the name of that option from EasyFlash to EasyBrick.

Useful changelog…

2012-02-11T08:49:36+00:00

Here it goes:

Tell me if I should upgrade it or not…

Magic entries for BTRFS and Software RAID

2012-02-11T07:21:07+00:00

I’ve just discovered that the magic database for the file(1) command in Debian/Unstable has no support for Linux Software RAID and that it’s support for BTRFS is lacking (no reporting of space used, number of devices, or the UUID). Below is my first draft of a change to fix these problems. I would appreciate it if someone with a big-endian system could test these out and let me know how they go, I suspect that I will have to change the “lelong” types to “long” but I’m not sure.

4096 lelong 0xa92b4efc Linux Software RAID
>4100 lelong x version 1.2 (%d)
>4112 belong x UUID=%8x:
>4116 belong x \b%8x:
>4120 belong x \b%8x:
>4124 belong x \b%8x
>4128 string x name=%s
>4168 lelong x level=%d
>4188 lelong x disks=%d

0 lelong 0xa92b4efc Linux Software RAID
>4 lelong x version 1.1 (%d)
>16 belong x UUID=%8x:
>20 belong x \b%8x:
>24 belong x \b%8x:
>28 belong x \b%8x
>32 string x name=%s
>72 lelong x level=%d
>92 lelong x disks=%d

# BTRFS
0×10040 string _BHRfS_M BTRFS Filesystem
>0x1012b string >\0 label "%s",
>0×10090 lelong x sectorsize %d,
>0×10094 lelong x nodesize %d,
>0×10098 lelong x leafsize %d,
>0×10020 belong x UUID=%8x-
>0×10024 beshort x \b%4x-
>0×10026 beshort x \b%4x-
>0×10028 beshort x \b%4x-
>0x1002a beshort x \b%4x
>0x1002c belong x \b%8x,
>0×10078 lequad x %lld/
>0×10070 lequad x \b%lld bytes used,
>0×10088 lequad x %lld devices

Label vs UUID vs Device Someone asked on a mailing list about the issues related...
Starting with BTRFS Based on my investigation of RAID reliability [1] I have...
Software vs Hardware RAID Should you use software or hardware RAID? Many people claim...

Shell Scripts For Preschoolers

2012-02-11T04:19:13+00:00

It probably comes as no surprise to anybody that Jacob has had a computer since he was 3. Jacob and I built it from spare parts, together.

It may come as something of a surprise that it has no graphical interface, and Jacob uses the command line and loves it — and did even before he could really read.

A few months ago, I wrote about the fun Jacob had with speakers and a microphone, and posted a copy of the cheat sheet he has with his computer. Lately, Jacob has really enjoyed playing with the speech synthesizer — both trying to make it say real words and nonsense words. Sometimes he does that for an hour.

I was asked for a copy of the scripts I wrote. They are really simple. I gave them names that would be easy for a preschooler to remember and spell, even if they conflicted with existing Unix/Linux commands. I put them in /usr/local/bin, which occurs first on the PATH, so it doesn’t matter if they conflict.

First, for speech systhesis, /usr/local/bin/talk:

#!/bin/bash echo "Press Ctrl-C to stop." espeak -v en-us -s 150

espeak comes from the espeak package. It seemed to give the most consistenly useful response.

Now, on to the sound-related programs. Here’s /usr/local/bin/ssl, the “sound steam locomotive”. It starts playing a train sound if one isn’t already playing:

#!/bin/bash pgrep mpg321 > /dev/null || mpg321 -q /usr/local/trainsounds/main.mp3 & sl "$@"

And then there’s /usr/local/bin/record:

#!/bin/bash cd $HOME/recordings echo "Now recording. Press Ctrl-C to stop." DATE=`date +%Y-%m-%dT%H-%M-%S` FILENAME="$DATE-$$.wav" chmod a-w *.wav exec arecord -c 1 -f S16_LE -c 1 -r 44100 "$FILENAME"

This simply records in a timestamped file. Then, its companion, /usr/local/bin/play. Sorry about the indentation; for whatever reason, it is being destroyed by the blog, but you get the idea.

#!/bin/bash case "$1" in train) mpg321 /usr/local/trainsounds/main.mp3 ;; song) /usr/bin/play /usr/local/trainsounds/traindreams.flac ;; *) cd $HOME/recordings exec aplay `ls -tr| tail -n 1` ;; esac

So, Jacob can run just “play”, which will play back his most recent recording. As something of a bonus, the history of recordings is saved for us to listen to later. If he types “play train”, there is the sound of a train passing. And, finally, “play song” plays Always a Train in My Dreams by Steve Gillette (I heard it on the radio once and bought the CD).

Some of these commands kick off sound playing in the background, so here is /usr/local/bin/bequiet:

#!/bin/bash killall mpg321 &> /dev/null killall play &> /dev/null killall aplay &> /dev/null killall cw &> /dev/null

Livin’ on the edge

2012-02-10T18:26:15+00:00

zenbook ~ # uname -r
3.3.0-rc3-fenio
zenbook ~ #

Let’s see how many sudden shutdowns will be on that kernel

Rant: Your custom rounding code is wrong

2012-02-10T18:20:00+00:00

Maxims (or “TL;DR” if you want):

Any time you write your own rounding code, it's most likely wrong.
Any time you think your rounding code is faster than what the standard library can give you, you're most likely wrong. (This one has exceptions, though.)
Any time your rounding code involves the constant 0.5, it's almost certainly wrong.

Why?

So, let's start off by assuming you have a float and you want to convert that to an int. (Or maybe you have five million of them.) I'll assume a more or less x86/Linux environment, although of course, the real world is a bit more complex.

OK, so you've decided to write your own rounding function.

Anyway, first of all, what are you going to do about the floating-point numbers that don't represent real numbers? Positive and negative infinity? NaN? OK, maybe only floating-point buffs ever care about those. (I'm not one of them.) Maybe it would be nice to have some sort of consistent behavior for, say, the value 2^48, but maybe you also don't care, since you have a certain amount of control as of where these numbers come from anyway. Sure.

So, you write your function. Most likely, someone taught you somewhere that you can do proper rounding in Excel by adding 0.5, so you write:

return (int)(x + 0.5f);

Now, of course, the (int) cast is a rounding in itself, in truncation mode (this is, by the way, my single least favorite decision by K&R). If you're unlucky enough to be on 32-bit, this makes the compiler spew out tons of stuff for changing the rounding mode, so you pretty much lost. Perhaps you write something similar using inline assembler and the x87 FISTP instruction; now you just made sure that on 64-bit, all your floats have to round-trip from SSE to x87 just to get to ints. (Or maybe you use some dodgy trick you learned from a web forum at some point and thought were exceedingly cool and elegant, which destroys your range. Who knows.)

OK, so maybe you can get the int-truncate step fast and correct. After all, eventually CPUs started to get their own instructions for convert-to-int-with-truncate, since C has made that so common. So your function converts 0.4f to 0, and 0.5f to 1. Fine. Now, what do you do with negative values? -0.5f becomes 0, which means you just violated that round(-x) = -round(x). (Even worse, -0.6f becomes 0.) OK. Maybe you don't care about that (in fact, maybe you think, correctly or not, that this is exactly what you want). Maybe you change your inline assembler to test if the number is negative or not, and add -0.5f instead, in which case you just incurred a (costly) branch. But OK, let's say you're happy about the behavior with negative numbers.

Maybe you also don't worry that this introduces a bias towards rounding up, both for positive and negative numbers; slightly more floats will be rounded up than down. After all, you're most likely not in the hair-splitting business. It should be noted, though, that IEEE has a solution for this, called “round-to-nearest-even”, where numbers ending in .5 are rounded up and down every other time to avoid just this issue (-1.5f → -2, -0.5f → 0, 0.5f → 0, 1.5f → 2, and so on).

OK, but even if you're happy with your behavior with numbers ending in about 0.5, at least it should round the right way for things between 0.0 and less and 0.5, right? Here comes another maxim: Rounding twice is doomed to fail. This might seem intuitive, but not everybody understands this; when I was ten, I had a maths teacher that insisted that 1.46 could be rounded to 2 (“I wouldn't grade that as an error”, in her own words), because you could round 1.46 → 1.5 → 2.0. But whenever you do an add, you also do a round, so you do get rounding twice. The case in point is ~0.4999999701976776123046875, the largest floating-point number below 0.5. If you had 0.5 to this, you'll end up with 0.999999975, except that you don't have enough mantissa bits to represent this exactly, so it gets rounded off to 1.0, so now your function yields round(0.49999997) = 1! Oops.

OK. So maybe you don't care that your function rounds the wrong way every now and then. (And if you think this won't ever happen; I've seen it happen in real code. For doubles.) Sometimes speed is more important than correctness. But now you've basically said that you do not care, so allow me to suggest just calling lrintf(), the standard library's method for converting floats to ints. The implementation looks like this:

cvtss2si %xmm0, %eax
ret

If you run with -fno-math-errno (implied by -ffast-math), it will even get inlined to this single instruction! ¹ Do you really think you can beat that? It will even follow whatever rounding mode you've set, which is most likely the default, sane, bias-free round-to-even, with no issues with 0.499999975 or similar madness. The inlined version will take you all of three cycles (at least two of which can be overlapped with other stuff), and there's no way on Earth you can beat that for speed. (I mean, your CPU has special circuitry for doing this. If adding 0.5 were the fastest way, maybe it would just do that internally.) You'd think people actually noticed this and didn't write their own functions that are slower than what the standard library gives, but a) maybe the code was written ten years ago on an entirely different platform where this was not the case, and b) people generally don't measure their code; a profiler seems to be a completely unheard of tool for most developers, even if they write performance-critical code, so they miss the fact that 70% of their time is spent in their brilliant inlined rounding function.

Granted, this is the most dodgy part of my assertions; for instance, if you're on ARM, lrintf() is vastly more complex, and the compiler matters here. (I did take a reservation. :-) ) But even if so, the other maxims really come into play; if you really care about speed, the right thing to do here on ARM is to do a single round-to-int instruction (VCVTR), not to add 0.5 as a fudge factor.

¹ Unfortunately GCC won't output this sequence without -fno-math-errno, due to language-lawyering reasons — basically it seemingly is not allowed to make a conversion that doesn't set errno on overflow, even though glibc is, since it has communicated it to the program through other means.

Update: Sam pointed out that the constant in question is not 0.499999975, but slightly larger.

kvm and product_uuid

2012-02-10T18:08:17+00:00

While looking for something to use as a system-unique fall-back when a TPM is not available, I looked at /sys/devices/virtual/dmi/id/product_uuid (same as dmidecode‘s “System Information / UUID”), but was disappointed when, under KVM, the file was missing (and running dmidecode crashes KVM *cough*). However, after a quick check, I noticed that KVM supports the “-uuid” option to set the value of /sys/devices/virtual/dmi/id/product_uuid. Looks like libvirt supports this under capabilities / host / uuid in the XML, too.

host# kvm -uuid 12345678-ABCD-1234-ABCD-1234567890AB ...
host# ssh localhost ...
...
guest# cat /sys/devices/virtual/dmi/id/product_uuid
12345678-ABCD-1234-ABCD-1234567890AB

People behind Debian: Ana Beatriz Guerrero López, member of the Debian KDE team

2012-02-10T18:00:00+00:00

If you met Ana, you’ll easily remember her. She has a great and pronounced Spanish accent… I’m glad that the existence of the Debian Women project helped her to join Debian because she has been doing a great job.

From KDE packaging to publicity/marketing work, her interests shifted over the years but this allowed her to stay very involved. As she explains it very well, Debian is big enough so that you can stop doing something which is no longer fun for you, and still find something new to do in another part of Debian!

Read on to learn more about Ana, the KDE team, Debian’s participation to the Google Summer of Code, and more.

Raphael: Who are you?

Ana: I’m Ana Guerrero López and I’m in my early 30s. I was born and raised in the wonderful city of Sevilla, Spain and I live in Lyon, France. I share my life with another Debian Developer and my paid work is doing Debian support and integration, so you won’t be surprised to read that Debian is a big part of my life.

Raphael: How did you start contributing to Debian?

Ana: Although I knew about the existence of Linux since 1997 or so, I didn’t really start using Linux until the summer 2001 when I finally got a computer on my own and an Internet link at home. In the beginning, I was using Mandrake in a dual boot with Windows and later around 2003, I happily moved to only using Debian and ditching the Windows partition. Once settled as a Debian user, I knew anybody could help improve the distribution but I hesitated to join mostly due to two reasons, my perception of Debian was the one of a very elitist and aggressive club and who wants to join this kind of cult^wproject? And even if I wanted to join, I did not know how to get started.

By the summer of 2004, the Debian Women project started, it made me seeing Debian as a more welcoming project, and I started maintaining my first packages. The following summer 2005, I attended akademy 2005 (the annual KDE conference) where I had the pleasure to meet there some of the people from the KDE team and this really made a difference for me. Christopher Martin and Adeodato Simó, with the help of other people, have started the maintenance of KDE as a team a few months before and by that time most of the KDE modules where under the maintenance umbrella of the team. This was a very good move since it allowed easily to share the KDE maintenance in a more coordinated way and also eased having non-DDs, like me at that time, to join in and help.

The Debian Women project started, it made me seeing Debian as a more welcoming project.

Raphael: You’re part of the Debian KDE team. What’s your role in the team and what are your plans for Wheezy?

Ana: Nowadays, I am not as active in the KDE team as I used to be in the past. The KDE 3 to KDE 4 transition was quite tiring and changes on the KDE side like the successive marketing renames, the shorter 6 months schedule (it used to be at least 9) or the uncoordinated KDE releases mostly burnt me out. Currently, I am mostly working in helping others to get started within the team, some small fixes here and there, and helping with the uploads: an upload of the full KDE suite to the archive requires some building power and upload bandwidth not everybody have.

For Wheezy, with the tentative freeze date in June, the plan is to try to ship the latest possible point release of the KDE 4.8 series. The first release of the series, 4.8.0 was released a couple of weeks ago and while writing these lines, the packaging work for 4.8 hasn’t started yet. The next move for the team is getting 4.7.4 in unstable, currently sitting in experimental.

For Wheezy, […] the plan is to try to ship the latest possible point release of the KDE 4.8 series.

Besides the KDE packages, there is some software which users perceive as KDE, such as amarok, digikam, etc., which are not part of KDE but fall under its umbrella. These other programs have their own maintainers and their updates depend greatly in the availability of them. For the KDE office suite, we have right now KOffice in the archive. KOffice got a fork some time ago named Calligra and we should replace KOffice by Calligra in the archive before the release of Wheezy. Sadly there isn’t yet a final release of Calligra to use.

My personal goal for Wheezy was to finish the removal of all the remaining packages depending on KDE 3 and Qt 3 that Squeeze still contained. The removal of the KDE 3 libraries and all the packages using them was quickly achieved after the release of Squeeze. The removal of Qt 3 soon showed that it was task harder than expected since some popular packages (sometimes not in the Debian archive, e.g. third-party scientific software) depend on it, and also Qt 3 is a requirement for LSB compatibility. Right now, Qt 3 has been orphaned for 9 months and nobody has shown any interest in adopting it.

Raphael: KDE, much like GNOME, has been forked by people who were unhappy by the direction that the project has taken since version 4 (cf Trinity). What’s your personal opinion on KDE 4.x and what’s the position of the Debian KDE team concerning this fork?

Ana: I use KDE 4 on my laptop and I think it is a solid desktop environment and platform. However I am finding it less and less attractive for me. On one side, my usage of the computer has been slightly changing and on the other side, I do not like how the new developments in KDE are evolving, things like plasmoids or activities are not attractive for me. I have switched my other 2 systems to awesome although I continue to use mainly a bunch of KDE applications: dolphin, konsole, kate, juk, kmix, etc. So you might say my desktop environment is an awesome KDE.

Regarding the Trinity project, a lot of users complained very loudly when KDE 3 got replaced by KDE 4 in testing/unstable, so I find quite laudable the decision of some users to act instead and try to continue with a forked development of KDE 3. However the Trinity team seems to be about 3 persons (funny for a project named Trinity ) while KDE 3 is big. In perspective, it does not look that big because KDE 4 is even larger, but it is still too much for such small team. In addition those developers need to maintain Qt3 that has been end-of-lifed years ago by Nokia/Trolltech¹. So my guess is that sooner or later the project will fade away.

Nobody from the KDE team is interested in Trinity and in case someone wants to package it for Debian, they would have to make a new team. For the reasons mentioned above: Qt3 maintenance and reduced upstream group, this would be a bad idea.

My advice if you do not like KDE 4 and you miss KDE 3, would be taking a look at razor-qt based on Qt4 and quite similar to KDE 3.

¹ I read they have plans to port it to Qt4, but frankly that could take some years… same it took to the KDE project for KDE 4.0.0

Raphael: You used to maintain news.debian.net, a WordPress blog dedicated to Debian, but you stopped a while ago. A few months later you started to maintain a Debian page on Google+. Why did you stop the blog and what’s your goal with the Google+ page?

Ana: I blogged about the reasons I started news.debian.net. In short, I thought Debian needed a better system to publish news, something like a blog. I first tried to suggest the idea to the press/publicity team but they weren’t interested, so I started the project alone. IMHO the blog worked quite well and I was feeling like it should be made official. I talked about this with some people but at the time I wasn’t pushing it because I had other priorities and I knew pushing it to become official would need some extra time and energy.

Stefano decided to start the discussion about making news.debian.net official (that’s moving it to a debian.org domain) in its own initiative. After the public discussion and some private exchange of emails with DSA, the situation became frustrating and I decided to close news.debian.net after the release of Squeeze.

Later, during DebConf, an officer from the press team announced they were launching a blog and I asked Stefano if he could try to have a discussion about this to see if it could still somehow fit my ideas, and maybe contributing myself, but nobody from the press team answered Stefano’s email and the blog hasn’t started yet either.

Irony that communication didn’t work when wanting to improve communication.

About the Google+ page, everyday I follow what is going in Debian and quite often I find things I want to share. I do not want to clutter my own profiles with Debian stuff or have people following me because of that, so I decided to create the Debian page when Google+ made them available. I like the fact that people can follow that without having an account in Google+ although they can not comment anonymously. I am not happy about the fact that Google+ is a closed platform but hopefully the data will become easier to export in the near future. Right now, there are some services that provides RSS feeds of Google+ pages if you want to follow the page and you are not in Google+ (or I could setup one if several people ask me).

Raphael: Last year you helped to manage Debian’s participation to the Google Summer of Code. How did it went? Is there something that you can improve for this year?

Ana: I think last year we managed to have people in Debian more aware about what the students were doing. That also helped students to get more feedback and therefore get to know more people in the project and get more integrated. Students were sending periodic public reports available to everybody interested in the status of the projects and some of them also held their own sessions in DebConf.

We still failed to start looking for mentors early enough and to give them information about how the GSoC worked and how they could have a successful project. Having good projects in Debian is harder than in other projects because the GSoC mostly promotes having students started in Open Source *coding* for a project, while Debian is more a project about integrating software and we overall do not have so many parts that has to be coded.

My personal goal for this year is to try getting the projects earlier to attract good students from the very beginning, even if that means we have less projects than in other years.

Raphael: What motivates you to continue to contribute year after year?

Ana: Three things. I like improving the OS I use, I like the friends I have made while working in Debian through the years and because I have fun.

Also Debian is quite a big project, so if you become tired or burn out working in some area, you always can easily find interesting things to do somewhere else.

Raphael: Is there someone in Debian that you admire for their contributions?

Ana: Adeodato Simó, he is now in a long leave from the project, but it is one of those persons who made a difference in the project in his job in the release team some years ago. Aurélien Jarno because of his tireless work in (e)glibc and porting of several architectures.

I also have special admiration for all those people who have been very active in the project for more than 7-8 years because I know it is not always easy to combine it with real life.

Thank you to Ana for the time spent answering my questions. I hope you enjoyed reading her answers as I did. Note that older interviews are indexed on wiki.debian.org/PeopleBehindDebian.

Subscribe to my newsletter to get my monthly summary of the Debian/Ubuntu news and to not miss further interviews. You can also follow along on Identi.ca, Google+, Twitter and Facebook

4 comments | Liked this article? Click here. | My blog is Flattr-enabled.

Dpkg: edit debian/control description without worrying about leading ‘.’ and ‘ ‘

2012-02-10T13:24:36+00:00

Hello

In Debian package, the debian/control file has a specific syntax where each line must begin with a space and paragraphs are separated by a single ‘.’. If you want to follow DEP-5 syntax for debian/copyright files, you have a similar problem when comes the time to fill the license text for creative-common license: it’s easy to miss a space or a dot.

Let’s say I want to add a new section in the description of my package. Here’s the current description:

Description: module to automate definition of a DBIx::Class::Schema
 DBIx::Class::Schema::Loader is an extension to DBIx::Class that automates the
 definition of a DBIx::Class::Schema by scanning table schemas and setting up
 columns and primary keys appropriately. It supports MySQL, PostgreSQL, SQLite
 and DB2.
 .
 Bare table definitions are fairly straightforward, but relationship creation
 is somewhat heuristic, especially with respect to choosing relationship types
 and names, as well as join types. The relationships generated by this module
 will probably never be as well-defined as hand-generated ones.

I want to add “This package is awesome” but I don’t want to be bothered by leading dots and white spaces.

First, I go in the package directory, then check the syntax with cme:

$ cme check dpkg

Then create a loop directory, where the dpkg content will be mapped to a fuse directory:

$ mkdir loop
$ cme fusefs dpkg -fuse-dir loop/
Mounting config on loop/ in background.
Use command 'fusermount -u loop/' to unmount
$ cat loop/control/binary/libdbix-class-schema-loader-perl/Description
DBIx::Class::Schema::Loader is an extension to DBIx::Class that automates the
definition of a DBIx::Class::Schema by scanning table schemas and setting up
columns and primary keys appropriately. It supports MySQL, PostgreSQL, SQLite
and DB2.

Bare table definitions are fairly straightforward, but relationship creation
is somewhat heuristic, especially with respect to choosing relationship types
and names, as well as join types. The relationships generated by this module
will probably never be as well-defined as hand-generated ones.

Then edit loop/control/binary/libdbix-class-schema-loader-perl/Description with your favorite editor.

For the sake of this blog, I will use echo to edit this file:

$ echo -e "\nThis module is awesome" >> loop/control/binary/libdbix-class-schema-loader-perl/Description

Now umount the loop (note that the actual debian/control is not changed before this step):

fusermount -u loop

Now the result:

$ cat debian/control
[...]
Description: module to automate definition of a DBIx::Class::Schema
 DBIx::Class::Schema::Loader is an extension to DBIx::Class that automates the
 definition of a DBIx::Class::Schema by scanning table schemas and setting up
 columns and primary keys appropriately. It supports MySQL, PostgreSQL, SQLite
 and DB2.
 .
 Bare table definitions are fairly straightforward, but relationship creation
 is somewhat heuristic, especially with respect to choosing relationship types
 and names, as well as join types. The relationships generated by this module
 will probably never be as well-defined as hand-generated ones.
 .
 This module is awesome

Here’s another example to fill the license text for a license while respecting DEP-5 syntax, (done after the cme command):

# creating this dir will create a new DEP-5
# standalone license section with name CC-BY-30
mkdir loop/copyright/License/CC-BY-30
# fill the text part of the  license
wget http://creativecommons.org/licenses/by/3.0/legalcode -O - \
 | html2text -style pretty > loop/copyright/License/CC-BY-30/text
$ fusermount -u loop
$ cat debian/copyright
[...]
License: CC-BY-30
 .
 .
 .
                                Creative_Commons
 .
 .
 Creative Commons Legal Code
 .
 .
 Attribution 3.0 Unported
[...]

And that’s it. (note that cme command is new and is available only from version 2.004 of Config::Model)

Of course, there’s more than a awy to do it: you can also run cme edit dpkg and add or edit license text in the GUI (or in an editor launched by the GUI). cme will take care of the syntax details.

All the best

Release Critical Bug report for Week 06

2012-02-10T12:01:00+00:00

The bug webinterface of the Ultimate Debian Database currently knows about the following release critical bugs:

In Total:	1340
Affecting Wheezy:	777
Wheezy only:	122
Remaining to be fixed in Wheezy:	655

Of these 655 bugs, the following tags are set:

Pending in Wheezy:	41
Patched in Wheezy:	86
Duplicates in Wheezy:	45
Can be fixed in a security Update:	21
Contrib or non-free in Wheezy:	16
Claimed in Wheezy:	0
Delayed in Wheezy:	6
Otherwise fixed in Wheezy:	49

Ignoring all the above (multiple tags possible) 447 bugs need to be fixed by Debian Contributors to get Debian 7.0 Wheezy released.

However, with the view of the Release Managers, 687 need to be dealt with for the release to happen.

Please see Interpreting the release critical bug statistics for an explanation of the different numbers.

Book about phpMyAdmin 3.4

2012-02-10T11:00:00+00:00

With quite usual delay after release, book Mastering phpMyAdmin 3.4 for Effective MySQL Management has been published.

Unlike the previous edition, I was not doing technical review here, but it is anyway special book for me - the cover photo is coming from me, what is actually for the very first time this has happened.

As usual with Packt opensource books, phpMyAdmin project receives some money for each sold copy of the book, so you're welcome to buy it :-).

Filed under: English Phpmyadmin | 2 comments | Flattr this!

Starting with BTRFS

2012-02-10T09:52:16+00:00

Based on my investigation of RAID reliability [1] I have determined that BTRFS [2] is the Linux storage technology that has the best potential to increase data integrity without costing a lot of money. Basically a BTRFS internal RAID-1 should offer equal or greater data protection than RAID-6.

As BTRFS is so important and so very different to any prior technology for Linux it’s not something that can be easily deployed in the same way as other filesystems. It is possible to easily switch between filesystems such as Ext4 and XFS because they work in much the same way, you have a single block device which the filesystem uses to create a single mount-point. While BTRFS supports internal RAID so it may have multiple block devices and it may offer multiple mountable filesystems and snapshots. Much of the functionality of Linux Software RAID and LVM is covered by BTRFS. So the sensible way to deploy BTRFS is to give it all your storage and not make use of any other RAID or LVM.

So I decided to do a test installation. I started with a Debian install CD that was made shortly before the release of Squeeze (it was first to hand) and installed with BTRFS for the root filesystem, I then upgraded to Debian/Unstable to get the latest kernel as BTRFS is developing rapidly. The system failed on the first boot after upgrading to Unstable because the /etc/fstab entry for the root filesystem had the FSCK pass number set to 1 – which wasn’t going to work as no FSCK program has been written. I changed that number to 0 and it then worked.

The initial install was on a desktop system that had a single IDE drive and a CD-ROM drive. For /boot I used a degraded RAID-1 and then after completing the installation I removed the CD-ROM drive and installed a second hard drive, after that it was easy to add the other device to the RAID-1. Then I tried to add a new device to the BTRFS group with the command “btrfs device add /dev/sdb2 /dev/sda2” and was informed that it can’t do that to a mounted filesystem! That will decrease the possibilities for using BTRFS on systems with hot-swap drives, I hope that the developers regard it as a bug.

Then I booted with an ext3 filesystem for root and tried the “btrfs device add /dev/sdb2 /dev/sda2” again but got the error message “btrfs: sending ioctl 5000940a to a partition!” which is not even found by Google.

The next thing that I wanted to do was to put a swap file on BTRFS, the benefits for having redundancy and checksums on swap space seem obvious – and other BTRFS features such as compression might give a benefit too. So I created a file by using dd to take take from /dev/zero, ran mkswap on it and then tried to run swapon. But I was told that the file has holes and can’t be used. Automatically making zero blocks into holes is a useful feature in many situations, but not in this case.

So far my experience with BTRFS is that all the basic things work (IE storing files, directories, etc). But the advanced functions I wanted from BTRFS (mirroring and making a reliable swap space) failed. This is a bit disappointing, but BTRFS isn’t described as being ready for production yet.

Discovering OS Bugs and Using Snapshots I’m running Debian/Unstable on an EeePC 701, I’ve got an...
Reliability of RAID ZDNet has an insightful article by Robin Harris predicting the...
How I Partition Disks Having had a number of hard drives fail over the...

NIST::NVD::Store::SQLite3

2012-02-10T06:54:55+00:00

I published an SQLite3 storage back-end to NIST::NVD on the CPAN. It’s pretty quick. About as fast as the DB_File one, but without the down side of being tied to DB_File. It shouldn’t be too difficult to re-factor this code to any DBI-based database. MariaDB anyone?

I know it works on Debian. The nightly CPAN test results should come back shortly and I’ll find out how well it works on other platforms.

Daffodils

2012-02-10T05:57:38+00:00

Dear Lazyweb,

I'd like a video of the 90s commercial "That's what daffodils do". If you do that, I'll release a basic library for the iPhone to use SPDY[1].

[1] I said basic!

Comments with OpenID

2012-02-10T01:08:10+00:00

Readers who look at our blog itself (rather than one of the lovely sites that reprint our articles) may have noticed that you can now comment in either the usual WordPress way (Name/Email/Link) or by logging in with a social media profile from one of a large range of providers, including WordPress, Livejournal, Yahoo, Google and many more.

This uses the broadly-cooperative openID system. If you run a website that accepts reader contributions, you should allow comments with openid because it helps people to use their existing social media membership without you having to surrender any control to facebook, twitter, or anyone else (unless you choose to). You also don’t have to ask your readers to weaken their security settings like with disqus (which requires javascript and third-party cookies).

The comment form on our site is powered by the openid plugin, together with our co-op’s version of the comments-with-openid plugin which can be downloaded from our site. Please download them if you’d find them useful for your WordPress site. (I’d love to adopt the official comments-with-openid at wordpress.org because the previous maintainer doesn’t answer – anyone know how to do that? I’m surprised it’s not in the FAQ.)

Do you use some other platform? What tools have let you add openid logins to it? For example, Drupal has some openID support in its core distribution: what else is out there?

2012-02-09: Thursday

2012-02-09T22:57:00+00:00

Michael Meeks gave awesome talks at FOSDEM, so Kate was inspired to hack on LibreOffice. I was inspired to write this blog entry in a list. She probably wins.
Building LibreOffice master on Debian stable failed for her with a segmentation fault in GNU Make. A bit of searching threw up Savannah bug #20033, which is hitting everyone on the upstream mailing list.
Bumped severity and offered to NMU Debian bug #622644.
Then actually tried building make-dfsg in cowbuilder, and aclocal fails in the clean environment because /usr/share/aclocal does not exist. I think it's related to Debian bug #565663, but I'm still poking it.

Is GPL usage really declining?

2012-02-09T22:33:55+00:00

Matthew Aslett wrote about how the proportion of projects released under GPL-like licenses appears to be declining, at least as far as various sets of figures go. But what does that actually mean? In absolute terms, GPL use has increased - any change isn't down to GPL projects transitioning over to liberal licenses. But an increasing number of new projects are being released under liberal licenses. Why is that?

The figures from Black Duck aren't a great help here, because they tell us very little about the software they're looking at. FLOSSmole is rather more interesting. I pulled the license figures from a few sites and found the following proportion of GPLed projects:

RubyForge: ~30%
Google Code: ~50%
Launchpad: ~70%

I've left the numbers rough because there's various uncertainties - should proprietary licenses be included in the numbers, is CC Sharealike enough like the GPL to count it there, that kind of thing. But what's clear is that these three sites have massively different levels of GPL use, and it's not hard to imagine why. They all attract different types of developer. The RubyForge figures are obviously going to be heavily influenced by Ruby developers, and that (handwavily) implies more of a bias towards web developers than the general developer population. Launchpad, on the other hand, is going to have a closer association with people with an Ubuntu background - it's probably more representative of Linux developers. Google Code? The 50% figure is the closest to the 56.8% figure that Black Duck give, so it's probably representative of the more general development community.

The impression gained from this is that the probability of you using one of the GPL licenses is influenced by the community that you're part of. And it's not a huge leap to believe that an increasing number of developers are targeting the web, and the web development community has never been especially attached to the GPL. It's not hard to see why - the benefits of the GPL vanish pretty much entirely when you're never actually obliged to distribute the code, and while Affero attempts to compensate from that it also constrains your UI and deployment model. No matter how strong a believer in Copyleft you are, the web makes it difficult for users to take any advantage of the freedoms you'd want to offer. It's as easy not to bother.
So it's pretty unsurprising that an increase in web development would be associated with a decrease in the proportion of projects licensed under the GPL.

This obviously isn't a rigorous analysis. I have very little hard evidence to back up my assumptions. But nor does anyone who claims that the change is because the FSF alienated the community during GPLv3 development. I'd be fascinated to see someone spend some time comparing project type with license use and trying to come up with a more convincing argument.

(Raw data from FLOSSmole: Howison, J., Conklin, M., & Crowston, K. (2006). FLOSSmole: A collaborative repository for FLOSS research data and analyses. International Journal of Information Technology and Web Engineering, 1(3), 17–26.)

comments

Debian packaging for beginners @ FOSDEM

2012-02-09T18:31:11+00:00

After not attending FOSDEM for a few years, this year I decided to attend and also give a talk about “Debian packaging for beginners”, a replay of a talk given by Gergely Nagy in Debconf11 (video). As the per distribution devrooms were replaced a few years ago with the cross-distributions rooms, I thought it might be a good chance to have this kind of introduction talk to help people started contributing to Debian (or derived distributions).

The talk wasn’t meant to replace reading the documentation (new maintainer’s guide, developer’s references and the debian policy), but it’s a good start for those who want some hands-on experience. The idea is to start with a just the upstream directory and progress trough the various errors and warnings we get during the build process using dpkg-buildpackage. The outcome is of course very basic, but enough to get people ready to do things on their own, including various QA tests on the package (e.g. lintian and debdiff).

The presentation covers the important points of the 3 main files in the debian directory: control, changelog and rules. It addition it holds some information about the various tools one can used to test the packages. I hope to make another version of the presentation to be more standalone than just having the main points during the talk itself.

Filed under: Debian GNU/Linux, FOSDEM

Recipes for extending Net-SNMP

2012-02-09T18:30:35+00:00

SNMP stands for Simple Network Management Protocol. It allows a manager to query information from an agent. A popular use of SNMP is to retrieve network interface counters to plot a bandwidth graph. While the “S” in SNMP stands for simple, SNMP can be quite difficult to deal with. However, it is still the de facto standard for retrieving metrics in an heterogeneous environment. Net-SNMP is a suite of SNMP applications including an agent. Out of the box, this agent exports a lot of information but if something is missing, there are several ways to extend it. Which ones?

tl;dr: in my opinion, extending Net-SNMP should be done with either of those methods:

using extend directive;
using pass_persist directive;
using AgentX protocol.

SNMP in a nutshell

The variables accessible with SNMP through the agent are organized in a tree. Each variable is typed and associated to an object identifier (or OID). For example, .1.3.6.1.2.1.31.1.1.1.1.2 is the OID of the name of the second network card.

Usually, SNMP is operated over UDP, port 161. For the purpose of this article, we consider a manager can issue three kinds of requests:

GET: retrieve a variable.
GETNEXT: retrieve the next variable, in lexical order of their OID. This operation enables an SNMP manager to walk through all available variables, for example to discover the list of interfaces of an agent. This operation is what makes an SNMP agent difficult to implement but this is also a critical feature.
SET: request the modification of a variable.

Since such an OID may be difficult to handle by humans, available variables are described by a MIB module which is a file that should be parsed to get a mapping between OID and variable names and types. For example, the above OID can also be referred as IF-MIB::ifName.2. Those MIB modules are described using a subset of ASN.1 defined in RFC 2578.

A MIB module allows to group several variables into a conceptual table. For example, IF-MIB::ifDescr, IF-MIB::ifType, IF-MIB::ifSpeed are columns of IF-MIB::ifTable. Each row is associated to an index. In the case of IF-MIB::ifTable, it is the interface index. Net-SNMP includes snmptable allowing one to display such a table in a natural manner¹:

$ snmptable localhost IF-MIB::ifTable
SNMP table: IF-MIB::ifTable

 ifIndex ifDescr           ifType ifMtu   ifSpeed
       1      lo softwareLoopback 16436  10000000
       2    eth0   ethernetCsmacd  1500 100000000
       3    eth1   ethernetCsmacd  1500  10000000
       4     br0   ethernetCsmacd  1500         0

Variables inside a table are called columnar objects while those outside are scalar objects. SNMP, as a protocol, does not care of this distinction since it does not know the concept of MIB modules.

Other useful tools from Net-SNMP includes snmpget to get a specific variable, snmpwalk to discover available variables (with GETNEXT operation) from an OID and snmptranslate to translate between object names and OID.

$ snmpget localhost IF-MIB::ifDescr.2
IF-MIB::ifDescr.2 = STRING: eth0
$ snmpwalk localhost IF-MIB::ifDescr
IF-MIB::ifDescr.1 = STRING: lo
IF-MIB::ifDescr.2 = STRING: eth0
IF-MIB::ifDescr.3 = STRING: eth1
IF-MIB::ifDescr.4 = STRING: br0
$ snmpwalk -On public localhost .1.3.6.1.2.1.2.2.1.2
.1.3.6.1.2.1.2.2.1.2.1 = STRING: lo
.1.3.6.1.2.1.2.2.1.2.2 = STRING: eth0
.1.3.6.1.2.1.2.2.1.2.3 = STRING: eth1
.1.3.6.1.2.1.2.2.1.2.4 = STRING: br0
$ snmptranslate -On IF-MIB::ifDescr.3
.1.3.6.1.2.1.2.2.1.2.3

Extending Net-SNMP

ethtool -S allows us to access various statistics from a network card, like the number of octets received and transmitted or the number of collisions. While some of those statistics are defined in IF-MIB, in RMON-MIB and in EtherLike-MIB, some of them are not available and very specific. Therefore, we would like to export those statistics, unmodified, through SNMP.

To the best of my knowledge, no MIB module exists for such an usage. The first task is to write one. It can be called ETHTOOL-MIB. It only contains one table indexed by the interface index and the name of the statistic. Here is the expected output:

$ snmpwalk localhost ETHTOOL-MIB::ethtoolStat.2
ethtoolStat.2.'align_errors' = Counter64: 0
ethtoolStat.2.'broadcast' = Counter64: 25
ethtoolStat.2.'multicast' = Counter64: 345
ethtoolStat.2.'rx_errors' = Counter64: 0
ethtoolStat.2.'rx_missed' = Counter64: 0
ethtoolStat.2.'rx_packets' = Counter64: 262011
ethtoolStat.2.'tx_aborted' = Counter64: 0
ethtoolStat.2.'tx_errors' = Counter64: 0
ethtoolStat.2.'tx_multi_collisions' = Counter64: 0
ethtoolStat.2.'tx_packets' = Counter64: 296170
ethtoolStat.2.'tx_single_collisions' = Counter64: 0
ethtoolStat.2.'tx_underrun' = Counter64: 0
ethtoolStat.2.'unicast' = Counter64: 261641

I have coded various implementations of this MIB module and made them available in a GitHub repository.

With arbitrary commands

The extend directive allows the agent to execute an arbitrary command and to provide its output and status through NET-SNMP-EXTEND-MIB module. If we want to export the statistics for eth0, we could add something like this in snmpd.conf, using a custom script to format the output of ethtool -S in a way that is more suitable for our use:

# Export eth0 statistics
extend eth0-names  /full/path/to/ethtool-stats eth0 names
extend eth0-values /full/path/to/ethtool-stats eth0 values

The result can be retrieved through NET-SNMP-EXTEND-MIB::‌nsExtendOutput2Table:

$ snmpwalk localhost NET-SNMP-EXTEND-MIB::nsExtendOutput2Table
nsExtendOutLine."eth0-names".1 = STRING: tx_packets
nsExtendOutLine."eth0-names".2 = STRING: rx_packets
nsExtendOutLine."eth0-names".3 = STRING: tx_errors
nsExtendOutLine."eth0-names".4 = STRING: rx_errors
nsExtendOutLine."eth0-names".5 = STRING: rx_missed
nsExtendOutLine."eth0-names".6 = STRING: align_errors
nsExtendOutLine."eth0-names".7 = STRING: tx_single_collisions
nsExtendOutLine."eth0-names".8 = STRING: tx_multi_collisions
nsExtendOutLine."eth0-values".1 = STRING: 246309
nsExtendOutLine."eth0-values".2 = STRING: 223941
nsExtendOutLine."eth0-values".3 = STRING: 0
nsExtendOutLine."eth0-values".4 = STRING: 0
nsExtendOutLine."eth0-values".5 = STRING: 0
nsExtendOutLine."eth0-values".6 = STRING: 0
nsExtendOutLine."eth0-values".7 = STRING: 0
nsExtendOutLine."eth0-values".8 = STRING: 0

It is also possible to configure such a command using SET requests. For example, if we want to add statistics for eth2, we could issue the following command:

$ snmpset -m +NET-SNMP-EXTEND-MIB localhost \
>    'nsExtendStatus."eth2-names"'  = createAndGo \
>    'nsExtendCommand."eth2-names"' = /full/path/to/ethtool-stats \
>    'nsExtendArgs."eth2-names"'    = 'eth2 names' \
>    'nsExtendStatus."eth2-values"'  = createAndGo \
>    'nsExtendCommand."eth2-values"' = /full/path/to/ethtool-stats \
>    'nsExtendArgs."eth2-values"'    = 'eth2 values'
nsExtendStatus."eth2-names" = INTEGER: createAndGo(4)
nsExtendCommand."eth2-names" = STRING: /full/path/to/ethtool-stats
nsExtendArgs."eth2-names" = STRING: eth2 names
nsExtendStatus."eth2-values" = INTEGER: createAndGo(4)
nsExtendCommand."eth2-values" = STRING: /full/path/to/ethtool-stats
nsExtendArgs."eth2-values" = STRING: eth2 values
$ snmpgetnext -m +NET-SNMP-EXTEND-MIB localhost \
>    'nsExtendOutLine."eth2-names"'
nsExtendOutLine."eth2-names".1 = STRING: tx_packets

This can be very convenient but also a huge security risk. You should disable this functionality or enable it only for some SNMPv3 user.

Using extend does not allow to implement an arbitrary MIB module. If the cache is not disabled, extend is a good solution for various simple needs, like providing the content of a one-line file or the output of some Nagios plugin. However, exporting anything tabular, like interface statistics, is not a good fit.

With pass-through scripts

Net-SNMP allows one to extend the agent by delegating some sub-tree to a script defined by the pass_persist directive. Such a script will receive requests on its standard input with a very simple line-oriented protocol described in the manual page of snmpd.conf. Here is an example of interaction:

→  PING
←  PONG
→  get
→  .1.3.6.1.4.1.39178.100.1.1.1.2.2.109.117.108.116.105.99.97.115.116
←  .1.3.6.1.4.1.39178.100.1.1.1.2.2.109.117.108.116.105.99.97.115.116
←  counter64
←  223941
→  getnext
→  .1.3.6.1.4.1.39178.100.1.1.1.2.2
←  .1.3.6.1.4.1.39178.100.1.1.1.2.2.97.108.105.103.110.95.101.114
←  counter64
←  0

The protocol is simple enough to implement it from scratch in any scripting language. However, in Perl, you may prefer to use SNMP::‌ExtensionSNMP::‌PassPersist extension from Sébastien Aperghis-Tramoni. You can find the complete example on Github.

use SNMP::Extension::PassPersist;
my $extsnmp = SNMP::Extension::PassPersist->new(
    backend_collect => \&update_tree
    );
$extsnmp->run;

sub update_tree {
    my @interfaces = </sys/class/net/*>;
    foreach my $interface (@interfaces) {
        $interface =~ s/^.*\///;
        open(ETHTOOL, "ethtool -S $interface 2>/dev/null |") or next;
        while (<ETHTOOL>) {
            /^\s+(\w+): (\d+)$/ or next;
            my $name  = $1;
            my $value = int($2);
            my $oid   = oid_compute($interface, $name);
            $extsnmp->add_oid_entry($oid, "counter", $value);
        }
        close(ETHTOOL);
    }
}

With Python, snmp-passpersist module, from Nicolas Agius, provides a similar interface. Look at the complete example on GitHub.

pass_persist allows you to implement an arbitrary MIB module with honest performances. Here are some important things to know about this directive:

Unless you have a recent version of Net-SNMP, 64bit types are converted to 32bit equivalents. The counters may overflow quickly.
While handling a request, you are not allowed to pause (for example, to fetch a remote information): the agent would become unresponsive. If you need slow-to-get data or data from the agent, grab them in a separate thread and ensure you have them beforehand.

With AgentX protocol

AgentX is a protocol defined in RFC 2741 allowing a master agent to be extended by independent sub-agents. For snmpd to become a master agent, you just need to add master agentx in snmpd.conf. Here are some of the benefits of an AgentX sub-agent over pass_persist:

No configuration is needed for the master agent to accept an additional sub-agent. A sub-agent registers to the master agent the MIB modules (or part of them) it wants to take care of.
A sub-agent is decoupled from the master agent. It can run with a different identity or be integrated into another daemon to export its internal metrics, send traps or allow remote configuration through SNMP.
AgentX protocol can be carried over TCP. Sub-agents can therefore run on a foreign host or in a jailed environment.
64bit types are fully supported. Traps are also supported.

Perl and Python

Net-SNMP provides a Perl module, NetSNMP::agent, to write a sub-agent using AgentX protocol. This module is not as convenient as the module for pass_persist described above since it does not provide a convenient layer to put required information into some datastore. Basically, it mimics the available C API. Here are a few examples of its use:

An implementation of ETHTOOL-MIB I have done in the same way as the pass_persist version. My Perl skills are a bit rusty and the code has many rough edges.
An implementation of a sub-agent for Redis servers.
An implementation of BRIDGE-MIB shipped with Net-SNMP.

There is no similar binding for Python but there is an agentx extension using ctypes. It provides a more high-level view. I did not write an example for this one but you can look at the example provided by the author.

C with “new” API

To write a sub-agent in C, you have to choose between two API:

the API inherited from UCD-SNMP also known as the “traditional” API;
the new API developed for Net-SNMP 5.x.

With both API, an handler is registered to take care of one or several sub-trees. With the traditional API, the handler is invoked with very little information. It is then up to you to locate the appropriate variable by mapping the index part of the OID to the objects you want to export. It is quite simple for scalar objects but columnar objects with complex indexes are a pain with such an API.

The new API provides various helpers. Some of them allow you to copy objects into an array or a list and let Net-SNMP library do the hard work of serving them. Another helper will provide methods to serve objects without putting them in a special structure but by providing an iterator. Some of those helpers can also help you to cache some variables.

On top of this new API, Net-SNMP comes with mib2c, a tool that will help you convert a MIB module into C code: after a few questions, you will get a skeleton C code you will have to complete with the logic to retrieve real objects. A generated file also provides directions on what to do (with some magic command to have a step-by-step cookbook).

I was able to implement ETHTOOL-MIB by just running it through mib2c, choosing the table container helper with cache and adding some code in ethtoolStatTable_data_access.c. You can look at the result on Github. Here is a stripped-down version² of what has been modified:

/* Socket for ioctl */
skfd = socket(AF_INET, SOCK_DGRAM, 0);

/* Iterate through all interfaces */
getifaddrs(&ifap);
for (ifa = ifap; ifa != NULL; ifa = ifa->ifa_next) {
    /* Grab statistics name and values */
    strcpy(ifr.ifr_name, ifa->ifa_name);
    drvinfo.cmd         = ETHTOOL_GDRVINFO;
    ifr.ifr_data        = (caddr_t) &drvinfo;
    ioctl(skfd, SIOCETHTOOL, &ifr);
    n_stats             = drvinfo.n_stats;
    strings->cmd        = ETHTOOL_GSTRINGS;
    strings->string_set = ETH_SS_STATS;
    strings->len        = n_stats;
    ifr.ifr_data        = (caddr_t) strings;
    ioctl(skfd, SIOCETHTOOL, &ifr);
    stats->cmd          = ETHTOOL_GSTATS;
    stats->n_stats      = n_stats;
    ifr.ifr_data        = (caddr_t) stats;
    ioctl(skfd, SIOCETHTOOL, &ifr);

    ifIndex = if_nametoindex(ifa->ifa_name);

    /* Iterate through statistics */
    for (i = 0; i < n_stats; i++) {

        /* Declare the appropriate index */
        strncpy(ethtoolStatName,
                (char *)&strings->data[i * ETH_GSTRING_LEN],
                ETH_GSTRING_LEN);
        ethtoolStatName[sizeof(ethtoolStatName) - 1] = '\0';
        ethtoolStatName_len = strlen(ethtoolStatName);

        /* Append this new variable to the container. */
        rowreq_ctx = ethtoolStatTable_allocate_rowreq_ctx();
        ethtoolStatTable_indexes_set(rowreq_ctx,
                    ifIndex,
                    ethtoolStatName, ethtoolStatName_len);
        rowreq_ctx->data.ethtoolStat.high = stats->data[i] >> 32;
        rowreq_ctx->data.ethtoolStat.low = (uint32_t)stats->data[i];

        CONTAINER_INSERT(container, rowreq_ctx);
    }
}

See? It is pretty easy. I have only enumerated all the objects I wanted to expose. I didn’t even have to convert statistics names to an OID, mib2c built a ethtoolStatTable_indexes_set() for this purpose. OpenHPI project provides an extensive documentation to write a sub-agent this way.

Now, on the downside, to implement a single table, I have 14 generated files whose code style is usually unfit to integrate into an existing project : some people, me included, do not like automatically generated code, all the more when it is so verbose. The solution would be to use the new API without mib2c; unfortunately, while the documentation exists, all provided examples rely exclusively on mib2c tool. Unless you are able to grasp the new API without it, I would restrict its use to two cases:

Writing a new module for inclusion into Net-SNMP. Since most modules are now written with the help of mib2c, there is no coding style problem.
Writing a standalone sub-agent for some MIB. As a standalone project, you won’t run into coding style problems either.

For other uses, let’s have a look at the traditional API.

C with “traditional” API

The traditional API is way simpler and even if it is quite old, it is easier to find associated examples. Here is how to initialize the agent, again, in a stripped-down version (the complete version is on GitHub):

static oid ethtool_oid[] = {1, 3, 6, 1, 4, 1, 39178, 100, 1};
static struct variable3 ethtool_vars[] = {
  {1, ASN_COUNTER64, RONLY, ethtool_stat, 3, {1, 1, 2}}
};

int main()
{
  netsnmp_enable_subagent();
  snmp_disable_log();
  snmp_enable_stderrlog();
  init_agent("ethtoolAgent");
  REGISTER_MIB("ethtoolStatTable", ethtool_vars,
               variable3, ethtool_oid);
  init_snmp("ethtoolAgent");
  while (1)
    agent_check_and_process(1);
}

The REGISTER_MIB() macro is used to register the provided root OID (ethtool_oid) to the master agent. It also registers handlers associated to various sub-OID through the ethtool_vars[] array. Each member of this array features the following fields:

magic is an integer allowing one to discriminate OID handled by the same handler. For a table, this allows to handle several columns with the same handler since they share a common index.
type is the type of the variable that will be returned.
acl tells if the object is read-only (RONLY) or read-write (RWRITE).
findVar is the handler associated to the object. Its task will be to find the appropriate object and return its value.
The two last fields are the relative OID this entry is responsible for along with its length.

The prototype of ethtool_stat() handler function is the following:

static u_char*
ethtool_stat(struct variable *vp, oid *name, size_t *length,
             int exact, size_t *var_len, WriteMethod **write_method);

This function should return the value of the requested variable or NULL if no appropriate object was found. The value can be statically stored since it will be copied. Here is the description of the arguments:

struct variable *vp is the structure the handle was associated with, except that vp->name is now a full OID, not a relative one. You can access vp->magic to determine which column or which scalar to handle if this handler is common to several objects.
oid *name and size_t *length describe the OID requested. If you want to return another OID (for a GETNEXT request for example), you can modify them. name points to an OID array large enough to fit any OID you want to return.
int exact tells if the requested OID should be exactly matched (case of GET or SET) or not (GETNEXT).
size_t *var_len is used to tell the size of the returned object.
WriteMethod **write_method is only used if the variable is modifiable. When a SET request is received, the API will call the handler with exact set to 1 and expects to get a pointer to the function that will handle the write operation³.

To implement this function, you are on your own. When relying on externally fetched data, you need to handle a cache yourself. I have used a red-black tree for this purpose (with code stolen from OpenBSD). This also enables an easy implementation of the GETNEXT operation.

Here are three projects using this API to provide SNMP support:

Keepalived is a VRRP daemon and a monitoring daemon for LVS clusters. I have added a complete SNMP support. The most interesting files are core/snmp.c, check/check_snmp.c and vrrp/vrrp_snmp.c. It features a tight integration into Keepalived event loop that should be reusable by other projects. It includes read-only support of all data structures (configuration, state and statistics), ability to send traps on events and a write support for some values.
lldpd is an implementation of 802.1AB (LLDP) which is a protocol allowing an equipment to advertise itself to its neighbors on Ethernet level. 802.1AB comes with a huge MIB module and many extensions. lldpd provides a fairly complete read-only implementation of this MIB module. Some tables have complex indexes. Have a look at agent.c. There is currently no write support and no traps but lldpd features privilege separation and some code allows it to still use an Unix socket to speak with the master agent.
Asterisk, an open source PBX, includes an AgentX sub-agent allowing one to grab some metrics. Look at res/snmp/agent.c.

ColorHug with non English locales

2012-02-09T11:00:00+00:00

Since infamous erasing of factory calibration in my ColorHug device and restoring calibration matrix, I noticed it did screen calibration wrong. However I did not find time to properly investigate the issue. Yesterdays mail from Richard was actually trigger for me so I've opened up this topic.

In the end it turned out to be caused by Little CMS wrongly parsing CCMX in case you are using locales which use something else than . as decimal point.

After lot of googling, I've realized there is probably no good way of parsing floats independent on current locales, so I used one of hacks I found and I think it's less intrusive - get current decimal point by printing float string using printf and then convert the string to it. I know it looks ugly, but including own implementation of strtod is also not nice and playing with locales is definitely something not thread safe to do within widely used library.

Anyway I've asked upstream to merge my patches, so let's see what they think of it.

Filed under: Debian English Linux Photography Suse | 8 comments | Flattr this!

Can aptitude show older versions of packages if available in /var/cache/apt/archives?

2012-02-09T10:10:24+00:00

Dear Lazyweb,

is it possible to tell aptitude to show older versions of a package next to the currently available one if it is still present in /var/cache/apt/archives? Like it does when you use unstable and experimental side by side? I know that aptitude does not really support downgrades of packages, but showing those packages directly in aptitude if available in the cache is a lot easier than searching them in the file system and installing them manually, especially if you don’t know where they hide.

SPI Feb 2012

2012-02-09T01:47:32+00:00

Software in the Public Interest, the mass-membership association that supports some great Free and Open Source Software projects, will hold a public board of directors meeting today, Thursday 9th February 2012 at 21:00 UTC. The day and time of SPI meetings has changed recently, so maybe different people can get to them now.

They’re held online, on irc.spi-inc.org (the OFTC network). The agenda for the meeting is open and available at http://www.spi-inc.org/meetings/agendas/2012/2012-02-09/ and there’s been a bit of discussion of back office support on the SPI email list.

I’ll link to a meeting summary from the comments in this blog post after it happens.

[life/americania] On making parking easier

2012-02-08T22:21:00+00:00

Paul Wayper writes about the merits of using toll transponders to pay for parking.

I can report that I'm able to use my FasTrak tag to pay for parking at San Francisco International Airport, and it does indeed rock.

I'm unaware of anywhere else accepting it as a form of payment though.

SELinux on Wheezy

2012-02-08T17:35:01+00:00

So, Collier Technologies LLC needs to pass annual audits to operate a certification authority recognized by the SoS. To this end, I’m working with the fine group of developers who maintain SELinux. It seems that the configuration of Xorg that I’m using while typing this here blog post does not have a policy set up for it in the Debian packages. Or if it does, I don’t know enough about it to figure it out.

I’ve been keeping logs and publishing them here:

http://www.colliertech.org/federal/nsa/

I’ll update this post as progress is made.

[edit 20120608T1042]

It looks like loading all .pp files (except alsa) makes X run:

cjac@foxtrot:/usr/share/selinux/default$ time sudo \
semodule -i `ls *.pp | grep -v -e 'base.pp' -e 'alsa.pp'`

real	0m24.148s
user	0m23.249s
sys	0m0.628s

I had to boot into single user mode and load the policies before switching to runlevel 2. To get the kernel args added to the grub command line, I modified /etc/default/grub to include this line:

cjac@foxtrot:/usr/share/selinux/default$ grep -i selinux /etc/default/grub
GRUB_CMDLINE_LINUX=" selinux=1 security=selinux"

Next steps:

get the policies loaded at boot time
get seinfo working

[edit 20120208T1305]

It looks like the seinfo package has not been updated in the last 18 months.

cjac@foxtrot:/usr/src/git/debian/setools$ grep url .git/config
	url = git://git.debian.org/git/users/srivasta/debian/setools.git
cjac@foxtrot:/usr/src/git/debian/setools$ git log | head -4
commit 22a5d3e451d8a1e60a3c746466c865e63089a92a
Merge: fa238f0 149e283
Author: Manoj Srivastava <srivasta>
Date:   Tue Jul 20 23:10:06 2010 -0700

[edit 20120208T1346]

Stephen tells me that the modules are persistent across re-boots.

> What's the best way to do this at boot?

You just do it once and it remains until/unless you remove it with
semodule -r.  No need to do it on each boot.  Normally it is done when
you install the policy package, but since your policy package apparently
didn't install all modules, I'm suggesting that you do so manually.

More DRBD Performance tests

2012-02-08T13:24:03+00:00

I’ve previously written Some Notes on DRBD [1] and a post about DRBD Benchmarking [2].

Previously I had determined that replication protocol C gives the best performance for DRBD, that the batch-time parameters for Ext4 aren’t worth touching for a single IDE disk, that barrier=0 gives a massive performance boost, and that DRBD gives a significant performance hit even when the secondary is not connected. Below are the results of some more tests of delivering mail from my Postal benchmark to my LMTP server which uses the Dovecot delivery agent to write it to disk, the rates are in messages per minute where each message is an average of 70K in size. The ext4 filesystem is used for all tests and the filesystem features list is “has_journal ext_attr resize_inode dir_index filetype extent flex_bg sparse_super large_file huge_file uninit_bg dir_nlink extra_isize“.

	p4-2.8
Default Ext4	1663
barrier=0	2875
DRBD no secondary al-extents=7	645
DRBD no secondary default	2409
DRBD no secondary al-extents=1024	2513
DRBD no secondary al-extents=3389	2650
DRBD connected	1575
DRBD connected al-extents=1024	1560
DRBD connected al-extents=1024 Gig-E	1544

The al-extents option determines the size of the dirty areas that need to be resynced when a failed node rejoins the cluster. The default is 127 extents of 4M each for a block size of 508MB to be synchronised. The maximum is 3389 for a synchronisation block size of just over 13G. Even with fast disks and gigabit Ethernet it’s going to take a while to synchronise things if dirty zones are 13GB in size. In my tests using the maximum size of al-extents gives a 10% performance benefit in disconnected mode while a size of 1024 gives a 4% performance boost. Changing the al-extents size seems to make no significant difference for a connected DRBD device.

All the tests on connected DRBD devices were done with 100baseT apart from the last one which was a separate Gigabit Ethernet cable connecting the two systems.

Conclusions

For the level of traffic that I’m using it seems that Gigabit Ethernet provides no performance benefit, the fact that it gave a slightly lower result is not relevant as the difference is within the margin of error.

Increasing the al-extents value helps with disconnected performance, a value of 1024 gives a 4% performance boost. I’m not sure that a value of 3389 is a good idea though.

The ext4 barriers are disabled by DRBD so a disconnected DRBD device gives performance that is closer to a barrier=0 mount than a regular ext4 mount. With the significant performance difference between connected and disconnected modes it seems possible that for some usage scenarios it could be useful to disable the DRBD secondary at times of peak load – it depends on whether DRBD is used as a really current backup or a strict mirror.

Future Tests

I plan to do some tests of DRBD over Linux software RAID-1 and tests to compare RAID-1 with and without bitmap support. I also plan to do some tests with the BTRFS filesystem, I know it’s not ready for production but it would still be nice to know what the performance is like.

But I won’t use the same systems, they don’t have enough CPU power. In my previous tests I established that a 1.5GHz P4 isn’t capable of driving the 20G IDE disk to it’s maximum capacity and I’m not sure that the 2.8GHz P4 is capable of running a RAID to it’s capacity. So I will use a dual-core 64bit system with a pair of SATA disks for future tests. The difference in performance between 20G IDE disks and 160G SATA disks should be a lot less than the performance difference between a 2.8GHz P4 and a dual-core 64bit CPU.

DRBD Benchmarking I’ve got some performance problems with a mail server that’s...
Some Notes on DRBD DRBD is a system for replicating a block device across...
Ethernet bonding Bonding is one of the terms used to describe multiple...

Migrating code to github

2012-02-08T11:00:00+00:00

As you might have noticed from my previous posts, we've moved phpMyAdmin code to github. Now I'm going to describe some things which might be useful for you if you are about do to similar switch.

While using git, moving to another hosting provider should be pretty straightforward. Just add another remote, push code there and it should be done. On the other side you probably have dozens of things in your infrastructure which you need to adjust. So the first thing to do is write down what all services are connected to your current git repositories. Let me name some which might be worth checking:

continuous integration server
snapshots generating
demo server (in case you're providing something like we do)
commit notifications
various statistics services such as cia.vc
website generating
references in wiki, website and documetation

Once you think you have remembered all important things (the less important will probably show up anyway, but majority of things should work), you're ready to make the move.

I've decided to make the move in few steps. First push all data to new location, what can take some time. I'll get in more details to that later. In the same time I asked all contributors to give me their login information, so that I can allow them access to new repositories. Once all recently active developers were migrated, it was time to push all remaining commits to new git repositories and make the switch for real.

Pushing git repo to another location, should be pretty easy. On the other side if you have many branches, it get's slightly more complex, I've ended up with following shell snippet (pushing all branches present in origin to github remote):

git branch -r | grep origin/ | grep -v HEAD | sed 's@.*/@@' | while read b ; do git checkout $b ; git push github $b:$b ; done

Please ensure that you check output of this, because you may hit network problems somewhere in the middle and you end up with few branches than you expect. As the code is pretty much idempotent, you can safely run it several times until there is nothing to push. You should also push all tags to new location:

git push --tags github

Okay, we've all data on right place, so let's switch all our users to new location:

git remote set-url origin git@github.com:phpmyadmin/phpmyadmin.git # read/write
git remote set-url origin git://github.com/phpmyadmin/phpmyadmin.git # read only

Of course everybody has to do this manually.

Next good thing is to let people know when they are using wrong repo (which will stay there for some time). Unfortunately there is AFAIK no way to warn them on pull, so let's warn at least on push:

$ cat > hooks/update
#!/bin/sh
echo "phpMyAdmin git repositories have moved to https://github.com/phpmyadmin"
exit 1

I think this is pretty much all. You can find some more bits in our Git migration wiki page.

PS: Thanks to github for offering us hosting and sorry for breaking their branch displaying page by too many divergent branches.

Filed under: Coding English Phpmyadmin | 4 comments | Flattr this!

Dependency order matters !

2012-02-08T10:34:36+00:00

Recently I've discovered a subtle consequence of how the order in which dependencies are specified in debian actually matters. While re-factoring the code of dose3, I changed the order in which dependencies are considered by our sat solver (of edos-fame) . I witnessed a twofold performance loss just by randomizing how variables were presented to our sat solver. This highlights, on one hand, how our solver is strongly dependent on the structure of the problem and, on the other hand the standard practice of debian maintainers to assign an implicit priority in the disjunctive dependencies where the first is the most preferred packages (and maybe the most tested, at least dependency-wise).

The basic idea of distcheck is to encode the dependencies information contained in a Packages file in CNF format and then to feed them to a sat solver to find out if a package has broken dependencies or if its dependencies are such that no matter what, it would be impossible to install this package on a user machine.

Conflicts are encoded as binary clauses. So if package A conflicts with package B, I add a constraint they says "not (A and B)" , that is A and B cannot be considered together. The dependencies encoding associates to each disjunction of the depends field a clause that says "A implies B". If a package foo depends on A,B | C,D , I'll add "foo implies A and B" & "foo implies C and D" . This encoding is pretty standard and it is easy to understand.

The problem is how the sat solver will search for a solution to the problem "Is is possible to install package foo in an empty repository". The solver we use is very efficient and can easily deal with 100K packages or more. But in general is not very good at dealing with random CNF instances. The reason because edos-debcheck is so efficient lies in the way it exploits the structure of the sat problems.

The goal of a sat solver is to find a model (that is a variable assignment list) that is compatible with the given set of constraints. So if my encoding of the debian repository is a set of constraints R, the installability problem boils down to add an additional constraint to R imposing that the variable associated to the package foo must be true, and then ask the solver to find a model to make this possible. This installation, in sat terms, would be just an array of variables that must be true in order to satisfy the given set of constraints.

If you look at the logic problem as a truth table, the idea is to find a row in this table. This is the solution of your problem. Brute force of course is not an option and modern sat solvers use a number of strategies and heuristic to guide the search in the most intelligent way possible. Some of them try to learn from previous attempts, some of them, when they are lost try to pick a random variable to proceed.

If we consider problems that have a lot of structure, award winning sat solver do not back track very much. By exploiting the structure of the problem, their algorithm allows them to considerably narrow down the search only to those variables that are really important to find a solution.

All this long introduction was to talk about the solver that is currently used in edos-debcheck and distcheck (that is a rewrite of the edos-debcheck).

So why dependency order matters ? If we consider any package, even if the policy does not specify any order in the dependencies, it's common practice to write disjunctive dependencies specifying the most probable and tested alternative first and all other, more esoteric choices later. Moreover real packages are considered *before* virtual packages. Since every developer seems be doing the same, some kind of structure might be hidden in the order in which dependencies are specified.

Part of the efficiency of the the solver used in our tools is actually due to the fact that its search strategy is strongly dependent on the order in which literal are specified in each clause. Saying the package foo depends on A and B is "different" then saying it depends on B and A, even if semantically equivalent.

In my tests, I found about a twofold performance loss if the order of literals is either randomized or inverted. This is clearly a specific problem related to our solvers, while other solvers might not be susceptible to such small structural changes. Sat competitions often employs some form of obfuscation strategies of well known problems with well known structures in order to make useless to encode a search strategy that exploits the specific structure of a problem.

Since here we're not trying to win a sat competition, but to provide tool to solve a specific problem, we are of course very happy to exploit this structure.

Squeeze RCs's Squashing 2012 #2

2012-02-08T09:18:00+00:00

This is the second entry in my series about squeeze release critical bug squashing. In response to my last blog post it was asked whether this is proper release critical bug squashing. Indeed there haven't been any patches or upload involved in this, only BTS handling, but this doesn't mean that these bugs weren't considered to be affecting squeeze. You can see this effort currently as weeding out the "wrong" bugs so that the list gets more useful and actually be able to ask maintainers to address the real issues.

You can at least see in this graph that the blue line is going down constantly since the year change instead of rising up like before. And I hope I will be able to keep it below the green line for a while still. Also thanks to the release-team and ftpmasters that it was possible to keep the massbugs about waf binary blob not being preferred source for modification out of affecting squeeze and ignore it for the current stable release—the required changes for those would rather be a fair bit intrusive for a stable update.

637213: doesn't fail in squeeze
648989: doesn't fail in squeeze
584568: most stuff (BTS, PTS, ...) use maintainer information from unstable only
624464: most stuff (BTS, PTS, ...) use maintainer information from unstable only
632106: most stuff (BTS, PTS, ...) use maintainer information from unstable only
632108: most stuff (BTS, PTS, ...) use maintainer information from unstable only
632262: most stuff (BTS, PTS, ...) use maintainer information from unstable only
632264: most stuff (BTS, PTS, ...) use maintainer information from unstable only
632265: most stuff (BTS, PTS, ...) use maintainer information from unstable only
632266: most stuff (BTS, PTS, ...) use maintainer information from unstable only
595298: squeeze has linux 2.6.32
613732: doesn't fail in squeeze
616694: doesn't fail in squeeze
620264: doesn't fail in squeeze
620573: doesn't fail in squeeze
621440: libdb4.8 not obsolete in squeeze
624593: doesn't fail in squeeze
626868: doesn't fail in squeeze
628475: libdb4.8 not obsolete in squeeze
629838: boost1.46 not in squeeze
628475: boost1.46 not in squeeze
631623: doesn't fail in squeeze
639990: doesn't fail in squeeze
632763: no libevent 2.0 in squeeze
627000: not affecting squeeze
658216: doesn't fail in squeeze
654870: doesn't fail in squeeze
655115: this is explicitly about testing
657092: working in squeeze
655710: doesn't fail in squeeze
644691: no nautilus 3 in squeeze

I am glad that I managed to keep it up and even have a nice margin in case I can't put any effort into it some day but still have more bugs squashed than days there are in the year so far. Currently I am at 60 bugs in 39 days. This gives a warm feeling. :)

Enjoy!

/debian | permanent link | Comments: 3 | Flattr this

Python and --prefix

2012-02-08T05:16:00+00:00

Something interesting I discovered about Python and --prefix that I can't see a lot of documentation on...

When you build Python you can use the standard --prefix flag to configure to home the installation as you require. You might expect that this would hard-code the location to look for the support libraries to the value you gave; however in reality it doesn't quite work like that.

Python will only look in the directory specified by prefix after it first searches relative to the path of the executing binary. Specifically, it looks at argv[0] and works through a few steps — is argv[0] a symlink? then dereference it. Does argv[0] have any slashes in it? if not, then search the $PATH for the binary. After this, it starts searching for dirname(argv[0])/lib/pythonX.Y/os.py, then dirname(argv[0])/../lib and so on, until it reaches the root. Only after these searches fail does the interpreter then fall back to the hard-coded path specified in the --prefix when configured.

What is the practical implications? It means you can move around a python installation tree and have it all "just work", which is nice. In my situation, I noticed this because we have a completely self-encapsulated build toolchain, but we wish to ship the same interpreter on the thing that we're building (during the build, we run the interpreter to create .pyc files for distribution, and we need to be sure that when we did this we didn't accidentally pick up any of the build hosts python; only the toolchain python).

The PYTHONHOME environment variable does override this behaviour; if it is set then the search stops there. Another interesting thing is that sys.prefix is therefore not the value passed in by --prefix during configure, but the value of the current dynamically determined prefix value.

If you run an strace, you can see this in operation.

readlink("/usr/bin/python", "python2.7", 4096) = 9
readlink("/usr/bin/python2.7", 0xbf8b014c, 4096) = -1 EINVAL (Invalid argument)
stat64("/usr/bin/Modules/Setup", 0xbf8af0a0) = -1 ENOENT (No such file or directory)
stat64("/usr/bin/lib/python2.7/os.py", 0xbf8af090) = -1 ENOENT (No such file or directory)
stat64("/usr/bin/lib/python2.7/os.pyc", 0xbf8af090) = -1 ENOENT (No such file or directory)
stat64("/usr/lib/python2.7/os.py", {st_mode=S_IFREG|0644, st_size=26300, ...}) = 0
stat64("/usr/bin/Modules/Setup", 0xbf8af0a0) = -1 ENOENT (No such file or directory)
stat64("/usr/bin/lib/python2.7/lib-dynload", 0xbf8af0a0) = -1 ENOENT (No such file or directory)
stat64("/usr/lib/python2.7/lib-dynload", {st_mode=S_IFDIR|0755, st_size=4096, ...}) = 0

Firstly it dereferences symlinks. Then it looks for Modules/Setup to see if it is running out of the build tree. Then it starts looking for os.py, walking its way upwards. One interesting thing that may either be a bug or a feature, I haven't decided, is that if you set the prefix to / then the interpreter will not go back to the root and then look in /lib. This is probably pretty obscure usage though!

All this is implemented in Modules/getpath.c which has a nice big comment at the top explaining the rules in detail.

FOSDEM 2012, the aftermath

2012-02-07T22:20:24+00:00

FOSDEM 2012 was very nice, as expected.

This year, I decided to stop consuming only and getting more involved. I helped staff the token sale and held two Lightning Talks. This turned out to be an awesome experience.

During the beer event, we sold 4600 beer tokens (FOSDEM does not earn anything from this; all proceeds go to Cafe Delirium). The sale was staffed with one to three people at all times which made for a constant but mostly bearable workload. While the others made good-natured fun of me and my German efficiency, I like to think that I helped make things run a tad more smoothly. Along those lines, we will have a sign asking people for exact change and/or to buy stacks that can be paid in bills if possible, next year; that should help streamline the whole thing even more. All in all, it turned out to be quite stressful, but tons of fun. I am looking forward to chuck in again next year.

The two Lighting Talks (vcsh on Saturday and git-annex on Sunday) I held very rather well received. Feedback after the talks and online was very positive which is always nice. There's been a slight increase in help requests and generic questions, so I assume people are trying vcsh and git-annex as a consequence of those talks, which is even nicer.

Also, I met a lot of people. Most of the meetings were pre-arranged, some by chance. Carrying a name tag at all times, I was approached by several people who knew me online but didn't know my face. I will try to always carry a name tag on conferences now and I suggest others do the same.

Finally, a huge thank you to everyone who helped make FOSDEM happen; you rock!

Make sure glue isn't stripped

2012-02-07T21:38:48+00:00

If you ever get this cryptic error when loading an Android native app:

FATAL EXCEPTION: main
java.lang.RuntimeException: Unable to start activity ComponentInfo{org.wikibooks.OpenGL/android.app.NativeActivity}: java.lang.IllegalArgumentException: Unable to load native library: /data/data/org.wikibooks.OpenGL/lib/libnative-activity.so
       at android.app.ActivityThread.performLaunchActivity(ActivityThread.java:1768)
       at android.app.ActivityThread.handleLaunchActivity(ActivityThread.java:1784)
       at android.app.ActivityThread.access$1500(ActivityThread.java:123)
       at android.app.ActivityThread$H.handleMessage(ActivityThread.java:939)
       at android.os.Handler.dispatchMessage(Handler.java:99)
       at android.os.Looper.loop(Looper.java:130)
       at android.app.ActivityThread.main(ActivityThread.java:3835)
       at java.lang.reflect.Method.invokeNative(Native Method)
       at java.lang.reflect.Method.invoke(Method.java:507)
       at com.android.internal.os.ZygoteInit$MethodAndArgsCaller.run(ZygoteInit.java:847)
       at com.android.internal.os.ZygoteInit.main(ZygoteInit.java:605)
       at dalvik.system.NativeStart.main(Native Method)
Caused by: java.lang.IllegalArgumentException: Unable to load native library: /data/data/org.wikibooks.OpenGL/lib/libnative-activity.so
       at android.app.NativeActivity.onCreate(NativeActivity.java:199)
       at android.app.Instrumentation.callActivityOnCreate(Instrumentation.java:1047)
       at android.app.ActivityThread.performLaunchActivity(ActivityThread.java:1722)
       ... 11 more

This may mean that Java couldn't find the ANativeActivity_onCreate function in your code, because it was stripped by the compiler.

If you use the native_app_glue NDK module, you may have noticed this strange code:

    // Make sure glue isn't stripped.
    app_dummy();

Let's experiment what happens with and without this line:

Calling app_dummy:

$ arm-linux-androideabi-objdump -T libs/armeabi/libnative-activity.so  | grep ANativeActivity_onCreate
000067fc g    DF .text  000000f8 ANativeActivity_onCreate

Not calling app_dummy :

$ arm-linux-androideabi-objdump -T libs/armeabi/libnative-activity.so | grep ANativeActivity_onCreate
$   # nothing

native_app_glue mainly defines Android callbacks. Since none of them are called directly by your code, the compiler strips the android_native_app_glue.o module entirely. If you use app_dummy however, it embeds it. Fortunately the compiler cannot strip the module on a per-function basis

That's why you need to call app_dummy when using the native_app_glue NDK module.

This looks like a ugly work-around though - isn't there a cleaner way?

fatrace: report system wide file access events

2012-02-07T19:53:05+00:00

Part of our efforts to reduce power consumption is to identify processes which keep waking up the disk even when the computer is idle. This already resulted in a few bug reports (and some fixes, too), but we only really just began with this.

Unfortunately there is no really good tool to trace file access events system-wide. powertop claims to, but its output is both very incomplete, and also wrong (e. g. it claims that read accesses are writes). strace gives you everything you do and don’t want to know about what’s going on, but is per-process, and attaching strace to all running and new processes is cumbersome. blktrace is system-wide, but operates at a way too low level for this task: its output has nothing to do any more with files or even inodes, just raw block numbers which are impossible to convert back to an inode and file path.

So I created a little tool called fatrace (“file access trace”, not “fat race” ) which uses fanotify, a couple of /proc lookups and some glue to provide this. By default it monitors the whole system, i. e. all mounts (except the virtual ones like /proc, tmpfs, etc.), but you can also tell it to just consider the mount of the current directory. You can write the log into a file (stdout by default), and run it for a specified number of seconds. Optional time stamps and PID filters are also provided.

$ sudo fatrace
rsyslogd(967): W /var/log/auth.log
notify-osd(2264): O /usr/share/pixmaps/weechat.xpm
compiz(2001): R device 8:2 inode 658203
[...]

It shows the process name and pid, the event type (Rread, Write, Open, or Close), and the path. Sometimes its’ not possible to determine a path (usually because it’s a temporary file which already got deleted, and I suspect mmaps as well), in that case it shows the device and inode number; such programs then need closer inspection with strace.

If you run this in gnome-terminal, there is an annoying feedback loop, as gnome-terminal causes a disk access with each output line, which then causes another output line, ad infinitum. To fix this, you can either redirect output to a file (-o /tmp/trace) or ignore the PID of gnome-terminal (-p `pidof gnome-terminal`).

So to investigate which programs are keeping your disk spinning, run something like

  $ sudo fatrace -o /tmp/trace -s 60

and then do nothing until it finishes.

My next task will be to write an integration program which calls fatrace and powertop, and creates a nice little report out of that raw data, sorted by number of accesses and process name, and all that. But it might already help some folks as it is right now.

The code lives in bzr branch lp:fatrace (web view), you can just run make and sudo ./fatrace. I also uploaded a package to Ubuntu Precise, but it still needs to go through the NEW queue. I also made a 0.1 release, so you can just grab the release tarball if you prefer. Have a look at the manpage and --help, it should be pretty self-explanatory.

phpMyAdmin is now at github

2012-02-07T14:04:22+00:00

I've just updated phpMyAdmin repos on github and enabled notifications hooks there, so the earlier announced switch to github is done.

All you need to do is point your repositories to pull/push to github, for main repository it can be done using:

git remote set-url origin git@github.com:phpmyadmin/phpmyadmin.git

If you are using just read only access then use following:

git remote set-url origin git://github.com/phpmyadmin/phpmyadmin.git

For other repositories just replace last part of the URL with repository name (they have not changed).

Everything should work as before, pushes should be now faster, because all notifications are done in background on github and they don't block pushing.

Filed under: English Phpmyadmin | 2 comments | Flattr this!

Video at FOSDEM 2012

2012-02-07T13:17:28+00:00

A year ago, during FOSDEM 2011, I walked up to the NamurLUG folks who'd been doing video coverage of FOSDEM for years, and suggested that this year, maybe they should consider using dvswitch for video coverage. While they seemed agreeable to this idea, they simply had not had the time to look at it in detail, and were therefore not using it. Since I've used dvswitch in the past, both as part of the DebConf video team and for my own concert recordings, I reasoned this a problem that could be solved by joining the video team.

However, when it came to be time to start preparing for FOSDEM 2012, we found that many of the NamurLUG people were not going to have as much time to prepare for video coverage as they had during past years, and therefore responsibility of FOSDEM video coverage fell entirely to me. This was fairly unexpected, though not too daunting.

For those of you who don't know dvswitch very well: a typical dvswitch installation for coverage of a talk requires:

Two cameras; one to capture the speaker, one to capture audience members asking questions.
A twinpact 100, used to convert a VGA output into a DV stream that we can load into dvswitch
Three laptops: one for the audience camera, one for the twinpact, and one for the speaker camera and the dvswitch installation
A GigE network between the three laptops, with an uplink to do streaming
A large amount of diskspace to store the recordings on. Since dvswitch records in raw DV, you require approximately 13GB per hour of recorded video.
Some powerful systems to do transcoding from DV to a streaming format such as Ogg Theora or WebM
A fairly extensive audio setup: one wireless headset microphone for the speaker, one or two wireless handheld microphones for people in the audience, and a number of microphones at select positions in the room to pick up ambient noise (which for obvious reasons you don't want to overdo on, but which if done well makes the audio on your recording sound much more natural). This also requires a mixing console.
And last but not least a (network of) streaming server(s) to do the streaming to people watching from home.

This is a rather large amount of work, and therefore it is not too unexpected that for DebConf, the DebConf video team takes a week to properly set up two rooms so that they can be used for recording. However, for FOSDEM we only have less than a day on-site to set up things, and we were going to cover five rooms. So, I tried to cut corners as much as possible:

We were going to work with one camera per room, rather than two. Not only would this reduce the number of required volunteers (of which I wasn't sure yet at that point how I'd find them), it would also reduce the amount of preparation needing to be done, and it would reduce the number of things that could go wrong.
We were going to have the network team do our network, rather than try to do it ourselves. The result was that we were not going to have to lay out cables, cut them, crimp connectors on them, test them, etc, and would only need to worry about using the network, instead of building it.
Since I'd read Thomas' announcement about an ''offer I cannot refuse'', I contacted him to see about that offer and get it all set up for FOSDEM. This way, I wouldn't have to worry about streaming servers etc, instead just about getting a stream on the network, into a flumotion server running somewhere; and after that, it would no longer be my problem.
I decided to simplify the audio setup somewhat, and not use ambient mics. These are nice, but not essential, so we could do without them, and that would be (at least) two microphones per room that the team wouldn't have to rent, place, wire up, and tune.

So in the end, I would need to take care of ten firewire-capable laptops, five cameras, five mixing consoles, several microphones, a number of transcoding machines, and shitloads of cabling (power, firewire, and audio). And about a week or two before the conference, as the massiveness of what we were about to do started to sink in, I was starting to have bad dreams of what would happen, wondering what I'd gotten myself into this time.

Now, one day after FOSDEM, I have to say it all turned out okay, but there are some things where there's still room for improvement. So that I don't forget, I thought I'd make a list of things that went well, and one of things that didn't go well. And since that might be interesting for other people, I thought I'd do it here, rather than in a private file.

First, things that did not go so well:

Set-up for some rooms took more time than we had, and as such some rooms did not get properly streamed or recorded for their first few talks. This was due to the fact that there was some confused communication between some members of the team, which meant that we lost a day that we'd planned for preparation, and as such we couldn't test as much as we'd hoped. We need to improve on that next year. Ideally, we should run the full set-up somewhere, with all cameras and all laptops running, and making sure that everything is ready to be set up and that we have every cable we need, before dividing everything over a number of boxes (one per room) and bringing it to the conference.
While some of our cameras were semi-professional Panasonic cameras that had a balanced XLR audio input, others were much lower-level camcorder-style cameras that did not have such a thing. When a camera has a good audio input, it's fairly easy to set up a link from the mixing console to the camera and get the audio into the stream that way. When a camera does not have such a thing, the set-up gets much more complex. Since we had so much to worry about, we did not notice that in one room, on saturday, something went wrong with audio. To combat this, we should improve on our audio set-up for next year, and also verify the streams every once in a while (I did the latter on sunday, so everything should be fine there).
Inherent to an ad-hoc network, there were some network-related issues. For instance, on sunday evening, during the last session in that room, Ben informed me that the link to room H.1301 had gone down. As we were investigating, we easily found the source of the problem:

The cable had been put under a door, and had been 'protected' with loads and loads of duct tape, but apparently that wasn't enough. This wasn't something we wanted to fix anymore, at that point; instead, we just let the cable be and focused on other things.
Storage. None of the systems used for the recordings were my own; instead, some were owned by IRILL and some were rented. If I want to do anything with my recordings, that means I have to copy the data off. The system I chose was to bring a USB3 hard disk and copy everything over; but for saturday, this took an hour and thirty minutes. On sunday, my USB3 disk was fairly full, so I had to revert to a USB2 one, which increased the time by much. In the end, we had to abort copying files, and we'll now have to be a bit creative to get them.
Volunteer management. I originally set up a wiki page to allow people to sign up for talks that they wanted to help out with. Unfortunately, that didn't work so well as it did for DebConf; there were several talks that did not have enough people, resulting in the volunteers from the previous talk remaining on post and finishing their talk. Also, there were some people that signed up without contacting me, or without telling me what name they used on the wiki page, which made it somewhat harder to know who to talk to. I lost count of who had signed up to help out. On Sunday, I decided to set up an IRC channel instead, through which volunteers could communicate more directly and just ask for people to take over, instead of registering for a talk first and then hoping nobody would forget. Also, since I'm terribly bad with faces and video volunteers did not have anything that made them stand out from the rest, I couldn't just walk around until I would see someone involved with video work and ask them to take over in a room. Having a separate 'video team' t-shirt could help there.

After the bad news, here's the list of things that did go well:

Streaming. Even though choosing flumotion rather than icecast (as we'd done for streaming in the past) involved extra work for me before the conference (I had to write some code so that I could stream from dvswitch into flumotion, which wasn't possible out of the box), the guys at Flumotion decided to send over one of their support engineers. As such, I simply did not have to worry about streaming servers crashing or failing to do what they were supposed to do. While I have no reason to think any of the Flumotion servers were having issues, even if they were I wouldn't have known about it, since Francesc took care of it all and never ever required my involvement about something on the flumotion side. This was extremely valuable.
Training right before the welcome talk. Explaining to people in a few words how dvswitch works, and then immediately following that up with a live demonstration, isn't a bad way for them to understand.
Despite all the problems we did have, once everything had been set up and all the gear was more or less manned, actually recording and streaming did go fairly well. The feedback I received so far was mostly positive, which makes me very happy.

In the end, while there certainly is still room for improvement and things did not go as smoothly as I'd hoped, they have gone much better than my worst fears. There's still some work to do, which I'll be doing with the NamurLUG people this week, but all in all, it's gone pretty good.

As to the usage of the streams, I asked Flumotion for some statistics. Since I was fairly late in asking, they could only provide me with statistics about sunday:

Number of clients connecting:
Traffic:

Games Screenshot Party

2012-02-07T10:36:00+00:00

The following announce is lazily copied from Paul Wise's announce. There is only one thing I like to add: the screenshots that are submitted and collected on screenshots.debian.net are visible on the packages websites (both Debian and Ubuntu) and are also used by the software-center package, so they help people to get a first impression of the package they might want to install.

Have you ever wondered how to start getting involved in Debian/Ubuntu? Do you enjoy discovering new games and playing them? You might want to come to the games screenshot party! We hope that the party will be a fun, easy, low-commitment way to get involved.

The Debian/Ubuntu Games Team is organizing a half-day screenshots party on the weekend of 25th-26th February for creating screenshots for all the games that are available in Debian/Ubuntu.

If you are interested in attending, please add your availability to the poll linked from the announcement so that we can get some idea of attendance and when is a good time for the people who are interested.

Look forward to lots of game playing, screenshots and merry time, hope to see you all there!

/debian | permanent link | Comments: 0 | Flattr this

Stop ACTA

2012-02-07T10:26:26+00:00

I hope by now you have heard of ACTA. In any case, here is a nice 6:30 minute video giving a good overview.

Please help stop ACTA. Our freedom is at risk. Whether you tell people about it, write about it, use services like Twitter to tell the world about #StopACTA, or whether you take the time to march against what corporate entities are lobbying politicians to do against their people — please help protect the Internet as we know it.

NP: God is an Astronaut: Moment of Stillness

Dpkg with multiarch support available in Debian experimental

2012-02-07T08:29:23+00:00

As I announced on debian-devel, Guillem Jover uploaded a snapshot of dpkg’s multiarch branch to experimental (version 1.16.2~wipmultiarch). Beware: There will
likely be some small “interface” changes between this version and the version that will be released later in unstable (possibly in the output of dpkg --get-selections, dpkg --list, maybe other commands).

multiarch allows you to install packages from different architectures on the same machine. This can be useful if your computer can run programs from 2 architectures (eg. x86 CPU supporting i386 and amd64), or if you often need to cross-compile software and thus need the libraries of your target architecture.

Test dpkg with multiarch support

If you want to test multiarch support in dpkg, install the package from experimental (apt-get install dpkg/experimental assuming you have experimental in your sources.list).

Then you can add a supplementary architecture to your system by doing sudo dpkg --add-architecture <arch> (e.g. i386 if you are on amd64, and vice-versa). APT will automatically pick up the new architecture and start downloading the Packages file for the new architecture (it uses dpkg --print-foreign-architectures to know about them).

From there on you can install packages from the “foreign” architectures with “apt-get install foo:<arch>“. Many packages will not be installable because some of their dependencies have not yet been updated to work with in a multiarch world (libraries must be installed in a multiarch-compliant path so as to be co-installable, and then marked “Multi-Arch: same“). Other dependencies might need to be marked “Multi-Arch: foreign“. See wiki.debian.org/Multiarch/Implementation for more HOWTO-like explanations.

Now is a good time to see if you can install the foreign packages that you could need in such a setup and to help to convert the required libraries.

You can also read Cyril Brulebois’ article which quickly shows how to hunt for the problematic packages which have not been converted to multiarch (in his sample, “ucf” is not ready. Since it’s an “Architecture: all” package which can run on any architecture, it means that it’s lacking a “Multi-Arch: foreign” field).

Report bugs

If you discover any bug in dpkg’s multiarch implementation, please report it to the Bug Tracking System (against “dpkg” with the version “1.16.2~wipmultiarch”).

If you notice important libraries or packages which are not yet multiarch ready, please open wishlist bug reports requesting the conversion and point the maintainers towards the wiki page linked above. Even better, prepare patches and submit those with your bug reports.

Again, you can follow the lead of Cyril Brulebois who filed 6 bugs!

Review the multiarch implementation

If you’re a C programmer and have some good knowledge of dpkg (or are willing to learn more of it), we would certainly benefit from more eyes reviewing the multiarch branch. If you want to discuss some design issues of the multiarch implementation in dpkg (or have questions related to your review), please get in touch via debian-dpkg@lists.debian.org.

The latest version of the branch is pu/multiarch/master in Guillem’s personal repository. I have my own version of the branch (pu/multiarch/full) which is usually a snapshot of Guillem’s branch with my own submitted fixes.

$ git clone git://git.debian.org/dpkg/dpkg.git
$ cd dpkg
$ git remote add guillem git://git.hadrons.org/git/debian/dpkg/dpkg.git
$ git remote add buxy git://git.debian.org/~hertzog/dpkg.git
$ git fetch guillem && git fetch buxy

If you followed the instructions above, the relevant branches are thus guillem/pu/multiarch/master and buxy/pu/multiarch/full. Both branches are regularly rebased on top of master where Guillem merges progressively the commits from the multi-arch branch as his review progresses.

Thank you in advance for your help bringing multiarch in shape for Debian Wheezy,

6 comments | Liked this article? Click here. | My blog is Flattr-enabled.

5 Principles of Backup Software

2012-02-07T07:26:04+00:00

Everyone agrees that backups are generally a good thing. But it seems that there is a lot less agreement about how backups should work. Here is a list of 5 principles of backup software that seem to get ignored most of the time:

(1/5) Backups should not be Application Specific

It’s quite reasonable for people to want to extract data from a backup on a different platform. Maybe someone will want to extract data a few decades after the platform becomes obsolete. I believe that vendors of backup software have an ethical obligation to make it possible for customers to get their data out with minimal effort regardless of the circumstances.

Often when writing a backup application there will be good reasons for not using the existing formats for data storage (tar, cpio, zip, etc). But ideally any data store which involves something conceptually similar to a collection of files in one larger file will use one of those formats. There have been backward compatible extensions to tar and zip for SE Linux contexts and for OS/2 EAs – the possibility of extending archive file formats with no consequence other than warnings on extraction with an unpatched utility has been demonstrated.

For a backup which doesn’t involve source files (EG the contents of some sort of database) then it should be in a format that can be easily understood and parsed. Well designed XML is generally a reasonable option. Generally the format should involve plain text that is readable and easy to understand which is optionally compressed with a common compression utility (pkzip is a reasonable choice).

(2/5) Data Store Formats should be Published

For every data store there should be public documentation about it’s format to allow future developers to write support for it. It really isn’t difficult to release some commented header files so that people can easily determine the data structures. This includes all data stores including databases and filesystems. If I suddenly find myself with a 15yo image of a NTFS filesystem containing a proprietary database I should be able to find official header files for the version of NTFS and the database server in question so I can decode the data if it’s important enough.

When an application vendor hides the data formats it gives the risk of substantial data loss at some future time. Imposing such risk on customers to try and prevent them from migrating to a rival product is unethical.

(3/5) Backups should be forward and backward compatible

It is entirely unreasonable for a vendor to demand that all their users install the latest versions of their software. There are lots of good reasons for not upgrading which includes hardware not supporting new versions of the OS, lack of Internet access to perform the upgrade, application compatibility, and just liking the way the old version works. Even for the case of a critical security fix it should be possible to restore data without applying the fix.

For any pair of versions of software that are only separated by a few versions it should be possible to backup data from one and restore to the other. Even if the data can’t be used directly (EG a backup of AMD64 programs that is restored on an i386 system) it should still be accessible. If a new version of the software doesn’t support the ancient file formats then it should be possible for the users to get a slightly older version which talks to both the old and new versions.

Backups made on 64bit systems running the latest development version of Linux and on 10yo 32bit proprietary Unix systems are interchangeable. Admittedly Unix is really good at preserving file format compatibility, but there is no technical reason why other systems can’t do the same. Source code to cpio, tar, and gnuzip, is freely available!

Apple TimeMachine fails badly in this regard, even a slightly older version of Mac OS can’t do a restore. It is however nice that most of the TimeMachine data is a tree of files which could be just copied to another system.

(4/5) Backup Software should not be Dropped

Sony Ericsson has made me hate them even more by putting the following message on their update web site:

The Backup and Restore app will be overwritten and cannot be used to restore data. Check out Android Market for alternative apps to back up and restore your data, such as MyBackup.

So if you own a Sony Ericsson phone and it is lost, stolen, or completely destroyed and all you have is a backup made by the Sony Ericsson tool then the one thing you absolutely can’t do is to buy a new Sony Ericsson phone to restore the data.

I believe that anyone who releases backup software has an ethical obligation to support restoring to all equivalent systems. How difficult would it be to put a new free app in the Google Market that has as it’s sole purpose recovering old Sony Ericsson backups onto newer phones? It really can’t be that difficult, so even if they don’t want to waste critical ROM space by putting the feature in all new phones they can make it available to everyone who needs it. When compared to the cost of developing a new Android release for a series of phones the cost of writing such a restore program would be almost nothing.

It is simply mind-boggling that Sony Ericsson go against their own commercial interests in this regard. Surely it would make good business sense to be able to sell replacements for all the lost and broken Sony Ericsson phones, but instead customers who get burned by broken backups are given an incentive to buy a product from any other vendor.

(5/5) The greater the control over data the greater the obligation for protecting it

If you have data stored in a simple and standard manner (EG the /DCIM directory containing MP4 and JPEG files that is on the USB accessible storage in every modern phone) then IMHO it’s quite OK to leave customers to their own devices in terms of backups. Typical users can work out that if they don’t backup their pictures then they risk losing them, and they can work out how to do it.

My Sony Ericsson phones have data stored under /data (settings for Android applications) which is apparently only accessible as root. Sony Ericsson have denied me root access which prevents me running backup programs such as Titanium Backup, therefore I believe that they have a great obligation to provide a way of making a backup of this data and restoring it on a new phone or a phone that has been updated. To just provide phone upgrade instructions which tell me that my phone will be entirely wiped and that I should search the App Market for backup programs is unacceptable.

I believe that there are two ethical options available to Sony Ericsson at this time, one is to make it easy to root phones so that Titanium Backup and similar programs can be used, and the other option is to release a suitable backup program for older phones. Based on experience I don’t expect Sony Ericsson to choose either option.

Now it is also a bad thing for the Android application developers to make it difficult or impossible to backup their data. For example the Wiki for one Android game gives instructions for moving the saved game files to a new phone which starts with “root your phone”. The developers of that game should have read the Wiki, realised that rooting a phone for the mundane task of transferring saved game files is totally unreasonable, and developed a better alternative.

The best thing for developers to do is to allow the users to access their own data in the most convenient manner. Then it becomes the user’s responsibility to manage it and they can concentrate on improving their application.

Why Freedom is Important

Installing CyanogenMod on my Galaxy S was painful, but having root access so I can do anything I want is a great benefit. If phone vendors would do the right thing then I could recommend that other people use the vendor release, but it seems that vendors can be expected to act unethically. So I can’t recommend that anyone use an un-modded Android phone at any time. I also can’t recommend ever buying a Sony Ericsson product, not even when it’s really cheap.

Google have done a great thing with their Data Liberation Front [1]. Not only are they providing access to the data they store on our behalf (which is a good thing) but they have a mission statement that demands the same behavior from other companies – they make it an issue of competitive advantage! So while Sony Ericsson and other companies might not see a benefit in making people like me stop hating them, failing to be as effective in marketing as Google is a real issue. Data Liberation is something that should be discussed at board elections of IT companies.

Keep in mind the fact that ethics are not just about doing nice things, they are about establishing expectations of conduct that will be used by people who deal with you in future. Sony Ericsson has shown that I should expect that they will treat the integrity of my data with contempt and I will keep this in mind every time I decline an opportunity to purchase their products. Google has shown that they consider the protection of my data as an important issue and therefore I can be confident when using and recommending their services that I won’t get stuck with data that is locked away.

While Google has demonstrated that corporations can do the right thing, the vast majority of evidence suggests that we should never trust a corporation with anything that we might want to retrieve when it’s not immediately profitable for the corporation. Therefore avoiding commercial services for storing important data is the sensible thing to do.

[1] http://www.dataliberation.org/

Galaxy S vs Xperia X10 and Android Network Access Galaxy S Review I’ve just been given an indefinite loan...
Standardising Android Don Marti wrote an amusing post about the lack of...
On Burning Platforms Nokia is in the news for it’s CEO announcing that...

Planet Debian

Mini-sendmail... in bash

Quasi-Private Resources

eSpeak GUI 0.4 is out!

phpMyAdmin translations status

Debian Wheezy: US$19 Billion. Your price… FREE!

Zooming in on the Linux “Kernel”

Individual Projects

Debian Wheezy by Programming Language

Other Tools and Comparisons

Summary

Mexico City Metro project

Cooling a Thinkpad

Dear fellows of the SJCARS

Branching FreeGLUT SVN with Git

Playing with the Trim-Slice

Some things you may have heard about Secure Boot which aren't entirely true

Secure Boot provides no additional security

Secure Boot is just another name for Trusted Boot

Microsoft are just requiring that vendors implement the specification

Secure Boot can be used to implement DRM

Secure Boot provides physical security

Secure Boot only defines the interaction between the firmware and the bootloader. It doesn't specify any higher policy

Only machines that want to boot Windows need to carry Microsoft's keys

This is only a problem for clients, not servers

Secure Boot is required by NIST

It's easy for Linux to implement Secure Boot

It's only a problem for hobbyist Linux, not the real Linux market

Conclusion

I’ve been away from Windows for more than a decade but …

perf stats for doing nothing

Perf stats for "doing nothing"

Test setup

Results

Compiled languages

Shells

Text processing

Interpreted languages

Java

Comparative analysis

Summary

RC bugs 2012/05-06

Multi-Arch progress

FACTA

2012 update 9 for Debian Installer localization

XMPP for Children

Express Sessions with connect-redis + Socket.IO in CoffeeScript

On arranging meetings

multiarch (and) hacks

mutt-notmuch 0.2

It hurts…

Useful changelog…

Magic entries for BTRFS and Software RAID

Shell Scripts For Preschoolers

Livin’ on the edge

Rant: Your custom rounding code is wrong

kvm and product_uuid

People behind Debian: Ana Beatriz Guerrero López, member of the Debian KDE team

Dpkg: edit debian/control description without worrying about leading ‘.’ and ‘ ‘

Release Critical Bug report for Week 06

Book about phpMyAdmin 3.4

Starting with BTRFS

NIST::NVD::Store::SQLite3

Daffodils

Comments with OpenID

2012-02-09: Thursday

Is GPL usage really declining?

Debian packaging for beginners @ FOSDEM

Recipes for extending Net-SNMP

SNMP in a nutshell

Extending Net-SNMP

With arbitrary commands

With pass-through scripts

With AgentX protocol

Perl and Python

C with “new” API

C with “traditional” API

Other solutions

ColorHug with non English locales

Can aptitude show older versions of packages if available in /var/cache/apt/archives?