January 21, 2020

hackergotchi for Keith Packard

Keith Packard


Linux.conf.au 2020

I just got back from linux.conf.au 2020 on Saturday and am still adjusting to being home again. I had the opportunity to give three presentations during the conference and wanted to provide links to the slides and videos.


My first presentation was part of the Open ISA miniconf on Monday. I summarized the work I've been doing on a fork of Newlib called Picolibc which targets 32- and 64- bit embedded processors.


Wednesday morning, I presented on my snek language, which is a small Python designed for introducing programming in an embedded environment. I've been using this for the last year or more in a middle-school environment (grades 5-7) as a part of a LEGO robotics class.

X History and Politics

Bradley Kuhn has been encouraging me to talk about the early politics of X and how that has shaped my views on the benefits of copyleft licenses in building strong communities, especially in driving corporate cooperation and collaboration. I would have loved to also give this talk as a part of the Copyleft Conference being held in Brussels after FOSDEM, but I won't be at that event. This talk spans the early years of X, covering events up through 1992 or so.

21 January, 2020 11:02PM

hackergotchi for Chris Lamb

Chris Lamb

Tour d'Orwell: Sutton Courtenay

(Previously in George Orwell-themed travel posts: Marrakesh, Hampstead, Paris, Southwold & Ipswich.)

George Orwell spent the last chapter of his life at the University College Hospital in London. Despite being gravely ill, arrangements were underway for him to travel to Switzerland for treatment. He had clearly not surrendered all hope as he had acquired his own fishing rod as well as some "proper" English tea to accompany him on the trip, although this is more likely to be a rare failure of facing unpleasant facts. In the end, he died in the early hours of 21st January 1950 from complications resulting from his chronic pneumonia.

He was buried in Sutton Courtenay, a small village approximately ten miles south of Oxford. Orwell had no personal connection to the village, but a lifelong love of the countryside must have encouraged a collaboration between David Astor (a longtime editor of The Observer newspaper) and Malcolm Muggeridge, better known today for introducing Mother Teresa to an international audience and some inexpensive commentary on Monty Python's The Life Of Brian.

I was expecting a few more fellow travellers to be there seventy years to the day of his death but in recompense I had a frosty yet beautifully quiet churchyard to myself. The surrounding Thames had swollen its banks onto the floodplain and the winter sunset a few hours later had a quality all its own.

21 January, 2020 10:44PM

Michael Stapelberg

distri: 20x faster initramfs (initrd) from scratch

In case you are not yet familiar with why an initramfs (or initrd, or initial ramdisk) is typically used when starting Linux, let me quote the wikipedia definition:

“[…] initrd is a scheme for loading a temporary root file system into memory, which may be used as part of the Linux startup process […] to make preparations before the real root file system can be mounted.”

Many Linux distributions do not compile all file system drivers into the kernel, but instead load them on-demand from an initramfs, which saves memory.

Another common scenario, in which an initramfs is required, is full-disk encryption: the disk must be unlocked from userspace, but since userspace is encrypted, an initramfs is used.


Thus far, building a distri disk image was quite slow:

This is on an AMD Ryzen 3900X 12-core processor (2019):

distri % time make cryptimage serial=1
80.29s user 13.56s system 186% cpu 50.419 total # 19s image, 31s initrd

Of these 50 seconds, dracut’s initramfs generation accounts for 31 seconds (62%)!

Initramfs generation time drops to 8.7 seconds once dracut no longer needs to use the single-threaded gzip(1) , but the multi-threaded replacement pigz(1) :

This brings the total time to build a distri disk image down to:

distri % time make cryptimage serial=1
76.85s user 13.23s system 327% cpu 27.509 total # 19s image, 8.7s initrd

Clearly, when you use dracut on any modern computer, you should make pigz available. dracut should fail to compile unless one explicitly opts into the known-slower gzip. For more thoughts on optional dependencies, see “Optional dependencies don’t work”.

But why does it take 8.7 seconds still? Can we go faster?

The answer is Yes! I recently built a distri-specific initramfs I’m calling minitrd. I wrote both big parts from scratch:

  1. the initramfs generator program (distri initrd)
  2. a custom Go userland (cmd/minitrd), running as /init in the initramfs.

minitrd generates the initramfs image in ≈400ms, bringing the total time down to:

distri % time make cryptimage serial=1
50.09s user 8.80s system 314% cpu 18.739 total # 18s image, 400ms initrd

(The remaining time is spent in preparing the file system, then installing and configuring the distri system, i.e. preparing a disk image you can run on real hardware.)

How can minitrd be 20 times faster than dracut?

dracut is mainly written in shell, with a C helper program. It drives the generation process by spawning lots of external dependencies (e.g. ldd or the dracut-install helper program). I assume that the combination of using an interpreted language (shell) that spawns lots of processes and precludes a concurrent architecture is to blame for the poor performance.

minitrd is written in Go, with speed as a goal. It leverages concurrency and uses no external dependencies; everything happens within a single process (but with enough threads to saturate modern hardware).

Measuring early boot time using qemu, I measured the dracut-generated initramfs taking 588ms to display the full disk encryption passphrase prompt, whereas minitrd took only 195ms.

The rest of this article dives deeper into how minitrd works.

What does an initramfs do?

Ultimately, the job of an initramfs is to make the root file system available and continue booting the system from there. Depending on the system setup, this involves the following 5 steps:

1. Load kernel modules to access the block devices with the root file system

Depending on the system, the block devices with the root file system might already be present when the initramfs runs, or some kernel modules might need to be loaded first. On my Dell XPS 9360 laptop, the NVMe system disk is already present when the initramfs starts, whereas in qemu, we need to load the virtio_pci module, followed by the virtio_scsi module.

How will our userland program know which kernel modules to load? Linux kernel modules declare patterns for their supported hardware as an alias, e.g.:

initrd# grep virtio_pci lib/modules/5.4.6/modules.alias
alias pci:v00001AF4d*sv*sd*bc*sc*i* virtio_pci

Devices in sysfs have a modalias file whose content can be matched against these declarations to identify the module to load:

initrd# cat /sys/devices/pci0000:00/*/modalias

Hence, for the initial round of module loading, it is sufficient to locate all modalias files within sysfs and load the responsible modules.

Loading a kernel module can result in new devices appearing. When that happens, the kernel sends a uevent, which the uevent consumer in userspace receives via a netlink socket. Typically, this consumer is udev(7) , but in our case, it’s minitrd.

For each uevent messages that comes with a MODALIAS variable, minitrd will load the relevant kernel module(s).

When loading a kernel module, its dependencies need to be loaded first. Dependency information is stored in the modules.dep file in a Makefile-like syntax:

initrd# grep virtio_pci lib/modules/5.4.6/modules.dep
kernel/drivers/virtio/virtio_pci.ko: kernel/drivers/virtio/virtio_ring.ko kernel/drivers/virtio/virtio.ko

To load a module, we can open its file and then call the Linux-specific finit_module(2) system call. Some modules are expected to return an error code, e.g. ENODEV or ENOENT when some hardware device is not actually present.

Side note: next to the textual versions, there are also binary versions of the modules.alias and modules.dep files. Presumably, those can be queried more quickly, but for simplicitly, I have not (yet?) implemented support in minitrd.

2. Console settings: font, keyboard layout

Setting a legible font is necessary for hi-dpi displays. On my Dell XPS 9360 (3200 x 1800 QHD+ display), the following works well:

initrd# setfont latarcyrheb-sun32

Setting the user’s keyboard layout is necessary for entering the LUKS full-disk encryption passphrase in their preferred keyboard layout. I use the NEO layout:

initrd# loadkeys neo

3. Block device identification

In the Linux kernel, block device enumeration order is not necessarily the same on each boot. Even if it was deterministic, device order could still be changed when users modify their computer’s device topology (e.g. connect a new disk to a formerly unused port).

Hence, it is good style to refer to disks and their partitions with stable identifiers. This also applies to boot loader configuration, and so most distributions will set a kernel parameter such as root=UUID=1fa04de7-30a9-4183-93e9-1b0061567121.

Identifying the block device or partition with the specified UUID is the initramfs’s job.

Depending on what the device contains, the UUID comes from a different place. For example, ext4 file systems have a UUID field in their file system superblock, whereas LUKS volumes have a UUID in their LUKS header.

Canonically, probing a device to extract the UUID is done by libblkid from the util-linux package, but the logic can easily be re-implemented in other languages and changes rarely. minitrd comes with its own implementation to avoid cgo or running the blkid(8) program.

4. LUKS full-disk encryption unlocking (only on encrypted systems)

Unlocking a LUKS-encrypted volume is done in userspace. The kernel handles the crypto, but reading the metadata, obtaining the passphrase (or e.g. key material from a file) and setting up the device mapper table entries are done in user space.

initrd# modprobe algif_skcipher
initrd# cryptsetup luksOpen /dev/sda4 cryptroot1

After the user entered their passphrase, the root file system can be mounted:

initrd# mount /dev/dm-0 /mnt

5. Continuing the boot process (switch_root)

Now that everything is set up, we need to pass execution to the init program on the root file system with a careful sequence of chdir(2) , mount(2) , chroot(2) , chdir(2) and execve(2) system calls that is explained in this busybox switch_root comment.

initrd# mount -t devtmpfs dev /mnt/dev
initrd# exec switch_root -c /dev/console /mnt /init

To conserve RAM, the files in the temporary file system to which the initramfs archive is extracted are typically deleted.

How is an initramfs generated?

An initramfs “image” (more accurately: archive) is a compressed cpio archive. Typically, gzip compression is used, but the kernel supports a bunch of different algorithms and distributions such as Ubuntu are switching to lz4.

Generators typically prepare a temporary directory and feed it to the cpio(1) program. In minitrd, we read the files into memory and generate the cpio archive using the go-cpio package. We use the pgzip package for parallel gzip compression.

The following files need to go into the cpio archive:

minitrd Go userland

The minitrd binary is copied into the cpio archive as /init and will be run by the kernel after extracting the archive.

Like the rest of distri, minitrd is built statically without cgo, which means it can be copied as-is into the cpio archive.

Linux kernel modules

Aside from the modules.alias and modules.dep metadata files, the kernel modules themselves reside in e.g. /lib/modules/5.4.6/kernel and need to be copied into the cpio archive.

Copying all modules results in a ≈80 MiB archive, so it is common to only copy modules that are relevant to the initramfs’s features. This reduces archive size to ≈24 MiB.

The filtering relies on hard-coded patterns and module names. For example, disk encryption related modules are all kernel modules underneath kernel/crypto, plus kernel/drivers/md/dm-crypt.ko.

When generating a host-only initramfs (works on precisely the computer that generated it), some initramfs generators look at the currently loaded modules and just copy those.

Console Fonts and Keymaps

The kbd package’s setfont(8) and loadkeys(1) programs load console fonts and keymaps from /usr/share/consolefonts and /usr/share/keymaps, respectively.

Hence, these directories need to be copied into the cpio archive. Depending on whether the initramfs should be generic (work on many computers) or host-only (works on precisely the computer/settings that generated it), the entire directories are copied, or only the required font/keymap.

cryptsetup, setfont, loadkeys

These programs are (currently) required because minitrd does not implement their functionality.

As they are dynamically linked, not only the programs themselves need to be copied, but also the ELF dynamic linking loader (path stored in the .interp ELF section) and any ELF library dependencies.

For example, cryptsetup in distri declares the ELF interpreter /ro/glibc-amd64-2.27-3/out/lib/ld-linux-x86-64.so.2 and declares dependencies on shared libraries libcryptsetup.so.12, libblkid.so.1 and others. Luckily, in distri, packages contain a lib subdirectory containing symbolic links to the resolved shared library paths (hermetic packaging), so it is sufficient to mirror the lib directory into the cpio archive, recursing into shared library dependencies of shared libraries.

cryptsetup also requires the GCC runtime library libgcc_s.so.1 to be present at runtime, and will abort with an error message about not being able to call pthread_cancel(3) if it is unavailable.

time zone data

To print log messages in the correct time zone, we copy /etc/localtime from the host into the cpio archive.

minitrd outside of distri?

I currently have no desire to make minitrd available outside of distri. While the technical challenges (such as extending the generator to not rely on distri’s hermetic packages) are surmountable, I don’t want to support people’s initramfs remotely.

Also, I think that people’s efforts should in general be spent on rallying behind dracut and making it work faster, thereby benefiting all Linux distributions that use dracut (increasingly more). With minitrd, I have demonstrated that significant speed-ups are achievable.


It was interesting to dive into how an initramfs really works. I had been working with the concept for many years, from small tasks such as “debug why the encrypted root file system is not unlocked” to more complicated tasks such as “set up a root file system on DRBD for a high-availability setup”. But even with that sort of experience, I didn’t know all the details, until I was forced to implement every little thing.

As I suspected going into this exercise, dracut is much slower than it needs to be. Re-implementing its generation stage in a modern language instead of shell helps a lot.

Of course, my minitrd does a bit less than dracut, but not drastically so. The overall architecture is the same.

I hope my effort helps with two things:

  1. As a teaching implementation: instead of wading through the various components that make up a modern initramfs (udev, systemd, various shell scripts, …), people can learn about how an initramfs works in a single place.

  2. I hope the significant time difference motivates people to improve dracut.

Appendix: qemu development environment

Before writing any Go code, I did some manual prototyping. Learning how other people prototype is often immensely useful to me, so I’m sharing my notes here.

First, I copied all kernel modules and a statically built busybox binary:

% mkdir -p lib/modules/5.4.6
% cp -Lr /ro/lib/modules/5.4.6/* lib/modules/5.4.6/
% cp ~/busybox-1.22.0-amd64/busybox sh

To generate an initramfs from the current directory, I used:

% find . | cpio -o -H newc | pigz > /tmp/initrd

In distri’s Makefile, I append these flags to the QEMU invocation:

-kernel /tmp/kernel \
-initrd /tmp/initrd \
-append "root=/dev/mapper/cryptroot1 rdinit=/sh ro console=ttyS0,115200 rd.luks=1 rd.luks.uuid=63051f8a-54b9-4996-b94f-3cf105af2900 rd.luks.name=63051f8a-54b9-4996-b94f-3cf105af2900=cryptroot1 rd.vconsole.keymap=neo rd.vconsole.font=latarcyrheb-sun32 init=/init systemd.setenv=PATH=/bin rw vga=836"

The vga= mode parameter is required for loading font latarcyrheb-sun32.

Once in the busybox shell, I manually prepared the required mount points and kernel modules:

ln -s sh mount
ln -s sh lsmod
mkdir /proc /sys /run /mnt
mount -t proc proc /proc
mount -t sysfs sys /sys
mount -t devtmpfs dev /dev
modprobe virtio_pci
modprobe virtio_scsi

As a next step, I copied cryptsetup and dependencies into the initramfs directory:

% for f in /ro/cryptsetup-amd64-2.0.4-6/lib/*; do full=$(readlink -f $f); rel=$(echo $full | sed 's,^/,,g'); mkdir -p $(dirname $rel); install $full $rel; done
% ln -s ld-2.27.so ro/glibc-amd64-2.27-3/out/lib/ld-linux-x86-64.so.2
% cp /ro/glibc-amd64-2.27-3/out/lib/ld-2.27.so ro/glibc-amd64-2.27-3/out/lib/ld-2.27.so
% cp -r /ro/cryptsetup-amd64-2.0.4-6/lib ro/cryptsetup-amd64-2.0.4-6/
% mkdir -p ro/gcc-libs-amd64-8.2.0-3/out/lib64/
% cp /ro/gcc-libs-amd64-8.2.0-3/out/lib64/libgcc_s.so.1 ro/gcc-libs-amd64-8.2.0-3/out/lib64/libgcc_s.so.1
% ln -s /ro/gcc-libs-amd64-8.2.0-3/out/lib64/libgcc_s.so.1 ro/cryptsetup-amd64-2.0.4-6/lib
% cp -r /ro/lvm2-amd64-2.03.00-6/lib ro/lvm2-amd64-2.03.00-6/

In busybox, I used the following commands to unlock the root file system:

modprobe algif_skcipher
./cryptsetup luksOpen /dev/sda4 cryptroot1
mount /dev/dm-0 /mnt

21 January, 2020 05:19PM

January 20, 2020

hackergotchi for Jonathan Dowland

Jonathan Dowland

Self-hosted web fonts

Today on Lobsters I found a link to Kev Quirk's blog post How to self-host your web fonts. For the last nine years I've been using Google's font-hosting service, which whilst very convenient, carried some privacy concerns (which Joey Hess originally brought to my attention) and (it turns out) does not appear to have been faster, in network terms, than bundling what I was using locally. This is something I've been meaning to get around to doing for almost that long.

20 January, 2020 03:08PM

hackergotchi for Dirk Eddelbuettel

Dirk Eddelbuettel

anytime 0.3.7

A fresh minor release of the anytime package is arriving on CRAN right now. This is the eighteenth release, and it comes roughly five months after the previous showing the relative feature-stability we have now.

anytime is a very focused package aiming to do just one thing really well: to convert anything in integer, numeric, character, factor, ordered, … format to either POSIXct or Date objects – and to do so without requiring a format string. See the anytime page, or the GitHub README.md for a few examples.

This release brings a clever new option, thanks to Stephen Froehlich. If you know your input has (lots) of duplicates you can now say so and anytime() (and the other entry points for times and dates, UTC or not) will only parse the unique entries leading to potentially rather large speed gains (as in Stephen’s case where he often has more than 95% of the data as duplicates). We also tweaked the test setup some more, but as we are still unable to replicate what is happening with the Fedora test boxen at CRAN due to the non-reproducible setup so this remains a bit of guess work. Lastly, I am making use of a new Rcpp #define to speed up compilation a little bit too.

The full list of changes follows.

Changes in anytime version 0.3.7 (2019-01-20)

  • Test and possibly condition away one more test file.

  • Small enhancement for compilation by setting no-rtti define via Rcpp.

  • New option calcUnique for speed-up by parseing only unique timestamps (Stephen Froehlich in #110 fixing #109).

Courtesy of CRANberries, there is a comparison to the previous release. More information is on the anytime page. The issue tracker tracker off the GitHub repo can be use for questions and comments.

If you like this or other open-source work I do, you can now sponsor me at GitHub. For the first year, GitHub will match your contributions.

This post by Dirk Eddelbuettel originated on his Thinking inside the box blog. Please report excessive re-aggregation in third-party for-profit settings.

20 January, 2020 01:58PM

hackergotchi for Matthew Garrett

Matthew Garrett

Verifying your system state in a secure and private way

Most modern PCs have a Trusted Platform Module (TPM) and firmware that, together, support something called Trusted Boot. In Trusted Boot, each component in the boot chain generates a series of measurements of next component of the boot process and relevant configuration. These measurements are pushed to the TPM where they're combined with the existing values stored in a series of Platform Configuration Registers (PCRs) in such a way that the final PCR value depends on both the value and the order of the measurements it's given. If any measurements change, the final PCR value changes.

Windows takes advantage of this with its Bitlocker disk encryption technology. The disk encryption key is stored in the TPM along with a policy that tells it to release it only if a specific set of PCR values is correct. By default, the TPM will release the encryption key automatically if the PCR values match and the system will just transparently boot. If someone tampers with the boot process or configuration, the PCR values will no longer match and boot will halt to allow the user to provide the disk key in some other way.

Unfortunately the TPM keeps no record of how it got to a specific state. If the PCR values don't match, that's all we know - the TPM is unable to tell us what changed to result in this breakage. Fortunately, the system firmware maintains an event log as we go along. Each measurement that's pushed to the TPM is accompanied by a new entry in the event log, containing not only the hash that was pushed to the TPM but also metadata that tells us what was measured and why. Since the algorithm the TPM uses to calculate the hash values is known, we can replay the same values from the event log and verify that we end up with the same final value that's in the TPM. We can then examine the event log to see what changed.

Unfortunately, the event log is stored in unprotected system RAM. In order to be able to trust it we need to compare the values in the event log (which can be tampered with) with the values in the TPM (which are much harder to tamper with). Unfortunately if someone has tampered with the event log then they could also have tampered with the bits of the OS that are doing that comparison. Put simply, if the machine is in a potentially untrustworthy state, we can't trust that machine to tell us anything about itself.

This is solved using a procedure called Remote Attestation. The TPM can be asked to provide a digital signature of the PCR values, and this can be passed to a remote system along with the event log. That remote system can then examine the event log, make sure it corresponds to the signed PCR values and make a security decision based on the contents of the event log rather than just on the final PCR values. This makes the system significantly more flexible and aids diagnostics. Unfortunately, it also means you need a remote server and an internet connection and then some way for that remote server to tell you whether it thinks your system is trustworthy and also you need some way to believe that the remote server is trustworthy and all of this is well not ideal if you're not an enterprise.

Last week I gave a talk at linux.conf.au on one way around this. Basically, remote attestation places no constraints on the network protocol in use - while the implementations that exist all do this over IP, there's no requirement for them to do so. So I wrote an implementation that runs over Bluetooth, in theory allowing you to use your phone to serve as the remote agent. If you trust your phone, you can use it as a tool for determining if you should trust your laptop.

I've pushed some code that demos this. The current implementation does nothing other than tell you whether UEFI Secure Boot was enabled or not, and it's also not currently running on a phone. The phone bit of this is pretty straightforward to fix, but the rest is somewhat harder.

The big issue we face is that we frequently don't know what event log values we should be seeing. The first few values are produced by the system firmware and there's no standardised way to publish the expected values. The Linux Vendor Firmware Service has support for publishing these values, so for some systems we can get hold of this. But then you get to measurements of your bootloader and kernel, and those change every time you do an update. Ideally we'd have tooling for Linux distributions to publish known good values for each package version and for that to be common across distributions. This would allow tools to download metadata and verify that measurements correspond to legitimate builds from the distribution in question.

This does still leave the problem of the initramfs. Since initramfs files are usually generated locally, and depend on the locally installed versions of tools at the point they're built, we end up with no good way to precalculate those values. I proposed a possible solution to this a while back, but have done absolutely nothing to help make that happen. I suck. The right way to do this may actually just be to turn initramfs images into pre-built artifacts and figure out the config at runtime (dracut actually supports a bunch of this already), so I'm going to spend a while playing with that.

If we can pull these pieces together then we can get to a place where you can boot your laptop and then, before typing any authentication details, have your phone compare each component in the boot process to expected values. Assistance in all of this extremely gratefully received.

comment count unavailable comments

20 January, 2020 12:53PM

Russ Allbery

DocKnot 3.03

DocKnot is the software that I use to generate package documentation and web pages, and increasingly to generate release tarballs.

The main change in this release is to use IO::Uncompress::Gunzip and IO::Compress::Xz to generate a missing xz tarball when needed, instead of forking external programs (which causes all sorts of portability issues). Thanks to Slaven Rezić for the testing and report.

This release adds two new badges to README.md files: a version badge for CPAN packages pushed to GitHub, and a Debian version badge for packages with a corresponding Debian package.

This release also makes the tarball checking done as part of the release process (to ensure all files are properly included in the release) a bit more flexible by adding a distribution/ignore metadata setting containing a list of regular expressions matching files to ignore for checking purposes.

Finally, this release fixes a bug that leaked $@ modifications to the caller of App::DocKnot::Config.

You can get the latest version from the DocKnot distribution page.

20 January, 2020 03:45AM

January 19, 2020

Enrico Zini

hackergotchi for Dirk Eddelbuettel

Dirk Eddelbuettel

RPushbullet 0.3.3

RPpushbullet demo

Release 0.3.3 of the RPushbullet package just got to CRAN. RPushbullet offers an interface to the neat Pushbullet service for inter-device messaging, communication, and more. It lets you easily send (programmatic) alerts like the one to the left to your browser, phone, tablet, … – or all at once.

This release further robustifies operations via two contributed PRs. The first by Chan-Yub ensures we set UTF-8 encoding on pushes. The second by Alexandre permits to downgrade from http/2 to http/1.1 which he needed for some operations with a particular backend. I made that PR a bit more general by turning the downgrade into one driven by a new options() toggle. Special thanks also to Jeroen in help debugging this issue. See below for more details.

Changes in version 0.3.3 (2020-01-18)

  • UTF-8 encoding is now used (Chan-Yub Park in #55).

  • Curl can use HTTP/1.1 (Alexandre Shannon in #59 fixing #57, plus Dirk in #60 making it optional).

Courtesy of CRANberries, there is also a diffstat report for this release. More details about the package are at the RPushbullet webpage and the RPushbullet GitHub repo.

If you like this or other open-source work I do, you can now sponsor me at GitHub. For the first year, GitHub will match your contributions.

This post by Dirk Eddelbuettel originated on his Thinking inside the box blog. Please report excessive re-aggregation in third-party for-profit settings.

19 January, 2020 06:14PM

January 18, 2020

Ingo Juergensmann

XMPP - Fun with Clients

As I already wrote in my last blog post there's much development in XMPP, not only on the server side, but also on the client side. It's surely not exaggerated to say that Conversations on Android is the de-facto standard client-wise. So, if you have an Android phone, that's the client you want to try&use. As I don't have Android, I can't comment on it. The situation on Linux is good as well: there are such clients as Gajim, which is an old player in the "market" and is available on other platforms as well, but there is with Dino a new/modern client as well that you may want to try out.

The situation for macOS and iOS users are not that good as for Windows, Linux or Android users. But in the end all clients have their pro and cons... I'll try to summarize a few clients on Linux, macOS and iOS...


Fully featured multiprotol client with lots of available plugins. If you want to use OMEMO with Gajim you need to enable it in your plugin settings. There is even a plugin for letting the keyboard LED blink when there are new/unread messages. I found that a little bit annoying, so I disabled that. Gajim is an old-style client with the well-known layout of a contact list window and one for the actual chats. Gajim has some nice features like service discovery on your or remote servers.


Dino is available in Debian as dino-im and is a quite new client, which you will find out at first start: it's a single window app where the focus is on the chats. There is no contact list at first glance where you can see whether or not your contacts are online. You can find your contacts when you want to start a conversation with your contact. I don't find this bad or good. It's just different and puts the chat into focus, as said, maybe similar to WhatApp, Signal or other messengers nowadays where you just sent messages back and forth and don't care if the contact is online. The contact will receive and read your message when the contact is online and will eventually answer your message.


Monal is an actively developed and maintained client for macOS and iOS. If you want to try out Monal, don't waste your time with the older client, but focus on Monal Catalyst (direct download link). Catalyst shares the same code as the iOS version of Monal and will become the default Monal download within the next few weeks. It's far easier for the developers to focus on one codebase than on two different ones. Monal has great potential, but also has some issues. For some reason it seems that some messages from time to time will be sent multiple times to a contact or a MUC. The developers are very helpful and supportive. So when you find a bug or issue, please report back. 


BeagleIM is a free XMPP client by Tigase, which business is to sell their XMPP Communication suite and professional support for it. They provide clients for Android, macOS and iOS. Maybe for that reason their clients seems to be very mature, but of course will work best with their own server software. That doesn't mean that the clients won't work well with other 3rd party XMPP servers, just that their main focus will be their own server software, However, BeagleIM seems to work well with Prosody and ejabberd and when you have issues you can also reach out to Tigase on Mastodon, which I find a very big plus! They are really helpful there as well. The BeagleIM client is currently my main client on macOS and it works quite well. As you can see it's more or less chat-focused as well by default, but you can open a contact list window if you want to see all of available/all clients. Only issue I personally have at the moment is, that it seems to have problems with ejabberd: in the contact list and account preferences I see the accounts/contacts with ejabberd going offline/online every few minutes. There are some log entries in ejabberd that seem to be timeout related. I'm not sure whether this is an issue with ejabberd or with BeagleIM - or a rare combination of both.


ChatSecure is one of my first XMPP clients on iOS I installed and used for a long time. It works mostly very well, supports OMEMO (like all the other clients I mention here) and it seems to be able to work well with bookmarks (i.e. use a list of MUCs to join). Only issues I have with ChatSecure currently are: 1) when ChatSecure comes to front after a deep sleep state of the client on iPhone it presents an empty screen and no accounts in settings. You need to quit ChatSecure and restart it to have it working again. Quite annoying. 2) when restarted it polls all messages again from MAM (message archives) over and over again. A small annoying where you can decide if it's better to have duplicated messages or may miss a message.


What was valid for Monal Catalyst on macOS is also (more or less) true for Monal on iOS. As said: they share the same code base. It is usuable, but has sometimes issues with joining MUCs: it appears as if you can't join a MUC, but suddenly you receive a message in the MUC and then it is listed under Chats and you can use it.


Siskin is the iOS client from Tigase. It also seems to be very mature, but has some special caveats: in the settings you can configure Push Notifications and HTTP Uploads for each and every clients. Other clients make this automatically and I leave it to you to decide whether this is a nice feature that you can configure it or if it is a little bit annoying, because when you don't know that, you will be wondering why you can't upload files/fotos in a chat. Maybe uploading files will work with Tigase XMPP server, but it doesn't seem to work on my servers.


So, in the end, there are good/promising clients on iOS and macOS, but every client seem to have its own pitfalls. On iOS all three clients do support and use Apple Push Notifications, but you should choose carefully for which one you want to enable it. I can tell you it's a little bit annoying to test three clients and have Push Notifications turned on for all of them and have joined in several MUCs and get all notifications three times for every message... ;-)

MUCs are a special topic I need to investigate a little more in the future: when your server supports Bookmarks for MUCs I would assume that it should be working for all supporting clients and you only need to join MUC on one client and have that MUC at least in your list of bookmarks. If you want to join that MUC on every client might be another story. But I don't know if this is the intended behaviour of the XEP in question or if my assumption how it should work is just wrong.

In the end the situation for XMPP clients on macOS and iOS is much better than it was 1-2 years ago. Though, it is not as good as on Android, but you can help improving the situation by testing the available clients and give feedback to the developers by either joining the approrpriate MUCs or - even better - file issues on their Github pages!


18 January, 2020 05:58PM by ij

Russ Allbery

Term::ANSIColor 5.01

This is the module included in Perl core that provides support for ANSI color escape sequences.

This release adds support for the NO_COLOR environment variable (thanks, Andrea Telatin) and fixes an error in the example of uncolor() in the documentation (thanks, Joe Smith). It also documents that color aliases are expanded during alias definition, so while you can define an alias in terms of another alias, they don't remain linked during future changes.

You can get the latest release from CPAN or from the Term::ANSIColor distribution page.

18 January, 2020 03:20AM

Mike Hommey

Announcing git-cinnabar 0.5.3

Git-cinnabar is a git remote helper to interact with mercurial repositories. It allows to clone, pull and push from/to mercurial remote repositories, using git.

Get it on github.

These release notes are also available on the git-cinnabar wiki.

What’s new since 0.5.2?

  • Updated git to 2.25.0 for the helper.
  • Fixed small memory leaks.
  • Combinations of remote ref styles are now allowed.
  • Added a git cinnabar unbundle command that allows to import a mercurial bundle.
  • Experimental support for python >= 3.5.
  • Fixed erroneous behavior of git cinnabar {hg2git,git2gh} with some forms of abbreviated SHA1s.
  • Fixed handling of the GIT_SSH environment variable.
  • Don’t eliminate closed tips when they are the only head of a branch.
  • Better handle manifests with double slashes created by hg convert from Mercurial < 2.0.1, and the following updates to those paths with normal Mercurial operations.
  • Fix compatibility with Mercurial libraries >= 3.0, < 3.4.
  • Windows helper is now statically linked against libcurl.

18 January, 2020 02:49AM by glandium

January 17, 2020

hackergotchi for Jonathan McDowell

Jonathan McDowell

A beginner tries PCB assembly

I wrote last year about my experience with making my first PCB using JLCPCB. I’ve now got 5 of the boards in production around my house, and another couple assembled on my desk for testing. I also did a much simpler board to mount a GPS module on my MapleBoard - basically just with a suitable DIP connector and mount point for the GPS module. At that point I ended up having to pay for shipping; not being in a hurry I went for the cheapest option which mean the total process took 2 weeks from order until it arrived. Still not bad for under $8!

Just before Christmas I discovered that JLCPCB had expanded their SMT assembly option to beyond the Chinese market, and were offering coupons off (but even without that had much, much lower assembly/setup fees than anywhere else I’d seen). Despite being part of LCSC the parts library can be a bit limited (partly it seems there’s nothing complex to assemble such as connectors), with a set of “basic” components without setup fee and then “extended” options which have a $3 setup fee (because they’re not permanently loaded, AIUI).

To test out the service I decided to revise my IoT board. First, I’ve used a few for 12V LED strip control which has meant the 3.3V LDO is working harder than ideal, so I wanted to switch (ha ha) to a buck converter. I worked back from the JLCPCB basic parts list and chose an MP2451, which had a handy data sheet with an example implementation. I also upgraded the ESP module to an ESP32-WROOM - I’ve had some issues with non-flickery PWM on the ESP8266 and the ESP32 has hardware PWM. I also have some applications the Bluetooth would be useful for. Once again I turned to KiCad to draw the schematic and lay out the board. I kept the same form factor for ease, as I knew I could get a case for it. The more complex circuitry was a bit harder to lay out in the same space, and the assembly has a limitation of being single sided which complicates things further, but the fact it was being done for me meant I could drop to 0603 parts.

All-in-all I ended up with 17 parts for the board assembly, with the ESP32 module and power connectors being my responsibility (JLCPCB only have the basic ESP32 chip and I did not feel like trying to design a PCB antenna). I managed to get everything except the inductor from the basic library, which kept costs down. Total cost for 10 boards, parts, assembly, shipping + customs fees was just under $29 which is amazing value to me. What’s even better is that the DFM (design for manufacturing) checks they did realised I’d placed the MP2451 the wrong way round and they automatically rotated it 180° for me. Phew!

The order was placed in the middle of December and arrived just before New Year - again, about 2 weeks total time end to end. Very impressive. Soldering the ESP32 module on was more fiddly than the ESP-07, but it all worked first time with both 5V + 12V power supplies, so I’m very pleased with the results.


Being able to do cheap PCB assembly is a game changer for me. There are various things I feel confident enough to design for my own use that I’d never be able to solder up myself; and for these prices it’s well worth a try. I find myself currently looking at some of the basic STM32 offerings (many of them in JLCPCB’s basic component range) and pondering building a slightly more advanced dev board around one. I’m sure my PCB design will cause those I know in the industry to shudder, but don’t worry, I’ve no plans to do this other than for my own amusement!

17 January, 2020 07:34PM

hackergotchi for Steve Kemp

Steve Kemp

Announce: github2mr

myrepos is an excellent tool for applying git operations to multiple repositories, and I use it extensively.

I've written several scripts to dump remote repository-lists into a suitable configuration format, and hopefully I've done that for the last time.

github2mr correctly handles:

  • Exporting projects from Github.com
  • Exporting projects from (self-hosted installations of) Github Enterprise.
  • Exporting projects from (self-hosted installations of) Gitbucket.

If it can handle Gogs, Gitea, etc, then I'd love to know, otherwise patches are equally welcome!

17 January, 2020 06:30PM

Russ Allbery

Review: Lent

Review: Lent, by Jo Walton

Publisher: Tor
Copyright: May 2019
ISBN: 1-4668-6572-5
Format: Kindle
Pages: 381

It is April 3rd, 1492. Brother Girolamo is a Dominican and the First Brother of San Marco in Florence. He can see and banish demons, as we find out in the first chapter when he cleanses the convent of Santa Lucia. The demons appear to be drawn by a green stone hidden in a hollowed-out copy of Pliny, a donation to the convent library from the King of Hungary. That green stone will be central to the story, but neither we nor Girolamo find out why for some time. The only hint is that the dying Lorenzo de' Medici implies that it is the stone of Titurel.

Brother Girolamo is also a prophet. He has the ability to see the future, sometimes explicitly and sometimes in symbolic terms. Sometimes the events can be changed, and sometimes they have the weight of certainty. He believes the New Cyrus will come over the Alps, leading to the sack and fall of Rome, and hopes to save Florence from the same fate by transforming it into the City of God.

If your knowledge of Italian Renaissance history is good, you may have already guessed the relevant history. The introduction of additional characters named Marsilio and Count Pico provide an additional clue before Walton mentions Brother Girolamo's last name: Savonarola.

If, like me, you haven't studied Italian history but still think this sounds vaguely familiar, that may be because Savonarola and his brief religious rule of Florence is a topic of Chapter VI of Niccolò Machiavelli's The Prince. Brother Girolamo in Walton's portrayal is not the reactionary religious fanatic he is more often shown as, but if you know this part of history, you'll find many events of the first part of the book familiar.

The rest of this book... that's where writing this review becomes difficult.

About 40% of the way through Lent, and well into spoiler territory, this becomes a very different book. Exactly how isn't something I can explain without ruining a substantial portion of the plot. That also makes it difficult to talk about what Walton is doing in this novel, and to some extent even to describe its genre. I'll try, but the result will be unsatisfyingly vague.

Lent is set in an alternate historical universe in which both theology and magic work roughly the way that 15th century Christianity thought that they worked. Demons are real, although most people can't see them. Prophecy is real in a sense, although that's a bit more complicated. When Savonarola says that Florence is besieged by demons, he means that demons are literally arrayed against the walls of the city and attempting to make their ways inside. Walton applies the concreteness of science with its discoverable rules and careful analysis to prophecy, spiritual warfare, and other aspects of theology that would be spoilers.

Using Savonarola as the sympathetic main character is a bold choice. The historical figure is normally portrayed as the sort of villain everyone, including Machiavelli, loves to hate. Walton's version of the character is still arguably a religious fanatic, but the layers behind why he is so deeply religious and what he is attempting to accomplish are deep and complex. He has a single-minded belief in a few core principles, and he's acting on the basis of prophecy that he believes completely (for more reasons than either he or the reader knows at first). But outside of those areas of uncompromising certainty, he's thoughtful and curious, befriends other thoughtful and curious people, supports philosophy, and has a deep sense of fairness and honesty. When he talks about reform of the church in Lent, he's both sincere and believable. (This would not survive a bonfire of the vanities that was a literal book burning, but Walton argues forcefully in an afterward that this popular belief contradicts accounts from primary sources.)

Lent starts as an engrossing piece of historical fiction, pulling me into the fictional thoughts of a figure I would not have expected to like nearly as much as I did. I was not at all bored by the relatively straightforward retelling of Italian history and would have happily read more of it. The shifting of gears partway through adds additional intriguing depth, and it's fun to play what-if with medieval theology and explore the implications of all of it being literally true.

The ending, unfortunately, I thought was less successful, mostly due to pacing. Story progress slows in a way that has an important effect on Savonarola, but starts to feel a touch tedious. Then, Walton makes a bit too fast of a pivot between despair and success and didn't give me quite enough emotional foundation for the resolution. She also dropped me off the end of the book more abruptly than I wanted. I'm not sure how she could have possibly continued beyond the ending, to be fair, but still, I wanted to know what would happen in the next chapter (and the theology would have been delightfully subversive). But this is also the sort of book that's exceedingly hard to end.

I would call Lent more intriguing than fully successful, but I enjoyed reading it despite not having much inherent interest in Florence, Renaissance theology, or this part of Italian history. If any of those topics attracts you more than it does me, I suspect you will find this book worth reading.

Rating: 7 out of 10

17 January, 2020 04:13AM

January 16, 2020

hackergotchi for Steve Kemp

Steve Kemp

Exporting github repositories to myrepos

myrepos is an excellent tool for applying git operations to multiple repositories, and I use it extensively.

Given a configuration file like this:


checkout = git clone git@github.com:skx/asql.git

checkout = git clone git@github.com:skx/bookmarks.public.git

checkout = git clone git@github.com:skx/Buffalo-220-NAS.git

checkout = git clone git@github.com:skx/calibre-plugins.git


You can clone all the repositories with one command:

mr -j5 --config .mrconfig.github checkout

Then pull/update them them easily:

mr -j5 --config .mrconfig.github update

It works with git repositories, mercurial, and more. (The -j5 argument means to run five jobs in parallel. Much speed, many fast. Big wow.)

I wrote a simple golang utility to use the github API to generate a suitable configuration including:

  • All your personal repositories.
  • All the repositories which belong to organizations you're a member of.

Currently it only supports github, but I'll update to include self-hosted and API-compatible services such as gitbucket. Is there any interest in such a tool? Or have you all written your own already?

(I have the feeling I've written this tool in Perl, Ruby, and even using curl a time or two already. This time I'll do it properly and publish it to save effort next time!)

16 January, 2020 07:19PM

hackergotchi for Dirk Eddelbuettel

Dirk Eddelbuettel

RcppRedis 0.1.10: Switch to tinytest

Another minor release of RcppRedis just arrived on CRAN, following a fairly long break since the last release in October 2018.

RcppRedis is one of several packages connecting R to the fabulous Redis in-memory datastructure store (and much more). RcppRedis does not pretend to be feature complete, but it may do some things faster than the other interfaces, and also offers an optional coupling with MessagePack binary (de)serialization via RcppMsgPack. The package has carried production loads for several years now.

This release switches to the fabulous tinytest package, allowing for very flexible testing during development and deployment—three cheers for easily testing installed packages too.

Changes in version 0.1.10 (2020-01-16)

  • The package now uses tinytest for unit tests (Dirk in #41).

Courtesy of CRANberries, there is also a diffstat report for this release. More information is on the RcppRedis page.

If you like this or other open-source work I do, you can now sponsor me at GitHub. For the first year, GitHub will match your contributions.

This post by Dirk Eddelbuettel originated on his Thinking inside the box blog. Please report excessive re-aggregation in third-party for-profit settings.

16 January, 2020 11:43AM

François Marier

Sharing your WiFi connection with a NetworkManager hotspot

In-flight and hotel WiFi can be quite expensive and often insist on charging users extra to connect multiple devices. In order to avoid that, it's possible to easily create a WiFi hotspot using NetworkManager and a external USB WiFi adapter.

Creating the hotspot

The main trick is to right-click on the NetworkManager icon in the status bar and select "Edit Connections..." (not "Create New WiFi Network..." despite the promising name).

From there click the "+" button in the lower right then "WiFi" as the Connection Type. I like to use the computer name as the "Connection name".

In the WiFi tab, set the following:

  • SSID: machinename_nomap
  • Mode: hotspot
  • Device: (the device name of the USB WiFi adapter)

The _nomap suffix is there to opt out of the Google and Mozilla location services which could allow anybody to lookup sightings of your device around the World.

In the WiFi Security tab:

  • Security: WPA & WPA2 Personal
  • Password: (a 63-character random password generated using pwgen -s 63)

While you may think that such a long password is inconvenient, it's now possible to add the network automatically by simply scanning a QR code on your phone.

In the IPv4 Settings tab:

  • Method: Shared to other computers

Finally, in the IPv6 Settings tab:

  • Method: Ignore

I ended up with the following config in /etc/NetworkManager/system-connections/machinename:

uuid=<long UUID string>


psk=<63-character password>



Firewall rules

In order for the packets to flow correctly, I opened up the following ports on my machine's local firewall:

-A FORWARD -d -m conntrack --ctstate RELATED,ESTABLISHED -j ACCEPT
-A INPUT -d -s -j ACCEPT
-A INPUT -d -s -j ACCEPT
-A INPUT -d -s -j ACCEPT
-A INPUT -d -s -j ACCEPT

16 January, 2020 02:15AM

January 15, 2020

Enrico Zini

Himblick one day later

This is part of a series of posts on the design and technical steps of creating Himblick, a digital signage box based on the Raspberry Pi 4.

One day after the first deploy, we went to check how the system was doing, and noticed some fine tuning to do, some pretty much urgent.


Since the system runs on a readonly rootfs with a writable tempfs overlay, one can inspect the contents of /live/cow and see exactly what files were written since the last boot. ncdu -x /live/cow is a wonderful, wonderful thing.

In this way, we can quickly identify disk/memory usage leaks, and other possible unexpected surprises, like an unexpectedly updated apt package database.

An unexpectedly updated apt package database, with apt sources that may publish broken software, raised very loud alarm bells.

Disable apt timers

It looks like Raspbian ships with the automatic apt update/upgrade timer services enabled. In our case, that would give us a system that works when turned on, then upgrades overnight, and the next day won't play videos, until rebooted, when the tmpfs overlay will be reset and it will work again, until the next nightly upgrade, and so on.

In other words, a flaky system, that would thankfully fix itself at boot but break one day after booting. A system that would be very hard to debug. A system that would soon lose the trust of its users.

The first hotfix after deployment of Himblick was then to update the provisioning procedure to disable automatic package updates:

systemctl disable apt-daily.timer
systemctl mask apt-daily.timer
systemctl disable apt-daily-upgrade.timer
systemctl mask apt-daily-upgrade.timer

Of course, the first system to be patched was on top of a very tall ladder close to a museum ceiling.

journald disk usage

Logging takes an increasing amount of space. In theory, using a systemd.volatile setup, journald does the right thing by default. Since we need to use dracut's hack instead of systemd.volatile, we need to take manual steps to bound the amount of disk space used.

Thanfully, it looks easy to fine-tune journald's disk usage

Limit the growth of .xsession-errors

The .xsession-errors file grows indefinitely during the X session, and it cannot be rotated without restarting X. Deleting it won't help, as the X session still has the file open and keeps it allocated and growing on disk. At most, it can be occasionally truncated.

The file is created by /etc/X11/Xsession before sourcing other configuration files, so one cannot override its location with, say, /dev/null, or a pipe to some command, without editing the Xsession script itself.

Still, .xsession-errors is extremely useful for finding unexpected error output from X programs when something goes wrong.

In our case, himblick-player is the only program run in the X session. We can greatly limit the growth of .xsession-errors by making it log to a file instead of stderr, and using one of python's rotating logging handlers to limit the amount of Himblick's stored logging, or send himblick's log directly to journald, and let journald take care of disk allocation.

Once that is sorted, we can change Himblick to capture the players' stdout and stderr, and log it, to avoid it going to .xsession-errors.

15 January, 2020 03:03PM

Dmitry Shachnev

Qt packages built with OpenGL ES support are now available

Some time ago, there was a thread on debian-devel where we discussed how to make Qt packages work on hardware that supports OpenGL ES, but not the desktop OpenGL.

My first proposal was to switch to OpenGL ES by default on ARM64, as that is the main affected architecture. After a lengthy discussion, it was decided to ship two versions of Qt packages instead, to support more (OpenGL variant, architecture) configurations.

So now I am announcing that we finally have the versions of Qt GUI and Qt Quick libraries that are built against OpenGL ES, and the release team helped us to rebuild the archive for compatibility with them. These packages are not co-installable together with the regular (desktop OpenGL) Qt packages, as they provide the same set of shared libraries. So most packages now have an alternative dependency like libqt5gui5 (>= 5.x) | libqt5gui5-gles (>= 5.x). Packages get such a dependency automatically if they are using ${shlibs:Depends}.

These Qt packages will be mostly needed by ARM64 users, however they may be also useful on other architectures too. Note that armel and armhf are not affected, because there Qt was built against OpenGL ES from the very beginning. So far there are no plans to make two versions of Qt on these architectures, however we are open to bug reports.

To try that on your system (running Bullseye or Sid), just run this command:

# apt install libqt5gui5-gles libqt5quick5-gles

The other Qt submodule packages do not need a second variant, because they do not use any OpenGL API directly. Most of the Qt applications are installable with these packages. At the moment, Plasma is not installable because plasma-desktop FTBFS, but that will be fixed sooner or later.

One major missing thing is PyQt5. It is linking against some Qt helper functions that only exist for desktop OpenGL build, so we will probably need to build a special version of PyQt5 for OpenGL ES.

If you want to use any OpenGL ES specific API in your package, build it against qtbase5-gles-dev package instead of qtbase5-dev. There is no qtdeclarative5-gles-dev so far, however if you need it, please let us know.

In case you have any questions, please feel free to file a bug against one of the new packages, or contact us at the pkg-kde-talk mailing list.

15 January, 2020 02:55PM by Dmitry Shachnev

hackergotchi for Dirk Eddelbuettel

Dirk Eddelbuettel

RQuantLib 0.4.11: More polish

New year, new RQuantLib! A new release 0.4.11 of RQuantLib arrived overnight on CRAN; and a Debian upload will follow shortly.

QuantLib is a very comprehensice free/open-source library for quantitative finance; RQuantLib connects it to the R environment and language.

This version does three new things. First, we fixed an oversight on our end and now allow a null calendar (as the C++ API). Second, the package switched to tinytest as a few of my other packages have done, allowing for very flexible testing during development and deployment—three cheers for easily testing installed packages too. Third, and per a kind nag from Kurt Hornik I updated a few calls which the current QuantLib 1.17 marks as deprecated. That lead to a compile issue with 1.16 so the change is conditional in one part. The complete set of changes is listed below:

Changes in RQuantLib version 0.4.11 (2020-01-15)

  • Changes in RQuantLib code:

    • The 'Null' calendar without weekends or holidays is now recognized.

    • The package now uses tinytest for unit tests (Dirk in #140).

    • Calls deprecated-in-QuantLib 1.17 were updated (Dirk in #144).

Courtesy of CRANberries, there is also a diffstat report for the this release. As always, more detailed information is on the RQuantLib page. Questions, comments etc should go to the new rquantlib-devel mailing list. Issue tickets can be filed at the GitHub repo.

If you like this or other open-source work I do, you can now sponsor me at GitHub. For the first year, GitHub will match your contributions.

This post by Dirk Eddelbuettel originated on his Thinking inside the box blog. Please report excessive re-aggregation in third-party for-profit settings.

15 January, 2020 12:22PM

hackergotchi for Wouter Verhelst

Wouter Verhelst

Running SReview in minikube

I spent the last week or so building Docker images and a set of YAML files that allows one to run SReview, my 99%-automated video review and transcode system, inside minikube, a program that sets up a mini Kubernetes cluster inside a VM for development purposes.

I wish the above paragraph would say "inside Kubernetes", but alas, unless your Kubernetes implementation has a ReadWriteMany volume that can be used from multiple nodes, this is not quite the case yet. In order to fix that, I am working on adding an abstraction layer that will transparently download files from an S3-compatible object store; but until that is ready, this work is not yet useful for large installations.

But that's fine! If you're wanting to run SReview for a small conference, you can do so with minikube. It won't have the redundancy and reliability things that proper Kubernetes provides you, but then you don't really need that for a conference of a few days.

Here's what you do:

  • Download minikube (see the link above)
  • Run minikube start, and wait for it to finish
  • Run minikube addon enable ingress
  • Clone the SReview git repository
  • From the toplevel of that repository, run perl -I lib scripts/sreview-config -a dump|sensible-pager to see an overview of the available configuration options.
  • Edit the file dockerfiles/kube/master.yaml to add your configuration variables, following the instructions near the top
  • Once the file is configured to your liking, run kubectl apply -f master.yaml -f storage-minikube.yaml
  • Add sreview.example.com to /etc/hosts, and have it point to the output of minikube ip.
  • Create preroll and postroll templates, and download them to minikube in the location that the example config file suggests. Hint: minikube ssh has wget.
  • Store your raw recorded assets under /mnt/vda1/inputdata, using the format you specified for the $inputglob and $parse_re configuration values.
  • Profit!

This doesn't explain how to add a schedule to the database. My next big project (which probably won't happen until after the next FOSDEM is to add a more advanced administrator's interface, so that you can just log in and add things from there. For now though, you have to run kubectl port-forward svc/sreview-database 5432, and then use psql to localhost to issue SQL commands. Yes, that sucks.

Having said that, if you're interested in trying this out, give it a go. Feedback welcome!

(many thanks to the people on the #debian-devel IRC channel for helping me understand how Kubernetes is supposed to work -- wouldn't have worked nearly as nice without them)

15 January, 2020 08:12AM

January 14, 2020

Enrico Zini

Raspberry Pi 4: force video mode at boot

Testig himblick automatic media replication

This is part of a series of posts on the design and technical steps of creating Himblick, a digital signage box based on the Raspberry Pi 4.

Another surprise hits us at the last moment: if the system boots without an HDMI monitor plugged in, no framebuffer device is ever created, and X will not start, lightdm will give up after some tries, and even if one plugs in a monitor afterwards, it will stay blank until a reboot or some kind of manual intervention.

As a workaround, one can configure the bootloader to force a specific HDMI configuration. This post documents how we did it.

Find out what video mode one needs

We plugged the target monitor into a laptop and ran xrandr to see the selection of video modes:

   1920x1080     60.00*+  50.00    59.94    30.00    25.00    24.00    29.97    23.98
   1920x1080i    60.00    50.00    59.94

Then we looked up the video mode in the hdmi_mode table, for DMT video modes, in the Video options in config.txt documentation.

Then, since the Raspberry Py 4 has two HDMI outputs, one can append :0 or :1 to each video option to select the output for which it applies.

The resulting bit of config.txt that did the trick for us was this:

# Pretend that a monitor is attached on HDMI0
# Pretend that the monitor is a monitor and not a TV
# Pretend that the monitor has resolution 1920x1080

With that X started, but for some reason it started with different (lower) monitor resolution. Thankfully, a call to xrandr on startup fixed that too, and now everything works as expected whether the system boots with a monitor attached or not.

14 January, 2020 03:25PM

hackergotchi for Jonathan Dowland

Jonathan Dowland

data-types for representing stream-processing programs

This year I want to write much more about my PhD work on my blog, and here's my first effort. Most of this material has been languishing as a draft for over a year, so it's past time to get it out!

1 + 2

As part of my PhD work, I've been looking at data structures for representing stream-processing programs. The intention for our system is to take a user-supplied stream-processing program, rewrite it in order to alter its behaviour and partition it up into sub-programs which could be deployed and executed on different computers, connected together via TCP/IP.

1 * 2

To help familiarise myself with the existing system, when I started working on this I begun to explore different ways of representing both a stream-processing program and a set of interconnected, partitioned programs. Graph data structures seem like a natural fit for these, with stream-processing programs as a graph and interconnected programs as a graph-of-graphs1.

1 * (2 + 3)

There are a number of different graph libraries for Haskell. The most common approach they use for representation is "tabular": lists of edges as pairs of vertices, or similar. This isn't the only approach. One of the older, more established libraries — fgl — uses inductive types. But the one I have initially settled on is Algebra.Graph, which defines an algebra of graphs with which you can construct your instances2.

The USP for Algebra.Graph is that the four provided constructors are all total functions, so certain types of invalid graph are impossible to represent with the library (such as those where an edge does not point to a vertex).

The four basic constructors are3:

  • Vertex x, a single vertex, containing x
  • Overlay x y, which overlays one graph upon another
  • Connect x y, which connects all the vertices from Graph x to all of the vertices in Graph y.
  • Empty, for an empty graph

The Graph type implements the Num type-class, so Overlay can be abbreviated to + and connect to *. I've included some example graph definitions, encoded using + and * for brevity, and images of their corresponding renderings within this blog post.

I didn't perform an exhaustive search — nor evaluation — of all the available graph libraries. There's no definitive "right" answer to the question of which to choose: the graphs I will be dealing with are relatively small, so raw performance is not a major consideration.

So, what does a stream-processing program look like, encoded in this way? Here's a real example of a simple 5-node path graph (from here), simplified a little for clarity:

λ> foldg Empty (Vertex . vertexId) Overlay Connect graph
Overlay (Connect (Vertex 1) (Vertex 2)) (Overlay (Connect (Vertex 2)
(Vertex 3)) (Overlay (Connect (Vertex 3) (Vertex 4)) (Connect (Vertex 4)
(Vertex 5))))

Rendering it graphically is more clear:

simple 5-node stream graph

  1. Graphs are not the only data-type that could be used, of course. I've started out using a graph representation in order to bootstrap the implementation and get further along with a proof-of-concept, but there are shortcomings that might be addressed by other approaches. I'll write more about those in another blog post.
  2. By coincidence, Andrey Mokhov, the author of Algebra.Graph was a Senior Lecturer at Newcastle University, where I am a student, and was also co-author of a draft paper that was responsible for me getting interested in pursuing this work in the first place. Later, Andrey briefly became my second supervisor, but has now moved on to work for Jane Street. He remains a visiting fellow at Newcastle.
  3. Different variants of the grammar can vary these constructors to achieve different results. For example, you can forbid empty graphs by removing the Empty constructor. An adjustment to the types is made to support edge-labelling.

14 January, 2020 10:21AM

Russ Allbery

New year's haul

Accumulated book purchases for the past couple of months. A rather eclectic mix of stuff.

Becky Albertalli — Simon vs. the Homo Sapiens Agenda (young adult)
Ted Chiang — Exhalation (sff collection)
Tressie McMillan Cottom — Thick (nonfiction)
Julie E. Czerneda — This Gulf of Time and Stars (sff)
Katharine Duckett — Miranda in Milan (sff)
Sarah Gailey — Magic for Liars (sff)
Carol Ives Gilman — Halfway Human (sff)
Rachel Hartman — Seraphina (sff)
Isuna Hasekura — Spice and Wolf, Volume 1 (sff)
Elizabeth Lim — Spin the Dawn (sff)
Sam J. Miller — Blackfish City (sff)
Tamsyn Muir — Gideon the Ninth (sff)
Sylvain Neuvel — The Test (sff)
K.J. Parker — Sixteen Ways to Defend a Walled City (sff)
Caroline Criado Perez — Invisible Women (nonfiction)
Delia Sherman — The Porcelain Dove (sff)
Connie Willis — All About Emily (sff)

Several sales on books that I wanted to read for various reasons, several recommendations, one book in an ongoing series, and one earlier book in a series that I want to read.

We'll see if, in 2020, I can come closer to reading all the books that I buy in roughly the same year in which I buy them.

14 January, 2020 04:05AM

January 13, 2020

Enrico Zini

Creating a Raspberry PI SD from tar files

Pile of Raspberry Pi 4 boxes

This is part of a series of posts on the design and technical steps of creating Himblick, a digital signage box based on the Raspberry Pi 4.

Provisioning a SD card starting from the official raspbian-lite is getting quite slow, since there are a lot of packages to install.

It would be significantly faster if we could take a SD card, partition it from scratch, then untar the boot and rootfs partition contents into them.

Here's how.

Partitioning a SD card from scratch

We can do almost everything with pyparted.

See this LinuxVoice article for a detailed introduction to pyparted, and the C parted documentation for some low-level reference.

Here is the pyparted recipe for the SD card, plus a media directory at the end:

def partition_reset(self, dev: Dict[str, Any]):
    Repartition the SD card from scratch
        import parted
    except ModuleNotFoundError:
        raise Fail("please install python3-parted")

    device = parted.getDevice(dev["path"])

    disk = parted.freshDisk(device, "msdos")

    # Add 256M fat boot
    optimal = device.optimumAlignment
    constraint = parted.Constraint(
            start=parted.sizeToSectors(4, "MiB", device.sectorSize),
            end=parted.sizeToSectors(16, "MiB", device.sectorSize)),
            start=parted.sizeToSectors(256, "MiB", device.sectorSize),
            end=parted.sizeToSectors(512, "MiB", device.sectorSize)),
        minSize=parted.sizeToSectors(256, "MiB", device.sectorSize),
        maxSize=parted.sizeToSectors(260, "MiB", device.sectorSize))
    geometry = parted.Geometry(
        length=parted.sizeToSectors(256, "MiB", device.sectorSize),
    geometry = constraint.solveNearest(geometry)
    boot = parted.Partition(
            disk=disk, type=parted.PARTITION_NORMAL, fs=parted.FileSystem(type='fat32', geometry=geometry),
    disk.addPartition(partition=boot, constraint=constraint)

    # Add 4G ext4 rootfs
    constraint = parted.Constraint(
            end=geometry.end + parted.sizeToSectors(16, "MiB", device.sectorSize)),
            start=geometry.end + parted.sizeToSectors(4, "GiB", device.sectorSize),
            end=geometry.end + parted.sizeToSectors(4.2, "GiB", device.sectorSize)),
        minSize=parted.sizeToSectors(4, "GiB", device.sectorSize),
        maxSize=parted.sizeToSectors(4.2, "GiB", device.sectorSize))
    geometry = parted.Geometry(
        length=parted.sizeToSectors(4, "GiB", device.sectorSize),
    geometry = constraint.solveNearest(geometry)
    rootfs = parted.Partition(
            disk=disk, type=parted.PARTITION_NORMAL, fs=parted.FileSystem(type='ext4', geometry=geometry),
    disk.addPartition(partition=rootfs, constraint=constraint)

    # Add media partition on the rest of the disk
    constraint = parted.Constraint(
            end=geometry.end + parted.sizeToSectors(16, "MiB", device.sectorSize)),
            start=geometry.end + parted.sizeToSectors(16, "MiB", device.sectorSize),
        minSize=parted.sizeToSectors(4, "GiB", device.sectorSize),
    geometry = constraint.solveMax()
    # Create media partition
    media = parted.Partition(
            disk=disk, type=parted.PARTITION_NORMAL,
    disk.addPartition(partition=media, constraint=constraint)


Setting MBR disk identifier

So far so good, but /boot/cmdline.txt has root=PARTUUID=6c586e13-02, and we need to change the MBR disk identifier to match:

# Fix disk identifier to match what is in cmdline.txt
with open(dev["path"], "r+b") as fd:
    buf = bytearray(512)
    buf[0x1B8] = 0x13
    buf[0x1B9] = 0x6e
    buf[0x1BA] = 0x58
    buf[0x1BB] = 0x6c

Formatting the partitions

Formatting is reasonably straightforward, and although we've tried to match the way raspbian formats partitions, it may be that not all of these options are needed:

# Format boot partition with 'boot' label
run(["mkfs.fat", "-F", "32", "-n", "boot", disk.partitions[0].path])

# Format rootfs partition with 'rootfs' label
run(["mkfs.ext4", "-F", "-L", "rootfs", "-O", "^64bit,^huge_file,^metadata_csum", disk.partitions[1].path])

# Format exfatfs partition with 'media' label
run(["mkexfatfs", "-n", "media", disk.partitions[2].path])

Now the SD card is ready for a simple untarring of the boot and rootfs partition contents.

Useful commands

These commands were useful in finding out differences between how the original Raspbian image partitions were formatted, and how we were formatting them:

sudo minfo -i /dev/sdb1 ::
sudo tune2fs -l /dev/sdb2

13 January, 2020 05:45PM

January 12, 2020

Russ Allbery

Review: Guardians of the West

Review: Guardians of the West, by David Eddings

Series: The Malloreon #1
Publisher: Del Rey
Copyright: April 1987
Printing: October 1991
ISBN: 0-345-35266-1
Format: Mass market
Pages: 438

Technically speaking, many things in this review are mild spoilers for the outcome of The Belgariad, the previous series set in this world. I'm not going to try to avoid that because I think most fantasy readers will assume, and be unsurprised by, various obvious properties of the ending of that type of epic fantasy.

The world has been saved, Garion is learning to be king (and navigate his domestic life, but more on that in a moment), and Errand goes home with Belgarath and Polgara to live the idyllic country life of the child he never was. That lasts a surprisingly long way into the book, with only occasional foreshadowing, before the voice in Garion's head chimes in again, new cryptic prophecies are discovered, and the world is once again in peril.

I can hear some of you already wondering what I'm doing. Yes, after re-reading The Belgariad, I'm re-reading The Malloreon. Yes, this means I'm arguably reading the same series four times. I was going through the process of quitting my job and wrapping up projects and was stressed out of my mind and wanted something utterly predictable and unchallenging that I could just read and enjoy without thinking about. A re-read of Eddings felt perfect for that, and it was.

The Malloreon is somewhat notorious in the world of epic fantasy because the plot... well, I won't say it's the same plot as The Belgariad, although some would, but it has eerie similarities. The overarching plot of The Belgariad is the battle between the Child of Light and the Child of Dark, resolved at the end of Enchanters' End Game. The kickoff of the plot of The Malloreon near the middle of this book is essentially "whoops, there was another prophecy and you have to do this all again." The similarities don't stop there: There's a list of named figures who have to go on the plot journey that's only slightly different from the first series, a mysterious dark figure steals something important to kick off the plot, and of course there is the same "free peoples of the west" versus "dictatorial hordes of the east" basic political structure. (If you're not interested in more of that in your fantasy, I don't blame you a bit and Eddings is not the author to reach for.)

That said, I've always had a soft spot for this series. We've gotten past the introduction of characters and gotten to know an entertaining variety of caricatures, Eddings writes moderately amusing banter, and the characters can be fun if you treat them like talking animals built around specific character traits. Guardians of the West moves faster and is less frustrating than Pawn of Prophecy by far. It also has a great opening section where Errand, rather than Garion, is the viewpoint character.

Errand is possibly my favorite character in this series because he takes the plot about as seriously as I do. He's fearless and calm in the face of whatever is happening, which his adult guardians attribute to his lack of understanding of danger, but which I attribute to him being the only character in the book who realizes that the plot is absurd and pre-ordained and there's no reason to get so worked up about it. He also has a casual, off-hand way of revealing that he has untapped plot-destroying magical powers, which for some reason I find hilarious. I wish the whole book were told from Errand's point of view.

Sadly, two-thirds of it returns to Garion. That part isn't bad, exactly, but it features more of his incredibly awkward and stereotyped relationship with Ce'Nedra, some painful and obvious stupidity around their attempt to have a child, and possibly the stupidest childbirth scene I've ever seen. (Eddings is aiming for humorous in a way that didn't work for me at all.) That's followed by a small war (against conservative religious fanatics; Eddings's interactions with cultural politics are odd and complicated) that wasn't that interesting.

That said, the dry voice in Garion's head was one of my favorite characters in the first series and that's even more true here when he starts speaking again. I like some of what Eddings is doing with prophecy and how it interacts with the plot. I'm also endlessly amused when the plot is pushed forward by various forces telling the main characters what to do next. Normally this is a sign of lazy writing and poor plotting, but Eddings is so delightfully straightforward about it that it becomes oddly metafictional and, at least for me, kind of fun. And more of Errand is always enjoyable.

I can't recommend this series (or Eddings in general). I like it for idiosyncratic reasons and can't defend it as great writing. There are a lot of race-based characterization, sexism, and unconsidered geographic stereotypes (when you lay the world map over a map of Europe, the racism is, uh, kind of blatant, even though Eddings makes relatively even-handed fun of everyone), and while you could say the same for Tolkien, Eddings is not remotely at Tolkien levels of writing in compensation. But Guardians of the West did exactly what I wanted from it when I picked it up, and now part of me wants to finish my re-read, so you may be hearing about the rest of the series.

Followed by King of the Murgos.

Rating: 6 out of 10

12 January, 2020 05:18AM

January 11, 2020

hackergotchi for Romain Perier

Romain Perier

Add support for F2FS filesystem to GRUB and initramfs-tools

Hi there,

For these like me who want to change their root filesystem to F2FS, I have enabled support for adding the F2FS module in the EFI signed image of grub in Debian (commit). So the grub EFI image can load configuration, kernel images and initrd from a /boot that is formatted in F2FS (the upstream grub supports the filesystem since 2.04).

Now that the kernel is loading it must be able to mount the rootfs. In Debian, a lot of features like some filesystems or some drivers are built as modules, this allow to be able to boot and work on a lot of different machines without have to build-in statically everything into the linux kernel image. This is why we use an initramfs , it offers a variety of cool features and detects magically some details for you like "load the brtfs module or your favorite emmc driver as module". If you want to use F2FS as your main filesystem on your rootfs, we need to add F2FS as base module into initramfs-tools (that handles all the scripts and the magic stuffs for your initramfs). It has been done by this commit.

See you !

11 January, 2020 07:20PM by Romain Perier (noreply@blogger.com)

hackergotchi for Markus Koschany

Markus Koschany

My Free Software Activities in December 2019

Welcome to gambaru.de. Here is my monthly report that covers what I have been doing for Debian. If you’re interested in Java, Games and LTS topics, this might be interesting for you.

Debian Games

  • I started the month by backporting the latest version of minetest to buster-backports.
  • New versions of Springlobby, the single and multiplayer lobby for the Spring RTS engine, and Freeciv (now at 2.6.1) were packaged.
  • I had to remove python-pygccxml as a build-dependency from spring because of the Python 2 removal and there was also another unrelated build failure that got fixed as well.
  • I also released a new version of the debian-games metapackages. A considerable number of games were removed from Debian in the past months, in parts due to the ongoing Python 2 removal but also because of inactive maintainers or upstreams. There were also some new games though. Check out the 3.1 changelog for more information. As a consequence of our Python 2 goal, the development metapackage for Python 2 is gone now.

Debian Java


  • The imlib2 image library was updated to version 1.6.1 and now supports the webp image format.
  • I backported the Thunderbird addon dispmua to Buster and Stretch because the new Thunderbird ESR version had made it unusable.
  • I also updated binaryen, a compiler and library for WebAssembly and asked upstream if they could relax the build-dependency on Git which they did.

Debian LTS

This was my 46. month as a paid contributor and I have been paid to work 16,5 hours on Debian LTS, a project started by Raphaël Hertzog. In that time I did the following:

From 23.12.2019 until 05.01.2020 I was in charge of our LTS frontdesk. I investigated and triaged CVE in sudo, shiro, waitress, sa-exim, imagemagick, nss, apache-log4j1.2, sqlite3, lemonldap-ng, libsixel, graphicsmagick, debian-lan-config, xerces-c, libpodofo, vim, pure-ftpd, gthumb, opencv, jackson-databind, pillow, fontforge, collabtive, libhibernate-validator-java, lucene-solr and gpac.

  • DLA-2051-1. Issued a security update for intel-microcode fixing 2 CVE.
  • DLA-2058-1. Issued a security update for nss fixing 1 CVE.
  • DLA-2062-1. Issued a security update for sa-exim fixing 1 CVE.
  • I prepared a security update for tomcat7 by updating to the latest upstream release in the 7.x series. It is pending review by Mike Gabriel at the moment.


Extended Long Term Support (ELTS) is a project led by Freexian to further extend the lifetime of Debian releases. It is not an official Debian project but all Debian users benefit from it without cost. The current ELTS release is Debian 7 “Wheezy”. This was my nineteenth month and I have been assigned to work 15 hours on ELTS.

  • I was in charge of our ELTS frontdesk from 23.12.2019 until 05.01.2020 and I triaged CVE in sqlite3, libxml2 and nss.
  • ELA-200-2. Issued a security update for intel-microcode.
  • Worked on tomcat7, CVE-2019-12418 and CVE-2019-17563, and finished the patches prepared by Mike Gabriel. We have discovered some unrelated test failures and are currently investigating the root cause of them.
  • Worked on nss, which is required to build OpenJDK 7 and also needed at runtime for the SunEC security provider. I am currently investigating CVE-2019-17023 which has been assigned only a few days ago.
  • ELA-206-1. Issued a security update for apache-log4j1.2 fixing 1 CVE.

Thanks for reading and see you next time.

11 January, 2020 05:36PM by apo

hackergotchi for Ritesh Raj Sarraf

Ritesh Raj Sarraf

Laptop Mode Tools 1.73

Laptop Mode Tools 1.73

I am pleased to announce the release of Laptop Mode Tools version 1.73

This release includes many bug fixes. For user convenience, 2 command options have been added.

rrs@priyasi:~$ laptop_mode -h
Following user commands are understood
status      :   Display a Laptop Mode Tools power savings status
power-stats  :  Display the power statistics on the machine
power-events :  Trap power related events on the machine
help        :   Display this help message (--help, -h)
version     :   Display program version (--version, -v)
15:22 â™’ ༐  â˜ş đŸ˜„    

rrs@priyasi:~$ sudo laptop_mode status
[sudo] password for rrs: 
   /dev/mapper/nvme0n1p4_crypt on / type btrfs (rw,noatime,compress=zstd:3,ssd,space_cache,autodefrag,subvolid=5,subvol=/)
   /dev/nvme0n1p3 on /boot type ext4 (rw,relatime)
   /dev/nvme0n1p1 on /boot/efi type vfat (rw,relatime,fmask=0077,dmask=0077,codepage=437,iocharset=ascii,shortname=mixed,utf8,errors=remount-ro)
   /dev/fuse on /run/user/1000/doc type fuse (rw,nosuid,nodev,relatime,user_id=1000,group_id=1000)
Drive power status:
   Cannot read /dev/[hs]d[abcdefgh], permission denied - /usr/sbin/laptop_mode needs to be run as root
(NOTE: drive settings affected by Laptop Mode cannot be retrieved.)
Readahead states:
   /dev/mapper/nvme0n1p4_crypt: 128 kB
   /dev/nvme0n1p3: 128 kB
   /dev/nvme0n1p1: 128 kB
Laptop Mode Tools is allowed to run: /var/run/laptop-mode-tools/enabled exists.

   state:      open
15:22 â™’ ༐  â˜ş đŸ˜„    

rrs@priyasi:~$ laptop_mode power-stats
Power Supply details for /sys/class/power_supply/AC

P: /devices/LNXSYSTM:00/LNXSYBUS:00/ACPI0003:00/power_supply/AC
L: 0
E: DEVPATH=/devices/LNXSYSTM:00/LNXSYBUS:00/ACPI0003:00/power_supply/AC
E: SUBSYSTEM=power_supply

Power Supply details for /sys/class/power_supply/BAT0

P: /devices/LNXSYSTM:00/LNXSYBUS:00/PNP0C0A:00/power_supply/BAT0
L: 0
E: DEVPATH=/devices/LNXSYSTM:00/LNXSYBUS:00/PNP0C0A:00/power_supply/BAT0
E: SUBSYSTEM=power_supply

15:23 â™’ ༐  â˜ş đŸ˜„    

rrs@priyasi:~$ laptop_mode power-events
Running Laptop Mode Tools in event tracing mode. Press ^C to interrupt
monitor will print the received events for:
UDEV - the event which udev sends out after rule processing
KERNEL - the kernel uevent

KERNEL[140321.536870] change   /devices/LNXSYSTM:00/LNXSYBUS:00/ACPI0003:00/power_supply/AC (power_supply)

KERNEL[140321.569526] change   /devices/LNXSYSTM:00/LNXSYBUS:00/PNP0C0A:00/power_supply/BAT0 (power_supply)

UDEV  [140321.577770] change   /devices/LNXSYSTM:00/LNXSYBUS:00/ACPI0003:00/power_supply/AC (power_supply)

UDEV  [140321.582123] change   /devices/LNXSYSTM:00/LNXSYBUS:00/PNP0C0A:00/power_supply/BAT0 (power_supply)

KERNEL[140324.857185] change   /devices/LNXSYSTM:00/LNXSYBUS:00/ACPI0003:00/power_supply/AC (power_supply)

UDEV  [140324.916156] change   /devices/LNXSYSTM:00/LNXSYBUS:00/ACPI0003:00/power_supply/AC (power_supply)

KERNEL[140324.917955] change   /devices/LNXSYSTM:00/LNXSYBUS:00/PNP0C0A:00/power_supply/BAT0 (power_supply)

UDEV  [140324.922916] change   /devices/LNXSYSTM:00/LNXSYBUS:00/PNP0C0A:00/power_supply/BAT0 (power_supply)

15:24 â™’ ༐   ☚ đŸ˜Ÿ=> 130  

A filtered list of changes is mentioned below. For the full log, please refer to the git repository.

1.73 - Sat Jan 11 14:52:11 IST 2020

* Respect black/white lists when disabling autosuspend
* Add newer power supply names
* Fix crash due external battery of mouse
* Honor configuration setting for battery level polling
* cpufreq: intel_pstate should use performance governors
* runtime-pm: Speed up by avoiding fork in echo_to_file
* runtime-pm: Inline echo_to_file_do
* runtime-pm: Fix echo_to_file* indentation
* runtime-pm: Speed up by avoiding fork in listed_by_{id,type}
* runtime-pm: Simplify vendor/product match
* add help and verison user commands
* Add a power-stats status command
* Separate power sysfs attributes and add sysfs status attribute
* Add device type 'sd' to default blacklist
* Fix rpm spec file for new installable files

Source tarball, Feodra/SUSE RPM Packages available at: https://github.com/rickysarraf/laptop-mode-tools/releases

Debian packages will be available soon in Unstable.

Homepage: https://github.com/rickysarraf/laptop-mode-tools/wiki

Mailing List: https://groups.google.com/d/forum/laptop-mode-tools

What is Laptop Mode Tools

Description: Tools for Power Savings based on battery/AC status
 Laptop mode is a Linux kernel feature that allows your laptop to save
 considerable power, by allowing the hard drive to spin down for longer
 periods of time. This package contains the userland scripts that are
 needed to enable laptop mode.
 It includes support for automatically enabling laptop mode when the
 computer is working on batteries. It also supports various other power
 management features, such as starting and stopping daemons depending on
 power mode, automatically hibernating if battery levels are too low, and
 adjusting terminal blanking and X11 screen blanking
 laptop-mode-tools uses the Linux kernel's Laptop Mode feature and thus
 is also used on Desktops and Servers to conserve power

11 January, 2020 09:44AM by Ritesh Raj Sarraf (rrs@researchut.com)

January 10, 2020

hackergotchi for Anisa Kuci

Anisa Kuci

Outreachy post 3 - Midterm report

Time passes by quickly when you do the things that you like. And so have passed by very quickly the first six weeks of Outreachy. The first half of the internship has been an amazing experience for me. I have worked and learned so many new things. I got familiar more closely with the Debian project that I was already contributing to in the past, but less intensively. I am very happy to get to know more people from the community, feel so welcomed and find such a warm environment.

Since the first weeks of the internship I started working on fundraising materials for DebConf20 as part of my tasks, using LaTeX which is an amazing tool to work on creating different types of documents. My skills on using LaTeX are improved, and the more I use it the more I discover how powerful a tool it is and the variety of things that you can do with it. Lately I worked on the flyer and brochure that will be sent to potential sponsors.

DebConf20 sponsorship flyer

On the flyer I removed the translation elements, since this year the materials will be only in English. I updated the content making it relevant for this year, and also updated the logo to the winning entry of a contest the local team ran. Matching to the dominant color of the DebConf20 logo I created a color scale that we are using for headlines and decorative elements within the fundraising material and the conference web page.

DebConf20 color scale

As for the fundraising brochure, I took the content from a Google doc, which was carefully created by my mentor Karina and converted it into LaTeX. I adapted it with the new logo, colors and monetary values in the local currency. For this I needed to create a TeX \newcommand as the ILS currency symbol (₪) is not supported natively. This also led to a restriction in the choice of fonts available because the ILS symbol needs to be part of the font. With support from the wider DebConf team we settled on Liberation Sans. As we are working on the visual identity of DebConf20, we are almost finalizing the fundraising materials for this edition.

I have also worked on the draft email templates that I have proposed for the next phases of contacting sponsors, hoping I will receive a good feedback from the team. They are available on a private DebConf git repo. The basic idea is to provide new aspects of the benefits of sponsoring a DebConf with each contact that we have reaching out to sponsors.

Initial commit of the DebConf20 sponsorship brochure

Beside practicing LaTeX I have also worked a lot on git and it has been very helpful for me to practice. There is so much information to work on and so much you can do with git. I am trying to get beyond the common level of understanding git:

xkcd on git

Another task I have is documentation, so, I have worked on this too, in parallel. As each DebConf is organized every year in another country, you might imagine that for the local team not everything is familiar, even if they might be part of Debian, and of course depending also on the experience they might have on organizing events or specifically fundraising. So, working on fundraising now, I have had many things that I was not completely familiar with and I have started documenting the workflow so it will be hopefully more convenient and smooth process for future DebConf local organizing teams.

As mentioned on my last blog post, I have already joined the main communication channels that the Debian community uses. I try to be as much available as I can and try to stay updated with all the info that might be relevant information for my internship. I participate in all the biweekly team meetings for DebConf20, giving updates about my progress and staying in the loop of the current situation regarding organizational topics related to the conference.

Updating the DebConf20 sponsorship flyer in git

I stay in contact with my mentors Daniel and Karina via IRC and emails. I would like to take a moment and thank them for all their encouragement, support and feedback which has helped me improve and has motivated me a lot to continue working in this awesome project. I keep connection to the wider community as well via IRC, Planet Debian or constantly following the mailing lists.

Last but not least, I also participate in the Outreachy webchats where I had the chance to have a little bit of background from other Outreachy interns and meet the people who are running the Outreachy program. I am so glad to see what a safe, easygoing and inclusive environment they have created for everyone.

My experience so far has been a blast!

10 January, 2020 06:37PM

hackergotchi for Dirk Eddelbuettel

Dirk Eddelbuettel

rfoaas 2.1.0: New upstream so new access point!

rfoaas greed example

FOAAS, having been resting upstream for some time, released version 2.1.0 of its wonderful service this week! So without too much further ado we went to work and added support for it. And now we are in fact thrilled to announce that release 2.1.0 of rfoaas is now on CRAN as of this afternoon (with a slight delay as yours truly managed to state the package release date as 2019-01-09 which was of course flagged as ‘too old’).

The new 2.1.0 release of FOAAS brings a full eleven new REST access points, namely even(), fewer(), ftfty(), holygrail(), idea(), jinglebells(), legend(), logs(), ratsarse(), rockstar(), and waste(). On our end, documentation and tests were updated.

As usual, CRANberries provides a diff to the previous CRAN release. Questions, comments etc should go to the GitHub issue tracker. More background information is on the project page as well as on the github repo

If you like this or other open-source work I do, you can now sponsor me at GitHub. For the first year, GitHub will match your contributions.

This post by Dirk Eddelbuettel originated on his Thinking inside the box blog. Please report excessive re-aggregation in third-party for-profit settings.

10 January, 2020 12:37AM

January 09, 2020

hackergotchi for Lisandro Damián Nicanor Pérez Meyer

Lisandro Damián Nicanor Pérez Meyer

Qt 4 removed from Debian bullseye (current testing)

Today Qt 4 (aka src:qt4-x11) has been removed from Debian bullseye, what as of today we know as "testing". We plan to remove it from unstable pretty soon.

09 January, 2020 10:31PM by Lisandro Damián Nicanor Pérez Meyer (noreply@blogger.com)

January 08, 2020

hackergotchi for Steve Kemp

Steve Kemp

I won't write another email client

Once upon a time I wrote an email client, in a combination of C++ and Lua.

Later I realized it was flawed, and because I hadn't realized that writing email clients is hard I decided to write it anew (again in C++ and Lua).

Nowadays I do realize how hard writing email clients is, so I'm not going to do that again. But still .. but still ..

I was doing some mail-searching recently and realized I wanted to write something that processed all the messages in a Maildir folder. Imagine I wanted to run:

 message-dump ~/Maildir/people-foo/ ~/Maildir/people-bar/  \
     --format '${flags} ${filename} ${subject}'

As this required access to (arbitrary) headers I had to read, parse, and process each message. It was slow, but it wasn't that slow. The second time I ran it, even after adjusting the format-string, it was nice and fast because buffer-caches rock.

Anyway after that I wanted to write a script to dump the list of folders (because I store them recursively so ls -1 ~/Maildir wasn't enough):

 maildir-dump --format '${unread}/${total} ${path}'

I guess you can see where this is going now! If you have the following three primitives, you have a mail-client (albeit read-only)

  • List "folders"
  • List "messages"
  • List a single message.

So I hacked up a simple client that would have a sub-command for each one of these tasks. I figured somebody else could actually use that, be a little retro, be a little cool, pretend they were using MH. Of course I'd have to write something horrid as a bash-script to prove it worked - probably using dialog to drive it.

And then I got interested. The end result is a single golang binary that will either:

  • List maildirs, with a cute format string.
  • List messages, with a cute format string.
  • List a single message, decoding the RFC2047 headers, showing text/plain, etc.

And now I wonder, am I crazy? Is writing an email client hard? I can't remember

Probably best to forget the GUI exists. Probably best to keep it a couple of standalone sub-commands for "scripting email stuff".

But still .. but still ..

08 January, 2020 07:19PM

hackergotchi for Thomas Lange

Thomas Lange

20 years of FAI and a new release

20 years ago, on December 20, 1999 FAI 1.0 was released. Many things have happened since then. Some milestones:

  • 1999: version 1.0
  • 2000: first official Debian package
  • 2001: first detailed user report ("No complete installs. Teething problems.")
  • 2005: Wiki page and IRC
  • 2005: FAI CD
  • 2006: fai dirinstall
  • 2007: new partitioning tool setup-storage
  • 2009: new web design
  • 2014: brtfs support
  • 2016: autodiscover function, profiles menu
  • 2016: fai-diskimage, cloud images
  • 2017: cross architecture builds
  • 2017: Fai.me web service
  • 2020: UEFI support

Besides that, a lot of other things happened in the FAI project. Apart from the first report, we got more than 300 detailed reports containing positive feedback. We had several FAI developers meetings and I did more than 40 talks about FAI all over the world. We had a discussion about an alleged GPL violation of FAI in the past, I did several attempts to get a logo for FAI, but we still do not have one. We moved from subversion to git, which was very demanding for me. The FAI.me service for customized installation and cloud images was used more than 5000 times. The Debian Cloud team now uses FAI to build the official Debian cloud images.

I'm very happy with the outcome of this project and I like to thank all people who contributed to FAI in the past 20 years!

This week, I've released the new FAI version 5.9. It supports UEFI boot from CD/DVD and USB stick. Also two new tools were added:

  • fai-sed - call sed on a file but check for changes before writing
  • fai-link - create symlink idempotent

UEFI support in fai-cd only used grub, no syslinux or isolinux is needed. New FAI installation images are also available from


The FAI.me build service is also using the newest FAI version and the customized ISO images can now be booted in an UEFI environment.


08 January, 2020 12:21PM

January 07, 2020

Ingo Juergensmann

XMPP - Prosody & Ejabberd

In my day job I'm responsible of maintaining the VoIP and XMPP infrastructure. That's about approx. 40.000 phones and several thousand users using Enterprise XMPP software. Namely it is Cisco CUCM and IM&P on the server side and Cisco Jabber on the client side. There is also Cisco Webex and Cisco Telepresence infrastructure to maintain.

On the other hand I'm running an XMPP server myself for a few users. It all started with ejabberd more than a decade ago or so. Then I moved to Openfire, because it was more modern and had a nice web GUI for administration. At some point there was Prosody as a new shiny star. This is now running for many users, mostly without any problems, but without much love and attention as well.

It all started as "Let's see what this Jabber stuff is..." on a subdomain like jabber.domain.com - it was later that I discovered the benefits of SRV records and the possibility of having the same address for mail, XMPP and SIP. So I began to provide XMPP acounts as well for some of my mail domains.

A year ago I enabled XMPP for my Friendica node on Nerdica.net, the second largest Friendica node according to the-federation.info. Although there are hundreds of monthly active users on Friendica, only a handful of users are using XMPP. XMPP has a hard stand since Google and Facebook went from open federation to closing in their user base.

My personal impression is that there is a lot of development in the last years in regards of XMPP - thanks to the Conversations client on Android - and its Compliance Tester. With that tool it is quite easy to have a common ground for the most needed features of todays user expectation in a mobile world. There is also some news in regards to XMPP clients on Apple iOS, but that's for another article.

This is about the server side, namely Prosody and Ejabberd. Of course there are already several excellent comparisons between these two server softwares. So, this is just my personal opinion and personal impressions about the two softwares I got in the past two weeks.

As I have the most experience with Prosody I'll start with it. Prosody has the advantage of being actively maintained and having lots of community modules to extend its functionality. This is a big win - but there is also the other side of truth: you'll need to install and configure many contrib modules to pass 100% in the Compliance Tester. Some modules might be not that well maintained. Another obstacle I faced with Prosody is the configuration style: usually you have the main config file where you can configure common settings, modules for all virtual hosts and components like PubSub, MUC, HTTP Upload and such. And then there are the config files for the virtual hosts, which feature the same kind of configuration. Important to all is (apparently): order does matter! This can get confusing: Components are similar to loading modules, using both for the same purpose can be, well, interesting. and configuration of modules and components can be challenging as well. When trying to get mod_http_upload working in the last days I experienced that a config on one virtual host was working, but the same config on a different host was not working. This was when I thought I might give Ejabberd a chance...

Contrary to Prosody there is a company behind Ejabberd. And this is often perceived as being good and bring some stability to Ejabberd. However, when I joined Ejabberd chat room, I learned in the first minutes by regarding the chat log that the main developer of that company left and the company itself seemed to have lost interest in Ejabberd. However the people in the chat room were relaxed: it's not the end of the world and there are other developers working on the code. So, no issue in the end, but that's not something you expect to read when you join a chat room for the first time. ;)
Contrary to Prosody Ejabberd seems to be well-prepared to pass the Compliance Tester without installing (too many) modules. Large sites such as conversations.im are running on Ejabberd. It is also said that Ejabberd doesn't need restarts of the server for certain config changes as Prosody does. The config file itself appears to be more straightforward and doesn't differentiate between modules and components which makes it a little more easy to understand.

Currently I haven't been able to deal much with Ejabberd, but one other difference is: there is a Debian repository on Prosody.im, but for Ejabberd there is no such repository. You'll have to use backports.debian.org for a newer version of Ejabberd on Debian Buster. It's up to you to decide what is better for you.

I'm still somewhat undecided whether or not to proceed with Ejabberd and migrate from Prosody. The developer of Prosody is very helpful and responsive and I like that. On the other hand, the folks in the Ejabberd chat rooms are very supportive as well. I like the flexibility and the various number of contrib modules for Prosody, but then again it's hard to find the correct/best one to load and to configure for a given task and to satisfy the Compliance Tester. Then again, both servers do feature a Web GUI for some basic tasks, but I like the one of Ejabberd more.

So, in the end, I'm also open for suggestions about either one. Some people will state of course that neither is the best way and I should consider Matrix, Briar or some other solutions, but that's maybe another article comparing XMPP and other options. This one is about XMPP server options: Prosody or Ejabberd. What do you prefer and why?



07 January, 2020 08:21PM by ij

January 06, 2020

Reproducible Builds

Reproducible Builds in December 2019

Welcome to the December 2019 report from the Reproducible Builds project!

In these reports we outline the most important things that we have been up to over the past month. As a quick recap, whilst anyone can inspect the source code of free software for malicious flaws, almost all software is distributed to end users as pre-compiled binaries.

The motivation behind the reproducible builds effort is to ensure no flaws have been introduced during this compilation process by promising identical results are always generated from a given source, thus allowing multiple third-parties to come to a consensus on whether a build was compromised.

In this report for December, we cover:

  • Media coverageA Google whitepaper, The Update Framework graduates within the Cloud Native Computing Foundation, etc.
  • Reproducible Builds Summit 2019What happened at our recent meetup?
  • Distribution workThe latest reports from Arch, Debian and openSUSE, etc.
  • Software developmentPatches, patches, patches…
  • Mailing list summary
  • Contact — *How to contribute

If you are interested in contributing to our project, please visit the Contribute page on our website.

Media coverage

Google published Binary Authorization for Borg, a whitepaper on how they reduce exposure of user data to unauthorised code as well as methods for verifying code provenance using their Borg cluster manager. In particular, the paper notes how they attempt to limit their “insider risk”, ie. the potential for internal personnel to use organisational credentials or knowledge to perform malicious activities.

The Linux Foundation announced that The Update Framework (TUF) has graduated within the Cloud Native Computing Foundation (CNCF) and thus becomes the first specification and first security-focused project to reach the highest maturity level in that group. TUF is a technology that secures software update systems initially developed by Justin Cappos at the NYU Tandon School of Engineering.

Andrew “bunnie” Huang published a blog post asking Can We Build Trustable Hardware? Whilst it concludes pessimistically that “open hardware is precisely as trustworthy as closed hardware” it does mention that reproducible builds can:

Enable any third-party auditor to download, build, and confirm that the program a user is downloading matches the intent of the developers.

At the 36th Chaos Communication Congress (36C3) in Leipzig, Hannes Mehnert from the MirageOS project gave a presentation called Leaving legacy behind which talks generally about Mirage system offering a potential alternative and minimalist approach to security but has a section on reproducible builds (at 38m41s).

Reproducible Builds Summit 2019

We held our fifth annual Reproducible Builds summit between the 1st and 8th December at Priscilla, Queen of the Medina in Marrakesh, Morocco.

The aim of the meeting was to spend time discussing and working on Reproducible Builds with a widely diverse agenda and the event was a huge success.

During our time together, we updated and exchanged the status of reproducible builds in our respective projects, improved collaboration between and within these efforts, expanded the scope and reach of reproducible builds to yet more interested parties, established and continued strategic long-term thinking in a way not typically possible via remote channels, and brainstormed designs for tools to enable end-users to get the most benefit from reproducible builds.

Outside of these achievements in the hacking sessions kpcyrd made a breakthrough in Alpine Linux by producing the first reproducible package — specifically, py3-uritemplate — in this operating system. After this, progress was accelerated and by the denouement of our meeting the reproducibility status in Alpine reached 94%. In addition, Jelle van der Waa, Mattia Rizzolo and Paul Spooren discussed and implemented substantial changes to the database that underpins the testing framework that powers tests.reproducible-builds.org in order to abstract the schema in a distribution agnostic way, for example to allow submitting the results of attempts to verify officially distributed Arch Linux packages.

Lastly, Jan Nieuwenhuizen, David Terry and Vagrant Cascadian used three entirely-separate distributions (GNU Guix, NixOS and Debian) to produce a bit-for-bit identical GNU Mes binary despite using three different major versions of GCC and other toolchain components to build an initial binary, which was then used to build a final, bit-for-bit identical, binary of Mes.

The event was held at Priscilla, Queen of the Medina in Marrakesh, a location sui generis that stands for gender equality, female empowerment and the engagement of vulnerable communities locally through cultural activism. The event was open to anybody interested in working on Reproducible Builds issues, with or without prior experience.

A number of reports and blog posts have already been written, including for:

… as well as a number of tweets including ones from Jan Nieuwenhuizen celebrating progress in GNU Guix [] and Hannes [].

Distribution work

Within Debian, Chris Lamb categorised a large number of packages and issues in the Reproducible Builds notes.git repository, including identifying and creating markdown_random_email_address_html_entities and nondeterministic_devhelp_documentation_generated_by_gtk_doc.

In openSUSE, Bernhard published his monthly Reproducible Builds status update and filed the following patches:

Bernhard also filed bugs against:

The Yocto Project announced that it is running continuous tests on the reproducibility of its output which can observed through the oe-selftest runs on their build server. This was previously limited to just the mini images but this has now been extended to the larger graphical images. The test framework is available for end users to use against their own builds. Of particular interest is the production of binary identical results — despite arbitrary build paths — to allow more efficient builds through reuse of previously built objects, a topic covered in more-depth in a recent LWN article.

In Arch Linux, the database structure on tests.reproducible-builds.org was changed and the testing jobs updated to match and work has been started on a verification test job which rebuilds the officially released packages and verifies if they are reproducible or not. In the “hacking” time after our recent summit, several key packages were made reproducible, raising the amount of reproducible packages by approximately 1.5%. For example libxslt was patched with the patch originating from Debian and openSUSE.

Software development


diffoscope is our in-depth and content-aware diff-like utility that can locate and diagnose reproducibility issues. It is run countless times a day on our testing infrastructure and is essential for identifying fixes and causes of non-deterministic behaviour.

This month, diffoscope version 134 was uploaded to Debian unstable by Chris Lamb. He also made the following changes to diffoscope itself, including:

  • Always pass a filename with a .zip extension to zipnote otherwise it will return with an UNIX exit code of 9 and we fallback to displaying a binary difference for the entire file. []
  • Include the libarchive file listing for ISO images to ensure that timestamps – and not just dates – are visible in any difference. (#81)
  • Ensure that our autopkgtests are run with our pyproject.toml present for the correct black source code formatter settings. (#945993)
  • Rename the text_option_with_stdiout test to text_option_with_stdout [] and tidy some unnecessary boolean logic in the ISO9660 tests [].

In addition, Eli Schwartz fixed an error in the handling of the progress bar [] and Vagrant Cascadian added external tool reference for the zstd compression format for GNU Guix [] as well as updated the version to 133 [] and 134 [] in that distribution.

Project website & documentation

There was more work performed on our website this month, including:

In addition, Paul Spooren added a new page overviewing our Continuous Tests overview [], Hervé Boutemy made a number of improvements to our Java and JVM documentation expanding and clarifying various definitions as well as adding external links [][][][] and Mariana Moreira added a .jekyll-cache entry to the .gitignore file [].

Upstream patches

The Reproducible Builds project detects, dissects and attempts to fix as many currently-unreproducible packages as possible. We endeavour to send all of our patches upstream where appropriate. This month, we wrote a large number of such patches, including:

Test framework

We operate a comprehensive Jenkins-based testing framework that powers tests.reproducible-builds.org. This month, the following changes were made:

  • Holger Levsen:

    • Alpine:

      • Indicate where Alpine is being built on the node overview page. []
      • Turn off debugging output. []
      • Sleep longer if no packages are to be built. []
    • Misc:

      • Add some help text to our script to powercycle IONOS (neé Profitbricks) nodes. []
      • Install mosh everywhere. []
      • Only install ripgrep on Debian nodes. []
  • Mattia Rizzolo:

    • Arch Linux:

      • Normalise the suite names in the database. [][][][][]
      • Drop an unneeded line in the scheduler. []
    • Debian:

      • Fix a number of SQL errors. [][][][]
      • Use the debian.debian_support Python library over apt_pkg to perform version comparisons. []
    • Misc:

      • Permit other distributions to use our web-based package scheduling script. []
      • Reformat our power-cycling script using Black and use the Python logging module. []
      • Introduce a dsources database view to simplify some queries [] and add a build_type field to support both “doublerebuilds” and verification rebuilds [].
      • Move (almost) all the timestamps in the database schema from raw strings to “real” timestamp data types. []
      • Only block bots on jenkins.debian.net and tests.reproducible-builds.org, not any other sites. []

  • kpcyrd (for Alpine Linux):

    • Patch/install the abuild utility to one that is reproducible. [][][][]
    • Bump the number of build workers and collect garbage more frequently. [][][][]
    • Classify and display build results consistently. [][][]
    • Ensure that tmux and ripgrep is installed. [][]
    • Support building packages in the future. [][][]

Lastly, Paul Spooren removed the project overview from the bottom-left of the generated pages [] and the usual node maintenance was performed by Holger Levsen [] and Mattia Rizzolo [][].

Mailing list summary

There was considerable activity on our mailing list this month. Firstly, Bernhard M. Wiedemann posted a thread asking What is the goal of reproducible builds? in order to encourage refinements, extra questions and other contributions to what an end-user experience of reproducible builds should or even could look like.

Eli Schwartz then resurrected a previous thread titled Progress in rpm and openSUSE in 2019 to clarify some points around Arch Linux and Python package installation. Hans-Christoph Steiner followed-up to a separate thread originally started by Hervé Boutemy announcing the status of .buildinfo file support in the Java ecosystem, and Paul Spooren then informed the list that Google Summer of Code is now looking for projects for the latest cohort.

Lastly, Lars Wirzenius enquired about the status of Reproducible system images which resulted in a large number of responses.


If you are interested in contributing to the Reproducible Builds project, please visit our Contribute page on our website. However, you can get in touch with us via:

This month’s report was written by Arnout Engelen, Bernhard M. Wiedemann, Chris Lamb, Hervé Boutemy, Holger Levsen, Jelle van der Waa, Lukas Puehringer and Vagrant Cascadian. It was subsequently reviewed by a bunch of Reproducible Builds folks on IRC and the mailing list.

06 January, 2020 12:54PM

hackergotchi for Julien Danjou

Julien Danjou

Atomic lock-free counters in Python

Atomic lock-free counters in Python

At Datadog, we're really into metrics. We love them, we store them, but we also generate them. To do that, you need to juggle with integers that are incremented, also known as counters.

While having an integer that changes its value sounds dull, it might not be without some surprises in certain circumstances. Let's dive in.

The Straightforward Implementation

class SingleThreadCounter(object):
	def __init__(self):
    	self.value = 0
    def increment(self):
        self.value += 1

Pretty easy, right?

Well, not so fast, buddy. As the class name implies, this works fine with a single-threaded application. Let's take a look at the instructions in the increment method:

>>> import dis
>>> dis.dis("self.value += 1")
  1           0 LOAD_NAME                0 (self)
              2 DUP_TOP
              4 LOAD_ATTR                1 (value)
              6 LOAD_CONST               0 (1)
              8 INPLACE_ADD
             10 ROT_TWO
             12 STORE_ATTR               1 (value)
             14 LOAD_CONST               1 (None)
             16 RETURN_VALUE

The self.value +=1 line of code generates 8 different operations for Python. Operations that could be interrupted at any time in their flow to switch to a different thread that could also increment the counter.

Indeed, the += operation is not atomic: one needs to do a LOAD_ATTR to read the current value of the counter, then an INPLACE_ADD to add 1, to finally STORE_ATTR to store the final result in the value attribute.

If another thread executes the same code at the same time, you could end up with adding 1 to an old value:

Thread-1 reads the value as 23
Thread-1 adds 1 to 23 and get 24
Thread-2 reads the value as 23
Thread-1 stores 24 in value
Thread-2 adds 1 to 23
Thread-2 stores 24 in value

Boom. Your Counter class is not thread-safe. 😭

The Thread-Safe Implementation

To make this thread-safe, a lock is necessary. We need a lock each time we want to increment the value, so we are sure the increments are done serially.

import threading

class FastReadCounter(object):
    def __init__(self):
        self.value = 0
        self._lock = threading.Lock()
    def increment(self):
        with self._lock:
            self.value += 1

This implementation is thread-safe. There is no way for multiple threads to increment the value at the same time, so there's no way that an increment is lost.

The only downside of this counter implementation is that you need to lock the counter each time you need to increment. There might be much contention around this lock if you have many threads updating the counter.

On the other hand, if it's barely updated and often read, this is an excellent implementation of a thread-safe counter.

A Fast Write Implementation

There's a way to implement a thread-safe counter in Python that does not need to be locked on write. It's a trick that should only work on CPython because of the Global Interpreter Lock.

While everybody is unhappy with it, this time, the GIL is going to help us. When a C function is executed and does not do any I/O, it cannot be interrupted by any other thread. It turns out there's a counter-like class implemented in Python: itertools.count.

We can use this count class as our advantage by avoiding the need to use a lock when incrementing the counter.

If you read the documentation for itertools.count, you'll notice that there's no way to read the current value of the counter. This is tricky, and this is where we'll need to use a lock to bypass this limitation. Here's the code:

import itertools
import threading

class FastWriteCounter(object):
    def __init__(self):
        self._number_of_read = 0
        self._counter = itertools.count()
        self._read_lock = threading.Lock()

    def increment(self):

    def value(self):
        with self._read_lock:
            value = next(self._counter) - self._number_of_read
            self._number_of_read += 1
        return value

The increment code is quite simple in this case: the counter is just incremented without any lock. The GIL protects concurrent access to the internal data structure in C, so there's no need for us to lock anything.

On the other hand, Python does not provide any way to read the value of an itertools.count object. We need to use a small trick to get the current value. The value method increments the counter and then gets the value while subtracting the number of times the counter has been read (and therefore incremented for nothing).

This counter is, therefore, lock-free for writing, but not for reading. The opposite of our previous implementation

Measuring Performance

After writing all of this code, I wanted to make sure how the different implementations impacted speed. Using the timeit module and my fancy laptop, I've measured the performance of reading and writing to this counter.

Operation SingleThreadCounter FastReadCounter FastWriteCounter
increment 176 ns 390 ns 169 ns
value 26 ns 26 ns 529 ns
Atomic lock-free counters in Python

I'm glad that the performance measurements in practice match the theory 😅. Both SingleThreadCounter and FastReadCounter have the same performance for reading. Since they use a simple variable read, it makes absolute sense.

The same goes for SingleThreadCounter and FastWriteCounter, which have the same performance for incrementing the counter. Again they're using the same kind of lock-free code to add 1 to an integer, making the code fast.


It's pretty obvious, but if you're using a single-threaded application and do not have to care about concurrent access, you should stick to using a simple incremented integer.

For fun, I've published a Python package named fastcounter that provides those classes. The sources are available on GitHub. Enjoy!

06 January, 2020 10:47AM by Julien Danjou

January 05, 2020

hackergotchi for Michael Prokop

Michael Prokop

Revisiting 2019

Mika on the Drums, picture by Gregor

Mainly to recall what happened last year and to give thoughts and to plan for the upcoming year(s) I’m once again revisiting my previous year (previous editions: 2018, 2017, 2016, 2015, 2014, 2013 + 2012).

In terms of IT events, I attended Grazer Linuxdays 2019 and gave a talk (Best Practices in der IT-Administration, Version 2019) and was interviewed by Radio Helsinki there. With the Grml project, we attended the Debian Bug Squashing Party in Salzburg in April. I also visited a meeting of the Foundation for Applied Privacy in Vienna. Being one of the original founders I still organize the monthly Security Treff Graz (STG) meetups. In 2020 I might attend DebConf 20 in Israel (though not entirely sure about it yet), will definitely attend Grazer Linuxdays (maybe with a talk about »debugging for sysadmins« or alike) and of course continue with the STG meetups.

I continued to play Badminton in the highest available training class (in german: “Kader”) at the University of Graz (Universitäts-Sportinstitut, USI). I took part in the Zoo run in Tiergarten Schönbrunn (thanks to an invitation by a customer).

I started playing the drums at the »HTU Big Band Graz« (giving a concert on 21st of November). Playing in a big band was like a dream come true, being a big fan of modern Jazz big bands since being a kid and I even played the drums in a big band more than 20 years ago, so I’m back™. I own a nice e-drum set and recently bought a Zildjian Gen16 cymbal set and also own a master-keyboard (AKA MIDI keyboard) for many years, which is excellent for recording. But in terms of “living room practicality”, I wanted something more piano alike, and we bought a Yamaha CLP-645 B digital piano, which my daughters quite regularly use and now and then I manage to practice on it as well. As you might guess, I want to make music a more significant part of my life again.

I visited some concerts, including Jazz Redoute, Jazzwerkstatt Graz, Billy Cobham’s Crosswinds Project, Jazz Night Musikforum Viktring, Gnackbruch evening with AMMARITE, a concert of the Kärntner Sinfonieorchester, Steven Wilson’s To The Bone tour, Sting’s My Songs tour and the Corteo show of Cirque du Soleil. I took some local trips in Graz, including a Murkraftwerk Graz tour and a »Kanalführung«.

Business-wise it was the sixth year of business with SynPro Solutions, and we moved the legal form of our company from GesnbR to GmbH. No big news but steady and ongoing work with my other business duties Grml Solutions and Grml-Forensic.

I also continued with taking care of our kids every Monday and half another day of the week – which is still challenging now and then with running your own business, but so absolutely worth it. With a kid going to school, it was quite some change for my schedule and day planning as well. Now having a fixed schedule for most of the days, the Sonos soundbox wakes us up with Ö1 news and its Ö1 signature tune Monday to Friday. Thanks to Pavlovian conditioning, when waking up on Saturdays and Sundays I also hear the Ö1 signature tune in my head while no radio is present then. :)

I tried to minimize my Amazon usage as much as possible and will try to continue doing so in 2020 as well.

I had quite some troubles with my Vespa PX125, hopefully things are sorted out nowadays though. *knockingonwood*

After being ~20 years in the Usenet (mostly de.* + at.* + tu-graz.*) I pretty much gave it up.

Book reading became more of a habit again, and I managed to complete 42 books (see Bookdump 01/2019 and Bookdump 02/2019). I noticed that what felt like good days for me always included reading books, and want to keep my reading pace for 2020.

05 January, 2020 10:58PM by mika

hackergotchi for Marco d'Itri

Marco d'Itri

Debian support for libxcrypt

glibc 2.29-7 and libxcrypt 1:4.4.10-10 today entered Debian testing: crypt(3) and the other related library functions in libcrypt from now on will be provided by libxcrypt instead of glibc.

After 18 months of packaging work, Debian finally supports modern password hashing methods like yescrypt: the details about them are documented in crypt(5).

For the time being there is still no support for libxcrypt in our release of PAM, but hopefully the Debian maintainer will update the package soon and this will allow using yescrypt by default.

If you want to test now the new algorithms then you can generate a password using my mkpasswd program and copy it to /etc/shadow:

# echo "marco:$(echo 12345 | mkpasswd --stdin)" | chpasswd --encrypted

05 January, 2020 02:18AM

January 04, 2020

hackergotchi for Shirish Agarwal

Shirish Agarwal

Indian Economy, NPR, NRC and Crowd Control Part – II

Protests and their history

A Happy New Year to all. While I would have loved to start on a better note, situations are the way they are. Before starting with the prickly process of NPR, NRC let me just focus on the protests themselves. Now protests either in India or abroad are not a new thing. While thinking on this, I found one of the modern, medieval recorded entries of protests to be in 12th century Paris, and just like most strikes, this one was for rights of liberty, freedom and price increase. The famous or infamous University of Paris strike 1229 , There have been so many strikes and protests worldwide which changed the world, in recent memory the protests against American involvement in Vietnam , The Montgomery bus boycott by Rosa Parks , the protests that I became aware in South Africa, UCT about Rhodes must fall movement . I would be forever grateful to Bernelle for sharing with us the protests that had happened the year before near the Sarah Bartman Hall. Closer home i.e. in India, we have had a rich history of protests especially during the Colonial period and the Indian Freedom movement, as well as afterwards, i.e. after india became free. Whether it was the Navnirman Andolan or the Great Bombay Textile Strike which actually led to industries moving out of Mumbai. My idea of sharing above strikes and protests has been that protests are not a new thing in India and have been part of India socio-political culture throughout its history. I am sure there were also protests during the medieval period but not going that far as it would not add value to the post currently. It may be a good idea to share about that in some other blog post perhaps.

The protests against NPR, NRC and CAA

So, let’s start with what these acronyms are and why people are protesting are against it . The first acronym is National Population Register (NPR) . Now while the Government says that NPR is nothing but the census which is done by GOI every year, there is a difference. There are few things which make it different from earlier years, those are, birth certificates, ‘date and birth of parents’ and ‘last place of residence’ . The problem starts and ends with these two points for NPR apart from biometric information which again has issues, both for rich and poor alike . Let me explain what is the problem therein, using my own use-case or history which probably can be multiplied by probably millions of people of my age and lesser and elder to me.

Now one could ask naively, what is wrong in birth certificates, in theory and today’s day and age, perhaps not, but everything has a context and a time and place. While I was born in Pune, in 1975, the Registration and Births Act had been recently passed in 1969. So neither the Municipal Corporation of that time was active and nor was that a statutory requirement in those times. To add to that, my mother had to go at least 10-15 times to the Municipal Corporation in order to secure my birth certificate even though there was no name. This brings to another issue, in those times, almost till date, the mortality rate of newborns have been high. While we have statistics of the last 20 odd years which do show some change, one can only guess what the infant mortality rates would have been in the 60’s and the 70’s . Apart from that, most deliveries happened at home with a mid-wife rather than in a nursing home. This is still the norm today in many cities and hinterland as well. Recently, there was news of 100 babies who died in Kota, Rajasthan. While earlier they were being given clean chits, it seems most neonatal units which should house only one child were housing three children. Hence the large number of deaths. Probably, most of the parents were poor and the administration while showing that each child was given a separate neonatal unit were put together. The corruption whether in Indian public or private hospitals deserves its own blog post. I had shared some of it in the blog post no country for women or doctors.

But that apart, in those days because babies died, many children didn’t get the name till s/he was of kinder-garden, school going age. In my situation was a bit more different and difficult as my parents had separated and there was possibility that my father may go for a custody battle which never happened. Now apart from proving my own identity even though I have all the papers, I might still need to procure more, I would need papers or documentation proving the relationship between mother and I . Now I can’t produce father because he is no more apart from his death certificate, with my mother, I would have to get and submit most probably a DNA test which is expensive to say the least. I know of some labs who also charge depending upon many genetic markers you are looking for, the more the better and costs go up like that. Now here itself two questions arise with the NPR itself.

a. How many children would have proper birth certificates, more so the ones who live either below poverty line or just above ? Many of them don’t have roof over their head or have temporary shelters, where would they have place to get and put such documents. Also, as many of them cannot either read or write, they are usually suspicious of Government papers (and with good reason) . Time and again, the Government of the day has promised one thing and done another. Mostly to do with property rights and exclusion. Why to go far, just a few days back Sonam wangchuck, the engineer, who shared his concerns about Ladakh and the fragile ecosystem of Ladakh in Himalayas due to unchecked ‘development’ as been promised by the Prime Minister bringing it par to Delhi. Sadly, many people do not know that Himalayas is the youngest geologically speaking while the oldest are in South Africa (thanks to Debconf for that bit of info. as well.) . While they celebrate the political freedom from Kashmir, they do have concerns as being with Kashmir, they enjoyed land rights and rights to admission to Indians and foreigners alike, this they have now lost. A region which is predominantly buddhist in nature is being introduced to sex tourism and massage parlours which are not needed. If possible, people should see Mr. Wangchuk’s talk on Ted talks or any of the work the gentleman has done but this is getting off-topic here.

b. What about people of my age or above me. Would all of us would become refugees in our own lands ? What about those who go from place to place for work ? What about Government servants themselves ? Those who work in Central Government, many of them have and are supposed to change places every 3-4 years. My grandfather (from mother’s side) toured all of India due to his job. My mother also got transferred a few times, although I stayed in Pune. This puts up questions which make it difficult for many to answer if they were to do it truthfully as nobody would have full papers. There are also no guarantees that just having the right papers would make it right. It is very much possible that the ‘babu’ or beareaucrat sitting at the other end would demand money. I am sure there was lot of black money generated during NRC in Assam. This would happen at NPR stage itself.

c. The third question is what would happen to those who are unable to prove their citizenship at the beginning itself. They probably would need to go to court of law. What happens to their job, property and life as it is. As it is, most businesses have a slack of 40-50%, this will become more pronounced. Would we be stateless citizens in our own land ?


Somehow, let’s say you managed to have yourself included in NPR, it doesn’t mean that you will be included in NRC or National Register of Citizens. For this, one may have to rinse and repeat. They may ask more invasive questions as never before. The possible exploitation of people by the state would be as never seen before. Most of the majority in India think of Germany and how it became an industrious house and they think, they became industrious because they persecuted Jews. In fact, I believe it was the opposite. As have shared before on the blog itself, numerous times, if the Jews had been part of Germany, Germany may have prospered many times over. If the Jews could make Israel so powerful, where would Germany would have been today ? Also not many people know about the Marshall Plan which perhaps laid the foundation of the European Union as we know today but that may be part of another blog post, another day. I would probably do a part III as there are still aspects of the whole issue which I haven’t touched upon. I might do it tomorrow or few days after. Till later.

04 January, 2020 10:14PM by shirishag75

Iustin Pop

System load and ping latency strangeness

So, instead of a happy new year post or complaining about Debian’s mailing list threads (which make me very sad), here’s an interesting thing (I think).

Having made some changes to the local network recently, I was surprised at the variability in ping latency on the local network but also to localhost! I thought, well, such is Linux, yada yada, it’s a kernel build with CONFIG_NO_HZ_IDLE=y, etc. However looking at the latency graph an hour ago showed something strange: latencies stabilised… and then later went bad again. Huh?

This is all measured via smokeping, which calls fping 10 times in a row, and records both average and spread of the values. For “stable”, I’m talking here about a somewhat even split between 10µsec and 15µsec (for the 10-ping average), with very consistent values, and everything between 20µsec and 45µsec, which is a lot.

For the local-lan host, it’s either consistently 200µsec vs 200-300µsec with high jitter (outliers up to 1ms). This is very confusing.

The timing of the “stable” periods aligned with times when I was running heavy disk I/O. Testing quickly confirmed this:

  • idle system: localhost : 0.03 0.04 0.04 0.03 0.03 0.03 0.03 0.03 0.03 0.03
  • pv /dev/md-raid5-of-hdds: localhost : 0.02 0.01 0.01 0.01 0.01 0.01 0.03 0.03 0.03 0.02
  • pv /dev/md-raid5-of-ssds: localhost : 0.03 0.01 0.01 0.01 0.01 0.02 0.02 0.02 0.02 0.02
  • with all CPUs at 100%, via stress -c $N: localhost : 0.02 0.00 0.01 0.00 0.01 0.01 0.01 0.01 0.01 0.01
  • with CPUs idle, but with governor performance so there’s no frequency transition: localhost : 0.01 0.15 0.03 0.03 0.03 0.03 0.03 0.03 0.03 0.03

So, this is not CPU frequency transitions, at least as seen by Linux. This is purely CPU load, and, even stranger, it’s about single core load. Running the following in parallel:

  • taskset -c 8 stress -c 1 and
  • taskset -c 8 fping -C 10 localhost

Results in the awesome values of:

localhost : 0.01 0.01 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.01

Now, that is what I would expect :) Even more interesting, running stress on a different CPU (any different CPU) seems to improve things, but only by half (using ping which has better resolution).

To give a more graphical impression of the latencies involved (staircase here is due to fping resolution bug, mentioned below):

Smokeping: localhost Smokeping: localhost
Smokeping: host on local net Smokeping: host on local net
Localhost CPU usage Localhost CPU usage

Note that plain I/O wait (the section at the top) doesn’t affect latency; only actual CPU usage, as seen at Fri 02:00-03:00 and then later 11:00-21:00 and (much higher) Sat 10:15-20:00.

If you squint, you can even correlate lower CPU usage on Fri 16:00-21:00 to slightly increased latencies.

Localhost CPU frequency caling Localhost CPU frequency caling

Does this all really matter? Not really, not in any practical sense. Would I much prefer clean, stable ping latencies? Very much so.

I’ve read the documentation on no HZ, which tells me I should be rebooting about 20 or 30 times with all kinds of parameter combinations and kernel builds, and that’s a bit too much from my free time. So maybe someone has some idea about this, would be very happy to learn what I can tune to make my graphs nicer :)

I’ve also tested ping from another host to this host, and high CPU usage results in lower latencies. So it seems to be not user-space related, but rather kernel latencies?!

I’ve also thought this might be purely an fping issue; however, I can clearly reproduce it simply by watching ping localhost which running (or not) stress -c $N; the result is ~10-12µsec vs. ~40µsec.

Thanks in advance for any hints.

04 January, 2020 09:55PM

Thorsten Alteholz

My Debian Activities in December 2019

FTP master

This month I accepted 450 packages and rejected 61. The overall number of packages that got accepted was 481.

Debian LTS

This was my sixty sixth month that I did some work for the Debian LTS initiative, started by Raphael Hertzog at Freexian.

This month my all in all workload has been 16.5h. During that time I did LTS uploads of:

  • [DLA 2035-1] libpgf security update for one CVE
  • [DLA 2039-1] libvorbis security update for two CVEs
  • [DLA 2040-1] harfbuzz security update for one CVE
  • [DLA 2043-1] gdk-pixbuf security update for five CVEs
  • [DLA 2043-2] gdk-pixbuf regression update
  • [DLA 2047-1] cups security update for one CVE
  • [DLA 2050-1] php5 security update for four CVEs
  • [DLA 2052-1] libbsd security update for one CVE
  • [DLA 2055-1] igraph security update for one CVE

Last but not least I did some days of frontdesk duties and started to work on the sqlite3 package.

Debian ELTS

This month was the nineteenth ELTS month.

During my allocated time I uploaded:

  • ELA-202-1 for gdk-pixbuf
  • ELA-202-2 for gdk-pixbuf
  • ELA-204-1 for php5

I also did some days of frontdesk duties.

Other stuff

This month I uploaded new upstream versions of …

I improved packaging of …

As nobody really used them, I removed the lam4 and mpich2 version of meep. Now only the serial version, the openmpi- and the mpi-default-version are available. Please complain in case you need one of the other versions again.

I also uploaded all meep packages, libctl and mpb to unstable.

On my Go challenge I uploaded the source-only versions of golang-github-boj-redistore, golang-github-dchest-uniuri, golang-github-jackc-fake, golang-github-joyent-gocommon, golang-github-mattetti-filebuffer, golang-github-nrdcg-goinwx, golang-github-pearkes-dnsimple, golang-github-soniah-dnsmadeeasy, golang-github-vultr-govultr, golang-github-zorkian-go-datadog-api.
New Go packages I uploaded were: golang-github-hashicorp-terraform-svchost, golang-github-apparentlymart-go-cidr, golang-github-bmatcuk-doublestar, golang-github-cactus-go-statsd-client, golang-github-corpix-uarand, golang-github-cyberdelia-heroku-go

04 January, 2020 07:15PM by alteholz

hackergotchi for Jonathan Carter

Jonathan Carter

Free Software Activities (2019-12)

Watching people windsurf at Blouberg beach

A lot has happened in Debian recently, I wrote seperate blog entries about that but haven’t had the focus to finish them up, maybe I’ll do that later this month. In the meantime, here are some uploads I’ve done during the month of December…

Debian packaging work

2019-12-02: Upload package calamares (3.2.17-1) to Debian unstable.

2019-12-03: Upload package calamares ( to Debian unstable.

2019-12-04: Upload package python3-flask-caching to Debian unstable.

2019-12-04: File removal request for python3-flask-cache (BTS: #946139).

2019-12-04: Upload package gamemode (1.5~git20190812-107d469-3) to Debian unstable.

2019-12-11: Upload package gnome-shell-extension-draw-on-your-screen (5-1) to Debian unstable.

2019-12-11: Upload package xabacus (8.2.3-1) to Debian unstable.

2019-12-11: Upload package gnome-shell-extension-gamemode (4-1) to Debian unstable.

2019-12-11: Upload package gamemode (1.5~git20190812-107d469-4) to Debian unstable.

Debian package sponsoring/reviewing

2019-12-02: Sponsor package scrcpy (1.11+ds-1) for Debian unstable (mentors.debian.net request).

2019-12-03: Sponsor package python3-portend (2.6-1) for Debian unstable (Python team request).

2019-12-04: Merge MR#1 for py-postgresql (DPMT).

2019-12-04: Merge MR#1 for pyphen (DPMT).

2019-12-04: Merge MR#1 for recommonmark (DPMT).

2019-12-04: Merge MR#1 for python-simpy3 (DPMT).

2019-12-04: Merge MR#1 for gpxpy (DPMT).

2019-12-04: Sponsor package gpxpy (1.3.5-2) (Python team request).

2019-12-04: Merge MR#1 for trac-subcomponents (DPMT).

2019-12-04: Merge MR#1 for debomatic (PAPT).

2019-12-04: Merge MR#1 for archmage (PAPT).

2019-12-04: Merge MR#1 for ocrfeeder (PAPT).

2019-12-04: Sponsor package python3-tempura (1.14.1-2) for Debian unstable (Python team request).

2019-12-04: Sponsor package python-sabyenc (4.0.1-1) for Debian experimental (Python team request).

2019-12-04: Sponsor package python-yenc (0.4.0-7) for Debian unstable (Python team request).

2019-12-05: Sponsor package python-gntp (1.0.3-1) for Debian unstable (Python team request).

2019-12-05: Sponsor package python-cytoolz (0.10.1-1) for Debian unstable (Python team request).

2019-12-22: Sponsor package mwclient (0.10.0-2) for Debian unstable (Python team request).

2019-12-22: Sponsor package hyperlink (19.0.0-1) for Debian unstable (Python team request).

2019-12-22: Sponsor package drf-generators (0.4.0-1) for Debian unstable (Python team request).

2019-12-22: Sponsor package python-mongoengine (0.18.2-1) for Debian unstable (Python team request).

2019-12-22: Sponsor package libcloud (2.7.0-1) for Debian unstable (Python team request).

2019-12-22: Sponsor package pep8-naming (0.9.1-1) for Debian unstable (Python team request).

2019-12-23: Sponsor package python-django-braces (1.13.0-2) for Debian unstable (Python team request).

04 January, 2020 11:34AM by jonathan

Elana Hashman

KubeCon NA 2019 Talk Resources

At KubeCon + CloudNativeCon North America 2019, I co-presented "Weighing a Cloud: Measuring Your Kubernetes Clusters" with Han Kang. Here's some links and resources related to my talk, for your reference.

Weighing a Cloud: Measuring Your Kubernetes Clusters

Related readings

I'm including these documents for reference to add some context around what's currently happening (as of 2019Q4) in the Kubernetes instrumentation SIG and wider ecosystem.

Note that GitHub links are pinned to their most recent commit to ensure they will not break; if you want the latest version, make sure to switch the branch to "master".

04 January, 2020 05:00AM by Elana Hashman

January 03, 2020

hackergotchi for Kurt Kremitzki

Kurt Kremitzki

November and December Update for FreeCAD & Debian Science

Hello again! This new year's update announces some interesting new beginnings for the FreeCAD project, though it's a little short since I got some much needed vacation time over the last two months.

OpenFOAM on One Core? Only 92 Hours! (for mipsel)

OpenFOAM & ParaView flow simulation.

In November a strange bug was found in the OpenFOAM package which led to only one core being used during builds, even though the logs reported an N core build. In the worst case scenario, on the mipsel architecture, this led to an increase in build times from 17 to 92 hours! I did some troubleshooting on this but found it a bit difficult since OpenFOAM uses a bespoke build system called wmake. I found myself wishing for the simplicity of CMake, and found there was an experimental repo implementing support for it but it didn't seem to work out of the box or with a bit of effort. I wonder if there's any consideration amongst OpenFOAM developers in moving away from wmake?

Anyway, OpenFOAM ended up getting removed from Debian Testing, but thankfully Adrian Bunk identified the problem, which is that the environment variable MAKEFLAGS was getting set to 'w' for some reason, and thus falling through the wmake code block that set up a proper parallel build for OpenFOAM. So, unsatisfyingly, as a workaround I uploaded the latest OpenFOAM version, 1906.191111, with unexport MAKEFLAGS. It would be nice to find an explanation, but I didn't spend much more time digging.

So, to end on the good news, the newest bugfix version of the OpenFOAM 1906 release, from November 11th 2019, is available for use going into 2020!

Trip to FOSDEM 2020 and MiniDebCamp at the Hackerspace Brussels

FOSDEM logo.

It was a bit last minute, but I finally decided to attend FOSDEM 2020. I had balked a bit at the cost since flights from the US are around $900, but decided it would be an important opportunity for FreeCAD developers and community to get together and possibly do some important work. Thankfully, Yorik and other senior FreeCAD developers thought it would be a good use of the project's Bountysource money to cover the cost of one ticket, split in half between myself and sliptonic, a developer from Missouri. He focuses on the Path workbench and FreeCAD in CAM applications, an area I'm interested in moving into as I now have such machining equipment available to me through my local ATX Hackerspace. The three of us will be giving a talk, "Open-source design ecosystems around FreeCAD", at 11:20 on Saturday, so please come by and say hi if you're able to!

I'll be staying for a few days before and after FOSDEM, including attending the MiniDebCamp at Hackerspace Bruxelles on Thursday & Friday, interested in anything Debian/FreeCAD related, so I look forward to getting a lot of work done indeed!

Looking at BRL-CAD for Debian

Developer working on BRL-CAD, circa 1980

Lead developer Mike Muuss works on the XM-1 tank in BRL‑CAD on a PDP‑11/70 computer system, circa 1980.

For the past several summers, FreeCAD has participated in the Google Summer of Code program under an umbrella organization led by Sean Morrison of BRL-CAD. BRL-CAD is a very interesting bit of software with a long history, in fact the oldest known public version-controlled codebase in the world still under development, dating back to 1983-12-16 00:10:31 UTC. It is inspired by the development ideas of the era, a sort of UNIX philosophy for CAD, made up of many small tools doing one thing well and meant to be used in a normal UNIXy way, being piped into one another and so forth, with a unifying GUI using those tools. Since it's made up of BSD/LGPL licensed code, it ought to be available as part of the Debian Science toolkit, where it may be useful for FreeCAD as an included alternative CAD kernel to the currently exclusive OpenCASCADE. For example, fillets in OpenCASCADE are somewhat buggy and unmaintainably implemented such that an upstream rewrite is the only hope for long-term improvement. BRL-CAD could potentially improve FreeCAD in areas like this.

It turns out a Debian Request for Packaging bug for BRL-CAD has been open since 2005. I plan to close it! It turns out there's already existing Debian packaging work, too, though it's quite a few years old and thus some adaptation still is required.

PySide 2 and KDE Maintenance in Debian

Recently, FreeCAD has been unbuildable in Debian Sid because of issues related to PySide 2 and the Python 3.8 migration. This is complicated by the fact that the upstream fix has been released but in version 5.14.0, which builds fine with Qt 5.14, although Sid currently has 5.12. Furthermore, the PySide 2 package itself isn't building at the moment either! Since FreeCAD depends on PySide 2 and Qt, and I use the Qt-based KDE as my desktop, it seems like taking on maintenance of PySide 2 is something I should do to get started in this realm. However, the Qt/KDE Team's packaging practices and tools are rather different than the ones I'm used to for Science Team packages. This makes sense: Science Team packages are very often a single Git repo, but Qt5 for example is really 44 Git submodules smushed together. As such, things are a bit different! Once I get things taken care of for the package, I will try to write up some notes to help others interested in getting started, especially since KDE packaging could use some help.

FreeCAD Sysadmin Woes Begone: DigitalOcean Sponsorship

DigitalOcean's "Powered By" blue badge lgoo.

I'm very happy to announce that the FreeCAD project is now among the many open source software sponsorships by DigitalOcean.

One of the first things I did when interested in FreeCAD was to try to take on the responsibility of maintaining the project's infrastructure, since that would free up time for people to work on FreeCAD itself. FreeCAD's 17 years old now, and some of our infrastructure stack is about as dated. However, it isn't easy to just move things, I had to get things up to speed first and try to minimize disruption, so it's been a slow process. I'll go into more details in a technical blog post on the matter after I've finished our migration, hopefully by the end of this month, including details on our new setup, with the goal of allowing people to get set up with a dev environment of our project tools so you can do some hacking on things yourself and help out if possible.

Thanks for your support

I appreciate any feedback you might have.

You can get in touch with me via Twitter @thekurtwk.

If you'd like to donate to help support my work, there are several methods available on my site.

03 January, 2020 03:25AM by Kurt Kremitzki

January 02, 2020

hackergotchi for Ben Hutchings

Ben Hutchings

Debian LTS work, December 2019

I was assigned 16.5 hours of work by Freexian's Debian LTS initiative and carried over 3.75 hours from November. I worked all 20.25 hours this month.

I prepared and, after review, released Linux 3.16.79. I rebased the Debian package onto 3.16.79 and sent out a request for testing.

I also released Linux 3.16.80, but haven't yet rebased the Debian package onto this.

02 January, 2020 04:24PM

hackergotchi for Jonathan Dowland

Jonathan Dowland

Linux Desktop

Happy New Year!

It's been over two years since writing back on the Linux desktop, and I've had this draft blog post describing my desktop setup sitting around for most of that time. I was reminded of it by two things recently: an internal work discussion about "the year of the linux desktop" (or similar), and upon discovering that the default desktop choice for the current Debian release ("Buster") uses Wayland, and not the venerable X. (I don't think that's a good idea).

GNOME 3 Desktop

I already wrote a little bit about my ethos and some particulars, so I'll not repeat myself here. The version of GNOME I am using is 3.30.2. I continue to rely upon Hide Top Bar, but had to disable TopIcons Plus which proved unstable. I use the Arc Darker theme to shrink window title bars down to something reasonable (excepting GTK3 apps that insist on stuffing other buttons into that region).

Although I mostly remove or hide things, I use one extension to add stuff: Suspend Button, to add a distinct "Suspend" button. The GNOME default was, and seem to remain, to offer only a "Power off" button, which seems ludicrous to me.

I spend a lot of time inside of Terminals. I use GNOME terminal, but I disable or hide tabs, the menubar and the scrollbar. Here's one of my top comfort tips for working in terminals: I set the default terminal size to 120x32, up from 80x24. It took me a long time to realise that I habitually resized every terminal I started.

I've saved the best for last: The Put Windows GNOME shell extension allows you to set up keyboard shortcuts for moving and resizing the focussed window to different regions of the desktop. I disable the built-in shortcuts for "view splits" and rely upon "Put Windows" instead, which is much more useful: with the default implementation, once "snapped", you can't resize windows (widen or narrow them) unless you first "unsnap" them. But sometimes you don't want a 50/50 split. "Put Windows" doesn't have that restriction; but it also lets you cycle between different (user-configurable) splits: I use something like 50/50, 30/70, 70/30. It also lets you move things to corners as well as sides, and also top/bottom splits, which is very useful for comparing spreadsheets (as I pointed out eight years ago).

"Put Windows" really works marvels and entirely replaces SizeUp that I loved on Mac.

02 January, 2020 03:29PM

Tim Retout

Sylvain Beucler

Debian LTS and ELTS - December 2019

Debian LTS Logo

Here is my transparent report for my work on the Debian Long Term Support (LTS) and Debian Extended Long Term Support (ELTS), which extend the security support for past Debian releases, as a paid contributor.

In December, the monthly sponsored hours were split evenly among contributors depending on their max availability - I was assigned 16.5h for LTS (out of 30 max) and 16.5h for ELTS (max).

This is less than usual, AFAICS due to having more team members requesting more hours (while I'm above average), and less unused hours given back (or given back too late).

ELTS - Wheezy

  • libonig: finish work started in November:
  • CVE-2019-19203/libonig: can't reproduce, backport non-trivial likely to introduce bugs,
  • CVE-2019-19012,CVE-2019-19204,CVE-2019-19246/libonig: security upload
  • libpcap: attempt to recap vulnerabilities mismatch (possibly affecting ELA-173-1/DLA-1967-1); no follow-up from upstream
  • CVE-2019-19317,CVE-2019-19603,CVE-2019-19645/sqlite3: triage: not-affected (development version only)
  • CVE-2019-1551/openssl: triage: not-affected; discuss LTS triage rationale
  • CVE-2019-14861,CVE-2019-14870/samba: triage: not-affected
  • CVE-2019-19725/sysstat: triage: not-affected (vulnerable code introduced in v11.7.1)
  • CVE-2019-15845,CVE-2019-16201,CVE-2019-16254,CVE-2019-16255/ruby1.9.1: security upload

LTS - Jessie

  • CVE-2019-19012,CVE-2019-19204,CVE-2019-19246/libonig: shared work with ELTS, security upload
  • libpcap: shared work with ELTS
  • libav: finish work started in November:
  • CVE-2018-18829/libav: triage: postponed (libav-specific issue, no patch)
  • CVE-2018-11224/libav: triage: postponed (libav-specific issue, no patch)
  • CVE-2017-18247/libav: triage: ignored (not reproducible, no targeted patch)
  • CVE-2017-18246/libav: triage: ignored (not reproducible)
  • CVE-2017-18245/libav: reproduce, track down fix in ffmpeg
  • CVE-2017-18244/libav: triage: ignored (not reproducible)
  • CVE-2017-18243/libav: triage: ignored (not reproducible)
  • CVE-2017-18242/libav: triage: ignored (not reproducible)
  • CVE-2017-17127/libav: reproduce, track down fix in ffmpeg
  • CVE-2016-9824/libav: triage: ignored: usan (undefined sanitized) warning only, no patch
  • CVE-2016-9823/libav: triage: ignored: usan (undefined sanitized) warning only, no patch
  • CVE-2016-5115/libav: triage: postpone due different (indirect mplayer) vulnerability and lack of time
  • CVE-2017-17127,CVE-2017-18245,CVE-2018-19128,CVE-2018-19130,CVE-2019-14443,CVE-2019-17542/libav: security upload


02 January, 2020 10:18AM

January 01, 2020

Balasankar 'Balu' C

FOSS contributions in 2019


I have been interested in the concept of Freedom - both in the technical and social ecosystems for almost a decade now. Even though I am not a harcore contributor or anything, I have been involved in it for few years now - as an enthusiast, a contributor, a mentor, and above all an evangelist. Since 2019 is coming to an end, I thought I will note down what all I did last year as a FOSS person.


My job at GitLab is that of a Distribution Engineer. In simple terms, I have to deal with anything that a user/customer may use to install or deploy GitLab. My team maintains the omnibus-gitlab packages for various OSs, docker image, AWS AMIs and Marketplace listings, Cloud Native docker images, Helm charts for Kubernetes, etc.

My job description is essentially dealing with the above mentioned tasks only, and as part of my day job I don’t usually have to write and backend Rails/Go code. However, I also find GitLab as a good open source project and have been contributing few features to it over the year. Few main reasons I started doing this are

  1. An opportunity to learn more Rails. GitLab is a pretty good project to do that, from an engineering perspective.
  2. Most of the features I implemented are the ones I wanted from GitLab, the product. The rest are technically simpler issues with less complexity(relates to the point above, regarding getting better at Rails).
  3. I know the never-ending dilemma our Product team goes through to always maintain the balance of CE v/s EE features in every release, and prioritizing appropriate issues from a mountain of backlog to be done on each milestone. In my mind, it is easier for both them and me if I just implemented something rather than asked them to schedule it to be done by a backend team, so that I cane enjoy the feature. To note, most of the issues I tackled already had Accepting Merge Requests label on them, which meant Product was in agreement that the feature was worthy of having, but there were issues with more priority to be tackled first.

So, here are the features/enhancements I implemented in GitLab, as an interested contributor in the selfish interest of improving my Rails understanding and to get features that I wanted without much waiting:

  1. Add number of repositories to usage ping data
  2. Provide an API endpoint to get GPG signature of a commit
  3. Add ability to set project path and name when forking a project via API
  4. Add predefined CI variable to provide GitLab FQDN
  5. Ensure changelog filenames have less than 99 characters
  6. Support notifications to be fired for protected branches also
  7. Set X-GitLab-NotificationReason header in emails that are sent due to explicit subscription to an issue/MR
  8. Truncate recommended branch name to a sane length
  9. Support passing CI variables as push options
  10. Add option to configure branches for which emails should be sent on push

Swathanthra Malayalam Computing

I have been a volunteer at Swathanthra Malayalam Computing for almost 8 years now. Most of my contributions are towards various localization efforts that SMC coordinates. Last year, my major contributions were improving our fonts build process to help various packaging efforts (well, selfish reason - I wanted my life as the maintainer of Debian packages to be easier), implementing CI based workflows for various projects and helping in evangelism.

  1. Ensuring all our fonts build with Python3
  2. Ensuring all our fonts have proper appstream metadata files
  3. Add an FAQ page to Malayalam Speech Corpus
  4. Add release workflow using CI for Magisk font module


I have been a Debian contributor for almost 8 years, became a Debian Maintainer 3 years after my first stint with Debian, and have been a Debian Developer for 2 years. My activities as a Debian contributor this year are:

  1. Continuing maintenance of fonts-smc-* and hyphen-indic packages.
  2. Packaging of gopass password manager. This has been going on very slow.
  3. Reviewing and sponsoring various Ruby and Go packages.
  4. Help GitLab packaging efforts, both as a Debian Developer and a GitLab employee.

Other FOSS projects

In addition to the main projects I am a part of, I contributed to few FOSS last year, either due to personal interest, or as part of my job. They are:

  1. Calamares - I initiated and spearheaded the localization of Calamares installer to Malayalam language. It reached 100% translated status within a month.
  2. Chef
    1. Fix openSUSE Leap and SLES detection in Chef Ohai 14
    2. Make runit service’s control commands configurable in Chef Runit cookbook
  3. Mozilla - Being one of the Managers for Malayalam Localization team of Mozilla, I helped coordinate localizations of various projects, interact with Mozilla staff for the community in clarifying their concerns, getting new projects added for localization etc.


I also gave few talks regarding various FOSS topics that I am interested/knowledgeable in during 2019. List and details can be found at the talks page.

Overall, I think 2019 was a good year for the FOSS person in me. Next year, I plan to be more active in Debian because from the above list I think that is where I didn’t contribute as much as I wanted.

01 January, 2020 06:00AM

hackergotchi for Junichi Uekawa

Junichi Uekawa

Happy new year.

Happy new year. Last year I kept on practicing piano. I started Tea. Wondering what awaits.

01 January, 2020 05:21AM by Junichi Uekawa

Paul Wise

FLOSS Activities December 2019





  • Debian wiki: whitelist email addresses, help with auth issues
  • FOSSJobs: forwarding jobs, approving jobs


  • Respond to queries from Debian users and developers on the mailing lists and IRC


Some of the lintian-brush issues, the devscripts tagpending issue and the libpst work were sponsored by my employer. All other work was done on a volunteer basis.

01 January, 2020 01:18AM

Utkarsh Gupta

Debian Activities for December 2019

Here’s my (third) monthly update about the activities I’ve done in Debian this December.

Debian LTS

This was my third month as a Debian LTS paid contributor.
I was assigned 16.50 hours and worked on the following things:

CVE Fixes and Announcements:

  • Issued DLA 2024-1, fixing CVE-2019-19617, for phpmyadmin.
    Details here:

    phpMyAdmin before 4.9.2 does not escape certain Git information, related to libraries/display_git_revision.lib.php and libraries/Footer.class.php.

    For Debian 8 “Jessie”, this has been fixed in 4:4.2.12-2+deb8u7.
    Furthermore, sent a patch to the Security team for fixing the same in Stretch.

  • Issued DLA 2025-1, fixing CVE-2017-17833 and CVE-2019-5544, for openslp-dfsg.
    Details here:

    OpenSLP releases in the 1.0.2 and 1.1.0 code streams have a heap-related memory corruption issue which may manifest itself as a denial-of-service or a remote code-execution vulnerability.
    OpenSLP as used in ESXi and the Horizon DaaS appliances has a heap overwrite issue. VMware has evaluated the severity of this issue to be in the critical severity range.

    For Debian 8 “Jessie”, this has been fixed in 1.2.1-10+deb8u2.

  • Issued DLA 2026-1, fixing CVE-2019-19630, for htmldoc.
    Details here:

    In HTMLDOC, there was a one-byte underflow in htmldoc/ps-pdf.cxx caused by a floating point math difference between GCC and Clang.

    For Debian 8 “Jessie”, this has been fixed in 1.8.27-8+deb8u1.
    Furthermore, sent a patch to the Security team for Stretch and Buster.

  • Issued DLA 2046-1, fixing CVE-2019-19479, for opensc.
    Details here:

    An issue was discovered in libopensc/card-setcos.c in OpenSC, which has an incorrect read operation during parsing of a SETCOS file attribute.

    For Debian 8 “Jessie”, this has been fixed in 0.16.0-3+deb8u2.


  • Triage luajit, python-oslo.utils, davical, sqlite3, phpmyadmin, openssl, htmldoc, and opensc for Jessie.

  • Pinged upstream of libexif, ruby-rack, and ruby-rack-cors for more clarification of the patches provided.

  • Clarified more about CVE-2019-1551/openssl triage to the Security Team and the Debian LTS ML.

  • Took a deeper look at CVE-2019-16782/ruby-rack; the patch itself introduces regression and induces a backdoor on its own. Notified the Security Team and the ML to avoid its upload.

  • Discuss the state of CVE-2019-19479/opensc with Roberto and process its upload. Also opened upstream issue out of frustration for “hiding” most of their report.

  • In midst of fixing test failures of ruby-rack-cores. And also WIP for ruby-excon.

Debian Uploads

Most importantly, I became a DD this month! \o/
Here’s my NM process. Many, many thanks to Thomas (zigo) for being so nice and patient! :D

New Version:

  • ruby-reverse-markdown ~ 1.3.0-1 (to unstable).
  • ruby-behance ~ 0.6.1-1 (to unstable).
  • ruby-unidecode ~ 1.0.0-1 (to unstable).
  • micro ~ 1.4.1-1 (to unstable).
  • golang-code.cloudfoundry-bytefmt ~ 0.0~git20190818.854d396-1 (to unstable).

Source-Only and Other Uploads:

  • micro ~ 1.4.1-2 (to unstable).
  • golang-github-flynn-json5 ~ 0.0~git20160717.7620272-2 (to unstable).
  • golang-github-zyedidia-pty ~ 1.1.1+git20180126.3036466-2 (to unstable).
  • golang-github-zyedidia-terminal ~ 0.0~git20180726.533c623-2 (to unstable).
  • golang-golang-x-text ~ 0.3.2-3 (to unstable).
  • golang-github-yuin-gopher-lua ~ 0.0~git20170915.0.eb1c729-4 (to unstable).
  • golang-github-sergi-go-diff ~ 1.0.0-2 (to unstable).

Bug Fixes:

  • #946859 for ruby-reverse-markdown (ITP).
  • #946895 for ruby-behance (ITP).
  • #946945 for ruby-unidecode (ITP).
  • #947724 for golang-code.cloudfoundry-bytefmt (ITP).
  • #889196 golang-github-yuin-gopher-lua.
  • #889209 for golang-github-sergi-go-diff.

Reviews and Sponsored Uploads:

  • easygen ~ 4.1.0-1 for Tong Sun.
  • node-webpack ~ 4.30.0-1 for Pirate Praveen.
  • node-timeago.js ~ 4.0.2-1 for Sakshi Sangwan.
  • Outreachy mentoring for GitLab project.
  • Grant DM access for easygen to Tong Sun.
  • Grant DM access for golang-github-danverbraganza-varcaser to Tong Sun.
  • Help James Montgomery for Golang packaging (wrt #945524).
  • Migrated all my -guest accounts and certificates to use my new, shiny account associated with the DD status.
  • With regards to this tweet, I assisted the following people:
    • Shubhank Saxena with 1:1 Hangouts call.
    • Shreya Gupta with 1:1 Hangouts call.
    • Eshaan Bansal with 1:1 Hangouts call.
      P.S. It was lovely to interact with such lovely people :)

Until next time.
:wq for today.

01 January, 2020 12:16AM

December 31, 2019

hackergotchi for Chris Lamb

Chris Lamb

Favourite books of 2019

I managed to read 74 books in 2019 (up from 53 in 2018 and 50 in 2017) but here follows ten of my favourites this year, in no particular order.

Disappointments included The Seven Deaths of Evelyn Hardcastle (2018) which started strong but failed to end with a bang; all of the narrative potential energy tightly coiled in the exposition was lazily wasted in a literary æther like the "whimper" in the imagined world of T. S Eliot. In an adjacent category whilst I really enjoyed A Year in Provence (1989) last year, Toujours Provence (1991) did not outdo its predecessor but was still well worth the dégustation. I was less surprised to be let down by Jon Ronson's earliest available book, The Men Who Stare At Goats (2004), especially after I had watched the similarly off-key film of the same name, but it was at least intellectually satisfying to contrast the larval author of this work and comparing him the butterfly he is today but I couldn't recommend the experience to others who aren't fans of him now.

The worst book that I finished this year was Black Nowhere (2019), a painful attempt at a cyberthriller based on the story of the Silk Road marketplace. At many points I seriously pondered whether I was an unwitting participant in a form of distributed performance art or simply reading an ironic takedown of inexpensive modern literature.

As a slight aside, choosing which tomes to write about below was an interesting process but likely not for the reasons you might think; I found it difficult to write so publically anything interesting about some books that remain memorable to this day without essentially inviting silent censure or, worse still, the receipt of tedious correspondence due to their topics of contemporary politics or other vortexes of irrationality, assumed suspicion and outright hostility. (Given Orwell's maxim that "the only test of artistic merit is survival," I find this somewhat of disservice to my integrity, yet alone to the dear reader.)

In the Woods (2007), The Likeness (2008) & Faithful Place (2010)

Tana French

I always feel a certain smug pleasure attached to spotting those gaudy "Now a major TV series!" labels appearing upon novels I have already digested. The stickers do not merely adhere to the book themselves, but in a wider sense stick to myself too as if my own refined taste had been given approval and blessing of its correctness. Not unlike as if my favourite local restaurant had somehow been granted a Michelin star, the only problem then becomes the concomitant difficulty in artfully phrasing that one knew about it all along...

But the first thing that should probably be said about the books that comprise the Dublin Murder Squad ("Now a major TV series!") is the underlying scaffolding of the series: whilst the opening novel details Irish detectives Rob Ryan and Cassie Maddox investigating a murder it is told in from the first-person perspective of the former. However, the following book then not only recounts an entirely different Gardaí investigation it is told from the point of view of the latter, Cassie, instead. At once we can see how different (or not) the characters really are, how narrow (or not) their intepretation of events are, but moreover we get to enjoy replaying previous interaction between the two both, implicitly in our minds and even sometimes explicitly on the page. This fount of interest continues in the third of the series which is told from the viewpoint of a yet another character introduced in the second book and so forth.

I feel I could write a fair amount about these novels, but in the interest of brevity I will limit my encomium to the observation that the setting of Ireland never becomes a character itself, now curiously refreshing as most series feel the need to adopt this trope which overshot cliché some time ago. Authors, by all means set your conceits in well-trodded locations but please refrain from boasting or namedropping your knowledge at seemingly every opportunity (the best/worst example being Ben Aaronovitch's Rivers of London series or, by referencing street and pub names just a few too many times for comfort, Irvine Welsh's Edinburgh). Viewer's of the BBC Spooks series will likely know what I mean too - it isn't that the intelligence officers couldn't meet in the purview of St Paul's or under the watchful London Eye but the unlikelihood that all such clandestine conventicles would happen with the soft focus of yet another postcard-worthy landmark in the background forces at least this particular ex-Londoner of the plot somewhat.

Anyway, highly recommend. I believe I have three more in this series, all firmly on my 2020 list.

The Ministry of Truth (2019)

Dorian Lynskey

It should hopefully come as no surprise to anyone that I would read this "biography" of George Orwell's Nineteen Eighty-Four (NB. not "1984"...) after a number of Orwell-themed travel posts this year (Marrakesh, Hampstead, Paris, Southwold, Ipswich, etc.).

Timed to coincide with the book's publication 70 years ago, Lynskey celebrates its platinum anniversary with an in-depth view into the book's literary background in the dystopian fiction of the preceding generation including Yevgeny Zamyatin's 1921 We and H. G. Well's output more generally. It is a bête noire of mine that the concepts in the original book are taken too literally by most (as if by pointing out the lack of overt telescreens somehow discredits the work or — equally superficially in analysis — has been "proven right" by the prevalence of the FAANGs throughout our culture) but Lynskey does no such thing and avoids this stubbornly sophomoric and narrow view of Nineteen... and does not neglect the wider, more delicate and more interesting topics such as the slippage between deeds, intentions, thoughts, veracity and language.

Thorough and extremely comprehensive, this biography remains a wonderfully easy read and is recommended to all interested in one of the most influencial novels of the 20th century and furthermore should not be considered the exclusive domain of lovers of trivial Orwellania, not withstanding that such folks will undoubtably find something charming in Lynskey's research in any case: Who knew that the original opening paragraph of this book was quite so weak? Or a misprint resulted in an ambiguous ending...? This book shouldn't just make you want to read the novel again, it will likely pique your interest into delving deeper into Orwell’s writing for yourself. And if you don't, Big Brother is...

City of the Dead (2011) & The Bohemian Highway (2013)

Sara Gran

Imagine a Fleabag with more sass, more drug abuse, and — absent the first person narrative — thankfully hold the oft-distracting antics with the fourth wall. Throw in the perceptive insight of Sherlock and finish with the wistful and mystical notes of a Haruki Murakami novel and you've got Claire DeWitt, our plucky protagonist.

In post-Katrani New Orleans, where we lay our scene, this troubled private detective has been tasked with looking for a local prosecutor who has been missing since the hurricane. Surprisingly engrossing and trenchant, my only quibble with the naked, fast-paced and honest writing of City of the Dead is that the ends of chapters are far too easily signposted as the tone of the prose changes in a reliable manner, disturbing the unpredictability of the rest of the text.

The second work I include here (The Bohemian Highway) is almost on-par with the first with yet more of Claire's trenchant observations about herself and society (eg. "If you hate yourself enough, you’ll start to hate anyone who reminds you of you", etc.). However, it was quite the disappointment to read the third in this series (The Infinite Blacktop (2018) which had almost all the aforementioned ingredients but somehow fell far, far short of the target. Anyway, if someone has not optioned the rights for an eight-part television series of the first two novels, I would be willing to go at least, say, 90:10 in with you.

Never Split the Difference (2016)

Chris Voss

I was introduced to Chris Voss earlier in the year via an episode of The Tim Ferriss Show (and if that wasn't enough of a eyebrow-raising introduction he was just on an episode of Lance Armstrong's own podcast...) but regardless of its Marmite-esque route into my world I could not help but be taken hostage by this former FBI negotiator's approach to Negotiating as if Your Life Depended On It, as its subtitle hyperbolically claims.

My initial interest in picking up this how-to-negotiate volume lied much deeper than its prima facie goal of improving my woefully-lacking skills as I was instead intellectually curious about the socio-anthropology and to learn more about various facets of human connections and communication in general. However, the book mixes its "pop psych" with remarkably simple and highly practical tips for all levels of negotiations. Many of these arresting ideas, at least in the Voss school, are highly counter-intuitive yet he argues for them all persuasively, generally preferring well-reasoned argument over relying on the langue du bois of the "amygdala" and other such concepts borrowed superficial from contemporary psychology that will likely be rendered the phrenology of the early 21st-century anyway.

Whilst the book's folksy tone and exhale-inducing approach to pedagogy will put many off (I thought I left academia and its "worksheets" a long time ago…) it certainly passes the primary test of any book of negotiation: it convinced me.

The Way Inn (2014) & Plume (2019)

Will Wiles

I really enjoyed this authors take on modern British culture but I am unsure if I could really communicate exactly why. However, I am certain that I couldn't explain what his position really is beyond using misleading terms such as "surreal" or "existential" because despite these labels implying an inchoate and nebulous work I also found it simultaneously sharp and cuttingly incisive.

Outlining the satirical and absurd plot of The Way Inn would do little to communicate the true colour palette of the volume too (our self-absorbed protagonist attends corporate conferences on the topic, of course, of conferences themselves) but in both of these books Wiles ruthlessly avoiding all of the tired takedowns of contemporary culture, somehow finding new ways to critique our superficial and ersatz times.

The second of Wiles' that I read this year, Plume, was much darker and even sinister in feel but remains peppered with enough microscopic observations on quotidian life ("the cloying chemical reek of off-brand energy drinks is a familiar part of the rush-hour bouquet"...) that somehow made it more, and not less, harrowing in tone. You probably need to have lived in the UK to get the most out of this, but I would certainly recommend it.

Chasing the Scream (2015)

Johann Hari

It is commonplace enough to find RT ≠ endorsement in a Twitter biography these days but given that Hari's book documents a Search For The Truth About Addiction I am penning this review with more than a soupçon of trepidation. As in, if it would be premature to assume that if someone has chosen to read something then they are implicitly agreeing with its contents it would also be a similar error to infer that reader is looking for the same answers. This is all to say that I am not outing myself as an addict here, but then again, this is precisely what an addict would say...

All throat clearing aside, I got much from reading Johann Hari's book which, I think, deliberately does not attempt to break new ground in any of the large area it surveys and prefers to offer a holistic view of the war on drug [prohibition] through a series of long vignettes and stories about others through the lens of Hari himself on his own personal journey.

Well-written and without longueur, Hari is careful to not step too close to the third-rail of the medication—mediation debate as the most effective form of treatment. This leads to some equivocation at points but Hari's narrative-based approach generally lands as being more honest than many similar contemporary works that cede no part of the complex terrain to anything but their prefered panacea, all deliciously ironic given his resignation from the Independent newspaper in 2011. Thus acting as a check against the self-assured tones of How to Change Your Mind (2018) and similar, Chasing the Scream can be highly recommended quite generally but especially for readers in this topic area.

The Sellout (2016)

Paul Beatty

"I couldn't put it down…" is the go-to cliché for literature so I found it amusing to catch myself in quite-literally this state at times. Winner of the 2016 Man Booker Prize, the first third of this were perhaps the most engrossing and compulsive reading experience I've had since I started "seriously" reading.

This book opens in medias res within the Supreme Court of the United States where the narrator lights a spliff under the table. As the book unfolds, it is revealed that this very presence was humbly requested by the Court due to his attempt to reinstate black slavery and segregation in his local Los Angeles neighbourhood. Saying that, outlining the plot would be misleading here as it is far more the ad-hoc references, allusions and social commentary that hang from this that make this such an engrossing work.

The tranchant, deep and unreserved satire might perhaps be merely enough for an interesting book but where it got really fascinating to me (in a rather inside baseball manner) is how the latter pages of the book somehow don't live up the first 100. That appears like a straight-up criticism, but this flaw is actually part of this book's appeal to me — what actually changed in these latter parts? It's not overuse of the idiom or style and neither is it that it strays too far from the original tone or direction, but I cannot put my finger on why which has meant the book sticks to this day in my mind. I can almost, just almost, imagine a devilish author such as Paul deliberately crippling one's output for such an effect…

Now, one cannot unreservedly recommend this book. The subject matter itself, compounded by being dealt with in such an flippant manner will be unpenetrable to many and deeply offensive to others, but if you can see your way past that then you'll be sure to get something—whatever that may be—from this work.

Diary of a Somebody (2019)

Brian Bilston

The nom de plume of the "unofficial poet laureate of Twitter", Brian Bilston is an insufferable and ineffectual loser who decides to write a poem every day for a year. A cross between the cringeworthiness of Alan Partridge and the wit and wordplay of Spike Milligan, the eponymous protagonist documents his life after being "decruited" from his job.

Halfway through this book I came to the realisation that I was technically reading a book of poetry for fun, but far from being Yeats, Auden or The Iliad, "Brian" tends to pen verse along the lines of:

No, it's not Tennyson and "plot" ties itself up a little too neatly at the end, but I smiled out loud too many times whilst reading this book to not include it here.

Stories of Your Life and Others (2014) & Exhalation (2019)

Ted Chiang

This compilation has been enjoying a renaissance in recent years due the success of the film Arrival (2016) which based on on the fourth and titular entry in this amazing collection. Don't infer too much from that however as whilst this is prima facie just another set of sci-fi tales, it is science fiction in the way that Children of Men is, rather than Babylon 5.

A well-balanced mixture of worlds are evoked throughout with a combination of tales that variously mix the Aristotelian concepts of spectacle (opsis), themes (dianoia), character (ethos) and dialogue (lexis), perhaps best expressed practically in that some stories were extremely striking at the time — one even leading me to rebuff an advance at a bar — and a number were not as remarkable at the time yet continue to occupy my idle thoughts.

The opening tale which reworks the Tower of Babel into a construction project probably remains my overall favourite, but the Dark Materials-esque world summoned in Seventy-Two Letters continues to haunt my mind and lips of anyone else who has happened to come across it, perhaps becoming the quite-literal story of my life for a brief period. Indeed it could be said that, gifted as a paperback, whilst the whole collection followed me around across a number of locales, it continues to follow me — figuratively speaking that is — to this day.

Highly recommended to all readers but for those who enjoy discussing books with others it would more than repay any investment.

Operation Mincemeat (2010)

Ben MacIntyre

In retrospect it is almost obvious that the true story of an fictitious corpse whose invented love letters, theatre life and other miscellania stuffed into the pockets of a calculatingly creased Captain's uniform would make such a captivating tale. Apparently drowned and planted into the sea off Huelva in 1943, this particular horse was not exactly from Troy but was rather a Welsh vagrant called Glyndwr who washed up — or is that washed out? — on the Andalusian shoreline along with information on a feigned invasion of Sicily in an attempt to deceive the Wehrmacht. However, this would be to grosslly misprice Ben MacIntyre's ability to not get in the way of telling the story as well the larger picture about the bizarre men who concocted the scheme and the bizarre world they lived in.

In such a Bond-like plot where even Ian Fleming (himself a genuine British naval officer) makes an appearance it seems prudent to regularly recall yet again that truth can be stranger than fiction, but the book does fall foul of the usual sin of single-issue WW2 books in overestimating the importance in the larger context of a conflict. (Indeed, as a diversionary challenge to the reader of this review I solicit suggestions for any invention, breakthrough or meeting that has not been identified as "changing the course of World War II". Victor Davis Hanson rather handsomeley argues in his 2017 The Second World Wars that is best approached as multiple wars, anyway…)

Likely enjoyed by those not typically accustomed to reading non-fiction history, this is genuinely riveting account nonetheless and well worth the reading.

31 December, 2019 06:42PM

hackergotchi for Jonathan McDowell

Jonathan McDowell

Free Software Activities for 2019

As a reader of Planet Debian I see a bunch of updates at the start of each month about what people are up to in terms of their Free Software activities. I’m not generally active enough in the Free Software world to justify a monthly report, and this year in particular I’ve had a bunch of other life stuff going on, but I figured it might be interesting to produce a list of stuff I did over the course of 2019. I’m pleased to note it’s longer than I expected.


I’m not a big conference attendee; I’ve never worked somewhere that paid travel/accommodation for Free Software conferences so I end up covering these costs myself. That generally means I go to local things and DebConf. This year was no exception to that; I attended BelFOSS, an annual free software conference held in Belfast, as well as DebConf19 in Curitiba, Brazil. (FOSDEM was at an inconvenient time this year for me, or I’d have made it to that as well.)


Most of my contributions to Free software happen within Debian.

As part of the Data Protection Team I responded to various minor requests for advice from within the project.

The Debian Keyring was possibly my largest single point of contribution. We’re in a roughly 3 month rotation of who handles the keyring updates, and I handled 2019.03.24, 2019.06.25, 2019.08.23, 2019.09.24 + 2019.12.23.

For Debian New Members I handled a single applicant, Marcio de Souza Oliveira, as an application manager. I had various minor conversations throughout the year as part of front desk.

I managed to get binutils-xtensa-lx106 + gcc-xtensa-lx106 packages (1 + 1) for cross building ESP8266 firmware uploaded in time for the buster release, as well as several updates throughout the year (2, 3 + 2, 3, 4). There was a hitch over some disagreements on the package naming, but it conforms with the generally accepted terms used for this toolchain.

Last year I ended up fixing an RC bug in ghdl, so this year having been the last person to touch the package I did a couple of minor uploads (0.35+git20181129+dfsg-3, 0.35+git20181129+dfsg-4). I’m no longer writing any VHDL as part of my job so my direct interest in this package is limited, but I’ll continue to try and fix the easy things when I have time.

Although I requested the package I originally uploaded it for, l2tpns, to be removed from Debian (#929610) I still vaguely maintain libcli, which saw a couple of upstream driven uploads (1.10.0-1, 1.10.2-1).

OpenOCD is coming up to 3 years since its last stable release, but I did a couple (0.10.0-5, 0.10.0-6) of minor uploads this year. I’ve promised various people I’ll do a snapshot upload and I’ll try to get that into experimental at some point. libjaylink, a dependency, also saw a couple of minor uploads (0.1.0-2, 0.1.0-3).

I pushed an updated version of libtorrent into experimental (0.13.8-1), as a pre-requisite for getting rtorrent updated. Once that had passed through NEW I uploaded 0.13.8-2 and then rtorrent 0.9.8-1.

The sigrok project produced a number of updates, sigrok-firmware-fx2lafw 0.1.7-1, libsigrok 0.5.2-1 + libsigrokdecode 0.5.3-1.

sdcc was the only package I did sponsored uploads of this year - (3.8.0+dfsg-2, 3.8.0+dfsg-3). I don’t have time to take over maintainership of this package fully, but sigrok-firmware-fx2lafw depends on it to build so I upload for Gudjon and try to help him out a bit.

Personal projects

In terms of personal projects I finally pushed my ESP8266 Clock to the outside world (and wrote it up). I started learning Go and as part of that wrote gomijia, a tool to passively listen for Bluetooth LE broadcasts from Xiaomi Mijia devices and transmits them over MQTT. I continued to work on onak, my OpenPGP key server, adding support for the experimental v5 key format, dkg’s abuse resistant keystore proposal and finally merged in support for signature verification. It’s due a release, but the documentation really needs improved before I’d be happy to do that.


Back when picolibc was newlib-nano I had a conversation with Keith Packard about getting the ESP8266 newlib port (largely by Max Filippov based on the Tensilica work) included. Much time has passed since then, but I finally got time to port this over and test it this month. I’m hopeful the picolibc-xtensa-lx106-elf package will appear in Debian at some point in the next few months.


As part of my work at Titan IC I did some work on Snort3, largely on improving its support for hardware offload accelerators (ignore the fact my listed commits were all last year, Cisco generally do a bunch of squashed updates to the tree so the original author doesn’t always show).

Software in the Public Interest

While I haven’t sat on the board of SPI since 2015 I’m still the primary maintainer of the membership website (with Martin Michlmayr as the other active contributor). The main work carried out this year was fixing up some issues seen with the upgrade from Stretch to Buster.


I talked about my home automation, including my use of Home Assistant, at NIDC 2019, and again at DebConf with more emphasis on the various aspects of Debian that I’ve used throughout the process. I had a couple of other sessions at DebConf with the Data Protection and Keyring teams. I did a brief introduction to Reproducible Builds for BLUG in October.


I had a one liner accepted to systemd to make my laptop keyboard work out of the box. I fixed up Xilinx XRT to be able to build .debs for Debian (rather than just Ubuntu), have C friendly header files and clean up some GCC 8.3 warnings. I submitted a fix to Home Assistant to accept 202 as a successful REST notification response. And I had a conversation on IRC which resulted in a tmux patch to force detach (literally I asked how do to this thing and I think Colin had whipped up a patch before the conversation was even over).

31 December, 2019 06:20PM

hackergotchi for Chris Lamb

Chris Lamb

Free software activities in December 2019

Software Freedom Conservancy (the fiscal sponsor for the Reproducible Builds project) have announced their fundraising season with a huge pledge to match donations from a number of illustrious individuals. If you have ever considered joining as a supporter, now would be the time to do so.

Whilst it was a busy month away from the keyboard for me, here is my update covering what I have been doing in the free software world during December 2019 (previous month):

  • Attended the fifth Reproducible Builds summit meeting in Marrakesh, Morocco.

  • As part of my duties of being on the board of directors of the Open Source Initiative and Software in the Public Interest (SPI) I attended and prepared for their respective monthly meetings, participated in various licensing and other free software related topics occurring on the internet, as well as the usual internal discussions regarding logistics, policy, etc.

  • Opened a pull request against the Chart.js JavaScript charting library to make the build reproducible. [...]

  • Updated my django-slack library that provides a convenient library between projects using the Django and the Slack chat platform to drop Python 2.7 support prior to its uncoming deprecation [...] and add support for Python 3.8 [...]...][...].

  • Made some changes to my tickle-me-email library which implements Gettings Things Done-like behaviours in IMAP inboxes including fixing an issue where we could add a duplicate empty Subject header that would result in emails being rejected as invalid by mail servers. [...]

  • Opened a pull request to make the build reproducible in infernal, a tool for analysing RNA molecule data. [...]

  • Even more hacking on the Lintian static analysis tool for Debian packages including a considerable amount of issue and merge request triage, as well as:

    • Bug fixes:

      • Don't attempt to check manual section if we don't know the section number in order to silence Perl warnings on the commandline. (#946471)
    • Cleanups:

    • Reporting:

      • Add missing tag summary checks to debian/changelog and fix our generate-tag-summary script to match our newer style of changelog entry placeholder. [...]
      • Update the long description of debian-rules-not-executable tag to not imply that precisely 0755 permissions are required. [...]

Reproducible builds

Whilst anyone can inspect the source code of free software for malicious flaws almost all software is distributed pre-compiled to end users. The motivation behind the Reproducible Builds effort is to ensure no flaws have been introduced during this compilation process by promising identical results are always generated from a given source, thus allowing multiple third-parties to come to a consensus on whether a build was compromised.

The initiative is proud to be a member project of the Software Freedom Conservancy, a not-for-profit 501(c)(3) charity focused on ethical technology and user freedom.

Conservancy acts as a corporate umbrella allowing projects to operate as non-profit initiatives without managing their own corporate structure. If you like the work of the Conservancy or the Reproducible Builds project, please consider becoming an official supporter.

I made the following changes to diffoscope, our in-depth and content-aware diff utility that can locate and diagnose reproducibility issues:

  • Always pass a filename with a .zip extension to zipnote otherwise it will return with an UNIX exit code of 9 and we fallback to displaying a binary difference for the entire file. [...]
  • Include the libarchive file listing for ISO images to ensure that timestamps -- and not just dates -- are visible in any difference. (#81)
  • Ensure that our autopkgtests are run with our pyproject.toml present for the correct black source code formatter settings. (#945993)
  • Rename the text_option_with_stdiout test to text_option_with_stdout [...] and tidy some unnecessary boolean logic in the ISO9660 tests [...].

I also:


Debian LTS

This month I have worked 16½ hours on Debian Long Term Support (LTS) and 12 hours on its sister Extended LTS project.

You can find out more about the project via the following video:


  • For the Tails privacy-oriented operating system, I uploaded obfs4proxy (0.0.8-1)

FTP Team

As a Debian FTP assistant I ACCEPTed eight packages: fluidsynth, golang-github-bmatcuk-doublestar, golang-github-pearkes-cloudflare, librandomx, meep, meep-mpi-default, meep-openmpi & node-webassemblyjs. I additionally filed two RC bugs against packages that had potentially-incomplete debian/copyright files against fluidsynth & meep.

31 December, 2019 01:50PM

December 30, 2019

Ian Jackson

subdirmk 0.3 - ergonomic preprocessing assistant for non-recursive make

I have released subdirmk 0.3.


Peter Miller's 1997 essay Recursive Make Considered Harmful persuasively argues that it is better to arrange to have a single make invocation with the project's complete dependency tree, rather than the currently conventional $(MAKE) -C subdirectory approach.

However, I have found that actually writing a project's build system in a non-recursive style is not very ergonomic. So with some help and prompting from Mark Wooding, I have made a tool to help.

What's new

I have overhauled and regularised some of the substitution syntaxes. The filenames have changed. And there is a new $-doubling macro help facility.


It's still 0.x. I'm still open to comments about details of syntax and naming. Please make them here on this blog, or by posting to sgo-software-discuss.

But it's looking quite good I think and I intend to call it 1.0 RSN.

Further reading

see the README.

edited 2019-12-30 16:39 Z to fix some formatting issues with Dreamwidth's HTML "sanitiser"

comment count unavailable comments

30 December, 2019 04:40PM

December 29, 2019

François Marier

Encoding your WiFi access point password into a QR code

Up until recently, it was a pain to defend againt WPA2 brute-force attacks by using a random 63-character password (the maximum in WPA-Personal) mode). Thanks to Android 10 and iOS 11 supporting reading WiFi passwords from a QR code, this is finally a practical defense.

Generating the QR code

After installing the qrencode package, run the following:

qrencode -o wifi.png "WIFI:T:WPA;S:<SSID>;P:<PASSWORD>;;"

substituting <SSID> for the name of your WiFi network and <PASSWORD> for the 63-character password you hopefully generated with pwgen -s 63.

If your password includes a semicolon, then escape it like this:


since iOS won't support the following (which works fine on Android):


The only other pitfall I ran into is that if you include a trailing newline character (for example piping echo "..." into qrencode as opposed to echo -n "...") then it will fail on both iOS and Android.

Scanning the QR code

On iOS, simply open the camera app and scan the QR code to bring up a notification which allows you to connect to the WiFi network:

On Android, go into the WiFi settings and tap on the WiFi network you want to join:

then click the QR icon in the password field and scan the code:

In-browser alternative

If you can't do this locally for some reason, there is also an in-browser QR code generator with source code available.

29 December, 2019 03:25AM