November 21, 2019

hackergotchi for Jonathan Dowland

Jonathan Dowland


On Yesterday's Mary Anne Hobbs radio show she debuted a new track by Squarepusher, "Vortrack [Fracture Remix]", which you can now watch/listen to on YouTube:

This track really grabbed my attention. Later that day you you could buy it in a variety of formats and quality levels on Squarepusher's website.

One of the format/quality options was "8-bit Lossless WAV", which I thought was a joke, a poke in the eye of audiophiles. I was aware that he likely used some 8-bit instruments/equipment to write the tracks, but surely it was mixed in a more modern environment, and resampling down to 8-bit would result in something that sounded like mush.

But it seems the jokes on me; I bought the track and it's seemingly indistinguishable to the audio track on that YouTube video. And it really is 8-bit:

Input #0, wav, from 'Vortrack-001-Squarepusher-Vortrack (Fracture Remix).wav':
  Duration: 00:08:02.99, bitrate: 705 kb/s
    Stream #0:0: Audio: pcm_u8 ([1][0][0][0] / 0x0001),
    44100 Hz, 2 channels, u8, 705 kb/s

It even — losslessly — compressed down to a bitrate lower than a typical MP3:

Input #0, flac, from 'Vortrack-001-Squarepusher-Vortrack (Fracture Remix).flac':
  Duration: 00:08:02.99, start: 0.000000, bitrate: 313 kb/s
    Stream #0:0: Audio: flac, 44100 Hz, stereo, s16 (8 bit)

Sorry, comments on my site seem to be broken at the moment. I'm hoping to fix them soon.

21 November, 2019 11:55AM

Ian Jackson

Debian init systems - what, another GR ?

Sam Hartman, the Debian Project Leader, has proposed a General Resolution (a plebiscite of the whole project) about init systems. In this posting I am going to try to summarise the situation. This will necessarily be a personal view but I will try to be fair. Also, sorry that it's so long but there is a lot of ground to cover.

Pithy summary

The current political atmosphere has been turning bugs into fights. We need to stop this, and try to turn our fights back into bugs. The parts that need to be nailed down are:

  • whose job it is to fix the bug? (or, what happens if the bug is not fixed?, which amounts to the same thing);
  • when the bug is fixed take the patch already!; and
  • everyone stop with the dissing, negging, and concern trolling, already

I think we can do this (well, the first two, at least) with my proposal.


My starting point for all of this is what I have written at the top of my proposed alternative to the DPL's GR texts:

It is primarily for the communities in each software ecosystem to maintain and develop their respective software - but with the active support of other maintainers and gatekeepers where needed.
This is particularly important for a project like Debian. Debian plays a very central role for a lot of people. It is at least as much a clearinghouse or a marketplace, as it is a unified system. Supporting a different init system or a different menu system or whatever is fairly easy to do if done within Debian (if the Debian maintainers will take your patches) but a lot harder to do downstream.

Conversely, specialising on particular use cases is very easy to do in a derivative. This means that Debian needs to remain as general as possible, even if that means it suffers a bit in dominant use cases.


In 2014 Debian changed to use systemd as the default init system. The intent (at least of those on the TC making that decision) was that other init systems would continue to be supported.

Since then, Debian has still continued to support sysvinit and other systems such as runit and OpenRC. sysvinit support was very good in Debian 9 ("stretch").

In Debian 10 ("buster", currently "stable") there are some snags with desktop software, much of which nowadays usually wants interfaces provided by systemd's logind component. elogind can provide this under sysvinit, but the package dependencies are awkward and there are some other wrinkles. Also the sysvinit package was looking rather undermaintained.

The Debian Ecosystem Init Diversity Team

In response, we assembled a team of people who cared about these issues. The team includes Debian Developers, developers from Devuan and we also have contributions from (for example) the recently revitalised sysvinit upstream. Working with the folks from Devuan has been great. I want in particular to say a huge thank you to Mark Hindley.

In bullseye ("testing"), sysvinit is getting plenty of love and is in reasonably good shape. We have a close relationship with sysvinit upstream. Support for desktop software is provided by elogind, which is a logind derivative developed for use in Guix.

Unresolved issues

However, the Debian Project as a whole remains very unsettled about init systems. Everyone has a lot of baggage, and there are some specific unresolved issues too.

Toxic atmosphere

One huge problem is that conversations on the Debian mailing lists about sysvinit or systemd-related features are generally extremely toxic.

There is a significant contingent within Debian who seem to be hoping that non-systemd systems would just go away so they could have the beautiful sunlit uplands of homogeneity. Comments predicting the demise of sysvinit, with a subtext that systemd sceptics have lost and should shut up and go away, or at least that they are doomed, are common. The implication (usually not stated so baldly) is "we decided for systemd n years ago, isn't it time to throw that other stuff out yet?"

Conversely, systemd sceptics in these conversations have been defensive, embattled and bitter. Predictions of systemd hegemony lead naturally to responses citing alleged systemd misfeatures, bugs and absurdities. And sometimes the responses from systemd sceptics just have been very nasty.

Debian has generally suffered from problems with toxic culture and has struggled to improve and to manage the behaviour of some of its contributors, and struggled to deal with obstructive maintainers. On this particular issue the political context, and the ongoing fights (read on), make it much much worse.

Blocking of init diversity work

An underlying reason for the bad feeling amongst systemd sceptics has been a pattern of obstructive behaviours by some people within Debian.

For example there have been vague and unactionable RC bug reports against pieces of the non-systemd ecosystem. Requests to core package maintainers to accomodate non-systemd setups in their dependencies are approved very slowly if at all. [1]

Blocking of policy updates; init scripts

Debian Policy still talks basically only about init scripts. We need policy that reflects reality and gives useful technical guidance on how to make things work properly with systemd, and how to make things compatible.

Nominally, policy says that you MUST have an init script if you have a systemd daemon service unit. But no-one is filing these bugs. This is probably because a contributor would not expect a useful response; also, Debian's processes for review of maintainer decisions can be quite unpleasant [2]. The current policy wording de facto encourages the mere filing of an RC bug (ie, a demand that the maintainer do the work) rather than the contribution of a patch.

The policy discussion process is unsuited to controversial issues. Usually policy changes only by consensus. It is clearly not possible to achieve consensus to drop the mandatory requirement for init scripts, since systemd opponents are unhappy with that - in particular because RC bug status is Debian's only really useful mechanism for dealing with an obstructive maintainer.

The policy maintainers are quite reasonably reluctant to make changes in this area without a clear indication of direction from the project.

Blocking of use of useful, portable, interfaces from systemd

systemd has some useful features that we could probably make use of in Debian without giving up support for non-systemd init systems. Particularly it offers declarative syntaxes for various common tasks which we currently do with debhelper scripts.

Many of these do not depend on systemd itself, and are not particularly complex. They would not be hard to reimplement (that being the standard approach to providing compatibility when the atmosphere is less fraught). Some of us have been hoping for more declarative approaches for some time, so there would be value here.

Normally in Debian someone who wanted to transition to such new scheme would provide the relevant implementations (perhaps several), do a few demo packages, document it in policy, and eventually (after some years) we would have lintian warnings about the old scheme. People who want to improve Debian must generally file patches[3] even when the maintainers are sympathetic but maybe busy.

However, people who think that systemd is the future (and that other init systems are obsolete) see reimplementing systemd features in other programs as wasted effort, and systemd itself is not structured to make it easy to tease out its implementations of some of its features. Conversely, people who dislike systemd are not likely to agree to the inclusion in policy of requirements they see as extending the systemd hegemony.

It is not unusual in Debian to see valuable changes blocked [4], sometimes for many years. But it is not a good thing - and this stuck issue provides additional fuel for the political fires.

The upcoming General Resolution and the position of the DPL

The current DPL thinks that trying to resolve the unresolved tensions over systemd is necessary for the health of the project. That is why he has brought this issue to a head by making a set of GR proposals.

Sam is framing this as a question of the project's priorities. The solution, for him, seems to be to simply and definitively answer the question of whether the project is going to support only systemd, and how much of a commitment we are to make to it.

I think that is the wrong framing. As I say above I think Debian needs to be the place where you can come to do your work, and we need to welcome everyone's contributions even if we don't think they're particularly valuable. That is the basic bargain underlying the common project.

Sam's middle option is not a compromise, and sets us up for more fighting

The most likely outcome of a choice just between the DPL's three options would probably the second one, which has been placed in the "middle" and labelled "systemd but" (as if dropping systemd were even on the table). But it is a poor text - vague and unbalanced.

It does not clearly and specifically address many of the problematic behaviours I discuss above, so it would set us up for more years of fighting over details.

Although it reads vaguely like a compromise ("however") and has been placed in the middle of the list, it will in fact make things things quite difficult for contributors and users who don't want systemd.

It doesn't clearly contradict the notion that "systemd is the future and other things are obsolete or experimental", which continually undermines work on systemd alternatives (for example, we have had maintainers ask on debian-devel whether they should remove their init script now they have a systemd unit file). It legitimises uncoordinated adoption of systemd features and make it unnecessarily hard for sysvinit etc. to keep up. It does nothing to help contributors whose non-systemd support contributions are rejected: for example, init script contributors would still need to fight those unreasonable rejections piecemeal.

On the approach to process

Sam has taken the unusual approach of drafting three alternative options himself. He has also taken a rather autocratic approach to the timetable. I strongly disapprove of this.

He has put himself in the position of trying to frame the debate along the lines that he sees, which is not right. Different constituencies have different framings, and are entitled to have those represented. People who see things differently should not be put in the position of opposing the leadership, and scrabbling for resolution seconders on needlessly tight timescales. The drafting of options should be led by the people whose opinions they are.

My position: Keep Debian a do-ocracy

The question of whether Debian supports running without systemd needs to be answered not by strategic decisions over priorities. That amounts to to to deliberately excluding people's contributions. I can see how that is tempting, because inevitably diversity comes with a price for everyone. But that diversity is how we built our community, and it is particularly important for Debian's role as a central clearinghouse, and indeed our role as an upstream to a whole ecosystem of derivatives.

We should look at the question we usually ask (eg about whether to support architectures): are there enough people who want to support non-systemd setups, and are they willing and able to do the work ?

If there are we should support them. We should take their patches and support and enable their work. If there are not, then quite significant obsolete software can and will die its death fairly quietly, without the need for blocking, fighting, negging and concern trolling.

Unfortunately it appears that enough people in Debian aren't able to be reasonable and constructive, at least without being set clear and specific boundaries. Setting those boundaries is what I am trying to do with my proposed option. If you are a full voting Debian Member, I would appreciate it if you would please formally Second my proposal. I have more than the minimum of five already - thanks Russ, Sean, Simon, Kyle, Dmitry, Noodles - but lots more would be good.

How my proposal helps

Toxic atmosphere

The introduction says we want to support multiple init systems "for the foreseeable future". That directly confronts the narrative that "systemd has won and therefore other things should go away". The introduction explicitly states the principle I started this posting with. There are some specific rules towards the end, where I name particular common toxic patterns; I don't know if our communication fora operators have the will to deal with this but this is the best a GR can do, I think.

Additionally I'm hoping that providing workable rules of engagement will mean that people are less frustrated, and that we don't have to try to wear each other down (which is how fights are traditionally won in Debian - urgh.)

Unblocking non-systemd support

The compromise for init scripts etc., is to encourage submission of patches by making the patches count as RC bugs. Although this is the current formal state in policy, it is not effective because it lacks political legitimacy. This GR gives it that legitimacy. In return we relax the policy MUST in cases where there is no patch. So we are not forcing anyone to do the work.

There are more complex situations where systemd package dependencies, or release team decisions, and so on, are involved. My option gives general guidance (which really ought to be uncontroversial, but has not always been followed). If this is not effective in particular cases then unfortunately the Technical Committee will probably end up having to resolve the disputes. But I think there will be relatively few of these and because of the complexity of the issues it's not possible to decide them in advance.

Unblocking policy development

There is a compromise on systemd's non-init-related features ("declarative" things). Fastest progress could be made for systemd systems if every maintainer was free to simply use every systemd feature. But that is not how we normally do things in Debian. We try to bring the whole community with us, even if that means some delays.

Because questions of timescale are going to be particularly controversial, and are a matter of balancing the interests of different groups, I put in a specific timescale to avoid further arguments. In practice I think when useful systemd syntaxes etc. are approved, the init diversity community will want to implement them quickly to demonstrate their dynamism and relevance.

On the "Init Diversity" counter-proposal

Dmitry Bogatov has put forward an option of his own. I think what Dmitry is trying to do makes sense. His option is the strongest protector of diversity and most strongly reestablishes the usual (albeit not universal) Debian norm that people who want to change things should do all the work. It gives clear political reinforcement to the current de jure policy on init scripts.

In most situations Dmitry's proposal is rather stronger than mine. I think in practical terms my proposal will be quite sufficient to have good support for sysvinit (and indeed other init systems) for the long term. But if you supported my strong init diversity position in 2014 you will want to consider seconding and/or voting for Dmitry's proposal.

This section largely rewritten 2019-11-21 11:03 since Dmitry updated and improved his proposal

Looking to the future

Frankly, there is a lot not to like about init scripts. I have never liked the "detach" model of daemon management (even though I wrote start-stop-daemon to fix its worst bugs). I much prefer setups with closer supervision.

I hope that we can find a better lingua franca than handwritten init scripts, at least for the most common cases. Exactly what that looks like will have to be discussed within the init diversity community.

We do have two other init systems: runit and OpenRC. I haven't tried either of them and frankly I have little idea if I would like them if I did. But we have good relationships between all of the relevant maintainers and users. I hope these projects will thrive.

But so long as the community of systemd sceptics have to be fighting for their very existence, there is little time and little emotional energy for development and experimentation, and adoption of novel ideas.


Thanks very much to the several people who helped by reviewing drafts of this posting. Your support and frank comments have been invaluable. You know who you are. All errors and missteps are mine, of course.


[1] I don't mean to be calling people out, but I feel these kind of assertions need to be substantiated so here are some representative examples.

[2] The Technical Committee process is quite stressful. Here's a particularly bad example (where I have mixed feelings about my own part).

[3] all reproducible builds bugs (warning: huge)

[4] Debian is famously slow but here are a couple of my pet examples.

Edited 2019-11-20 12:20 UTC to fix some HTML formatting issues with some blog aggregators.

comment count unavailable comments

21 November, 2019 11:08AM

hackergotchi for Joachim Breitner

Joachim Breitner

Faster Winter 4: Export lists

(This is the forth optimization presented in the “faster winter” series, please see that post for background information.)

This is on a funny one: You can speed up your code by adding export lists to your modules!

Why is that so?

Without an export, the compiler has to assume that every top-level function can possibly called from the outside, even functions that you think of as “internal”. If you have a function that you do not export, like instr, step_work and step after my change, the compiler can see all the places the function is called. If the function is only called in one place, it may inline it (copy its definition into where it is called), and simplify the code around the edges. And even if it does not inline the function, it might learn something about how the functions are used, and optimize them based on that (e.g. based on Demand Analysis).

Improvement: Allocations: -22.59% Memory: +0.00% Time: -11.79% (Commit bbe8af7...6f2ba09)

Besides being a performance win, an explicit export list is also very helpful to those who want to read or edit your code: they can refactor with greater ease while maintaining the external interface of the module.

21 November, 2019 07:52AM by Joachim Breitner (

hackergotchi for Kees Cook

Kees Cook

experimenting with Clang CFI on upstream Linux

While much of the work on kernel Control Flow Integrity (CFI) is focused on arm64 (since kernel CFI is available on Android), a significant portion is in the core kernel itself (and especially the build system). Recently I got a sane build and boot on x86 with everything enabled, and I’ve been picking through some of the remaining pieces. I figured now would be a good time to document everything I do to get a build working in case other people want to play with it and find stuff that needs fixing.

First, everything is based on Sami Tolvanen’s upstream port of Clang’s forward-edge CFI, which includes his Link Time Optimization (LTO) work, which CFI requires. This tree also includes his backward-edge CFI work on arm64 with Clang’s Shadow Call Stack (SCS).

On top of that, I’ve git a few x86-specific patches that get me far enough to boot a kernel without warnings pouring across the console. Along with that are general linker script cleanups, CFI cast fixes, and x86 crypto fixes, all in various states of getting upstreamed. The resulting tree is here.

On the compiler side, you need a very recent Clang and LLD (i.e. “Clang 10”, or what I do is build from the latest git). For example, here’s how to get started. First, checkout, configure, and build Clang (and include a RISC-V target just for fun):

# Check out latest LLVM
mkdir -p $HOME/src
cd $HOME/src
git clone
mkdir llvm-build
cd llvm-build
# Configure
cmake -DCMAKE_BUILD_TYPE=Release \
      -DLLVM_ENABLE_PROJECTS='clang;lld' \
# Build!
make -j$(getconf _NPROCESSORS_ONLN)
# Install cfi blacklist template (why is this missing from "make" above?)
mkdir -p $(echo lib/clang/*)/share
cp ../llvm-project/compiler-rt/lib/cfi/cfi_blacklist.txt lib/clang/*/share/cfi_blacklist.txt

Then checkout, configure, and build the CFI tree. (This assumes you’ve already got a checkout of Linus’s tree.)

# Check out my branch
cd ../linux
git remote add kees
git fetch kees
git checkout kees/kspp/cfi/x86 -b test/cfi
# Configure (this uses "defconfig" but you could use "menuconfig"), but you must
# include CC and LD in the make args or your .config won't know about Clang.
make defconfig \
     CC=$HOME/src/llvm-build/bin/clang LD=$HOME/src/llvm-build/bin/ld.lld
# Enable LTO and CFI.
scripts/config \
     -e CONFIG_LTO \
# Enable LKDTM if you want runtime fault testing:
scripts/config -e CONFIG_LKDTM
# Build!
make -j$(getconf _NPROCESSORS_ONLN) \
     CC=$HOME/src/llvm-build/bin/clang LD=$HOME/src/llvm-build/bin/ld.lld

Do not be alarmed by various warnings, such as:

ld.lld: warning: cannot find entry symbol _start; defaulting to 0x1000
llvm-ar: error: unable to load 'arch/x86/kernel/head_64.o': file too small to be an archive
llvm-ar: error: unable to load 'arch/x86/kernel/head64.o': file too small to be an archive
llvm-ar: error: unable to load 'arch/x86/kernel/ebda.o': file too small to be an archive
llvm-ar: error: unable to load 'arch/x86/kernel/platform-quirks.o': file too small to be an archive
WARNING: EXPORT symbol "page_offset_base" [vmlinux] version generation failed, symbol will not be versioned.
WARNING: EXPORT symbol "vmalloc_base" [vmlinux] version generation failed, symbol will not be versioned.
WARNING: EXPORT symbol "vmemmap_base" [vmlinux] version generation failed, symbol will not be versioned.
WARNING: "__memcat_p" [vmlinux] is a static (unknown)
no symbols

Adjust your .config as you want (but, again, make sure the CC and LD args are pointed at Clang and LLD respectively). This should(!) result in a happy bootable x86 CFI-enabled kernel. If you want to see what a CFI failure looks like, you can poke LKDTM:

# Log into the booted system as root, then:
cat <(echo CFI_FORWARD_PROTO) >/sys/kernel/debug/provoke-crash/DIRECT

Here’s the CFI splat I see on the console:

[   16.288372] lkdtm: Performing direct entry CFI_FORWARD_PROTO
[   16.290563] lkdtm: Calling matched prototype ...
[   16.292367] lkdtm: Calling mismatched prototype ...
[   16.293696] ------------[ cut here ]------------
[   16.294581] CFI failure (target: lkdtm_increment_int$53641d38e2dc4a151b75cbe816cbb86b.cfi_jt+0x0/0x10):
[   16.296288] WARNING: CPU: 3 PID: 2612 at kernel/cfi.c:29 __cfi_check_fail+0x38/0x40
[   16.346873] ---[ end trace 386b3874d294d2f7 ]---
[   16.347669] lkdtm: Fail: survived mismatched prototype function call!

The claim of “Fail: survived …” is due to CONFIG_CFI_PERMISSIVE=y. This allows the kernel to warn but continue with the bad call anyway. This is handy for debugging. In a production kernel that would be removed and the offending kernel thread would be killed. If you run this again with the config disabled, there will be no continuation from LKDTM. :)

Enjoy! And if you can figure out before me why there is still CFI instrumentation in the KPTI entry handler, please let me know and help us fix it. ;)

© 2019, Kees Cook. This work is licensed under a Creative Commons Attribution-ShareAlike 3.0 License.
Creative Commons License

21 November, 2019 05:09AM by kees

November 20, 2019

Antoine Beaupré

a quick review of file watchers

File watchers. I always forget about those and never use then, but I constantly feel like I need them. So I made this list to stop searching everywhere for those things which are surprisingly hard to find in a search engine.


Those tools an watch files or trees of files and execute whatever.


  • 2013-2019, GNU project since 2014
  • C
  • GPL-3
  • Debian package, since 2015 (stretch), out of date (5.1 vs 5.2)
  • requires a config file to operate


  • 2012-2019
  • C
  • ISC
  • Debian package since 2014 (jessie), up to date
  • finds file from stdin
  • has special hacks to reload browser
  • can clear screen between calls


  • 2014-2018
  • C++
  • GPL-3
  • Debian package since 2017 (buster), up to date
  • outputs changesets using a specific syntax, so requires more commandline voodoo



  • 2006-2012, 2014-2015
  • C++
  • GPL-2
  • Debian package since 2007
  • Watches files and launches command in a cron-like (and rather obscure) syntax
  • no event deduplication


  • 2007-2012
  • C
  • GPL-2
  • Debian package is upstream
  • watches directory, fires command
  • yet another wrapper for inotify
  • built for reprepro
  • no event deduplication


  • 2012-2016
  • Debian package since 2013 (jessie)
  • Perl
  • GPL-1+
  • yet another wrapper for inotify
  • no event deduplication, but can "buffer" multiple events together with a timeout


systemd .path units

  • 2010-2019
  • C
  • GPL-2
  • Debian package since 2010
  • activates a system or user "unit" on inotify changes


  • 2016-2019
  • Rust
  • Apache-2.0
  • No Debian package
  • supports .gitignore files and filename patterns
  • merges multiple events
  • fast polling
  • can clear screen between calls


  • 2013-2019
  • C++, Python
  • Apache-2
  • Facebook
  • Debian package, since 2019 (bullseye)
  • can watch multiple directories
  • waits for settling

Web development

grip (markdown)

  • 2012-2018
  • Python
  • MIT
  • Debian package since 2016 (stretch)
  • rebuilds markdown files and servers them over HTTP with autoreload
  • relies on GitHub's API for rendering (???)


  • 2012-2019
  • Python
  • BSD-3
  • Debian package since 2013 (jessie), up to date
  • watches a directory and triggers a web browser livereload extension
  • also supports arbitrary commands through custom Python scripts
  • integration with Python frameworks (Django, Flask, Bottle)


Many frameworks automatically reload files when they change. A few examples:

Unit tests


  • 2012-2018 (no official release)
  • Python
  • No license
  • No Debian package
  • Reruns tests (but also any command) on file change in the current directory
  • Part of a continuous testing strategy on the desktop


  • 2008-2014 (no official release)
  • Python
  • MIT
  • No Debian package
  • inspiration for the above
  • restricted to a subset of Python test frameworks

File synchronization

I will not go through a list of all the file synchronization tools here. Most of them have some sort of "wake-up" system to notify file changes, but they are considered out of scope here unless they can also launch custom commands. But just for the sake of the argument, I am aware of:


  • 2008-2010 (archived upstream)
  • Python
  • BSD
  • Debian package since 2009 (jessie), in RFA
  • combination of rsync and inotify
  • similar to lsyncd


  • 2012-2018
  • Lua, C
  • GPL-2
  • Debian package since 2010 (jessie), up to date
  • spawns rsync on file changes
  • Lua configuration can be leveraged to do other things than sync

Integrity checks

Here again, there are many filesystem integrity checkers, but they are not relevant here unless they can also execute arbitrary commands. But, again, here's an idea of the other stuff that is out there that might be a better fit than trying to fit a square peg in this round hole:


  • 2006-2009
  • Perl
  • GPL-2+
  • Debian package since 2006
  • emails on file changes
  • can also execute commands, but primarily designed as an intrusion detection system
  • XML configuration (???)


kfmon (kobo launcher)

  • 2016-2019
  • C
  • GPL-3
  • No Debian package
  • Launches programs when icons are activated on a Kobo reader

I've probably missed half a million such programs so this might be expanded with time and your comments.

Update: I've added a bunch of tools. I was relunctant to add all those old inotify wrappers because I don't find them as interesting as the newer ones - they're really hard to use! - but I guess it's worth mentioning them even if just to criticise them. ;)

20 November, 2019 09:04PM

Birger Schacht

Converting ikiwiki to hugo

Sometimes I play around with Tails and on rare occasions I also build a Tails image myself. One thing that makes the build of Tails a bit tedious is that it a also builds the Tails Website, which contains the whole documentation (which is really cool, because that way users have the most up to date documentation on their desktop!). The problem is, that the website takes a looooong time to build- on my Laptop (i7-5600U) it takes around 11 minutes.

I was curious if it was possible to convert the whole website, which is based on ikiwiki, to the hugo static site generator which is known to be pretty fast (”with its amazing speed and flexibility, Hugo makes building websites fun again” as the hugo website puts it ;)). I did some research if there was some tooling to do so- the Hugo website lists some migration tools but nothing for ikiwiki, but I stumbled upon anarcat’s conversion notes which has a lot of information and also links to the write up jak did on his conversion. Anarcat also published a python script to convert ikiwiki to hugo which I tried, but there were some important parts missing. As it happened I also had to prepare for an exam at that time and I am a bit of a procrastinator, so I started adapting the script for the Tails wiki and then rewrote some parts and now its just another ikiwiki2hugo script:

What does it do?

For every ikiwiki *.mdwn or *.html file, it creates a Markdown file with hugo frontmatter. The information for the hugo frontmatter comes from parsing the [[!meta directives in the files.

If there is a *.po file with the same name, the script creates a * file with the strings translated.

Then the content is converted:

The content of the Markdown files is a copy of the content of the ikiwiki files with the ikiwiki directives replaced by hugo shortcodes. There are a lot of directives out there and I only implemented replacements for some of them. My test ikiwiki instance was the the Tails ikiwiki source, so I mostly implemented directives used there. Merge requests or patches for additional directives are welcome- to add a replacement for a directive, look in the directives folder, all the replacement python modules inherit from the Directive class. The shortcode files are assets/shortcodes folder.

What does it not do?

Probably the more important question ;)

There are a lot of directives whose replacements are not implemented. Most important probably the [[!inline directive with multiple (and negative) arguments and the [[!map directive.

Also, Hugo is more strict regarding Markdown syntax. It does not convert markdown syntax that is embedded in <div>...</div> and other block-level elements.

The result

Disclaimer: This comparison only looks at the build time- there is a lot of functionality of ikiwiki that hugo does not provide, for example the possibility to edit websites through the browser, which is used for the Tails blueprints. Another feature are the traillink directives which i simple replaced by normal markdown links.

Anarcat also provides a nice oneliner to check which directives are used in the ikiwiki repository:

grep -h -r '\[\[!' * | sed 's/\[\[!/\n[[!/g' | grep '\[\[!' | sed 's/ .*//' | sort | uniq -c | sort -n

Before running the script (I’ve removed listings of directives that occure less than 100 times):

    122 [[!wikipedia\n"
    124 [[!debsa2012
    130 [[!debsa2018
    134 [[!tails_roadmap
    152 [[!debsa2014
    188 [[!debsa2015
    204 [[!debsa2017
    219 [[!tails_gitweb_branch
    222 [[!debsa2016
    247 [[!tails_website
    283 [[!mfsa
    604 [[!cve
    655 [[!toggleable
    696 [[!tails_roadmap]]
    775 [[!"
    806 [[!tails_gitweb
   1143 [[!toggle
   1203 [[!wikipedia
   1704 [[!traillink
   2411 [[!pagetemplate
   2434 [[!toc
   2572 [[!tag
   5627 [[!inline
   5747 [[!tails_ticket
   8673 [[!img
  13577 [[!meta

After the conversion :

      1 [[!immagine
      1 [[!parent]]`.
      1 [[!tails_gitweb
      1 [[!tails_ticket
     30 [[!map
     66 [[!inline
(I think the immagine is a typo in a translation; I also found a couple of other typos which are fixed now; the tails_ticket directive is a code-quote from the release process document and the tails_gitweb is the one occurence of this directive which lists the description before the path ;))

Hugo takes nearly two seconds to build the website:

                   | DE  |  EN  | ES  | FA  | FR  | IT  | PT
  Pages            | 286 | 1023 | 286 | 286 | 286 | 286 | 286
  Paginator pages  |   0 |    0 |   0 |   0 |   0 |   0 |   0
  Non-page files   |   9 |  400 |   9 |   9 |  14 |   9 |   9
  Static files     |   0 |    0 |   0 |   0 |   0 |   0 |   0
  Processed images |   0 |    0 |   0 |   0 |   0 |   0 |   0
  Aliases          |   1 |    0 |   0 |   0 |   0 |   0 |   0
  Sitemaps         |   2 |    1 |   1 |   1 |   1 |   1 |   1
  Cleaned          |   0 |    0 |   0 |   0 |   0 |   0 |   0

Total in 1711 ms

Finally, a screeshot of the Tails website built with hugo- it looks pretty broken. Most of the syntax that does not work is because its embedded in HTML tags. Apparently hugo thinks about switching to another markdown renderer, so that could make a conversion from systems with a lot of mixed HTML/Markdown code easier.

Screenshot of the Tails website built with hugo

20 November, 2019 05:30PM

hackergotchi for Rhonda D'Vine

Rhonda D'Vine

Trans Day of Remembrance 2019

Today, November 20th, is Trans Day of Remembrance. It is about remembering victims of hate crimes that aren't amongst us anymore. Last year we learned about at least 331 murdered trans people, the real number is like always higher. Also like always it affects mostly trans women of color who are the target of multiple discriminatory patterns.

What is also a pattern is that Brazil has a fair chunk of those murders. Unfortunately the country since a while is on the top in that statistic, but the election of an right winged outspoken queer hating person as president of the country last year did make those who feel the hate having some sort of legitimacy to it, which makes it obviously harder to survive these days. My thoughts thus are specifically with the people of Brazil who fight for their survival.

Right-winged parties though rise all around the globe spreading hate, and as our Debian Free Software Guidelines say in #5, "No Discrimination Against Persons or Groups", and this is something that we can't limit only to software licenses but also have to extend to the way we work as community.

If you ask what you can do: Support your local community spaces and support groups. I had the pleasure to meet Grupo Dignidade during my stay in Curitiba for DebConf 19, and was very thankful for a representative of that group to join my Debian Diversity BoF. Thanks again, Ananda, it was lovely having you!

Meu Corpo é Político - my body is political.

/personal | permanent link | Comments: 0 | Flattr this

20 November, 2019 01:33PM by Rhonda

John Goerzen

The Incredible Disaster of Python 3

I have long noted issues with Python 3’s bytes/str separation, which is designed to have a type “bytes” that is a simple list of 8-bit characters, and “str” which is a Unicode string. After apps started using Python 3, I started noticing issues: they couldn’t open filenames that were in ISO-8859-1, gpodder couldn’t download podcasts with 8-bit characters in their title, etc. I have files on my system dating back to well before widespread Unicode support in Linux.

Due to both upstream and Debian deprecation of Python 2, I have been working to port pygopherd to Python 3. I was not looking forward to this task. It turns out that the string/byte types in Python 3 are even more of a disaster than I had at first realized.

Background: POSIX filenames

On POSIX platforms such as Unix, a filename consists of one or more 8-bit bytes, which may be any 8-bit value other than 0x00 or 0x2F (‘/’). So a file named “test\xf7.txt” is perfectly acceptable on a Linux system, and in ISO-8859-1, that filename would contain the division sign ÷. Any language that can’t process valid filenames has serious bugs – and Python is littered with these bugs.

Inconsistencies in Types

Before we get to those bugs, let’s look at this:

>>> "/foo"[0]
>>> "/foo"[0] == '/'
>>> b"/foo"[0]
>>> b"/foo"[0] == '/'     # this will fail anyhow because bytes never equals str
>>> b"/foo"[0] == b'/'
>>> b"/foo"[0] == b'/'[0]

Look at those last two items. With the bytes type, you can’t compare a single element of a list to a single character, even though you still can with a str. I have no explanation for this mysterious behavior, though thankfully the extensive tests I wrote in 2003 for pygopherd did cover it.

Bugs in the standard library

A whole class of bugs arise because parts of the standard library will accept str or bytes for filenames, while other parts accept only str. Here are the particularly egregious examples I ran into.

Python 3’s zipfile module is full of absolutely terrible code. As I reported in Python bug 38861, even a simple zipfile.extractall() fails to faithfully reproduce filenames contained in a ZIP file. Not only that, but there is egregious code like this in

            if flags & 0x800:
                # UTF-8 file names extension
                filename = filename.decode('utf-8')
                # Historical ZIP filename encoding
                filename = filename.decode('cp437')

I can assure you that zip on Unix was not mystically converting filenames from iso-8859-* to cp437 (which was from DOS, and almost unheard-of on Unix). Or how about this gem:

    def _encodeFilenameFlags(self):
            return self.filename.encode('ascii'), self.flag_bits
        except UnicodeEncodeError:
            return self.filename.encode('utf-8'), self.flag_bits | 0x800

This combines to a situation where perfectly valid filenames cannot be processed by the zipfile module, valid filenames are mangled on extraction, and unwanted and incorrect character set conversions are performed. zipfile has no mechanism to access ZIP filenames as bytes.

How about the dbm module? It simply has no way to specify a filename as bytes, and absolutely can’t open a file named “text\x7f”. There is simply no way to make that happen. I reported this in Python bug 38864.

Update 2019-11-20: As is pointed out in the comments, there is a way to encode this byte in a Unicode string in Python, so “absolutely can’t open” was incorrect. However, I strongly suspect that little code uses that approach and it remains a problem.

I should note that a simple open(b"foo\x7f.txt", "w") works. The lowest-level calls are smart enough to handle this, but the ecosystem built atop them is uneven at best. It certainly doesn’t help that things like b"foo" + "/" are runtime crashers.

Larger Consequences of These Issues

I am absolutely convinced that these are not the only two modules distributed with Python itself that are incapable of opening or processing valid files on a Unix system. I fully expect that these issues are littered throughout the library. Nobody appears to be testing for them. Nobody appears to care about them.

It is part of a worrying trend I have been seeing lately of people cutting corners and failing to handle valid things that have been part of the system for years. We are, by example and implementation, teaching programmers that these shortcuts are fine, that it’s fine to use something that is required to be utf-8 to refer to filenames on Linux, etc. A generation of programmers will grow up writing code that is incapable of processing files with perfectly valid names. I am thankful that grep, etc. aren’t written in Python, because if they were, they’d crash all the time.

Here are some other examples:

  • When running “git status” on my IBM3151 terminal connected to Linux, I found it would clear the screen each time. Huh. Apparently git assumes that if you’re using it from a terminal, the terminal supports color, and it doesn’t bother using terminfo; it just sends ANSI sequences assuming that everything uses them. The IBM3151 doesn’t by default. (GNU tools like ls get this right) This is but one egregious example of a whole suite of tools that fail to use the ncurses/terminfo libraries that we’ve had for years to properly abstract these things.
  • A whole suite of tools, including ssh, tmux, and so forth, blindly disable handling of XON/XOFF on the terminal, neglecting the fact that this is actually quite important for some serial lines. Thankfully I can at least wrap things in GNU Screen to get proper XON/XOFF handling.
  • The Linux Keyspan USB serial driver doesn’t even implement XON/XOFF handling at all.

Now, you might make an argument “Well, ISO-8859-* is deprecated. We’ve all moved on to Unicode!” And you would be, of course, wrong. Unix had roughly 30 years of history before xterm supported UTF-8. It would be quite a few more years until UTF-8 reached the status of default for many systems; it wasn’t until Debian etch in 2007 that Debian used utf-8 by default. Files with contents or names in other encoding schemes exist and people find value in old files. “Just rename them all!” you might say. In some situations, that might work, but consider — how many symlinks would it break? How many scripts that refer to things by filenames would it break? The answer is most certainly nonzero. There is no harm in having files laying about the system in other encoding schemes — except to buggy software that can’t cope. And this post doesn’t even concern the content of files, which is a whole additional problem, though thankfully the situation there is generally at least somewhat better.

There are also still plenty of systems that can’t handle multibyte characters (and in various embedded or mainframe contexts, can’t even handle 8-bit characters). Not all terminals support ANSI. It requires only correct thinking (“What is a valid POSIX filename? OK, our datatypes better support that then”) to do the right thing.

20 November, 2019 09:48AM by John Goerzen

November 19, 2019

hackergotchi for Joachim Breitner

Joachim Breitner

Faster Winter 3: Difference Lists

(This is the third optimization presented in the “faster winter” series, please see that post for background information.)

Today, we will tackle the big bad memory leak that eats up my laptop’s memory.

A quick look at the heap profile (+RTS -h) showed that the memory was filling up with lists. Not very helpful, as lists are everywhere. So I looked through the hot code of the interpreter, eval, step and instr in Wasm.Exec.Eval to see if anything fishy is going on. I found some uses of the list concatenation operator (++) – always a bad sign, as it has to traverse the list on its left completely!

And often the solution is pretty simple: Use difference lists! It’s even simpler than the name makes it sound like. It does not require you to import anything new, and works well everywhere where you assemble a list in multiple stages, but use it, in its full form, only once at the end. The trick is easy:

  • In the types, replace [t] with [t] -> [t] (or introduce an alias type DList a = [a] -> [a])
  • Replace [] with id
  • Replace [x] with (x:)
  • Replace xs ++ ys with xs . ys, if ys is also a difference list
  • Replace xs ++ ys with xs ys, if ys is a normal list
  • To turn the difference list xs into a normal list, write xs []

That’s it, and suddenly you have a list like data structure with constant-time concatenation!

Improvement: Allocations: -9.67% Memory: -99.08% Time: -47.36% (Commit 2e284f8...f9bbe8e)

Memory consumption is now at 60MB. I found some more places where I could use difference lists, and then it was finally at essentially zero, which is what you would expect for the program at hand:

Improvement: Allocations: -0.21% Memory: -96.83% Time: -2.91% (Commit f9bbe8e...bbe8af7)

To be honest, I am not actually sure why this fixed a space leak: Difference lists are not more memory efficient than normal lists! But maybe somehow these lists were more strictly evaluated once they were difference lists? Anyways, I was happy with the result and did not investigate further.

19 November, 2019 11:09PM by Joachim Breitner (

John Goerzen

A Mystery of Unix History

I wrote recently about buying a Digital (DEC) vt420 and hooking it up to Linux. Among my observations on the vt420, which apparently were among the most popular to use with Unix systems, are these:

  • DEC keyboards had no Esc key
  • DEC keyboards had no key that could be used as Alt or Meta

The two most popular historic editors on Unix, vi and emacs, both make heavy use of these features (Emacs using Esc when Alt or Meta is unavailable). Some of the later entries in the DEC terminal line, especially the vt510, supported key remapping or alternative keyboards, which can address the Esc issue, but not entirely.

According to the EmacsOnTerminal page and other research, at least the vt100 through the vt420 lacked Esc by default. Ctrl-3 and Ctrl-[ could send the character. However, this is downright terrible for both vi and Emacs (as this is the only way to trigger meta commands in Emacs).

What’s more, it seems almost none of these old serial terminal support hardware flow control, and flow control is an absolute necessity on many. That implies XON/XOFF, which use Ctrl-S and Ctrl-Q — both of which are commonly used in Emacs.

Both vi and Emacs trace their roots back to the 1970s and were widely used in the serial terminal era, running on hardware dominated by DEC and its serial terminals.

So my question is: why would both of these editors be developed in such a way that they are downright inconvenient to use on the hardware on which they most frequently ran?

Update 2019-11-20: It appears that the vt100 did have the Esc key, but it was dropped with the vt220. At least the vt420 and later, and possibly as far back as the vt220, let you map one of a few other keys to be Esc. This still leaves the Ctrl-S mystery in Emacs though.

19 November, 2019 07:21PM by John Goerzen

Mark Brown

Audio Miniconf 2019 Report

This year’s audio miniconference happened last month in Lyon, sponsored by Intel. Thanks to everyone who attended, this event is all about conversations between people, and to Alexandre Belloni for organizing the conference dinner.

We started off with Curtis Malainey talking us through some UCM extensions that ChromeOS has been using. There was general enthusiasm for the changes that were proposed, discussion mainly revolved around deployment issues with handling new features, especially where topology files are involved. If new controls are added in a topology file update then some system is needed to ensure that UCM files to configure them are updated in sync or work without the update. No clear answers were found, one option would be to combine UCM and topology files but it’s not clear that this is useful in general if the configuration is done separately to the DSP development.

Daniel Baluta then started some discussion of topics related to Sound Open Firmware (slides). The first was issues with loading firmware before the filesystems are ready, we agreed that this can be resolved through the use of the _nowait() APIs. More difficult was resolving how to deal with card initialization. Currently the only complete in-tree users are x86 based so have to deal with the problems with the incomplete firmware descriptions provided by ACPI, there’s nothing standards based like we have for device tree systems, and assumptions about that have crept into how the code works. It’s going to take a bunch of work to implement but we came to a reasonable understanding of how this should work, with the DSP represented as a device in the device tree and bound to the card like any other component.

Continuing on the DSP theme Patrick Lai then lead a discussion of gapless playback with format switches, we agreed that allowing set_params() to be called multiple times on a single stream when the driver could support it was the most sensible approach. The topic of associating controls with PCM streams was also discussed, there are some old APIs for this but so little hardware has implemented them that we agreed that a convention for control names based on the stream names was probably easier to support with current userspace software.

Patrick also lead a discussion of time synchronization for audio streams, both compressed and PCM. A number of systems, especially those dealing with broadcast media, want to bring the local audio clocks as closely in sync with other system and external clocks for both long term streams and A/V sync. As part of doing this there is a desire to embed timestamps into the audio stream for use by DSPs. There was an extensive discussion on how to do this, the two basic options being to add an extended audio format which includes timestamps (in the compressed API) or additional API calls to go along with the data. The in band data is easier to align with the audio but the format modifications will make it much harder to implement in a standard way while the out of band approach is harder to align but easier to standardize. We came to a good understanding of the issues and agreed that it’s probably best to evaluate this in terms of concrete API proposals on the list.

Liam Girdwood then took over and gave an overview of the status of SoundWire. This will be reaching products soon, with a gradual transition so all current interfaces are still in active use for the time being. Immediate plans for development include a lot of work on hardening the framework to deal with corner cases and missing corners of the spec that are identified in implementation, support for applications where the host system is suspended while a DSP and CODEC implement features like hotword detection and support for dynamic routing on the SoundWire bus. Those with SoundWire hardware agreed that this was a good set of priorities.

We then moved on to a discussion of ABI stability and consistency issues with control names, lead by Takashi Iwai. The main focus was around control and stream naming where we have a specification which userspace does use but we currently only verify that it’s being followed manually during kernel development which isn’t great. We decided that a testing tool similar to v4l-complaince which is successfully used by the video4linux community would be the most viable option here, though there were no immediate volunteers to write the tool.

The next topic was virtualization, this was mainly a heads up that there is some discussion going on in OASIS around a VirtIO specification for audio.

Jerome Brunet then talked through his experiences as a new contributor implementing audio support for Amlogic SoCs. Their audio subsystems are relatively simple by modern SoC standards but still more complex than the simple DAIs that ASoC currently supports well, needing elements of DPCM and CODEC-CODEC link support to work with the current system all of which presented an excessively tough learning curve to him. Sadly all the issues he faced were already familiar and we even have some good ideas for improving the situation through moving to a component based model. Morimoto-san (who was sadly unable to attend) has been making great strides in converting all the drivers into component drivers which makes the core changes tractable but we still need someone to take up the core work. Charles Keepax has started some work on this before and says he hopes to find some time soon, with several other people indicating an interest in helping out so hopefully we might see some progress on this in the next year.

The final topic on the agenda was spreading DSP load throughout the system, including onto the sound server running on the host CPU if the dedicated DSP hardware was overloaded. Ideally we’d be able to do this in a transparent fashion, sharing things like calibration coefficients between DSPs. The main sticking point with implementing this is Android systems since Android doesn’t use full alsa-lib and therefore doesn’t share any interfaces above the kernel layer with other systems. It’s likely that something can be implemented for most other systems but it’ll almost certainly need separate Android integration even if plugins can be shared.

We did have some further discussions on a number of topics, including testing, after working through the agenda but sadly minutes weren’t being kept for those so . Thanks again to the attendees, and to Intel for sponsoring.

19 November, 2019 12:15PM by broonie

hackergotchi for Wouter Verhelst

Wouter Verhelst

extrepo followup

My announcement the other day has resulted in a small amount of feedback already (through various channels), and a few extra repositories to be added. There was, however, enough feedback (and the manner of it unstructured enough) that I think it's time for a bit of a follow-up:

  • If you have any ideas about future directions this should take, then thank you! Please file them as issues against the extrepo-data repository, using the wishlist tag (if appropriate).
  • If you have a repository that you'd like to see added, awesome! Ideally you would just file a merge request; Vincent Bernat already did so, and I merged it this morning. If you can't figure out how to do so, then please file a bug with the information needed for the repository (deb URI, gpg key signing the repository, description of what type of software the repository contains, supported suites and components), and preferably also pointing to the gap in the documentation that makes it hard for you to understand, so I can improve ;-)
  • One questioner asked why we don't put third-party repository metadata into .deb packages, and ship them that way. The reason for this is that the Debian archive doesn't change after release, which is the point where most external repositories would be updated for the new release. As such, while it might be useful to have something like that in certain cases (e.g., the (now-defunct) pkg-mozilla-archive-keyring package follows this scheme), it can't work in all cases. In contrast, extrepo is designed to be updateable after the release of a particular Debian suite.

19 November, 2019 11:36AM

Mike Hommey

Five years of git-cinnabar

On this very day five years ago, I committed the initial code of what later became git-cinnabar. It is kind of an artificial anniversary, because I didn’t actually publish anything until 3 weeks later, and I also had some prototypes months earlier.

The earlier prototypes of what I’ll call “pre-git-cinnabar” could handle doing git clone hg:: (that is, creating a git clone of a Mercurial repository), but they couldn’t git pull later. That pre-git-cinnabar initial commit, however, was the first version that did.

The state of the art back then was similar git helpers, the most popular choice being Felipec’s git-remote-hg, or the opposite tool: hg-git, a mercurial plugin that allows to push to a git repository.

They both had the same caveats: they were slow to handle a repository the size of mozilla-central back then, and both required a local mercurial repository (hidden in the .git directory in the case of Felipec’s git-remote-hg).

This is what motivated me to work on pre-git-cinnabar, which was also named git-remote-hg back then because of how git requires a git-remote-hg executable to handle hg::-prefixed urls.

Fast forward five years, mozilla-central has grown tremendously, and another mozilla-unified repository was created that aggregates the various mozilla branches (esr*, release, beta, central, integration/*).

git-cinnabar went through multiple versions, multiple changes to the metadata it keeps, and while I actually haven’t cumulatively worked all that much on it considering the number of years, a lot of progress has been made.

But let’s go back to the 19th of November 2014. Thankfully, Mercurial allows to strip everything past a certain date, artificially allowing to restore the state of the repository at that date. Unfortunately, pre-git-cinnabar only supports the old Mercurial bundle format, which both the mozilla-central and mozilla-unified repositories now don’t allow. So pre-git-cinnabar can’t clone them out of the box anymore. It’s still possible to allow it in mirror repositories, but because they now use generaldelta, that incurs a server-side conversion that is painfully slow (the server rejects clients that don’t support the new format for this reason).

So for testing purposes, I setup a nginx reverse-proxy and cache, such that the conversion only happens once, and performed clones multiple times, removing any bundling and conversion cost out of the equation. And tested the current version of Felipec’s git-remote-hg, the current version of hg-git, pre-git-cinnabar, and last git-cinnabar release (0.5.2 as of writing), on some AWS instances, with Xeon Platinum 8124M 3Ghz CPUs. That’s a different CPU from what I had back in 2014, yielding some different results from what I wrote in that first announcement.

I’ve thus cloned both mozilla-central (denoted m-c) and mozilla-unified (denoted m-u), with simulated old states of the repositories. Mozilla-unified didn’t exist before 2016, but it’s still interesting to simulate its content as if it had existed because it allows to learn how the tools perform with the additional branches it contains, with the implication they have on how the data is stored in the repository.

Note: I didn’t test older versions of git-remote-hg or hg-git to see how they performed at the time, and how things evolved for them.

Clone times in 2014

There are multiple things of note in the results above:

  • I wrote back then that cloning took 12 hours with git-remote-hg and 30 minutes with pre-git-cinnabar on the machine I used. And while cloning with pre-git-cinnabar on more modern hardware was much faster (16 minutes), cloning with git-remote-hg wasn’t. The pre-git-cinnabar improvement could, though, be attributed in part to improvements in git-fast-import itself (I haven’t tested older versions). But it’s also not impossible that git-remote-hg regressed. Only further testing would tell.
  • mozilla-unified is bigger than mozilla-central, because it is a superset, and that reflects on the clone times, but hg-git and pre-git-cinnabar are both much slower to clone mozilla-unified than you’d expect from the difference in size, especially hg-git. git-cinnabar made a lot of progress in that regard.
  • I hadn’t tested hg-git back then, but while it’s not as slow as git-remote-hg, it’s still horribly slow for a repository this size.

Let’s now look at the .git sizes:

.git sizes in 2014

Those are the sizes for the .git directory fresh after cloning. In all cases, git gc --aggressive would make the clone smaller, at the cost of CPU time (although not significantly smaller in the git-cinnabar case). And after you spent 12 hours cloning, are you really going to spend another large number of hours on a git gc to save disk space?

It is worth noting that in the case of hg-git, this doesn’t include the size of the mercurial repository required to maintain the git repository, while it is included for git-remote-hg, where it is hidden in .git, as mentioned earlier. That puts them about on par w.r.t size.

It’s interesting how close hg-git and git-remote-hg are in disk usage, when the former uses dulwich, a pure Python implementation of Git, and the latter uses git-fast-import. pre-git-cinnabar used git-fast-import too, but optimized the data it sent to git-fast-import to allow for a more compact .git. Recent git-cinnabar made it even better, although it doesn’t use git-fast-import directly, instead using a helper derived from git-fast-import.

But that was 2014. Let’s look at how things evolved over time, by taking “snapshots” of the repositories at one year interval, starting in November 2007.

Clone times over time

Of note:

  • pre-git-cinnabar somehow invalidated the nginx cache for years >= 2016 for mozilla-unified, which didn’t allow to get reliable measures.
  • Things went well out of hand with git-remote-hg and hg-git, so much so that I wasn’t able to get results for git-remote-hg clones for 2019 in time for this post. They’re now getting clone times that count in days rather than hours.
  • Things are getting much worse with mozilla-unified, relatively to mozilla-central, for hg-git than they do for git-remote-hg or git-cinnabar, while they were really bad with pre-git-cinnabar.
  • pre-git-cinnabar clone times for mozilla-central are indistinguishable from git-cinnabar’s at this scale (but see further below).
  • the progression is not linear, but the progression in repository size wasn’t linear either. In order to get a slightly better picture, it is better to look at the clone times vs. the size of the repositories. One measure of that size is number of objects (changeset, manifests and file revisions they contain).

Clone times over repo size

The progression here looks more linear, but still not quite linear. The difference between the mozilla-central and mozilla-unified clone times is the most damning, especially for hg-git and pre-git-cinnabar. At this scale things don’t look so bad for git-cinnabar, but looking closer, they aren’t actually much better:

Clone times over repo size, pre-git-cinnabar and git-cinnabar only

mozilla-central clone times have slightly improved since pre-git-cinnabar days, at least more than the comparison with hg-git and git-remote-hg suggested. mozilla-unified clone times, however, have dramatically improved (notwithstanding the fact that it’s currently not possible to clone with pre-git-cinnabar at all directly from

But clone times are starting to get a little out of hand, especially for mozilla-unified, which is why I’ve recently added support for “clone bundles”. But I also have work in progress that I expect will make non-bundle clones faster too, and hopefully more linear.

As for .git sizes:

.git sizes over repo size

  • hg-git and git-remote-hg are still hand in hand.
  • Here the progression is mostly linear, with almost no difference between mozilla-central and mozilla-unified, as one could expect.
  • I think the larger increase in size between what would be 2017 and 2018 is largely due to the testing/web-platform/meta/MANIFEST.json file.
  • People who try to clone the Mozilla repositories with hg-git or git-remote-hg at this point better have a lot of time and a lot of free disk space.

While git-cinnabar is demonstrably significantly faster than both git-remote-hg and hg-git by a large margin for the Mozilla repositories (more than an order of magnitude), looking at the data more closely revealed something interesting that can be pictured in the following graph, plotting how much slower than git-cinnabar the other tools are.

Clone time ratios against git-cinnabar

The ratio is not constant, and has surprisingly been decreasing steadily since 2016, correlating with the observation that clone times are getting slower more quickly than the repositories are growing. But they are doing more so with git-cinnabar than with the other tools. Clone times with git-cinnabar have multiplied by more than 5 in five years, for a repository that only has 2.5 times more objects. At this pace, in five more years, clones will take well above 10 hours, and that’s not counting for the accelerated slowdown. Hopefully, the current work in progress will help.

It’s also interesting to see how the ratios changed after 2011 between mozilla-central and mozilla-unified. 2011 is when Firefox 4 was released and the release process switched to multiple repositories, which mozilla-unified, well, unified in a single repository. So mozilla-unified and mozilla-central were largely identical when stripped of commits after 2011 and started diverging afterwards.

To conclude this rather long post, pre-git-cinnabar improved the state of the art to clone large Mercurial repositories, and git-cinnabar went further in the past five years. But without more work, things will get out of hand. And that only accounts for clone times. I haven’t spent much time working on other aspects, like negotiation times (the time it takes to agree with a Mercurial server what the git clone has in common with it), or bundling times (the time it takes to generate a bundle to send a Mercurial server). Both are the relevant parts of push times.

19 November, 2019 09:04AM by glandium

hackergotchi for Dirk Eddelbuettel

Dirk Eddelbuettel

ttdo 0.0.4: Extension

A first update release to the still very new (and still very small) ttdo package arrived on CRAN today. Introduced about two months ago in September, the ttdo package extends the most excellent (and very minimal / zero depends) unit testing package tinytest by Mark van der Loo with the very clever and well-done diffobj package by Brodie Gaslam.

ttdo screenshot

Just how the package creation was motivated by our needs in teaching STAT 430 at Illinois, so does the extension code in this release which generalized how we extend the tinytest test predicates with additional arguments which help in the use of the PrairieLearn system (developed at Illinois) to provide tests, quizzes or homework. This release is mostly the work of Alton who is now also a coauthor.

The NEWS entries follow.

Changes in ttdo version 0.0.4 (2019-11-18)

  • Generalize tinytest extensions with additional arguments for test predicates (Alton in #2).

  • Use Travis CI for continuous integration off GitHub (Dirk).

Please use the GitHub repo and its issues for any questions.

If you like this or other open-source work I do, you can now sponsor me at GitHub. For the first year, GitHub will match your contributions.

This post by Dirk Eddelbuettel originated on his Thinking inside the box blog. Please report excessive re-aggregation in third-party for-profit settings.

19 November, 2019 01:12AM

November 18, 2019

hackergotchi for Joachim Breitner

Joachim Breitner

Faster Winter 2: SPECIALIZE

(This is the second optimization presented in the “faster winter” series, please see that post for background information.)

The evaluator of the WebAssembly interpreter is highly polymorphic, as you can tell from the type signature of its main function:

type CEvalT f m a = ReaderT (Config f m) (ExceptT EvalError m) a
eval :: (Regioned f, MonadRef m, Show1 f)
     => Code f m -> CEvalT f m (Stack Value)

This mean the caller can choose whatever monad m this should use. This is very flexible, but it is also really hard to make that run fast: Every line in do notation is a call to the monadic bind operator (>>=), and since this operator is provided by the caller, the code of eval has to do do many calls to this unknown function.

GHC can do a much better job at optimizing this code if it knows which concrete monad is used, so that it knows the actual definition of that monad’s (>>=). Because then the compiler can inline it and make the whole monad overhead go away.

The wasm-invoke program actually uses the plain IO monad for m. We could now rewrite the whole module to not be polymorphic in m, but use IO instead. But that’s a lot of work and some user might genuinely want a different monad (e.g. ST s, or something else).

Luckily we can tell the compiler: “Let’s keep eval with this general type signature, but please also provide versions specifically for IO, and use that if possible”, by giving a SPECIALIZE pragma:

{-# SPECIALIZE eval ::
       Code Identity IO -> CEvalT Identity IO (Stack Value) #-}

This runs much faster now:

Improvement: Allocations: -48.65% Memory: +52.56% Time: -50.29% (Commit 4061fe6...2e284f8)

Again, as when we introduced mutable vectors, memory usage increases a bit, likely due to more stuff happening between garbage collections.

The code runs faster, but memory consumption is still absurd. So it is high time to hunt down the space leak, as the interpreter should not need any significant amounts of memory, and instead execute the program using a constant and low amount of memory. More about that in the next entry of this series.

18 November, 2019 10:41PM by Joachim Breitner (

hackergotchi for Steve Kemp

Steve Kemp

Adventures optimizing a bytecode based scripting language

I've recently spent some time working with a simple scripting language. Today I spent parts of the day trying to make it faster. As a broad overview we'll be considering this example script:

  if ( 1 + 2 * 3 == 7 ) { return true; }

  return false;

This gets compiled into a simple set of bytecode instructions, which can refer to a small collection of constants - which are identified by offset/ID:

  000000	    OpConstant	0	// load constant: &{1}
  000003	    OpConstant	1	// load constant: &{2}
  000006	    OpConstant	2	// load constant: &{3}
  000009	         OpMul
  000010	         OpAdd
  000011	    OpConstant	3	// load constant: &{7}
  000014	       OpEqual
  000015	 OpJumpIfFalse	20
  000018	        OpTrue
  000019	      OpReturn
  000020	       OpFalse
  000021	      OpReturn

  000000 Type:INTEGER Value:1
  000001 Type:INTEGER Value:2
  000002 Type:INTEGER Value:3
  000003 Type:INTEGER Value:7

The OpConstant instruction means to load the value with the given ID from the constant pool and place it onto the top of the stack. The multiplication and addition operations both pop values from the stack, apply the appropriate operation and push the result back. All standard stuff.

Of course these constants are constant so it seemed obvious to handle the case of integers as a special case. Rather than storing them in the constant-pool, where booleans, strings, etc live, we just store them inline. That meant our program would look like this:

  000000	        OpPush	1
  000003	        OpPush	2
  000006	        OpPush	3
  000009	         OpMul
  000010	         OpAdd

At this point the magic begins: we can scan the program from start to finish. Every time we find "OpPush", "OpPush", then "Maths" we can rewrite the program to contain the appropriate result already. So this concrete fragment:

OpPush 2
OpPush 3

is now replaced by:

OpPush	6
OpNop     ; previous opconstant
OpNop     ; previous arg1
OpNop     ; previous arg2
OpNop     ; previous OpMul

Repeating the process, and applying the same transformation to our comparision operation we now have the updated bytecode of:

  000000	         OpNop
  000001	         OpNop
  000002	         OpNop
  000003	         OpNop
  000004	         OpNop
  000005	         OpNop
  000006	         OpNop
  000007	         OpNop
  000008	         OpNop
  000009	         OpNop
  000010	         OpNop
  000011	         OpNop
  000012	         OpNop
  000013	         OpNop
  000014	        OpTrue
  000015	 OpJumpIfFalse	20
  000018	        OpTrue
  000019	      OpReturn
  000020	       OpFalse
  000021	      OpReturn

The OpTrue instruction pushes a "true" value to the stack, while the OpJumpIfFalse only jumps to the given program offset if the value popped off the stack is non-true. So we can remove those two instructions.

Our complete program is now:

  000000	         OpNop
  000001	         OpNop
  000002	         OpNop
  000003	         OpNop
  000004	         OpNop
  000005	         OpNop
  000006	         OpNop
  000007	         OpNop
  000008	         OpNop
  000009	         OpNop
  000010	         OpNop
  000011	         OpNop
  000012	         OpNop
  000013	         OpNop
  000014	         OpNop
  000015	         OpNop
  000018	        OpTrue
  000019	      OpReturn
  000020	       OpFalse
  000021	      OpReturn

There are two remaining obvious steps:

  • Remove all the OpNop instructions.
  • Recognize the case that there are zero jumps in the generated bytecode.
    • In that case we can stop processing code once we hit the first OpReturn

With that change made our input program:

if ( 1 + 2 * 3 == 7 ) { return true; } return false;

Now becomes this compiled bytecode:

  000000	        OpTrue
  000001	      OpReturn

Which now runs a lot faster than it did in the past. Of course this is completely artificial, but it was a fun process to work through regardless.

18 November, 2019 05:45PM

Sylvain Beucler

SCP Foundation needs you!

SCP is a mind-blowing, diverse, high-quality collection of writings and illustrations, all released under the CC-BY-SA free license.
If you never read horror stories written with scientific style -- have a try :)

[obviously this has nothing to do with OpenSSH Secure CoPy ;)]

Faced with a legal threat through the aggressive use of a RU/EU trademark, the SCP project is raising a legal fund.
I suggest you have a look.

18 November, 2019 12:57PM

Russell Coker

4K Monitors

A couple of years ago a relative who uses a Linux workstation I support bought a 4K (4096*2160 resolution) monitor. That meant that I had to get 4K working, which was 2 years of pain for me and probably not enough benefit for them to justify it. Recently I had the opportunity to buy some 4K monitors at a low enough price that it didn’t make sense to refuse so I got to experience it myself.

The Need for 4K

I’m getting older and my vision is decreasing as expected. I recently got new glasses and got a pair of reading glasses as a reduced ability to change focus is common as you get older. Unfortunately I made a mistake when requesting the focus distance for the reading glasses and they work well for phones, tablets, and books but not for laptops and desktop computers. Now I have the option of either spending a moderate amount of money to buy a new pair of reading glasses or just dealing with the fact that laptop/desktop use isn’t going to be as good until the next time I need new glasses (sometime 2021).

I like having lots of terminal windows on my desktop. For common tasks I might need a few terminals open at a time and if I get interrupted in a task I like to leave the terminal windows for it open so I can easily go back to it. Having more 80*25 terminal windows on screen increases my productivity. My previous monitor was 2560*1440 which for years had allowed me to have a 4*4 array of non-overlapping terminal windows as well as another 8 or 9 overlapping ones if I needed more. 16 terminals allows me to ssh to lots of systems and edit lots of files in vi. Earlier this year I had found it difficult to read the font size that previously worked well for me so I had to use a larger font that meant that only 3*3 terminals would fit on my screen. Going from 16 non-overlapping windows and an optional 8 overlapping to 9 non-overlapping and an optional 6 overlapping is a significant difference. I could get a second monitor, and I won’t rule out doing so at some future time. But it’s not ideal.

When I got a 4K monitor working properly I found that I could go back to a smaller font that allowed 16 non overlapping windows. So I got a real benefit from a 4K monitor!

Video Hardware

Version 1.0 of HDMI released in 2002 only supports 1920*1080 (FullHD) resolution. Version 1.3 released in 2006 supported 2560*1440. Most of my collection of PCIe video cards have a maximum resolution of 1920*1080 in HDMI, so it seems that they only support HDMI 1.2 or earlier. When investigating this I wondered what version of PCIe they were using, the command “dmidecode |grep PCI” gives that information, seems that at least one PCIe video card supports PCIe 2 (released in 2007) but not HDMI 1.3 (released in 2006).

Many video cards in my collection support 2560*1440 with DVI but only 1920*1080 with HDMI. As 4K monitors don’t support DVI input that meant that when initially using a 4K monitor I was running in 1920*1080 instead of 2560*1440 with my old monitor.

I found that one of my old video cards supported 4K resolution, it has a NVidia GT630 chipset (here’s the page with specifications for that chipset [1]). It seems that because I have a video card with 2G of RAM I have the “Keplar” variant which supports 4K resolution. I got the video card in question because it uses PCIe*8 and I had a workstation that only had PCIe*8 slots and I didn’t feel like cutting a card down to size (which is apparently possible but not recommended), it is also fanless (quiet) which is handy if you don’t need a lot of GPU power.

A couple of months ago I checked the cheap video cards at my favourite computer store (MSY) and all the cheap ones didn’t support 4K resolution. Now it seems that all the video cards they sell could support 4K, by “could” I mean that a Google search of the chipset says that it’s possible but of course some surrounding chips could fail to support it.

The GT630 card is great for text, but the combination of it with a i5-2500 CPU (rating 6353 according to [3]) doesn’t allow playing Netflix full-screen and on 1920*1080 videos scaled to full-screen sometimes gets mplayer messages about the CPU being too slow. I don’t know how much of this is due to the CPU and how much is due to the graphics hardware.

When trying the same system with an ATI Radeon R7 260X/360 graphics card (16* PCIe and draws enough power to need a separate connection to the PSU) the Netflix playback appears better but mplayer seems no better.

I guess I need a new PC to play 1920*1080 video scaled to full-screen on a 4K monitor. No idea what hardware will be needed to play actual 4K video. Comments offering suggestions in this regard will be appreciated.

Software Configuration

For GNOME apps (which you will probably run even if like me you use KDE for your desktop) you need to run commands like the following to scale menus etc:

gsettings set org.gnome.settings-daemon.plugins.xsettings overrides "[{'Gdk/WindowScalingFactor', <2>}]"
gsettings set org.gnome.desktop.interface scaling-factor 2

For KDE run the System Settings app, go to Display and Monitor, then go to Displays and Scale Display to scale things.

The Arch Linux Wiki page on HiDPI [2] is good for information on how to make apps work with high DPI (or regular screens for people with poor vision).


4K displays are still rather painful, both in hardware and software configuration. For serious computer use it’s worth the hassle, but it doesn’t seem to be good for general use yet. 2560*1440 is pretty good and works with much more hardware and requires hardly any software configuration.

18 November, 2019 10:56AM by etbe

hackergotchi for Matthew Garrett

Matthew Garrett

Extending proprietary PC embedded controller firmware

I'm still playing with my X210, a device that just keeps coming up with new ways to teach me things. I'm now running Coreboot full time, so the majority of the runtime platform firmware is free software. Unfortunately, the firmware that's running on the embedded controller (a separate chip that's awake even when the rest of the system is asleep and which handles stuff like fan control, battery charging, transitioning into different power states and so on) is proprietary and the manufacturer of the chip won't release data sheets for it. This was disappointing, because the stock EC firmware is kind of annoying (there's no hysteresis on the fan control, so it hits a threshold, speeds up, drops below the threshold, turns off, and repeats every few seconds - also, a bunch of the Thinkpad hotkeys don't do anything) and it would be nice to be able to improve it.

A few months ago someone posted a bunch of fixes, a Ghidra project and a kernel patch that lets you overwrite the EC's code at runtime for purposes of experimentation. This seemed promising. Some amount of playing later and I'd produced a patch that generated keyboard scancodes for all the missing hotkeys, and I could then use udev to map those scancodes to the keycodes that the thinkpad_acpi driver would generate. I finally had a hotkey to tell me how much battery I had left.

But something else included in that post was a list of the GPIO mappings on the EC. A whole bunch of hardware on the board is connected to the EC in ways that allow it to control them, including things like disabling the backlight or switching the wifi card to airplane mode. Unfortunately the ACPI spec doesn't cover how to control GPIO lines attached to the embedded controller - the only real way we have to communicate is via a set of registers that the EC firmware interprets and does stuff with.

One of those registers in the vendor firmware for the X210 looked promising, with individual bits that looked like radio control. Unfortunately writing to them does nothing - the EC firmware simply stashes that write in an address and returns it on read without parsing the bits in any way. Doing anything more with them was going to involve modifying the embedded controller code.

Thankfully the EC has 64K of firmware and is only using about 40K of that, so there's plenty of room to add new code. The problem was generating the code in the first place and then getting it called. The EC is based on the CR16C architecture, which binutils supported until 10 days ago. To be fair it didn't appear to actually work, and binutils still has support for the more generic version of the CR16 family, so I built a cross assembler, wrote some assembly and came up with something that Ghidra was willing to parse except for one thing.

As mentioned previously, the existing firmware code responded to writes to this register by saving it to its RAM. My plan was to stick my new code in unused space at the end of the firmware, including code that duplicated the firmware's existing functionality. I could then replace the existing code that stored the register value with code that branched to my code, did whatever I wanted and then branched back to the original code. I hacked together some assembly that did the right thing in the most brute force way possible, but while Ghidra was happy with most of the code it wasn't happy with the instruction that branched from the original code to the new code, or the instruction at the end that returned to the original code. The branch instruction differs from a jump instruction in that it gives a relative offset rather than an absolute address, which means that branching to nearby code can be encoded in fewer bytes than going further. I was specifying the longest jump encoding possible in my assembly (that's what the :l means), but the linker was rewriting that to a shorter one. Ghidra was interpreting the shorter branch as a negative offset, and it wasn't clear to me whether this was a binutils bug or a Ghidra bug. I ended up just hacking that code out of binutils so it generated code that Ghidra was happy with and got on with life.

Writing values directly to that EC register showed that it worked, which meant I could add an ACPI device that exposed the functionality to the OS. My goal here is to produce a standard Coreboot radio control device that other Coreboot platforms can implement, and then just write a single driver that exposes it. I wrote one for Linux that seems to work.

In summary: closed-source code is more annoying to improve, but that doesn't mean it's impossible. Also, strange Russians on forums make everything easier.

comment count unavailable comments

18 November, 2019 08:19AM

hackergotchi for Joachim Breitner

Joachim Breitner

Faster Winter 1: Vectors

(This is the first optimization presented in the “faster winter” series, please see that post for background information.)

Immutable vectors

The first very low hanging fruit was to be found in the Wasm.Runtime.Memory module. The memory of the Wasm module was defined as follows:

import Data.Vector (Vector)

data MemoryInst m = MemoryInst
  { _miContent :: Mutable m (Vector Word8)
  , _miMax :: Maybe Size

An immutable vector in a mutable box (Mutable m is something like IORef or STRef, whatever is appropriate for the monad m).

This means that every write to the memory has to copy the whole memory into a new vector. This is nice from a semantic point of view (e.g. you can take a snapshot of the memory at a specific point in time, evaluate the code some more, and the immutable vector is untouched), but obviously very inefficient.

Mutable vectors

So I changed it to a mutable vector (MVector) that allows in-place updates:

import Data.Vector (MVector)

data MemoryInst m = MemoryInst
  { _miContent :: Mutable m (MVector (PrimState m) Word8)
  , _miMax :: Maybe Size

I still need to place that mutable vector in a mutable box, because an MVector still has fixed size but Wasm programs can resize their memory. This mean we still copy the whole memory when the memory grows, but not during normal writes.

Improvement: Allocations: -97.71% Memory: +16.87% Time: -98.88% (Commit af6adfc...254d700)

This is much more … usable!

Oddly, memory consumption increases,and I am not entirely sure why: My guess is that because it runs faster, the garbage collector runs less often, the program can allocate more stuff in between garbage collector runs.

Unboxed vectors

In a second step I also replaced the vector with an unboxed vector: Above we have the normal Data.Vector vector, which stores pointers (8 bytes!) to elements, even if the element is a single byte like here. By importing Data.Vector.Unboxed (no other change!) I get a nicely packed array of bytes.

Interestingly, while it improved maximum memory usage quite a little bit, it looks like it made the program actually a bit slower – potentially because it now had to box and unbox these Word8 values.

Improvement: Memory: -27.04% Time: +15.82% (4061fe6)

There may be more improvements possible here (e.g. when writing a Word32, maybe that can be done in one go, instead of writing the four bytes separately), but I did not follow this clue any further for now.

Instead, the next thing that looked like it was worth investigating was overly polymorphic code.

18 November, 2019 12:02AM by Joachim Breitner (

Faster Winter 1/7: Vectors

(This is the first optimization presented in the “faster winter” series, please see that post for background information.)

Immutable vectors

The first very low hanging fruit was to be found in the Wasm.Runtime.Memory module. The memory of the Wasm module was defined as follows:

import Data.Vector (Vector)

data MemoryInst m = MemoryInst
  { _miContent :: Mutable m (Vector Word8)
  , _miMax :: Maybe Size

An immutable vector in a mutable box (Mutable m is something like IORef or STRef, whatever is appropriate for the monad m).

This means that every write to the memory has to copy the whole memory into a new vector. This is nice from a semantic point of view (e.g. you can take a snapshot of the memory at a specific point in time, evaluate the code some more, and the immutable vector is untouched), but obviously very inefficient.

Mutable vectors

So I changed it to a mutable vector (MVector) that allows in-place updates:

import Data.Vector (MVector)

data MemoryInst m = MemoryInst
  { _miContent :: Mutable m (MVector (PrimState m) Word8)
  , _miMax :: Maybe Size

I still need to place that mutable vector in a mutable box, because an MVector still has fixed size but Wasm programs can resize their memory. This mean we still copy the whole memory when the memory grows, but not during normal writes.

Improvement: Allocations: -97.71% Memory: +16.87% Time: -98.88% (Commit af6adfc...254d700)

This is much more … usable!

Oddly, memory consumption increases,and I am not entirely sure why: My guess is that because it runs faster, the garbage collector runs less often, the program can allocate more stuff in between garbage collector runs.

Unboxed vectors

In a second step I also replaced the vector with an unboxed vector: Above we have the normal Data.Vector vector, which stores pointers (8 bytes!) to elements, even if the element is a single byte like here. By importing Data.Vector.Unboxed (no other change!) I get a nicely packed array of bytes.

Interestingly, while it improved maximum memory usage quite a little bit, it looks like it made the program actually a bit slower – potentially because it now had to box and unbox these Word8 values.

Improvement: Memory: -27.04% Time: +15.82% (4061fe6)

There may be more improvements possible here (e.g. when writing a Word32, maybe that can be done in one go, instead of writing the four bytes separately), but I did not follow this clue any further for now.

Instead, it was time to hunt down the space leak, as the interpreter should not need any significant amounts of memory, and instead execute the program using a constant and low amount of memory. More about that in the next entry of this series.

18 November, 2019 12:02AM by Joachim Breitner (

November 17, 2019

hackergotchi for Shirish Agarwal

Shirish Agarwal


This will be short as I have been having a touch of cold and flu over the last week. As I shared, I moved to mastodon from twitter and quite a number of people have moved on mastodon and many other instance which run mastodon. The best thing IMHO was that –
a. I came to know the wide range of projects that fediverse is attacking or working on. While I’m not going to dwelve much into fediverse in this blog post, maybe some future blog post either here or on might be used to share some of the features which I thought were far better than twitter and the possibility of people challenging the status quo, especially since if you have a bit of experience servicing a web-server. It is very LAMPisque in nature, except of course they use postgres ( much better choice and probably much more scalable) although they are still trying to fix and figure out response times and such, being a non-corporate, non-ad service, they rely on people’s donations.


I just started seeing Unbelievable which was produced and shown on netflix. While I probably need a week or two and would be staggering my viewing so that it doesn’t affect me emotionally as have been told by friends. I just viewed the first twenty minutes of the 1st episode and there itself, it seemed lot of things had gone wrong. Some of my observations –

a. What is not told is when the first call to 911 went, if it was told that the assault involved a young woman shouldn’t a lady officer have been sent to the scene to investigate. While I have heard that the cop defended the girl, perhaps it would have been better if a lady cop would have gone to the scene. She might have been a tad more empathatic and it probably would have been easier for the victim, in this case a young girl to share her experience.

b. When it was told it was rape, there were again two male detectives. Shouldn’t it be women handling this. I would have seen multiple advantages in having a woman detective here. Probably most women are able to have a good reading about other women. Probably women in law-enforcement much more so. She would also have been empathatic and usually girl/women are usually much more relaxed around other women. So for the detective it would also have helped her in the investigation for signs of ‘tells’ to know if the victim was lying.

c. No legal help for the victim – After the lead detective questions the girl again in the police station, he asks her to sign over the results of the tests of the testing from the lab as well as to write the statement again. Having an attorney there perhaps would have advised her as to what are her rights and whether she should or not either sign those documents or start writing by herself right in the police station which was stressful for her. As she does tell in the beginning of the interview, she has a headache. If such a statement was indeed needed by her, she should have been told to write in the confines of her home or where she was staying with her guardian. I know of so many times while I think of writing a blog post, there are so many ideas, when I start typing, half of them I simply have forgotten. Later, if I do even want to make a change or two, many a times the change itself would be so dramatic that I would have to re-write quite a bit of the blog post taking that additional info. which I had known but not voiced on the blog. So, when somebody like me who may miss formulations or ideas even when I’m not stressed, here is a young girl/woman who claims to have been raped/assaulted. Having had a lawyer even if she was not very sharp would probably told her of her rights and what she can or should not do in the cirumstances she is in. I probably will expand on it as I see more of the series either here itself or on other blog posts as I see more of it.

Hopefully we will meet in a more cheerful blog post 😦

17 November, 2019 08:26PM by shirishag75

hackergotchi for Wouter Verhelst

Wouter Verhelst

Announcing extrepo

Debian ships with a lot of packages. This allows our users to easily install software without too much effort -- just run apt-get install foo, and foo gets installed.

However, Debian does not ship with everything, and for that reason there sometimes are things that are not installable with just the Debian repositories. Examples include:

  • Software that is not (yet) packaged for Debian
  • A repository for software that is in Debian, but for the bleeding-edge version of that software (e.g., maintained by upstream, or maintained by the Debian packager for that software)
  • Software that is not DFSG-free, and cannot be included into Debian's non-free repository due to licensing issues, but that can freely be installed by Debian users.

In order to enable and use such repositories on a Debian system, a user currently has to perform some actions that may be insecure:

  • Some repositories provide a link to a repository and a key, and expect the user to perform all actions to enable that repository manually. This works, but mistakes are easy to make (especially for beginners), and therefore it is not ideal.
  • Some repositories provide a script to enable the repository, which must be run as root. Such scripts are not signed when run. In other words, we expect users to run untrusted code downloaded from the Internet, when everywhere else we tell people that doing so is a bad idea.
  • Some repositories provide a single .deb file that can be installed, and which enables the necessary repositories in the apt configuration. Since, in contrast to RPM files, Debian packages are not signed, this means that the configuration is not enabled securely.

While there is a tool to enable package signatures in Debian packages, the dpkg tool does not enforce the existence of such signatures, and therefore it is possible for an attacker to replace the (signed) .deb file with an unsigned variant, bypassing the whole signature.

In an effort to remedy this whole situation, I looked at creating extrepo, a package that would download repository metadata from a special-purpose repository, verify the signatures placed on that metadata, and if everything matches, enable the repository by creating the necessary apt configuration files.

This should allow users to enable external repository "foo" by running extrepo enable foo, rather than downloading a script from foo's website and executing it as root -- or other similarly insecure options.

The extrepo package has been uploaded to Debian; and so once NEW processing has finished, will be available in Debian unstable.

I might upload it to backports too if no issues are found, but that won't be an immediate thing.

17 November, 2019 09:34AM

Russ Allbery

Review: Rift in the Sky

Review: Rift in the Sky, by Julie E. Czerneda

Series: Stratification #3
Publisher: DAW
Copyright: July 2009
ISBN: 1-101-13317-1
Format: Kindle
Pages: 419

This is the third and final book of the Stratification trilogy and the bridge between it and the Trade Pact trilogy, of which it is a prequel. It's readable without reading the Trade Pact (although that series is overall better), but not very readable without the rest of the Stratification series.

Be warned that the publisher's summary of this book, found on Goodreads, Amazon, and other places you might read about it, is a spoiler for nearly all of the plot. It's not a very surprising spoiler if you've read the Trade Pact books, but still, publishers, don't do this.

Riders of the Storm left off with Aryl with a more stable situation, a new negotiated compromise with the Oud, and more information about the Om'ray, including the mysterious distant clan that no one visits. The origins of her world's balance of powers, and the goals of the outsider presence on her world, are both still opaque, but she's settled near the human Marcus and his archaeological site, so I was optimistic that we were about to learn more.

We don't. What the reader gets instead is more clan politics, worries about the new Om'ray powers that are eagerly adopted by the children of Aryl's clan, and only a few faint additional hints at the nature of the Cloisters. Aryl has more interactions with the Tiktik that finally lead to understanding more of the Agreement from the perspective of the other two races of Cersi (with rather dramatic consequences), and we learn quite a bit more about Om'ray reproduction. But there's next to nothing about how any of this strange and clearly artificial balance was created in the first place, or what the civilization that Marcus is so fascinated by truly is, or how it relates to the Om'ray, or even why the Om'ray clearly had advanced technology at one point in time that they no longer understand.

I hope the information that I wanted from this series is in the following Reunification series, since right now I'm irritated. (Although apparently not irritated enough to stop reading.)

On top of the frustrating lack of answers to nearly every question I had from the first book, this novel has a very odd structure, exacerbated by some strange decisions in how the Kindle version is configured. The last 100 pages are marked in the table of contents as "Teaser chapter" and appear after the dramatis personae. The Kindle reader even pops up the "rate this book" screen before that chapter, exactly as if the novel were over and this material were preview chapters of some subsequent book. I assumed it was a (surprisingly long) excerpt from the start of This Gulf of Time and Stars, the first book of the next trilogy.

It's not, although I admit I bought that book just to check for this review (I was going to buy it eventually anyway). That's a very good thing, since that last hundred pages was the only thing that salvaged this story for me, even though it uses my least favorite fictional trope.

The conclusion of the main story is weirdly abrupt. After lots of Aryl navigating the leadership of her clan, there's a very abrupt political shift (for magical reasons that I never found adequately explained) and the intensity of the pace picks up dramatically. Following some developments in the subplot with Marcus and some dramatic changes in the Agreement, Aryl and her people take drastic steps to resolve the situation, leading to the event that series readers will have been anticipating for the trilogy. But it's sudden and entirely unsatisfying, and if the book had actually ended there, this review would be much more negative.

Instead, the last part of the book takes us back into the science fiction setting of the previous trilogy and a reasonably entertaining cultural conflict. It felt much more like a Trade Pact story, and since my enjoyment of those stories is why I was reading this trilogy in the first place, I'm totally fine with that. Unfortunately, it's also infuriating because Czerneda forecloses on any answers to the historical questions I've been puzzling over for the whole trilogy. The details of how are spoilers, so I won't get into it except to say that I thought it was a rather cheap and unsatisfying device.

If you came this far in this trilogy, you may as well read the ending. Some pieces of it were moving, and I did enjoy the last part of the story. But this whole trilogy focused on the parts of Aryl's life that I found the least interesting and consistently avoided the questions that Marcus was asking and that I really wanted answered. I hope the following Reunification trilogy revisits this world and provides some resolution.

This is the last book of its trilogy, but there is a sequel trilogy to the earlier Trade Pact trilogy in the same universe that starts with This Gulf of Time and Stars, and it promises to tie the two trilogies together.

Rating: 5 out of 10

17 November, 2019 06:25AM

November 16, 2019

Andrew Cater

Seems like I only ever write something at CD release time ... Buster release 2 is happening round about now

Updating CD images again for Buster release 2. I've spent a happy day chasing down various CD images and discovering my ignorance once again :)

Thanks to Sledge for having me around and also to Schweer - who wasn't feeling so well. No RattusRattus or Isy this time round - that meant I got to do a little more but I do miss their company.

All looks set fair for CD, DVD images and so on: the Jigdo16GB and the BD-1 isos built from jigdo files also worked well for me. This is getting to be a slick process and it's good to be a part of it.

And another one in a few months, I suspect :)

16 November, 2019 11:28PM by Andrew Cater (

Molly de Blanc


We spend a lot of time focusing on the epic side of free software and user freedom: joys come from providing encrypted communication options to journalists and political dissidents; losses are when IoT devices are used to victimize and abuse.

I think a lot about the little ways technology interacts with our lives, the threats to or successes for user freedom we encounter in regular situations that anyone can find themselves able to understand: sexting with a secure app, sharing  DRM-free piece of media, or having your communications listened to by a “home assistant.”

When I was writing a talk about ethics and IoT, I was looking for these small examples of the threats posed by smart doorbells. False arrests and racial profiling, deals with law enforcement to monitor neighborhoods, the digital panopticon — these are big deals. I remembered something I read about kids giving their neighbor a pair of slippers for Christmas. This sort of anonymous gift giving becomes impossible when your front door is constantly being monitored. People laughed when I shared this idea with them — that we’re really losing something by giving up the opportunity to anonymously leave presents.

We are also giving up what my roommate calls “benign acts of rebellion.” From one perspective, making it harder for teenagers to sneak out at night is a good thing. Keeping better tabs on your kids and where they are is a safety issue. Being able to monitor what they do on their computer can prevent descent into objectively bad communities and behavior patterns, but it can also prevent someone from participating in the cultural coming of age narratives that help define who we are as a society and give us points of connection across generations.

People sneak out. People go places their parents don’t want them to. People stay up late at night reading or playing video games. People explore their sexuality by looking at porn when they’re underage. People do things their parents don’t want them to, and these are things their parents are increasingly able to prevent them from doing using technology.

I met someone at a conference who was talking about potentially installing a camera into the bedroom of their pubescent child — the same kind designed to allow parents to monitor their babies at night — because their child was playing video games when they “should be sleeping.”

This appalled me, but one of the things that really struck me was how casually they said it. Technology made it not a big deal. They already had one in their baby’s room, putting another in seemed simple.

I would happily argue all the epic points that come out of this: creating a surveillance state, normalizing the reality of being monitored, controlling behavior and creating a docile population. These are real threats, but also, seriously, poor sleep hygiene is just a thing teenagers do and it’s okay.

These benign acts of rebellion — staying up later than we’re told to, chatting with our friends when we’re not “supposed to” — are not just important points of cultural connection, but also just important for our own individual development. Making mistakes, doing dumb things, acting the fool, and learning from all of this is important in the process of defining ourselves. Technology should not be used to hinder our personal growth, especially when it offers to many opportunities for us to better explore who we are, or makes it safer for us to continue to rebel in the myriad ways we always have. Rebellion is important to our narratives — it’s certainly integral to mine. I hope that people younger than me don’t lose that because of the fear others hold.

16 November, 2019 07:05PM by mollydb

hackergotchi for Dirk Eddelbuettel

Dirk Eddelbuettel


A new minor release of RcppEigen arrived on CRAN today (and just went to Debian too) bringing support for Eigen 3.3.7 to R.

This release comes almost a year after the previous minor release Besides the upgrade to the new upstream version, it brings a few accumulated polishes to the some helper and setup functions, and switches to the very nice tinytest package for unit tests; see below for the full list. As before, we carry a few required changes to Eigen in a diff. And as we said before at the previous two releases:

One additional and recent change was the accomodation of a recent CRAN Policy change to not allow gcc or clang to mess with diagnostic messages. A word of caution: this may make your compilation of packages using RcppEigen very noisy so consider adding -Wno-ignored-attributes to the compiler flags added in your ~/.R/Makevars.

The complete NEWS file entry follows.

Changes in RcppEigen version (2019-11-16)

  • Fixed skeleton package creation listing RcppEigen under Imports (James Balamuta in #68 addressing #16).

  • Small RNG use update to first example in skeleton package used by package creation helper (Dirk addressing #69).

  • Update vignette example to use RcppEigen:::eigen_version() (Dirk addressing #71).

  • Correct one RcppEigen.package.skeleton() corner case (Dirk in #77 fixing #75).

  • Correct one usage case with pkgKitten (Dirk in #78).

  • The package now uses tinytest for unit tests (Dirk in #81).

  • Upgraded to Eigen 3.3.7 (Dirk in #82 fixing #80).

Courtesy of CRANberries, there is also a diffstat report for the most recent release.

If you like this or other open-source work I do, you can now sponsor me at GitHub. For the first year, GitHub will match your contributions.

This post by Dirk Eddelbuettel originated on his Thinking inside the box blog. Please report excessive re-aggregation in third-party for-profit settings.

16 November, 2019 03:41PM

Molly de Blanc

Free software activities, October 2019

A belated hello! I was traveling at the end of October and missed this. Apologies!

A beautiful, green Japanese maple tree in front of a Buddhist shrine.

In October, work was quite busy, though a lot of it was behind-the-scenes stuff I cannot yet update you on. It was also very busy with a very exciting trip I took that had absolutely nothing to do with free software. If you’re ever going to Kyoto or Tokyo and looking for some recommendations for coffee, cocktail bars, restaurants, or general things to do, hmu.

Free software activities (personal)

  • I have regular meetings with Patrick Masson, the general manager of the OSI. We made most of them in October.
  • I did some writing for the OSI. Not all of it is published at this point.
  • I worked on crafting drafts of organizational policies for the OSI, including staffing, travel, and a whistle blower policy. I hope to be able to arrange for an HR specialist or employment lawyer to review these.
  • The OSI has two new board members! In order to make this happen, I contacted all of the nominees for whom I had contact information. I spoke with them about the OSI, the Board and it’s activities, and how they saw their potential involvement. Basically I interviewed a bunch of ~fancy~ people. It was so much fun talking with every one of them and I learned so much during the process.
  • The Debian Community Team had some meetings, wrote some emails, and discussed The Future together and with the greater Debian community.
  • I attended All Things Open and spoke about ethics and IoT devices. My slides were puppy themed.
  • I did some philosophy based writing. I got a  lot out of this and hope you did too.
  • I also found out that my brother’s company does some open source work!
  • I submitted to the Open Source Festival 2020 CfP. And you can too!

Free software activities (professional)

  • I attended All Things Open and had one of the most awesome tabling experiences I have had to date! It was such a great crowd at ATO! They took all of our stickers!
  • I had a lot of meetings with some more great people. Thank you everyone who made some time for me!
  • We launched a Patent Troll Defense Fund! I cannot thank the donors enough! It’s so inspiring for me to see the community come together to support a project I really believe in.
  • We’ve been doing a lot of work on this Rothschild Imaging thing.
  • We did some fundraising for Linux Application Summit (which happened this week!).

16 November, 2019 12:32AM by mollydb

November 15, 2019

hackergotchi for Romain Perier

Romain Perier

Capitole du Libre 2019

Hi !

The Capitole Du Libre is a french event that takes place at INP-ENSEEIHT in Toulouse. It is an event dedicated to free and open source softwares. The Capitole Du Libre 2019 will happen this weekend on 16th-17th November.

There will be a Debian booth with the DebianFrance team, I will be there to help them. A lot of interesting talks will be presented, see the schedule here.

Viveris, my company, is a gold sponsor this year. During the year, some of my contributions are sponsorised by my company. Viveris also sponsorised the demo I am going to present at the Debian booth, many thanks to her !

15 November, 2019 08:19PM by Romain Perier (

hackergotchi for Rapha&#235;l Hertzog

Raphaël Hertzog

Freexian’s report about Debian Long Term Support, October 2019

A Debian LTS logo

Like each month, here comes a report about
the work of paid contributors
to Debian LTS.

Individual reports

In October, 214.50 work hours have been dispatched among 15 paid contributors. Their reports are available:

  • Abhijith PA did 8.0h (out of 14h assigned) and gave the remaining 6h back to the pool.
  • Adrian Bunk didn’t get any hours assigned as he had been carrying 26h from September, of which he gave 8h back, so thus carrying over 18h to November.
  • Ben Hutchings did 22.25h (out of 22.75h assigned), thus carrying over 0.5h to November.
  • Brian May did 10h (out of 10h assigned).
  • Chris Lamb did 18h (out of 18h assigned).
  • Emilio Pozuelo Monfort did 46.25h (out of 21.75h assigned at the beginning of the month and 24.5h assigned at the end of the month).
  • Hugo Lefeuvre did 46.5h (out of 22.75h assigned and 23.75h from September).
  • Jonas Meurer didn’t get any hours assigned and gave back the 14.5h he was carrying from September as he did nothing.
  • Markus Koschany did 22.75h (out of 22.75h assigned).
  • Mike Gabriel did 11.75h (out of 10h assigned and 1.75h from September).
  • Ola Lundqvist did 8.5h (out of 8h assigned and 14h from September), thus carrying over 13.5h to November.
  • Roberto C. Sánchez did 8h (out of 8h assigned).
  • Sylvain Beucler did 22.75h (out of 22.75h assigned).
  • Thorsten Alteholz did 22.75h (out of 22.75h assigned).
  • Utkarsh Gupta did 10.0h (out of 10h assigned).

Evolution of the situation

In October Emilio spent many hours bringing firefox-esr 68 to jessie and stretch, thus expanding the impact from Debian LTS to stable security support. For jessie firefox-esr needed these packages to be backported: llvm-toolchain, gcc-mozilla, cmake-mozilla, nasm-mozilla, nodejs-mozilla, cargo, rustc and rust-cbindgen.
October was also the month were we saw the first paid contributions from Utkarsh Gupta, who was a trainee in September.

Starting in November we also have a new trainee, Dylan Aïssi. Welcome to the team, Dylan!

We currently have 59 LTS sponsors sponsoring 212h per month. Still, as always we are welcoming new LTS sponsors!

The security tracker currently lists 35 packages with a known CVE and the dla-needed.txt file has 35 packages needing an update.

Thanks to our sponsors

New sponsors are in bold.

No comment | Liked this article? Click here. | My blog is Flattr-enabled.

15 November, 2019 02:26PM by Raphaël Hertzog

hackergotchi for Keith Packard

Keith Packard


Picolibc Version 1.1

Picolibc development is settling down at last. With the addition of a simple 'hello world' demo app, it seems like a good time to stamp the current code as 'version 1.1'.

Changes since Version 1.0

  • Semihosting helper library. Semihosting lets an application running under a debugger or emulator communicate through the debugger or emulator with the environment hosting those. It's great for platform bringup before you've got clocking and a serial driver. I'm hoping it will also make running tests under qemu possible. The code works on ARM and RISC-V systems and offers console I/O and exit() support (under qemu).

  • Hello World example. This is a stand-alone bit of code with a Makefile that demonstrates how to build a complete application for both RISC-V and ARM embedded systems using picolibc after it has been installed. The executables run under QEMU using a provided script. Here's all the source code you need; the rest of the code (including semihosting support) is provided by picolibc:

    #include <stdio.h> #include <stdlib.h>

    int main(void) { printf("hello, world\n"); exit(0); }

  • POSIX file I/O support. For systems which have open/close/read/write, picolibc's tinystdio can now provide stdio functions that use them, including fopen and fdopen.

  • Updated code from newlib. I've merged current upstream newlib into the tree. There were a few useful changes there, including libm stubs for fenv on hosts that don't provide their own.

Where To Get Bits

You can find picolibc on my personal server's git repository:

There's also a copy on github:

If you like tarballs, I also create those:

I've create tags for 1.1 (upstream) and 1.1-1 (debian packaging included) and pushed those to the git repositories.

Filing Issues, Making Contributions

There's a mailing list at

Or you can file issues using the github tracker.

15 November, 2019 06:39AM

hackergotchi for Kees Cook

Kees Cook

security things in Linux v5.3

Previously: v5.2.

Linux kernel v5.3 was released! I let this blog post get away from me, but it’s up now! :) Here are some security-related things I found interesting:

heap variable initialization
In the continuing work to remove “uninitialized” variables from the kernel, Alexander Potapenko added new init_on_alloc” and “init_on_free” boot parameters (with associated Kconfig defaults) to perform zeroing of heap memory either at allocation time (i.e. all kmalloc()s effectively become kzalloc()s), at free time (i.e. all kfree()s effectively become kzfree()s), or both. The performance impact of the former under most workloads appears to be under 1%, if it’s measurable at all. The “init_on_free” option, however, is more costly but adds the benefit of reducing the lifetime of heap contents after they have been freed (which might be useful for some use-after-free attacks or side-channel attacks). Everyone should enable CONFIG_INIT_ON_ALLOC_DEFAULT_ON=1 (or boot with “init_on_alloc=1“), and the more paranoid system builders should add CONFIG_INIT_ON_FREE_DEFAULT_ON=1 (or “init_on_free=1” at boot). As workloads are found that cause performance concerns, tweaks to the initialization coverage can be added.

pidfd_open() added
Christian Brauner has continued his pidfd work by creating the next needed syscall: pidfd_open(), which takes a pid and returns a pidfd. This is useful for cases where process creation isn’t yet using CLONE_PIDFD, and where /proc may not be mounted.

-Wimplicit-fallthrough enabled globally
Gustavo A.R. Silva landed the last handful of implicit fallthrough fixes left in the kernel, which allows for -Wimplicit-fallthrough to be globally enabled for all kernel builds. This will keep any new instances of this bad code pattern from entering the kernel again. With several hundred implicit fallthroughs identified and fixed, something like 1 in 10 were missing breaks, which is way higher than I was expecting, making this work even more well justified.

x86 CR4 & CR0 pinning
In recent exploits, one of the steps for making the attacker’s life easier is to disable CPU protections like Supervisor Mode Access (and Execute) Prevention (SMAP and SMEP) by finding a way to write to CPU control registers to disable these features. For example, CR4 controls SMAP and SMEP, where disabling those would let an attacker access and execute userspace memory from kernel code again, opening up the attack to much greater flexibility. CR0 controls Write Protect (WP), which when disabled would allow an attacker to write to read-only memory like the kernel code itself. Attacks have been using the kernel’s CR4 and CR0 writing functions to make these changes (since it’s easier to gain that level of execute control), but now the kernel will attempt to “pin” sensitive bits in CR4 and CR0 to avoid them getting disabled. This forces attacks to do more work to enact such register changes going forward. (I’d like to see KVM enforce this too, which would actually protect guest kernels from all attempts to change protected register bits.)

additional kfree() sanity checking
In order to avoid corrupted pointers doing crazy things when they’re freed (as seen in recent exploits), I added additional sanity checks to verify kmem cache membership and to make sure that objects actually belong to the kernel slab heap. As a reminder, everyone should be building with CONFIG_SLAB_FREELIST_HARDENED=1.

KASLR enabled by default on arm64
Just as Kernel Address Space Layout Randomization (KASLR) was enabled by default on x86, now KASLR has been enabled by default on arm64 too. It’s worth noting, though, that in order to benefit from this setting, the bootloader used for such arm64 systems needs to either support the UEFI RNG function or provide entropy via the “/chosen/kaslr-seed” Device Tree property.

hardware security embargo documentation
As there continues to be a long tail of hardware flaws that need to be reported to the Linux kernel community under embargo, a well-defined process has been documented. This will let vendors unfamiliar with how to handle things follow the established best practices for interacting with the Linux kernel community in a way that lets mitigations get developed before embargoes are lifted. The latest (and HTML rendered) version of this process should always be available here.

Those are the things I had on my radar. Please let me know if there are other things I should add! Linux v5.4 is almost here…

© 2019, Kees Cook. This work is licensed under a Creative Commons Attribution-ShareAlike 3.0 License.
Creative Commons License

15 November, 2019 01:36AM by kees

November 14, 2019

Uwe Kleine-König

Philips Hue Bridge v2.1

I recently bought a Hue Bridge to experiment a bit with Zigbee and 802.15.4. Following two posts for the hardware version 2.0 and some comments about the differences to version 2.1 I was able to get shell access on my 2.1 hardware.

As there is up to now no complete guide I describe here, what I did:

Opening the case is straigth forward. Just remove the two lower nubsis at the bottom and unscrew the two torx screws; then carefully unclip the bottom.


The header on the left side on the image gives access to the UART. Pin 1 is ground, pin 4 is the board's RX (so connect your adapter's TX) and pin 5 is the board's TX. The voltage level is 3.3 V. Note however that the board doesn't boot (after a cold start) with its TX connected so leave that unconnected until after the bootloader is up.

To stop the bootloader short the test point marked with a red cycle on the image near the NAND flash to ground for a few seconds when plugging in the power cord. I used pin 14 of SJ8 (the unassembled header on the left above the factory reset button). This makes the bootloader fail to read the kernel image from NAND and then drop to a shell.

Then after connecting RX you can do there:

setenv bootdelay 3
setenv security '$5$uaS2DwZOsch$2H76tTh0yLzH8LfIkg7z1icoYZLeLGV.Gpo3quhZSe2'

After that the machine boots up and you can login with username root and password lala. (If you want a different password, set security in U-Boot accordingly. I did

mkpasswd -m sha-256

to generate the necessary string.)

I already asked Philips for the GPL code running on that machine, didn't hear back from them yet.

I will keep you updated on any progress here.

14 November, 2019 11:00AM

November 13, 2019

hackergotchi for Norbert Preining

Norbert Preining

Python 3 deprecation imminent

OSS Journal, November 2026. In less than two month, with the end of the year 2026, Python 3 will be deprecated and will not obtain any further security updates. Despite the announcement of deprecation back in summer 2020, shortly after the deprecation of Python 2, still thousands of software projects, in particular in data science, seem to be still based on Python 3.

After the initially quick uptake of Python 4, announced in June 2020, most developers have switched to the clearly superior version which resolves long-standing discrepancies in the Python variable semantics. Unfortunately, Python 4 is not backward compatible with Python 3 (not to think of Python 2).

In a recent interview with the OSS Journal, the Python Head Developer stated:

The future is with Python 4 – we have worked hard to make Python 4 the best programming language out there, and we expect it to serve the community for a long future. Having announced the deprecation of Python 3 well in advance (5 years ago), we expect everyone to have updated their code by now.

The Python developer community has enthusiastically embraced Python 4, and we see no reason to prolongue the outdated Python 3 language just for a few data scientists.

The Python 3 deprecation has created a whole new branch of companies providing only Python upgrade services, but despite the abundance of these services, many programs are still available only for Python 3, some – like Calibre – even only for Python 2.

So let us use the remaining month to fix the billions of lines of code still not compatible with Python 4, for a better future! Rest assured, it will be the last incompatible Python upgrade (for now).

13 November, 2019 08:23AM by Norbert Preining

November 12, 2019

hackergotchi for Dirk Eddelbuettel

Dirk Eddelbuettel

RcppAnnoy 0.0.14

annoy image

A new minor release of RcppAnnoy is now on CRAN, following the previous 0.0.13 release in September.

RcppAnnoy is the Rcpp-based R integration of the nifty Annoy library by Erik Bernhardsson. Annoy is a small and lightweight C++ template header library for very fast approximate nearest neighbours—originally developed to drive the famous Spotify music discovery algorithm.

This release once again allows compilation on older compilers. The 0.0.13 release in September brought very efficient 512-bit AVX instruction to accelerate computations. However, this could not be compiled on older machines so we caught up once more with upstream to update to conditional code which will fall back to either 128-bit AVX or no AVX, ensuring buildability “everywhere”.

Detailed changes follow below.

Changes in version 0.0.14 (2019-11-11)

  • RcppAnnoy again synchronized with upstream to ensure builds with older compilers without AVX512 instructions (Dirk #53).

  • The cleanup script only uses /bin/sh.

Courtesy of CRANberries, there is also a diffstat report for this release.

If you like this or other open-source work I do, you can now sponsor me at GitHub. For the first year, GitHub will match your contributions.

This post by Dirk Eddelbuettel originated on his Thinking inside the box blog. Please report excessive re-aggregation in third-party for-profit settings.

12 November, 2019 12:13PM

November 11, 2019

hackergotchi for Jonathan Dowland

Jonathan Dowland

∑(No,12k,Lg,17Mif) New Order + Liam Gillick: So It Goes..

For the Manchester International Festival in 2017, New Order teamed up with visual designer Liam Gillick to conceive of a new and very visual way of presenting their material. They performed a short run of concerts with a 12-piece accompanying "synth orchestra", housed in a specially designed stage that was more than a little influenced by Kraftwerk. For the set list, New Order deviated significantly from their usual staples, pulling some deep cuts and re-arranging them all for the five-piece band plus 12-piece orchestra set up.

The stage set-up

The stage set-up

How the shows were designed, built and delivered, as well as some background on the synth orchestra performers, their rehersals, and good coverage of the visual elements of the show: these aspects were all covered pretty well in a documentary called "Decades" that aired on Sky Arts and a few other places but is not in general circulation.

And so it came as a little bit of a surprise that the band's official release marking this event is an audio only CD/digital/vinyl double (or triple) album. The booklet that accompanies the physical releases features some photos of the stage set up but doesn't really do a good enough job of capturing that aspect of the work.

So on to the music. The set list for the shows was largely static (as you might expect) and, for a fan wanting to hear something a bit different from the set you'd expect at a "regular" New Order live gig, very compelling. There's no Blue Monday here; not even Temptation, although the regular favourite Bizarre Love Triangle does sneak in, as does one of my personal favourites Your Silent Face. However, their presentation, at least without the visual element, is pretty much exactly what you'd expect from a regular gig: BLT is re-arranged along the lines of the classic Richard X remix, and YSF's arpeggio is the more simplistic one that has always been played live, relative to the record, but Bernard's melodica solo is significantly extended. You can enjoy almost the same arrangements on the recent NOMC15 live album.

BLT does end with the lovely synth orchestra motif from Vanishing Point, the next song on the set list, and the two are segued together, although not entirely seamlessly, as the motif stops for a few bars before resuming a few octaves higher.

For the less common songs, it's a great pleasure to hear, in particular, Ultraviolence, In a Lonely Place, All Day Long and Decades. For some inexplicable reason IaLP is performed a measure or two quicker than the single version, which frustrates me. I'm tempted to slow it down in a DAW and see if it sounds better. Of all the arrangements, Decades benefits the most from the synth orchestra, sounding sublime. Bernard's vocal delivery also works best on the Joy Division tracks from the set (the other ones are Disorder and Heart and Soul). He's not having one of his best days on most of the others.

To wrap up: the best way to have experienced these shows would undoubtedly have been to attend them in person, which I sadly was unable to do. Failing that, it seems very strange for the official record of what took place to be, well, a record, rather than a video, especially since there is a video in the form of the Sky Arts documentary. Having said all that, there are some lovely arrangements captured here, even if the ebb-and-flow of the set list as sequenced is unusual.

11 November, 2019 12:28PM

Russ Allbery

Review: Mary Poppins

Review: Mary Poppins, by P.L. Travers

Series: Mary Poppins #1
Illustrator: Mary Shepard
Publisher: Houghton Mifflin Harcourt
Copyright: 1934
Printing: 2014
ISBN: 0-544-57475-3
Format: Kindle
Pages: 202

I read this book as part of Mary Poppins: 80th Anniversary Collection, which includes the first four books of the series.

I have a long-standing irritation with movies that become so famous that they swallow the book on which they were based, largely because of three examples from my childhood: Bambi (the book is so much better), The Wizard of Oz (the book is... not really better, but the rest of the series certainly is), and Mrs. Frisby and the Rats of NIMH (the book is so much more). That irritation is sometimes misplaced, however. Even Disney has been known to make a mediocre book into a better movie on occasion (The Hundred and One Dalmations). When Mary Poppins came up recently (a movie I adored as a kid), I vaguely remembered having read the book long ago, couldn't remember anything about it, and wondered what side of the fence it would come down on. Since an anniversary collection of the first four books was free with Amazon Prime, it was easy to find out.

Answer: perhaps the series improves in later books, but the movie is totally different in tone from the first book, and much better.

I am surprised that I'd forgotten as much about this book as I had, even though it's been at least thirty years since I've read it, since it is extremely odd. I suspect the highly episodic structure is to blame. Mary Poppins never develops into a proper story; instead, it's a series of vignettes about the titular character, the Banks family, and other people on Cherry-Tree Lane. Some of these stories will be familiar from the movie (Uncle Albert floating up into the air because of his laughter). Some will definitely not be, such as a (very brief) trip around the world via a magical compass, a visit from a star who is Christmas shopping, or a truly bizarre birthday celebration for Mary Poppins in the zoo. Unlike the movie, there is no unifying theme of Mary Poppins fixing the Banks's family problems; quite to the contrary, she seems entirely uninterested and even oblivious to them.

This is not Julie Andrews's kind, gentle, and magically competent nurse. There aren't two separate advertisements for her job; this Mary Poppins appears after Mrs. Banks sent letters to the papers advertising for a position and blithely dismisses her request for references. She is neither kind nor gentle, although by the end of the book one gets the feeling she's brought a sort of gruff stability to the household. Like the movie character, she does take the children on adventures, but they seem almost accidental, a side effect of being around Mary Poppins and thus inadvertantly involved in her business (which she rarely, if ever, explains). It's a more intimidating and fae strangeness, only slightly explained by a chapter that reveals that all children know how to talk to the sun and the wind and the animals but forget when they turn one, except for Mary Poppins. (Ode: Intimations of Immortality has a lot of depressing philosophy to answer for.)

Perhaps the oddest difference from the movie for me is that Travers's Mary Poppins is endlessly vain. She's constantly admiring herself in shop windows or finding just the right clothes, much to the frequent boredom of the children. It's an amusing take on a child's view of adult shopping trips, but the vanity and preening feels weirdly out of place for such a magical character.

There is no change in the Banks household in this book; perhaps there is more material in the later books. (The whole series appears to be eight volumes.) When the wind changes, Mary Poppins disappears as mysteriously as she appears, not even saying goodbye, although she does leave some gifts. By that point, Jane and Michael do seem fond of her, although I'm not entirely sure why. Yes, there are adventures, but outside of them, and even during them, Mary Poppins is short, abrupt, demanding, and fond of sharp and dismissive aphorisms. Gregory Maguire proclaims his preference for the books in the foreword on the grounds that they show more glimmers of mystery and danger, and I can see that if I squint. But I mostly found her unpleasant, dictatorial, irritating, and utterly unwilling to explain anything to curious children.

On this point, I'll dare to disagree with Maguire and prefer the Disney version.

A few of the stories here were fun and entertaining in ways not provided by the movie, particularly "Miss Lark's Andrew" (the successful escape of a neighbor dog from enforced isolation and unnatural pampering) and "Christmas Shopping" (I do hope the Pleiades liked their presents!). But when I get the urge for Mary Poppins, I think I'll turn to the movie with no regrets. This is an interesting curiosity, and perhaps subsequent books add more depth (and make Mary less obnoxious), but I don't think it's worth seeking out.

Followed by Mary Poppins Comes Back.

Rating: 6 out of 10

11 November, 2019 04:04AM

November 10, 2019

hackergotchi for Keith Packard

Keith Packard


Picolibc Hello World Example

It's hard to get started building applications for embedded RISC-V and ARM systems. You need to at least:

  1. Find and install the toolchain

  2. Install a C library

  3. Configure the compiler for the right processor

  4. Configure the compiler to select the right headers and libraries

  5. Figure out the memory map for the target device

  6. Configure the linker to place objects in the right addresses

I've added a simple 'hello-world' example to picolibc that shows how to build something that runs under qemu so that people can test the toolchain and C library and see what values will be needed from their hardware design.

The Source Code

Getting text output from the application is a huge step in embedded system development. This example uses the “semihosting” support built-in to picolibc to simplify that process. It also explicitly calls exit so that qemu will stop when the demo has finished.

#include <stdio.h>
#include <stdlib.h>

    printf("hello, world\n");

The Command Line

The hello-world documentation takes the user through the steps of building the compiler command line, first using the picolibc.specs file to specify header and library paths:

gcc --specs=picolibc.specs

Next adding the semihosting library with the --semihost option (this is an option defined in picolibc.specs which places -lsemihost after -lc):

gcc --specs=picolibc.specs --semihost

Now we specify the target processor (switching to the target compiler here as these options are target-specific):

riscv64-unknown-elf-gcc --specs=picolibc.specs --semihost -march=rv32imac -mabi=ilp32


arm-none-eabi-gcc --specs=picolibc.specs --semihost -mcpu=cortex-m3

The next step specifies the memory layout for our emulated hardware, either the 'spike' emulation for RISC-V:

riscv64-unknown-elf-gcc --specs=picolibc.specs --semihost -march=rv32imac -mabi=ilp32 -Thello-world-riscv.ld

with hello-world-riscv.ld containing:

__flash = 0x80000000;
__flash_size = 0x00080000;
__ram = 0x80080000;
__ram_size = 0x40000;
__stack_size = 1k;
INCLUDE picolibc.ld

or the mps2-an385 for ARM:

arm-none-eabi-gcc --specs=picolibc.specs --semihost -mcpu=cortex-m3 -Thello-world-arm.ld

with hello-world-arm.ld containing:

__flash =      0x00000000;
__flash_size = 0x00004000;
__ram =        0x20000000;
__ram_size   = 0x00010000;
__stack_size = 1k;
INCLUDE picolibc.ld

Finally, we add the source file name and target elf output:

riscv64-unknown-elf-gcc --specs=picolibc.specs --semihost
-march=rv32imac -mabi=ilp32 -Thello-world-riscv.ld -o
hello-world-riscv.elf hello-world.c

arm-none-eabi-gcc --specs=picolibc.specs --semihost
-mcpu=cortex-m3 -Thello-world-arm.ld -o hello-world-arm.elf


Picolibc tries to make things a bit simpler by offering built-in compiler and linker scripts along with default startup code to try and make building your first embedded application easier.

10 November, 2019 10:54PM

hackergotchi for Markus Koschany

Markus Koschany

My Free Software Activities in October 2019

Welcome to Here is my monthly report that covers what I have been doing for Debian. If you’re interested in Java, Games and LTS topics, this might be interesting for you.

Debian Games

  • Python 3 ports: I reviewed and sponsored krank and solarwolf for Reiner Herrmann. Thanks to his diligent work two more Python games were ported to Python 3. He also packaged a new upstream release of hyperrogue and improved the build system. Less memory is required to build hyperrogue now and some buildd are thankful for that.
  • The bullet transition got finally approved and completed successfully.
  • I uploaded a new version of pygame-sdl2 to experimental which supports Python 3 now. However the library is still exclusively needed for renpy but upstream hasn’t finished the porting work to Python 3 yet. Hopefully this will be done next year. That means the new version of renpy which I also packaged this month still depends on Python 2.
  • I fixed two bugs in Freeciv, the famous strategy game, by replacing fonts-noto-cjk with fonts-unfonts-core. (#934588) The latter fonts looks apparently better on ordinary screens. The second one was simple to fix, I just had to remove an unneeded Python 2 build-dependency. (#936553)
  • The strategy game asc, a neat clone of Battle Isle 2, also needed some attention this month. I had to replace libwxgtk3.0-dev with libwxgtk3.0-gtk3-dev. (#943439)
  • I did a QA upload of open-invaders because the maintainer email address was bouncing. The game needs a new maintainer.

Debian Java


  • I packaged a new version of privacybadger, and backported ublock-origin  to Stretch and Buster because the addon was incompatible with the latest Firefox ESR release.

Debian LTS

This was my 44. month as a paid contributor and I have been paid to work 22,75 hours on Debian LTS, a project started by Raphaël Hertzog. In that time I did the following:

  • From 14.10.2019 until 20.10.2019 and from 28.10.2019 until 03.11.2019 I was in charge of our LTS frontdesk. I investigated and triaged CVE in wordpress, ncurses, opencv, pillow, poppler, golang, gdal, lz4, python-reportlab, ruby-haml, vips, rdesktop, modsecurity-crs, zabbix, polarssl and tika.
  • DLA-1960-1. Issued a security update for wordpress fixing 7 CVE.
  • DLA-1966-1. Issued a security update for aspell fixing 1 CVE.
  • DLA-1973-1. Issued a security update for libxslt fixing 1 CVE.
  • DLA-1978-1. Issued a security update for python-ecdsa fixing 2 CVE.
  • DLA-1982-1. Issued a security update for openafs fixing 2 CVE.
  • I triaged 17 CVE in libgig and forwarded the result upstream. After the investigation I decided to mark these issues as no-dsa because all in all the security risk was low. (#931309)


Extended Long Term Support (ELTS) is a project led by Freexian to further extend the lifetime of Debian releases. It is not an official Debian project but all Debian users benefit from it without cost. The current ELTS release is Debian 7 “Wheezy”. This was my seventeenth month and I have been assigned to work 15 hours on ELTS plus five hours from September. I used 8 of them for the following:

  • ELA-185-1. Issued a security update for libxslt fixing 1 CVE.
  • ELA-186-1. Issued a security update for libssh2 fixing 1 CVE.
  • ELA-187-1. Issued a security update for cpio fixing 1 CVE. The update was prepared by Ola Lundqvist.
  • ELA-188-1. Issued a security update for djvulibre fixing 1 CVE.
  • I worked on OpenJDK 7. I contacted upstream and asked for a new IcedTea release on which we rely for packaging new upstream releases of OpenJDK. The release is still delayed.

10 November, 2019 10:17PM by apo

Russ Allbery

Review: Happy Ever After

Review: Happy Ever After, by Paul Dolan

Publisher: Penguin
Copyright: 2019
ISBN: 0-241-28445-7
Format: Kindle
Pages: 186

Paul Dolan is a professor of behavioral science at the London School of Economics, but grew up a working-class kid in a council estate (UK public housing; the US equivalent is the projects) in London. This intentionally provocative book looks at a list of nine things that we have been taught to believe will make us happy and presents evidence against that assumption. Dolan's goal is to question social narratives of success and to advocate for a different philosophical basis for life decisions: minimizing misery (negative utilitarianism), rather than trying to maximize happiness.

Happy Ever After is an argument against rules and, specifically, against judging people by rules rather than outcomes:

There is nothing inherently good or bad in a social narrative in itself; it can only ever be judged according to the costs and benefits of adhering to it in a given context. I therefore adopt a consequentialist position in contrast to a deontological one. A consequentialist's view on theft would be that it is only ever wrong when it causes more misery than it promotes happiness, whereas a deontologist would be duty bound to argue that theft is always wrong because moral value lies in certain rules of conduct. A deontological perspective typically does not allow for the importance of context. And yet I would contend that it is morally right to steal to feed your hungry child.

This is obviously a drastically simplified explanation of a complex philosophical debate, but those of you who know my political beliefs probably see why I picked up this book.

Before I dive into the details, though, one note about accuracy. One of Dolan's most provocative claims is that marriage does not, on average, make women happy, a claim repeated in an article in The Guardian (now amended to remove the claim). This claim is not supported by the data he references in this book. It was based on a misunderstanding of the coding of results in the American Time Use Survey and has been subsequently retracted by Dolan.

This is a good caution to have in the back of your mind. Dolan, as is typical for a book of this sort, cites a lot of surveys and statistics. At least some of those citations are wrong. Many more are probably unreproducible. This is not a problem unique to Dolan; as the Vox article points out, most books are fact-checked only by the author, and even academic papers have fallen prey to the replication crisis. Hard factual data, particularly about psychology, is hard to come by.

How fatal this is for Dolan's book is a judgment for the reader. Personally, I'm dubious of most psychological studies and read books like this primarily for opportunities of insight into my own life and my own decision-making processes. Whether or not statistics say that marriage makes women happier on average, Dolan's actual point stands: there is no reason to believe that marriage will necessarily make any specific woman happier, and thus pursuit of marriage as a universal life goal is on dubious ground. The key contention of Happy Ever After, in my reading of it, is that we measure ourselves and others against universal social narratives and mete out punishment for falling short, even if there is no reason to believe that social narrative should be universal. That in turn is a cause of unnecessary misery in the world that we could avoid.

Dolan divides his material into three meta-narratives, each with three sub-narratives: reaching (composed of wealthy, successful, and educated), related (married, monogamous, and children), and responsible (altruistic, healthy, and volitional). For each, he provides some data questioning whether following that narrative truly makes us happy, and looks at ways where the narrative itself may be making us unhappy. Each chapter starts with a simple quiz that asks the reader to choose between a life (first for oneself and then for one's friend) that fulfills that narrative but makes them feel miserable frequently and a life that does not fulfill that narrative but in which they rarely feel miserable. At the end of each section, Dolan shows the results of that survey, all of which show at least some support (surprising to me) for choosing the narrative despite the cost of being miserable.

Some of these chapters I found unsurprising. I'm unmarried and don't intend to have children, so the chapters on marriage and children struck me as relatively obvious. Similarly, the lack of positive happiness benefit of wealth beyond a rather modest level is well-known, although I thought Dolan failed to engage sufficiently with the risk of misery from being poor. A significant motivation for pursuing modest wealth for many people is to acquire a form of self-insurance against financial disasters, particularly in the US with our appalling lack of a safety net.

I had the most mental arguments with Dolan over education. Apparently (and not very surprisingly) this is the social narrative that I buy into the most strongly. But Dolan makes good points about how pushing a working-class kid to go to a middle-class or upper-class university can sever them from their friendship ties and emotional support network and force a really miserable adjustment, and it's not clear that the concrete benefits of education in their life are worth that. This would be even clearer if we hadn't started using college degree attainment as a credentialing system for many jobs that are not reliant on specialized education only attainable in college. (I'm looking at nearly the entire field of computing, for example.) Dolan goes farther than I would in arguing that no college education should be state-subsidized because it's inherently unfair for working-class people to be taxed to pay for middle-class educational structures. Still, I keep thinking back to this chapter during US political discussions about how important it is that we create some economic path for every US child to attend college. Is that really the correct public education policy? (See also Hofstadter's point in Anti-Intellectualism in American Life that some of the US obsession with college education is because, by comparison to Germany, our high-school and middle-school education is slow, relaxed, unchallenging, and insufficient.)

Altruistic and volitional may require a bit of additional explanation. Dolan's point with altruism is that we value a social narrative of giving for its own sake, without ego or reward. (I personally would trace this to Christianity; this was the interpretation of Matthew 6:6 that I was taught.) He argues that letting people show off their good deeds encourages more good deeds and helps others increases personal happiness. People who are more self-oriented in their motivations for volunteering stick with volunteer projects for longer. I thought of free software here, where self-interested reasons for volunteering are commonplace and accepted (scratching your own itch) rather than socially shunned, and considered part of the healthy texture of the community.

The chapter on volition recaps some of the evidence (which I've also seen in other books) that less of our life and our decisions stem from individual choice than we would like to think, and that some of our perception of free will is probably a cognitive illusion. Dolan isn't too interested in trying to undermine the reader's own sense of free will, but does want to undermine our belief in the free will of other people. His target here is the abiding political belief that other people get the life outcomes they deserve, and that poor people are poor because they're lazy or make bad choices. If we let go of the social narrative of volition and instead judge interventions solely by their results, we have fewer excuses to not collectively tackle problems and fewer justifications for negatively judging other people for their own misery.

I'm not sure I recommend this whole book. It's delightfully contrarian, but somewhat slim on new ideas (particularly if you've read broadly about happiness and life satisfaction) and heavy on studies that you should be somewhat dubious about. I'm still thinking about the chapter on education, though. How much you get out of it may depend on how many of Dolan's narratives you agree with going into the book.

Also, although I didn't discuss it in detail, mad props to Dolan for taking on the assumption that striving to be healthy is a life goal that should override happiness. We need a lot more questioning of that specific narrative.

Rating: 7 out of 10

10 November, 2019 04:58AM

Hideki Yamane

fontforge package update

I've uploaded fontforge package into experimental. It needs huge changes in debian packaging.

> $ git diff debian/1%20170731_dfsg-2 HEAD debian|wc -l
> 2565
It'll help python2 removal since it provides python3-fontforge for building font packages.

Other work: follow several pacakges to latest upstream releases, fix Multi-Arch things and lintian warnings, add salsa-ci.yml and enable CI on salsa, etc.

Next: eliminate "repo in alioth" warning, more enabling salsa-ci, then digging bug reports.

10 November, 2019 02:50AM by Hideki Yamane (

Rodrigo Siqueira

Status Update and XDC 2019, October 2019

It has been a while since my last post, but there is a simple reason for that: on August 5th, I had to move from Brazil to Canada. Why did I move? Thanks to Harry Wentland recommendation, I got an interview for a software engineering position at AMD (Markham), and I got hired to work on the display team. From now on, I suppose that I’ll be around the DRM subsystem for a long time :). Even though I’m now employed by AMD this post reflect my personal thoughts only and should not be construed to represent AMD in any way.

I have a few updates about my work with the community since I have been busy with my relocation and adaptation. However, my main updates came from XDC 2019 [1] and I want to share it here.

XDC 2019 - Montréal (Concordia University Conference)

This year I had the great luck joining XDC again. However, at this time, I was with Harry Wentland, Nicholas Kazlauskas, and Leo Li (we worked together at AMD). We put effort into learning from other people’s experiences, and we tried to know what the compositor developers wanted to see in our driver. We also used this opportunity to try to explain a little bit more about our hardware features. In particular, we had conversations about Freesync, MST, DKMS, and so forth. Thinking of that, I’ll share my view of the most exciting moments that we had.


As usual, I tried my best to understand what people want to see in VKMS soon or later. For example, from the XDC 2018, I focused on fixing some bugs but mainly in add writeback support cause it could provide a visual output (this work is almost done, see [2]). This year I collected feedback from multiple people (special thanks to Marten, Lyude, Hiler, and Harry), and from these conversations I tend to work in the following tasks:

  1. Finish the writeback feature and enable visual output;
  2. Add support for adaptive refresh rate;
  3. Add support for “dynamic connectors”, which can enable the MST test.

Additionally, Martin Peres gave a talk that he shared his views for the CI and test. In his presentation, he suggested using VKMS to validate the API, and I have to admit that I’m really excited about this idea. I hope that I can help with this.


The amdgpu drivers support a technology named Freesync [3]. In a few words, this feature allows the dynamic change of the refreshes rate, which can bring benefits for games and for power saving. Harry Wayland talked about that feature and you can see it here:

Video 1: Freesync, Adaptive Sync & VRR

After Harry’s presentation, many people asked interesting questions related to this subject, this situation caught my attention, and for this reason, I added the VRR to my roadmap in the VKMS. Roman Gilg, from KDE, was one of the developers that asked for a protocol extension in Wayland for support Freesync; additionally, compositor developers asked for mechanisms that enable them to know in advance if the experience of a specific panel will be good or not. Finally, there were some discussions about the use of Freesync for power saving and in an application that requires time-sensitive.

IGT and CI

This year I kept my tradition of asking thousands of questions to Hiler with the goal of learning more about IGT, and as usual, he was extremely kind and gentle with my questions (thanks Hiler). One of the concepts that Hiler explained to me, it is the use of podman ( for prepare IGT image, for example, after a few minutes of code pair with him I could run IGT on my machine after executing the following commands:

sudo su
podman run --privileged
podman run --privileged \
                        igt_runner -t core_auth
podman run --privileged \
                        igt_runner -t core_auth /tmp
podman run --privileged -v /tmp/results:/results \ igt_runner -t core_auth /results

We also had a chance to discuss CI with Martin Peres, and he explained his work for improving the way that the CI keep track of bugs. In particular, he introduced a fantastic tool named cibuglog, which is responsible for keep tracking of test failures and using this data for building a database. Cibuglog has many helpful filters that enable us to see test problems associated with a specific machine and bugs in the Bugzilla. The huge insights from cibuglog it is the idea of using data for helping with bug tracking. Thanks Martin, for showing us this amazing tool.


I just want to finish this post with brief updates from my work with free software, starting with kw and finish with VKMS.

Kernel Workflow (kw)

When I started to work with VKMS, I wrote a tool named kworkflow, or simply kw, for helping me with basic tasks related to Kernel development. These days kw reborn to me since I was looking for a way to automate my work with amdgpu; as a result, I implemented the following features:

  • Kernel deploy in a target machine (any machine reachable via IP);
  • Module deploy;
  • Capture .config file from a target machine;

Unfortunately, the code is not ready for merging in the main branch, I’m working on it; I think that in a couple of weeks, I can release a new version with these features. If you want to know a little bit more about kw take a look at


I was not working in VKMS due to my change of country; however, now I am reworking part of the IGT test related to writeback, and as soon as I finish it, I will try to upstream it again. I hope that I can also have the VKMS writeback merged into the drm-misc-next by the end of this month. Finally, I merged the prime supported implemented by Oleg Vasilev (huge thanks!).


[1] “First discussion in the Shayenne’s patch about the CRC problem”. URL:

[2] “Introduces writeback support”. URL:

[3] “FreeSync”. URL:

10 November, 2019 02:00AM

November 09, 2019

Russ Allbery

Python dataclasses and typing

I'm going to preach the wonders of Python dataclasses, but for reasons of interested to those who have already gone down the delightful rabbit-hole of typed Python. So let me start with a quick plug for mypy if you haven't heard about it.

(Warning: this is going to be a bit long.)

Type Checking

mypy is a static type-checker for Python. In its simplest form, instead of writing:

def hello(name):
    return f"Hello, {name}"

you write:

def hello(name: str) -> str:
    return f"Hello, {name}"

The type annotations are ignored at runtime, but the mypy command makes use of them to do static typing. So, for instance:

$ cat >
def hello(name: str) -> str:
    return f"Hello {name}"
$ mypy error: Argument 1 to "hello" has incompatible type "int"; expected "str"

If you're not already using this with your Python code, I cannot recommend it highly enough. It's somewhat tedious to add type annotations to existing code, particularly at first when you have to look up how mypy represents some of the more complicated constructs like decorators, but once you do, mypy starts finding bugs in your code like magic. And it's designed to work incrementally and tolerate untyped code, so you can start slow, and the more annotations you add, the more bugs it finds. mypy is much faster than a comprehensive test suite, so even if you would have found the bug in testing, you can iterate faster on changes. It can even be told which variables may be None and then warn you if you use them without checking for None in a context where None isn't allowed.

But mypy can only help with code that's typed, so once you get the religion, the next goal is to push typing ever deeper into complicated corners of your code.

Typed Data Structures

Python code often defaults to throwing any random collection of data into a dict. For a simple example, suppose you have a paginated list of strings (a list, an offset from the start of the list, a limit of the number of strings you want to see, and a total number of strings in the underlying data). In a lot of Python code, you'll see something like:

strings = {
    "data": ["foo", "bar", "baz"],
    "offset": 5,
    "limit": 3,
    "total": 10,

mypy is good, but it's not magical. It has no way to keep track of the fact that strings["data"] is a list of strings, but strings["offset"] is a int. Instead, it decides the type of each value is the superclass of the types it sees in the initializer (in this case, object, which provides almost no type checking).

There are two traditional solutions: an object, and a NamedTuple (the typing-enhanced version of collections.namedtuple). An object is tedious:

class Strings:
    def __init__(
        self, data: List[str], offset: int, limit: int, total: int
    ) -> None: = data
        self.offset = offset
        self.limit = limit = total

This provides perfect typing, but who wants to write all that. A NamedTuple is a little better:

Strings = NamedTuple(
    [("data", List[str]), ("offset", int), ("limit", int), ("total", int)],

but still kind of tedious and has other quirks, such as the fact that your object can now be used as a tuple, which can introduce some surprising bugs.

Enter dataclasses, which are new in Python 3.7 (although inspired by attrs, which have been around for some time). The equivalent is:

class Strings:
    data: List[str]
    offset: int
    limit: int
    total: int

So much nicer, and the same correct typing. And unlike NamedTuple, dataclasses support default values, expansion via inheritance, and are full classes so you can attach short methods and do other neat tricks. You can also optionally mark them as frozen, which provides the NamedTuple behavior of making them immutable after creation.

A Detailed Example

Using dataclasses for those random accumulations of data is already great, but today I found a way to use them for a trickier typing problem.

I work on a medium-sized (about 75 routes) Tornado web UI using Jinja2 and WTForms for templating. Returning a page to the user's browser involves lots of code that looks something like this:

self.render("template.html", service=service, owner=owner)

Under the hood, this loads that template, builds a dictionary of template variables, and tells Jinja2 to render the template with those variables. The problem is the typing: the render method has no idea what sort of data you want to pass to a given template, so it uses the dreaded **kargs: Any, so you can pass anything you want. And mypy can't look inside Jinja2 template code.

Forget to pass in owner? Exception or silent failure during template rendering depending on your Jinja2 options. Pass in the name of a service when the template was expecting a rich object? Exception or silent failure. Typo in the name of the parameter? Exception or silent failure. Better hope your test suite is thorough.

What I did today was wrap each template in a dataclass:

class Template(BaseTemplate):
    service: str
    owner: str
    template: InitVar[str] = "template.html"

Now, the code to render it looks like:

template = Template(service, owner)

and now I have type-checking of all of the template arguments and only need to ensure the dataclass definition matches the needs of the template implementation.

The magic happens in the base template class:

class BaseTemplate:
    def render(self, handler: RequestHandler) -> str:
        template_name = getattr(self, "template")
        template = handler.environment.get_template(template_name)
        namespace = handler.get_template_namespace()
        for field in fields(self):
            namespace[] = getattr(self,
        return template.render(namespace)

(My actual code is a bit different and more complicated since I move some other template setup to the render() method.)

There's some magic here to work around dataclass limitations that warrants some explanation.

I pass the Tornado handler class into the template render() method so that I have access to the template environment and the (overridden) Tornado get_template_namespace() call to get default variable settings. Passing them into the dataclass constructor would make the code less clean and is harder to implement, mostly due to limitations on attributes with default values, mentioned below.

The name of the template file should be a property of the template definition rather than something the caller needs to know, but that means it has to be given last since dataclasses require that all attributes without default values come before ones with default values. That in turn also means that the template attribute cannot be defined in BaseTemplate, even without a default value, because if a child class sets a default value, @dataclass then objects. Hence the getattr to hide from mypy the fact that I'm breaking type rules and assuming all child classes are well-behaved.

template in the child classes is marked as InitVar so that it won't be included in the fields of the dataclass and thus won't be passed down to Jinja2.

Finally, it would be nice to be able to use dataclasses.asdict() to turn the object into a dictionary for passing into Jinja2, but unfortunately asdict tries to do a deep copy of all template attributes, which causes all sorts of problems. I want to pass functions and WTForms form objects into Jinja2, which resulted in asdict throwing all sorts of obscure exceptions. Hence the two lines that walk through the fields and add a shallow copy of each field to the template namespace.

I've only converted four templates so far (this code base is littered with half-finished transitions to better ways of doing things that I try to make forward progress on when I can), but I'm already so much happier. All sorts of obscure template problems will now be caught at mypy even before needing to run the test suite.

09 November, 2019 11:02PM

hackergotchi for Dirk Eddelbuettel

Dirk Eddelbuettel

Rcpp 1.0.3: More Spit and Polish

rcpp logo

The third maintenance release 1.0.3 of Rcpp, following up on the 10th anniversary and the 1.0.0. release both pretty much exactly one year ago, arrived on CRAN yesterday. This deserves a special shoutout to Uwe Ligges who was even more proactive and helpful than usual. Rcpp is a somewhat complex package with many reverse dependencies, and both the initial check tickles one (grandfathered) NOTE, and the reverse dependencies typically invoke a few false positives too. And in both cases did he move the process along before I even got around to replying to the auto-generated emails. So just a few hours passed between my upload, and the Thanks, on its way to CRAN email—truly excellent work of the CRAN team. Windows and macOS binaries are presumably being built now. The corresponding Debian package was also uploaded as a source package, and binaries have since been built.

Just like for Rcpp 1.0.1 and Rcpp 1.0.2, we have a four month gap between releases which seems appropriate given both the changes still being made (see below) and the relative stability of Rcpp. It still takes work to release this as we run multiple extensive sets of reverse dependency checks so maybe one day we will switch to six month cycle. For now, four months seem like a good pace.

Rcpp has become the most popular way of enhancing R with C or C++ code. As of today, 1832 packages on CRAN depend on Rcpp for making analytical code go faster and further, along with 190 in BioConductor. And per the (partial) logs of CRAN downloads, we are currently running at 1.1 millions downloads per month.

This release features a number of different pull requests by five different contributors as detailed below.

Changes in Rcpp version 1.0.3 (2019-11-08)

  • Changes in Rcpp API:

    • Compilation can be sped up by skipping Modules headers via a toggle RCPP_NO_MODULES (Kevin in #995 for #993).

    • Compilation can be sped up by toggling RCPP_NO_RTTI which implies RCPP_NO_MODULES (Dirk in #998 fixing #997).

    • XPtr tags are now preserved in as<> (Stephen Wade in #1003 fixing #986, plus Dirk in #1012).

    • A few more temporary allocations are now protected from garbage collection (Romain Francois in #1010, and Dirk in #1011).

  • Changes in Rcpp Modules:

    • Improved initialization via explicit Rcpp:: prefix (Riccardo Porreca in #980).
  • Changes in Rcpp Deployment:

    • A unit test for Rcpp Class exposure was updated to not fail under r-devel (Dirk in #1008 fixing #1006).
  • Changes in Rcpp Documentation:

    • The Rcpp-modules vignette received a major review and edit (Riccardo Porreca in #982).

    • Minor whitespace alignments and edits were made in three vignettes following the new pinp release (Dirk).

    • New badges for DOI and CRAN and BioConductor reverse dependencies have been added to (Dirk).

    • Vignettes are now included pre-made (Dirk in #1005 addressing #1004)).

    • The Rcpp FAQ has two new entries on 'no modules / no rtti' and exceptions across shared libraries (Dirk in #1009).

Thanks to CRANberries, you can also look at a diff to the previous release. Questions, comments etc should go to the rcpp-devel mailing list off the R-Forge page. Bugs reports are welcome at the GitHub issue tracker as well (where one can also search among open or closed issues); questions are also welcome under rcpp tag at StackOverflow which also allows searching among the (currently) 2255 previous questions.

If you like this or other open-source work I do, you can now sponsor me at GitHub. For the first year, GitHub will match your contributions.

This post by Dirk Eddelbuettel originated on his Thinking inside the box blog. Please report excessive re-aggregation in third-party for-profit settings.

09 November, 2019 02:06PM

Utkarsh Gupta

Debian Activities for October 2019

Here’s my (first) monthly update about the activities I’ve done in Debian this October.

Debian LTS

This was my first month as a Debian LTS paid contributor.
I was assigned 10 hours and worked on the following things:

CVE Fixes and Announcements:

  • Issued DLA 1948-1, fixing CVE-2019-13574, for ruby-mini-magick.
    Details here:

    In lib/mini_magick/image.rb in ruby-mini-magick, a fetched remote image filename could cause remote command execution because input is directly passed to Kernel#open, which accepts a pipe character followed by a command.

    For Debian 8 “Jessie”, this has been fixed in 3.8.1-1+deb8u1.

  • Issued DLA 1961-1, fixing CVE-2019-14464, CVE-2019-14496, and CVE-2019-14497, for milkytracker.
    Details here:

    XMFile::read in XMFile.cpp in milkyplay in MilkyTracker had a heap-based buffer overflow.
    LoaderXM::load in LoaderXM.cpp in milkyplay in MilkyTracker had a stack-based buffer overflow.
    ModuleEditor::convertInstrument in tracker/ModuleEditor.cpp in MilkyTracker had a heap-based buffer overflow.

    For Debian 8 “Jessie”, this has been fixed in 0.90.85+dfsg-2.2+deb8u1.
    Furthermore, sent a patch to the maintainer, James, for fixing the same in Bullseye, Sid. Commit here. Fixed in 1.02.00+dfsg-2.

  • Issued DLA 1962-1, fixing CVE-2017-18638, for graphite-web.
    Details here:

    The “send_email” function in graphite-web/webapp/graphite/composer/ in Graphite is vulnerable to SSRF. The vulnerable SSRF endpoint can be used by an attacker to have the Graphite web server request any resource. The response to this SSRF request is encoded into an image file and then sent to an e-mail address that can be supplied by the attacker. Thus, an attacker can exfiltrate any information.

For Debian 8 “Jessie”, this has been fixed in 0.9.12+debian-6+deb8u1.
Furthermore, sent a patch to the maintainer, Zigo, for fixing the same in Bullseye, Sid. Commit here. Fixed in 1.1.4-5.
Also, sent a patch to the Security Team for fixing the same in Buster, but uploaded by Zigo himself. Commit here. Fixed in 1.1.4-3+deb10u1.


  • Actually fix CVE-2019-11027 upstream for ruby-openid. Pull request here.
    Whilst this has been merged and released as v2.9.2, there are other login problems, as reported here.

  • Discuss with LTS Team Members about best practices for CVE-2019-11027 for ruby-openid’s actual fix. Thread here.

  • Triage Ansible for CVE-2019-14846 (which seems to be an easy fix) and CVE-2019-14858 (this kinda looks unaffected for Jessie, but not sure yet).

Debian Uploads

New Upstream Version:

  • ruby-fog-aws ~ 3.5.2-1.
  • librole-tiny-perl ~ 2.001001-1.
  • gitlab ~ 12.1.14-1 (to experimental).
  • libmail-box-perl ~ 3.008-1.
  • ruby-invisible-captcha ~ 0.12.2-1.
  • ruby-gnome ~ 3.4.0-1.
  • gitlab-shell ~ 9.3.0+dfsg-1 (to experimental).
  • gitlab-workhorse ~ 8.8.1+debian-1 (to experimental).
  • gitaly ~ 1.59.3+dfsg-1.
  • python-marshmallow-sqlalchemy ~ 0.19.0-1.
  • gitlab ~ 12.2.9-1 (to experimental).

Bug Fixes:

  • #942125 for ruby-invisible-captcha.
  • #941795 for ruby-gnome.
  • #940352 for golang-github-davecgh-go-spew.
  • #942456 for python-flask-marshmallow.
  • Autopkgtest failure for python-flask-marshmallow.
  • CVE-2019-{18446 to 18463} for gitlab.

Reviews and Sponsored Uploads:

  • node-d3-geo ~ 1.11.6-1 for Abhijith Sheheer.
  • d3-format ~ 1:1.4.1-2 for Samyak Jain.
  • Reviewed node-regex-cache for Abhijith Sheheer.
  • Reviewed node-ansi-align for Abhijith Sheheer.
  • Reviewed node-color-name for Priyanka Saggu.
  • Reviewed node-webpack for Priyanka Saggu.

Fasttrack Repo (

  • Uploaded and ACCEPTED gitlab.
  • Uploaded and ACCEPTED ruby-jwt for Nilesh Patra.
  • Uploaded and ACCEPTED ruby-gitlab-sidekiq-fetcher.
  • Uploaded and ACCEPTED ruby-fog-aws for Samyak Jain.

Until next time.
:wq for today.

09 November, 2019 12:02AM

November 07, 2019

hackergotchi for Shirish Agarwal

Shirish Agarwal

Move from twitter to mastodon

After last 2-3 nights of writing that blog post, was thinking that would be taking nice and easy for a while. But unfortunately, events have a habit overtaking the best of plans. Hopefully, it won’t be too long. It all started innocently enough. A gentleman, a Senior advocate who advocates in the Supreme Court, Dr. Sanjay Hegde had the following cover on his twitter account/handle.

August Landmasser – source – twitter, wikipedia,

Now, the photograph is about a gentleman called August Landmesser, a german national who according to Wikipedia was imprisoned, eventually drafted into penal military service and eventually killed in action according to Wikipedia . This erupted as a row in twitter as the gentleman while known for his anti-establishment views has been in all aspects a gentleman on twitter. His twitter account was suspended under the view of ‘hateful imagery’ . While one could argue that it was done right, but he was not only the only one, over the last several days, lot of people on the left-side of the spectrum, sane voices have been suspended while some twitters even after giving rape or death threats on twitter from the right, no action has been taken.

So two things happened, while Advocate Hegde was reinstated over the hue and cry, he put the cover back up and was again suspended and now has served a notice to Twitter Inc. where the senior counsel is being represented by Mr. Panjal Kishore. While I don’t want to get into the legal notice itself, I would say it makes for some pretty interesting reading and makes some very valid points. The poem of poet Gorakh Pandey in its english translation provides icing on the cake. The counsel representing Dr. Hegde also points to constitutional law and previous judgements as well as references Alexander Meiklejohn and some of the statements he made in his work ‘Political Freedom’ . The notice also reminds about Article 19 (1) (a) which ensures each person the right to free speech while restraining the Govt. The gentleman also goes on to talk about censorship and its practise and asks the courts to direct twitter Inc. to unblock him while at the same time issue some guidelines which follow both in spirit and form what Article 19 was all about.

While one could argue about the merits of the case either way, it at least to my knowledge is one of the very rare detailed notice against a social media company talking about social liberties in such a manner. I, unreservadly support the notice because the notice is about democracy and ensuring voices, even of those with whom you might agree or disagree. How serious does the Supreme Court take this and what sort of action or inaction will the Govt. take will tell where the Government is heading. There are some people who feel the judiciary is under influence of the current regime but there are still others like Dr. Hegde who still seek to challenge it. If anything, have to give him points for fighting the good fight.

Movement from Twitter to mastodon

Because of the hue and cry, quite a bit of people have started moving from twitter to mastodon. People, by default don’t like toxic situations. People enjoy constructive criticism and different ways of looking at things but don’t enjoy death threats etc. from anonymous handles which had been happening. So what started with a small wave is steadily turning into a stream, people are moving from twitter to mastadon. While I’m sort of have been on mastadon for quite sometime, it was nice to see an influx of lot of reporters, journalists, constitutional lawyers, techies as well as well as just social media junkies turning to and other masto related sites.

The altnews founder Pratik Sinha made a thread on twitter where he asked people to ask questions. While, many of the questions were easy, many of them were also thought-provoking. For instance, the idea that couldn’t a federated system be gamed. While I don’t see how, I have enough faith in human ingenuinity to concede that any system can be gamed. We know that all and any software will have bugs and security issues. Add to that lot of such new media may use either derivatives or patches to library or libraries which have not been tested enough on security or tested in variety of combinations. But this is not a new issue but an age-old issue. I did share mastadon issue tracker . There were lot of conspirancy theories, some real, some fanciful and some portending how the rich and powerful could take over. All of which are possible, I had to admit. I did enjoy sharing which helps people in choosing instances. One of the big changes which I wasn’t aware of but became aware of due to somebody named Neha was that now it’s possible to export identities, followers, blocks from one instance to another. This at least, IMHO is a big boost to the features and ability in improvement in the federated structures. Of course, the more functionality we have, the more we want 😉 . Somebody also shared a cross-posting utility but seems to only work with mastodon and not any of it derivatives, at least not the network I’m on atm. I also came to know of another mastodon instance where a single toot can be of 1111 letters. For people asking about the cons, the biggest might be security. For instance, if I were to use the cross-posting uility and somehow it is breached, then possibility of two accounts is possible, not just one. Something to consider. Still though, all in all, a very productive evening 🙂

Update 10/11/19 – The exodus which was a sliver is has turned into a stream. First BBC reported it, then bloombergquint, News18, even the chinese picked it up as well as theprint, mumbai mirror which is part of the times group as well as moneycontrol and outlookbusiness . If you read most of the mainstream press you will see that most of them gave a sort of left-handed compliment which is not surprising. Many of them don’t understand the medium and almost all have a vested interest as they are present on twitter and their business-models is based on twitter. So, perhaps to expect them to give a kind word would perhaps be expecting a bit too much.

Some quotations or interesting comments from users and the creator

There were also some interesting comments from users as to why they moved to mastodon and they were varied a bit. For e.g.

As a media researcher, I find it amazing how new digital platforms have been shaped by a crisis of credibility. Birdsite really exploded after the Radia Tapes and its subsequent blackout by mainstream media. A crisis in the credibility of Twitter foced us to switch to Mastodon. This is important in itself because a lot of people thought that Twitter and Facebook have become the final pillers of social media and are no longer replaceable.

Garvit Garg

Let’s get something clear…The exodus from Twitter is not because of Trolls. Those were easily handled. The problem was that Twitter was being paid by the Govt to turn a blind eye to the radicalisation of the Majority community.The exodus is not in fear but in disgust.

Comrade Nambiar

And finally from the creator of the software Mastodon who had to figure out how to deal with all the new traffic which was coming in –

It’s now up to 18.8K new users this week. For reference, growth was about 1K per week for a long time.


There are of course many other aspects of fediverse and mastodon that we may explore as time passes by. Till later.

07 November, 2019 10:53PM by shirishag75

A tale of unfortunate coincidences and incidents

This would be a biggie so please relax as this is going to be a really long one. So make sure you have something to chill out with.

People talk about ‘ease of doing business’ or what not for quite sometime. From last few months, there has been a slow-down and due to certain issues I had to make some purchases. This blog post will share the travails from a consumer perspective only as the horrors from the other side are also well-known But for this blog post, will share the series of unfortunate coincidences and incidents which happened with me.


For quite few months, my old UPS had been giving issues hence I decided to see what is available in the market. As my requirements are small, two desktops and a laptop sometimes, I settled on APC BR1000G-IN . Instead of buying it from Amazon I decided to try my luck at the local vendors. None of them had this specific UPS which I wanted. Before I continue I want to share the trivia that I had known that Schiedner Electric was buying APC the last time I bought an APC UPS. That was the big news at the time, 2007. At that time when I bought the APC UPS I was given 5 years warranty and the UPS worked till 7-8 years so was pretty happy with it. I also had a local brand I used which worked but didn’t offer anything special such as LED interface on the UPS.

Anyways, with those factors in mind, I went to the APC site and saw the partner list and saw that they were something like 20-25 odd partners in my city, Pune. I sent one or two e-mails to each of the partners of APC, while some were generic with my requirements while in others I was more specific in what I want. I was hoping that at least a few would respond. To my horror, most e-mails bounced back and some were in black-hole which means no one answered. I even tried calling some of the numbers given of the partners and even they turned out to be either fake or not working, which of the two it is I don’t know.

Luckily, I have been to Mumbai quite often and have few contacts with some people who are into IT sales. One of them, had numbers of some of the people in Mumbai, one of which worked, who in turn directed or shared number of a vendor in Pune. One thing led to another and soon I had BR1000G-IN at my place. Sadly, the whole process was draining and took almost a week to get it at my end. I read the user guide fully, did all the connections and found that the the start/reset button of the UPS is depressed, it doesn’t connect well with my stubby fingers. I asked mother and even she was not able to push the button.

APC Back-UPS Pro 1000 Copyright – APC

As can be seen while it is at the upper corner when trying to on it and do the required settings, it just wouldn’t work. I was not sure whether it was the UPS at fault or my stubby finger at fault. As shared, even my mom was not able to push it. After trying for a day or two, turned to the vendor, had to escalate it up and finally was assigned an engineer who came to check it. When you buy a unit for around INR 10k/- this is the least they can do. Somehow his finger was able to penetrate. For both mum and me, the start/reset button was a design fail. While I can understand it why the original design might have been so, I am sure lot of people like me would have problems with it. Coincidentally, the engineer was from one of the vendors whom I had e-mailed earlier. I showed him the e-mail I had sent to the firm. Sadly, till date the e-mail address hasn’t been corrected. is still a blackhole while I was told this will be corrected soon but two months and it has still not been corrected 😦

Sadly, my problems still continue somewhat. While I’m thankful to whoever the apcupsd Debian wiki page for some reason it doesn’t work for me. While I have asked the good people at apcupsd-users mailing list I am hopeful will get an answer in a day or two. The good part is that the cable at least is working and giving some status information as shared. Hopefully, it will be some minor thing rather than something major but that only time will tell.

Another bit of trivia – While some of you may have known, some might not, I was also looking if APC bought out the Li-Ion batteries too. As fate would have it, the date I bought the UPS, the same date the Li-Ion batteries were launched or shown on youtube.

While it probably will take a few years, would be looking forward to it. There is also possibility of supercapacitors as well but that is well into the future.

Cooler Master G650M

I remember writing about having a better SMPS about a year back. At the time I was having power problems I also thought it would be prudent to change the SMPS as well. While it didn’t take me much time, while I was looking for 700-750W SMPS I finally had to settle for Cooler Master G650M Bronze SMPS as that was the one which was available. There were others too, 1000W ones (gold) but out of my price range. Even the 650W SMPS costed me around INR 7.5k. This also worked for few months and then shorted out. I was highly surprised when the SMPS conked out, as the warranty is supposed to run for five whole years. While buying I had checked the labelling and it was packed only couple of months before purchase, so not that old. What is most peculiar is that now that product is no longer in the market and in fact had been replaced by Cooler Master MWE Bronze 650 which has 3 year warranty 😦 . Why is it so is beyond me. Usually Products which have 5 year warranty or more are usually in the market for a much longer time. Unlike other brands, Cooler Master doesn’t believe in repair but offer replacement but takes anywhere between 2 to 3 weeks which I didn’t knew at the time of purchase 😦

Just to be clear, I wasn’t sure what was wrong. I had bought the ASUS Prime Z270-P motherboard which has LED lighting all around it. I have blogged about it before last year in the same blog post above. What was peculiar that the stock fan above the CPU was not running and nor the cabinet power button, although rest of the motherboard it was showing power so it was peculiar as to what could the problem might be. I have an old voltage detector, something like this from which I could ascertain that I was getting power at points but still couldn’t figure out what was wrong. I did have the old stock SMPS but as have shared before it has lot lesser wattage, says 400 on the label but probably more like 300-325 watts. I removed few of the components from the system before taking it the vendor so it would make it easier for the vendor to tell what is wrong. I assumed that it most probably might be the switch as I usually use reboot all the time whenever I feel the need for reboot, usually after I have some updates and need to refresh my session. The vendor was able to diagnoze within few minutes that the fault lay in the SMPS and not in the switch or anywhere else and asked me to take the unit to the service center for RMA.

While I sent it for RMA, thought could survive for the required time without any internet. But I was wrong. As news in most news channels in India is so stale and biased I found it unbearable to be without news in 2-3 days. I again wondered how people in Kashmir would be without all the facilities that we have.

GRUB 2 missing, UEFI and supergrub2

I went back to the vendor with my old stock SMPS and it worked but found that grub2 menu was missing. It was just plain booting to windows 10. I started a thread at debian-user trying to figure out if there was some issue at my end, maybe some grub variable had got lost or something but the responses seemed to suggest that something else had happened. I also read through some of the UEFI documentation on wikipedia and web, I didn’t go to much depth as that would have been distracting as the specification itself is evolving and is subject to change. I did find some interesting bits and pieces but that is for a later date perhaps. One of the things I remembered from my previous run-ins with grub2 issues is that supergrub2 had been immensely useful. Sadly though, the version which I tried as stable was dumping me to grub rescue instead of the grub menu when I used the ISO image on a thumb drive. I could have tried to make a go for it but was too lazy. On an off-chance I looked at supergrub2 support and did find that somebody else also have had the same exact issue and it was reported. I chimed in and tried one of the beta versions and it worked which made me breathe easier. After getting into debian, I tried the old $ sudo update-grub which usually fixed the issues. I again tried to boot without using the help of the usb disk but failed as it again booted me into MS-Windows environment.

Firmware update

I dunno how or where or how I thought it might be a firmware issue. While trawling via the web I had come across issues which stated similar issues especially either with dual-booting or multi-booting and firmware was one of the issues which was found. Apart from waiting 2 weeks and then perhaps getting a hdd I had no other option than to update the firmware.

Using inxi I was able to get firmware details which I also shared in the github issue tracker page before update.

$ inxi -M Machine: Type: Desktop Mobo: ASUSTeK model: PRIME Z270-P v: Rev X.0x serial: UEFI: American Megatrends v: 0808 date: 06/16/2017

I would ask you to look at the version number and the date. I used Asus’s EZ update utility and downloaded the new UEFI BIOS .pcap file . In EZ update, I just had to give the correct path and couple of restarts later I had the new version of UEFI BIOS as can be seen below.

Asus UEFI BIOS 1205 Copyright – Asus

One thing to note is that there is no unix way that I least I know of updating an UEFI BIOS. If anybody knows, please let me know. I did look for ‘unix .pcacp update’ but most of the tutorials I found out were for network packet sniffing rather than UEFI BIOS update. Maybe it’s an Asus issue. Does anybody know or can point to something ?

The update fixed the EFI code which was also not appearing, as can be seen now via efibootmgr.

$ efibootmgr
BootCurrent: 0004
Timeout: 1 seconds
BootOrder: 0004,0000,0003,0001,0002,0005
Boot0000* Windows Boot Manager
Boot0001* UEFI:CD/DVD Drive
Boot0002* UEFI:Removable DeviceBoot
0003* Hard Drive
Boot0004* debian
Boot0005* UEFI:Network Device

While I’m not going to go into more details but this should be enough –

$ efibootmgr -v | grep debianBoot0004* debian HD(2,GPT,xxxxx)/File(\EFI\DEBIAN\GRUBX64.EFI)..BO

I have hidden some details for privacy sake such as address space as well as GPT hash etc. Finally, Grub 2 menu came to me in all its loveliness –

Grub 2.04-3 debian-edu CC-SA-3.0

There are still some things I wanna fix through, for instance I hope to help out adrian in testing some of his code. I wasn’t able to do because nowadays we get cheap multi-level cell, see for e.g. this paper. I might have mentioned before.

Ending Notes – Powershell

To end I did try to make home even in MS-Windows but the usefulness of shell far outlives anything that is on MS-Windows. I used powershell 5 and even downloaded and installed powershell 6 and even manage to figure out how to get quite some of the utilities to behave similar to how they behave under GNU/Linux but still the Windowing environment itself was much of an irritant than anything else. One of the biggest letdowns was not being able to use touch. While somebody made a powershell module for it but it still needs to be imported for every session. While I’m sure I could have written a small script for the same, my time was beneficial in finding the solution to it. As shared I also learnt about UEFI a bit in the process. Sharing screenshots of powershell 5 and 6 .

Powershell 5 Copyright – Microsoft Corporation
Powershell-6 Copyright – Microsoft Corporation

Conclusion – While it doesn’t even cover probably even a quarter of the issues or use-cases but even if one person finds it useful, that is good enough. I have taken lot of shortcuts and not shared whole lot otherwise this would have been lot longer. One of the things I forgot to mention is that I did find some mentions of MS-Windows overwriting, this was in October 2018 as well as October 2019 Security Updates as well. How much to trust the issues that people posted don’t really know.

07 November, 2019 08:40AM by shirishag75

November 06, 2019

hackergotchi for Gunnar Wolf

Gunnar Wolf

Made with Creative Commons ⇒ Hecho con Creative Commons. To the printer!

I am very happy to tell you that, around 2.5 years after starting the translation project, today we sent to the presses the Spanish translation for Creative Commons' book, Made with Creative Commons!

This has been quite a feat, on many fronts — Social, technical, organizational. Of course, the book is freely redistributable, and you can get it at IIEc-UNAM's repository.

As we are producing this book from DocBook sources, we will also be publishing an EPUB version. Only... We need to clear some processes first (i.e. having the right department approve it, get a matching ISBN record, etc.) and will probably only be done by early next year. Of course, you can clone our git repository and build it at home :-]

Of course, I cannot celebrate until the boxes of brand new books land in my greedy hands... But it will happen soon™.

06 November, 2019 11:09PM by gwolf

Reproducible Builds

Reproducible Builds in October 2019

Welcome to the October 2019 report from the Reproducible Builds project. 👌

In our monthly reports we attempt outline the most important things that we have been up to recently. As a reminder on what our little project is all about, whilst anyone can inspect the source code of free software for malicious changes most software is distributed to end users or servers as precompiled binaries. Reproducible builds tries to ensure that no changes have been made during these compilation processes by promising identical results are always generated from a given source, allowing multiple third-parties to come to a consensus on whether a build was compromised.

In this month’s report, we will cover:

  • Media coverage & conferencesReproducible builds in Belfast & science
  • Reproducible Builds Summit 2019Registration & attendees, etc.
  • Distribution workThe latest work in Debian, OpenWrt, openSUSE, and more…
  • Software developmentMore diffoscope development, etc.
  • Getting in touchHow to contribute & get in touch

If you are interested in contributing to our venture, please visit our Contribute page on our website.

Media coverage & conferences

Jonathan McDowell gave an introduction on Reproducible Builds in Debian at the Belfast Linux User Group:

Whilst not strictly related to reproducible builds, Sean Gallagher from Ars Technica wrote an article entitled Researchers find bug in Python script may have affected hundreds of studies:

A programming error in a set of Python scripts commonly used for computational analysis of chemistry data returned varying results based on which operating system they were run on.

Reproducible Builds Summit 2019

Registration for our fifth annual Reproducible Builds summit that will take place between the 1st and 8th December in Marrakesh, Morocco has opened and invitations have been sent out.

Similar to previous incarnations of the event, the heart of the workshop will be three days of moderated sessions with surrounding “hacking” days and will include a huge diversity of participants from Arch Linux, coreboot, Debian, F-Droid, GNU Guix, Google, Huawei, in-toto, MirageOS, NYU, openSUSE, OpenWrt, Tails, Tor Project and many more. We are still seeking additional sponsorship for the event. Sponsoring enables us to enable the attendance of people who would not otherwise be able to attend. If you or your company would be able to sponsor the event, please contact

If you would like to learn more about the event and how to register, please visit our our dedicated event page.

Distribution work

GNU Guix announced that they had significantly reduced the size of their “bootstrap seed” by replacing binutils, GCC and glibc with smaller alternatives resulting in the package manager “possessing a formal description of how to build all underlying software” in a reproducible way from a mere 120MB seed.

OpenWrt is a Linux-based operating system targeting wireless network routers and other embedded devices. This month Paul Spooren (aparcar) posted a patch to their mailing list adding KCFLAGS to the kernel build flags to make it easier to rebuild the official binaries.

Bernhard M. Wiedemann posted his monthly Reproducible Builds status update for the openSUSE distribution which describes how rpm was updated to run most builds with the -flto=auto argument, saving mirror disk space/bandwidth. In addition, maven-javadoc-plugin received a toolchain patch (originating from Debian) in order to normalise a date.


In Debian this month Didier Raboud (OdyX) started a discussion on the debian-devel mailing list regarding building Debian source packages in a reproducible manner (thread index). In addition, Lukas Pühringer prepared an upload of in-toto, a framework to protect supply chain integrity by the Secure Systems Lab at New York University which was uploaded by Holger Levsen.

Holger Levsen started a new section on the Debian wiki to centralise to document the progress made on various Debian-specific reproducibility issues and noticed that the “essential” package set in the bullseye distribution became unreproducible again, likely due to a a bug in Perl itself. Holger also restarted a discussion on Debian bug #774415 which requests that the devscripts collection of utilities that “make the life of a Debian package maintainer easier” adds a script/wrapper to enable easier end-user testing of whether a package is reproducible.

Johannes Schauer (josch) explained that their mmdebstrap tool can create bit-for-bit identical Debian chroots of the unstable and buster distributions for both the essential and minbase bootstrap “variants”, and Bernhard M. Wiedemann contributed to a discussion regarding adding a “global” build switch to enable/disable Profile-Guided Optimisation (PGO) and Link-time optimisation in the dpkg-buildflags tool, nothing that “overall it is still very hard to get reproducible builds with PGO enabled.”

64 reviews of Debian packages were added, 10 were updated and 35 were removed this month adding to our knowledge about identified issues. Three new types were added by Chris Lamb (lamby): nondeterministic_output_in_code_generated_by_ros_genpy, nondeterministic_ordering_in_include_graphs_generated_by_doxygen & nondeterministic_defaults_in_documentation_generated_by_python_traitlets.

Lastly, there was a far-reaching discussion regarding the correctness and suitability of setting the TZ environment variable to UTC when it was noted that the value UTC0 was “technically” more correct.

Software development

Upstream patches

The Reproducible Builds project detects, dissects and attempts to fix as many currently-unreproducible packages as possible. We endeavour to send all of our patches upstream where appropriate. This month, we wrote a large number of such patches, including:

Lastly, a request from Steven Engler to sort fields in the PKG-INFO files generated by the setuptools Python module build utilities was resolved by Jason R. Coombs and Vagrant Cascadian added SOURCE_DATE_EPOCH support to LTSP’s manual page generation.

strip-nondeterminism & reprotest

strip-nondeterminism is our tool to remove specific non-deterministic results from successful builds. This month, Chris Lamb made a number of changes including uploading version 1.6.1-1 was to Debian unstable. This dropped the test fixture as it is no longer compatible with the latest version of Perl’s Archive::Zip module (#940973).

reprotest is our end-user tool to build same source code twice in widely differing environments and then checks the binaries produced by each build for any differences. This month, Iñaki Malerba updated our Salsa CI scripts [] as well as adding a --control-build parameter []. Holger Levsen uploaded the package as 0.7.10, bumping the Debian “standards version” to 4.4.1 [].


diffoscope is our in-depth and content-aware diff utility that can locate and diagnose reproducibility issues. It is run countless times a day on our testing infrastructure and is essential for identifying fixes and causes of non-deterministic behaviour.

This month, Chris Lamb (lamby) made the following changes, including uploading versions 126, 127, 128 and 129 to the Debian unstable distribution:

  • Disassembling and reporting on files related to the R (programming language):

    • Expose an .rdb file’s absolute paths in the semantic/human-readable output, not hidden deep in a hexdump. []
    • Rework and refactor the handling of .rdb files with respect to locating the parallel .rdx prior to inspecting the file to ensure that we do not add files to the user’s filesystem in the case of directly comparing two .rdb files or — worse — overwriting a file in is place. []
    • Query the container for the full path of the parallel .rdx file to the .rdb file as well as looking in the same directory. This ensures that comparing two Debian packages shows any varying path. []
    • Correct the matching of .rds files by also detecting newer versions of this file format. []
    • Don’t read the site and user environment when comparing .rdx, .rdb or .rds files by using Rscript’s --vanilla option. [][]
    • Ensure all object names are displayed, including ones beginning with a fullstop (.) [] and sort package fields when dumping data from .rdb files [].
    • Mask/hide standard error when processing .rdb files [] and don’t include useless/misleading NULL when dumping data from them. []
    • Format package contents as foo = bar rather than using ugly and misleading brackets, etc. [] and include the object’s type [].
    • Don’t pass our long script to parse .rdb files via the command line; use standard input instead. []
    • Call the deparse function to ensure that we do not error out and revert to a binary diff when processing .rdb files with internal “vector” types; they do not automatically coerce to strings. []
    • Other misc/cosmetic changes. [][][]
  • Output/logging:
    • When printing an error from a command, format the command for the user. []
    • Truncate very long command lines when displaying them as an external source of data. []
    • When formatting command lines ensure newlines and other metacharacters appear escaped as \n, etc. [][]
    • When displaying the standard error from commands, ensure we use the escaped version. []
    • Use “exit code” over “return code” terminology when referring to UNIX error codes in displayed differences. []
  • Internal API:
    • Add ability to pass bytestring input to external commands. []
    • Split out command-line formatting into a separate utility function. []
    • Add support for easily masking the standard error of commands. [][]
    • To match the libarchive container, raise a KeyError exception if we request an invalid member from a directory. []
    • Correct string representation output in the traceback when we cannot locate a specific item in a container. []
  • Misc:
    • Move build-dependency on python-argcomplete to its Python 3 equivalent to facilitate Python 2.x removal. (#942967)
    • Track and report on missing Python modules. (#72)
    • Move from deprecated $ADTTMP to $AUTOPKGTEST_TMP in the autopkgtests. []
    • Truncate the tcpdump expected diff to 8KB (from ~600KB). []
    • Try and ensure that new test data files are generated dynamically, ie. at least no new ones are added without “good” reasons. []
    • Drop unused BASE_DIR global in the tests. []

In addition, Mattia Rizzolo updated our tests to run against all supported Python versions [] and to exit with a UNIX exit status of 2 instead of 1 in case of running out of disk space []. Lastly Vagrant Cascadian updated diffoscope 126 and 129 in GNU Guix, and updated inputs for additional test suite coverage.

trydiffoscope is the web-based version of diffoscope and this month Chris Lamb migrated the tool to depend on the python3-docutils package over python-docutils to allow for Python 2.x removal (#943293) as well as updating the packaging to the latest Debian standards and conventions [][][].

Project website

There was yet more effort put into our our website this month, including Chris Lamb improving the formatting of reports  [][][][][] and tidying the new “Testing framework” links [], etc.

In addition, Holger Levsen add the Tor Project’s Reproducible Builds Manager to our “Who is Involved?” page and Mattia Rizzolo dropped a literal <br> HTML element [].

Test framework

We operate a comprehensive Jenkins-based testing framework that powers This month, the following changes were made:

  • Holger Levsen:
    • Debian-specific changes:
      • Add a script to ease powercycling x86 and arm64 nodes. [][]
      • Don’t create suite-based directories for []
      • Make all four suites being tested shown in a single row on the performance page. []
    • OpenWrt changes:
      • Only run jobs every third day. []
      • Create jobs to run the script today and in the future. []
  • Mattia Rizzolo:
    • Add some packages that were lost while updating to buster. []
    • Fix the auto-offline functionality by checking the content of the permalinks file instead of following the lastSuccessfulBuild that no longer being updated. []
  • Paul Spooren (OpenWrt):
    • Add a reproducible_common utilities file. []
    • Update the openwrt-rebuild script to to use schroot. []
    • Use unbuffered Python output [] as well as fixing newlines [][]

The usual node maintenance was performed by Holger Levsen [][], Mattia Rizzolo [][][] and Vagrant Cascadian [][][].

Getting in touch

If you are interested in contributing the Reproducible Builds project, please visit our Contribute page on our website. However, you can get in touch with us via:

This month’s report was written by Bernhard M. Wiedemann, Chris Lamb, Holger Levesen and Vagrant Cascadian. It was subsequently reviewed by a bunch of Reproducible Builds folks on IRC and the mailing list.

06 November, 2019 06:25PM

November 05, 2019

hackergotchi for Jaldhar Vyas

Jaldhar Vyas

My Brexit Joke

As it is Bonfire Night (aka Guy Fawkes Night) and I might not get another chance to tell it given recent events, without further ado I present my Brexit joke.

Race relations in Britain have improved tremendously in recent years for Blacks and Asians but sadly bigots have a new target: the large number of people who have recently emigrated from Eastern Europe. Why Boris Johnson only made a pierogi in Parliament and one and half million people signed a petition to stop him.

There you have it, my Brexit joke. Thankyou very much and goodnight!

05 November, 2019 11:36PM

hackergotchi for Norbert Preining

Norbert Preining

SUSI.AI Smart Speaker release 20191105

After long and painful times due to the switch to buster, we at the FOSSASIA Team finally can release a new stable build of SUSI.AI for our RaspberryPi based smart speaker.

SUSI.AI aims at bringing a privacy aware personal assistant onto all of your devices. It runs on Android phones, desktop computers, and RaspberryPi based smart speakers. With the switch to Debian/buster, a lot of problems creeped in and the main application became extremely unstable, crashing in libportaudio2 with very unclear symptoms. Thanks to hint by Felix Yan we were able to fix libportaudio2 locally, and finally got a working and stable image.

During the last summer, a great team of GSoC students have worked on SUSI.AI in general, and on the smart speaker in particular. At the moment SUSI.AI can be installed onto RaspberryPi as well as any Debian based distribution (in particular Debian/buster and Ubuntu 19.04 upward). Core components of the SUSI.AI system are:

  • SUSI Server: the brain of the SUSI.AI system, which interprets questions and provides answers using a variety of built-in and user-created skills (Java)
  • SUSI Linux: the frontend for desktops and raspberry pis, listening to hotwords, and using speech-to-text to convert to text, querying the SUSI Server, and responding via text-to-speech conversion on the smart speaker (Python)
  • SUSI Skills: User editable skill database, where writing new skills is as easy as editing Wiki pages

plus a lot of setup, update, sync daemons.

After downloading the image and flashing it onto a sufficiently large SD card (at least 4Gb) via:

unxz susibian-20191105.img.xz
sudo dd if=susibian-20191105.img of=/dev/XXX bs=4M conv=fsync

(note that the device /dev/XXX needs to be adjusted for your SD card reader!), further setup is explained in this document. We are working on improving the documentation!

We hope by liberating the personal assistant market from the privacy-ignorant Google and Amazon forces, users will gain more freedom to choose and freedom to share, and freedom to create!

We are looking forward to feedback and suggestions, improvements, pull request! Please send either issues to one of the git repositories, or join us at one of the gitter channels (susi_hardware, susi_server, susi_android, susi_webclient, or generally at fossasia)!

05 November, 2019 11:49AM by Norbert Preining

Russ Allbery

Review: Cold Steel

Review: Cold Steel, by Kate Elliott

Series: Spiritwalker #3
Publisher: Orbit
Copyright: June 2013
ISBN: 0-316-21515-5
Format: Kindle
Pages: 597

Cold Steel is the third book in the closely-linked Spiritwalker trilogy. This is one long, sprawling story rather than separable books. If you want to read it, start with Cold Magic, and beware of spoilers for Cold Fire in even cursory descriptions of Cold Steel.

Cold Steel opens with Cat in the hot Caribbean, where most of Cold Fire was set, but does not stay there. This is a proper end to the trilogy and wraps up all the loose ends: the peril Andevai was left in after the semi-cliffhanger ending of Cold Fire, the conflict over governance in Europe, the power of the cold mage Houses, the politics of the radicals, the conflict between cold and fire magic, Bee's entanglements in the Caribbean, Camjiata's war of conquest, and of course the Wild Hunt. There's a lot of plot to finish.

Somewhat unfortunately, what Cold Steel also contains in large quantity is Andevai being an idiot, Cat and Andevai failing to communicate for new (but dumb) reasons, and one of the characters developing into a full-blown arch-villain.

This is a series with a fascinating premise (a very complex alternate history with dragons, magic, humanoid dinosaurs, and a very odd sort of elven court) that I wanted to be a political drama but that the author wrote primarily as a romance. This is not entirely the author's fault; it's a serviceable romance if you enjoy two vastly stubborn people butting heads, falling in love, discovering that being in love doesn't prevent them from butting heads, and finally reaching some hard-fought compromises. But I thought the romance was the least interesting (and least original) thing that was happening in this series and wanted to get it over with so that the series could move on to the more interesting bits, and that opinion was not shared by the author.

The part that caught my attention, as noted in my review of Cold Fire, is the political cause of radicalism and egalitarianism. The history varies wildly from ours, but the politics are clearly late French Revolution writ large (without the church, which is interesting): princes and cold mages colluding to maintain an aristocratic, almost feudal order, resented by laborers and political radicals who are trying to form new governments of equality and to undermine historical privilege and systems of indenture. Into that mix, Elliott drops Camjiata, a brilliantly ambiguous figure who is this world's Napoleon. He uses the radicals, speaks on their terms, and claims to fight for equality, but he personally wants to conquer Europe and bring back the glories of the Roman Empire. He's a strategic genius, one of the smartest people in the book (that's a difficult characterization to write well and Elliott does a good job), but his agenda isn't that of the protagonists or, really, anyone else. Elliott gives the impression of Camjiata as something that happens to Europe more than a force that comes out of the politics everyone else cares about, which I found intriguing throughout the series.

Unfortunately, despite radicalism's central role in the plot and despite Cat's sister Bee finding an unexpected but believable place for herself as a leader, the politics don't go anywhere satisfying. There is little resolution for Europe here, just some personal impact for Andevai and Cat and a parallel mythical plot line with the Wild Hunt that I thought was a bit too obvious (although the resolution of it is satisfying). A series that wrestled with the political complexities of defining radicalism as a constructive form of government instead of an opposition resistance force (the shoal that, to simplify greatly, the real French Revolution ran aground on) would have been messy and challenging to write, but that was the book I wanted to read. Cold Steel alas turns somewhat away from that to personalize the problems and solve them primarily in the realms of magic and romance.

The other difficulty with this final book is that it's structurally a mess. There are too many loose ends to weave together, which results in some odd pacing and story structure. The book starts as a continuation of Cold Fire, involving Caribbean politics and magic, and builds up to a climax with the fairy world a third of the way into the book. Then comes a quiet interlude with Cat and Andevai that stumbles into a pacing gap where Cat is unhappy but not communicating, Andevai is ignoring her, no forward progress is made on the major plots of the novel, and there is a lot of frustrating talking. Pacing for the rest of the book comes in fits and starts, building up to a weirdly unsatisfying war and turning the broader politics of the series into a too-simplistic arc of personal animus, mustache-twirling evil, and magical penis-measuring. Even the ending, although eventually satisfying, provides another excuse for Cat to not talk to people about things that are important.

I'm probably making this sound worse than it is. I am glad I read the whole series. It's hugely ambitious and succeeds in being something different than the typical fantasy trilogy. The world background is one of the most creative I've seen, and if I found the politics unsatisfying in the end, that's only because Elliott attempted something far more ambitious and radical than most fantasy will go near. I think the first book was the strongest and the series sputtered to a conclusion, but it is a proper and satisfying conclusion and both Cat and Andevai, by the end of the series, are characters I enjoy spending time with.

Cold Steel by itself is a bit of a mess, but I still recommend the series as a whole, as long as you're in the mood for something long and complicated and not completely successful.

Rating: 6 out of 10

05 November, 2019 04:48AM

November 04, 2019

hackergotchi for Julien Danjou

Julien Danjou

Finding definitions from a source file and a line number in Python

Finding definitions from a source file and a line number in Python

My job at Datadog keeps me busy with new and questioning challenges. I recently stumbled upon a problem that sounded easy but was more difficult than I imagined.

Here's the thing: considering a filename and a line number, can you tell which function, method or class this line of code belongs to?

I started to dig into the standard library, but I did not find anything solving this problem. It sounded like I had to write this myself.

The first steps sound easy. Open a file, read it, find the line number. Right.

Then, how do you know which functions this line is in? You don't, expect if you parse the whole file and keep tracks of function definitions. A regular expression parsing each line might be a solution?

Well, you had to be careful as function definitions can span multiple lines.

Using the AST

I decided that a good and robust strategy was not going to use manual parsing or the like, but using Python abstract syntax tree (AST) directly. By leveraging Python's own parsing code, I was sure I was not going to fail while parsing a Python source file.

This can be simply be accomplished with:

import ast

def parse_file(filename):
    with open(filename) as f:
        return ast.parse(, filename=filename)

And you're done. Are you? No, because that only works in 99.99% of the case. If your source file is using an encoding that is now ASCII or UTF-8, then the function fails. I know you think I'm crazy to think about this but I like my code to be robust.

It turns out Python has a cookie to specify the encoding in the form of # encoding: utf-8 as defined in PEP 263. Reading this cookie would help to find the encoding.

To do that, we need to open the file in binary mode, use a regular expression to match the data, and… Well, it's dull, and somebody already implemented it for us so let's use the fantastic function provided by Python:

import ast
import tokenize

def parse_file(filename):
    with as f:
        return ast.parse(, filename=filename)

That should work in 100% of the time. Until proven otherwise.

Browsing the AST

The parse_file function now returns a Python AST. If you never played with Python AST, it's a gigantic tree that represents your source code just before it is compiled down to Python bytecode.

In the tree, there should be statements and expression. In our case, we're interested in finding the function definition that is the closest to our line number. Here's an implementation of that function:

def filename_and_lineno_to_def(filename, lineno):
    candidate = None
    for item in ast.walk(parse_file(filename)):
        if isinstance(item, (ast.FunctionDef, ast.AsyncFunctionDef, ast.ClassDef)):
            if item.lineno > lineno:
                # Ignore whatever is after our line
            if candidate:
                distance = lineno - item.lineno
                if distance < (lineno - candidate.lineno):
                    candidate = item
                candidate = item

    if candidate:

This iterates over all the node of the AST and returns the node where the line number is the closest to our definition. If we have a file that contains:

class A(object):
    X = 1
    def y(self):
        return 42

the function filename_and_lineno_to_def returns for the lines 1 to 5:

Return of filename_and_lineo_to_def for lines 1 to 5

It works!


The naive approach described earlier likely works for 90% of your code, but there are some edge cases. For example, when defining function closures, the above algorithm fails. With the following code:

class A(object):
   X = 1
   def y(self):
       def foo():
           return 42
       return foo

the function filename_and_lineno_to_def returns for lines 1 to 7:

Return of filename_and_lineo_to_def for lines 1 to 7

Oops. Clearly, lines 6 and 7 do not belong to the foo function. Our approach is too naive to see that starting at line 6, we're back in the y method.

Interval Trees

The correct way of handling that is to consider each function definition as an interval:

Finding definitions from a source file and a line number in PythonPiece of code seen as interval.

Whatever the line number we request is, we should return the node that is responsible for the smallest interval that the line is in.

What we need in this case is a correct data structure to solve our problem: an interval tree fits perfectly our use case. It allows for searching rapidly pieces of code that match our line number.

To solve our problem we need several things:

  • A way to compute the beginning and end line numbers for a function.
  • A tree that is fed with the intervals we computed just before.
  • A way to select the best matching intervals if a line is part of several functions (closure).

Computing Function Intervals

The interval of a function is the first and last lines that compose its body. It's pretty easy to find those by walking through the function AST node:

def _compute_interval(node):
    min_lineno = node.lineno
    max_lineno = node.lineno
    for node in ast.walk(node):
        if hasattr(node, "lineno"):
            min_lineno = min(min_lineno, node.lineno)
            max_lineno = max(max_lineno, node.lineno)
    return (min_lineno, max_lineno + 1)

Given any AST node, the function returns a tuple of the first and last line number of that node.

Building The Tree

Rather than implementing an interval tree, we'll use the intervaltree library. We need to create a tree and feed it with the computed interval:

def file_to_tree(filename):
    with as f:
        parsed = ast.parse(, filename=filename)
    tree = intervaltree.IntervalTree()
    for node in ast.walk(parsed):
        if isinstance(node, (ast.FunctionDef, ast.AsyncFunctionDef, ast.ClassDef)):
            start, end = _compute_interval(node)
            tree[start:end] = node
    return tree

Here you go: the function parses the Python file passed as an argument and converts it to its AST representation. It then walks it and feeds the interval tree with every class and function definition.

Querying the Tree

Now that the tree is built, it should be queried with the line number. This is pretty simple:

matches = file_to_tree(filename)[lineno]
if matches:
    return min(matches, key=lambda i: i.length())

The build tree might return several matches if there are several intervals containing our line number. In that case, we pick the smallest interval and return the name of the node — which is our class or function name!

Mission Success

We did it! We started with a naive approach and iterated to a final solution covering 100% of our cases. Picking the right data structure, interval trees here, helped us solving this in an intelligent approach.

04 November, 2019 10:21AM by Julien Danjou

November 03, 2019

hackergotchi for Ben Hutchings

Ben Hutchings

Debian LTS work, October 2019

I was assigned 22.75 hours of work by Freexian's Debian LTS initiative. I worked almost all those hours this month, but will carry over 0.5 hours to November.

I prepared and, after review, released Linux 3.16.75, including various important fixes. I then rebased the Debian package onto that, and sent out a request for testing. I prepared and sent out Linux 3.16.76-rc1 for review.

I handled a misdirected request to update the tzdata package, adding that and the related Perl library to the dla-needed.txt file. I responded to a support request regarding Intel microcode updates for security issues. I also spent some time working on security issues that are still under embargo.

03 November, 2019 10:54PM

hackergotchi for Chris Lamb

Chris Lamb

Tour d'Orwell: No. 1 South End Road

In yet one more George Orwell-themed outing, No. 1 South End Road in South Hampstead in north London is the former location of the bookshop that George Orwell worked at from 1934-1935.

He would write rather touchingly about his experiences in an essay titled Bookshop Memories where he, inter alia, described it as a place patronised by those who…

… would be a nuisance anywhere, but have special opportunities in a bookshop.

And in a further comment that has modern day echoes in Netflix, iPlayer and friends:

In a lending library you see people's real tastes, not their pretended ones.

I was meeting with fellow Blairite — Eric…! — to whom I was finally able to gift a copy of Burmese Days that I had acquired in Southwold, the Suffolk village in which he wrote it.

In a nod to Orwell's multiculturalism today the bookshop is now a (Belgian) «Le Pain Quotidien» and big glass windows ensure it is no longer a "gloomy cave of a place." I had a fairly passable and somewhat ironic shakshuka, something that Orwell himself might have sampled whilst residing in Marrakech...

03 November, 2019 07:26PM

hackergotchi for Urvika Gola

Urvika Gola

Grace Hopper Celebration’19, Orlando

The first week of October 2019, which happens to be my Birthday as well, I got an opportunity to attend Grace Hoppers Conference.

Being in there with 26,000 other women in tech at the same place was thrilling, overwhelming and quite tiring at the same time. I say tiring because of the 2 hours wait in the long queues just to pick up a paper-badge and later for Keynotes, regular sessions, food, even to get a picture with the great GHC signboard. Waking up at 5 am and going to the convention center to pick my badge is a story for another day..

Intel_group_GHC.jpgIntel Women In Tech at GHC #IamIntel

About Grace Hoppers Celebration (GHC) – It’s the world’s largest gathering of Women and other underrepresented groups in Computing / Technology. This year they had a massive 26 thousand amazing and lively group of individuals attending GHC. I got to know about the conference when I was pursuing my Bachelors, GHC-India version of it which of course happens in India every year. I was selected as a Student Scholar for GHC-India. Honestly, I always wanted to be a part of GHC and meet like-minded people sharing different yet similar challenges to be a part of Technology.

This year GHC happened in Orlando, Florida at Orange County Convention Center from 1st – 4th October 2019.

The conference kickstarted with inspiring Keynote sessions and awards. This year’s GHC theme was “We will..” and you could fill in your own interpretation in what ways the statement could be filled.

One of the topics of discussion I really liked was igniting the genius in every child using technology to bring the best quality education to communities around the world, especially the most underserved.

When this video was previewed at GHC keynote, It quite literally gave me goosebumps seeing high school kids talking about what solutions they wish to build via technology. I was pondering on the very insistence by society and all the hype about “Growing up” and doing wonders while we could still do wonders without growing up.

Unfortunately, all the sessions of GHC are not recorded. There were sessions that were happening in parallel, equally interesting and then you have to make this choice of just watching either of them. It’s a pain but that’s how it is. Well, you can still watch some of the keynotes here, which is great!

One of my favorite keynote session amongst the many was by Dr. Fei Fei Lee where she talked about the remarks she received from her admiring colleagues about her idea about image-net, Now, it has a key contribution towards Deep Learning revolution and foundation for the way Machine Learning projects are developed.  She said “It is okay to feel small but together we can be big enough to accomplish anything” and it takes conviction to find your circle.

One unexpected performance which gave everyone chills was themed as how women in tech we have been told false narrative who we are and what we can do. Together we will flip the script and change the status quo. It was uber cool and inspiring to watch this performance. Few snapshots below.



I was looking forward to the keynote by Aicha Evans. She was one of my most admired women in tech in a leadership role. She talked about her vision and necessity for Autonomous Mobility. She concluded by mentioning the famous proverb.

“If you want to go FAST, go ALONE.  If you want to go FAR, go TOGETHER. ”

It’s always a good feeling to meet Outreachy people, I met one of a former Outreachy Intern, we both interned during the same round under the same org, Debian.

Intel’s been a proud supporter for GHC for years and has a strong presence at GHC.

Intel_Hiring_Booth_GHCIntel’s Hiring Booth at GHC



Intel is committed to advancing the uses of AI that most positively impact the world. The hardware, software, and algorithms that we create and utilize on a daily basis can help identify children at risk of exploitation, improve medical diagnosis, and protect our natural resources. Intel is passionate about equalizing AI resources, capabilities, and services to benefit all. Source – AI4Good

Few products we put out there as a part of Technology Showcase at GHC:-

Intel’s AI TrailGaurd Camera

  • Non-profit RESOLVE’s* new TrailGuard AI* camera uses Intel-powered artificial intelligence technology to detect poachers entering Africa’s wildlife reserves and alert park rangers in near real-time so poachers can be stopped before killing endangered animals.
  • As per stats, 25k – 35k elephants killed annually for their ivory, every day a Rhino is poached for its horn. Lions have lost 43% of their population over the past 20 years. While many of these animals live on protected reserves, the areas are vast, the infrastructure limited and park ranger resources spread thin that it is difficult to stop poachers before they kill.
  • TrailGuard AI builds on anti-poaching prototypes funded by Leonardo DiCaprio Foundation and National Georaphic Society.
  • Find more details here ->

Intel’s Volumetric Capture

  • Since Intel has moved its dial from PC-centric to Data-centric, Intel continues to position itself as the leader in immersive media across key segments including sports and entertainment.
  • Scenes inside the dome from all directions at once, a technique called “volumetric capture.”
  • Data captured by each camera is shaped into voxels (think 3D pixels), which render the virtual environment in multi-perspective 3D.
  • It allows audiences to view a scene from any angle – even the middle of the action.
    This Grease piece was captured at the world’s largest volumetric stage – Intel Studios.
  • Find more details here ->

At the end of it, attending GHC was truly an extraordinary experience. I felt grateful to be a part of the celebration. Thanks to my employer – Intel and Intel’s team for the opportunity.

03 November, 2019 06:48PM by urvikagola

hackergotchi for Jonathan Carter

Jonathan Carter

Free Software Activities (2019-10)

Bits and pieces

AIMS Desktop talk: On the 1st of October I gave a talk at AIMS titled “Why AIMS Desktop?” where I talked about the Debian-based system we use at AIMS, and went into some detail about what Linux is, what Debian is, and why it’s used within the AIMS network. I really enjoyed the reaction from the students and a bunch of them are interested in getting involved directly with Debian. I intend to figure out a way to guide them into being productive community members without letting it interfere with their academic program.

Catching up with Martin: On the 12th of October I had lunch with Martin Michlmayr. Earlier this year we were both running for DPL, which was an interesting experience and the last time we met neither of us had any intentions to do so. This was the first time we talked in person since then and it was good reflecting over the last year and we also talked about a bunch of non-Debian stuff.

Cover art of our band?

Activity log

2019-10-08: Upload package bundlewrap (3.7.0-1) to Debian unstable.

2019-10-08: Upload package calamares (3.2.14-1) to Debian unstable.

2019-10-08: Sponsor package python3-fastentrypoints (0.12-2) for Debian unstable (Python team request).

2019-10-08: Sponsor package python3-cheetah (3.2.4-1) for Debian unstable (Python team request).

2019-10-14: Upload package calamares (3.2.15-1) to Debian unstable.

2019-10-14: Upload package kpmcore (4.0.1-1) to Debian unstable.

2019-10-14: Upload package gnome-shell-extension-disconnect-wifi (21-1) to Debian unstable.

2019-10-15: Upload package partitionmanager (4.0.0-1) to Debian unstable.

2019-10-15: Sponsor package python-sabyenc (3.3.6-1) for Debian unstable (Python team request).

2019-10-15: Sponsor package python-jaraco.functools (2.0-1) for Debian unstable (Python team request).

2019-10-15: Sponsor package python3-gntp (1.0.3-1) for Debian unstable (Python team request).

2019-10-15: Review package python3-portend (2.5-1) (Not yet ready) (Python team request).

2019-10-15: Review package python3-tempora (1.14.1) (Net yet ready) (Python team request))

2019-10-15: Upload package python3-flask-silk (0.2-15) to Debian unstable.

2019-10-15: Upload package tuxpaint (0.9.24~git20190922-f7d30d-1~exp2) to Debian experimental.

2019-10-15: Upload package python3-flask-silk (0.2-16) to Debian unstable.

2019-10-16: Upload package gnome-shell-extension-multi-monitors (19-1) to Debian unstable.

2019-10-16: Upload package python3-flask (0.6.2-5) to Debian unstable.

2019-10-16: Sponsor package buildbot (2.4.1-1) for Debian unstable (Python team request).

2019-10-16: Signed/sent keys from DebConf19 KSP.

2019-10-17: Publish blog entry “Calamares plans for Debian 11“.

2019-10-17: Upload package kpmcore (4.0.1-2) to Debian unstable (Thanks to Alf Gaida for the merge request with fixes) (Closes: #942522, #942528, #942512).

2019-10-22: Sponsor package membernator (1.1.0-1) for Debian unstable (Python team request).

2019-10-22: Sponsor package isbg (2.1.5-1) for Debian unstable (Python team request).

2019-10-22: Sponsor package python-pluggy (0.13.0-1) for Debian unstable (Python team request).

2019-10-22: Sponsor package python-pyqt5chart (5.11.3+dfsg-2) for Debian unstable (Python team request).

2019-10-23: Upload package tetzle (2.1.4+dfsg1-3) to Debian unstable.

2019-10-23: Upload package partitionmanager (4.0.0-2) to Debian unstable.

2019-10-24: Upload package tetzle (2.1.5+dfsg1-1) to Debian unstable.

2019-10-24: Upload package xabacus (8.2.2-1) to Debian unstable.

2019-10-24: Review package (fpylll) (Needs some more work) (Python team request).

2019-10-28: Upload package gnome-shell-extension-dash-to-dock (25-1) to Debian unstable.

03 November, 2019 06:18PM by jonathan

Thorsten Alteholz

My Debian Activities in October 2019

FTP master

This month I accepted 257 packages and rejected 17. The overall number of packages that got accepted was 314.

Debian LTS

This was my sixty fourth month that I did some work for the Debian LTS initiative, started by Raphael Hertzog at Freexian.

This month my all in all workload has been 22.75h. During that time I did LTS uploads of:

    [DLA 1969-1] file security update for one CVE
    [DLA 1971-1] libarchive security update for one CVE
    [DLA 1972-1] mosquitto security update for four CVEs
    [DLA 1974-1] proftpd-dfsg security update for one CVE
    [DLA 1975-1] spip security update for four CVEs
    [DSA-4557-1] libarchive security update for three CVEs in Stretch and one CVE in Buster

I also started to work on tiff.

Last but not least I did some days of frontdesk duties.

Debian ELTS

This month was the seventeenth ELTS month.

During my allocated time I uploaded:

  • ELA-181-1 of blktrace
  • ELA-182-1 of file
  • ELA-184-1 of libarchive

I also had a look at wireshark and ncurses but did not do an upload yet.

I also did some days of frontdesk duties.

Other stuff

This month I uploaded new upstream versions of …

I improved packaging of …

03 November, 2019 03:27PM by alteholz

hackergotchi for Steinar H. Gunderson

Steinar H. Gunderson

Chess960 opening recalculation

Congratulations to Wesley So as the very first official Chess960 world champion!

A while back, I set Stockfish to analyze every opening position in Chess960 (also knows as Fischer Random chess, or Fischerandom, or several other names). For the World Championship, I redid this analysis with a newer version of Stockfish to higher depths (d41 on most positions, d42 on others)—and this time, I kept the PV. (I also fixed a fairly important bug; thanks to Peter Heine Nielsen for spotting it.)

You can find the updated version at It completely supersedes the old one.

03 November, 2019 02:00PM

Russell Coker

KMail Crashing and LIBGL

One problem I’ve had recently on two systems with NVideo video cards is KMail crashing (SEGV) while reading mail. Sometimes it goes for months without having problems, and then it gets into a state where reading a few messages (or sometimes reading one particular message) causes a crash. The crash happens somewhere in the Mesa library stack.

In an attempt to investigate this I tried running KMail via ssh (as that precludes a lot of the GL stuff), but that crashed in a different way (I filed an upstream bug report [1]).

I have discovered a workaround for this issue, I set the environment variable LIBGL_ALWAYS_SOFTWARE=1 and then things work. At this stage I can’t be sure exactly where the problems are. As it’s certain KMail operations that trigger it I think that’s evidence of problems originating in KMail, but the end result when it happens often includes a kernel error log so there’s probably a problem in the Nouveau driver. I spent quite a lot of time investigating this, including recompiling most of the library stack with debugging mode and didn’t get much of a positive result. Hopefully putting it out there will help the next person who has such issues.

Here is a list of environment variables that can be set to debug LIBGL issues (strangely I couldn’t find documentation on this when Googling it). If you are stuck with a problem related to LIBGL you can try setting each of these to “1” in turn and see if it makes a difference. That can either be for the purpose of debugging a problem or creating a workaround that allows you to run the programs you need to run. I don’t know why GL is required to read email.


03 November, 2019 12:39AM by etbe