December 06, 2019

hackergotchi for Rhonda D'Vine

Rhonda D'Vine

Oxa

It's been a while. And to be honest, I'm overdue with a few things that I want to get out. One of those things is … Brazil doesn't let me go. I'm watching this country since over a year now, hopefully understandable with the political changes last year and this year's debconf being there, and I promise to go into more details with that in the future because there is more and more to it …

Because one of those things that showed me that Brazil doesn't want to let me go was stumbling upon this artist. They were shared by some friends, and I instantly fell for them. This is about Oxa, but see for yourself:

  • Toy: Their first performance at the show »The Voice of Germany«, where they also stated that they are non-binary. And the song is lovely.
  • Born This Way: With this one, the spoken word interlude gave me goosebumps and I'm astonished that this was possible to get into the show. Big respect!
  • I'm Still Standing: The lyrics in this song are also just as powerful as the other chosen ones. Extremely fine selection!

I'm absolute in love with the person on so many levels–and yes, they are from Brazil originally. Multo brigado, Brazil!

/music | permanent link | Comments: 0 | Flattr this

06 December, 2019 11:01PM by Rhonda

Enrico Zini

Read only rootfs

This is part of a series of post on the design and technical steps of creating Himblick, a digital signage box based on the Raspberry Pi 4.

Another nice to have in a system like Himblick is the root filesystem mounted readonly, with a volatile tempfs overlay on top. This would kind of always guarantee a clean boot without leftovers from a previous run, especially in a system where the most likely mode of shutdown is going to be pulling the plug.

This won't be a guarantee about SD issues developing over time in such a scenario, but it should at least cover the software side of things.

In theory, systemd supports this out of the box with the systemd.volatile=overlay kernel command line option, including integrating this with journald and the way other things get mounted.

In practice:

While things catch up, dracut has a rootovl option that implements something similar.

The procedure becomes, roughly:

# apt install --no-install-recommends dracut
# echo filesystems+=overlay > /etc/dracut.conf.d/overlay.conf
# dracut /boot/initrd.img
# sed -ri -e '/$/ rootovl/' /boot/cmdline.txt" rootovl" in cmdline
# echo "initramfs initrd.img" >> /boot/config.txt

Here's how it ended up in python:

    def setup_readonly_root(self):
        """
        Setup a readonly root with a tempfs overlay
        """
        # Set up a readonly root using dracut's 'rootovl' feature.
        # Eventually do this with systemd's systemd.volatile=overlay option.
        # See https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=945596
        self.apt_install("dracut")
        self.write_file("/etc/dracut.conf.d/overlay.conf", "filesystems+=overlay\n")
        self.run(["dracut", "--force", "/boot/initrd.img", "4.19.75-v7l+"])
        with self.edit_kernel_commandline("/boot/cmdline.txt") as parts:
            # Add 'rootovl' to /etc/cmdline
            if "rootovl" not in parts:
                parts.append("rootovl")

        # Add initramfs initrd.img to config.txt
        with self.edit_text_file("/boot/config.txt") as lines:
            if "initramfs initrd.img" not in lines:
                lines.append("initramfs initrd.img")

06 December, 2019 11:00PM

hackergotchi for Vincent Bernat

Vincent Bernat

Replacing Orange Livebox router by a Linux box

A few months ago, I moved back to France and I settled for Orange as an ISP with a bundle combining Internet and mobile subscription. In Switzerland, I was using my own router instead of the box provided by Swisscom. While there is an abundant documentation to replace the box provided by Orange, the instructions around a plain Linux box are kludgy. I am exposing here my own variation. I am only interested in getting IPv4/IPv6 access: no VoIP, no TV.

Hardware​

Orange is using GPON for its FTTH deployment. Therefore, an ONT is needed to encapsulate and decapsulate Ethernet frames into GPON frames. Two form-factors are available. It can be small Huawei HG8010H box also acting as a media converter to Ethernet 1000BASE-T:

Huawei ONT rebranded as Orange
The rebranded Huawei HG8010H is acting as an ONT and media converter

With a recent Livebox, Orange usually provides an SFP to be plugged inside the Livebox. For some reason I got the external ONT instead of the SFP version. As I have a Netgear GS110TP with two SFP ports, I have bought an SFP GPON FGS202 on eBay. It is the same model than Orange is providing with its Livebox 4. However, I didn’t get the motivation to test it.1

Sercomm SFP ONT
The Sercomm FGS202 GPON SFP ONT

IPv4 configuration​

Internet is provided over VLAN 832 and configured with DHCPv4. The first step is to setup the DHCP client to send some additional information, notably the RFC 3118 authentication string. It includes the alphanumeric connection identifier prefixed by fti/ and provided by snail mail. /etc/dhcp/dhclient.conf looks like this:

option rfc3118-authentication code 90 = string;
interface "internet" {
  timeout 60;
  retry 1;
  select-timeout 0;
  send vendor-class-identifier "sagem";
  send user-class "+FSVDSL_livebox.Internet.softathome.Livebox4";
  # fti/xxxxxx identifier can be converted to hexadecimal with:
  #  echo -n 123456 | od -A n -t x1
  send rfc3118-authentication 00:00:00:00:00:00:00:00:00:00:00:1a:09:00:00:05:58:01:03:41:01:0d:66:74:69:2f:xx:xx:xx:xx:xx:xx:xx;
  request subnet-mask, routers,
          domain-name-servers, domain-name,
          broadcast-address,
          dhcp-lease-time, dhcp-renewal-time, dhcp-rebinding-time,
          rfc3118-authentication;
}

Orange expects some control packets, notably DHCP, to be tagged with 802.1p PCP 6. This is a 3-bit field within the Ethernet frame when using VLANs. By default, Linux leaves this field blank. With ip link, we can translate Linux’s skb->priority to a PCP. On Debian, here is how to declare the VLAN interface:2

auto internet
iface internet inet dhcp
  pre-up    ip link add link eno1 name internet type vlan id 832 egress-qos-map 0:0 6:6
  pre-up    /etc/firewall/run
  post-down ip link del internet

The last step is to add the appropriate code in /etc/firewall/run to ensure DHCP, ARP, IGMP and ICMP packets have an internal priority of 6. Netfilter’s CLASSIFY target would be the easiest solution. However, ISC DHCP client is using raw sockets and the packets it sent won’t pass throught Netfilter. A clean solution is to use tc to modify packets just before handing them to the network card. The skbedit action allows to change the priority associated to a packet:

# We need a qdisc to set filters
tc qdisc replace dev internet root handle 1: prio
tc filter del dev internet

# DHCP (raw sockets, do not specify "protocol ip")
tc filter add dev internet parent 1: prio 1 u32 \
     match ip protocol 17 ff \
     match ip dport 67 ffff \
     action skbedit priority 0:6
# ARP
tc filter add dev internet parent 1: prio 2 protocol 0x806 u32 \
     match u32 0 0 \
     action skbedit priority 0:6
# IGMP
tc filter add dev internet parent 1: prio 3 protocol ip u32 \
     match ip protocol 2 ff \
     action skbedit priority 0:6
# ICMP
tc filter add dev internet parent 1: prio 4 protocol ip u32 \
     match ip protocol 1 ff \
     action skbedit priority 0:6

With this configuration in place, ifup internet should get you connected through IPv4.

IPv6 configuration​

Native IPv6 is also available over the same VLAN. SLAAC autoconfiguration should be used to get a default route, but not the IP address. Instead, Orange is providing a /60 prefix through DHCPv6 “prefix delegation.”

The DHCP configuration is completed to send the DHCPv6 equivalents for vendor class, user class and authentication string:

# […]
option dhcp6.auth code 11 = string;
option dhcp6.userclass code 15 = string;
option dhcp6.vendorclass code 16 = string;
interface "internet" {
  timeout 60;
  retry 1;
  select-timeout 0;
  # […]
  send dhcp6.vendorclass 00:00:04:0e:00:05:73:61:67:65:6d;
  send dhcp6.userclass 00:2b:46:53:56:44:53:4c:5f:6c:69:76:65:62:6f:78:2e:49:6e:74:65:72:6e:65:74:2e:73:6f:66:74:61:74:68:6f:6d:65:2e:6c:69:76:65:62:6f:78:34;
  send dhcp6.auth 00:00:00:00:00:00:00:00:00:00:00:1a:09:00:00:05:58:01:03:41:01:0d:66:74:69:2f:xx:xx:xx:xx:xx:xx:xx;
  also request dhcp6.auth, dhcp6.vendorclass, dhcp6.userclass;
}

The firewall script is amended to classify DHCPv6 and ICMPv6 packets with priority 6:

# DHCPv6
tc filter add dev internet parent 1: prio 5 protocol ipv6 u32 \
     match ip6 protocol 17 ff \
     match ip6 dport 547 ffff \
     action skbedit priority 0:6
# ICMPv6
tc filter add dev internet parent 1: prio 6 protocol ipv6 u32 \
     match ip6 protocol 58 ff \
     action skbedit priority 0:6

The definition of the internet interface is updated to invoke the DHCPv6 client:

auto internet
iface internet inet dhcp
  pre-up    ip link add link eno1 name internet type vlan id 832 egress-qos-map 0:0 6:6
  pre-up    /etc/firewall/run
  post-down ip link del internet
  post-up   /lib/ifupdown/wait-for-ll6.sh && \
            dhclient -6 -P -pf /run/dhclient6.$IFACE.pid \
                           -lf /var/lib/dhcp/dhclient6.$IFACE.leases \
                           -df /var/lib/dhcp/dhclient.$IFACE.leases \
                           $IFACE
  post-down dhclient -6 -r -pf /run/dhclient6.$IFACE.pid dhclient \
                           -lf /var/lib/dhcp/dhclient6.$IFACE.leases \
                           -df /var/lib/dhcp/dhclient.$IFACE.leases \
                           $IFACE || true

The /lib/ifupdown/wait-for-ll6.sh script waits for the interface to get a link-local address before continuing. The -P option for the DHCPv6 client enables prefix delegation and disables the normal address query.

It is not over: the DHCPv6 client will receive a /60 prefix but there is nothing configured to make use of it. You need to drop a script in /etc/dhcp/dhclient-exit-hooks.d to actually distribute this prefix to your internal network. Here is a simplified non-tested version of this script:

#!/bin/sh
IA_PD_IFACES="lan-trusted lan-guest lan-games"

case $reason in
  BOUND6|EXPIRE6|REBIND6|REBOOT6|RENEW6)
    offset=0
    for iface in $IA_PD_IFACES; do
      # Remove old /64 prefix if there is a change
      [ -n "$old_ip6_prefix" ] && \
        [ "$old_ip6_prefix" != "$new_ip6_prefix" ] && \
        ip -6 addr flush dev $iface scope global
      # Compute and add new /64 prefix
      [ -n "$new_ip6_prefix" ] && {
        offset=$((offset + 1))
        address=$(sipcalc --v6split=64 --split-verbose "$new_ip6_prefix" \
                   | grep '^Compressed' \
                   | awk "(NR == $offset)"' { print $NF }')1/64
        ! ip -6 addr show dev $iface | grep -qwF $address || \
          ip -6 addr add $address dev $iface
    done
esac

At the top of the script, the IA_PD_IFACES variable contains the list of internal interfaces. From the /60 provided in $new_ip6_prefix, the script will assign a /64 to each of them—along with the first address. For example, when being assigned 2001:db8:f:b00::/60, we get:

$ ip -brief -6 a show scope global
lan-trusted@eno1  UP  2001:db8:f:b00::1/64
lan-guest@eno1    UP  2001:db8:f:b01::1/64
lan-games@eno1    UP  2001:db8:f:b02::1/64

I am using dnsmasq to offer IPv6 router advertisements to hosts in each network. This is done through the dhcp-range directive:

dhcp-range=::,constructor:lan-trusted,ra-names
dhcp-range=::,constructor:lan-guest,ra-names
dhcp-range=::,constructor:lan-games,ra-names

The script also handles the default route by switching accept_ra to 2 for the internet interface to accept IPv6 router advertisements even when forwarding is enabled and sending an IPv6 router discovery packet using rdisc6:

case $old_ip6_prefix,$new_ip6_prefix in
  *,)
    # No IPv6 prefix delegation, remove old route
    sysctl -qw net/ipv6/conf/$interface/accept_ra=0
    ip -6 route del default proto ra || true
    ;;
  *)
    # Otherwise, get a default route
    sysctl -qw net/ipv6/conf/$interface/accept_ra=2
    rdisc6 $interface
    ;;
esac

Be sure to use the complete script instead of the shortened code above! If after ifdown internet && ifup internet, you don’t get a /60 prefix, you may have to reboot the ONT to clear an old DHCP lease.

  1. As Orange is using the serial number to authorize the ONT, my plan is to call Orange customer service, pretend I have got a replacement and provide the new serial number. ↩︎

  2. There is no need to have the VLAN number in the interface name. I usually leaves them out as it doesn’t help to describe the interface. The VLAN number can still be recovered with ip -d link show. ↩︎

06 December, 2019 11:15AM by Vincent Bernat

hackergotchi for Gunnar Wolf

Gunnar Wolf

My greedy hands are full of books! (Made With Creative Commons) @ccmx @creativecommons @xattack @scannopolis

FINALLY!

Made with Creative Commons is translated to Spanish, printed, and available!

Over two years after starting the project, 976 commits, getting involved in long processes (besides the scope we originally envisioned, such as waiting for the translation to be refereed or going over two quite long rounds of copyediting), after leading a team of five translators to Spanish and working closely with a similar team that's quite close to publishing the equivalent translation in Norwegian... Behold!

I won't get again in details on the contents of this book, as I have repeatedly talked about it in the blog. The photo above is of the pages where the CC licensing schemes are presented. And the following is a page I like including in all of my (so far, three) published books:

I have made a point of requiring my university's editorial department to use the legal page to be very explicit regarding the expected usage of this book, by inviting every person that comes across it to copy this book.

So... Where can you get your paws on one of them? Well, of course, you are welcome to come to our institute's bookstore and buy one. For people in general, MX$280 (≈US$15), for UNAM-community, MX$140 (≈US$7.50).

Of course, that's quite inconvenient for people living over 15Km from me, right? What about those living in other countries?

The book can also be downloaded from the Institute's repository. And I will soon upload it to an online, on-demand printing site (probably lulu.com or something like that. Can you suggest one?).

AttachmentSize
Finally: a box full of books80.66 KB
How to use CC licenses?78.48 KB
Copy this book!70.5 KB

06 December, 2019 12:44AM by gwolf

December 05, 2019

Enrico Zini

Temporarily suspending automount on a device

This is part of a series of post on the design and technical steps of creating Himblick, a digital signage box based on the Raspberry Pi 4.

After seeing lots of automatic mount/umount notifications during provisioning, we wondered if it would be possibile to temporarily disable them while we're working on the SD card.

It turns out that it's possible, and here's a convenient python context manager to do it cleanly, based on /usr/lib/udisks2/udisks2-inhibit, but adding the possibility of inhibiting automounting only on one specific device:

    @contextmanager
    def pause_automounting(self, dev: Dict[str, Any]):
        """
        Pause automounting on the device for the duration of this context
        manager
        """
        # See /usr/lib/udisks2/udisks2-inhibit
        devpath = dev["path"]
        rules_dir = "/run/udev/rules.d"
        os.makedirs(rules_dir, exist_ok=True)
        rule_file = os.path.join(rules_dir, "90-udisks-inhibit-" + devpath.replace("/", "_") + ".rules")
        with open(rule_file, "wt") as fd:
            print('SUBSYSTEM=="block", ENV{DEVNAME}=="' + devpath + '*", ENV{UDISKS_IGNORE}="1"', file=fd)
            fd.flush()
            os.fsync(fd.fileno())
        run(["udevadm", "control", "--reload"])
        run(["udevadm", "trigger", "--settle", "--subsystem-match=block"])
        try:
            yield
        finally:
            os.unlink(rule_file)
            run(["udevadm", "control", "--reload"])
            run(["udevadm", "trigger", "--settle", "--subsystem-match=block"])

05 December, 2019 11:00PM

Molly de Blanc

Free software activities (November 2019)

November brings two things very demanding of my time: Thanksgiving and the start of fundraising season.

Free software activities (personal)

  • The Open Source Initiative had it’s twice-a-year face to face board meeting! Good times all around.
  • Debian is having a GR. I’ve been following the development of proposals and conversation, which is basically a part time job in and of itself.
  • Participated in Debian Community Team meetings.
  • I started drafting Bits from the Debian Community Team.
  • Wrote some blog posts! I liked them this month.
  • Wearing multiple hats I attended SustainNYC, talking about sustainability in free and open source software.
  • I submitted to some CFPs — SCaLE, FOSSASIA, and OSCON.
  • I am serving on the papers committee for CopyLeftConf, and for this I reviewed proposals.

Free software activities (professional)

  • We launched a fundraiser! (About a patent infringement case)
  • Funding a legal case is an expensive proposition, so I am also meeting with companies and potential large donors interested in helping out with the case.
  • We launched another fundraiser! (About general Foundation activities)
  • I participated in the hiring process to fill two roles at the GNOME Foundation.

05 December, 2019 03:14PM by mollydb

Jonas Meurer

switch to sway

Switching from Gnome to a tiling window manager

After having thought about it since "forever", I finally decided to switch to a tiling window manager. I went with sway since it runs on wayland and since it seems to be the recommended "wayland version of i3", a tiling window manager that many of my tech friends use ;)

After a few days of using sway, I'm pretty sure that I won't switch back anytime soon. It feels super convenient to have all windows tiled on the screen and being able to rearrange and resize them easily with a few keyboard shortcuts.

There's still some things that didn't work instantly, so I'll try to document them here in hope that it's useful to others. Feedback welcome!

This blog post covers the following topics:

Install sway on Debian Buster

I run Debian Buster on my work machine. The sway components aren't available in Buster or buster-backports yet, so I went with installing the packages from Unstable or experimental manually. I'll probably help with backporting them to buster-backports once I settled on using sway.

Lucky enough, sway packages only bring one dependency that's not satisfied in Buster, which is libjson-c4. So for now, to install the sway Debian packages on Buster, you have to do the following:

mkdir ~/devel/sway && cd ~/devel/sway

wget http://ftp.de.debian.org/debian/pool/main/w/wlroots/libwlroots3_0.7.0-2_amd64.deb
wget http://ftp.de.debian.org/debian/pool/main/s/scdoc/scdoc_1.10.0-1_amd64.deb
wget http://ftp.de.debian.org/debian/pool/main/s/swaybg/swaybg_1.0-2_amd64.deb
wget http://ftp.de.debian.org/debian/pool/main/s/swaylock/swaylock_1.4-1_amd64.deb
wget http://ftp.de.debian.org/debian/pool/main/s/swayidle/swayidle_1.5-1_amd64.deb
wget http://ftp.de.debian.org/debian/pool/main/s/sway/sway-backgrounds_1.2-1_all.deb
wget http://ftp.de.debian.org/debian/pool/main/j/json-c/libjson-c4_0.13.1+dfsg-6_amd64.deb
wget http://ftp.de.debian.org/debian/pool/main/s/sway/sway_1.2-1_amd64.deb

apt install ./libwlroots3_0.7.0-2_amd64.deb ./scdoc_1.10.0-1_amd64.deb ./swaybg_1.0-2_amd64.deb ./swaylock_1.4-1_amd64.deb ./swayidle_1.5-1_amd64.deb ./sway-backgrounds_1.2-1_all.deb ./libjson-c4_0.13.1 ./sway_1.2-1_amd64.deb

apt install dunst i3status suckless-tools

Basic sway configuration

Sway brings a good basic configuration at /etc/sway/config. In order to customize it, copy the file over to ~/.config/sway/config. First things I changed were the following:

# Disable windows title bars
default_borter pixel

# Use tilix wrapper as terminal emulator (more on that later)
set $term ~/.config/sway/scripts/tilix-wrapper.sh

# My internal laptop screen
set $laptop_screen eDP-1

# Command to lock screen
set $lock 'swaylock -F -f -e -K -l -c 000000'

# Default wallpaper
output * bg ~/Pictures/favourite_background.jpg fill

# Idle configuration
exec swayidle -w \
         timeout 300 $lock \
         timeout 600 'swaymsg "output * dpms off"' \
         resume 'swaymsg "output * dpms on"' \
         before-sleep $lock

# Internal Thinkpad Keyboard
input "1:1:AT_Translated_Set_2_keyboard" {
    xkb_layout de,us
    # Change keyboard layouts on <Super>+<Space>
    xkb_options grp:win_space_toggle
}

# Cherry Keyboard
input "1130:275:Cherry_GmbH_CHERRY_Wired_Keyboard" {
    xkb_layout de,us
    # Change keyboard layouts on <Super>+<Space>
    xkb_options grp:win_space_toggle
}

# Internal Thinkpad Touchscreen
input "2:7:SynPS/2_Synaptics_TouchPad" natural_scroll "enabled"

# Status Bar
bar {
    position top
    # Use i3status as status bar
    status_command i3status
}

# Custom key bindings

# Lock screen
bindsym $mod+Escape exec $lock

# Audio and brightness key bindings
bindsym XF86AudioRaiseVolume exec pactl set-sink-volume @DEFAULT_SINK@ +5%
bindsym XF86AudioLowerVolume exec pactl set-sink-volume @DEFAULT_SINK@ -5%
bindsym XF86AudioMute exec pactl set-sink-mute @DEFAULT_SINK@ toggle
bindsym XF86AudioMicMute exec pactl set-source-mute @DEFAULT_SOURCE@ toggle
bindsym XF86MonBrightnessDown exec brightnessctl set 5%-
bindsym XF86MonBrightnessUp exec brightnessctl set +5%
bindsym XF86AudioPlay exec playerctl play-pause
bindsym XF86AudioNext exec playerctl next
bindsym XF86AudioPrev exec playerctl previous

# Bindings for Firefox and Thunderbird
bindsym $mod+Shift+b exec "env MOZ_ENABLE_WAYLAND=1 firefox"
bindsym $mod+Shift+m exec "thunderbird"

# Autostart

# Start dunst, a notification daemon
exec dunst

# Start some programs in fixed worspaces
assign [app_id="firefox"] → 1
exec "env MOZ_ENABLE_WAYLAND=1 firefox"
assign [class="thunderbird"] → 2
exec "thunderbird"

Picking an application launcher

The default application launcher to be used is dmenu (from suckless-tools). While it works okayish, I don't particularly like it. In my eyes, it looks rather old-fashioned, and even worse, it doesn't seem to have support for freedesktop.org desktop entries.

I looked around a bit and wofi sounded pretty promising. It's not in Debian yet but was easy to compile. A big downer though is that it depends on a newer libglib2.0 version (2.60) than in Debian Buster. I still compiled it in a Bullseye schroot and got a first impression. I like it's look and feel (after a bit CSS customization) and probably I'll go with packaging it for Debian.

For the moment, I'm stuck with dmenu on my working system, though.

Configure the status bar

I decided to go with the i3status status bar and it serves my purposes pretty well. Here's my config (/.config/i3status/config):

# i3status configuration file.
# see "man i3status" for documentation.

# It is important that this file is edited as UTF-8.
# The following line should contain a sharp s:
# ß
# If the above line is not correctly displayed, fix your editor first!

general {
        #colors = true
        colors = false
        interval = 5
}

order += "load"
order += "wireless _first_"
order += "ethernet _first_"
order += "path_exists VPN"
order += "battery all"
order += "tztime local"

# Customized wireless status
wireless _first_ {
        format_up = "W: (%quality at %essid) %ip"
        format_down = "W: down"
}

# Only show ethernet status when connected
ethernet _first_ {
        # if you use %speed, i3status requires root privileges
        format_up = "E: %ip"
        format_down = ""
}

# Display VPN status
path_exists VPN {
        # path exists when a VPN tunnel launched by nmcli/nm-applet is active
        path = "/proc/sys/net/ipv4/conf/tun0"
}

# Customized battery status
battery all {
        format = "%status %percentage"
        status_chr = "⚡"
        status_bat = "🔋"
        status_full = "☻"
}

# Localized time format
tztime local {
        #format = "%Y-%m-%d %H:%M:%S"
        format = "%a %d. %b %Y %H:%M"
}

load {
        format = "L: %1min"
}

Configure a notification daemon

I'm really used to getting notifications by my chat programs (XMPP, IRC, Signal), and I don't want to dismiss this. So I installed dunst and configured sway to auto-start it (see above). That's it, it worked instantly. Well, that was easy :)

Preserve working directory in new terminal instances

One thing that really annoyed me after switching to sway was, that the working directory wasn't preserved when spawning new terminal instances. I often open five or more terminal instances in parallel when working on a complex project, and I'm very used to just open a new terminal and continue working in the same directory there immediately.

So I was really eager to find a solution here. Turned out that it's not that easy and needs a bit of dirty scripting, but I found a solution (with help from some nice folks in #sway on Freenode).

First some words about the problem: spawning a new terminal in sway doesn't use whatever sophisticated means to spawn new instances of the same terminal process. Instead, it just spawns a fresh process of your favourite terminal emulator. While I really like tilix and used it as a tiling terminal emulator, I no longer want to use it's tiling features when I now have a tiling window manager. I'll stick for tilix for now as I like its look and feel, though.

So if the new terminal emulator process doesn't know about the working directory of your former terminal, what to do about it?

The solution: Luckily, it's possible to identify the PID of your focused window in sway using swaymsg -t get_tree. In case that the focused window is a terminal emulator, it's parent ID should be your shell. And the shells PWD can easily be determined by reading the symlink /proc/$PID/cwd.

So let's put this in a wrapper script under ~/.config/sway/scripts/tilix-wrapper.sh:

#!/bin/sh

# Small script that tries to determine the PWD of the focused terminal
# (in sway tiling window manager) and pass it to the newly spawned one.

TERMINAL_CMD="tilix --new-process"

FOCUSED_PID=""
if [ ! type jq 2>/dev/null ]; then
    echo "ERROR: jq not installed" >&2
else
    FOCUSED_PID="$(swaymsg -t get_tree | jq '.. | select(.type?) |
        select(.type=="con") | select(.focused==true).pid')"
fi

FOCUSED_PWD=""
# Check if $FOCUSED_PID is an integer
if [ "$FOCUSED_PID" -eq "$FOCUSED_PID" 2>/dev/null ]; then
    FOCUSED_PPID="$(ps -o pid= --ppid "$FOCUSED_PID" | awk '{print $1}')"
    if [ "$FOCUSED_PPID" -eq "$FOCUSED_PPID" 2>/dev/null ]; then
        FOCUSED_PWD="$(readlink "/proc/$FOCUSED_PPID/cwd")"
    fi
fi

# Spawn terminal in background
if [ -d "$FOCUSED_PWD" ]; then
    $TERMINAL_CMD --working-directory="$FOCUSED_PWD" $@ &
else
    $TERMINAL_CMD $@ &
fi

Finally, we have to set the script as $term in sways config (see above). Yay, now I've a solution to preserve my working directory when spawning new terminals!

Use gnome-keyring as SSH agent with sway

Another super annoying thing was that my SSH agent no longer worked with sway, mostly because I used gnome-keyring before and it wasn't spawned automatically when starting sway. So let's change that. I found it a bit complicated to get this working as docs on the internet said a lot of different things, but in the end, the following worked.

Since I still use gdm3 as desktop manager, gnome-keyring-daemon is started automatically during login. So the only thing that's missing is to initalize the gnome-keyring-daemon when starting a terminal. To do so, add the following to ~/.profile (in order to only do it on a login shell):

# Connect to and initalize gnome-keyring-daemon when in sway session
if [ "$DESKTOP_SESSION" = "sway" ]; then
    export $(gnome-keyring-daemon --start)
fi

What's missing

  • I want to start profanity (XMPP client) and irssi (IRC client) automatically in workspace 3, but so far I failed to find a working filter for sways assign feature to identify tilix instances with profanity/irssi (in order to automatically assign those terminals to workspace 3).
  • I miss the redshift feature of gnome 3. redshift itself doesn't support wayland yet. There's a fork with wayland support, but I didn't find time to look into it yet.
  • I'll probably switch from i3status to py3status soon as it's list of modules looks really promising.

05 December, 2019 02:10PM

December 04, 2019

Enrico Zini

himblick media player

This is part of a series of post on the design and technical steps of creating Himblick, a digital signage box based on the Raspberry Pi 4.

Finally, we have enough pieces to start working on the media player. It's been way more work that expected getting to this point, and I hope that this series of posts could help others getting a faster start.

To begin with, we'd like to be able to show:

  • PDF files automatically looping through the pages
  • Image galleries automatically looping
  • Looping videos
  • ODP presentations

Configuring the screen

The first thing to do on startup is to configure the screen based on the himblick.conf settings:

    def configure_screen(self):
        """
        Configure the screen based on himblick.conf
        """
        # Set screen orientation
        orientation = self.settings.general("screen orientation")
        if orientation:
            run(["xrandr", "--orientation", orientation])

        mode = self.settings.general("screen mode")
        if mode:
            res = run(["xrandr", "--query"], capture_output=True, text=True)
            re_output = re.compile(r"^(\S+) connected ")
            for line in res.stdout.splitlines():
                mo = re_output.match(line)
                if mo:
                    output_name = mo.group(1)
                    break
            else:
                output_name = None
            run(["xrandr", "--output", output_name, "--mode", mode])

This had the extra complication of needing to parse xrandr --query output to figure out the name of the HDMI output in use, since the RaspberryPi 4 has two of them. It would be nice if xrandr could join the ranks of tools with a machine parsable output, like for example lsblk is doing.

Scanning the media directory

The next step is finding what to play. We scan the media directory looking at file mimetypes, to avoid having to hardcode all the possible file extension that image and video files can have. Then we group media by type, and pick the group with the most recent files:

    def find_presentation(self, path):
        """
        Find the presentation to play from a given media directory
        """
        if not os.path.isdir(path):
            return None
        pdf = PDFPresentation()
        videos = VideoPresentation()
        images = ImagePresentation()
        odp = ODPPresentation()
        all_players = [pdf, videos, images, odp]

        for fn in os.listdir(path):
            abspath = os.path.abspath(os.path.join(path, fn))
            base, ext = os.path.splitext(fn)
            mimetype = mimetypes.types_map.get(ext)
            if mimetype is None:
                log.info("%s: mime type unknown", fn)
                continue
            else:
                log.info("%s: mime type %s", fn, mimetype)
            if mimetype == "application/pdf":
                pdf.add(abspath)
            elif mimetype.startswith("image/"):
                images.add(abspath)
            elif mimetype.startswith("video/"):
                videos.add(abspath)
            elif mimetype == "application/vnd.oasis.opendocument.presentation":
                odp.add(abspath)

        player = max(all_players, key=lambda x: x.mtime)
        if not player:
            return None
        return player

Caffeinate

We don't want power management to kick in and turn our signage screens black, so we are running all the media players under caffeine.

We automated with with a simple subprocess.run wrapper. Note the -- to prevent caffeinate from choking on the options passed to the actual media players:

def run(cmd: List[str], check: bool = True, **kw) -> subprocess.CompletedProcess:
    """
    Logging wrapper to subprocess.run.

    Also, default check to True.
    """
    log.info("Run %s", " ".join(shlex.quote(x) for x in cmd))
    return subprocess.run(cmd, check=check, **kw)


class Presentation:
    """
    Base class for all presentation types
    """
    def run_player(self, cmd, **kw):
        """
        Run a media player command line, performing other common actions if
        needed
        """
        # Run things under caffeinate
        # See also: https://stackoverflow.com/questions/10885337/inhibit-screensaver-with-python
        cmd = ["caffeinate", "--"] + cmd
        run(cmd, **kw)

Showing PDFs

okular seems to be the only PDF reader in Debian that can be convinced to do looping non interactive full screen presentations, with only a bit of tampering with its configuration files:

class PDFPresentation(SingleFileMixin, Presentation):
    def run(self):
        log.info("%s: PDF presentation", self.fname)

        confdir = os.path.expanduser("~/.config")
        os.makedirs(confdir, exist_ok=True)

        # TODO: configure slide advance time

        # Configure okular
        with open(os.path.expanduser(os.path.join(confdir, "okularpartrc")), "wt") as fd:
            print("[Core Presentation]", file=fd)
            print("SlidesAdvance=true", file=fd)
            print("SlidesAdvanceTime=2", file=fd)
            print("SlidesLoop=true", file=fd)
            print("[Dlg Presentation]", file=fd)
            print("SlidesShowProgress=false", file=fd)
            # print("SlidesTransition=GlitterRight", file=fd)

    # Silence a too-helpful first-time-run informational message
        with open(os.path.expanduser(os.path.join(confdir, "okular.kmessagebox")), "wt") as fd:
            print("[General]", file=fd)
            print("presentationInfo=4", file=fd)

        # Remove state of previous okular runs, so presentations begin at the
        # beginning
        docdata = os.path.expanduser("~/.local/share/okular/docdata/")
        if os.path.isdir(docdata):
            shutil.rmtree(docdata)

        self.run_player(["okular", "--presentation", "--", self.fname])

I was surprised at how looping a PDF presentation doesn't seem to be a well supported use case in PDF viewers. If it's somewhat painful to do it in okular, it's downright impossible to do it with evince: try evince --fullscreen --presentation: slides won't advance, and it still shows a toolbar!

Showing images

class ImagePresentation(FileGroupMixin, Presentation):
    def run(self):
        self.files.sort()
        log.info("Image presentation of %d images", len(self.files))
        with tempfile.NamedTemporaryFile("wt") as tf:
            for fname in self.files:
                print(fname, file=tf)
            tf.flush()

            # TODO: adjust slide advance time
        self.run_player(["feh", "--filelist", tf.name, "--fullscreen",
                         "--hide-pointer", "--slideshow-delay", "1.5"])

feh does everything needed and more. It seems to support our use case explictly, with useful knobs exposed on the command line, clean, straightforward, beautiful!

Showing videos

Most internet posts about playing media on Raspberry Pi, suggest omxplayer. After trying it it looked quite worrysome, as it seemed to fail with any media format not supported in hardware, there did not seem to be a way to ask it whether a file would be in a playable format or not, and one of the failures left the screen in the wrong resolution.

We would like the media player to be able to play the widest possible range of media, hardware accelerated if possible, software if not.

Luckily, it turned out that vlc can use the Raspberry Pi 4 hardware acceleration, and playing a 1920x1080 video full screen on it would consume only 4% of CPU, which is the same that omxplayer was using.

That was very relieving, as vlc can also play a wide range of media, has excellent support for gapless looping, can be invoked without a UI, and can even do playlists of multiple media.

Here is the corresponding player code:

class VideoPresentation(FileGroupMixin, Presentation):
    def run(self):
        self.files.sort()
        log.info("Video presentation of %d videos", len(self.files))
        with tempfile.NamedTemporaryFile("wt", suffix=".vlc") as tf:
            for fname in self.files:
                print(fname, file=tf)
            tf.flush()

            self.run_player(
                    ["cvlc", "--no-audio", "--loop", "--fullscreen",
                        "--video-on-top", "--no-video-title-show", tf.name])

Showing presentations

The code here is quite straightforward, but it took a while to put together that command line:

class ODPPresentation(SingleFileMixin, Presentation):
    def run(self):
        log.info("%s: ODP presentation", self.fname)
        self.run_player(["loimpress", "--nodefault", "--norestore", "--nologo", "--nolockcheck", "--show", self.fname])

I was surprised that I could not find a way to tell Impress to just play a presentation without other things getting in the way. Even like that, there is a moment in which the UI can be seen to come up on the screen before being covered by the full screen presentation.

There is also no way to force a presentation to loop or to advance slides after a given timeout: both features needs to be set in the presentation itself.

People will have to do a test run of their presentations with a checklist before putting them on the player. It would have been nice to have an easy way to guarantee that a presentation wouldn't get stuck on the player.

That is not a requirement for now anyway. If it ever becomes one, I guess we can always write code to check and tweak the .odp presentation file: it's thankfully a well known and open format.

Auditing recommends

So far we installed everything with --no-install-recommends, but it's risky to do so when dealing with packages with many dependencies like vlc, okular, and impress.

Aptitude offers the possibility to audit recommends: Views / Audit Recommendations will show a list of recommended but not installed packages.

That turned out some font packages that it's maybe nice to have, and libreoffice-avmedia-backend-vlc that may come in handy if people decide to play presentations with embedded videos.

Existing presentation software

Our needs for media playing so far have been simple. Should they become more complex, here are some pointers to existing, more featureful projects:

04 December, 2019 11:00PM

hackergotchi for Vincent Bernat

Vincent Bernat

Self-hosted videos with HLS: subtitles

In a previous article, I have described a solution to self-host videos while offering a delivery adapted to each user’s bandwith, thanks to HLS and hls.js. Subtitles1 were not part of the game. While they can be declared inside the HLS manifest or embedded into the video, it is easier to include them directly in the <video> element, using the WebVTT format:

<video poster="poster.jpg"
       controls preload="none">
  <source src="index.m3u8"
          type="application/vnd.apple.mpegurl">
  <source src="progressive.mp4"
          type='video/mp4; codecs="avc1.4d401f, mp4a.40.2"'>
  <track src="de.vtt"
         kind="subtitles" srclang="de" label="Deutsch">
  <track src="en.vtt"
         kind="subtitles" srclang="en" label="English">
</video>

Watch the following demonstration, featuring Agent 327: Operation Barbershop, a video created by Blender Animation Studio and currently released under the Creative Commons Attribution No Derivatives 2.0 license:

You may want to jump to 0:12 for the first subtitle. Most browsers should display a widget to toggle subtitles. This works just fine with Chromium but Firefox will not show the menu until the video starts playing, unless you enable preloading. Another annoyance: there is no simple way to specify safe margins for subtitles and they get stuck at the bottom. These two issues seem minor enough to not warrant pulling hundred of kilobytes of JavaScript for a custom player.

Update (2019.12)

This does not seem to work with Firefox 68 on Android. The browser makes no attempt to download the selected subtitle.

  1. Some people may be picky over the difference between closed captions and subtitles. Closed captions are usually targeted at people with hearing impairment and they include non-speech information like sound effects. Subtitles assume the viewer can hear but may not understand the language. ↩︎

04 December, 2019 08:13AM by Vincent Bernat

Dima Kogan

tee is broken?

Just found a highly surprising behavior in a core tool I've used for decades, so clearly I'm making a note here. None of these are surprising: 

$ seq 1000 | wc -l

1000


$ seq 1000 | tee /dev/null | wc -l

1000


$ seq 1000 | tee >( true ) | wc -l

1000


$ seq 1000 > >( true ) | wc -l

1000

I.e. I can write 1000 lines into tee, do stuff in one of the children, and the other child get my 1000 lines still. The last one uses multios in zsh for the tee. But check out what happens when I bump up the data size: 

$ seq 100000 | wc -l

100000


$ seq 100000 | tee /dev/null | wc -l

100000


$ seq 100000 | tee >( true ) | wc -l

14139


$ seq 100000 > >( true ) | wc -l

1039

Whoa. What the hell? When I stumbled on this I had another, unrelated problem breaking things in this area, which made for a long debugging session. Here're some runs that give a hint of what's going on: 

$ seq 100000 | tee >( true ) | wc -c

73728


$ seq 100000 > >( true ) | wc -c

4092


$ seq 100000 | tee >( cat > /dev/null ) | wc -l

100000

Figure it out?

Answer time! After a tee, a single writer parent feeds two reader children. If a child exits before reading all the data, then when the parent tries to feed that dead child, the parent will get a SIGPIPE. And apparently the default behavior of tee in GNU coreutils (and in the zsh multios redirection) is to give up and to stop feeding all the children at that point. So the second child (wc -l in the examples) ends up with incomplete input. No errors are thrown anywhere, and there's no indication at all that any data was truncated. Lots of the data is just silently missing.

The GNU coreutils implementation of tee has an innocuous-looking option: 

-p     diagnose errors writing to non pipes

I read the manpage several times, and it's still not obvious to me that -p does anything more than change something about diagnostic printing. But it does: tee -p feeds all the children as much as it can until they're all dead (i.e. what everybody was assuming it was doing the whole time): 

$ seq 100000 | tee -p >( true ) | wc -l

100000

There's also pee, specific tee-to-process utility in the Debian moreutils package. This utility can be used here, and it does the reasonable thing by default: 

$ seq 100000 | pee true 'wc -l'

100000

So yeah. I'm not the first person to discover this, but I'm certain this was quite surprising to each of us.

04 December, 2019 06:31AM by Dima Kogan

December 03, 2019

Enrico Zini

X autologin

This is part of a series of post on the design and technical steps of creating Himblick, a digital signage box based on the Raspberry Pi 4.

When powered on, the Pi units should go straight into X, and start the media player.

X autologin has long been a gripe of mine, and surprisingly hard to do right and reliably, integrating well with PAM, with no greeters or switches to text mode flashing quickly on the screen, and so on.

I once wrote nodm. Then lightdm-autologin-greeter, so I didn't have to maintain a whole display manager for this. Lightdm's autologin used to be insufficient, because when the X session ended, lightdm assumed one wanted to log out and showed the greeter again.

Now I'm very pleased to see that, in 2019, almost 2020, just setting autologin-user in lightdm does the right thing, and I'm very happy to be able to remove some of the kludges from the geological strata of kludges that accreted over the years to work around these kinds of things.

First thing first, do not wait for network to be online during boot. We don't need it in a media player that should be able to work just as well offline:

       # Do not wait for being online to finish boot
       chroot.systemctl_disable("systemd-networkd-wait-online.service", mask=True)

Setting up autologin now really is quite beautifully straightforward:

 - name: "Install X dependencies"
   apt:
      pkg:
      # basic X server
       - xserver-xorg
       - lightdm
      state: present
      update_cache: no
      install_recommends: no

 - name: "Enable lightdm autologin"
   lineinfile:
      path: /etc/lightdm/lightdm.conf
      regexp: ".*autologin-user\\s*=\\s*($|pi)"
      line: "autologin-user = pi"

A nice trick for a media display: X without a mouse cursor accidentally hovering like a fly over things:

 - name: "Disable mouse cursor in X"
   lineinfile:
      path: /etc/lightdm/lightdm.conf
      regexp: "(^|#)xserver-command\\s*=\\s*"
      line: "xserver-command = X -nocursor"

Finally, we need to start the player in the X session.

There are quite a lot of ways to autostart things in X sessions, and the Xsession page in the Debian Wiki has excellent documentation on it.

Since the machine will only be a media player, as a first attempt we decided to try and be the X session, starting the player directly with no desktop environment, no window manager, nothing, ohne alles:

 - name: "Use himblick player as X session"
   copy:
      dest: /home/pi/.xsession
      owner: pi
      group: pi
      mode: '0644'
      content: |
          exec /usr/bin/himblick player

This way, the player can stay there and do its thing without anything popping up in its way, and if it dies, X ends, and lightdm restarts it all.

03 December, 2019 11:00PM

hackergotchi for Norbert Preining

Norbert Preining

Debian breaking Unison (again)

Congratulations – Debian/sid now contains a unison binary that is incompatible with Debian/buster, the stable release. That means, everyone who relies on unison for file synchronization across servers (running buster) and development machines (running sid) is now busted. Trying to use the new binary from sid on buster also doesn’t work, due to GLIBC incompatibility.

For now the only solution I see is using the versions from Debian/buster and hold them, never to be upgraded for the next 2 years. At least that worked for me.

And BTW, the warning message that should be in NEWS(.Debian) didn’t make it into the binary builds …

03 December, 2019 09:59PM by Norbert Preining

hackergotchi for Ben Hutchings

Ben Hutchings

Debian LTS work, November 2019

I was assigned 24.5 hours of work by Freexian's Debian LTS initiative and carried over 0.5 hours from October. I worked 21.25 hours this month, so will carry over 3.75 hours to December.

I released Linux 3.16.76, rebased the Debian package onto that, and sent out a request for testing.

I backported the mitigation for TSX Asynchronous Abort (CVE-2019-11135) and reporting of iTLB multihit (CVE-2018-12207) to 3.16 (this work started in October). I applied these and a GPU security fix, uploaded the Debian package and issued DLA-1989-1.

I backported the latest security update for Linux 4.9 from stretch to jessie and issued DLA-1990-1 for that.

I prepared and, after, review, released Linux 3.16.77 and 3.16.78. I rebased the Debian package onto 3.16.78 and sent out a request for testing.

03 December, 2019 06:16PM

December 02, 2019

Enrico Zini

ssh setup

This is part of a series of post on the design and technical steps of creating Himblick, a digital signage box based on the Raspberry Pi 4.

Time to setup ssh. We want to have admin access to the pi user, and we'd like to have a broader access to a different, locked down user, to use to manage media on the boxes via sftp.

The first step is to mount the exFAT media partition into /srv/media:

---
 - name: "Install exfat drivers"
   apt:
      name: exfat-fuse,exfat-utils
      state: present
      update_cache: no

 - name: "Create /srv directory"
   file:
      path: "/srv"
      state: directory
      owner: root
      group: root
      mode: 0755

 - name: "Create /media mount point"
   file:
      path: "/srv/media"
      state: directory
      owner: pi
      group: pi
      mode: 0755

 - name: "Configure mounting media directory"
   copy:
      src: srv-media.mount
      dest: /etc/systemd/system/srv-media.mount
      owner: root
      group: root
      mode: 0644

Mounting exFAT before Linux kernel 5.4 requires FUSE. Using a mount unit allows us to bring up the mount after FUSE is up, and get it mounted at boot reliably.

We add a round of filesystem checking, too: if people plug the SD into a computer to load media into it, we can't be sure that they unmount it cleanly.

This is srv-media.mount; note that .mount unit names need to match the path of the mount point:

[Unit]
Description=Media Directory (/srv/media)
Before=local-fs.target
After=sys-fs-fuse-connections.mount

[Mount]
What=/dev/disk/by-label/media
Where=/srv/media
Type=exfat
Options=uid=0,gid=1001,fmask=117,dmask=007,rw,noatime,nosuid,nodev
ExecStartPre=-/sbin/fsck.exfat -a /dev/disk/by-label/media

[Install]
WantedBy=local-fs.target

gid 1001 is the media group id, shared by the pi user that runs the media player, and by the media user that does sftp. We make everything the media mount group-writable by the media user so both users can access it.

Next, we prepare a chroot jail for the media user. The root of the jail needs to be writable only by root, so we bind mount the media directory inside it:

 - name: "Create the chroot jail for media: /srv"
   file:
      path: "/srv"
      state: directory
      owner: root
      group: root
      mode: 0755

 - name: "Create the chroot jail for media: /srv/jail"
   file:
      path: "/srv/jail"
      state: directory
      owner: root
      group: root
      mode: 0755

 - name: "Create the chroot jail for media: /srv/jail/media"
   file:
      path: "/srv/jail/media"
      state: directory
      owner: root
      group: media
      mode: 0755

 - name: "Bind mount /srv/media under /srv/jail/media"
   copy:
      src: srv-jail-media.mount
      dest: /etc/systemd/system/srv-jail-media.mount
      owner: root
      group: root
      mode: 0644

This is the srv-jail-media.mount mount unit, neatly ordered to start after /srv/media is mounted:

[Unit]
Description=Media Directory in sftp jail (/srv/jail/media)
Before=local-fs.target
After=srv-media.target

[Mount]
What=/srv/media
Where=/srv/jail/media
Type=none
Options=bind

[Install]
WantedBy=local-fs.target

Finally, the ssh configuration:

---
 - name: "Disable ssh password authentication"
   lineinfile:
      path: /etc/ssh/sshd_config
      regexp: '\bPasswordAuthentication\b'
      line: 'PasswordAuthentication no'

 - name: "Install ssh admin access key"
   authorized_key:
      user: pi
      state: present
      key: "{{SSH_AUTHORIZED_KEY}}"
   when: SSH_AUTHORIZED_KEY is defined

 - name: "Install ssh media access key"
   authorized_key:
      user: media
      state: present
      key: "{{SSH_MEDIA_PUBLIC_KEY}}"
   when: SSH_MEDIA_PUBLIC_KEY is defined

 - name: "Install media access key for the pi user"
   copy:
      dest: "/home/pi/.ssh/id_media"
      content: "{{SSH_MEDIA_PRIVATE_KEY}}"
      owner: pi
      group: pi
      mode: 0600
   when: SSH_MEDIA_PRIVATE_KEY is defined

 - name: "Configure internal sftp, so ssh does not need binaries inside the jail"
   lineinfile:
      path: /etc/ssh/sshd_config
      regexp: ".*Subsystem\\s+sftp"
      line: "Subsystem sftp internal-sftp"

 - name: "Configure sftp chroot jail for user media"
   blockinfile:
      path: /etc/ssh/sshd_config
      block: |
         Match User media
              ChrootDirectory /srv/jail
              AllowTcpForwarding no
              X11Forwarding no
              ForceCommand internal-sftp

Don't forget to enable the media units:

       # Enable the /srv/media mount point, which ansible, as we run it
       # now, is unable to do
       chroot.systemctl_enable("srv-media.mount")
       chroot.systemctl_enable("srv-jail-media.mount")

02 December, 2019 11:00PM

Jelmer Vernooij

The Debian Janitor

There are a lot of small changes that can be made to the Debian archive to increase the overall quality. Many of these changes are small and have just minor benefits if they are applied to just a single package. Lintian encourages maintainers to fix these problems by pointing out the common ones.

Most of these issues are often trivially fixable; they are in general an inefficient use of human time, and it takes a lot of effort to keep up with. This is something that can clearly be automated.

Several tools (e.g. onovy's mass tool, and the lintian-brush tool that I've been working on) go a step further and (for a subset of the issues reported by lintian) fix the problems for you, where they can. Lintian-brush can currently fix most instances of close to 100 lintian tags.

Thanks to the Vcs-* fields set by many packages and the APIs provided by hosting platforms like Salsa, it is now possible to proactively attempt to fix these issues.

The Debian Janitor is a tool that will run lintian-brush across the entire archive, and propose fixes to lintian issues via pull request.

Objectives

The aim of Debian Janitor is to take some drudge work away from Debian maintainers where possible, so they can spend their time on more important packaging work. Its purpose is to make automated changes quick and easy to apply, with minimal overhead for package maintainers. It is essentially a bit of infrastructure to run lintian-brush across all of the archive.

The actions of the bot are restricted to a limited set of problems for which obviously correct actions can be taken. It is not meant to automate all packaging, or even to cover automating all instances of the issues it knows about.

The bot is designed to be conservative and delight with consistently correct fixes instead of proposing possibly incorrect fixes and hoping for the best. Considerable effort has been made to avoid the janitor creating pull requests with incorrect changes, as these take valuable time away from maintainers, the package doesn't actually improve (since the merge request is rejected) and it makes it likelier that future pull requests from the Debian Janitor bot are ignored or rejected.

In short: The janitor is meant to propose correct changes if it can, and back off otherwise.

Design

The Janitor finds package sources in version control systems from the Vcs*- control field in Debian source packages. If the packaging branch is hosted on a hosting platform that the Janitor has a presence on, it will attempt to run lintian-brush on the packaging branch and (if there are any changes made) build the package and propose a merge. It is based on silver-platter and currently has support for:

The Janitor is driven from the lintian and vcswatch tables in UDD. It queries for packages that are affected by any of the lintian tags that lintian-brush has a fixer script for. This way it can limit the number of repositories it has to process.

Ensuring quality

There are a couple of things I am doing to make sure that the Debian Janitor delights rather than annoys.

High quality changes

Lintian-brush has end-to-end tests for its fixers.

In order to make sure that merge requests are useful and high-value, the bot will only propose changes from lintian-brush that:

  • successfully build in a chroot and pass autopkgtest and piuparts;
  • are not completely trivial - e.g. only stripping whitespace

Changes for a package will also be reviewed by a human before they make it into a pull request.

One open pull request per package

If the bot created a pull request previously, it will attempt to update the current request by adding new commits (and updating the pull request description). It will remove and fix the branch when the pull request conflicts because of new upstream changes.

In other words, it will only create a single pull request per package and will attempt to keep that pull request up to date.

Gradual rollout

I'm slowly adding interested maintainers to receiving pull requests, before opening it up to the entire archive. This should help catch any widespread issues early.

Providing control

The bot will be upfront about its pull requests and try to avoid overwhelming maintainers with pull requests by:

  • Clearly identifying any merge requests it creates as being made by a bot. This should allow maintainers to prioritize contributions from humans.
  • Limiting the number of open proposals per maintainer. It starts by opening a single merge request and won't open additional merge requests until the first proposal has a response
  • Providing a way to opt out of future merge requests; just a reply on the merge request is sufficient.

Any comments on merge requests will also still be reviewed by a human.

Current state

Debian janitor is running, generating changes and already creating merge requests (albeit under close review). Some examples of merge requests it has created:

Using the janitor

The janitor can process any package that’s maintained in Git and has its Vcs-Git header set correctly (you can use vcswatch to check this).

If you're interested in receiving pull requests early, leave a comment below. Eventually, the janitor should get to all packages, though it may take a while with the current number of source packages in the archive.

By default, salsa does not send notifications when a new merge request for one of the repositories you're a maintainer for is created. Make sure you have notifications enabled in your Salsa profile, by ticking "New Merge Requests" for the packages you care about.

You can also see the number of open merge requests for a package repository on QA - it's the ! followed by a number in the pull request column.

It is also possible to download the diff for a particular package (if it's been generated) ahead of the janitor publishing it:

$ curl https://janitor.debian.net/api/lintian-fixes/pkg/PACKAGE/diff

E.g. for i3-wm, look at https://janitor.debian.net/api/lintian-fixes/pkg/i3-wm/diff.

Future Plans

The current set of supported hosting platforms covers the bulk of packages in Debian that is maintained in a VCS. The only other 100+ package platform that's unsupported is dgit. If you have suggestions on how best to submit git changes to dgit repositories (BTS bugs with patches? or would that be too much overhead?), let me know.

The next platform that is currently missing is bitbucket, but there are only about 15 packages in unstable hosted there.

At the moment, lintian-brush can fix close to 100 lintian tags. It would be great to add fixers for more common issues.

The janitor should probably be more tightly integrated with other pieces of Debian infrastructure, e.g. Jenkins for running jobs or linked to from the tracker or lintian.debian.org.

More information

See the FAQ on the homepage.

If you have any concerns about these roll-out plans, have other ideas or questions, please let me know in the comments.

02 December, 2019 09:45PM by Jelmer Vernooij

hackergotchi for Julien Danjou

Julien Danjou

Properly managing your .gitignore file

Properly managing your .gitignore file

There's not a single month where I don't have to explain this. I thought it'd be a good opportunity to write about this .gitignore file so everyone is up to date on this magic file.

The purpose of .gitignore

The .gitignore file is meant to be a list of files that Git should not track. It resides at the root directory of your repository. It can be a list of file path relative to the repository, or a list of wildcard. The file format and location is fully documented in Git documentation.

For example, this is a valid content for a .gitignore file:

foo
bar/*

When you're using Git commands such as git add, all the files matching what's listed in .gitignore are ignored. That makes sure you don't commit a file that should not be there by mistake. In the example above, any file in the bar directory or any file named foo will be completely ignored by all Git commands.

Awesome!

What's the problem with it?

Soon, developers realize that their directory is cluttered with temporary files. It might be from their build system, their editors or some test files they wrote.

So what do they do? They add those files to .gitignore for their project. You end up with a .gitignore file that contains entries like:

*~
.vscode
*.DS_Store
.idea

With that, you're sure to ignore backup files from vim, folders from MacOS and temporary files from Visual Studio code, etc.

Properly managing your .gitignore file

Don't do this. Not everybody uses your editor or favorite pet tool, and nobody cares. The repository you're working in is shared with a lot of others developers. Sending pull requests to just add this kind of entry to ignore files generated by your pet editor is wrong and annoying.

Wait, how do I ignore my editor files then?

If you read through Git documentation, the answer lies there: Git has a global ignore file that works for EVERY repository on your system. No need to hack each repository. By default, it's in ~/.config/git/ignore. Here's mine:

.#*
*.swp
.DS_Store
.dir-locals.el
.dir-locals-2.el

That's enough to ignore my editors and OS files in all my repositories so I don't git add wrong files by mistake. You can tweak this global file location by changing by tweaking core.excludesFile in your Git configuration.

So what should I put in .gitignore?

You should put in .gitignore all files and patterns that are generated by the build system of your project, or any file that it might output while running.

For example, for a Python project, it's common to have this:

*.pyc
__pycache__

With this, it makes sure that nobody is committing compiled Python files.

Properly managing your .gitignore file

Thanks for reading through this. I hope you'll write better .gitignore files in the future. 🤞

02 December, 2019 01:09PM by Julien Danjou

December 01, 2019

hackergotchi for Shirish Agarwal

Shirish Agarwal

No country for women – rehashed

This is not the first time I am writing about it, sadly I had written it twice before in the preceding months. But the scenario instead of changing for the better is changing for the worse. Sharing a screenshot shared by somebody today on social-media.

Indian Express page inside page screenshot Page 9, Decamber 1, 2019 Reportage of Rape only.

The most recent gruesome addition to the list was that of a 26 year-old veterinarian. From the various news reports it was a pre-planned rape and murder of the woman. What was startling to me when I read the report that it seems people still either don’t know or police officers who should know and inform people do not inform people of their rights such as zero FIR . Maybe some RTI activist can find out how many people have used zero FIR or not and make efforts to popularize the concept and its utility. If people are courageous enough, they can even live-shoot video when asking a police officer or constable to file a zero FIR . The other part is I do hope the police do investigate and do the DNA matching so they have conclusive proof other than the statements which the rapists may later claim as being under duress. The views of police is almost Victorian in nature irrespective of whether it’s an officer or a lower hawaldar/constable on the beat.

While there have been many attempts to try and figure out solutions, it somehow always falls short. There was the ‘Occupy streets by women’ movement in 2017 and so many small and major attempts but almost all have fallen short. Simply put, neither the society, nor the police or the judiciary seems to have any answers. I probably had shared about India’s Daughter , a documentary which shared an insight into the rapist’s mindset. Except for the juvenile, all the other rapists who had confessed to the crime shared their opinion on why the rape happened and most of it was victim-blaming or victim-shaming. The producer Leslee Udwin at the time she made and released the documentary shared that she herself had been raped. The statements made by various BJP leaders as well as some from the opposition were so in favor of the men who had committed the rape were such that they had to finally ban the documentary as lot of hypocricy was being unveiled.

From 2012, till date there probably would be a dozen or more apps. which are designed to help somehow never work. There have been few women cab services, driven by women for women but that too has not scaled. The couple I know of are kaola kabs based out of Bangalore and women cabs in Delhi . The problem with Kaola is that they don’t have night services when most of the rapes occur and when they are needed, about womencabs maybe somebody who may have used the services could perhaps share their their experience. There have been many experiments, right from 2012 onwards but more often than not, most have been more or less a failure. The problems stem from many systemic and societal problems. The problem is not just of women but men too. Places which are unsafe for women are unsafe for all vulnerable groups, whether it is children, the elderly and anybody else who is not a bully. This simple understanding somehow fails to connect or be understood by people. We had this happen everywhere, whether in Bollywood or whether they are women cops or the judiciary. The situation for women is the same everywhere. And before somebody accuses me of India-bashing even though I live here. We know the same happens in all countries, surprisingly more in European countries where I expected human rights to be a savior but it seems they have their own fights to win. I guess if you are a predator you can find your prey no matter what your race or orientation is as shared by a victim recently.

Sex Education

While we can’t take the excuse of Sex education, the short video does illustrate the views of most people. If they don’t want to be responsible about children’s understanding and influence about sex especially in the nuclear families most are, then who will ?

https://www.invidio.us/watch?v=feQocm2TSBc

I had to look back at how I had helped Dr. Shome in her book where we had gone back and forth on her book on sex education titled ‘Let’s talk about guys, girls and sex’ which is there on Amazon, Kindle and elsewhere. My discussions as well as discussions with others, she had shared when last we met that she had material for at least another 2 books if she decided to go that way. The problem even with that such books do not make for casual reading and nor would either most parents or schools will attempt to have discussions around a book like that.

Mobile application for women safety

There are attempts to make a mobile application but I don’t see how any mobile application can help when it’s society’s issues at large. I am guessing funding is also going to be a big issue as well as transportation of people from place X to Y itself carries a cost other than costs and expenses incurred at various places.

Ancedote

I can share an anecdotal tale I heard from a gentleman couple of years ago, and like many stories or incidents this one also didn’t make the newspaper. A few years ago, I had gone to a friend’s place who was staying at Baner, at a somewhat secluded place. We used to meet for talking about FOSS, sharing foss, eating, drinking and sleeping at his place at night and next morning having a chai before using public transport to come back. We used to do this once a month or once two months till he shifted to Bangalore. Anyways, one night after partying I didn’t sleep well and went outside to the bus station quite early, around 0600 hrs or something like that. There used to be quite a few cycle rickshaw shops ( similar to food carts that you see in the west but somewhat more rundown.) The thing about Baner and the area I was in, this was factory area so most factories have 3-4 shifts so most of these food carts (called thelas in the vernacular) make brisk business. Unlike other times, instead of 3-4 food carts, I saw only one and also a bit of eerie silence. After a bit of talking, cajoling the owner, came to know that a naked woman had come and created a scene and other thela walas had run away. Whether she was mentally unstable or had she been raped was not known. Even the police jeep for some reason didn’t want to take her as they suspected they may blame the policeman. Even he had run away but as he was old he couldn’t tow the food cart so he simply ran away because in his twilight he didn’t want to get into any such complication.

The reason I shared the above is, in such a scenario how are men supposed to act and react, I don’t know. Whether it’s just fight or flight or some other behavioral patterns that should be incultated by men in such a scenario ?

Anthillhacks

FWIW, there is anthillhacks happening in Bangalore. For people interested in hacking or doing something like above, this may be a good chance to connect with like-minded people and have some sort of rough prototype out or at least hash out much more face-to-face rather than using the web. I had been a part of this just as have been part of similar things in the past and my experience has been swell. Unfortunately travel budget is an issue this year otherwise for sure would have been part of it. Hopefully people enjoy and also blog about their experiences while staying there.

01 December, 2019 09:20PM by shirishag75

Thorsten Alteholz

Debian-Med Bug Squashing

As it is again this time of the year, I would also like to draw some attention to the Debian Med Advent Calendar. Like the past years, the Debian Med team starts a bug squashing event from the December 1st to 24th. Every bug that is closed will be registered in the calendar. So instead of taking something from the calendar, this special one will be filled and at Christmas hopefully every Debian Med related bug is closed. Don’t hestitate, start to squash :-).

The announcement on the mailing list can be found here.

01 December, 2019 07:08PM by alteholz

Sam Hartman

Voting Guide for Debian Init Systems GR

There are a lot of options on the ballot for the Init Systems GR.
There have been hundreds of messages on debian-vote, and more scattered
across debian-devel, debian-project, and bugs. Reading all that is no
easy task. so I would like to summarize my understanding of the options
and what they would mean. I've tried to remove as much bias as I can,
but this is still Sam's personal opinion.



I'm focused entirely on the effects of the proposals. Several options
(D, F, and G) spend significant time outlining principles. That is
something I'm ignoring here in this post, although I acknowledge it is
really important to some.



Areas of Agreement



One of the big surprises for me in this discussion is things that are
true of all the ballot options so far. We are agreed that programs
designed to use systemd features that only work with systemd are welcome
in Debian. That's true even if there is no way for these programs to
work without systemd. Under Proposal D, this is a bug, but not a
release-critical bug.



Init Diversity Options


Several options focus on encouraging or requiring packages to support
init systems others than systemd when that is possible. These include
Proposal
E, Proposal F,
and Proposal
A. Under proposal E, it is a release critical bug if a program does
not work when something other than systemd is pid 1 unless that program
is designed explicitly to work with systemd and no support for running
without systemd is available. Lack of an init script alone is not
sufficient to count as designed to work exclusively with systemd.



So, under this proposal, a maintainer must integrate support for running
without systemd if it is available. They are responsible for going out
and finding this support. If the support is as simple as writing an
init script, the maintainer has an RC bug until they write the init
script. If the support is more complex, the maintainer is not
responsible for writing it. Proposal A is the same as Proposal E,
except that the bug is not release-critical. I'll go into Proposal A in
more detail after discussing Proposal D.



Proposal D is similar to Proposal E. My interpretation is that Proposal D places
somewhat less of a burden on maintainers to go out and find existing
non-systemd support. My interpretation is that the bug becomes RC when
someone contributes that support. (Or if the support is present in the
upstream but turned off in the package). Proposal D requires that
non-systemd support not have a substantial effect on systemd
installations. So where as Proposal E uses the designed exclusively for
systemd criteria, Proposal D uses the no substantial effect on systemd
systems criteria to determine whether working only with systemd is
acceptable. The discussions seemed to imply that if Gnome uses systemd
features in excess of what technologies like elogind can handle, it is
likely to meet both criteria.



Proposal D goes into a lot more detail than Proposal E. Proposal E
would likely be a long-term block on using systemd facilities like
sysusers. Proposal D specifically proposes a mechanism whereby such
facilities can be documented in policy. This mechanism is only
available if it is likely that developers of non-systemd (including
non-Linux) systems will implement the facility. After a six-to-twelve
month transition time, the facility can be used even on non-systemd
systems. So, sufficiently low effort in the non-systemd community that
it is unreasonable to expect a facility could be implemented could still
permanently block adoption of such facilities. Proposal D is definitely
about a long-term commitment to non-systemd systems even if the effort
in the non-systemd community is not as high as we'd like to adopt new
features elsewhere.



Proposal D also includes a number of guidelines for proper behavior
around these emotionally charged issues.



The only difference between Proposal E and Proposal A is the severity of
the bug when non-systemd support is not in a package. In Proposal A,
this bug is important: potentially sufficient for a stable update, but
not release critical. As a practical matter, Proposal A allows the
non-systemd community to contribute (and NMU) patches for non-systemd
support. However, it does not place an obligation on maintainers to
write this support themselves. Proposal A would permit systemd
facilities like sysusers to be used, although doing so might be a bug.
In the specific case of sysusers, someone could justify NMUing a patch
to use adduser in a maintainer script. Unlike Proposal D, Proposal A
places the burden of keeping up with systemd facilities fully on the
non-systemd community. Proposal A does not have Proposal D's
requirement that it be reasonable to expect that the non-systemd
community can implement the facility.



Systemd Options


There are two systemd options: Proposal F
and Proposal
B. Proposal F replaces a previous Proposal C. As far as I can tell
Proposal F and C do the same thing, but Proposal F has some text
describing principles. As I said in the introduction, I'm not
discussing that here.



Under Proposal F, systemd is the only officially supported option.
Other init systems may be explored at wishlist priority. Systemd
facilities such as sysusers are encouraged, and we will use our usual
mechanisms to plan transitions from Debian facilities where appropriate.



Proposal B does not explicitly say that alternate init system work can
only be wishlist. Under Proposal B, I think it would be reasonable to
file some bugs at normal severity, but it would also be reasonable for a
maintainer to downgrade them. I don't consider that a significant
difference.



The big difference is that Proposal B commits us as a project to
reviewing integrations of technologies that are alternatives to
systemd facilities. The current example is elogind. But things like
a non-systemd implementation of sysusers, tmpfiles.d, etc, would also
qualify. The rationale is that sometimes alternatives like that touch
on core infrastructure, and even if other maintainers are doing the
work, gatekeeper review is needed. Under Proposal B, the alternate technologies would be available, but whether to use them in a specific package would be up to the maintainer. I've discussed in another, more opinionated blog post why this might be a good idea.



Proposal G


As best I can tell Proposal G is just a set of principles. so, in the context of the analysis I have set out to perform here, I think there is nothing to say.

01 December, 2019 05:22PM

hackergotchi for Thomas Goirand

Thomas Goirand

Upgrading an OpenStack Rocky cluster from Stretch to Buster

Upgrading an OpenStack cluster from one version of OpenStack to another has become easier, thanks to the versioning of objects in the rabbitmq message bus (if you want to know more, see what oslo.versionedobjects is). But upgrading from Stretch to Buster isn’t easy at all, event with the same version of OpenStack (it is easier to be running OpenStack Rocky backports on Stretch and upgrade to Rocky on Buster, rather than upgrading OpenStack at the same time as the system).

The reason it is difficult, is because rabbitmq and corosync in Stretch can’t talk to the versions shipped in Buster. Also, in a normal OpenStack cluster deployment, services on all machines are constantly doing queries to the OpenStack API, and exchanging messages through the RabbitMQ message bus. One of the dangers, for example, would be if a Neutron DHCP agent could not exchange messages with the neutron-rpc-server. Your VM instances in the OpenStack cluster then could loose connectivity.

If a constantly online HA upgrade with no downtime isn’t possible, it is however possible to minimize down time to just a few seconds, if following a correct procedure. It took me more than 10 tries to be able to do everything in a smooth way, understanding and working around all the issues. 10 tries, means installing 10 times an OpenStack cluster in Stretch (which, even if fully automated, takes about 2 hours) and trying to upgrade it to Buster. All of this is very time consuming, and I haven’t seen any web site documenting this process.

This blog post intends to document such a process, to save the readers the pain of hours of experimentation.

Note that this blog post asserts you’re cluster has been deployed using OCI (see: https://salsa.debian.org/openstack-team/debian/openstack-cluster-installer) however, it should also apply to any generic OpenStack installation, or even to any cluster running RabbitMQ and Corosync.

The root cause of the problem more in details: incompatible RabbitMQ and Corosync in Stretch and Buster

RabbitMQ in Stretch is version 3.6.6, and Buster has version 3.7.8. In theory, the documentation of RabbitMQ says it is possible to smoothly upgrade a cluster with these versions. However, in practice, the problem is the Erlang version rather than Rabbit itself: RabbitMQ in Buster will refuse to talk to a cluster running Stretch (the daemon will even refuse to start).

The same way, Corosync 3.0 in Buster will refuse to accept messages from Corosync 2.4 in Stretch.

Overview of the solution for RabbitMQ & Corosync

To minimize downtime, my method is to shutdown RabbitMQ on node 1, and let all daemons (re-)connect to node 2 and 3. Then we upgrade node 1 fully, and then restart Rabbit in there. Then we shutdown Rabbit on node 2 and 3, so that all daemons of the cluster reconnect to node 1. If done well, the only issue is if a message is still in the cluster of node 2 and 3 when daemons fail-over to node 1. In reality, this isn’t really a problem, unless there’s a lot of activity on the API of OpenStack. If this was the case (for example, if running a public cloud), then the advise would simply to firewall the OpenStack API for the short upgrade period (which shouldn’t last more than a few minutes).

Then we upgrade node 2 and 3 and make them join the newly created RabbitMQ cluster in node 1.

For Corosync, node 1 will not allow start the VIP resource before node 2 is upgraded and both nodes can talk to each other. So we just upgrade node 2, and turn off the VIP resource on node 3 immediately when it is up on node 1 and 2 (which happens during the upgrade of node 2).

The above should be enough reading for most readers. If you’re not that much into OpenStack, it’s ok to stop reading this post. For those who are move involved users of OpenStack on Debian deployed with OCI, let’s go more in details…

Before you start: upgrading OCI

In previous versions of OCI, the haproxy configuration was missing a “option httpcheck” for the MariaDB backend, and therefore, if a MySQL server on one node was going down, haproxy wouldn’t detect it, and the whole cluster could fail (re-)connecting to MySQL. As we’re going to bring some MySQL servers down, make sure the puppet-master is running with the latest version of puppet-module-oci, and that the changes have been applied in all OpenStack controller nodes.

Upgrading compute nodes

Before we upgrade the controllers, it’s best to start by compute nodes, which are the most easy to do. The easiest way is to live-migrate all VMs away from the machine before proceeding. First, we disable the node, so no new VM can be spawned on it:

openstack compute service set --disable z-compute-1.example.com nova-compute

Then we list all VMs on that compute node:

openstack server list –all-projects –host z-compute-1.example.com

Finally we migrate all VMs away:

openstack server migrate --live hostname-compute-3.infomaniak.ch --block-migration 8dac2f33-d4fd-4c11-b814-5f6959fe9aac

Now we can do the upgrade. First disable pupet, then tweak the sources.list, upgrade and reboot:

puppet agent --disable "Upgrading to buster"
apt-get remove python3-rgw python3-rbd python3-rados python3-cephfs librgw2 librbd1 librados2 libcephfs2
rm /etc/apt/sources.list.d/ceph.list
sed -i s/stretch/buster/g /etc/apt/sources.list
mv /etc/apt/sources.list.d/stretch-rocky.list /etc/apt/sources.list.d/buster-rocky.list
echo "deb http://stretch-rocky.debian.net/debian buster-rocky-proposed-updates main
deb-src http://stretch-rocky.debian.net/debian buster-rocky-proposed-updates main" >/etc/apt/sources.list/buster-rocky.list
apt-get update
apt-get dist-upgrade
reboot

Then we simply re-apply puppet:

puppet agent --enable ; puppet agent -t
apt-get purge linux-image-4.19.0-0.bpo.5-amd64 linux-image-4.9.0-9-amd64

Then we can re-enable the compute service:

openstack compute service set --enable z-compute-1.example.com nova-compute

Repeate the operation for all compute nodes, then we’re ready for the upgrade of controller nodes.

Removing Ceph dependencies from nodes

Most likely, if running with OpenStack Rocky on Stretch, you’d be running with upstream packages for Ceph Luminous. When upgrading to Buster, there’s no upstream repository anymore, and packages will use Ceph Luminous directly from Buster. Unfortunately, the packages from Buster are in a lower version than the packages from upstream. So before upgrading, we must remove all Ceph packages from upstream. This is what has been done just above for the compute nodes also. Upstream Ceph packages are easily identifiable, because upstream uses “bpo90” instead of what we do in Debian (ie: bpo9), so the operation can be:

apt-get remove $(dpkg -l | grep bpo90 | awk '{print $2}' | tr '\n' ' ')

This will remove python3-nova, which is fine as it is also running on the other 2 controllers. After switching the /etc/apt/sources.list to buster, Nova can be installed again.

In a normal setup by OCI, here’s the sequence of command that needs to be done:

rm /etc/apt/sources.list.d/ceph.list
sed -i s/stretch/buster/g /etc/apt/sources.list
mv /etc/apt/sources.list.d/stretch-rocky.list /etc/apt/sources.list.d/buster-rocky.list
echo "deb http://stretch-rocky.debian.net/debian buster-rocky-proposed-updates main
deb-src http://stretch-rocky.debian.net/debian buster-rocky-proposed-updates main" >/etc/apt/sources.list/buster-rocky.list
apt-get update
apt-get dist-upgrade
apt-get install nova-api nova-conductor nova-consoleauth nova-consoleproxy nova-placement-api nova-scheduler

You may notice that we’re replacing the Stretch Rocky backports repository by one for Buster. Indeed, even if all of Rocky is in Buster, there’s a few packages that are still pending for the review of the Debian stable release team before they can be uploaded to Buster, and we need the fixes for a smooth upgrade. See release team bugs #942201, #942102, #944594, #941901 and #939036 for more details.

Also, since we only did a “apt-get remove”, the Nova configuration in nova.conf must have stayed, and nova is already configured, so when we reinstall the services we removed when removing the Ceph dependencies, they will be ready to go.

Upgrading the MariaDB galera cluster

In an HA OpenStack cluster, typically, a Galera MariaDB cluster is used. That isn’t a problem when upgrading from Stretch to Buster, because the on-the-wire format stays the same. However, the xtrabackup library in Stretch is held by the MariaDB packages themselves, while in Buster, one must install the mariadb-backup. As a consequence, best is to simply turn off MariaDB in a node, do the Buster upgrade, install the mariadb-backup package, and restart MariaDB. To avoid that the MariaDB package attempts restarting the mysqld daemon, best is to mask the systemd unit:

systemctl stop mysql.service
systemctl disable mysql.service
systemctl mask mysql.service

Upgrading rabbitmq-server

Before doing anything, make sure all of your cluster is running with the python3-oslo.messaging version >= 8.1.4. Indeed, version 8.1.3 suffers from a bug where daemons would attempt reconnect constantly to the same server, instead of trying each of the servers described in the transport_url directive. Note that I’ve uploaded 8.1.4-1+deb10u1 to Buster, and that it is part of the 10.2 Buster point release. Though upgrading oslo.messaging will not restart daemons automatically: this must be done manually.

The strategy for RabbitMQ is to completely upgrade one node, start Rabbit on it, without any clustering, then shutdown the service on the other 2 node of the cluster. If this is performed fast enough, no message will be list in the message bus. However, there’s a few traps. Running “rabbitmqctl froget_cluster_node” only removes a node from the cluster for those who will still be running. It doesn’t remove the other nodes from the one which we want to upgrade. The way I’ve found to solve this is to simply remove the mnesia database of the first node, so that when it starts, RabbitMQ doesn’t attempt to cluster with the other 2 which are running a different version of Erlang. If it did, then it would just fail and refused to start.

However, there’s another issue to take care. When upgrading the 1st node to Buster, we removed Nova, because of the Ceph issue. Before we restart the RabbitMQ service on node 1, we need to install Nova, so that it will connect to either node 2 or 3. If we don’t do that, then Nova on node 1 may connect to the RabbitMQ service on node 1, which at this point, is a different RabbitMQ cluster than the one in node 2 and 3.

rabbitmqctl stop_app
systemctl stop rabbitmq-server.service
systemctl disable rabbitmq-server.service
systemctl mask rabbitmq-server.service
[ ... do the Buster upgrade fully ...]
[ ... reinstall Nova services we removed when removing Ceph ...]
rm -rf /var/lib/rabbitmq/mnesia
systemctl unmask rabbitmq-server.service
systemctl enable rabbitmq-server.service
systemctl start rabbitmq-server.service

At this point, since the node 1 RabbitMQ service was down, all daemons are connected to the RabbitMQ service on node 2 or 3. Removing the mnesia database removes all the credentials previously added to rabbitmq. If nothing is done, OpenStack daemons will not be able to connect to the RabbitMQ service on node 1. If like I do, one is using a config management system to populate the access rights, it’s rather easy: simply re-apply the puppet manifests, which will re-add the credentials. However, that isn’t enough: the RabbitMQ message queues are created when the OpenStack daemon starts. As I experienced, daemons will reconnect to the message bus, but will not recreate the queues unless daemons are restarted. Therefore, the sequence is as follow:

Do “rabbitmqctl start_app” on the first node. Add all credentials to it. If your cluster was setup with OCI and puppet, simply look at the output of “puppet agent -t –debug” to capture the list of commands to perform the credential setup.

Do a “rabbitmqctl stop_app” on both remaining nodes 2 and 3. At this point, all daemons will reconnect to the only remaining server. However, they wont be able to exchange messages, as the queues aren’t declared. This is when we must restart all daemons in one of the controllers. The whole operation normally doesn’t take more than a few seconds, which is how long your message bus wont be available. To make sure everything works, check the logs in /var/log/nova/nova-compute.log of one of your compute nodes to make sure Nova is able to report its configuration to the placement service.

Once all of this is done, there’s nothing to worry anymore about RabbitMQ, as all daemons of the cluster are connected to the service on node 1. However, one must make sure that, when upgrading node 2 and 3, they don’t reconnect to the message service on node 2 and 3. So best is to simply stop, disable and mask the service with systemd before continuing. Then, when restarting the Rabbit service on node 2 and 3, OCI’s shell script “oci-auto-join-rabbitmq-cluster” will make them join the new Rabbit cluster, and everything should be fine regarding the message bus.

Upgrading corosync

In an OpenStack cluster setup by OCI, 3 controllers are typically setup, serving the OpenStack API through a VIP (a Virtual IP). What we call a virtual IP is simply an IP address which is able to move from one node to another automatically depending on the cluster state. For example, with 3 nodes, if one goes down, one of the other 2 nodes will take over hosting the IP address which serves the OpenStack API. This is typically done with corosync/pacemaker, which is what OCI sets up.

The way to upgrade corosync is easier than the RabbitMQ case. The first node will refuse to start the corosync resource if it can’t talk to at least a 2nd node. Therefore, upgrading the first node is transparent until we touch the 2nd node: the openstack-api resource wont be started on the first node, so we can finish the upgrade in it safely (ie: take care of RabbitMQ as per above). The first thing to do is probably to move the resource to the 3rd node:

crm_resource --move --resource openstack-api-vip --node z-controller-3.example.com

Once the first node is completely upgraded, we upgrade the 2nd node. When it is up again, we can check the corosync status to make sure it is running on both node 1 and 2:

crm status

If we see the service is up on node 1 and 2, we must quickly shutdown the corosync resource on node 3:

crm resource stop openstack-api-vip

If that’s not done, then node 3 may also reclaim the VIP, and therefore, 2 nodes may it. If running with the VIP using L2 protocol, normally switches will connect only one of the machines declaring the VIP, so even if we don’t take care of it immediately, the upgrade should be smooth anyway. If, like I do in production, you’re running with BGP (OCI allows one to use BGP for the VIP, or simply use an IP on a normal L2 network), then the situation must be even better, as the peering router will continue to route to one of the controllers in the cluster. So no stress, this must be done, but no need to hurry as much as for the RabbitMQ service.

Finalizing the upgrade

Once node 1 and 2 are up, most of the work is done, and the 3rd node can be upgraded without any stress.

Recap of the procedure for controllers

  • Move all SNAT virtual routers running on node 1 to node 2 or 3 (note: this isn’t needed if the cluster has network nodes).
  • Disable puppet on node 1.
  • Remove all Ceph libraries from upstream on node 1, which also turn off some Nova services that runtime depend on them.
  • shutdown rabbitmq on node 1, including masking the service with systemd.
  • upgrade node 1 to Buster, fully. Then reboot it. This probably will trigger MySQL re-connections to node 2 or 3.
  • install mariadb-backup, start the mysql service, and make sure MariaDB is in sync with the other 2 nodes (check the log files).
  • reinstall missing Nova services on node 1.
  • remove the mnesia db on node 1.
  • start rabbitmq on node 1 (which now, isn’t part of the RabbitMQ cluster on node 2 and 3).
  • Disable puppet on node 2.
  • populate RabbitMQ access rights on node 1. This can be done by simply applying puppet, but may be dangerous if puppet restarts the OpenStack daemons (which therefore may connect to the RabbitMQ on node 1), so best is to just re-apply the grant access commands only.
  • shutdown rabbitmq on node 2 and 3 using “rabbitmqctl stop_app”.
  • quickly restart all daemons on one controller (for example the daemons on node 1) to declare message queues. Now all daemons must be reconnected and working with the RabbitMQ cluster on node 1 alone.
  • Re-enable puppet, and re-apply puppet on node 1.
  • Move all Neutron virtual routers from node 2 to node 1.
  • Make sure the RabbitMQ services are completely stopped on node 2 and 3 (mask the service with systemd).
  • upgrade node 2 to Buster (shutting down RabbitMQ completely, masking the service to avoid it restarts during upgrade, removing the mnesia db for RabbitMQ, and finally making it rejoin the newly node 1 single node cluster using oci-auto-join-rabbitmq-cluster: normally, puppet does that for us).
  • Reboot node 2.
  • When corosync on node 2 is up again, check corosync status to make sure we are clustering between node 1 and 2 (maybe the resource on node 1 needs to be started), and shutdown the corosync “openstack-api-vip” resource on node 3 to avoid the VIP to be declared on both nodes.
  • Re-enable puppet and run puppet agent -t on node 2.
  • Make node 2 rabbitmq-server has joined the new cluster declared on node 1 (do: rabbitmqctl cluster_status) so we have HA for Rabbit again.
  • Move all Neutron virtual routers of node 3 to node 1 or 2.
  • Upgrade node 3 fully, reboot it, and make sure Rabbit is connected to node 1 and 2, as well as corosync working too, then re-apply puppet again.

Note that we do need to re-apply puppet each time, because of some differences between Stretch and Buster. For example, Neutron in Rocky isn’t able to use iptables-nft, and puppet needs to run some update-alternatives command to select iptables-legacy instead (I’m writing this because this isn’t obvious, it’s just that sometimes, Neutron fails to parse the output of iptables-nft…).

Last words as a conclusion

While OpenStack itself has made a lot of progress for the upgrade, it is very disappointing that those components on which OpenStack relies (like corosync, who is typically used as the provider of high availability), aren’t designed with backward compatibility in mind. It is also disappointing that the Erlang versions in Stretch and Buster are incompatible this way.

However, with the correct procedure, it’s still possible to keep services up and running, with a very small down time, even to the point that a public cloud user wouldn’t even notice it.

As the procedure isn’t easy, I strongly suggest anyone attempting such an upgrade to train before proceeding. With OCI, it is easy to do run a PoC using the openstack-cluster-installer-poc package, which is the perfect environment to train on: it’s easy to reproduce, reinstall a cluster and restart the upgrade procedure.

01 December, 2019 04:45PM by Goirand Thomas

hackergotchi for Jonathan Carter

Jonathan Carter

Free Software Activities (2019-11)

TL;DR: Mostly a bunch of package sponsoring this month. :)

2019-11-11: Sponsor package python-tempora (1.14.1-1) for Debian unstable (Python team request).

2019-11-11: Sponsor package python-jaraco.functools (2.0-2) for Debian unstable (Python team request).

2019-11-11: Review package fpylll (Needs some more work) (Python team request).

2019-11-12: Sponsor package fpylll (0.4.1+ds1-7) for Debian unstable (Python team request).

2019-11-12: Review package python-flask-openid (Needs some more work) (Python team request).

2019-11-12: Upload package calamares (3.2.16-1) to Debian unstable.

2019-11-12: Review package python-six (Deferred to maintainer) (Python team request).

2019-11-12: Upload package gnome-shell-extension-draw-on-your-screen (14.1-1) to Debian unstable.

2019-11-12: Upload package vim-airline (11-1) to Debian unstable.

2019-11-12: Upload package gnome-shell-extension-arc-menu (38-dev-3) to Debian unstable.

2019-11-12: Sponsor package python-opentimestamps (0.4.1-1) for Debian unstable (Python team request).

2019-11-12: Sponsor package sphinx-autodoc-typehints (1.9.0-1) for Debian unstable (Python team request).

2019-11-12: Sponsor package flask-principal (0.4.0-2) for Debian unstable (Python team request).

2019-11-13: Sponsor package runescape (0.6-2) for Debian unstable (mentors.debian.net request).

2019-11-13: Sponsor package trace-cmd (2.8.3-1) for Debian unstable (mentors.debian.net request).

2019-11-13: Sponsor package gexiv (0.12.0-1) for Debian unstable (mentors.debian.net request).

2019-11-13: Sponsor package notepadqq (2.0.0~beta1-1) for Debian unstable (mentors.debian.net request).

2019-11-13: Sponsor package hijra (0.4.1-2) for Debian unstable (mentors.debian.net request).

2019-11-13: Sponsor package simple-scan (3.34.1-2) for Debian unstable (mentors.debian.net request).

2019-11-13: Sponsor package gyros (0.3.12) for Debian unstable (mentors.debian.net request).

2019-11-13: Sponsor package sysbench (1.0.18+ds-1) for Debian unstable (mentors.debian.net request).

2019-11-14: Sponsor package onedrivesdk (1.1.8-2) for Debian unstable (Python team request).

2019-11-14: Sponsor package pympler (0.7+dfsg1-1) for Debian unstable (Python team request).

2019-11-14: Sponsor package python3-portend (2.5-1) for Debian unstable (Python team request).

2019-11-14: Sponsor package clamfs (1.1.0-1) for Debian unstable (mentors.debian.net request).

2019-11-14: Sponsor package xautolock (2.2-6) for Debian unstable (mentors.debian.net request).

2019-11-14: Review package piper (0.3-1) (Needs some more work) (mentors.debian.net request).

2019-11-14: Review package srcpy (1.10+ds-1) (Needs some more work) (mentors.debian.net request).

2019-11-14: Sponsor package python-ebooklib (0.17-1) for Debian unstable (mentors.debian.net request).

2019-11-14: Sponsor package plowshare (2.1.7-4) for Debian unstable (mentors.debian.net request).

2019-11-14: Sponsor package py-libzfs (0.0+git20191113.2991805-1) for Debian unstable (Python team request).

2019-11-18: Sponsor package rpl (1.6.3-1) for Debian unstable (mentors.debian.net request).

2019-11-19: Upload new package feed2toot (0.12-1) to Debian unstable.

2019-11-21: Sponsor package isbg (2.2.1-2) for Debian unstable (Python team request).

2019-11-21: Sponsor package python-elasticsearch (7.1.0-1) for Debian unstable (Python team request).

2019-11-21: Review package python-fsspec (0.6.0-1) (Needs some more work) (Python team request).

2019-11-21: Sponsor package blastem (0.6.3.3-2) for Debian unstable (mentors.debian.net request).

2019-11-21: Review package ledmon (0.93-1) for Debian unstable (needs some more work) (mentors.debian.net request).

2019-11-21: Sponsor package ripser (1.1-1) for Debian unstable (mentors.debian.net request).

2019-11-21: Sponsor package surgescript (0.5.4-1) for Debian unstable (mentors.debian.net request).

2019-11-21: Upload package power (1.4+dfsg-4) to Debian unstable (Closes: #854887).

2019-11-22: Sponsor package sqlobject (3.7.3+dfsg-1) for Debian unstable (Python team request).

2019-11-22: Upload package foliate (1.5.3+dfsg1-1) to Debian experimental.

2019-11-28: Sponsor package micropython (1.11-1) (E-mail request).

2019-11-28: Sponsor package qosmic (1.6.0-2) (mentors.debian.net request).

01 December, 2019 12:14PM by jonathan

hackergotchi for Junichi Uekawa

Junichi Uekawa

Feeling some Fate in Coincidence.

Feeling some Fate in Coincidence. Just by chance I saw some sign that impressed me and I feel something in the chance.

01 December, 2019 03:39AM by Junichi Uekawa

Paul Wise

FLOSS Activities November 2019

Changes

Issues

Review

Administration

  • Chromium BSU: answer support requests
  • Funguloids: applied patches, ticket triage
  • Debian: updated apt repo key expiry
  • Debian website: fix directory permissions again
  • Debian wiki: diagnose DocBook output issue, unblacklist IP addresses, whitelist email addresses, whitelist email domains

Communication

Sponsors

The purple-discord, libpst work was sponsored by my employer. All other work was done on a volunteer basis.

01 December, 2019 02:39AM

hackergotchi for Keith Packard

Keith Packard

picolibc-float

Picolibc Without Double

Smaller embedded processors may have no FPU, or may have an FPU that only supports single-precision mode. In either case, applications may well want to be able to avoid any double precision arithmetic as that will drag in a pile of software support code. Getting picolibc to cooperate so that it doesn't bring in double-precision code was today's exercise.

Take a look at the changes in git

__OBSOLETE_MATH is your friend

The newlib math library, which is where picolibc gets its math code, has two different versions of some functions:

  • single precision sin, cos and sincos
  • single and double precision exp, exp2 and log, log2 and pow

The older code, which was originally written at Sun Microsystems (most of this code carries a 1993 copyright), is quite careful to perform single precision functions using only single precision intermediate values.

The newer code, which carries a 2018 copyright from Arm Ltd, uses double precision intermediate values for single precision functions.

I haven't evaluated the accuracy of either algorithm, but the newer code claims to be faster on machines which support double in hardware.

However, for machines with no hardware double support, especially for machines with hardware single precision support, I'm betting the code which avoids double will be faster. Not to mention all of the extra space in ROM that would be used by a soft double implementation.

I had switched the library to always use the newer code while messing about with some really stale math code last month, not realizing exactly what this flag was doing. I got a comment on that patch from github user 'innerand' which made me realize my mistake.

I've switched the default back to using the old math code on platforms that don't have hardware double support, and using the new math code on platforms that do. I also added a new build option, -Dnewlib-obsolete-math, which can be set to auto, true, or false. auto mode is the default, which selects as above.

Float vs Double error handling

Part of the integration of the Arm math code changed how newlib/picolibc handles math errors. The new method calls functions to set errno and return a specific value back to the application, like __math_uflow, which calls __math_xflow which calls __math_with_errno. All of these versions take double parameters and return double results. Some of them do minor arithmetic on these parameters. There are also float versions of these handlers, which are for use in float operations.

One float function, the __OBSOLETE_MATH version of log1pf, was mistakenly using the double error handlers, __math_divzero and __math_invalid. Just that one bug pulled in most of the soft double precision implementation. I fixed that in picolibc and sent a patch upstream to newlib.

Float printf vs C ABI

The C ABI specifies that float parameters to varargs functions are always promoted to doubles. That means that printf never gets floats, only doubles. Program using printf will end up using doubles, even if there are no double values anywhere in the code.

There's no easy way around this issue — it's hard-wired in the C ABI. Smaller processors, like the 8-bit AVR, “solve” this by simply using the same 32-bit representation for both double and float. On RISC-V and ARM processors, that's not a viable solution as they have a well defined 64-bit double type, and both GCC and picolibc need to support that for applications requiring the wider precision.

I came up with a kludge which seems to work. Instead of passing a float parameter to printf, you can pass a uint32_t containing the same bits, which printf can unpack back into a float. Of course, both the caller and callee will need to agree on this convention.

Using the same mechanism as was used to offer printf/scanf functions without floating point support, when the #define value, PICOLIBC_FLOAT_PRINTF_SCANF is set before including stdio.h, the printf functions are all redefined to reference versions with this magic kludge enabled, and the scanf functions redefined to refer to ones with the 'double' code disabled.

A new macro, printf_float(x) can be used to pass floats to any of the printf functions. This also works in the normal version of the code, so you can use it even if you might be calling one of the regular printf functions.

Here's an example:

#define PICOLIBC_FLOAT_PRINTF_SCANF
#include <stdio.h>
#include <stdlib.h>

int
main(void)
{
    printf("pi is %g\n", printf_float(3.141592f));
}

Results

Just switching to float-only printf removes the following soft double routines:

  • __adddf3
  • __aeabi_cdcmpeq
  • __aeabi_cdcmple
  • __aeabi_cdrcmple
  • __aeabi_d2uiz
  • __aeabi_d2ulz
  • __aeabi_dadd
  • __aeabi_dcmpeq
  • __aeabi_dcmpge
  • __aeabi_dcmpgt
  • __aeabi_dcmple
  • __aeabi_dcmplt
  • __aeabi_dcmpun
  • __aeabi_ddiv
  • __aeabi_dmul
  • __aeabi_drsub
  • __aeabi_dsub
  • __aeabi_f2d
  • __aeabi_i2d
  • __aeabi_l2d
  • __aeabi_ui2d
  • __aeabi_ul2d
  • __cmpdf2
  • __divdf3
  • __eqdf2
  • __extendsfdf2
  • __fixunsdfdi
  • __fixunsdfsi
  • __floatdidf
  • __floatsidf
  • __floatundidf
  • __floatunsidf
  • __gedf2
  • __gtdf2
  • __ledf2
  • __ltdf2
  • __muldf3
  • __nedf2
  • __subdf3
  • __unorddf2

The program shrank by 2672 bytes:

$ size double.elf float.elf
   text    data     bss     dec     hex filename
  48568     116   37952   86636   1526c double.elf
  45896     116   37952   83964   147fc float.elf

01 December, 2019 02:31AM

Utkarsh Gupta

Debian Activities for November 2019

Here’s my (second) monthly update about the activities I’ve done in Debian this November.

Debian LTS

This was my second month as a Debian LTS paid contributor.
I was assigned 18 hours and worked on the following things:

CVE Fixes and Announcements:

  • Issued DLA 1984-1, fixing CVE-2019-17545, for gdal.
    Details here:

    GDAL through 3.0.1 had a poolDestroy double free in OGRExpatRealloc in ogr/ogr_expat.cpp when the 10MB threshold was exceeded.

    For Debian 8 “Jessie”, this has been fixed in 1.10.1+dfsg-8+deb8u1.
    Furthermore, sent a patch to the Security team for fixing the same in Stretch. Relevant .dsc can be found here. Since I haven’t heard back from the team yet, the upload to Stretch is still pending.

  • Issued DLA 1986-1, fixing CVE-2012017-1002201, for ruby-haml.
    Details here:

    In haml, when using user input to perform tasks on the server, characters like < > “ ‘ must be escaped properly. In this case, the ‘ character was missed. An attacker can manipulate the input to introduce additional attributes, potentially executing code.

    For Debian 8 “Jessie”, this has been fixed in 4.0.5-2+deb8u1.

  • Issued DLA 2004-1, fixing CVE-2019-14824, for 389-ds-base.
    Details here:

    A flaw was found in the ‘deref’ plugin of 389-ds-base where it could use the ‘search’ permission to display attribute values. In some configurations, this could allow an authenticated attacker to view private attributes, such as password hashes.

    For Debian 8 “Jessie”, this has been fixed in 1.3.3.5-4+deb8u7.
    Furthermore, sent a patch to the maintainer, Timo, for fixing the same in Bullseye, Sid. And to the Security team for Stretch and Buster. The patch can be found here.

  • Issued DLA 2005-1, fixing CVE-2019-18849, for tnef.
    Details here:

    In tnef, an attacker may be able to write to the victim’s .ssh/authorized_keys file via an e-mail message with a crafted winmail.dat application/ms-tnef attachment, because of a heap-based buffer over-read involving strdup.

    For Debian 8 “Jessie”, this has been fixed in 1.4.9-1+deb8u4.
    Furthermore, sent a patch to the maintainer for fixing the same in Bullseye, Sid. And to the Security team for Stretch and Buster. The patch can be found here.

Miscellaneous:

  • Fixed CVE-2019-11027 for ruby-openid in unstable. News here. This is in reference with the DLA 1956-1, issued by Brian May.

  • Triage libexif, libjpeg-turbo, tnef, and ansible for Jessie.

  • Pinged upstream of libexif and 389-ds-base for relevant commits. Whilst 389-ds-base is now fixed, the maintainer of libexif is still working on the fix.

  • In midst of fixing CVE-2019-18978 for ruby-rack-cors and CVE-2019-2201 for libjpeg-turbo.

Debian Uploads

New Version:

  • ruby-openid ~ 2.9.2debian-1 (to unstable).
  • gitlab ~ 12.2.9-2 (to experimental).
  • node-yarnpkg ~ 1.19.1-1~bpo10+1 (to backports).
  • node-js-yaml ~ 3.13.1+dfsg-2~bpo10+1 (to backports).
  • ruby-sshkey ~ 2.0.0-2~bpo10+1 (to backports).
  • ruby-bootstrap-form ~ 4.2.0-2~bpo10+1 (to backports).

Bug Fixes:

  • #944906 for gitlab.
  • #930388 for ruby-openid.
  • #945232 for ruby-benchmark-suite.

Reviews and Sponsored Uploads:

  • node-hawk ~ 7.1.2+dfsg-1 for Sakshi Sangwan.
  • node-loud-rejection ~ 2.2.0-1 for Sakshi Sangwan.
  • node-lazy-cache ~ 2.0.2-1 for Sakshi Sangwan.

Until next time.
:wq for today.

01 December, 2019 12:05AM

November 30, 2019

Sam Hartman

The Case for Proposal B

This is my personal opinion, not that of the project leader. Tomorrow,
I'll write an essay trying to discuss the various options in with as
little bias as I can manage (although even that will be Sam's opinion).
Several people have asked me why I included Proposal B.
This is my answer.


While I was talking to people about systemd and init systems, people
seemed to inherently assume that being uncomfortable with systemd meant
that you were in favor of sysvinit, or at least init-script based
solutions. At least, people who were heavily involved in the issue made
that assumption. That didn't resonate with me.


Several concerns commonly raised with systemd resonate with me:


  1. It combines a bunch of things in one project; as an example how you
    start daemons ends up being tied to how you configure the network.

  2. This combination seems like it might reduce innovation at least
    outside of the systemd ecosystem, because interfaces are coupled.

  3. It is Linux specific


Of these, the biggest concern for me is the idea that systemd might
stifle innovation by becoming one point of control.


And yet, in my opinion, systemd is vastly superior to the current
alternatives. I'd far rather be writing service units than init
scripts. They are more declarative. Dependencies that I care about are
easier to express. There are better security isolation facilities. In
non-Debian work I've found that I depend heavily on systemd because it
is easier and more pleasurable to code to than the alternatives.
Declarative syntax for managing users is useful. I haven't personally
seen the huge joy of socket activation, but if I were writing somewhat
different things, perhaps I would. Given
the options today, I would pick systemd hands down and not look back.


But what about tomorrow? For me, one of the great things about Debian
has been that it's possible to integrate new technologies and to try
things out. Debian has been the OS where I and many others could try
out new technologies and figure out what it was like to fully integrate
them into the operating system. Systemd is the best we've got now, but
I'm reluctant to step away from Debian as a platform for innovation and
experimentation.


Yet I don't think focusing on sysvinit or other init-script based
solutions actually has anything to do with the kind of innovation I'm
talking about. I understand that for people who value sysvinit (or
something like runit) above systemd, that work is valuable. My
experience is that for my needs, systemd is a better fit. I wanted a
proposal that allowed us to maintain Debian as a platform for innovation
without focusing on the legacy of init scripts. I think that if there
is going to be something that some day replaces systemd, it will support
service units (or a significant subset) not init scripts. I suspect it
will have a way to handle socket activation and so on. And I cannot
imagine a future systemd replacement that does not have advanced
security isolation features.


How it Works


Proposal B is a systemd focused proposal. It's very similar to Proposal F.
The text is different, but the implications of both proposals are
similar. Maintainers can use whatever systemd facilities they choose.
Init scripts are not required. For most maintainers, even thinking
about alternate init systems or future experiments is something entirely
optional. That's true under both proposal F and Proposal B.


Where they differ is in how much support the project gives to
experiments involving alternate init systems. Under Proposal F, that's
entirely optional at each member's discretion. My experience is that's
not sufficient for Debian to remain a community for innovation. My
experience is that key maintainers and teams maintaining central
infrastructure or packages often need to work with people who are trying
to integrate new features. The difference between Proposal B and F is
that under Proposal B, we commit to making that happen for technologies
that are important in exploring alternatives to systemd.


Obviously, no member of our community is obligated to do work. In
practice this commitment might mean working to find new volunteers to
help out key packages or teams and do this work. Sadly, there are areas
where the history of interaction has not been so good; behavior on
multiple sides of discussions has not lived up to our standards. In
addition to making sure we have the necessary volunteers for key
packages and teams,
part of meeting this commitment may involve working with people who
want to explore alternatives to systemd to find advocates who foster a
climate where we can be excellent to each other.


The Risks


There are some real risks with Proposal B. The biggest is that we'll
spend time working on integrations and nothing innovative will come out
of it. A possible outcome is that we spend a lot of time integrating
elogind and similar technologies, but they end up not being useful
because packages start depending on service units and socket
activations. Unless something new comes along, we may waste our
effort. Yet we've often been willing to spend effort to enable people
to try things. For me, this proposal is about reaffirming that aspect
of Debian.


In the worst case, it's possible that we decrease the quality of our
systemd integration leaving room for something else, spend significant
emotional energy, and do not end up with interesting innovation.
I think it's much more likely that if there is no interesting
innovation, Proposal B will slowly morph into Proposal F.


Why did You Do this?


In the beginning of this post, I talked about how I personally
considered the concerns about systemd separate than the desire to keep
init-script based systems running. That view is uncommon among people
who have been spending a lot of time on this issue. In general people
who are spending a lot of time on init systems seem to be fairly
divided. If you are trying to get work done today, you are probably
either fully using systemd or using one of the existing init-script
based alternatives.


However, my concern resonated with developers I talk to who spend less
time involved in the issue. Not people who were going to go research
things enough to write a proposal. But people who weren't sure that
systemd was the way and the light of the future, but found it had a lot
of great things going for it.


I was one of the few people who was taking the time to really understand
the issues but who was somewhat flexible. I didn't even know how I was
going to rank the options on my ballot until this morning. Yes, I've
been systemd leaning in some ways, but I also very much see the
arguments in favor of enabling people to keep other init systems
working. I'd be happy with most of the options on this ballot winning.
So, I tried to listen and see if there were ways of splitting
disagreement that wouldn't work for the people most committed to one
position, but might appeal to people who are less involved.


Why are you Writing This Post?


I think it's dangerous for someone who is project leader to speak a
personal opinion, especially on a controversial issue. However, I've
heard people struggling with some of the issues I discuss here in our
community. What I say may give people another way of looking at
things. I do think I have a valuable prospective because I have spent a
lot of time thinking about the issues but have not been as intimately
involved as others who have spent similar time. I think my need to act
as a facilitator at least for this GR is over. And after spending a day
considering, I think it's more beneficial to specifically ask the
project to think about Debian as a community for experimentation than to
say nothing.

30 November, 2019 08:36PM

Sylvain Beucler

Debian LTS and ELTS - November 2019

Debian LTS Logo

Here is my transparent report for my work on the Debian Long Term Support (LTS) and Debian Extended Long Term Support (ELTS), which extend the security support for past Debian releases, as a paid contributor.

In November, the monthly sponsored hours were split evenly among contributors depending on their max availability - I was assigned 24.5h for LTS (out of 30 max) and 20h for ELTS (max).

Multiple vulnerabilities come from in-process fuzzing (library fuzzing with compiler instrumentation, as opposed to fuzzing a user executable). This is an interesting technique, though those are harder to reproduce, especially with older versions or (even worse) forks. A significant portion of such vulnerabilities comes from google's OSS-117Fuzz infrastructure.

data/CVE/list from the debian security-tracker repository reached 20M. With multiple changes per hour, git blame is consequently near-unusable: several minutes for a targetted, single-line look-up, if the entry is not too old. Despite this, the git commit messages are often used for triage justification or even as a substitute for personal communication, a practice I wouldn't recommend. #908678 looks stalled.

MITRE is still reactive when reporting issues on various free software project, and still very shy about changing the status of vulnerabilities. This is understandable when dealing with hard-to-reproduce issues, less understandable with legit-looking bogus vulnerabilities, which some people still like to throw at us so we have more work to do and get paid (seriously: please don't).

ELTS - Wheezy

  • Second part of my Front-Desk week, though only auto-triaged unsupported packages
  • CVE-2019-14866/cpio: help opal investigate reproducibility issue, contact cpio maintainer and security@gnu.org to get official patch/review
  • CVE-2019-18684/sudo: deconstruct bogus vulnerability; MITRE now marks it as DISPUTED
  • CVE-2019-5068/mesa: attempt to reproduce the issue, BTS update, testing, security upload
  • CVE-2019-3466/postgresql-common: triage: not-affected
  • libonig: start work on multiple vulnerabilities with non-trivial backports; to be completed in December
  • CVE-2019-19012/libonig: backport for 5.9, get maintainer review
  • CVE-2019-19246/libonig: register CVE for untracked vulnerability (discovered through upstream fuzzing, re-discovered through php-mbstring)
  • libonig: find embedded copy in php7.0 (Stretch) and php7.3 (Buster); LTS/ELTS not-affected

LTS - Jessie

  • CVE-2019-3689/nfs-util: ping upstream and debian sid, no pong
  • CVE-2019-14866/cpio: shared work with ELTS
  • CVE-2019-18684/sudo: shared work with ELTS
  • CVE-2019-5068/mesa: shared work with ELTS, security upload
  • CVE-2019-3466/postgresql-common: confirmed fix: jessie already fixed but I didn't notice due to late DLA
  • CVE-2019-11027/ruby-openid: provide requested second opinion
  • libav: start processing pending issues, package is a ffmpeg fork, was removed from newer dists and is unresponsive to security issues, requiring more work; to be completed in December
  • CVE-2019-17542/libav: heap-based buffer overflow: apply fix though libfuzzer-based reproducer not reproducible
  • CVE-2019-17539/libav: triage: not-affected (vulnerable code introduced later)
  • CVE-2019-14443/libav: reproduce, track down fix in ffmpeg, update libav bug
  • CVE-2019-14441/libav: mitre request: duplicate CVE-2018-19129 (got DISPUTED); fix attempt, update libav bug
  • CVE-2019-14371/libav: triage: already fixed through CVE-2018-11102
  • CVE-2019-9720/libav: triage: unimportant (stretching the definition of DoS)
  • CVE-2019-9719/libav: mitre request: rejection (got DISPUTED): generic warning, no vulnerability
  • CVE-2019-9717/libav: triage: unimportant (stretching the definition of DoS)
  • CVE-2018-20001/libav: jessie triage: postponed (not reproducible)
  • CVE-2018-19130/libav: mitre request: duplicate CVE-2017-17127 (got DISPUTED)
  • CVE-2018-19128/libav: reproduce, track down fix in ffmpeg
  • Welcome new trainee

Documentation/Scripts

30 November, 2019 07:59PM

Debian LTS and ELTS - November 2019

Debian LTS Logo

Here is my transparent report for my work on the Debian Long Term Support (LTS) and Debian Extended Long Term Support (ELTS), which extend the security support for past Debian releases, as a paid contributor.

In November, the monthly sponsored hours were split evenly among contributors depending on their max availability - I was assigned 24.5h for LTS (out of 30 max) and 20h for ELTS (max).

Multiple vulnerabilities come from in-process fuzzing (library fuzzing with compiler instrumentation, as opposed to fuzzing a user executable). This is an interesting technique, though those are harder to reproduce, especially with older versions or (even worse) forks. A significant portion of such vulnerabilities comes from google's OSS-117Fuzz infrastructure.

data/CVE/list from the debian security-tracker repository reached 20M. With multiple changes per hour, git blame is consequently near-unusable: several minutes for a targetted, single-line look-up, if the entry is not too old. Despite this, the git commit messages are often used for triage justification or even as a substitute for personal communication, a practice I wouldn't recommend. #908678 looks stalled.

MITRE is still reactive when reporting issues on various free software project, and still very shy about changing the status of vulnerabilities. This is understandable when dealing with hard-to-reproduce issues, less understandable with legit-looking bogus vulnerabilities, which some people still like to throw at us so we have more work to do and get paid (seriously: please don't).

ELTS - Wheezy

  • Second part of my Front-Desk week, though only auto-triaged unsupported packages
  • CVE-2019-14866/cpio: help opal investigate reproducibility issue, contact cpio maintainer and security@gnu.org to get official patch/review
  • CVE-2019-18684/sudo: deconstruct bogus vulnerability; MITRE now marks it as DISPUTED
  • CVE-2019-5068/mesa: attempt to reproduce the issue, BTS update, testing, security upload
  • CVE-2019-3466/postgresql-common: triage: not-affected
  • libonig: start work on multiple vulnerabilities with non-trivial backports; to be completed in December
  • CVE-2019-19012/libonig: backport for 5.9, get maintainer review
  • CVE-2019-19246/libonig: register CVE for untracked vulnerability (discovered through upstream fuzzing, re-discovered through php-mbstring)
  • libonig: find embedded copy in php7.0 (Stretch) and php7.3 (Buster); LTS/ELTS not-affected

LTS - Jessie

  • CVE-2019-3689/nfs-util: ping upstream and debian sid, no pong
  • CVE-2019-14866/cpio: shared work with ELTS
  • CVE-2019-18684/sudo: shared work with ELTS
  • CVE-2019-5068/mesa: shared work with ELTS, security upload
  • CVE-2019-3466/postgresql-common: confirmed fix: jessie already fixed but I didn't notice due to late DLA
  • CVE-2019-11027/ruby-openid: provide requested second opinion
  • libav: start processing pending issues, package is a ffmpeg fork, was removed from newer dists and is unresponsive to security issues, requiring more work; to be completed in December
  • CVE-2019-17542/libav: heap-based buffer overflow: apply fix though libfuzzer-based reproducer not reproducible
  • CVE-2019-17539/libav: triage: not-affected (vulnerable code introduced later)
  • CVE-2019-14443/libav: reproduce, track down fix in ffmpeg, update libav bug
  • CVE-2019-14441/libav: mitre request: duplicate CVE-2018-19129 (got DISPUTED); fix attempt, update libav bug
  • CVE-2019-14371/libav: triage: already fixed through CVE-2018-11102
  • CVE-2019-9720/libav: triage: unimportant (stretching the definition of DoS)
  • CVE-2019-9719/libav: mitre request: rejection (got DISPUTED): generic warning, no vulnerability
  • CVE-2019-9717/libav: triage: unimportant (stretching the definition of DoS)
  • CVE-2018-20001/libav: jessie triage: postponed (not reproducible)
  • CVE-2018-19130/libav: mitre request: duplicate CVE-2017-17127 (got DISPUTED)
  • CVE-2018-19128/libav: reproduce, track down fix in ffmpeg
  • Welcome new trainee

Documentation/Scripts

30 November, 2019 07:25PM

hackergotchi for Steinar H. Gunderson

Steinar H. Gunderson

More about the DDR arcade CDs

I'm continuing my journey throughout the world of the Dance Dance Revolution arcade CDs; it would be interesting to see how moddable they are, even though I don't have a machine myself (obviously, MAME is absolutely essential here).

One key fact that I didn't know about last time, but was eventually alerted to after looking at others' work, is that the software in flash runs off of a virtual filesystem (VFS). This makes things incredibly much easier than mucking around with offsets everywhere. It's sort of a strange hybrid, though; read on for more.

The System 573 mainboard has 16 MB (or 128 Mbit, if you wish) of onboard flash, spread over a few banks, and for the newer digital mixes, this is augmented with a 32 MB PCMCIA flash card (I believe the system can technically address 64 MB, but no software uses it, to the best of my knowledge). When installing the software from CD-ROM, it blits a file called GAME.DAT into the onboard flash and CARD.DAT into the PCMCIA card (plus sets up some checksums at 0xfe0000). Except for a few hard-coded items, they seem to largely be treated equivalently, simply as different backing stores for a single VFS.

When booting up regularly (SW4 set to booting from flash), it jumps to an address very early in the flash, which contains the bootloader (called boot/psx.bin in the VFS; but the VFS has a too short size for it, so if you trust the size when extracting it, it gets too short!). The bootloader reads the (encrypted) configuration file from 0xFE2000 (addressed as “/.raw=0x1fc4,0x2000” in the VFS, probably partially related ot that the flash is mapped up at 0x1f000000), which contains information about how to address the two flash devices and a bit more. It also reads the file table for the VFS at 0xFE4000, and from there, it's mostly filesystem time: The bootloader then loads the game itself from soft/s573/aout.exe and boots it.

Everything in the VFS is addressed by CRC32 only (a Konami-specific variant that uses only six bits per input character). Besides the CRC32 field, offset and size, there are a few curious fields: In particular, there's an encryption bit that specifies that the file is encrypted using a home-grown very simple stream cipher (using the CRC32 of the string “EXTREME” as key…); it's more like obfuscation. Only a few very central files, like the song index, are encrypted—and nothing must be, except for the configuration file (which doesn't have the encryption bit set!).

Furthermore, files can be marked as compressed, using a Konami-specific LZ77 algorithm. Curiously enough, even though loading appears to be fairly generic, some files that are eminently compressable, such as some textures on the PCMCIA card, cannot be compressed, or the game will crash. But many of the textures are indeed compressed, and without it, there probably wouldn't be space on the flash. There's generally lots of space on the PCMCIA card, though.

So, given all of this knowledge, can we use it for modding? Yes! There's a tool set called sys573tools, which, among others, can extract data from the file system. (It guesses file names based on known patterns and data found in other files, so that you don't only get a bunch of CRCs.) Given that, we can decrypt the music database, add more songs (for instance from other DDR mixes), pack everything back up again, flash, and then enjoy more songs. It's still pretty raw, but it seems to work pretty well.

How many songs is there room for? Nobody really knows. The machines can be fitted with with DVD drives, so there's plenty of CD space, but there's not infinite amount of RAM on the I/O board (which supposedly holds all the shorts, all the time), and nobody really knows what the limits of the internal structures are.

So, does anyone have a DWI-to-CSQ converter for neoMAX?

30 November, 2019 06:41PM

hackergotchi for Chris Lamb

Chris Lamb

Free software activities in November 2019

Software Freedom Conservancy, the fiscal sponsor for the Reproducible Builds project, have announced their fundraising season with a huge pledge to match donations. If you have been considering joining as a supporter, now would be the time to do so.


Here is my monthly update covering what I have been doing in the free software world during November 2019 (previous month):

  • As part of my duties of being on the board of directors of the Open Source Initiative I attended our autumn face-to-face meeting hosted by Zolando in Berlin, Germany. I also participated in various licensing and other discussions occurring on the internet, as well as the usual internal discussions regarding logistics, policy, etc. liasing at times with the ClearlyDefined project.

  • Started early conversations as the Head Judge for the next interation of the OpenUK awards to be given out in June 2020.

  • As part serving on board of the Software in the Public Interest, Inc. I attended my first face-to-face meeting in Manhattan, New York which was graciously hosted by Hudson River Trading. It was great to meet the rest of the board in person after talking at such length over the internet.

  • Opened pull requests to make the build reproducible in:


Reproducible builds

Whilst anyone can inspect the source code of free software for malicious flaws almost all software is distributed pre-compiled to end users.

The motivation behind the Reproducible Builds effort is to ensure no flaws have been introduced during this compilation process by promising identical results are always generated from a given source, thus allowing multiple third-parties to come to a consensus on whether a build was compromised.

The initiative is proud to be a member project of the Software Freedom Conservancy, a not-for-profit 501(c)(3) charity focused on ethical technology and user freedom.

Conservancy acts as a corporate umbrella, allowing projects to operate as non-profit initiatives without managing their own corporate structure. If you like the work of the Conservancy or the Reproducible Builds project, please consider becoming an official supporter.

Conservancy's fundraising season has begin in earnest with a huge pledge to match donations from a number of illustrious individuals. If you have ever considered joining as a supporter, now would be the time to do so.


This month, I:

  • I spent a few moments on our website this month too including dropping the duplicated use the term "community" and other words [...][...], correcting the capitalisation of GitHub & GitLab [...] and corrected the use of "an" [...].

  • Drafted, published and publicised our monthly report.

  • strip-nondeterminism is our tool to remove specific non-deterministic results from a completed build. This month, I added file as a dependency for libfile-stripnondeterminism-perl (#945212) and moved away from deprecated $ADTTMP variable [...].

  • Did some arrangement, organisation and financial administration regarding our upcoming summit meeting in Marrakesh, Morocco.

I also made the following changes to diffoscope, our in-depth and content-aware diff utility that can locate and diagnose reproducibility issues:

  • New features / improvements:

    • Allow all possible .zip file variations to return from external tools with non-zero exit codes, not just known types we can identify (eg. Java .jmod and .jar files). (#78)
    • Limit .dsc and .buildinfo file matching to files in ASCII or UTF-8 format. (#77)
    • Bump the previous max_page_size limit from 400 kB to 4 MB. [...]
    • Clarify in the HTML and text outputs that the limits are per-format, not global. (#944882)
    • Don't use line-base dbuffering when communucating with subprocesses in "binary" mode. (#75)

  • Regression fixes:

    • Correct the substitution/filtering of paths in ELF output to avoid unnecessary differences depending on the path name provided and commandline. (#945572)
    • Silence/correct a Python SyntaxWarning message due to incorrectly comparing an integer by identity vs. equality. (#945531)

  • Testsuite improvements:

    • Refresh the OCaml test fixtures to support versions greater than 4.08.1. [...]
    • Update an Android manifest test to reflect that parsed XML attributes are returned in a new/sorted manner under Python 3.8. [...]
    • Dramatically Truncate the tcpdump expected diff to 8KB from ~600KB to reduce the size of the release tarball. [...]
    • Add a self-test to encourage that new test data files are generated dynamically or at least no new ones are added without an explicit override. [...]
    • Add a comment that the text_ascii1 and text_ascii2 fixture files are used in multiple tests so is not trivial to remove/replace them. [...]
    • Drop two more test fixture files for the directory tests. [...]
    • Don't run our self-test against the output of the Black source code reformatter with versions earlier than ours as it will generate different results. [...]
    • Update an XML test for Python 3.8. [...]
    • Drop unused an unused BASE_DIR global. [...]

  • Code improvements:

    • Rework a long string of or statements into a loop with a break. [...]
    • Update code to reflect the latest version of the Black source code reformatter. [...]
    • Correct a reference to the .rdx extension suffix in a comment. [...]


Debian

Uploads

  • redis (5:5.0.7-1) — New upstream release.

  • python-django (2.2.7-1 & 3.0~rc1-1) — New upstream releases.

  • cpio (2.13+dfsg-1) — New upstream release.

  • libfiu (1.00-4) — Prevent a build failure when multiple Python version libraries exist in the build tree by manually deleting all but the version for the default Python version returned by py3versions prior to running the test suite. (#944911)

  • gunicorn (20.0.0-1 & 20.0.2-1) — New upstream releases.

  • memcached (1.5.20-1) — New upstream release.

I also sponsored an upload of adminer (4.7.5-1) of behalf of Alexandre Rossi.


Debian bugs filed

  • RFP (Request for Package) for container-diff. (#945524)

  • bowtie2: imp Python module deprecation warning is embedded into bowtie2-inspect(1) manpage. (#945422)

  • golang-github-nrdcg-goinwx: Please update/expand the "Andrew" copyright holder. (#944066)

  • pcb-rnd: /usr/lib symlinks point to an (absolute) build directory. (#943955)

Debian LTS

This month I have worked 18 hours on Debian Long Term Support (LTS) and 12 hours on its sister Extended LTS project.

You can find out more about the project via the following video:


FTP Team

As a Debian FTP assistant I ACCEPTed 21 packages: golang-github-boj-redistore, golang-github-dchest-uniuri, golang-github-jackc-fake, golang-github-joyent-gocommon, golang-github-mattetti-filebuffer, golang-github-nrdcg-goinwx, golang-github-pearkes-dnsimple, golang-github-soniah-dnsmadeeasy, golang-github-vultr-govultr, golang-github-zorkian-go-datadog-api, meep, meep-mpi-default, meep-openmpi, node-eslint-plugin-requirejs, node-i18next, node-node-sass, node-re2, ocplib-endian, python-asynctest, python-janus & python-matrix-nio.

30 November, 2019 02:55PM

Free software activities in November 2019

Software Freedom Conservancy, the fiscal sponsor for the Reproducible Builds project, have announced their fundraising season with a huge pledge to match donations. If you have been considering joining as a supporter, now would be the time to do so.


Here is my monthly update covering what I have been doing in the free software world during November 2019 (previous month):

  • As part of my duties of being on the board of directors of the Open Source Initiative I attended our autumn face-to-face meeting hosted by Zolando in Berlin, Germany. I also participated in various licensing and other discussions occurring on the internet, as well as the usual internal discussions regarding logistics, policy, etc. liasing at times with the ClearlyDefined project.

  • Started early conversations as the Head Judge for the next interation of the OpenUK awards to be given out in June 2020.

  • As part serving on board of the Software in the Public Interest, Inc. I attended my first face-to-face meeting in Manhattan, New York which was graciously hosted by Hudson River Trading. It was great to meet the rest of the board in person after talking at such length over the internet.

  • Opened pull requests to make the build reproducible in:


Reproducible builds

Whilst anyone can inspect the source code of free software for malicious flaws almost all software is distributed pre-compiled to end users.

The motivation behind the Reproducible Builds effort is to ensure no flaws have been introduced during this compilation process by promising identical results are always generated from a given source, thus allowing multiple third-parties to come to a consensus on whether a build was compromised.

The initiative is proud to be a member project of the Software Freedom Conservancy, a not-for-profit 501(c)(3) charity focused on ethical technology and user freedom.

Conservancy acts as a corporate umbrella, allowing projects to operate as non-profit initiatives without managing their own corporate structure. If you like the work of the Conservancy or the Reproducible Builds project, please consider becoming an official supporter.

Conservancy's fundraising season has begin in earnest with a huge pledge to match donations from a number of illustrious individuals. If you have ever considered joining as a supporter, now would be the time to do so.


This month, I:

  • I spent a few moments on our website this month too including dropping the duplicated use the term "community" and other words [...][...], correcting the capitalisation of GitHub & GitLab [...] and corrected the use of "an" [...].

  • Drafted, published and publicised our monthly report.

  • strip-nondeterminism is our tool to remove specific non-deterministic results from a completed build. This month, I added file as a dependency for libfile-stripnondeterminism-perl (#945212) and moved away from deprecated $ADTTMP variable [...].

  • Did some arrangement, organisation and financial administration regarding our upcoming summit meeting in Marrakesh, Morocco.

I also made the following changes to diffoscope, our in-depth and content-aware diff utility that can locate and diagnose reproducibility issues:

  • New features / improvements:

    • Allow all possible .zip file variations to return from external tools with non-zero exit codes, not just known types we can identify (eg. Java .jmod and .jar files). (#78)
    • Limit .dsc and .buildinfo file matching to files in ASCII or UTF-8 format. (#77)
    • Bump the previous max_page_size limit from 400 kB to 4 MB. [...]
    • Clarify in the HTML and text outputs that the limits are per-format, not global. (#944882)
    • Don't use line-base dbuffering when communucating with subprocesses in "binary" mode. (#75)

  • Regression fixes:

    • Correct the substitution/filtering of paths in ELF output to avoid unnecessary differences depending on the path name provided and commandline. (#945572)
    • Silence/correct a Python SyntaxWarning message due to incorrectly comparing an integer by identity vs. equality. (#945531)

  • Testsuite improvements:

    • Refresh the OCaml test fixtures to support versions greater than 4.08.1. [...]
    • Update an Android manifest test to reflect that parsed XML attributes are returned in a new/sorted manner under Python 3.8. [...]
    • Dramatically Truncate the tcpdump expected diff to 8KB from ~600KB to reduce the size of the release tarball. [...]
    • Add a self-test to encourage that new test data files are generated dynamically or at least no new ones are added without an explicit override. [...]
    • Add a comment that the text_ascii1 and text_ascii2 fixture files are used in multiple tests so is not trivial to remove/replace them. [...]
    • Drop two more test fixture files for the directory tests. [...]
    • Don't run our self-test against the output of the Black source code reformatter with versions earlier than ours as it will generate different results. [...]
    • Update an XML test for Python 3.8. [...]
    • Drop unused an unused BASE_DIR global. [...]

  • Code improvements:

    • Rework a long string of or statements into a loop with a break. [...]
    • Update code to reflect the latest version of the Black source code reformatter. [...]
    • Correct a reference to the .rdx extension suffix in a comment. [...]


Debian

Uploads

  • redis (5:5.0.7-1) — New upstream release.

  • python-django (2.2.7-1 & 3.0~rc1-1) — New upstream releases.

  • cpio (2.13+dfsg-1) — New upstream release.

  • libfiu (1.00-4) — Prevent a build failure when multiple Python version libraries exist in the build tree by manually deleting all but the version for the default Python version returned by py3versions prior to running the test suite. (#944911)

  • gunicorn (20.0.0-1 & 20.0.2-1) — New upstream releases.

  • memcached (1.5.20-1) — New upstream release.

I also sponsored an upload of adminer (4.7.5-1) of behalf of Alexandre Rossi.


Debian bugs filed

  • RFP (Request for Package) for container-diff. (#945524)

  • bowtie2: imp Python module deprecation warning is embedded into bowtie2-inspect(1) manpage. (#945422)

  • golang-github-nrdcg-goinwx: Please update/expand the "Andrew" copyright holder. (#944066)

  • pcb-rnd: /usr/lib symlinks point to an (absolute) build directory. (#943955)

Debian LTS

This month I have worked 18 hours on Debian Long Term Support (LTS) and 12 hours on its sister Extended LTS project.

You can find out more about the project via the following video:


FTP Team

As a Debian FTP assistant I ACCEPTed 21 packages: golang-github-boj-redistore, golang-github-dchest-uniuri, golang-github-jackc-fake, golang-github-joyent-gocommon, golang-github-mattetti-filebuffer, golang-github-nrdcg-goinwx, golang-github-pearkes-dnsimple, golang-github-soniah-dnsmadeeasy, golang-github-vultr-govultr, golang-github-zorkian-go-datadog-api, meep, meep-mpi-default, meep-openmpi, node-eslint-plugin-requirejs, node-i18next, node-node-sass, node-re2, ocplib-endian, python-asynctest, python-janus & python-matrix-nio.

30 November, 2019 02:55PM

November 29, 2019

hackergotchi for Mike Gabriel

Mike Gabriel

My Work on Debian LTS/ELTS (November 2019)

In November 2019, I have worked on the Debian LTS project for 15 hours (of 15 hours planned) and on the Debian ELTS project for 5 hours (of 5 hours planned) as a paid contributor.

For LTS, I, in fact, pulled over 1.7 hours from October, so I realy only did 13.3 hours for LTS in November.

(This is only half-true, I worked a considerable amount of hours on this libvncserver code bundle audit, but I am just not invoicing all of it).

LTS Work

  • Triage jhead, libapache2-mod-auth-openidc, mailutils, python-psutil, ruby-rack-cors during (actually one day after, in coordination with Thorsten Alteholz) my first LTS frontdesk week this month.
  • Triage asterisk, gnome-font-viewer, gnome-sushi, libjackson-json-java, proftpd-dfsg during my second week at LTS frontdesk this month. I also triaged nss and sqlite3 as part of my ELTS work (which dla-needed.txt benefitted from indirectly).
  • Provide a .debdiff for italc/stretch (3.0.3+dfsg1-1+deb9u1) to the Debian Security Team (feeback still pending)
  • Triage libvncserver issues in vino, provide a patch for vino/unstable (see Debian bug #945784 [1])
  • Upload to jessie-security: vino (DLA-2014-1 [2]), 3 CVEs
  • Upload to experimental: libvncserver 0.9.12-1 and 0.9.12-2, 1 CVE fixed, adopt the package, resurrect libvncserver.git on Salsa, give the package some love before uploading [3,4]
  • Triage libvncserver (actually: libvncclient) issues in ssvnc and work on patches for this old VNC client code base
  • Upload to jessie-security: ssvnc (DLA-2016-1 [5]), 4 CVEs
  • Notify the ssvnc maintainer about newly discovered vulnerabilities (see Debian bug #945827).
  • Provide .debdiff patches for libvncserver/stretch (0.9.11+dfsg-1.3~deb9u2) and libvncserver/buster (0.9.11+dfsg-1.3+deb10u1) to the Debian Security Team (feedback still pending)
  • Sponsor-upload to jessie-security: 389-ds-base (DLA-2004-1), on behalf of Utkarsh Gupta
  • Sponsor-upload to jessie-security: tnef (DLA-2005-1), on behalf of Utkarsh Gupta
  • Upload to jessie-security: nss (DLA-2015-1 [6]), 1 CVE

ELTS Work

  • Triage nss (affected by CVE-2019-17007) and sqlite3 (not affected by CVE-2019-19242, CVE-2019-19244)
  • Upload to wheezy-security: nss (ELA-197-1 [7]), 1 CVE

References

29 November, 2019 09:15PM by sunweaver

hackergotchi for Bits from Debian

Bits from Debian

Debian welcomes its new Outreachy interns

Outreachy logo

Debian continues participating in Outreachy, and we'd like to welcome our new Outreachy interns for this round, lasting from December 2019 to March 2020.

Anisa Kuci will work on Improving the DebConf fundraising processes, mentored by Karina Ture and Daniel Lange.

Sakshi Sangwan will work on Packaging GitLab's JS Modules, mentored by Utkarsh Gupta, Sruthi Chandran and Pirate Praveen.

Congratulations, Anisa and Sakshi! Welcome!

From the official website: Outreachy provides three-month internships for people from groups traditionally underrepresented in tech. Interns work remotely with mentors from Free and Open Source Software (FOSS) communities on projects ranging from programming, user experience, documentation, illustration and graphical design, to data science.

The Outreachy programme is possible in Debian thanks to the efforts of Debian developers and contributors who dedicate their free time to mentor students and outreach tasks, and the Software Freedom Conservancy's administrative support, as well as the continued support of Debian's donors, who provide funding for the internships.

Join us and help extend Debian! You can follow the work of the Outreachy interns reading their blogs (they are syndicated in Planet Debian), and chat with us in the #debian-outreach IRC channel and mailing list.

29 November, 2019 10:20AM by Daniel Lange, Laura Arjona Reina

Hideki Yamane

OSS Maintainers should...

https://github.com/PyMySQL/PyMySQL/issues/829

OSS Maintainers should better learn to read with understanding! And yes they are tech support, and yes because the software is free tech support also should be free.If you release software you take responsibility. Such idiots like you should never be maintainers to anything.
Oh, moron.

29 November, 2019 05:04AM by Hideki Yamane (noreply@blogger.com)

November 28, 2019

Dima Kogan

python-gnuplotlib knows about multiplots

I just released gnuplotlib 0.32. It can do this: 

import gnuplotlib as gp
import numpy      as np
import numpysane  as nps

th = np.linspace(0, np.pi*2, 30)

gp.plot( th, nps.cat( np.cos(th), np.sin(th)),
         _xrange = [0,2.*np.pi],
         _yrange = [-1,1])

sinusoids-single-plot.svg 

import gnuplotlib as gp
import numpy      as np
import numpysane  as nps

th = np.linspace(0, np.pi*2, 30)

gp.plot( (th, np.cos(th), dict(title="cos")),
         (th, np.sin(th), dict(title="sin")),
         _xrange = [0,2.*np.pi],
         _yrange = [-1,1],
         multiplot='title "multiplot sin,cos" layout 2,1')

sinusoids-multi-plot.svg

Long version

gnuplotlib and numpysane are becoming mature (i.e. I use them all the time, have done that for a while, and they work very well), so I'm going to start doing some proselytizing on this front. I want to do a talk in the near future, and looking forward to that, I'm going to expand the docs, and I'm implementing some long-envisioned-but-never-completed features. The first one of these is now complete: multiplot support for gnuplotlib.

Gnuplot multiplots are a way to create more than one plot in a single window (or hardcopy). These are a bit of a corner case, and I've been mostly getting by without ever using these, but sometimes they're really nice. Use cases:

  • Plotting several time series (or anything that shares an axis) on top of one another. You could just put all your stuff on the same plot, but if you have many different datasets, and many different y-axis scales, this becomes messy quickly
  • Getting around limitations in gnuplot. I recently discovered that gnuplot doesn't have a direct way to plot several stacked contours in 3D: more than one countour at a time is possible, but they'll all be drawn at the same z coordinate. One way to work around this is to use multiplots to plot each contour as a separate plot, and to tell gnuplot to put each subplot in the same location on the page

Making gnuplotlib support this was conceptually simple, but violated some core assumptions of the library, so lots of typing was required to add this feature. In the end, it came out nicely, and didn't break any of the previous APIs. The big update two-fold. First I needed to separate plot options into

  • process options: one per gnuplot process. Applies to the whole big plot. Stuff like terminal or hardcopy or wait
  • subplot options: one per subplot. With multiplots we'll have many of these at the same time. These control each plot. Stuff like xlabel, xrange, 3d, etc, etc

And then all the data passed to plot() needed to be wrappable into yet-another outer list. While with a single subplot you'd do this: 

gp.plot( (x0, y0, curve_options0),
         (x1, y1, curve_options1),
         ...,
         subplot_options, process_options)

In multiplot mode, you'd pass a tuple of subplots instead of a tuple of curves, where each suplot is more or less the whole argument set from each individual plot() command (minus the process options): 

gp.plot( ( (x0, y0, curve_options0),
           (x1, y1, curve_options1),
           ...
           subplot_options0 ),

         ( (x2, y2, curve_options2),
           (x3, y3, curve_options3),
           ...
           subplot_options1 ),
         ...,
         process_options )

The gnuplotlib README and the demo have some examples. Here're some stacked contours: 

import gnuplotlib as gp
import numpy      as np

xx,yy = np.meshgrid(np.linspace(-5,5,100),
                    np.linspace(-5,5,100))
zz0 = np.sin(xx) + yy*yy/8.
zz1 = np.sin(xx) + yy*yy/10.
zz2 = np.sin(xx) + yy*yy/12.

commonset = ( 'origin 0,0',
              'size 1,1',
              'view 60,20,1,1',
              'xrange [0:100]',
              'yrange [0:100]',
              'zrange [0:150]',
              'contour base' )

gp.plot3d( (zz0, dict(_set = commonset + ('xyplane at 10' ,))),
           (zz1, dict(_set = commonset + ('xyplane at 80' ,))),
           (zz2, dict(_set = commonset + ('xyplane at 150',))),

           tuplesize=3,
           _with = np.array(('lines nosurface',
                             'labels boxed nosurface')),
           square=1,
           multiplot=True)

stacked-contours.svg

This is pretty new, so if you try it out, and something is unclear or broken, please complain.

28 November, 2019 07:30PM by Dima Kogan

hackergotchi for Jonathan McDowell

Jonathan McDowell

Native IPv6: One Month Later

I wrote a month ago about getting native IPv6 over fibre. Given that this week RIPE announced they’d run out of IPv4 allocations and today my old connection was turned off, it seems an appropriate point to look at my v4 vs v6 usage over a month (as a reminder, I used my new FTTP connection for v6 only over the past month, with v4 still going over my old FTTC connection). I’m actually surprised at the outcome:

IPv4 traffic IPv6 traffic

That’s telling me I received 1.8 times as much traffic via IPv6 over the past month as I did over IPv4. Even discounting my backups (the 2 v6 peaks), which could account for up to half of that, that means IPv6 and IPv4 are about equal. That’s with all internal networks doing both and no attempt at traffic shaping between them - everything’s free to pick their preference.

I don’t have a breakdown of what went where, but if you run a network and you’re not v6 enabled, why not? From my usage at least you’re getting towards being in the minority.

28 November, 2019 07:17PM

hackergotchi for Holger Levsen

Holger Levsen

20191128-FOSDEM-MiniDebCamp

Mini-DebCamp pre FOSDEM 2020

This is to inform you about a Mini-DebCamp happening right before FOSDEM 2020, in Brussels, which is a city in Belgium in Europe on planet earth.

The Mini-DebCamp will take place at the Hackerspace Brussels (HSBXL) from Wednesday, the 29th of January 2020 until Friday, the 31st, as part of Byteweek and you should all come around if you can!

Not so much more information can be found at the event wiki page.

The event will become what you and we make out of it!

Further discussion on debian-events-eu@lists.debian.org or #debconf-fosdem (on irc.debian.org).

Happy hacking and hoping to see you there!

28 November, 2019 04:48PM

November 27, 2019

Vincent Sanders

Twice and thrice over, as they say, good is it to repeat and review what is good.

Three years ago I wrote about using the AFL fuzzer to find bugs in several NetSurf libraries. I have repeated this exercise a couple of times since then and thought I would summarise what I found with my latest run.

I started by downloading the latest version of AFL (2.52b) and compiling it. This went as smoothly as one could hope for and I experienced no issues although having done this several times before probably helps.

libnsbmp

I started with libnsbmp which is used to render windows bmp and ico files which remains a very popular format for website Favicons. The library was built with AFL instrumentation enabled, some output directories were created for results and a main and four subordinate fuzzer instances started.

vince@workshop:libnsbmp$ LD=afl-gcc CC=afl-gcc AFL_HARDEN=1 make VARIANT=debug test
afl-cc 2.52b by <lcamtuf@google.com>
afl-cc 2.52b by <lcamtuf@google.com>
afl-cc 2.52b by <lcamtuf@google.com>
 COMPILE: src/libnsbmp.c
afl-cc 2.52b by <lcamtuf@google.com>
afl-as 2.52b by <lcamtuf@google.com>
[+] Instrumented 633 locations (64-bit, hardened mode, ratio 100%).
      AR: build-x86_64-linux-gnu-x86_64-linux-gnu-debug-lib-static/libnsbmp.a
 COMPILE: test/decode_bmp.c
afl-cc 2.52b by <lcamtuf@google.com>
afl-as 2.52b by <lcamtuf@google.com>
[+] Instrumented 57 locations (64-bit, hardened mode, ratio 100%).
    LINK: build-x86_64-linux-gnu-x86_64-linux-gnu-debug-lib-static/test_decode_bmp
afl-cc 2.52b by <lcamtuf@google.com>
 COMPILE: test/decode_ico.c
afl-cc 2.52b by <lcamtuf@google.com>
afl-as 2.52b by <lcamtuf@google.com>
[+] Instrumented 71 locations (64-bit, hardened mode, ratio 100%).
    LINK: build-x86_64-linux-gnu-x86_64-linux-gnu-debug-lib-static/test_decode_ico
afl-cc 2.52b by <lcamtuf@google.com>
Test bitmap decode
Tests:1053 Pass:1053 Error:0
Test icon decode
Tests:609 Pass:609 Error:0
    TEST: Testing complete
vince@workshop:libnsbmp$ mkdir findings_dir graph_output_dir
vince@workshop:libnsbmp$ afl-fuzz -i test/ns-afl-bmp/ -o findings_dir/ -S f02 ./build-x86_64-linux-gnu-x86_64-linux-gnu-debug-lib-static/test_decode_bmp @@ /dev/null > findings_dir/f02.log >&1 &
vince@workshop:libnsbmp$ afl-fuzz -i test/ns-afl-bmp/ -o findings_dir/ -S f03 ./build-x86_64-linux-gnu-x86_64-linux-gnu-debug-lib-static/test_decode_bmp @@ /dev/null > findings_dir/f03.log >&1 &
vince@workshop:libnsbmp$ afl-fuzz -i test/ns-afl-bmp/ -o findings_dir/ -S f04 ./build-x86_64-linux-gnu-x86_64-linux-gnu-debug-lib-static/test_decode_bmp @@ /dev/null > findings_dir/f04.log >&1 &
vince@workshop:libnsbmp$ afl-fuzz -i test/ns-afl-bmp/ -o findings_dir/ -S f05 ./build-x86_64-linux-gnu-x86_64-linux-gnu-debug-lib-static/test_decode_bmp @@ /dev/null > findings_dir/f05.log >&1 &
vince@workshop:libnsbmp$ afl-fuzz -i test/ns-afl-bmp/ -o findings_dir/ -M f01 ./build-x86_64-linux-gnu-x86_64-linux-gnu-debug-lib-static/test_decode_bmp @@ /dev/null

The number of subordinate fuzzer instances was selected to allow the system in question (AMD 2600X) to keep all the cores in use with a clock of 4GHz which gave the highest number of
AFL master instance after six days
executions per second. This might be improved with better cooling but I have not investigated this.

After five days and six hours the "cycle count" field on the master instance had changed to green which the AFL documentation suggests means the fuzzer is unlikely to discover anything new so the run was stopped.

Just before stopping the afl-whatsup tool was used to examine the state of all the running instances.

vince@workshop:libnsbmp$ afl-whatsup -s ./findings_dir/
status check tool for afl-fuzz by <lcamtuf@google.com>

Summary stats
=============

       Fuzzers alive : 5
      Total run time : 26 days, 5 hours
         Total execs : 2873 million
    Cumulative speed : 6335 execs/sec
       Pending paths : 0 faves, 0 total
  Pending per fuzzer : 0 faves, 0 total (on average)
       Crashes found : 0 locally unique

Just for completeness there is also the graph of how the fuzzer performed over the run.

AFL fuzzer performance over libnsbmp run

There were no crashes at all (and none have been detected through fuzzing since the original run) and the 78 reported hangs were checked and all actually decode in a reasonable time. It seems the fuzzer "hang" detection default is simply a little aggressive for larger images.

libnsgif

I went through a similar setup with libnsgif which is used to render the GIF image format. The run was performed on a similar system running for five days and eighteen hours. The outcome was similar to libnsbmp with no hangs or crashes.


vince@workshop:libnsgif$ afl-whatsup -s ./findings_dir/
status check tool for afl-fuzz by <lcamtuf@google.com>

Summary stats
=============

       Fuzzers alive : 5
      Total run time : 28 days, 20 hours
         Total execs : 7710 million
    Cumulative speed : 15474 execs/sec
       Pending paths : 0 faves, 0 total
  Pending per fuzzer : 0 faves, 0 total (on average)
       Crashes found : 0 locally unique

libsvgtiny

AFL fuzzer results for libsvgtiny
I then ran the fuzzer on the SVG render library using a dictionary to help the fuzzer cope with a sparse textural input format. The run was allowed to continue for almost fourteen days with no crashes or hangs detected.

In an ideal situation this run would have been allowed to continue but the system running it required a restart for maintenance.

Conclusion

The aphorism "absence of proof is not proof of absence" seems to apply to these results. While the new fuzzing runs revealed no additional failures it does not mean there are no defects in the code to find. All I can really say is that the AFL tool was unable to find any failures within the time available.

Additionally the AFL test corpus produced did not significantly change the code coverage metrics so the existing set was retained.

Will I spend the time again in future to re-run these tests? perhaps, but I think more would be gained from enabling the fuzzing of the other NetSurf libraries and picking the low hanging fruit from there than expending thousands of hours preforming these runs again.

27 November, 2019 07:00PM by Vincent Sanders (noreply@blogger.com)

hackergotchi for Daniel Silverstone

Daniel Silverstone

Rust 2020

As in recent years, there was a call for posts about Rust in 2020. I've been sitting on my response for quite a few weeks because every time I try to write this, I think of other things I want to say, or a new "theme" I want to propose for Rust in 2020.

First, some history about myself in the Rust community -- I started learning Rust a few years ago, had a go at Advent Of Code 2016 in it, completed that, wrote a testing tool in Rust, and then promptly did nothing with Rust except read blogs and reddit for a year. Then I went and did Advent of Code 2017 in Rust and reminded myself of why I enjoyed the language. This time I decided I'd do more with Rust and started playing around. I wrote bits and bobs through 2018, but still failed to do anything major until once again, December rolled around and I did Advent of Code 2018 in Rust, learning an awful lot about procedural macros and other useful things. This was the time of the 2018 Rust edition which also spurred me on to do more useful stuff.

While I was enjoying the fun coding through December, I sat down and decided how I wanted 2019 to play out. I was very aware that I was failing a number of open source communities I was part of, and so I resolved (a) to be a net positive influence on any community I remained active in, and (b) to specifically work to be part of the wider Rust community because I wanted to give back to a community which had been welcoming to me and given me so much joy in coding. I went looking for ways to contribute, did a small patch to the panic wording as per an open issue, and then looked at the developer tools surrounding Rust. I noted that rustup needed some TLC and so I ended up filing my first PR on rustup, I then filed others, and on the 18th Of December 2018, my PR to update rustup to the 2018 edition was merged and thus began my time as a rustup contributor.

Fast-forward to mid-January 2019 and the Rustup working group is established and I was invited to be a member of that. Zoom on to mid-May and I'm asked to take on the mantle of leading the working group, and from there I ended up also helping on rustdoc and various other bits and bobs. I've ended up quite involved in an exciting and interesting community which has always made me feel welcome.

With that potted history given, I hope that you can appreciate where I'm coming from when I say that the one word which keeps coming up when I think about what the Rust community needs to look toward next, the "buzzword" for Rust 2020 that I'd propose, is "Inclusivity". Interestingly I don't mean in terms of being inclusive and welcoming to LGBT people, or women, or any number of other "minority" or "marginal" groups which online communities often find themselves needing to work to include; because Rust already is exceedingly welcoming to such groups -- I've never had so much as a misplaced blink when I say "my husband" in a Rust setting, and that has been so nice. What I mean is that in order to be more widely used, the Rust community needs to look toward ensuring that it can be included into other things.

I'll list the major topic areas I'm thinking of, from most-likely to least-likely to get somewhere concrete in the coming year. At least from my perspective. I'd love to be proven wrong about some of the later items…

Getting Rust

As a Debian developer I have long enjoyed the stability and reliability of the Debian operating system, the software supplied by it, and the trust I feel I can place in it as a software distribution point. Recently Debian started to include Rust in releases, and that is incredibly important I feel. However the Rust community expects things like rustup to "just work" and that means that if someone is using a Debian provided toolchain, and ask how to get rls working then the instruction rustup component add rls simply won't work properly for them. This gets even more interesting when you look at NixOS, or any Arch Linux derivative, because they distribute rustup but it cannot update it self, so other behaviours the community expects don't work, despite rustup being there.

Getting the Rust toolchain in the first place is a critical part of the flow of getting someone into Rust, and making that possible in a way which fits into our target user's worldview and meets their expectations for acquisition of tooling is critical to reducing the onboarding friction. So my first point of inclusivity is, amusingly, on me to spearhead and work out how to make it happen; though I'd welcome anyone wanting to make suggestions on how to make it work nicely.

Trust ergonomics

One thing which holds a lot of people back from Rust is being able to acquire it in a manner they already feel they can trust. This follows on from the point above because trust has to start somewhere, and people already trust the operating system they're running, at least to some extent. When you say that the recommended method for acquiring Rust toolchains starts with a curl | bash people have recoiled in horror. I think that the Rust project needs to start to come up with ways to improve the level of trust people can put in the tooling they acquire. Some of this has begun already, with a proposal to sign the crates index which is still in progress. However we need to extend that to the Rust toolchains, and to the installer (rustup) itself too, in order to be more usable to more people.

This kind of thing is something I am actively interested in, and I hope to be able to announce something in the new year at some point. Those of you who follow the rustup issues will have seen me discussing OpenPGP signatures on toolchains, and that will certainly form part of an infrastructure which people outside of the project can build trust upon. Maybe one day we'll get Debian to sign something which Debian users can use to trust a rustup that they downloaded in the curl | bash method which we know to be one of the most accepted of "current" approaches to getting non-distro-packaged software.

Reliable, deduplicated, dependencies

One criticism which is levelled at the crates ecosystem an awful lot in my earshot, and which I doubt anyone would really argue with, is that it feels very immature -- the preponderance of 0.x version numbers for "primary" crates is something that has been worked on, but is still a huge problem. In part this is because the community is so confident with the semantic versioning we all take care of, but also because there's an amount of release anxiety manifesting. What's worse than this though is the way that we end up with sets of incompatible dependency chains on many of these "primary" crates. It is not uncommon to end up with multiple version of rand, syn, quote, or others in your dependency tree. Not all of that is the fault of the authors of those libraries either, but because other crates are not yet up-to-date with changes in them. For example, syn is already in its 1.0 series and yet via various pathways, rustup ends up depending on 0.15 as well.

In order to be more includible into distributions such as Debian, it's critical that the Rust community as a whole looks to address this kind of thing. It seems odd to say that "gardening" your library's dependencies can be a way to lead to being more includible into other things, to enhance "inclusivity" as I'm choosing to define it for this post, but it is indeed one pathway.

An end-goal of this is that a majority of tools ought to be buildable with a unified singular set of library crate versions if they are going to be usefully included into a distribution.

Making Windows feel Tier One

The Windows platform is considered Tier One by the Rust project. This means that we care, as a project, that Rust and the developer tools all work nicely on the platform. Yet it's abundantly clear that the majority of the tools developers use either a Linux or MacOS X system. As such, concerns that Windows brings due to its non-POSIX nature are often either considered unimportant or simply ignored. I am very guilty of this myself. I am responsible for rustup and it's quite possibly the most UNIXy part of the Rust ecosystem, and yet is pretty much mandatory for every Windows user.

Another example of this is the rust-docs component which contains tens of thousands of small files which causes Windows based anti-virus systems significant indigestion. We ended up adding a "minimal" profile to rustup so that Windows users could skip installing the docs, but that's not great as a long-term solution.

As Rust tries to get included into more companies, this friction is going to be more and more important to resolve. A huge number of companies expect their developers to use Windows desktops, even when they're working on Linux hosted software, and making sure that's first-class will be really important as the number of users who have to use Rust (rather than those who choose to) increases.

I'm sure there are things we can do here, but I'm really going to need help to find ways to make this happen, I'm not a Windows person, but I'm very open to finding ways to improve matters on that platform.

Shared libraries

This is something I'm not sure that we can do anything particularly concrete on in the coming year, but is part of Rust's story which really needs a lot of thought. It's incredibly hard to provide a shared object which can be versioned at all similarly to how SONAME versioning currently works, due to the complexity of the types, and the instability of Rust's ABI. However at least some thought needs to be put into how we might begin to resolve this as it's yet another potential blocker to being included into distributions because it makes the security story for Rust programs so different from other languages which can and do make use of more traditional shared objects.

In summary, for me, for 2020, Rust's already very inclusive approach to its community needs to turn outward and look for ways to increase the chances that it can be included into other projects such as Linux distributions. I see this as increasing the inclusivity of the project by including into our worldview the particular needs of these other projects and communities and ensuring that by treating them as first-class consumers of Rust, we can become first-class members of their projects and communities as well.

27 November, 2019 03:53PM by Daniel Silverstone

November 26, 2019

hackergotchi for Olivier Berger

Olivier Berger

Displaying animated diagrams in org-reveal slide decks with SVG

org-reveal can be used to author slide decks with org-mode. The slides are displayed as an HTML document animated with reveal.js.

reveal.js allows to progressively display fragments which reveal parts of a page, for instance a bullet list.

It is also possible to display animated diagrams, as reveal.js fragments, provided that such diagrams are made as embedded SVG images included in the HTML.

Adding class="fragment" (and variations, including the associated data-fragment-index attributes) to the SVG elements helps animate parts of the diagram with the same JS actions as for regular bullet fragments.

The trick is to embed the SVG diagram inside the HTML source using the following code in the org-mode source:

#+BEGIN_EXPORT html
<svg ...>  
    ...
</svg>  
#+END_EXPORT

I’ve found inspiration for that trick seeing the way SVG content was animated in the following demo.

So far, I’ve edited my diagrams with inkscape, and finally copied the whole <svg> tag from the saved inkscape file, into the #+BEGIN_EXPORT embedded HTML of the org file.

You can see a demo of such an animation in the https://olberger.gitlab.io/org-teaching/slides.html document (slide 26: “Fragmented SVG display”).

Hope this helps.

26 November, 2019 09:55PM by Olivier Berger

hackergotchi for Bits from Debian

Bits from Debian

New Debian Developers and Maintainers (September and October 2019)

The following contributors got their Debian Developer accounts in the last two months:

  • Teus Benschop (teusbenschop)
  • Nick Morrott (nickm)
  • Ondřej Kobližek (kobla)
  • Clément Hermann (nodens)
  • Gordon Ball (chronitis)

The following contributors were added as Debian Maintainers in the last two months:

  • Nikos Tsipinakis
  • Joan Lledó
  • Baptiste Beauplat
  • Jianfeng Li

Congratulations!

26 November, 2019 02:00PM by Jean-Pierre Giraud

hackergotchi for Romain Perier

Romain Perier

RaspberryPi 3 desktop image running Debian Buster 10.2

Hi !

The Capitole Du Libre 2019 happened two weekend ago on 16th-17th November. I have helped the DebianFrance team on the Debian booth. It was a super event with a lot of cool and interesting people !
We promoted the Debian project and I have also presented a demo of a Raspberry PI 3 running a fresh debian 10.2 with a mainline linux kernel (for instance 4.19.x) with V3D or llvmpipe.  The different persons I met were very interested by the availability of the image, so I have decided to write a debos recipe from scratch for ARM64-based RaspberryPI.




This is a desktop oriented image. It will start an lxde session automatically with the "pi" user. The root accound is locked but you can use sudo with the "pi" user.

You can find the recipe here . You can build an image ready to flash with the following commands:

$ git clone https://salsa.debian.org/raspi-team/image-specs.git
$ cd image-specs/debos
$ debos raspberrypi-desktop_arm64.yaml

It should produce an image raspberrypi-desktop_buster_arm64.img.gz. Then, supposing that the device associated with your sdcard is /dev/mmcblk0 , you can then flash this image by typing:

$ zcat raspberrypi-desktop_buster_arm64.img.gz | sudo dd of=/dev/mmcblk0 bs=4M status=progress
$ sync

The VC4 graphical stack being still unstable in some cases, it is recommended to use llvmpipe for now (I have to investigate, I don't know yet if this is a bug in the DRI driver in mesa or a bug in the drm/kms driver).

A recipe for running bullseye should come quickly :)


Enjoy !

26 November, 2019 12:24PM by Romain Perier (noreply@blogger.com)

November 25, 2019

Petter Reinholdtsen

What would it cost to store all 2018 phone calls in Norway?

Four years ago, I did a back of the envelope calculation on how much it would cost to store audio recordings of all the phone calls in Norway, and came up with NOK 2.1 million / EUR 250 000 for the year 2013. It is time to repeat the calculation using updated numbers. The calculation is based on how much data storage is needed for each minute of audio, how many minutes all the calls in Norway sums up to, multiplied by the cost of data storage.

The number of phone call minutes for 2018 was fetched from the NKOM statistics site, and for 2018, land line calls are listed as 434 238 000 minutes, while mobile phone calls are listed with 7 542 006 000 minutes. The total number of minutes is thus 7 976 244 000. For simplicity, I decided to ignore any advantages in audio compression the last four years, and continue to assume 60 Kbytes/min as the last time.

Storage prices still varies a lot, but as last time, I decide to take a reasonable big and cheap hard drive, and double its price to include the surrounding costs into account. A 10 TB disk cost less than 4500 NOK / 450 EUR these days, and doubling it give 9000 NOK per 10 TB.

So, with the parameters in place, lets update the old table estimating cost for calls in a given year:

YearCall minutesSizePrice in NOK / EUR
200524 000 000 0001.3 PiB1 170 000 / 117 000
201218 000 000 0001.0 PiB900 000 / 90 000
201317 000 000 000950 TiB855 000 / 85 500
20187 976 244 000445 TiB401 100 / 40 110

Both the cost of storage and the number of phone call minutes have dropped since the last time, bringing the cost down to a level where I guess even small organizations can afford to store the audio recording from every phone call taken in a year in Norway. Of course, this is just the cost of buying the storage equipment. Maintenance, need to be included as well, but the volume of a single year is about a single rack of hard drives, so it is not much more than I could fit in my own home. Wonder how much the electricity bill would raise if I had that kind of storage? I doubt it would be more than a few tens of thousand NOK per year.

25 November, 2019 07:15PM

Ian Jackson

subdirmk - ergonomic preprocessing assistant for non-recursive make

Problem

Peter Miller's 1997 essay Recursive Make Considered Harmful persuasively argues that it is better to arrange to have a single make invocation with the project's complete dependency tree, rather than the currently conventional $(MAKE) -C subdirectory approach.

These problems are not theoretical for me. In the Xen Project we use recursive make and sadly suffer from occasional concurrency bugs. In my work on secnet (the currently rather unproductised greenend.org.uk vpn program) I have been frustrated by unreliability of the build system (which is developing fairly rapidly, as I overhaul secnet) .

However, actually writing a project's build system in a non-recursive style is not very ergonomic. I was complaining about this in the pub a week or two ago. Accepting my challenge, Mark Wooding demonstrated a proof of concept showing that it was possible to do better. I thought I had a better approach so I took his code and I ran with it.

Solution

I stole the sigil & to mean "directory local".

The result is subdirmk, which is an arrangment for preprocessing your per-subdirectory makefiles (really, makefile fragments) to prefix file and variable names with the name of the fragment's directory.

It's designed to be incorporated into your project via git  subtree (which is excellent and not to be confused with the awful git submodule). subdirmk is implemented in (very conservative) perl, which is ubiquitious nowadays. It expects you to be using autoconf, but not automake (I am not much of a fan of automake; I don't think its extra complexity buys much, especially now that make dist can be replaced with git archive).

I'm very pleased with the result. It's small and pleasant to use. Although it's not stabilised, I have already adopted it for secnet. But if you want to see an example of its use you probably want to look at the included toy example project (which is also used for subdirmk's self-tests).

There are a variety of &-escapes. The complete syntax etc. is documented in the current README.

Status

As I say, this is not yet stabilised. If you adopt it now, you should be prepared to make systematic changes to your project when you merge new versions of subdirmk. (git-subtree makes this fairly easy and I would be happy to advise eg with git-fu.)

Questions

I have a few things you might like to bikeshed:

Is my set of substitution syntaxes the best set? See "Tables of file reference syntaxes" and "Substitution Syntax" in the README. I'm particularly unsure about &=_ and &=/ (but there has to be a way to ask for these as they are the most primitive things the filter can provide) and the choice of ! in &!.

Please comment on my choices of filenames and extensions. (The input files to subdirmk must end in .mk, rather than being something like .mk.in.sd, so that editors choose makefile syntax mode.)

Should I introduce &$VARIABLE as a shorthand for $(&VARIABLE) ? If so I guess characters matching \w are part of the variable name.

The &TARGETS_foo thing is a bit of a wrinkle, especially the way that mentioning it in a #-comment has a side effect. (See line 22 in the provided clean.sd.mk for an example.)

Makefile fragment inclusion order is often relevant. Should I add a Perdir-top.sd.mk which works like Perdir.sd.mk but is included first rather than last?

I do not currently have a convenient way to write a reference to the build or install toplevel which is relative to one of the build subdirectories. This is sometimes needed for symlinks, or if a command wants to cd to the subdirectory and then refer to files elsewhere. autoconf's @top_builddir@ and @abs_top_srcdir@ and so on can be used. This seems rare enough that this is OK.

GNU make's metaprogramming facilities are rather awkward. You end up expanding macros with $(eval $(call ... )) and you have to $-double everything in the macro. (Just using $(call ...) often doesn't work because that construct cannot generate multiple separate rules etc.) I am tempted to add something to subdirmk to make that easier. It's not so simple because subdirmk's filtering engine is purely textual and does not understand make syntax, and runs before make so there is no access to make variables. (And I certainly don't want to add a third variable namespace to all the makefiles.) Also any facility of this kind would be available only in .sd.mk files and not .mk.in ones (but maybe that doesn't matter).

Edited 2019-11-25 15:19 to fix a few typos and make a minor clarification



comment count unavailable comments

25 November, 2019 03:21PM

Uwe Kleine-König

Philips Hue Bridge v2.1

I recently bought a Hue Bridge to experiment a bit with Zigbee and 802.15.4. Following two posts for the hardware version 2.0 and some comments about the differences to version 2.1 I was able to get shell access on my 2.1 hardware.

As there is up to now no complete guide I describe here, what I did:

Opening the case is straigth forward. Just remove the two lower nubsis at the bottom and unscrew the two torx screws; then carefully unclip the bottom.

PCB

The header on the left side on the image gives access to the UART. Pin 1 is ground, pin 4 is the board's RX (so connect your adapter's TX) and pin 5 is the board's TX. The voltage level is 3.3 V. Note however that the board doesn't boot (after a cold start) with its TX connected so leave that unconnected until after the bootloader is up.

To stop the bootloader short the test point marked with a red cycle on the image near the NAND flash to ground for a few seconds when plugging in the power cord. I used pin 14 of SJ8 (the unassembled header on the left above the factory reset button). This makes the bootloader fail to read the kernel image from NAND and then drop to a shell.

Then after connecting RX you can do there:

setenv bootdelay 3
setenv security '$5$uaS2DwZOsch$2H76tTh0yLzH8LfIkg7z1icoYZLeLGV.Gpo3quhZSe2'
saveenv
reset

After that the machine boots up and you can login with username root and password lala. (If you want a different password, set security in U-Boot accordingly. I did

mkpasswd -m sha-256

to generate the necessary string.)

After contacting Philips asking for the GPL code running on the bridge I got a code drop and looked a bit around in the source code. Here are some findings:

Having the boards UART TX connected to an USB adapter influences the register RST_BOOTSTRAP at offset 0x180600b0. This is used to determine how memory is initialized (in function ath_ddr_initial_config defined in qsdk/qca/src/qca-legacy-uboot/board/atheros/common/init-953x.c). The naming of the register together with the board's behaviour suggests that is it sampled at cold boot only and bit0 then contains the state of the TX pin. So I guess the machine not (cold) booting with TX connected could be fixed with a hardware modification removing (or weakening) a pullup on my USB adapter. But I'm more of a software guy with infinitesimal solder skills, so I will live with reconnecting TX for each cold boot. Alternatively patching the bootloader to ignore the RST_BOOTSTRAP might be an option.

I will keep you updated on any progress here.

25 November, 2019 03:00PM

NOKUBI Takatsugu

Cross Distibution Devloper Camp 2019 Fall

I had attend to Cross Distribution Developer Camp 2019 Fall at Manazuru-machi, Kanagawa at 16th Nov. 2019.
There was 8 developers from openSUSE, Ubuntu, LibreOffice and Debian. We had talked about some problem between such distributions, upstream software, especially Japanese (or CJK) specific things.

OLYMPUS DIGITAL CAMERA

The town is a fishing port so good sight, sashimi was tasty.

I had developed to nelogd-package, a tool to make mecab-ipadic-neologd as a debian package, but not completed.

25 November, 2019 06:05AM by knok

November 24, 2019

hackergotchi for Joachim Breitner

Joachim Breitner

Faster Winter: Statistics (the making-of)

(This is an appendix to the “faster winter” series, please see that post for background information.)

Did you like the graph and the stats that I produced? Just for completeness, I am including the various scripts I used. Nothing super exciting to see here, but maybe someone finds this useful.

This little shell one-liner collects the run-time statistics for each commit in the interesting range (line-wrapped for your convenience):

for h in $(git log 1cea7652f48fad348af914cb6a56b39f8dd99c6a^..5406efd9e057aebdcf94d14b1bc6b5469454faf3 --format=%H)
do
  echo -n "$h"
  git checkout -q "$h"
  cabal new-build -v0
  echo -n ":"
  rm -f stats/$h.txt
  for i in $(seq 1 5)
  do
    cabal -v0 new-run exe:wasm-invoke -- -w loop.wasm  -f canister_init +RTS -t >/dev/null 2>> stats/$h.txt
    echo -n .
  done
  echo
done

A small Perl script takes the minimum for each measurement across the five runs, and produces a CSV file:

#!/usr/bin/perl

use List::Util qw(min);

my @alloc;
my @in_use;
my @time;

while (<>) {
  m!<<ghc: (\d+) bytes, \d+ GCs, \d+/\d+ avg/max bytes residency \(\d+ samples\), (\d+)M in use, [\d.]+ INIT \(([\d.]+) elapsed\), [\d.]+ MUT \(([\d.]+) elapsed\), [\d.]+ GC \(([\d.]+) elapsed\) :ghc>>! or die $!;
  push @alloc, 0+$1;
  push @in_use, $2;
  push @time, $3+$4+$5;
}

printf "%d;%d;%f\n", min(@alloc), min(@in_use), min(@time);

To create a full file for all the commits in the range that have files, I used this bash one-liner (again line-wrapped for your convenience):

echo 'commit;allocations;memory;time' > stats.csv
for h in $(git log 1cea7652f48fad348af914cb6a56b39f8dd99c6a^..5406efd9e057aebdcf94d14b1bc6b5469454faf3 --format=%H|tac)
do
  git log -n 1 --oneline $h
  test -f stats/$h.txt && echo "$(echo $h|cut -c-7);$(./tally.pl < stats/$h.txt)" | tee -a stats.csv
done

The stats can be turned into the graphc using pgfplots by compiling this LaTeX file:

\documentclass[class=minimal]{standalone}
\usepackage{mathpazo}
\usepackage{pgfplots}
\definecolor{skyblue1}{rgb}{0.447,0.624,0.812}
\definecolor{scarletred1}{rgb}{0.937,0.161,0.161}
\pgfplotsset{width=12cm,compat=newest}

% From https://tex.stackexchange.com/a/63340/15107
\makeatletter
\pgfplotsset{
    /pgfplots/flexible xticklabels from table/.code n args={3}{%
        \pgfplotstableread[#3]{#1}\coordinate@table
        \pgfplotstablegetcolumn{#2}\of{\coordinate@table}\to\pgfplots@xticklabels
        \let\pgfplots@xticklabel=\pgfplots@user@ticklabel@list@x
    }
}
\makeatother

\begin{document}
\begin{tikzpicture}

\pgfplotsset{every axis/.style={ymin=0}}
\begin{semilogyaxis}[
  skyblue1,
  scale only axis,
  axis y line*=left,
  ylabel=Allocation (bytes),
  flexible xticklabels from table={stats.csv}{[index]0}{col sep=semicolon},
  xticklabel style={rotate=90, anchor=east, text height=1.5ex, font=\ttfamily, color=black},
  xtick=data,
  ]
\addplot[const plot mark mid, color=skyblue1]
  table [x expr=\coordindex+1, y index=1, col sep=semicolon] {stats.csv};
\end{semilogyaxis}

\begin{semilogyaxis}[
  green,
  scale only axis,
  axis y line*=right,
  ylabel=Memory (MB),
  x tick style={draw=none},
  xtick=\empty,
  ]
\addplot[const plot mark mid, color=green]
  table [x expr=\coordindex+1, y index=2, col sep=semicolon] {stats.csv};
\end{semilogyaxis}


\begin{semilogyaxis}[
  red,
  scale only axis,
  axis y line*=right,
  ylabel=Time (seconds),
  x tick style={draw=none},
  xtick=\empty,
  ]
\pgfplotsset{every outer y axis line/.style={xshift=2cm}, every tick/.style={xshift=2cm}, every y tick label/.style={xshift=2cm} }
\addplot[const plot mark mid, color=red]
  table [x expr=\coordindex+1, y index=3, col sep=semicolon] {stats.csv};
\end{semilogyaxis}
\end{tikzpicture}
\end{document}

And finally this Perl script allows me to paste any two lines from the CSV file and produces appropriate Markdown for the “improvement” lines in my posts:

#!/usr/bin/perl

my $first = 1;

my $commit;
my $alloc;
my $in_use;
my $time;

while (<>) {
  /(.*);(.*);(.*);(.*)/ or die;
  unless ($first) {
    printf "**Improvement**: Allocations: %+.2f%%  Memory: %+.2f%%  Time: %+.2f%% (Commit [%s...%s](http://github.com/dfinity/winter/compare/%s...%s))\n",
      (100 * ($2/$alloc - 1)),
      (100 * ($3/$in_use - 1)),
      (100 * ($4/$time - 1)),
      $commit,
      $1,
      $commit,
      $1;
  }
  $first = 0;
  $commit = $1;
  $alloc = $2;
  $in_use = $3;
  $time = $4;
}

24 November, 2019 10:03PM by Joachim Breitner (mail@joachim-breitner.de)

Faster Winter 7: The Zipper

(This is the seventh optimization presented in the “faster winter” series, please see that post for background information.)

The last bit of performance could be considered a domain-specific optimization, as one might describe it as “introducing a control stack to the interpreter”. But in a different light, one could consider it the introduction of a more appropriate data structure, by applying a “zipper”-like transformation to the existing data structure.

The problem is that the state of the system (datatype Code) is a stack of values and a stack of instructions. Simplifying this for this post, we might have

data Code = Code [Value] [Instr]
data Instr
  = Const Value | Add | Sub | Return
  | Labled Int Code

The interpreter gets such a Code, but does not always just execute the top-most instruction: If that is a Labeled instruction, it has to execute the next instruction in the argument of that Labeled. (It is not very important at this point to understand what a Labeled is used for). So we might have a Code that looks like the following:

c1 = Code [] [Labeled 1 (Code [2] [Labeled 2 (Code [3,4] [Add]), Sub]), Return]

The next instruction to execute is actually the Add. But in order to find that, the function step has to look under the first Labeled, look under the second Labeled, then execute step (Code [3,4] [Add]) = Code [7] [], and finally wrap it again in the two Labeled, to produce:

c2 = Code [] [Labeled 1 (Code [2] [Labeled 2 (Code [7] []), Sub]), Return]

Then eval calls step again, and step has to look inside the Labeled again to find the next instruction to execute.

It would be easier if the next instruction to execute would be presented to step more conveniently, right as a field of Code. But in order to do this, we have to move the Labeled entries “out of the way”. I do that by adding a new, first parameter to Code where I keep a stack of all the Label constructor that were in the way, in reverse order. So the c1 above would now be

data Code = Code Control [Value] Instr
data Control = Empty | Labeled Int Code

c1' = Code (Labeled 2 (Code (Labeled 1 (Code Empty [] [Return])) [2] [Sub]) [3,4] [Add]

Can you see how this relates to c1 above? The important piece here is that the interpreter finds the next instruction to execute always at the head of the instruction list right of the outermost code, and as long as there is something to execute there, it doesn't have to touch the Control stack at all.

This required touching some more lines of code than the previous optimizations, but doing so didn't require much creativity, as the old and new Code types are in clear correspondance, and that guides me in how to use adjust the code. And it’s certainly worth it:

Improvement: Allocations: -46.17% Time: -36.88% (Commit e66f1e0...57f3e9d)

24 November, 2019 09:57PM by Joachim Breitner (mail@joachim-breitner.de)

Faster Winter 6: Simpler Code

(This is the sixth optimization presented in the “faster winter” series, please see that post for background information.)

The Wasm reference interpreter has a function step that performs one single one step of evaluation, by taking the state before (in a type called code), and returning the state after the step. The function eval calls step, checks if the result is “done” (no instructions left to do), and if it is not done, recursively calls eval again. This way, evaluation progresses step by step until it is done.

The Haskell port follows the same pattern of a single-step step and a driver eval, but it chose to write the code continuation passing style: Instead of returning, the function step takes a function that it passes the result to. So the code looks like this (slightly simplified):

type CEvalT m a = ReaderT Config (ExceptT EvalError m) a

step :: PrimMonad => Code -> (Code -> CEvalT m r) -> CEvalT m r
step c k =  … k new_c …

eval :: PrimMonad => Code -> CEvalT m (Stack Value)
eval c =
    if is_done c
    then stack c
    else step c eval

There must have been a good reason to prefer this style over the plain style, but I was curious if it was really helpful. So I changed it to the following, more straight-forward code:

type CEvalT m a = ReaderT Config (ExceptT EvalError m) a

step :: PrimMonad => Code -> CEvalT m Code
step c k =  … new_c …

eval :: PrimMonad => Code f m -> CEvalT f m (Stack Value)
eval c =
    if is_done c
    then stack c
    else step c >>= eval

And indeed, the simpler code worked better:

Improvement: Allocations: -9.6 Time: -16.91% (Commit eb8add3...e66f1e0)

Again, avoiding function values (as we construct them as the contination closure) might have helped here.

Or maybe the simpler code allowed GHC to notice that the Code value, which is simply a tuple with a different name, is constructed by step but immediatelly deconstructed by eval, and thus GHC could optimize that away (by “unboxing” the argument and/or result of step and simply passing the components of the tuple).

And finally, independent of all the performance questions, it also makes the code easier to understand.

24 November, 2019 11:55AM by Joachim Breitner (mail@joachim-breitner.de)

November 23, 2019

Molly de Blanc

Health care

One of the most important issues for free software within the US is one we rarely talk about: healthcare. That is why I am going to write about it.

These days, sustainability in FOSS is a hot topic. In my experience, for many years this conversation focused nearly exclusively on making FOSS -profitable- for companies, in order to create jobs. Now, the conversation is shifting to ask: what conditions do we need to create so that everyone who wants to work in FOSS can do so?

The answer is not the same for everyone, nor is it the same in every country. Someone supporting a family of two, three, four, or however many has greater income needs than I do, as my biggest financial responsibilities are debt and a cat. However, I also have a condition with a mortality rate estimated at 15%. Access to affordable, comprehensive health care is not just a nice perk, but crucial for my continued survival.

Access to health insurance has been the primary factor in all of my professional decisions: staying places where I was miserable, doing work I hated, even choosing where to apply. Access to health insurance was a major factor in my moving to Massachusetts, which offers health insurance to all residents.

While my free software career has been amazing — I am extremely lucky that I was able to cultivate a skill set and social network that enabled me to work in the greater sphere of FOSS (and previously open ed) — I would have made different decisions had I not been reliant on employers to provide me with health insurance.

In the United States (and many, many other places), access to affordable, comprehensive healthcare (from here on: healthcare) is a major factor holding people back from FOSS contribution. When your access to health care is tied to your employer, your time — and literally your life — is dependent on your employer. This seriously impacts your ability to even have free time, let alone using that time to build FOSS.

Since the creation of software largely relies on people’s professional skill sets, we’re asking people to do in their free time what they do in their paid time — design, develop software, plan architecture, organize events, maintain systems and infrastructure, be a lawyer, manage finances, and everything else that strengthens FOSS and FOSS communities. In asking someone to take on a leadership role in a FOSS project or community, you’re asking them to take on another job — one that comes with neither pay nor benefits.

When people face constant threats to their existence due to fearing for their lives (i.e. their health), it can be hard, if not impossible to spend their time contributing to FOSS, or indeed to any activist project.

People who live in societies that rise to meet the basic material needs of all citizens are able to spend time contributing to the greater good. Those of us struggling to survive, however, must forgo opportunities to become participating members of communities that are trying to change the world. Instead, we look to our employers (usually with commercial interests) to meet our needs.

When you work in tech, meeting our basic material needs through employer-sponsored insurance comes at a steep price: non-compete agreements, signing away patent and intellectual property rights, fights to ensure your work is available under a free and/or open source license, and giving up more than 8 hours a day/40 hours a week. When we try to create good FOSS in addition to that, we burn out, we become miserable, and we’re trapped.

People are incapable of creating FOSS when they’re sick, burnt out, worried about their health, struggling with an ongoing condition or disability, or dead. It’s that simple. [powerful]

People fighting for access to healthcare should care about free software for many reasons, but we as a free software community also need to care about access to health care. This is for the sake of ourselves and the sake of our communities. We cannot build the tools and resources the world needs when we are struggling simply to live.

If you accept the notion that lack of access to comprehensive healthcare impacts our ability to have the resources necessary to create something like free software, then we can acknowledge that, by providing health care to everyone, everyone will then be in a better, more equitable position from which they can contribute to FOSS and lead safer, happier lives.

According to the KHN, 8.5% of U.S. Americans didn’t have health insurance in 2018. Un-insurance rates are even higher among non-white populations according to HHS. As a community, we’ve accepted that the lack of diversity and the over-representation of cis white folks is a problem. We need to create more equitable conditions — so that people come to FOSS from similar places of privilege, rather than having a huge disparity in privilege and oppression. Providing health care to everyone will help alleviate this, and will enable more people to do the things they are passionate about — or things they will become passionate about once they have the chance to do so.

If we are to create a world where FOSS is successful, access to health care is paramount and we need to take it seriously.

23 November, 2019 05:23PM by mollydb

hackergotchi for Dirk Eddelbuettel

Dirk Eddelbuettel

digest 0.6.23: More sha1 refinements

Another new version of digest got onto CRAN earlier today, and was also uploaded to Debian.

digest creates hash digests of arbitrary R objects (using the md5, sha-1, sha-256, sha-512, crc32, xxhash32, xxhash64, murmur32, and spookyhash algorithms) permitting easy comparison of R language objects. It is a fairly widely-used package (currently listed at 868k monthly downloads) as many tasks may involve caching of objects for which it provides convenient general-purpose hash key generation.

This release comes again one month after the previous release and contains further refinements for sha1 digests, thanks once more to Bill Denney and Thierry Onkelinx who added more support for formulae and better support for object attributes. I added some refinement for the encoding / utf8 filename tests from the previous to round things off.

CRANberries provides the usual summary of changes to the previous version.

For questions or comments use the issue tracker off the GitHub repo.

If you like this or other open-source work I do, you can now sponsor me at GitHub. For the first year, GitHub will match your contributions.

This post by Dirk Eddelbuettel originated on his Thinking inside the box blog. Please report excessive re-aggregation in third-party for-profit settings.

23 November, 2019 04:49PM

hackergotchi for Vincent Bernat

Vincent Bernat

Zsh prompt with asynchronous Git status

Zsh ships vcs_info, a function fetching information about the VCS state for the current directory and populating a variable that can be used in a shell prompt. It supports several VCS, including Git and SVN. Here is an example of configuration:

autoload -Uz vcs_info
zstyle ':vcs_info:*' enable git

() {
    local formats="${PRCH[branch]} %b%c%u"
    local actionformats="${formats}%{${fg[default]}%} ${PRCH[sep]} %{${fg[green]}%}%a"
    zstyle ':vcs_info:*:*' formats           $formats
    zstyle ':vcs_info:*:*' actionformats     $actionformats
    zstyle ':vcs_info:*:*' stagedstr         "%{${fg[green]}%}${PRCH[circle]}"
    zstyle ':vcs_info:*:*' unstagedstr       "%{${fg[yellow]}%}${PRCH[circle]}"
    zstyle ':vcs_info:*:*' check-for-changes true
}

add-zsh-hook precmd vcs_info

You can use ${vcs_info_msg_0_} in your prompt to display the current branch, the presence of staged and unstaged changes, as well as the ongoing action.1 Have a look at the documentation for more details.

Prompt with Git-related information, including branch name and presence of tracked and untracked changes
Example of prompt including information from the vcs_info function.

On large repositories, some information are expensive to fetch. While vcs_info queries Git, interactions with Zsh are stuck. A possible solution is to execute vcs_info asynchronously with zsh-async.

The first step is to define a wrapper around vcs_info. This wrapper will run into a separate process and should communicate its result using the standard output. It expects the current directory as its first argument.

_vbe_vcs_info() {
    cd -q $1
    vcs_info
    print ${vcs_info_msg_0_}
}

The second step is to define a worker, vcs_info, and attach a function to handle the result received from the wrapper. The registered function calls zle reset-prompt to force a refresh of the prompt with the updated information from ${vcs_info_msg_0_}.

source $ZSH/.../async.zsh
async_init
async_start_worker vcs_info
async_register_callback vcs_info _vbe_vcs_info_done

_vbe_vcs_info_done() {
    local stdout=$3
    vcs_info_msg_0_=$stdout
    zle reset-prompt
}

The last step is to schedule the wrapper function in the worker queue before displaying the prompt. This replaces the synchronous invocation of vcs_info:

add-zsh-hook precmd (){
    async_job vcs_info _vbe_vcs_info $PWD
}

That’s it!

Without relying on vcs_info, it should be possible to have a better experience by fetching the current branch name before retrieving the more expensive information. However, without much effort, this simple integration can make your prompt snappier! Have a look at the complete code: it contains some small enhancements.

  1. Actions include “merge,” “rebase” and “bisect.” ↩︎

23 November, 2019 02:21PM by Vincent Bernat

hackergotchi for Gunnar Wolf

Gunnar Wolf

Key migration: rsa4096/0x673A03E4C1DB921F → ed25519/0x2404C9546E145360

Oh, the joys of life... I see myself forced to do a key migration.

No, no — Don't worry! My key didn't land in any hostile party's hands. And I still kinda-sorta-have access to it.

Let me explain. I was quite a happy user of a Yubikey, kindly given to me in mid-2018. As the recommendation goes, I backed up my master key's secret material to an offline media, and kept the relevant subkeys in the Yubikey; I love knowing my computer does not have access to the private keys although it can use them — The Yubikey provides just the needed interfaces for them. And here they are:

$ gpg --list-secret-keys 
/home/gwolf/.gnupg/pubring.gpg
------------------------------
sec#  rsa4096/0x673A03E4C1DB921F 2009-07-09 [SC] [expires: 2020-12-19]
      Key fingerprint = AB41 C1C6 8AFD 668C A045  EBF8 673A 03E4 C1DB 921F
uid                   [ultimate] Gunnar Eyal Wolf Iszaevich 
uid                   [ultimate] Gunnar Eyal Wolf Iszaevich 
uid                   [ultimate] Gunnar Eyal Wolf Iszaevich (Instituto de Investigaciones Económicas UNAM) 
ssb>  rsa4096/0x92853D8CF7F6543F 2009-07-09 [E] [expires: 2020-12-19]
ssb>  rsa4096/0x80382A731F474556 2018-07-31 [E] [expires: 2020-12-19]
ssb>  rsa4096/0xA5F64FDEB981CD8C 2018-07-31 [S] [expires: 2020-12-19]
ssb>  rsa4096/0x49DD2A4E4979619C 2018-07-31 [S] [expires: 2020-12-19]
$ gpg --card-status 
(...)
Signature key ....: FA42 3AA0 6D8F E9ED 5D6C  5E42 A5F6 4FDE B981 CD8C
      created ....: 2018-07-31 03:29:09
Encryption key....: 0DE6 49DF 2778 E904 94B6  7952 9285 3D8C F7F6 543F
      created ....: 2009-07-09 23:20:40
Authentication key: 7C79 5E53 9968 8DDF 66F7  D620 49DD 2A4E 4979 619C
      created ....: 2018-07-31 03:31:16
General key info..: sub  rsa4096/0xA5F64FDEB981CD8C 2018-07-31 Gunnar Eyal Wolf Iszaevich 
sec#  rsa4096/0x673A03E4C1DB921F  created: 2009-07-09  expires: 2020-12-19
ssb>  rsa4096/0x92853D8CF7F6543F  created: 2009-07-09  expires: 2020-12-19
                                  card-no: 0006 05009847
ssb>  rsa4096/0x80382A731F474556  created: 2018-07-31  expires: 2020-12-19
                                  card-no: 0006 05009847
ssb>  rsa4096/0xA5F64FDEB981CD8C  created: 2018-07-31  expires: 2020-12-19
                                  card-no: 0006 05009847
ssb>  rsa4096/0x49DD2A4E4979619C  created: 2018-07-31  expires: 2020-12-19
                                  card-no: 0006 05009847

Until... One sad day, I discovered I could not decrypt documents sent to me anymore. While signing and encrypting do work:

$ date | gpg --encrypt --recipient 0x673A03E4C1DB921F --armor
-----BEGIN PGP MESSAGE-----

hQIMA4A4KnMfR0VWAQ/+J/Onblw3M/PwZ6ekhz3Ojnzf3pxlObLcNNNZMOlvUApK
uOAAMQ/YF/cueUbxSZT8+Yt4HV8iijVvI1Y4AK7ELvcjxdBhvmfc5QBG22lEaKPh
XNK+MUMv0xN4eIJh+uksZc6mdM6IZfvx1Io5RWiD6hABfQF7XyRwxF584PCqHMeq
oiYg1VY6epIWtunSfVa+CpF7MZ3KSTcs2VJdsutVuGxeS+NS7JVCApiXig4qk4yk
/Y/57RAy689vMNxu78noIP/mOnd7zV4DgRQ/l1iNaZnAMFgEeh1FtSJLH5odD7T0
KyfVTwdRHTJetedDjCIN+5bboNj4hIjjW4+l8KJP3aP/f8ppZuycHQ9mEf/YCuPn
KPeo7YKLGYKlerPylfENokaG7SkNyvd6UV1TdZ5eilTGBiSZpNAdlaHP0uaICbPA
Mpbn4COy52H2VTg/J/Wle1IyLPFJpGe87o/UEkylSoARMnhhtq5+Mm5mrL8k1Y5m
GvKqd8q7y7Bche+l5o2QRTZpLSfwZZ28r9qTX8S0vh+NvBD14iVNReR+RRZuf9FU
Qhgz3TzfSEwFngced24LmIXQNgtN/QBynRGJDgR0Kq9Su4/uv8VVt4N4I8wlU3qG
NwpK44FBlHPSZ6UQga/Y3Pckbrn8+RoOuIyMC0ExtDpB6qawfbssR+2EfNmPeWjS
WwHGJQ8PJ2fadWMJewhfXDVdio7LVZHqIUdJIasKB9Z3PCWoJUtrsEw0EHQxG3CZ
SN2QlLPBwveFtK0HQW6ZS29gGq6Z9Kj2la64NCJMtnYSq4RDV2wGaRr027E=
=bWqr
-----END PGP MESSAGE-----
$ date | gpg --clearsign 
gpg: using "C1DB921F" as default secret key for signing
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA256

Fri 22 Nov 2019 06:31:42 PM CST
-----BEGIN PGP SIGNATURE-----

iQIzBAEBCAAdFiEEfHleU5lojd9m99YgSd0qTkl5YZwFAl3Yfe4ACgkQSd0qTkl5
YZyTqhAAhToJg5byHBK37S4ixLc8w6mJWLlLOVvmDUmiXD2EbZoHEyCaVXJNCR+O
RveuDjfOpM/I4InrHaypGjzc7aO2yhcPnH+8adE6JBfakNdvk13XMynXggE3kqxL
oe0ZEx7sLiJaw/LzSr70kHBAqN4wnzx14+nKSuWYqZIdjR8eko867edBPWDZZgEn
nM+CSpd6XQbksf4BQR7S+/XmRKoeXuksPbmWC2uOsK9dMqGDhU91YY/j1eh3cMzq
4o/EOdc1grsfl4e8uV9z3nylOQP/PzXEp/fxXhU4lTVjVrjkBkOkDhO0DnGk8QD9
GQzdk3o6xt1gUxAGfthquRS/8ozdeL0lQ8CMaRGLRCiZKxO0kdtNkJ8QJNKxx/3M
CM/qHUnWWzyU078ncx0RkvZdwD1fq8eGM2RsptPjdeBc5Wgz7wAI5MpaUQezMrdA
/7rtJ7EnZL6WN33nwQjAC0MxoKSA073Fo8CqtBiqXkCtfp4x5x9FRpndgi7iKCNt
gQibgCIGYS6R09Q/F6r2uDcX84+SXzRyfftbP56k/4dyByn1ei7EvOM9Yi3ToU0p
qhAqLLqrG1omInTsQFGMgy/iRE7giT64YgKThdFNdT2DSNqoEkG581nabUoac8Hm
EFqGUDPn3rkPEZq6KNRTXKlMEFj7SF5v95iHWuh8OU+nPEWF1sY=
=2RSg
-----END PGP SIGNATURE-----

trying to decrypt the message does not get me very far:

$ date | gpg --encrypt --recipient 0x673A03E4C1DB921F --armor | gpg --decrypt
gpg: encrypted with 4096-bit RSA key, ID 0x80382A731F474556, created 2018-07-31
      "Gunnar Eyal Wolf Iszaevich "
gpg: public key decryption failed: Hardware problem
gpg: decryption failed: No secret key

And although the message is quite clear (public key decryption failed: Hardware problem), I spent far too many attempts at putting things upside down, trying and trying and trying to fix the issue. But no: Hardware problem means hardware problem. My Yubikey is somehow dead.

But it seems that... Even if I was able to bring it back from the dead, I would be doomed anyway: The USB key where I kept the backup for the master key material refuses to be read. Of course, I also gave it several attempts... All failed ☹ And, of course, I had it on just a single media ☹ So even getting the Yubikey decryption back to life would only allow me to use my key until 2020-12-19.

So... What's left for me to do? I just generated a shiny new elliptic-curve key, and will as soon as possible migrate my Debian credentials to use it. Please note, I am not able to sign my new key with the old one, as only the master key has Certification ability. So, the next best thing is a migration statement. I am inlining it here for convenience; if you want to check it, you can either:

$ wget https://gwolf.org/files/transition_statement.asc -O - | gpg --verify

Or just run gpg --verify and paste as its input the following text:

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA256

I am Gunnar Wolf, and I am transitioning away from my
rsa4096/0x673A03E4C1DB921F key, to ed25519/0x2404C9546E145360. The
reason for this transition is two simultaneous cases(!) of broken
hardware.

My old key is still usable until its expiry date, but I am unable to
use it for decryption; please use only my new key.

If you have signed my old key, please consider signing the new one;
this file is signed with both keys as a proof I do have control over
them. Please note my old key is unable to certify the new one, so it
is not yet signed.

 -={ Old key, which I am transitioning _away_ from }=-

pub   rsa4096/0x673A03E4C1DB921F 2009-07-09 [SC] [expires: 2020-12-19]
      Key fingerprint = AB41 C1C6 8AFD 668C A045  EBF8 673A 03E4 C1DB 921F
uid                   [ultimate] Gunnar Eyal Wolf Iszaevich 
uid                   [ultimate] Gunnar Eyal Wolf Iszaevich 
uid                   [ultimate] Gunnar Eyal Wolf Iszaevich (Instituto de Investigaciones Económicas UNAM) 

 -={ New key, which I am transitioning to }=-

pub   ed25519/0x2404C9546E145360 2019-11-22 [SC] [expires: 2022-11-21]
      Key fingerprint = 4D14 0506 53A4 02D7 3687  049D 2404 C954 6E14 5360
uid                   [ unknown] Gunnar Wolf 
uid                   [ unknown] Gunnar Eyal Wolf Iszaevich 
uid                   [ unknown] Gunnar Wolf 

The new key has been uploaded to pool.sks-keyservers.net. If you
decide to sign my new key, I'd prefer if you mail it to me via
(i.e. using caff).

Thank you very much,

      - Gunnar
-----BEGIN PGP SIGNATURE-----

iQIzBAEBCAAdFiEEfHleU5lojd9m99YgSd0qTkl5YZwFAl3YhWkACgkQSd0qTkl5
YZxc1A//fycReXF6pqHl+zsMs4YW5fdMBDDuglIblX3AH5Q+ZtYvUEcEjUqXpTp4
32B725nF3o+chRsMosbJ2wUh6m5zbSl4ET/gjw+9M2XmrmC5HgrKurRh/bDryvZ1
lPPJOD6rfH5JeIBEnyoVrTaAMO5g2UUd3UM27EemNfK0oGZhJwbV8pvhQxLdrHk+
VKXBGPFcX0AttkutuixF7ieS/DjaVIogKPhy7r6UNEFopnMcGZmmTdy8EOnePLNc
ocHSHgVylErywx2zCaumC7nvc5nCeS+iStro2Cwpx1wcgdV0+RfX81Sz5v3g4IQA
Zv9AuAPBEnSFlrAPUggeDVTfA4A7BSoHgYnIo/xcIWnkd7SMwL/XDSgW3v7lNCJy
DvKhP9m0EM0KOOzzV8N3tyhvXNxW9fb0dmc+pUNI7qzXrzu0DVRiwrzRiCTnrOUQ
uV9cM/FdL82IGe5lGO48FgCBEaluocMu3bh9qFOIuQukbEvRsdgknGfcaa1mWRxs
YbBPYafFCTCRR410it2Ck4dRunHRLvYecTcFIvxUJ+lgrYSlRXIe4nfVmQ3lWosM
mUM9DhCanIeeNk/Enzqf8qpKlLzpBnEpJPnRciNvuGfvXzWSeKBdah7t5TNoMfNz
oWmCb2XbYmf8t7XFc9SNxWFaOTzt//w3Foo0DbBa9qtvuo4giRaIdQQBFggAHRYh
BGCzCT2WEI5cuXFC7+L2O0NT9FmJBQJd2IVqAAoJEOL2O0NT9FmJXwMA/j8s6O85
phqdDmOjKcmpKoxBlCKpulkMvpzSpxxcmGMzAP49HE6yVsyxFYoCI50w3nASSqmt
5i/2Cv7TgtzOFXU1AA==
=JxNv
-----END PGP SIGNATURE-----

I will be soon meeting with two DDs, so in any case, this key will be in shape to enter our keyring. Thank you very much for following so far!

(...And yes — This time I made two separate offline media backups for my master key material :-Þ)

23 November, 2019 01:08AM by gwolf

November 22, 2019

John Goerzen

The Fundamental Problem in Python 3

“In the beginning the Universe was created. This has made a lot of people very angry and been widely regarded as a bad move.”

– Douglas Adams

This expands on my recent post The Incredible Disaster of Python 3. I seem to have annoyed the Internet…

Back in the mists of time, Unix was invented. Today the descendants of Unix, whether literal or in spirit, power the majority of the world’s cell phones, most of the most popular sites on the Internet, etc. And among this very popular architecture, there lies something that has made people very angry at times: on a Unix filesystem, 254 bytes are valid in filenames. The two that are not are 0x00 and the slash character. Otherwise, they are valid in virtually any combination (the special entries “.” and “..” being the exception).

This property has led to a whole host of bugs, particularly in shell scripts. A filename with a leading dash might look like a parameter to a tool. Filenames can contain newline characters, space characters, control characters, and so forth; running ls in a directory with maliciously-named files could certainly scramble one’s terminal. These bugs continue to persist, though modern shells offer techniques that — while optional — can be used to avoid most of these classes of bugs.

It should be noted here that not every valid stream of bytes constitutes a stream of bytes that can be decoded as UTF-8. This is a departure from earlier encoding schemes such as iso-8859-1 and cp437; you might get gibberish, but “garbage in, garbage out” was a thing and if your channel was 8-bit clean, your gibberish would survive unmodified.

Unicode brings many advantages, and has rightly become the predominant standard for text encoding. But the previous paragraph highlights one of the challenges, and this challenge, along with some others, are at the heart of the problem with Python 3. That is because, at a fundamental level, Python 3’s notion of a filename is based on a fiction. Not only that, but it tries to introduce strongly-typed strings into what is fundamentally a weakly-typed, dynamic language.

A quick diversion: The Rust Perspective

The Unicode problem is a problematic one, and it may be impossible to deal with it with complete elegance. Many approaches exist; here I will describe Rust’s string-related types, of which there are three for our purposes:

  • The String (and related &str) is used for textual data and contains bytes that are guaranteed to be valid UTF-8 at all times
  • The Vec<u8> (and related [u8]) is a representation of pure binary bytes, of which all 256 possible characters are valid in any combination, whether or not it forms valid UTF-8
  • And the Path, which represents a path name on the system.

The Path uses the underlying operating system’s appropriate data type (here I acknowledge that Windows is very different from POSIX in this regard, though I don’t go into that here). Compile-time errors are generated when these types are mixed without proper safe conversion.

The Python Fiction

Python, in contrast, has only two types; roughly analogous to the String and the Vec<u8> in Rust. Critically, most of the Python standard library treats a filename as a String – that is, a sequence of valid Unicode code points, which is a subset of the valid POSIX filenames.

Do you see what we just did here? We’ve set up another shell-like situation in which filenames that are valid on the system create unexpected behaviors in a language. Only this time, it’s not \n, it’s things like \xF7.

From a POSIX standpoint, the correct action would have been to use the bytes type for filenames; this would mandate proper encode/decode calls by the user, but it would have been quite clear. It should be noted that some of the most core calls in Python, such as open(), do accept both bytes and strings, but this behavior is by no means consistent in the standard library, and some parts of the library that process filenames (for instance, listdir in its most common usage) return strings.

The Plot Thickens

At some point, it was clearly realized that this behavior was leading to a lot of trouble on POSIX systems. Having a listdir() function be unable (in its common usage; see below) to handle certain filenames was clearly not going to work. So Python introduced its surrogate escape. When using surrogate escapes, when attempting to decode a binary byte that is not valid in UTF-8, it is replaced with a multibyte UTF-8 sequence from Unicode code space that is otherwise rarely used. Then, when converted back to a binary sequence, this Unicode code point is converted to the same original byte. However, this is not a systemwide default and in many cases must be specifically requested.

And now you see this is both an ugly kludge and a violation of the promise of what a string is supposed to be in Python 3, since this doesn’t represent a valid Unicode character at all, but rather a token for the notion that “there was a byte here that we couldn’t convert to Unicode.” Now you have a string that the system thinks is Unicode, that looks like Unicode, that you can process as Unicode — substituting, searching, appending, etc — but which is actually at least partially representing things that should rightly be unrepresentable in Unicode.

And, of course, surrogate escapes are not universally used by even the Python standard library either. So we are back to the problem we had in Python 2: what the heck is a string, anyway? It might be all valid Unicode, it might have surrogate escapes in it, it might have been decoded from the wrong locale (because life isn’t perfect), and so forth.

Unicode Realities

The article pragmatic Unicode highlights these facts:

  1. Computers are built on bytes
  2. The world needs more than 256 symbols
  3. You cannot infer the encoding of bytes — you must be told, or have to guess
  4. Sometimes you are told wrong

I have no reason to quibble with this. How, then, does that stack up with this code from Python? (From zipfile.py, distributed as part of Python)

            if flags & 0x800:
                # UTF-8 file names extension
                filename = filename.decode('utf-8')
            else:
                # Historical ZIP filename encoding
                filename = filename.decode('cp437')

There is a reason that Python can’t extract a simple ZIP file properly. The snippet above violated the third rule by inferring a cp437 encoding when it shouldn’t. But it’s worse; the combination of factors leads extracall() to essentially convert a single byte from CP437 to a multibyte Unicode code point on extraction, rather than simply faithfully reproducing the bytestream that was the filename. Oh, and it doesn’t use surrogate escapes. Good luck with that one.

It gets even worse

Let’s dissect Python’s disastrous documentation on Unicode filenames.

First, we begin with the premise that there is no filename encoding in POSIX. Filenames are just blobs of bytes. There is no filename encoding!

What about $LANG and friends? They give hints about the environment, languages for interfaces, and terminal encoding. They can often be the best HINT as to how we should render characters and interpret filenames. But they do not subvert the fundamental truth, which is that POSIX filenames do not have to conform to UTF-8.

So, back to the Python documentation. Here are the problems with it:

  • It says that there will be a filesystem encoding if you set LANG or LC_CTYPE, falling back to UTF-8 if not specified. As we have already established, UTF-8 can’t handle POSIX filenames.
  • It gets worse: “The os.listdir() function returns filenames, which raises an issue: should it return the Unicode version of filenames, or should it return bytes containing the encoded versions? os.listdir() can do both”. So we are somewhat tacitly admitting here that str was a poor choice for filenames, but now we try to have it every which way. This is going to end badly.
  • And then there’s this gem: “Note that on most occasions, you should can just stick with using Unicode with these APIs. The bytes APIs should only be used on systems where undecodable file names can be present; that’s pretty much only Unix systems now.” Translation: Our default advice is to pretend the problem doesn’t exist, and will cause your programs to be broken or crash on POSIX.

Am I just living in the past?

This was the most common objection raised to my prior post. “Get over it, the world’s moved on.” Sorry, no. I laid out the case for proper handling of this in my previous post. But let’s assume that your filesystems are all new, with shiny UTF-8 characters. It’s STILL a problem. Why? Because it is likely that an errant or malicious non-UTF-8 sequence will cause a lot of programs to crash or malfunction.

We know how this story goes. All the shell scripts that do the wrong thing when “; rm” is in a filename, for instance. Now, Python is not a shell interpreter, but if you have a program that crashes on a valid filename, you have — at LEAST — a vector for denial of service. Depending on the circumstances, it could turn into more.

Conclusion

  • Some Python 3 code is going to crash or be unable to process certain valid POSIX filenames.
  • Some Python 3 code might use surrogate escapes to handle them.
  • Some Python 3 code — part of Python itself even — just assumes it’s all from cp437 (DOS) and converts it that way.
  • Some people recommend using latin-1 instead of surrogate escapes – even official Python documentation covers this.

The fact is: A Python string is the WRONG data type for a POSIX filename, and so numerous, incompatible kludges have been devised to work around this problem. There is no consensus on which kludge to use, or even whether or not to use one, even within Python itself, let alone the wider community. We are going to continue having these problems as long as Python continues to use a String as the fundamental type of a filename.

Doing the right thing in Python 3 is extremely hard, not obvious, and rarely taught. This is a recipe for a generation of buggy code. Easy things should be easy; hard things should be possible. Opening a file correctly should be easy. Sadly I fear we are in for many years of filename bugs in Python, because this would be hard to fix now.

Resources

(For even more fun, consider command line parameters and environment variables! I’m annoyed enough with filenames to leave those alone for now.)

22 November, 2019 08:02PM by John Goerzen

Sylvain Beucler

Android Rebuilds updates

What is it already?

Android Rebuilds provides freely-licensed builds of Android development tools written by somebody else.

New builds

SDK 10 (API 29) and NDK 20 rebuilds are now available, as unattended build scripts as well as binaries you shan't trust.

sdkmanager integration will be complete when we figure out how to give our repo precedence over somebody else's.

Evolution of the situation

SDK build remains monolithic and growing (40GB .git, 7h multi-core build, 200GB build space).

But there are fewer build issues, thanks to newer "prebuilts" deps straight in Git, now including OpenJDK.
I expect we'll soon chroot in Git before build.

Also for the first time ever I could complete a NDK windows build.

Licensing

Official binaries are still click-wrapped with a proprietary license.

It was discovered that such a license is also covering past versions of android.jar & al. hidden in a prebuilts directory and somehow necessary to the builds.
Archeological work already successfully started to rebuild SDKs from the start of the decade.

Fanbase

Android Rebuilds is showcased in ungoogled-chromium-android, a lightweight approach to removing Google web service dependency.

F-Droid mirror

After some back and forth, the F-Droid mirror is stable and limited to the experimental sdkmanager repository.
F-Droid showed high dedication to implementing upload restrictions and establishing procedures.
I have great hope that they will soon show the same level of dedication dropping non-free licenses and freeing their build server.

22 November, 2019 01:35PM

hackergotchi for Joachim Breitner

Joachim Breitner

Faster Winter 5: Eta-Expanding ReaderT

(This is the fifth optimization presented in the “faster winter” series, please see that post for background information.)

Another good approach to performance turning is look at the code after GHC optimized it. So I planted a

{-# OPTIONS_GHC -ddump-simpl -dsuppress-coercions -dsuppress-unfoldings -dsuppress-module-prefixes #-}

at the top of Eval.hs, and looked through the code. This is not Haskell, but rather “GHC Core”, a simpler functional programming language that GHC uses internally. It is more verbose and less pretty, so it takes a bit of getting used to, but it’s nothing to be afraid of when you are a somewhat experienced Haskell developer.

There I found (much simplified) the following pattern:

step :: Instr -> Config -> IO Result
step e = case e of
  Add -> \c -> … do stuff …
  Del -> \c -> … do stuff …
  …

That’s bad! Yes, Haskell is a functional language, and passing around anonymous functions is very nice to write expressive code, and for most purposes it is not too slow … but in an inner loop, you really don’t want any such closures. So where did this come from? And as expected, the Haskell source did not have those inner lambdas. Instead, it was using a very innocent looking monad transformer:

step :: Instr -> ReaderT Config IO Result
step e = case e of
  Add -> do stuff …
  Del -> do stuff …
  …

A ReaderT r m a is just a different way of writing r -> m a that allows us to use do-notation or the monad combinators without having to pass the r around explicity, and as such it is indeed very convenient. But not as efficient as if we had written

step :: Instr -> Config -> IO Result
step e c = case e of
  Add -> … do stuff …
  Del -> … do stuff …
  …

where the step function takes two arguments right away, and no anonymous functions are created.

Why doesn’t our amazing Haskell compiler figure out that this would be better? Because it is not better in all situations: If we store step e :: ReaderT Config IO Result somewhere and and use it many times, with the same e but passing many different c :: Config, then we have to do the case e analysis only once. This can sometimes be better, so the compiler has to leave it in that form, in case we did it intentionally.

(Incidentially, the question of how to allow the compiler to eta-expand more functions seems to eternally haunt me, and its pursuit even led to a PhD thesis.

So how can we fix it? One relatively crude way is to shove it into the compiler face that we really want step to be a function with two parameters by wrapping the whole body in, well, a lambda.. But we still want to use the Reader monad in the body of step…

So I came up with this:

step :: Instr -> ReaderT Config IO Result
step e = ReaderT $ \c -> ($ c) $ runReaderT $ case e of
  Add -> … do stuff …
  Del -> … do stuff …
  …

Now the \c -> is outside the case, the compiler adds it to the arguments of step and we get the code that we want (confirmed by a quick peek at the Core).

Improvement: Allocations: -23.20% Time: -23.00% (Commit f5a0dd2...894070f)

I used this pattern in more than once place, so I wanted to abstract it into a little helper definition. But that’s not so easy: If I just write

etaReaderT :: ReaderT r m a -> ReaderT r m a
etaReaderT m = ReaderT $ \c -> ($ c) $ runReaderT m

step :: Instr -> ReaderT Config IO Result
step e = etaReaderT $ case e of
  Add -> … do stuff …
  Del -> … do stuff …
  …

then the whole thing doesn't work any more! Because now, the case e is again “outside” the \c ->.

I whined on twitter about this, and Sebastian Graf reminded me helpfully of GHC.Exts.oneShot, a little magic function that was added to GHC 5 years ago … by some forgetful person: me.

If we use this in the right place inside etaReaderT it tells GHC in a soothing voice “hey! it’s ok! you can move this lambda out of cases. believe me. it’s gonna be ok”. And with this, it works:

etaReaderT :: ReaderT r m a -> ReaderT r m a
etaReaderT = ReaderT . oneShot . runReaderT

I wonder if this function would make a good addition to Control.Monad.Trans.Reader.

Incidentally, if you look at the code at the end of all my optimizations, there is no mention of etaReaderT any more: Subsequent optimizations simplified the code so much that eventually GHC was able to be able to do this transformation without my help.

22 November, 2019 09:59AM by Joachim Breitner (mail@joachim-breitner.de)

November 21, 2019

hackergotchi for Jonathan Dowland

Jonathan Dowland

8-bit

On Yesterday's Mary Anne Hobbs radio show she debuted a new track by Squarepusher, "Vortrack [Fracture Remix]", which you can now watch/listen to on YouTube:

This track really grabbed my attention. Later that day you you could buy it in a variety of formats and quality levels on Squarepusher's website.

One of the format/quality options was "8-bit Lossless WAV", which I thought was a joke, a poke in the eye of audiophiles. I was aware that he likely used some 8-bit instruments/equipment to write the tracks, but surely it was mixed in a more modern environment, and resampling down to 8-bit would result in something that sounded like mush.

But it seems the jokes on me; I bought the track and it's seemingly indistinguishable to the audio track on that YouTube video. And it really is 8-bit:

Input #0, wav, from 'Vortrack-001-Squarepusher-Vortrack (Fracture Remix).wav':
  Duration: 00:08:02.99, bitrate: 705 kb/s
    Stream #0:0: Audio: pcm_u8 ([1][0][0][0] / 0x0001),
    44100 Hz, 2 channels, u8, 705 kb/s

It even — losslessly — compressed down to a bitrate lower than a typical MP3:

Input #0, flac, from 'Vortrack-001-Squarepusher-Vortrack (Fracture Remix).flac':
  Duration: 00:08:02.99, start: 0.000000, bitrate: 313 kb/s
    Stream #0:0: Audio: flac, 44100 Hz, stereo, s16 (8 bit)

Sorry, comments on my site seem to be broken at the moment. I'm hoping to fix them soon.

21 November, 2019 11:55AM

Ian Jackson

Debian init systems - what, another GR ?

Sam Hartman, the Debian Project Leader, has proposed a General Resolution (a plebiscite of the whole project) about init systems. In this posting I am going to try to summarise the situation. This will necessarily be a personal view but I will try to be fair. Also, sorry that it's so long but there is a lot of ground to cover.

Pithy summary

The current political atmosphere has been turning bugs into fights. We need to stop this, and try to turn our fights back into bugs. The parts that need to be nailed down are:

  • whose job it is to fix the bug? (or, what happens if the bug is not fixed?, which amounts to the same thing);
  • when the bug is fixed take the patch already!; and
  • everyone stop with the dissing, negging, and concern trolling, already

I think we can do this (well, the first two, at least) with my proposal.

Philosophy

My starting point for all of this is what I have written at the top of my proposed alternative to the DPL's GR texts:

It is primarily for the communities in each software ecosystem to maintain and develop their respective software - but with the active support of other maintainers and gatekeepers where needed.
This is particularly important for a project like Debian. Debian plays a very central role for a lot of people. It is at least as much a clearinghouse or a marketplace, as it is a unified system. Supporting a different init system or a different menu system or whatever is fairly easy to do if done within Debian (if the Debian maintainers will take your patches) but a lot harder to do downstream.

Conversely, specialising on particular use cases is very easy to do in a derivative. This means that Debian needs to remain as general as possible, even if that means it suffers a bit in dominant use cases.

Background

In 2014 Debian changed to use systemd as the default init system. The intent (at least of those on the TC making that decision) was that other init systems would continue to be supported.

Since then, Debian has still continued to support sysvinit and other systems such as runit and OpenRC. sysvinit support was very good in Debian 9 ("stretch").

In Debian 10 ("buster", currently "stable") there are some snags with desktop software, much of which nowadays usually wants interfaces provided by systemd's logind component. elogind can provide this under sysvinit, but the package dependencies are awkward and there are some other wrinkles. Also the sysvinit package was looking rather undermaintained.

The Debian Ecosystem Init Diversity Team

In response, we assembled a team of people who cared about these issues. The team includes Debian Developers, developers from Devuan and we also have contributions from (for example) the recently revitalised sysvinit upstream. Working with the folks from Devuan has been great. I want in particular to say a huge thank you to Mark Hindley.

In bullseye ("testing"), sysvinit is getting plenty of love and is in reasonably good shape. We have a close relationship with sysvinit upstream. Support for desktop software is provided by elogind, which is a logind derivative developed for use in Guix.

Unresolved issues

However, the Debian Project as a whole remains very unsettled about init systems. Everyone has a lot of baggage, and there are some specific unresolved issues too.

Toxic atmosphere

One huge problem is that conversations on the Debian mailing lists about sysvinit or systemd-related features are generally extremely toxic.

There is a significant contingent within Debian who seem to be hoping that non-systemd systems would just go away so they could have the beautiful sunlit uplands of homogeneity. Comments predicting the demise of sysvinit, with a subtext that systemd sceptics have lost and should shut up and go away, or at least that they are doomed, are common. The implication (usually not stated so baldly) is "we decided for systemd n years ago, isn't it time to throw that other stuff out yet?"

Conversely, systemd sceptics in these conversations have been defensive, embattled and bitter. Predictions of systemd hegemony lead naturally to responses citing alleged systemd misfeatures, bugs and absurdities. And sometimes the responses from systemd sceptics just have been very nasty.

Debian has generally suffered from problems with toxic culture and has struggled to improve and to manage the behaviour of some of its contributors, and struggled to deal with obstructive maintainers. On this particular issue the political context, and the ongoing fights (read on), make it much much worse.

Blocking of init diversity work

An underlying reason for the bad feeling amongst systemd sceptics has been a pattern of obstructive behaviours by some people within Debian.

For example there have been vague and unactionable RC bug reports against pieces of the non-systemd ecosystem. Requests to core package maintainers to accomodate non-systemd setups in their dependencies are approved very slowly if at all. [1]

Blocking of policy updates; init scripts

Debian Policy still talks basically only about init scripts. We need policy that reflects reality and gives useful technical guidance on how to make things work properly with systemd, and how to make things compatible.

Nominally, policy says that you MUST have an init script if you have a systemd daemon service unit. But no-one is filing these bugs. This is probably because a contributor would not expect a useful response; also, Debian's processes for review of maintainer decisions can be quite unpleasant [2]. The current policy wording de facto encourages the mere filing of an RC bug (ie, a demand that the maintainer do the work) rather than the contribution of a patch.

The policy discussion process is unsuited to controversial issues. Usually policy changes only by consensus. It is clearly not possible to achieve consensus to drop the mandatory requirement for init scripts, since systemd opponents are unhappy with that - in particular because RC bug status is Debian's only really useful mechanism for dealing with an obstructive maintainer.

The policy maintainers are quite reasonably reluctant to make changes in this area without a clear indication of direction from the project.

Blocking of use of useful, portable, interfaces from systemd

systemd has some useful features that we could probably make use of in Debian without giving up support for non-systemd init systems. Particularly it offers declarative syntaxes for various common tasks which we currently do with debhelper scripts.

Many of these do not depend on systemd itself, and are not particularly complex. They would not be hard to reimplement (that being the standard approach to providing compatibility when the atmosphere is less fraught). Some of us have been hoping for more declarative approaches for some time, so there would be value here.

Normally in Debian someone who wanted to transition to such new scheme would provide the relevant implementations (perhaps several), do a few demo packages, document it in policy, and eventually (after some years) we would have lintian warnings about the old scheme. People who want to improve Debian must generally file patches[3] even when the maintainers are sympathetic but maybe busy.

However, people who think that systemd is the future (and that other init systems are obsolete) see reimplementing systemd features in other programs as wasted effort, and systemd itself is not structured to make it easy to tease out its implementations of some of its features. Conversely, people who dislike systemd are not likely to agree to the inclusion in policy of requirements they see as extending the systemd hegemony.

It is not unusual in Debian to see valuable changes blocked [4], sometimes for many years. But it is not a good thing - and this stuck issue provides additional fuel for the political fires.

The upcoming General Resolution and the position of the DPL

The current DPL thinks that trying to resolve the unresolved tensions over systemd is necessary for the health of the project. That is why he has brought this issue to a head by making a set of GR proposals.

Sam is framing this as a question of the project's priorities. The solution, for him, seems to be to simply and definitively answer the question of whether the project is going to support only systemd, and how much of a commitment we are to make to it.

I think that is the wrong framing. As I say above I think Debian needs to be the place where you can come to do your work, and we need to welcome everyone's contributions even if we don't think they're particularly valuable. That is the basic bargain underlying the common project.

Sam's middle option is not a compromise, and sets us up for more fighting

The most likely outcome of a choice just between the DPL's three options would probably the second one, which has been placed in the "middle" and labelled "systemd but" (as if dropping systemd were even on the table). But it is a poor text - vague and unbalanced.

It does not clearly and specifically address many of the problematic behaviours I discuss above, so it would set us up for more years of fighting over details.

Although it reads vaguely like a compromise ("however") and has been placed in the middle of the list, it will in fact make things things quite difficult for contributors and users who don't want systemd.

It doesn't clearly contradict the notion that "systemd is the future and other things are obsolete or experimental", which continually undermines work on systemd alternatives (for example, we have had maintainers ask on debian-devel whether they should remove their init script now they have a systemd unit file). It legitimises uncoordinated adoption of systemd features and make it unnecessarily hard for sysvinit etc. to keep up. It does nothing to help contributors whose non-systemd support contributions are rejected: for example, init script contributors would still need to fight those unreasonable rejections piecemeal.

On the approach to process

Sam has taken the unusual approach of drafting three alternative options himself. He has also taken a rather autocratic approach to the timetable. I strongly disapprove of this.

He has put himself in the position of trying to frame the debate along the lines that he sees, which is not right. Different constituencies have different framings, and are entitled to have those represented. People who see things differently should not be put in the position of opposing the leadership, and scrabbling for resolution seconders on needlessly tight timescales. The drafting of options should be led by the people whose opinions they are.

My position: Keep Debian a do-ocracy

The question of whether Debian supports running without systemd needs to be answered not by strategic decisions over priorities. That amounts to to to deliberately excluding people's contributions. I can see how that is tempting, because inevitably diversity comes with a price for everyone. But that diversity is how we built our community, and it is particularly important for Debian's role as a central clearinghouse, and indeed our role as an upstream to a whole ecosystem of derivatives.

We should look at the question we usually ask (eg about whether to support architectures): are there enough people who want to support non-systemd setups, and are they willing and able to do the work ?

If there are we should support them. We should take their patches and support and enable their work. If there are not, then quite significant obsolete software can and will die its death fairly quietly, without the need for blocking, fighting, negging and concern trolling.

Unfortunately it appears that enough people in Debian aren't able to be reasonable and constructive, at least without being set clear and specific boundaries. Setting those boundaries is what I am trying to do with my proposed option. If you are a full voting Debian Member, I would appreciate it if you would please formally Second my proposal. I have more than the minimum of five already - thanks Russ, Sean, Simon, Kyle, Dmitry, Noodles - but lots more would be good.

How my proposal helps

Toxic atmosphere

The introduction says we want to support multiple init systems "for the foreseeable future". That directly confronts the narrative that "systemd has won and therefore other things should go away". The introduction explicitly states the principle I started this posting with. There are some specific rules towards the end, where I name particular common toxic patterns; I don't know if our communication fora operators have the will to deal with this but this is the best a GR can do, I think.

Additionally I'm hoping that providing workable rules of engagement will mean that people are less frustrated, and that we don't have to try to wear each other down (which is how fights are traditionally won in Debian - urgh.)

Unblocking non-systemd support

The compromise for init scripts etc., is to encourage submission of patches by making the patches count as RC bugs. Although this is the current formal state in policy, it is not effective because it lacks political legitimacy. This GR gives it that legitimacy. In return we relax the policy MUST in cases where there is no patch. So we are not forcing anyone to do the work.

There are more complex situations where systemd package dependencies, or release team decisions, and so on, are involved. My option gives general guidance (which really ought to be uncontroversial, but has not always been followed). If this is not effective in particular cases then unfortunately the Technical Committee will probably end up having to resolve the disputes. But I think there will be relatively few of these and because of the complexity of the issues it's not possible to decide them in advance.

Unblocking policy development

There is a compromise on systemd's non-init-related features ("declarative" things). Fastest progress could be made for systemd systems if every maintainer was free to simply use every systemd feature. But that is not how we normally do things in Debian. We try to bring the whole community with us, even if that means some delays.

Because questions of timescale are going to be particularly controversial, and are a matter of balancing the interests of different groups, I put in a specific timescale to avoid further arguments. In practice I think when useful systemd syntaxes etc. are approved, the init diversity community will want to implement them quickly to demonstrate their dynamism and relevance.

On the "Init Diversity" counter-proposal

Dmitry Bogatov has put forward an option of his own. I think what Dmitry is trying to do makes sense. His option is the strongest protector of diversity and most strongly reestablishes the usual (albeit not universal) Debian norm that people who want to change things should do all the work. It gives clear political reinforcement to the current de jure policy on init scripts.

In most situations Dmitry's proposal is rather stronger than mine. I think in practical terms my proposal will be quite sufficient to have good support for sysvinit (and indeed other init systems) for the long term. But if you supported my strong init diversity position in 2014 you will want to consider seconding and/or voting for Dmitry's proposal.

This section largely rewritten 2019-11-21 11:03 since Dmitry updated and improved his proposal

Looking to the future

Frankly, there is a lot not to like about init scripts. I have never liked the "detach" model of daemon management (even though I wrote start-stop-daemon to fix its worst bugs). I much prefer setups with closer supervision.

I hope that we can find a better lingua franca than handwritten init scripts, at least for the most common cases. Exactly what that looks like will have to be discussed within the init diversity community.

We do have two other init systems: runit and OpenRC. I haven't tried either of them and frankly I have little idea if I would like them if I did. But we have good relationships between all of the relevant maintainers and users. I hope these projects will thrive.

But so long as the community of systemd sceptics have to be fighting for their very existence, there is little time and little emotional energy for development and experimentation, and adoption of novel ideas.

Acknowledgements

Thanks very much to the several people who helped by reviewing drafts of this posting. Your support and frank comments have been invaluable. You know who you are. All errors and missteps are mine, of course.

Footnotes

[1] I don't mean to be calling people out, but I feel these kind of assertions need to be substantiated so here are some representative examples.

[2] The Technical Committee process is quite stressful. Here's a particularly bad example (where I have mixed feelings about my own part).

[3] all reproducible builds bugs (warning: huge)

[4] Debian is famously slow but here are a couple of my pet examples.

Edited 2019-11-20 12:20 UTC to fix some HTML formatting issues with some blog aggregators.



comment count unavailable comments

21 November, 2019 11:08AM

hackergotchi for Joachim Breitner

Joachim Breitner

Faster Winter 4: Export lists

(This is the forth optimization presented in the “faster winter” series, please see that post for background information.)

This is on a funny one: You can speed up your code by adding export lists to your modules!

Why is that so?

Without an export, the compiler has to assume that every top-level function can possibly called from the outside, even functions that you think of as “internal”. If you have a function that you do not export, like instr, step_work and step after my change, the compiler can see all the places the function is called. If the function is only called in one place, it may inline it (copy its definition into where it is called), and simplify the code around the edges. And even if it does not inline the function, it might learn something about how the functions are used, and optimize them based on that (e.g. based on Demand Analysis).

Improvement: Allocations: -22.59% Memory: +0.00% Time: -11.79% (Commit bbe8af7...6f2ba09)

Besides being a performance win, an explicit export list is also very helpful to those who want to read or edit your code: they can refactor with greater ease while maintaining the external interface of the module.

21 November, 2019 07:52AM by Joachim Breitner (mail@joachim-breitner.de)

hackergotchi for Kees Cook

Kees Cook

experimenting with Clang CFI on upstream Linux

While much of the work on kernel Control Flow Integrity (CFI) is focused on arm64 (since kernel CFI is available on Android), a significant portion is in the core kernel itself (and especially the build system). Recently I got a sane build and boot on x86 with everything enabled, and I’ve been picking through some of the remaining pieces. I figured now would be a good time to document everything I do to get a build working in case other people want to play with it and find stuff that needs fixing.

First, everything is based on Sami Tolvanen’s upstream port of Clang’s forward-edge CFI, which includes his Link Time Optimization (LTO) work, which CFI requires. This tree also includes his backward-edge CFI work on arm64 with Clang’s Shadow Call Stack (SCS).

On top of that, I’ve got a few x86-specific patches that get me far enough to boot a kernel without warnings pouring across the console. Along with that are general linker script cleanups, CFI cast fixes, and x86 crypto fixes, all in various states of getting upstreamed. The resulting tree is here.

On the compiler side, you need a very recent Clang and LLD (i.e. “Clang 10”, or what I do is build from the latest git). For example, here’s how to get started. First, checkout, configure, and build Clang (and include a RISC-V target just for fun):

# Check out latest LLVM
mkdir -p $HOME/src
cd $HOME/src
git clone https://github.com/llvm/llvm-project.git
mkdir llvm-build
cd llvm-build
# Configure
cmake -DCMAKE_BUILD_TYPE=Release \
      -DLLVM_ENABLE_PROJECTS='clang;lld' \
      -DLLVM_EXPERIMENTAL_TARGETS_TO_BUILD="RISCV" \
      ../llvm-project/llvm
# Build!
make -j$(getconf _NPROCESSORS_ONLN)
# Install cfi blacklist template (why is this missing from "make" above?)
mkdir -p $(echo lib/clang/*)/share
cp ../llvm-project/compiler-rt/lib/cfi/cfi_blacklist.txt lib/clang/*/share/cfi_blacklist.txt

Then checkout, configure, and build the CFI tree. (This assumes you’ve already got a checkout of Linus’s tree.)

# Check out my branch
cd ../linux
git remote add kees https://git.kernel.org/pub/scm/linux/kernel/git/kees/linux.git
git fetch kees
git checkout kees/kspp/cfi/x86 -b test/cfi
# Configure (this uses "defconfig" but you could use "menuconfig"), but you must
# include CC and LD in the make args or your .config won't know about Clang.
make defconfig \
     CC=$HOME/src/llvm-build/bin/clang LD=$HOME/src/llvm-build/bin/ld.lld
# Enable LTO and CFI.
scripts/config \
     -e CONFIG_LTO \
     -e CONFIG_THINLTO \
     -d CONFIG_LTO_NONE \
     -e CONFIG_LTO_CLANG \
     -e CONFIG_CFI_CLANG \
     -e CONFIG_CFI_PERMISSIVE \
     -e CONFIG_CFI_CLANG_SHADOW
# Enable LKDTM if you want runtime fault testing:
scripts/config -e CONFIG_LKDTM
# Build!
make -j$(getconf _NPROCESSORS_ONLN) \
     CC=$HOME/src/llvm-build/bin/clang LD=$HOME/src/llvm-build/bin/ld.lld

Do not be alarmed by various warnings, such as:

ld.lld: warning: cannot find entry symbol _start; defaulting to 0x1000
llvm-ar: error: unable to load 'arch/x86/kernel/head_64.o': file too small to be an archive
llvm-ar: error: unable to load 'arch/x86/kernel/head64.o': file too small to be an archive
llvm-ar: error: unable to load 'arch/x86/kernel/ebda.o': file too small to be an archive
llvm-ar: error: unable to load 'arch/x86/kernel/platform-quirks.o': file too small to be an archive
WARNING: EXPORT symbol "page_offset_base" [vmlinux] version generation failed, symbol will not be versioned.
WARNING: EXPORT symbol "vmalloc_base" [vmlinux] version generation failed, symbol will not be versioned.
WARNING: EXPORT symbol "vmemmap_base" [vmlinux] version generation failed, symbol will not be versioned.
WARNING: "__memcat_p" [vmlinux] is a static (unknown)
no symbols

Adjust your .config as you want (but, again, make sure the CC and LD args are pointed at Clang and LLD respectively). This should(!) result in a happy bootable x86 CFI-enabled kernel. If you want to see what a CFI failure looks like, you can poke LKDTM:

# Log into the booted system as root, then:
cat <(echo CFI_FORWARD_PROTO) >/sys/kernel/debug/provoke-crash/DIRECT
dmesg

Here’s the CFI splat I see on the console:

[   16.288372] lkdtm: Performing direct entry CFI_FORWARD_PROTO
[   16.290563] lkdtm: Calling matched prototype ...
[   16.292367] lkdtm: Calling mismatched prototype ...
[   16.293696] ------------[ cut here ]------------
[   16.294581] CFI failure (target: lkdtm_increment_int$53641d38e2dc4a151b75cbe816cbb86b.cfi_jt+0x0/0x10):
[   16.296288] WARNING: CPU: 3 PID: 2612 at kernel/cfi.c:29 __cfi_check_fail+0x38/0x40
...
[   16.346873] ---[ end trace 386b3874d294d2f7 ]---
[   16.347669] lkdtm: Fail: survived mismatched prototype function call!

The claim of “Fail: survived …” is due to CONFIG_CFI_PERMISSIVE=y. This allows the kernel to warn but continue with the bad call anyway. This is handy for debugging. In a production kernel that would be removed and the offending kernel thread would be killed. If you run this again with the config disabled, there will be no continuation from LKDTM. :)

Enjoy! And if you can figure out before me why there is still CFI instrumentation in the KPTI entry handler, please let me know and help us fix it. ;)

© 2019, Kees Cook. This work is licensed under a Creative Commons Attribution-ShareAlike 3.0 License.
Creative Commons License

21 November, 2019 05:09AM by kees