January 16, 2021

hackergotchi for Junichi Uekawa

Junichi Uekawa

It's been 20 years since I became a Debian Developer.

It's been 20 years since I became a Debian Developer. Lots of fun things happened, and I think fondly of the team. I am no longer active for the past 10 years due to family reasons, and it's surprising that I have been inactive for that long. I still use Debian, and I still participate in the local Debian meetings.

16 January, 2021 12:22AM by Junichi Uekawa

January 15, 2021

hackergotchi for Dirk Eddelbuettel

Dirk Eddelbuettel

Rcpp 1.0.6: Some Updates

rcpp logo

The Rcpp team is proud to announce release 1.0.6 of Rcpp which arrived at CRAN earlier today, and has been uploaded to Debian too. Windows and macOS builds should appear at CRAN in the next few days. This marks the first release on the new six-months cycle announced with release 1.0.5 in July. As reminder, interim ‘dev’ or ‘rc’ releases will often be available in the Rcpp drat repo; this cycle there were four.

Rcpp has become the most popular way of enhancing R with C or C++ code. As of today, 2174 packages on CRAN depend on Rcpp for making analytical code go faster and further (which is an 8.5% increase just since the last release), along with 207 in BioConductor.

This release features six different pull requests from five different contributors, mostly fixing fairly small corner cases, plus some minor polish on documentation and continuous integration. Before releasing we once again made numerous reverse dependency checks none of which revealed any issues. So the passage at CRAN was pretty quick despite the large dependency footprint, and we are once again grateful for all the work the CRAN maintainers do.

Changes in Rcpp patch release version 1.0.6 (2021-01-14)

  • Changes in Rcpp API:

    • Replace remaining few uses of EXTPTR_PTR with R_ExternalPtrAddr (Kevin in #1098 fixing #1097).

    • Add push_back and push_front for DataFrame (Walter Somerville in #1099 fixing #1094).

    • Remove a misleading-to-wrong comment (Mattias Ellert in #1109 cleaning up after #1049).

    • Address a sanitizer report by initializing two private bool variables (Benjamin Christoffersen in #1113).

    • External pointer finalizer toggle default values were corrected to true (Dirk in #1115).

  • Changes in Rcpp Documentation:

    • Several URLs were updated to https and/or new addresses (Dirk).
  • Changes in Rcpp Deployment:

    • Added GitHub Actions CI using the same container-based setup used previously, and also carried code coverage over (Dirk in #1128).
  • Changes in Rcpp support functions:

    • Rcpp.package.skeleton() avoids warning from R. (Dirk)

Thanks to my CRANberries, you can also look at a diff to the previous release. Questions, comments etc should go to the rcpp-devel mailing list off the R-Forge page. Bugs reports are welcome at the GitHub issue tracker as well (where one can also search among open or closed issues); questions are also welcome under rcpp tag at StackOverflow which also allows searching among the (currently) 2616 previous questions.

If you like this or other open-source work I do, you can sponsor me at GitHub. My sincere thanks to my current sponsors for me keeping me caffeinated.

This post by Dirk Eddelbuettel originated on his Thinking inside the box blog. Please report excessive re-aggregation in third-party for-profit settings.

15 January, 2021 11:24PM

hackergotchi for Michael Prokop

Michael Prokop

Revisiting 2020

*

Mainly to recall what happened last year and to give thoughts and plan for the upcoming year(s) I’m once again revisiting my previous year (previous editions: 2019, 2018, 2017, 2016, 2015, 2014, 2013 + 2012).

Due to the Coronavirus disease (COVID-19) pandemic, 2020 was special™ for several reasons, but overall I consider myself and my family privileged and am very grateful for that.

In terms of IT events, I planned to attend Grazer Linuxdays and DebConf in Haifa/Israel. Sadly Grazer Linuxdays didn’t take place at all, and DebConf took place online instead (which I didn’t really participate in for several reasons). I took part in the well organized DENOG12 + ATNOG 2020/1 online meetings. I still organize our monthly Security Treff Graz (STG) meetups, and for half of the year, those meetings took place online (which worked OK-ish overall IMO).

Only at the beginning of 2020, I managed to play Badminton (still playing in the highest available training class (in german: “Kader”) at the University of Graz / Universitäts-Sportinstitut, USI). For the rest of the year – except for ~2 weeks in October or so – the sessions couldn’t occur.

Plenty of concerts I planned to attend were cancelled for obvious reasons, including the ones I would have played myself. But I managed to attend Jazz Redoute 2020 – Dom im Berg, Martin Grubinger in Musikverein Graz and Emiliano Sampaio’s Mega Mereneu Project at WIST Moserhofgasse (all before the corona situation kicked in). The concert from Tonč Feinig & RTV Slovenia Big Band occurred under strict regulations in Summer. At the beginning of 2020, I also visited Literaturshow “Roboter mit Senf” at Literaturhaus Graz.

The lack of concerts and rehearsals also severely impacted my playing the drums (including at HTU BigBand Graz), which pretty much didn’t take place. :(

Grml-wise we managed to publish release 2020.06, codename Ausgehfuahangl. Regarding jenkins-debian-glue I tried to clarify its state and received some really lovely feedback.

I consider 2020 as the year where I dropped regular usage of Jabber (so far my accounts still exist, but I’m no longer regularly online and am not sure for how much longer I’ll keep my accounts alive as such).

Business-wise it was our seventh year of business with SynPro Solutions GmbH. No big news but steady and ongoing work with my other business duties Grml Solutions and Grml-Forensic.

As usual, I shared childcare with my wife. Due to the corona situation, my wife got a new working schedule, which shuffled around our schedule a bit on Mondays + Tuesdays. Still, we managed to handle the homeschooling/distance learning quite well. Currently we’re sitting in the third lockdown, and yet another round of homeschooling/distance learning is going on those days (let’s see how long…). I counted 112 actual school days in all of 2020 for our older daughter with only 68 school days since our first lockdown on 16th of March, whereas we had 213(!) press conferences by our Austrian government in 2020. (Further rants about the situation in Austria snipped.)

Book reading-wise I managed to complete 60 books (see “Mein Lesejahr 2020“). Once again, I noticed that what felt like good days for me always included reading books, so I’ll try to keep my reading pace for 2021. I’ll also continue with my hobbies “Buying Books” and “Reading Books”, to get worse at Tsundoku.

Hoping for vaccination and a more normal 2021, Schwuppdiwupp!

15 January, 2021 11:07PM by mika

January 14, 2021

hackergotchi for Steinar H. Gunderson

Steinar H. Gunderson

Bullseye freeze

Bullseye is freezing! Yay! (And Trondheim is now below -10.)

It's too late for that kind of change now, but it would have been nice if plocate could have been default for bullseye:

plocate popcon graph

Surprisingly enough, mlocate has gone straight downhill:

mlocate popcon graph

It seems that since buster, there's an override in place to change its priority away from standard, and I haven't been able to find anyone who could tell me why. (It was known that it was request moved away from standard for cloud images, which makes a lot of sense, but not for desktop/server images.)

Perhaps for bookworm, we can get a locate back in the default install? plocate really is a much better user experience, in my (admittely biased) opinion. :-)

14 January, 2021 08:22PM

January 13, 2021

Antoine Beaupré

New phone: Pixel 4a

I'm sorry to announce that I gave up on the Fairphone series and switched to a Google Phone (Pixel 4a) running CalyxOS.

Problems in fairy land

My fairphone2, even if it is less than two years old, is having major problems:

  • from time to time, the screen flickers and loses "touch" until I squeeze it back together
  • the camera similarly disconnects regularly
  • even when it works, the camera is... pretty bad: low light is basically unusable, it's slow and grainy
  • the battery can barely keep up for one day
  • the cellular coverage is very poor, in Canada: I lose signal at the grocery store and in the middle of my house...

Some of those problems are known: the Fairphone 2 is old now. It was probably old even when I got it. But I can't help but feel a little sad to let it go: the entire point of that device was to make it easy to fix. But alas, because it's sold only in Europe, local stores don't carry replacement parts. To be fair, Fairphone did offer to fix the device, but with a 2 weeks turnaround, I had to get another phone anyways.

I did actually try to buy a fairphone3, from Clove. But they did some crazy validation routine. By email, they asked me to provide a photo copy of a driver's license and the credit card, arguing they need to do this to combat fraud. I found that totally unacceptable and asked them to cancel my order. And because I'm not sure the FP3 will fix the coverage issues, I decided to just give up on Fairphone until they officially ship to the Americas.

Do no evil, do not pass go, do not collect 200$

So I got a Google phone, specifically a Pixel 4a. It's a nice device, all small and shiny, but it's "plasticky" - I would have prefered metal, but it seems you need to pay much, much more to get that (in the Pixel 5).

In any case, it's certainly a better form factor than the Fairphone 2: even though the screen is bigger, the device itself is actually smaller and thinner, which feels great. The OLED screen is beautiful, awesome contrast and everything, and preliminary tests show that the camera is much better than the one on the Fairphone 2. (The be fair, again, that is another thing the FP3 improved significantly. And that is with the stock Camera app from CalyxOS/AOSP, so not as good as the Google Camera app, which does AI stuff.)

CalyxOS: success

The Pixel 4a not not supported by LineageOS: it seems every time I pick a device in that list, I manage to miss the right device by one (I bought a Samsung S9 before, which is also unsupported, even though the S8 is). But thankfully, it is supported by CalyxOS.

That install was a breeze: I was hesitant in playing again with installing a custom Android firmware on a phone after fighting with this quite a bit in the past (e.g. htc-one-s, lg-g3-d852). But it turns out their install instructions, mostly using a AOSP alliance device-flasher works absolutely great. It assumes you know about the commandline, and it does require to basically curl | sudo (because you need to download their binary and run it as root), but it Just. Works. It reminded me of how great it was to get the Fairphone with TWRP preinstalled...

Oh, and kudos to the people in #calyxos on Freenode: awesome tech support, super nice folks. An amazing improvement over the ambiance in #lineageos! :)

Migrating data

Unfortunately, migrating the data was the usual pain in the back. This should improve the next time I do this: CalyxOS ships with seedvault, a secure backup system for Android 10 (or 9?) and later which backs up everything (including settings!) with encryption. Apparently it works great, and CalyxOS is also working on a migration system to switch phones.

But, obviously, I couldn't use that on the Fairphone 2 running Android 7... So I had to, again, improvised. The first step was to install Syncthing, to have an easy way to copy data around. That's easily done through F-Droid, already bundled with CalyxOS (including the privileged extension!). Pair the devices and boom, a magic portal to copy stuff over.

The other early step I took was to copy apps over using the F-Droid "find nearby" functionality. It's a bit quirky, but really helps in copying a bunch of APKs over.

Then I setup a temporary keepassxc password vault on the Syncthing share so that I could easily copy-paste passwords into apps. I used to do this in a text file in Syncthing, but copy-pasting in the text file is much harder than in KeePassDX. (I just picked one, maybe KeePassDroid is better? I don't know.) Do keep a copy of the URL of the service to reduce typing as well.

Then the following apps required special tweaks:

  • AntennaPod has an import/export feature: export on one end, into the Syncthing share, then import on the other. then go to the queue and select all episodes and download
  • the Signal "chat backup" does copy the secret key around, so you don't get the "security number change" warning (even if it prompts you to re-register) - external devices need to be relinked though
  • AnkiDroid, DSub, Nextcloud, and Wallabag required copy-pasting passwords

I tried to sync contacts with DAVx5 but that didn't work so well: the account was setup correctly, but contacts didn't show up. There's probably just this one thing I need to do to fix this, but since I don't really need sync'd contact, it was easier to export a VCF file to Syncthing and import again.

Known problems

One problem with CalyxOS I found is that the fragile little microg tweaks didn't seem to work well enough for Signal. That was unexpected so they encouraged me to file that as a bug.

The other "issue" is that the bootloader is locked, which makes it impossible to have "root" on the device. That's rather unfortunate: I often need root to debug things on Android. In particular, it made it difficult to restore data from OSMand (see below). But I guess that most things just work out of the box now, so I don't really need it and appreciate the extra security. Locking the bootloader means full cryptographic verification of the phone, so that's a good feature to have!

OSMand still doesn't have a good import/export story. I ended up sharing the Android/data/net.osmand.plus/files directory and importing waypoints, favorites and tracks by hand. Even though maps are actually in there, it's not possible for Syncthing to write directly to the same directory on the new phone, "thanks" to the new permission system in Android which forbids this kind of inter-app messing around.

Tracks are particularly a problem: my older OSMand setup had all those folders neatly sorting those tracks by month. This makes it really annoying to track every file manually and copy it over. I have mostly given up on that for now, unfortunately. And I'll still need to reconfigure profiles and maps and everything by hand. Sigh. I guess that's a good clearinghouse for my old tracks I never use...

Update: turns out setting storage to "shared" fixed the issue, see comments below!

Conclusion

Overall, CalyxOS seems like a good Android firmware. The install is smooth and the resulting install seems solid. The above problems are mostly annoyances and I'm very happy with the experience so far, although I've only been using it for a few hours so this is very preliminary.

13 January, 2021 08:50PM

Vincent Fourmond

Taking advantage of Ruby in QSoas

First of all, let me all wish you a happy new year, with all my wishes of health and succes. I sincerely hope this year will be simpler for most people as last year !

For the first post of the year, I wanted to show you how to take advantage of Ruby, the programming language embedded in QSoas, to make various things, like:
  • creating a column with the sum of Y values;
  • extending values that are present only in a few lines;
  • renaming datasets using a pattern.

Summing the values in a column

When using commands that take formulas (Ruby code), like apply-formula, the code is run for every single point, for which all the values are updated. In particulier, the state of the previous point is not known. However, it is possible to store values in what is called global variables, whose name start with an $ sign. Using this, we can keep track of the previous values. For instance, to create a new column with the sum of the y values, one can use the following approach:
QSoas> eval $sum=0
QSoas> apply-formula /extra-columns=1 $sum+=y;y2=$sum
The first line initializes the variable to 0, before we start summing, and the code in the second line is run for each dataset row, in order. For the first row, for instance, $sum is initially 0 (from the eval line); after the execution of the code, it is now the first value of y. After the second row, the second value of y is added, and so on. The image below shows the resulting y2 when used on:
QSoas> generate-dataset -1 1 x


Extending values in a column

Another use of the global variables is to add "missing" data. For instance, let's imagine that a files given the variation of current over time as the potential is changed, but the potential is only changed stepwise and only indicated when it changes:
## time	current	potential
0	0.1	0.5
1	0.2
2	0.3
3	0.2
4	1.2	0.6
5	1.3
...
If you need to have the values everywhere, for instance if you need to split on their values, you could also use a global variable, taking advantage of the fact that missing values are represented by QSoas using "Not A Number" values, which can be detected using the Ruby function nan?:
QSoas> apply-formula "if y2.nan?; then y2=$value; else $value=y2;end"
Note the need of quotes because there are spaces in the ruby code. If the value of y2 is NaN, that is it is missing, then it is taken from the global variable $value else $value is set the current value of y2. Hence, the values are propagated down:
## time	current	potential
0	0.1	0.5
1	0.2	0.5
2	0.3	0.5
3	0.2	0.5
4	1.2	0.6
5	1.3	0.6
...
Of course, this doesn't work if the first value of y2 is missing.

Renaming using a pattern

The command save-datasets can be used to save a whole series of datasets to the disk. It can also rename them on the fly, and, using the /mode=rename option, does only the renaming part, without saving. You can make full use of meta-data (see also a first post here)for renaming. The full power is unlocked using the /expression= option. For instance, for renaming the last 5 datasets (so numbers 0 to 4) using a scheme based on the value of their pH meta-data, you can use the following code:
QSoas> save-datasets /mode=rename /expression='"dataset-#{$meta.pH}"' 0..4
The double quotes are cumbersome but necessary, since the outer quotes (') prevent the inner ones (") to be removed and the inner quotes are here to indicate to Ruby that we are dealing with text. The bit inside #{...} is interpreted by Ruby as Ruby code; here it is $meta.pH, the value of the "pH" meta-data. Finally the 0..4 specifies the datasets to work with. So theses datasets will change name to become dataset-7 for pH 7, etc...

About QSoas

QSoas is a powerful open source data analysis program that focuses on flexibility and powerful fitting capacities. It is released under the GNU General Public License. It is described in Fourmond, Anal. Chem., 2016, 88 (10), pp 5050–5052. Current version is 3.0. You can download its source code there (or clone from the GitHub repository) and compile it yourself, or buy precompiled versions for MacOS and Windows there.

13 January, 2021 02:45PM by Vincent Fourmond (noreply@blogger.com)

January 12, 2021

John Goerzen

Remote Directory Tree Comparison, Optionally Asynchronous and Airgapped

Note: this is another article in my series on asynchronous communication in Linux with UUCP and NNCP.

In the previous installment on store-and-forward backups, I mentioned how easy it is to do with ZFS, and some of the tools that can be used to do it without ZFS. A lot of those tools are a bit less robust, so we need some sort of store-and-forward mechanism to verify backups. To be sure, verifying backups is good with ANY scheme, and this could be used with ZFS backups also.

So let’s say you have a shiny new backup scheme in place, and you’d like to verify that it’s working correctly. To do that, you need to compare the source directory tree on machine A with the backed-up directory tree on machine B.

Assuming a conventional setup, here are some ways you might consider to do that:

  • Just copy everything from machine A to machine B and compare locally
  • Or copy everything from machine A to a USB drive, plug that into machine B, and compare locally
  • Use rsync in dry-run mode and see if it complains about anything

The first two options are not particularly practical for large datasets, though I note that the second is compatible with airgapping. Using rsync requires both systems to be online at the same time to perform the comparison.

What would be really nice here is a tool that would write out lots of information about the files on a system: their names, sizes, last modified dates, maybe even sha256sum and other data. This file would be far smaller than the directory tree itself, would compress nicely, and could be easily shipped to an airgapped system via NNCP, UUCP, a USB drive, or something similar.

Tool choices

It turns out there are already quite a few tools in Debian (and other Free operating systems) to do this, and half of them are named mtree (though, of course, not all mtrees are compatible with each other.) We’ll look at some of the options here.

I’ve made a simple test directory for illustration purposes with these commands:

mkdir test
cd test
echo hi > hi
ln -s hi there
ln hi foo
touch empty
mkdir emptydir
mkdir somethingdir
cd somethingdir
ln -s ../there

I then also used touch to set all files to a consistent timestamp for illustration purposes.

Tool option: getfacl (Debian package: acl)

This comes with the acl package, but can be used with other than ACL purposes. Unfortunately, it doesn’t come with a tool to directly compare its output with a filesystem (setfacl, for instance, can apply the permissions listed but won’t compare.) It ignores symlinks and doesn’t show sizes or dates, so is ineffective for our purposes.

Example output:

$ getfacl --numeric -R test
...
# file: test/hi
# owner: 1000
# group: 1000
user::rw-
group::r--
other::r--
...

Tool option: fmtree, the FreeBSD mtree (Debian package: freebsd-buildutils)

fmtree can prepare a “specification” based on a directory tree, and compare a directory tree to that specification. The comparison also is aware of files that exist in a directory tree but not in the specification. The specification format is a bit on the odd side, but works well enough with fmtree. Here’s a sample output with defaults:

$ fmtree -c -p test
...
# .
/set type=file uid=1000 gid=1000 mode=0644 nlink=1
.               type=dir mode=0755 nlink=4 time=1610421833.000000000
    empty       size=0 time=1610421833.000000000
    foo         nlink=2 size=3 time=1610421833.000000000
    hi          nlink=2 size=3 time=1610421833.000000000
    there       type=link mode=0777 time=1610421833.000000000 link=hi

... skipping ...

# ./somethingdir
/set type=file uid=1000 gid=1000 mode=0777 nlink=1
somethingdir    type=dir mode=0755 nlink=2 time=1610421833.000000000
    there       type=link time=1610421833.000000000 link=../there
# ./somethingdir
..

..

You might be wondering here what it does about special characters, and the answer is that it has octal escapes, so it is 8-bit clean.

To compare, you can save the output of fmtree to a file, then run like this:

cd test
fmtree < ../test.fmtree

If there is no output, then the trees are identical. Change something and you get a line of of output explaining each difference. You can also use fmtree -U to change things like modification dates to match the specification.

fmtree also supports quite a few optional keywords you can add with -K. They include things like file flags, user/group names, various tipes of hashes, and so forth. I'll note that none of the options can let you determine which files are hardlinked together.

Here's an excerpt with -K sha256digest added:

    empty       size=0 time=1610421833.000000000 \
                sha256digest=e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
    foo         nlink=2 size=3 time=1610421833.000000000 \
                sha256digest=98ea6e4f216f2fb4b69fff9b3a44842c38686ca685f3f55dc48c5d3fb1107be4

If you include a sha256digest in the spec, then when you verify it with fmtree, the verification will also include the sha256digest. Obviously fmtree -U can't correct a mismatch there, but of course it will detect and report it.

Tool option: mtree, the NetBSD mtree (Debian package: mtree-netbsd)

mtree produces (by default) output very similar to fmtree. With minor differences (such as the name of the sha256digest in the output), the discussion above about fmtree also applies to mtree.

There are some differences, and the most notable is that mtree adds a -C option which reads a spec and converts it to a "format that's easier to parse with various tools." Here's an example:

$ mtree -c -K sha256digest -p test | mtree -C
. type=dir uid=1000 gid=1000 mode=0755 nlink=4 time=1610421833.0 flags=none 
./empty type=file uid=1000 gid=1000 mode=0644 nlink=1 size=0 time=1610421833.0 flags=none 
./foo type=file uid=1000 gid=1000 mode=0644 nlink=2 size=3 time=1610421833.0 flags=none 
./hi type=file uid=1000 gid=1000 mode=0644 nlink=2 size=3 time=1610421833.0 flags=none 
./there type=link uid=1000 gid=1000 mode=0777 nlink=1 link=hi time=1610421833.0 flags=none 
./emptydir type=dir uid=1000 gid=1000 mode=0755 nlink=2 time=1610421833.0 flags=none 
./somethingdir type=dir uid=1000 gid=1000 mode=0755 nlink=2 time=1610421833.0 flags=none 
./somethingdir/there type=link uid=1000 gid=1000 mode=0777 nlink=1 link=../there time=1610421833.0 flags=none 

Most definitely an improvement in both space and convenience, while still retaining the relevant information. Note that if you want the sha256digest in the formatted output, you need to pass the -K to both mtree invocations. I could have done that here, but it is easier to read without it.

mtree can verify a specification in either format. Given what I'm about to show you about bsdtar, this should illustrate why I bothered to package mtree-netbsd for Debian.

Unlike fmtree, the mtree -U command will not adjust modification times based on the spec, but it will report on differences.

Tool option: bsdtar (Debian package: libarchive-tools)

bsdtar is a fascinating program that can work with many formats other than just tar files. Among the formats it supports is is the NetBSD mtree "pleasant" format (mtree -C compatible).

bsdtar can also convert between the formats it supports. So, put this together: bsdtar can convert a tar file to an mtree specification without extracting the tar file. bsdtar can also use an mtree specification to override the permissions on files going into tar -c, so it is a way to prepare a tar file with things owned by root without resorting to tools like fakeroot.

Let's look at how this can work:

$ cd test
$ bsdtar --numeric -cf - --format=mtree .

. time=1610472086.318593729 mode=755 gid=1000 uid=1000 type=dir
./empty time=1610421833.0 mode=644 gid=1000 uid=1000 type=file size=0
./foo nlink=2 time=1610421833.0 mode=644 gid=1000 uid=1000 type=file size=3
./hi nlink=2 time=1610421833.0 mode=644 gid=1000 uid=1000 type=file size=3
./ormat\075mtree time=1610472086.318593729 mode=644 gid=1000 uid=1000 type=file size=5632
./there time=1610421833.0 mode=777 gid=1000 uid=1000 type=link link=hi
./emptydir time=1610421833.0 mode=755 gid=1000 uid=1000 type=dir
./somethingdir time=1610421833.0 mode=755 gid=1000 uid=1000 type=dir
./somethingdir/there time=1610421833.0 mode=777 gid=1000 uid=1000 type=link link=../there

You can use mtree -U to verify that as before. With the --options mtree: set, you can also add hashes and similar to the bsdtar output. Since bsdtar can use input from tar, pax, cpio, zip, iso9660, 7z, etc., this capability can be used to create verification of the files inside quite a few different formats. You can convert with bsdtar -cf output.mtree --format=mtree @input.tar. There are some foibles with directly using these converted files with mtree -U, but usually minor changes will get it there.

Side mention: stat(1) (Debian package: coreutils)

This tool isn't included because it won't operate recursively, but is a tool in the similar toolbox.

Putting It Together

I will still be developing a complete non-ZFS backup system for NNCP (or UUCP) in a future post. But in the meantime, here are some ideas you can reflect on:

  • Let's say your backup scheme involves sending a full backup every night. On the source system, you could pipe the generated tar file through something like tee >(bsdtar -cf bcakup.mtree @-) to generate an mtree file in-band while generating the tar file. This mtree file could be shipped over for verification.
  • Perhaps your backup scheme involves sending incremental backup data via rdup or even ZFS, but you would like to periodically verify that everything is good -- that an incremental didn't miss something. Something like mtree -K sha256 -c -x -p / | mtree -C -K sha256 would let you accomplish that.

I will further develop at least one of these ideas in a future post.

Bonus: cross-tool comparisons

In my mtree-netbsd packaging, I added tests like this to compare between tools:

fmtree -c -K $(MTREE_KEYWORDS) | mtree
mtree -c -K $(MTREE_KEYWORDS) | sed -e 's/\(md5\|sha1\|sha256\|sha384\|sha512\)=/\1digest=/' -e 's/rmd160=/ripemd160digest=/' | fmtree
bsdtar -cf - --options 'mtree:uname,gname,md5,sha1,sha256,sha384,sha512,device,flags,gid,link,mode,nlink,size,time,uid,type,uname' --format mtree . | mtree

12 January, 2021 05:53PM by John Goerzen

Molly de Blanc

1028 Words on Free Software

The promise of free software is a near-future utopia, built on democratized technology. This future is just and it is beautiful, full of opportunity and fulfillment for everyone everywhere. We can create the things we dream about when we let our minds wander into the places they want to. We can be with the people we want and need to be, when we want and need to.

This is currently possible with the technology we have today, but it’s availability is limited by the reality of the world we live in – the injustice, the inequity, the inequality. Technology runs the world, but it does not serve the interests of most of us. In order to create a better world, our technology must be transparent, accountable, trustworthy. It must be just. It must be free.

The job of the free software movement is to demonstrate that this world is possible by living its values now: justice, equity, equality. We build them into our technology, and we build technology that make it possible for these values to exist in the world.

At the Free Software Foundation, we liked to say that we used all free software because it was important to show that we could. You can do anything with free software, so we did everything with it. We demonstrated the importance of unions for tech workers and non-profit workers by having one. We organized collectively and protected our rights for the sake of ourselves and one another. We had non-negotiable salaries, based on responsibility level and position. That didn’t mean we worked in an office free from the systemic problems that plague workplaces everywhere, but we were able to think about them differently.

Things were this way because of Richard Stallman – but I view his influence on these things as negative rather than positive. He was a cause that forced these outcomes, rather than being supportive of the desires and needs of others. Rather than indulge in gossip or stories, I would like to jump to the idea that he was supposed to have been deplatformed in October 2019. In resigning from his position as president of the FSF, he certainly lost some of his ability to reach audiences. However, Richard still gives talks. The FSF continues to use his image and rhetoric in their own messaging and materials. They gave him time to speak at their annual conference in 2020. He maintains leadership in the GNU project and otherwise within the FSF sphere. The people who empowered him for so many years are still in charge.

Richard, and the continued respect and space he is given, is not the only problem. It represents a bigger problem. Sexism and racism (among others) run rampant in the community. This happens because of bad actors and, more significantly, by the complacency of organizations, projects, and individuals afraid of losing contributors, respect, or funding. In a sector that has so much money and so many resources, women are still being paid less than men; we deny people opportunities to learn and grow in the name of immediate results; people who aren’t men, who aren’t white, are abused and harassed; people are mentally and emotionally taken advantage of, and we are coerced into burn out and giving up our lives for these companies and projects and we are paid for tolerating all of this by being told we’re doing a good job or making a difference.

But we’re not making a difference. We’re perpetuating the worst of the status quo that we should be fighting against. We must not continue. We cannot. We need to live our ideals as they are, and take the natural next steps in their evolution. We cannot have a world of just technology when we live in a world of exclusion; we cannot have free software if we continue to allow, tolerate, and laud the worst of us. I’ve been in and around free software for seventeen years. Nearly every part of it I’ve participated in has members and leadership that benefit from allowing and encouraging the continuation of maleficence and systemic oppression.

We must purge ourselves of these things – of sexism, racism, injustice, and the people who continue and enable it. There is no space to argue over whether a comment was transphobic – if it hurt a trans person then it is transphobic and it is unacceptable. Racism is a global problem and we must be anti-racist or we are complicit. Sexism is present and all men benefit from it, even if they don’t want to. These are free software issues. These are things that plague software, and these are things software reinforces within our societies.

If a technology is exclusionary, it does not work. If a community is exclusionary, it must be fixed or thrown away. There is no middle ground here. There is no compromise. Without doing this, without taking the hard, painful steps to actually live the promise of user freedom and everything it requires and entails, our work is pointless and free software will fail.

I don’t think it’s too late for there to be a radical change – the radical change – that allows us to create the utopia we want to see in the world. We must do that by acknowledging that just technology leads to a just society, and that a just society allows us to make just technology. We must do that by living within the principles that guide this future now.

I don’t know what will happen if things don’t change soon. I recently saw someone comment that change doesn’t happens unless one person is willing to sacrifice everything to make that change, to lead and inspire others to play small parts. This is unreasonable to ask of or expect from someone. I’ve been burning myself out to meet other people’s expectations for seventeen years, and I can’t keep doing it. Of course I am not alone, and I am not the only one working on and occupied by these problems. More people must step up, not just for my sake, but for the sake of all of us, the work free software needs to do, and the future I dream about.

12 January, 2021 04:40PM by mollydb

Petter Reinholdtsen

Latest Jami back in Debian Testing, and scriptable using dbus

After a lot of hard work by its maintainer Alexandre Viau and others, the decentralized communication platform Jami (earlier known as Ring), managed to get its latest version into Debian Testing. Several of its dependencies has caused build and propagation problems, which all seem to be solved now.

In addition to the fact that Jami is decentralized, similar to how bittorrent is decentralized, I first of all like how it is not connected to external IDs like phone numbers. This allow me to set up computers to send me notifications using Jami without having to find get a phone number for each computer. Automatic notification via Jami is also made trivial thanks to the provided client side API (as a DBus service). Here is my bourne shell script demonstrating how to let any system send a message to any Jami address. It will create a new identity before sending the message, if no Jami identity exist already:

#!/bin/sh
#
# Usage: $0  
#
# Send  to , create local jami account if
# missing.
#
# License: GPL v2 or later at your choice
# Author: Petter Reinholdtsen


if [ -z "$HOME" ] ; then
    echo "error: missing \$HOME, required for dbus to work"
    exit 1
fi

# First, get dbus running if not already running
DBUSLAUNCH=/usr/bin/dbus-launch
PIDFILE=/run/asterisk/dbus-session.pid
if [ -e $PIDFILE ] ; then
    . $PIDFILE
    if ! kill -0 $DBUS_SESSION_BUS_PID 2>/dev/null ; then
        unset DBUS_SESSION_BUS_ADDRESS
    fi
fi
if [ -z "$DBUS_SESSION_BUS_ADDRESS" ] && [ -x "$DBUSLAUNCH" ]; then
    DBUS_SESSION_BUS_ADDRESS="unix:path=$HOME/.dbus"
    dbus-daemon --session --address="$DBUS_SESSION_BUS_ADDRESS" --nofork --nopidfile --syslog-only < /dev/null > /dev/null 2>&1 3>&1 &
    DBUS_SESSION_BUS_PID=$!
    (
        echo DBUS_SESSION_BUS_PID=$DBUS_SESSION_BUS_PID
        echo DBUS_SESSION_BUS_ADDRESS=\""$DBUS_SESSION_BUS_ADDRESS"\"
        echo export DBUS_SESSION_BUS_ADDRESS
    ) > $PIDFILE
    . $PIDFILE
fi &

dringop() {
    part="$1"; shift
    op="$1"; shift
    dbus-send --session \
        --dest="cx.ring.Ring" /cx/ring/Ring/$part cx.ring.Ring.$part.$op $*
}

dringopreply() {
    part="$1"; shift
    op="$1"; shift
    dbus-send --session --print-reply \
        --dest="cx.ring.Ring" /cx/ring/Ring/$part cx.ring.Ring.$part.$op $*
}

firstaccount() {
    dringopreply ConfigurationManager getAccountList | \
      grep string | awk -F'"' '{print $2}' | head -n 1
}

account=$(firstaccount)

if [ -z "$account" ] ; then
    echo "Missing local account, trying to create it"
    dringop ConfigurationManager addAccount \
      dict:string:string:"Account.type","RING","Account.videoEnabled","false"
    account=$(firstaccount)
    if [ -z "$account" ] ; then
        echo "unable to create local account"
        exit 1
    fi
fi

# Not using dringopreply to ensure $2 can contain spaces
dbus-send --print-reply --session \
  --dest=cx.ring.Ring \
  /cx/ring/Ring/ConfigurationManager \
  cx.ring.Ring.ConfigurationManager.sendTextMessage \
  string:"$account" string:"$1" \
  dict:string:string:"text/plain","$2" 

If you want to check it out yourself, visit the the Jami system project page to learn more, and install the latest Jami client from Debian Unstable or Testing.

As usual, if you use Bitcoin and want to show your support of my activities, please send Bitcoin donations to my address 15oWEoG9dUPovwmUL9KWAnYRtNJEkP1u1b.

12 January, 2021 04:00PM

Russell Coker

PSI and Cgroup2

In the comments on my post about Load Average Monitoring [1] an anonymous person recommended that I investigate PSI. As an aside, why do I get so many great comments anonymously? Don’t people want to get credit for having good ideas and learning about new technology before others?

PSI is the Pressure Stall Information subsystem for Linux that is included in kernels 4.20 and above, if you want to use it in Debian then you need a kernel from Testing or Unstable (Bullseye has kernel 4.19). The place to start reading about PSI is the main Facebook page about it, it was originally developed at Facebook [2].

I am a little confused by the actual numbers I get out of PSI, while for the load average I can often see where they come from (EG have 2 processes each taking 100% of a core and the load average will be about 2) it’s difficult to work out where the PSI numbers come from. For my own use I decided to treat them as unscaled numbers that just indicate problems, higher number is worse and not worry too much about what the number really means.

With the cgroup2 interface which is supported by the version of systemd in Testing (and which has been included in Debian backports for Buster) you get PSI files for each cgroup. I’ve just uploaded version 1.3.5-2 of etbemon (package mon) to Debian/Unstable which displays the cgroups with PSI numbers greater than 0.5% when the load average test fails.

System CPU Pressure: avg10=0.87 avg60=0.99 avg300=1.00 total=20556310510
/system.slice avg10=0.86 avg60=0.92 avg300=0.97 total=18238772699
/system.slice/system-tor.slice avg10=0.85 avg60=0.69 avg300=0.60 total=11996599996
/system.slice/system-tor.slice/tor@default.service avg10=0.83 avg60=0.69 avg300=0.59 total=5358485146

System IO Pressure: avg10=18.30 avg60=35.85 avg300=42.85 total=310383148314
 full avg10=13.95 avg60=27.72 avg300=33.60 total=216001337513
/system.slice avg10=2.78 avg60=3.86 avg300=5.74 total=51574347007
/system.slice full avg10=1.87 avg60=2.87 avg300=4.36 total=35513103577
/system.slice/mariadb.service avg10=1.33 avg60=3.07 avg300=3.68 total=2559016514
/system.slice/mariadb.service full avg10=1.29 avg60=3.01 avg300=3.61 total=2508485595
/system.slice/matrix-synapse.service avg10=2.74 avg60=3.92 avg300=4.95 total=20466738903
/system.slice/matrix-synapse.service full avg10=2.74 avg60=3.92 avg300=4.95 total=20435187166

Above is an extract from the output of the loadaverage check. It shows that tor is a major user of CPU time (the VM runs a ToR relay node and has close to 100% of one core devoted to that task). It also shows that Mariadb and Matrix are the main users of disk IO. When I installed Matrix the Debian package told me that using SQLite would give lower performance than MySQL, but that didn’t seem like a big deal as the server only has a few users. Maybe I should move Matrix to the Mariadb instance. to improve overall system performance.

So far I have not written any code to display the memory PSI files. I don’t have a lack of RAM on systems I run at the moment and don’t have a good test case for this. I welcome patches from people who have the ability to test this and get some benefit from it.

We are probably about 6 months away from a new release of Debian and this is probably the last thing I need to do to make etbemon ready for that.

12 January, 2021 06:50AM by etbe

RISC-V and Qemu

RISC-V is the latest RISC architecture that’s become popular. It is the 5th RISC architecture from the University of California Berkeley. It seems to be a competitor to ARM due to not having license fees or restrictions on alterations to the architecture (something you have to pay extra for when using ARM). RISC-V seems the most popular architecture to implement in FPGA.

When I first tried to run RISC-V under QEMU it didn’t work, which was probably due to running Debian/Unstable on my QEMU/KVM system and there being QEMU bugs in Unstable at the time. I have just tried it again and got it working.

The Debian Wiki page about RISC-V is pretty good [1]. The instructions there got it going for me. One thing I wasted some time on before reading that page was trying to get a netinst CD image, which is what I usually do for setting up a VM. Apparently there isn’t RISC-V hardware that boots from a CD/DVD so there isn’t a Debian netinst CD image. But debootstrap can install directly from the Debian web server (something I’ve never wanted to do in the past) and that gave me a successful installation.

Here are the commands I used to setup the base image:

apt-get install debootstrap qemu-user-static binfmt-support debian-ports-archive-keyring

debootstrap --arch=riscv64 --keyring /usr/share/keyrings/debian-ports-archive-keyring.gpg --include=debian-ports-archive-keyring unstable /mnt/tmp http://deb.debian.org/debian-ports

I first tried running RISC-V Qemu on Buster, but even ls didn’t work properly and the installation failed.

chroot /mnt/tmp bin/bash
# ls -ld .
/usr/bin/ls: cannot access '.': Function not implemented

When I ran it on Unstable ls works but strace doesn’t work in a chroot, this gave enough functionality to complete the installation.

chroot /mnt/tmp bin/bash
# strace ls -l
/usr/bin/strace: test_ptrace_get_syscall_info: PTRACE_TRACEME: Function not implemented
/usr/bin/strace: ptrace(PTRACE_TRACEME, ...): Function not implemented
/usr/bin/strace: PTRACE_SETOPTIONS: Function not implemented
/usr/bin/strace: detach: waitpid(1602629): No child processes
/usr/bin/strace: Process 1602629 detached

When running the VM the operation was noticably slower than the emulation of PPC64 and S/390x which both ran at an apparently normal speed. When running on a server with equivalent speed CPU a ssh login was obviously slower due to the CPU time taken for encryption, a ssh connection from a system on the same LAN took 6 seconds to connect. I presume that because RISC-V is a newer architecture there hasn’t been as much effort made on optimising the Qemu emulation and that a future version of Qemu will be faster. But I don’t think that Debian/Bullseye will give good Qemu performance for RISC-V, probably more changes are needed than can happen before the freeze. Maybe a version of Qemu with better RISC-V performance can be uploaded to backports some time after Bullseye is released.

Here’s the Qemu command I use to run RISC-V emulation:

qemu-system-riscv64 -machine virt -device virtio-blk-device,drive=hd0 -drive file=/vmstore/riscv,format=raw,id=hd0 -device virtio-blk-device,drive=hd1 -drive file=/vmswap/riscv,format=raw,id=hd1 -m 1024 -kernel /boot/riscv/vmlinux-5.10.0-1-riscv64 -initrd /boot/riscv/initrd.img-5.10.0-1-riscv64 -nographic -append net.ifnames=0 noresume security=selinux root=/dev/vda ro -object rng-random,filename=/dev/urandom,id=rng0 -device virtio-rng-device,rng=rng0 -device virtio-net-device,netdev=net0,mac=02:02:00:00:01:03 -netdev tap,id=net0,helper=/usr/lib/qemu/qemu-bridge-helper

Currently the program /usr/sbin/sefcontext_compile from the selinux-utils package needs execmem access on RISC-V while it doesn’t on any other architecture I have tested. I don’t know why and support for debugging such things seems to be in early stages of development, for example the execstack program doesn’t work on RISC-V now.

RISC-V emulation in Unstable seems adequate for people who are serious about RISC-V development. But if you want to just try a different architecture then PPC64 and S/390 will work better.

12 January, 2021 01:31AM by etbe

John Goerzen

The Good, Bad, and Scary of the Banning of Donald Trump, and How Decentralization Makes It All Better

It is undeniable that banning Donald Trump from Facebook, Twitter, and similar sites is a benefit for the moment. It may well save lives, perhaps lots of lives. But it raises quite a few troubling issues.

First, as EFF points out, these platforms have privileged speakers with power, especially politicians, over regular users. For years now, it has been obvious to everyone that Donald Trump has been violating policies on both platforms, and yet they did little or nothing about it. The result we saw last week was entirely forseeable — and indeed, WAS forseen, including by elements in those companies themselves. (ACLU also raises some good points)

Contrast that with how others get treated. Facebook, two days after the coup attempt, banned Benjamin Wittes, apparently because he mentioned an Atlantic article opposed to nutcase conspiracy theories. The EFF has also documented many more egregious examples: taking down documentation of war crimes, childbirth images, black activists showing the racist messages they received, women discussing online harassment, etc. The list goes on; YouTube, for instance, has often been promoting far-right violent videos while removing peaceful LGBTQ ones.

In short, have we simply achieved legal censorship by outsourcing it to dominant corporations?

It is worth pausing at this point to recognize two important princples:

First, that we do not see it as right to compel speech.

Secondly, that there exist communications channels and other services that nobody is calling on to suspend Donald Trump.

Let’s dive into those a little bit.

There have been no prominent calls for AT&T, Verizon, Gmail, or whomever provides Trump and his campaign with cell phones or email to suspend their service to him. Moreover, the gas stations that fuel his vehicles and the airports that service his plane continue to provide those services, and nobody has seriously questioned that, either. Even his Apple phone that he uses to post to Twitter remains, as far as I know, fully active.

Secondly, imagine you were starting up a small web forum focused on raising tomato plants. It is, and should be, well within your rights to keep tomato-haters out, as well as people that have no interest in tomatoes but would rather talk about rutabagas, politics, or Mars. If you are going to host a forum about tomatoes, you have the right to keep it a forum about tomatoes; you cannot be forced to distribute someone else’s speech. Likewise in traditional media, a newspaper cannot be forced to print every letter to the editor in full.

In law, there is a notion of a common carrier, that provides services to the general public without discrimination. Phone companies and ISPs fall under this.

Facebook, Twitter, and tomato sites don’t. But consider what happens if Facebook bans you. You might be using Facebook-owned Whatsapp to communicate with family and friends, and suddenly find yourself unable to ask someone to pick you up. Or your treasured family photos might be in Facebook-owned Instagram, lost forever. It’s not just Facebook; similar things happen with Google, locking people out of their phones and laptops, their emails, even their photos.

Is it right that Facebook and Google aren’t regulated as common carriers? Perhaps, or perhaps we need some line of demarcation between their speech-to-the-public services (Facebook timeline posts, YouTube) and private communication (Whatsapp, Gmail). It’s a thorny issue; should government be regulating speech instead? That’s also fraught. So is corporate control.

Decentralization Helps Dramatically

With email, you get to pick your email provider (yes, there are two or three big ones, but still plenty of others). Each email provider will have its own set of things it considers acceptable, and its own set of other servers and accounts it’s willing to exchange mail with. (It is extremely common for mail providers to choose not to accept mail from various other mail servers based on ISP, IP address, reputation, and so forth.)

What if we could do something like that for Twitter and Facebook?

Let you join whatever instance you like. Maybe one instance is all about art and they don’t talk about politics. Or another is all about Free Software and they don’t have advertising. And then there are plenty of open instances that accept anything that’s respectful. And, like email, people of one server can interact with those using another just as easily as if they were using the same one.

Well, this isn’t hypothetical; it already exists in the Fediverse. The most common option is Mastodon, and it so happens that a month ago I wrote about its benefits for other reasons, and included some links on getting started.

There is no reason that we must all let our online speech be controlled by companies with a profit motive to keep hate speech on their platforms. There is no reason that we must all have a single set of rules, or accept strong corporate or government control, either. The quality of conversation on Mastodon is far higher than either Twitter or Facebook; decentralization works and it’s here today.

12 January, 2021 01:04AM by John Goerzen

January 11, 2021

hackergotchi for Dirk Eddelbuettel

Dirk Eddelbuettel

BH 1.75.0-0: New upstream release, added Beast

Boost

Boost is a very large and comprehensive set of (peer-reviewed) libraries for the C++ programming language, containing well over 100 individual libraries. The BH package provides a sizeable subset of header-only libraries for use by R.

Version 1.75.0 of Boost was released in December, right on schedule with their April, August and December releases. I now try to follow these releases at a lower (annual) cadence and prepared BH 1.75.0-0 in mid-December. Extensive reverse-depends checks revealed a need for changes in a handful of packages whose maintainers I contacted then. With one exception, everybody responded in kind and brought updated packages to CRAN which permitted us to upload the package there two days ago. And thanks to this planned and coordinated upload, the package is now available on CRAN a mere two days later. My thanks to the maintainers of these packages for helping it along; this prompt responses really are appreciated. The version on CRAN is the same as the one the drat announced in this tweet asking for testing help. If you installed that version, you are still current as no changes were required since December and CRAN now contains same file.

This release adds one new library: Boost Beast, an http and websocket library built on top of Boost Asio. Other changes are highlighed below.

Changes in version 1.75.0-0 (2020-12-12)

  • Removed file NAMESPACE as the package has neither R code, nor a shared library to load

  • The file LICENSE_1_0.txt is now included (as requested in #73)

  • Added new beast library (as requested in #74)

  • Upgraded to Boost 1.75.0 (#75)

Via CRANberries, there is a diffstat report relative to the previous release. Its final line is quite impressive: 3485 files changed, 100838 insertions(+), 84890 deletions(-). Wow.

Comments and suggestions about BH are welcome via the issue tracker at the GitHub repo.

If you like this or other open-source work I do, you can now sponsor me at GitHub.

This post by Dirk Eddelbuettel originated on his Thinking inside the box blog. Please report excessive re-aggregation in third-party for-profit settings.

11 January, 2021 11:57PM

hackergotchi for Bastian Venthur

Bastian Venthur

Dear Apple,

In the light of WhatsApp’s recent move to enforce new Privacy Agreements onto its users, alternative messenger services like Signal are currently gaining some more momentum.

While this sounds good, it is hard to believe that this will be more than a dent in WhatsApp’s user base. WhatsApp is way too ubiquitous, and the whole point of using such a service for most users is to use the one that everyone is using. Unfortunately.

Convincing WhatsApp users to additionally install Signal is hard: they already have SMS for the few people that are not using WhatsApp, now expecting them to install a third app for the same purpose seems ridiculous.

Android mitigates this problem a lot by allowing to make other apps — like Signal — the default SMS/MMS app on the phone. Suddenly people are able to use Signal for SMS/MMS and Signal messages transparently. Signal is smart enough to figure out if the conversation partner is using Signal and enables encryption, video calls and other features. If not, it just falls back to plain old SMS. All in the same app, very convenient for the user!

I don’t really get why the same thing is not possible on iOS? Apple is well known for taking things like privacy and security for its users seriously, and this seems like a low-hanging fruit. So dear Apple, wouldn’t now be a good time to team up with WhatsApp-alternatives like Signal to help the users to make the right choice?

11 January, 2021 09:00PM by Bastian Venthur

January 10, 2021

Enrico Zini

Viewing OpenStreetMap

weeklyOSM posts lots of interesting links. Here are some beautiful views of OpenStreetMap edits and renderings:

10 January, 2021 11:00PM

Tim Retout

Iustin Pop

Dealing with evil ads

Background

I usually don’t mind ads, as not as they not very intrusive. I get that the current media model is basically ad-funded, and that unless I want to pay $1/month or so to 50 web sites, I have to accept ads, so I don’t run an ad-blocker.

Sure, sometimes are annoying (hey YT, mid-roll ads are borderline), but I’ve also seen many good ads, as in interesting or funny even. Well, I don’t think I ever bought anything as direct result from ads, so I don’t know how useful ads are for the companies, but hey, what do I care.

Except… there a few ad networks that run what I would say are basically revolting ads. Things I don’t want to ever accidentally see while eating, or things that are really make you go WTF? Maybe you know them, maybe you don’t, but I guess there are people who don’t know how to clean their ears, or people for whom a fast 7 day weight loss routine actually works.

Thankfully, most of the time I don’t browse sites which use this networks, but randomly they do “leak” to even sites I do browse. If I’m not very stressed already, I can ignore them, otherwise they really, really annoy me.

Case in point, I was on Slashdot, and because I was logged on and recently had mod points, the right side column had a check-box “disable ads”. That sidebar had some relatively meaningful ads, like a VPN subscription (not that I would use it, but it is a tech thing), or even a book about Kali Linux, etc. etc. So I click the “disable ads”, and the right column goes away. I scroll down happily, only to be met, at the bottom, by the “best way to clean your ear”, “the most 50 useless planes ever built” (which had a drawing of something that was for sure never ever built outside of in movies), “you won’t believe how this child actor looks today”, etc.

Solving the problem

The above really, really pissed me off, so I went to search “how to block ad network”. To my surprise, the fix was not that simple, for standard users at least.

Method 1: hosts file

The hosts file is reasonable as it is relatively cross-platform (Linux and Windows and Mac, I think), but how the heck do you edit hosts on your phone?

And furthermore, it has some significant downsides.

First, /etc/hosts lists individual hosts, so for an entire ad network, the example I had had two screens of host names. This is really unmaintainable, since rotating host names, or having a gazillion of them is trivial.

Second, it cannot return negative answers. I.e. you have to give each of those hosts a valid IPv4/IPv6, and have something either reply with 404 or another 4xx response, or not listen on port 80/443. Too annoying.

And finally, it’s a client-side solution, so one would have to replicate it across all clients in a home, and keep it in sync.

Method 2: ad-blockers

I dislike ad-blockers on principle, since they need wide permissions on all pages, but it is a recommended solution. However, to my surprise, one finds threads saying ad-blocker foo has whitelisted ad network bar, at which point you’re WTF? Why do I use an ad-blocker if they get paid by the lowest of the ad networks to show the ads?

And again, it’s a client-side solution, and one would have to deploy it across N clients, and keep them in sync, etc.

Method 3: HTTP proxy blocking

To my surprise, I didn’t find this mentioned in a quick internet search. Well, HTTP proxies have long gone the way of the dodo due to “HTTPs everywhere”, and while one can still use them even with HTTPS, it’s not that convenient:

  • you need to tunnel all traffic through them, which might result in bottlenecks (especially for media playing/maybe video-conference/etc.).
  • or even worse, there might be protocol issues/incompatibilities due to 100% tunneling.
  • running a proxy opens up some potential security issues on the internal network, so you need to harden the proxy as well, and maintain it.
  • you need to configure all clients to know about the proxy (via DHCP or manually), which might or might not work well, since it’s client-dependent.
  • you can only block at CONNECT level (host name), and you have to build up regexes for the host name.

On the good side, the actual blocking configuration is centralised, and the only distributed configuration is pointing the clients through the proxy.

While I used to run a proxy back in HTTP times, the gains were significant back them (media elements caching, downloads caching, all with a slow pipe, etc.), but today is not worth it, so I’ve stopped and won’t bring a proxy back just for this.

Method 4: DNS resolver filtering

After thinking through all the options, I thought - hey, a caching/recursive DNS resolver is what most people with a local network run, right? How difficult is to block at resolver level?

… and oh my, it is so trivial, for some resolvers at least. And yes, I didn’t know about this a week ago 😅

Response Policy Zones

Now, of course, there is a standard for this, called Response Policy Zone, and which is supported across multiple resolvers. There are many tutorials on how to use RPZs to configure things, some of them quite detailed - e.g. this one, or a simple/more straightforward one here.

The upstream BIND documentation also explains things quite well here, so you can go that route as well. It looks a bit hairy for me thought, but it works, and since it is a standard, it can be more easily deployed.

There are many discussions on the internet about how to configure RPZs, how to not even resolve the names (if you’re going to return something explicitly/statically), etc. so there are docs, but again it seems a bit overdone.

Resolver hooks

There’s another way too, if your resolver allows scripting. For example, the PowerDNS resolver allow Lua scripting, and has a relatively simple API—at least, to me it looks way, way simpler than the RPZ equivalent.

After 20 minutes of reading the docs, I ended up with this, IMO trivial, solution (in a file named e.g. rules.lua):

ads = newDS()
ads:add({'evilads.com', 'evilads.well-known-cdn.com', 'moreads.net'})

function preresolve(dq)
  if ads:check(dq.qname) then
    dq.rcode = pdns.NXDOMAIN
    return true;
  end
  return false;
end

… and that’s it. Well, enable it/load the file in the configuration, but nothing else. Syntax is pretty straightforward, matching by suffix here, and if you need more complex stuff, you can of course do it; it’s just Lua and a simple API.

I don’t see any immediate equivalent in Bind, so there’s that, but if you can use PowerDNS, then the above solution seems simple for simple cases, and could be extended if needed (not sure in which cases).

The only other thing one needs to do is to serve the local/custom resolver to all clients, whether desktop or mobile, and that’s it. DNS server is bread-and-butter in DHCP, so better support than proxy, and once the host name has been (mis)resolved, nothing is involved anymore in the communication path. True, your name server might get higher CPU usage, but for home network, this should not be a problem.

Can this filtering method (either RPZ or hooks) be worked around by ad networks? Sure, like anything. But changing the base domain is not fun. DNSSEC might break it (note Bind RPZ can be configure to ignore DNSSEC), but I’m more worried about DNS-over-HTTPS, which I thought initially it’s done for the user, but now I’m not so sure anymore. Not being in control even of your own DNS resolver seems… evil 😈, but what do I know.

Happy browsing!

10 lines of Lua, and now for sure I’m going to get even fatter without the “this natural method will melt your belly fat in 7 days” information. Or I will just throw away banana peels without knowing what I could do with hem.

After a few days, I asked myself “but ads are not so bad, why did I…” and then realised that yes, ads are not so bad anymore. And Slashdot actually loads faster 😜

So, happy browsing!

10 January, 2021 09:33PM

hackergotchi for Dirk Eddelbuettel

Dirk Eddelbuettel

RcppArmadillo 0.10.1.2.2: Minor update

armadillo image

Armadillo is a powerful and expressive C++ template library for linear algebra aiming towards a good balance between speed and ease of use with a syntax deliberately close to a Matlab. RcppArmadillo integrates this library with the R environment and language–and is widely used by (currently) 802 other packages on CRAN.

This release was needed because we use the Matrix package for some (optional) tests related to sparse matrices, and a small and subtle change and refinement in the recent 1.3.0 release of Matrix required us to make an update for the testing. Nothing has changed in how we set up, or operate on, sparse matrices. My thanks to Binxiang and Martin Maechler for feedback and suggestion on the initial fix both Binxiang and I set up independently. At the same time we upgrade some package internals related to continuous integration (for that, also see my blog post and video from earlier this week). Lastly Conrad sent in a one-line upstream fix for dealing with NaN in sign().

The full set of changes follows.

Changes in RcppArmadillo version 0.10.1.2.2 (2021-01-08)

  • Correct one unit test for Matrix 1.3.0-caused changed (Binxiang in #319 and Dirk in #322).

  • Suppress one further warning from Matrix (Dirk)

  • Apply an upstream NaN correction (Conrad in #321)

  • Added GitHub Actions CI using run.sh from r-ci (Dirk)

Courtesy of my CRANberries, there is a diffstat report relative to previous release. More detailed information is on the RcppArmadillo page. Questions, comments etc should go to the rcpp-devel mailing list off the R-Forge page.

If you like this or other open-source work I do, you can sponsor me at GitHub.

This post by Dirk Eddelbuettel originated on his Thinking inside the box blog. Please report excessive re-aggregation in third-party for-profit settings.

10 January, 2021 08:20PM

hackergotchi for Kentaro Hayashi

Kentaro Hayashi

Use external E-mail server for debian.net subdomain with Sakura Mailbox service

If you want to set up debian.net subdomain, you may setup E-mail server on your own. But if there is not afford to setup it by yourself, you need external E-mail server.

In this article, I'll explain how to use external mailbox service - Sakura Mailbox service.

Prerequisite

  • Owner of debian.net subdomain (fabre.debian.net)
  • Have an account of Sakura mailbox service

www.sakura.ne.jp

I've chosen Sakura mailbox service because of maintainance cost (87yen/month)

Configure dnsZoneEntry

Set dnZoneEntry, it means that gpg --clearsign and send to changes@db.debian.org. Here is an example for fabre.debian.net.

fabre IN A 140.227.125.189
mail.fabre IN A 49.212.207.226
fabre IN MX 10 mail.fabre.debian.net.

140.227.125.189 is Web server (https://fabre.debian.net) and 49.212.207.226 is E-mail server (Sakura mailbox service) for my case.

Note that It varies for your case.

DebianDotNet - Debian Wiki

Configure Sakura mailbox service

Set fabre.debian.net subdomain without transfer in Domain/SSL menu. Then create each account for @fabre.debian.net. After a while, you can use xxx@fabre.debian.net E-mail account. Yay!

10 January, 2021 01:20PM

January 09, 2021

hackergotchi for Jonathan McDowell

Jonathan McDowell

Free Software Activities for 2020

As a reader of Planet Debian I see a bunch of updates at the start of each month about what people are up to in terms of their Free Software activities. I’m not generally active enough in the Free Software world to justify a monthly report, but I did a report of my Free Software Activities for 2019 and thought I’d do another for 2020. I ended up not doing as much as last year; I put a lot of that down to fatigue about the state of the world and generally not wanting to spend time on the computer at the end of the working day.

Conferences

2020 was unsurprisingly not a great year for conference attendance. I was fortunate enough to make it to FOSDEM and CopyleftConf 2020 - I didn’t speak at either, but had plenty of interesting hallway track conversations as well as seeing some good talks. I hadn’t been planning to attend DebConf20 due to time constraints, but its move to an entirely online conference meant I was able to attend a few talks at least. I have to say I don’t like virtual conferences as much as the real thing; it’s not as easy to have the casual chats at them, and it’s also harder to carve out the exclusive time when you’re at home. That said I spoke at NIDevConf this year, which was also fully virtual. It’s not a Free Software focussed conference, but there’s a lot of crossover in terms of technologies and I spoke on my experiences with Go, some of which are influenced by my packaging experiences within Debian.

Debian

Most of my contributions to Free software happen within Debian.

As part of the Data Protection Team I responded to various inbound queries to that team. Some of this involved chasing up other project teams who had been slow to respond - folks, if you’re running a service that stores personal data about people then you need to be responsive to requests about it.

The Debian Keyring was possibly my largest single point of contribution. We’re in a roughly 3 month rotation of who handles the keyring updates, and I handled 2020.02.02, 2020.03.24, 2020.06.24, 2020.09.24 + 2020.12.24

For Debian New Members I’m mostly inactive as an application manager - we generally seem to have enough available recently. If that changes I’ll look at stepping in to help, but I don’t see that happening. I continue to be involved in Front Desk, having various conversations throughout the year with the rest of the team, but there’s no doubt Mattia and Pierre-Elliott are the real doers at present.

In terms of package uploads I continued to work on gcc-xtensa-lx106, largely doing uploads to deal with updates to the GCC version or packaging (5, 6 + 7). sigrok had a few minor updates, libsigkrok 0.5.2-2, libsigrokdecode 0.5.3-2 as well as a new upstream release of Pulseview 0.4.2-1 and a fix to cope with change to QT 0.4.2-2. Due to the sigrok-firmware requirement on sdcc I also continued to help out there, updating to 4.0.0+dfsg-1 and doing some fixups in 4.0.0+dfsg-2.

Despite still not writing an VHDL these days I continue to try and make sure ghdl is ok, because I found it a useful tool in the past. In 2020 that meant a new upstream release, 0.37+dfsg-1 along with a couple of more minor updates (0.37+dfsg-2 + 0.37+dfsg-3.

libcli had a new upstream release, 1.10.4-1, and I did a long overdue update to sendip to the latest upstream release, 2.6-1 having been poked about an outstanding bug by the Reproducible Builds folk.

OpenOCD is coming up to 4 years since its last stable release, but I did a snapshot upload to Debian experimental (0.10.0+g20200530-1) and a subsequent one to unstable (0.10.0+g20200819-1). There are also moves to produce a 0.11.0 release and I uploaded 0.11.0~rc1-1 as a result. libjaylink got a bump as a result (0.2.0-1) after some discussion with upstream.

OpenOCD

On the subject of OpenOCD I’ve tried to be a bit more involved upstream. I’m not familiar enough with the intricacies of JTAG/SWD/the various architectures supported to contribute to the core, but I pushed the config for my HIE JTAG adapter upstream and try and review patches that don’t require in depth hardware knowledge.

Linux

I’ve been contributing to the Linux kernel for a number of years now, mostly just minor bits here and there for issues I hit. This year I spent a lot of time getting support for the MikoTik RB3011 router upstreamed. That included the basic DTS addition, fixing up QCA8K to support SGMII CPU connections, adding proper 802.1q VLAN support to QCA8K and cleaning up an existing QCOM ADM driver that’s required for the NAND. There were a number of associated bugfixes/minor changes found along the way too. It can be a little frustrating at times going round the review loop with submitting things upstream, but I do find it quite satisfying when it all comes together and I have no interest in weird vendor trees that just bitrot over time.

Software in the Public Interest

I haven’t sat on the board of SPI since 2015 but I was still acting as the primary maintainer of the membership website (with Martin Michlmayr as the other active contributor) and hosting it on my own machine. I managed to finally extricate myself from this role in August. I remain a contributing member.

Personal projects

2020 finally saw another release (0.6.0, followed swiftly by 0.6.1 to allow the upload of 0.6.1-1 to Debian) of onak. This release finally adds various improvements to deal with the hostility shown to the OpenPGP keyserver network in recent years, including full signature verification as an option.

I fixed an oversight in my Digoo/1-wire temperature decoder and a bug that turned up on ARM but not MIPS in my mqtt-arp code. I should probably package it for Debian (even if I don’t upload it), as I’m running it on my RB3011 now.

09 January, 2021 06:09PM

Thorsten Alteholz

My Debian Activities in December 2020

FTP master

This month I only accepted 8 packages and like last month rejected 0. Despite the holidays 293 packages got accepted.

Debian LTS

This was my seventy-eighth month that I did some work for the Debian LTS initiative, started by Raphael Hertzog at Freexian.

This month my all in all workload has been 26h. During that time I did LTS uploads of:

  • [DLA 2489-1] minidlna security update for two CVEs
  • [DLA 2490-1] x11vnc security update for one CVE
  • [DLA 2501-1] influxdb security update for one CVE
  • [DLA 2511-1] highlight.js security update for one CVE

Unfortunately package slirp has the same version in Stretch and Buster. So I first had to upload slirp/1:1.0.17-11 to unstable, in order to be allowed to fix the CVE in Buster and to finally upload a new version to Stretch. Meanwhile the fix for Buster has been approved by the Release Team and I am waiting for the next point release now.

I also prepared a debdiff for influxdb, which will result in DSA-4823-1 in January.

As there appeared new CVEs for openjpeg2, I did not do an upload yet. This is planned for January now.

Last but not least I did some days of frontdesk duties.

Debian ELTS

This month was the thirtieth ELTS month.

During my allocated time I uploaded:

  • ELA-341-1 for highlight.js

As well as for LTS, I did not finish work on all CVEs of openjpeg2, so the upload is postponed to January.

Last but not least I did some days of frontdesk duties.

Unfortunately I also had to give back some hours.

Other stuff

This month I uploaded new upstream versions of:

I fixed one or two bugs in:

I improved packaging of:

Some packages just needed a source upload:

… and there have been even some new packages:

With these uploads I finished the libosmocom- and libctl-transitions.

The Debian Med Advent Calendar was again really successful this year. There was no new record, but with 109, the second most number of bugs has been closed.

year number of bugs closed
2011 63
2012 28
2013 73
2014 5
2015 150
2016 95
2017 105
2018 81
2019 104
2020 109

Well done everybody who participated. It is really nice to see that Andreas is no longer a lone wolf.

09 January, 2021 07:34AM by alteholz

hackergotchi for Louis-Philippe Véronneau

Louis-Philippe Véronneau

puppetserver 6: a Debian packaging post-mortem

I have been a Puppet user for a couple of years now, first at work, and eventually for my personal servers and computers. Although it can have a steep learning curve, I find Puppet both nimble and very powerful. I also prefer it to Ansible for its speed and the agent-server model it uses.

Sadly, Puppet Labs hasn't been the most supportive upstream and tends to move pretty fast. Major versions rarely last for a whole Debian Stable release and the upstream .deb packages are full of vendored libraries.1

Since 2017, Apollon Oikonomopoulos has been the one doing most of the work on Puppet in Debian. Sadly, he's had less time for that lately and with Puppet 5 being deprecated in January 2021, Thomas Goirand, Utkarsh Gupta and I have been trying to package Puppet 6 in Debian for the last 6 months.

With Puppet 6, the old ruby Puppet server using Passenger is not supported anymore and has been replaced by puppetserver, written in Clojure and running on the JVM. That's quite a large change and although puppetserver does reuse some of the Clojure libraries puppetdb (already in Debian) uses, packaging it meant quite a lot of work.

Work in the Clojure team

As part of my efforts to package puppetserver, I had the pleasure to join the Clojure team and learn a lot about the Clojure ecosystem.

As I mentioned earlier, a lot of the Clojure dependencies needed for puppetserver were already in the archive. Unfortunately, when Apollon Oikonomopoulos packaged them, the leiningen build tool hadn't been packaged yet. This meant I had to rebuild a lot of packages, on top of packaging some new ones.

Since then, thanks to the efforts of Elana Hashman, leiningen has been packaged and lets us run the upstream testsuites and create .jar artifacts closer to those upstream releases.

During my work on puppetserver, I worked on the following packages:

List of packages
  • backport9
  • bidi-clojure
  • clj-digest-clojure
  • clj-helper
  • clj-time-clojure
  • clj-yaml-clojure
  • cljx-clojure
  • core-async-clojure
  • core-cache-clojure
  • core-match-clojure
  • cpath-clojure
  • crypto-equality-clojure
  • crypto-random-clojure
  • data-csv-clojure
  • data-json-clojure
  • data-priority-map-clojure
  • java-classpath-clojure
  • jnr-constants
  • jnr-enxio
  • jruby
  • jruby-utils-clojure
  • kitchensink-clojure
  • lazymap-clojure
  • liberator-clojure
  • ordered-clojure
  • pathetic-clojure
  • potemkin-clojure
  • prismatic-plumbing-clojure
  • prismatic-schema-clojure
  • puppetlabs-http-client-clojure
  • puppetlabs-i18n-clojure
  • puppetlabs-ring-middleware-clojure
  • puppetserver
  • raynes-fs-clojure
  • riddley-clojure
  • ring-basic-authentication-clojure
  • ring-clojure
  • ring-codec-clojure
  • shell-utils-clojure
  • ssl-utils-clojure
  • test-check-clojure
  • tools-analyzer-clojure
  • tools-analyzer-jvm-clojure
  • tools-cli-clojure
  • tools-reader-clojure
  • trapperkeeper-authorization-clojure
  • trapperkeeper-clojure
  • trapperkeeper-filesystem-watcher-clojure
  • trapperkeeper-metrics-clojure
  • trapperkeeper-scheduler-clojure
  • trapperkeeper-webserver-jetty9-clojure
  • url-clojure
  • useful-clojure
  • watchtower-clojure

If you want to learn more about packaging Clojure libraries and applications, I rewrote the Debian Clojure packaging tutorial and added a section about the quirks of using leiningen without a dedicated dh_lein tool.

Work left to get puppetserver 6 in the archive

Unfortunately, I was not able to finish the puppetserver 6 packaging work. It is thus unlikely it will make it in Debian Bullseye. If the issues described below are fixed, it would be possible to to package puppetserver in bullseye-backports though.

So what's left?

jruby

Although I tried my best (kudos to Utkarsh Gupta and Thomas Goirand for the help), jruby in Debian is still broken. It does build properly, but the testsuite fails with multiple errors:

  • ruby-psych is broken (#959571)
  • there are some random java failures on a few tests (no clue why)
  • tests ran by raklelib/rspec.rake fail to run, maybe because the --pattern command line option isn't compatible with our version of rake? Utkarsh seemed to know why this happens.

jruby testsuite failures aside, I have not been able to use the jruby.deb the package currently builds in jruby-utils-clojure (testsuite failure). I had the same exact failure with the (more broken) jruby version that is currently in the archive, which leads me to think this is a LOAD_PATH issue in jruby-utils-clojure. More on that below.

To try to bypass these issues, I tried to vendor jruby into jruby-utils-clojure. At first I understood vendoring meant including upstream pre-built artifacts (jruby-complete.jar) and shipping them directly.

After talking with people on the #debian-mentors and #debian-ftp IRC channels, I now understand why this isn't a good idea (and why it's not permitted in Debian). Many thanks to the people who were patient and kind enough to discuss this with me and give me alternatives.

As far as I now understand it, vendoring in Debian means "to have an embedded copy of the source code in another package". Code shipped that way still needs to be built from source. This means we need to build jruby ourselves, one way or another. Vendoring jruby in another package thus isn't terribly helpful.

If fixing jruby the proper way isn't possible, I would suggest trying to build the package using embedded code copies of the external libraries jruby needs to build, instead of trying to use the Debian libraries.2 This should make it easier to replicate what upstream does and to have a final .jar that can be used.

jruby-utils-clojure

This package is a first-level dependency for puppetserver and is the glue between jruby and puppetserver.

It builds fine, but the testsuite fails when using the Debian jruby package. I think the problem is caused by a jruby LOAD_PATH issue.

The Debian jruby package plays with the LOAD_PATH a little to try use Debian packages instead of downloading gems from the web, as upstream jruby does. This seems to clash with the gem-home, gem-path, and jruby-load-path variables in the jruby-utils-clojure package. The testsuite plays around with these variables and some Ruby libraries can't be found.

I tried to fix this, but failed. Using the upstream jruby-complete.jar instead of the Debian jruby package, the testsuite passes fine.

This package could clearly be uploaded to NEW right now by ignoring the testsuite failures (we're just packaging static .clj source files in the proper location in a .jar).

puppetserver

jruby issues aside, packaging puppetserver itself is 80% done. Using the upstream jruby-complete.jar artifact, the testsuite fails with a weird Clojure error I'm not sure I understand, but I haven't debugged it for very long.

Upstream uses git submodules to vendor puppet (agent), hiera (3), facter and puppet-resource-api for the testsuite to run properly. I haven't touched that, but I believe we can either:

  • link to the Debian packages
  • fix the Debian packages if they don't include the right files (maybe in a new binary package that just ships part of the source code?)

Without the testsuite actually running, it's hard to know what files are needed in those packages.

What now

Puppet 5 is now deprecated.

If you or your organisation cares about Puppet in Debian,3 puppetserver really isn't far away from making it in the archive.

Very talented Debian Developers are always eager to work on these issues and can be contracted for very reasonable rates. If you're interested in contracting someone to help iron out the last issues, don't hesitate to reach out via one of the following:

As for I, I'm happy to say I got a new contract and will go back to teaching Economics for the Winter 2021 session. I might help out with some general Debian packaging work from time to time, but it'll be as a hobby instead of a job.

Thanks

The work I did during the last 6 weeks would be not have been possible without the support of the Wikimedia Foundation, who were gracious enough to contract me. My particular thanks to Faidon Liambotis, Moritz Mühlenhoff and John Bond.

Many, many thanks to Rob Browning, Thomas Goirand, Elana Hashman, Utkarsh Gupta and Apollon Oikonomopoulos for their direct and indirect help, without which all of this wouldn't have been possible.


  1. For example, the upstream package for the Puppet Agent vendors OpenSSL. 

  2. One of the problems of using Ruby libraries already packaged in Debian is that jruby currently only supports Ruby 2.5. Ruby libraries in Debian are currently expected to work with Ruby 2.7, with the transition to Ruby 3.0 planned after the Bullseye release. 

  3. If you run Puppet, you clearly should care: the .deb packages upstream publishes really aren't great and I would not recommend using them. 

09 January, 2021 05:00AM by Louis-Philippe Véronneau

January 08, 2021

hackergotchi for Dirk Eddelbuettel

Dirk Eddelbuettel

#32: Portable Continuous Integration using r-ci

Welcome to the 32th post in the rarely raucous R recommendations series, or R4 for short. This post covers continuous integration, a topic near and dear to many of us who have to recognise its added value.

The popular and widely-used service at Travis is undergoing changes driven by a hard-to-argue with need for monetization. A fate that, if we’re honest, lies ahead for most “free” services so who know, maybe one day we have to turn away from other currently ubiquitous service. Because one never knows, it can pay off to not get to tied to any one service. Which brings us to today’s post and my r-ci service which allows me to run CI at Travis, at GitHub, at Azure, and on local Docker containers as the video demonstrates. It will likely also work at GitLab and other services, I simply haven’t tried any others.

The slides are here. The r-ci website introduces r-ci at a high-level. This repo at GitHub contains run.sh, and can be used to raise issues, ask questions, or provide feedback.

If you like this or other open-source work I do, you can sponsor me at GitHub.

This post by Dirk Eddelbuettel originated on his Thinking inside the box blog. Please report excessive re-aggregation in third-party for-profit settings.

08 January, 2021 04:09AM

Reproducible Builds (diffoscope)

diffoscope 164 released

The diffoscope maintainers are pleased to announce the release of diffoscope version 164. This version includes the following changes:

[ Chris Lamb ]
* Truncate jsondiff differences at 512 bytes lest they consume the entire page.
* Wrap our external call to cmp(1) with a profile (to match the internal
  profiling).
* Add a note regarding the specific ordering of the new
  all_tools_are_listed test.

[ Dimitrios Apostolou ]
* Performance improvements:
  - Improve speed of has_same_content by spawning cmp(1) less frequently.
  - Log whenever the external cmp(1) command is spawn.ed
  - Avoid invoking external diff for identical, short outputs.
* Rework handling of temporary files:
  - Clean up temporary directories as we go along, instead of at the end.
  - Delete FIFO files when the FIFO feeder's context manager exits.

[ Mattia Rizzolo ]
* Fix a number of potential crashes in --list-debian-substvars, including
  explicitly listing lipo and otool as external tools.
 - Remove redundant code and let object destructors clean up after themselves.

[ Conrad Ratschan ]
* Add a comparator for Flattened Image Trees (FIT) files, a boot image format
  used by U-Boot.

You find out more by visiting the project homepage.

08 January, 2021 12:00AM

January 07, 2021

Russell Coker

Monopoly the Game

The Smithsonian Mag has an informative article about the history of the game Monopoly [1]. The main point about Monopoly teaching about the problems of inequality is one I was already aware of, but there are some aspects of the history that I learned from the article.

Here’s an article about using modified version of Monopoly to teach Sociology [2].

Maria Paino and Jeffrey Chin wrote an interesting paper about using Monopoly with revised rules to teach Sociology [3]. They publish the rules which are interesting and seem good for a class.

I think it would be good to have some new games which can teach about class differences. Maybe have an “Escape From Poverty” game where you have choices that include drug dealing to try and improve your situation or a cooperative game where people try to create a small business. While Monopoly can be instructive it’s based on the economic circumstances of the past. The vast majority of rich people aren’t rich from land ownership.

07 January, 2021 05:05AM by etbe

John Goerzen

This Is How Tyrants Go: Alone

I remember reading an essay a month or so ago — sadly I forget where — talking about how things end for tyrants. If I were to sum it up, it would be with the word “alone.” Their power fading, they find that they had few true friends or believers; just others that were greedy for power or riches and, finding those no longer to be had, depart the sinking ship. The article looked back at examples like Nixon and examples from the 20th century in Europe and around the world.

Today we saw images of a failed coup attempt.

But we also saw hope.

Already senior staff in the White House are resigning. Ones that had been ardent supporters. In the end, just 6 senators supported the objection to the legitimate electors. Six. Lindsay Graham, Mike Pence, and Mitch McConnel all deserted Trump.

CNN reports that there are serious conversations about invoking the 25th amendment and removing him from office, because even Republicans are to the point of believing that America should not have two more weeks of this man.

Whether those efforts are successful or not, I don’t know. What I do know is that these actions have awakened many people, in a way that nothing else could for four years, to the dangers of Trump and, in the end, have bolstered the cause of democracy.

Hard work will remain but today, Donald Trump is in the White House alone, abandoned by allies and blocked by Twitter. And we know that within two weeks, he won’t be there at all.

We will get through this.

07 January, 2021 04:00AM by John Goerzen

January 06, 2021

hackergotchi for Urvika Gola

Urvika Gola

Dog tails and tales from 2020

There is no denying in the fact that 2020 was a challenging year for everybody, including animals.

In India, animals such as dogs who mostly filled their bellies at street food stalls, were starving as there were no street eateries operating during a long long lockdown. I was in my home town, New Delhi, working from home like most of us.

During the month of July 2020, a dog near the place I live (we fondly called her Brownie) delivered 7 pups in wilderness.

I would never forget my first sight of them, inside a dirty, garbage filled land there were the cutest, cleanest, tiniest ball of fur! All of them were toppled as the land’s surface was uneven.. The first instinct was to put them all together on a flat surface. After the search mission completed, this was the sight..

Brownie and her litter put together in the same land where she gave birth.

The next day, I sought help from a animal-lover person to build a temporary shed for the puppies! We came and changed sheets, cleaned the surroundings and put fresh water for Brownie until…

..it started raining heavily one night and we were worried if the shed would sustain the heavy rainfall.
Next morning the first thing was to check on the pups, luckily, the pups were fine however, the entire area and their bed was damp.

Without any second thought, the pups were moved from there to a safe house shelter as it was predicted that the rains will continue for a few more weeks due to monsoon. Soon, 2 months went by, from observing the pups crawl over, their eyes open and to their first bark, despite the struggles, it was an beautiful experience.
Brownie weaned off the pups and thus, they were ready for adoption! However, my biggest fear was, will anyone come forward to adopt them??

With such thoughts parallelly running in my mind, I started to post about adoption for these 7 pups.
To my biggest surprise, one by one, 5 amazing humans came forward and decided to give these pups a better life than what they would get on the streets of India. I wouldn’t be able to express in words how grateful I will to be all the five dog parents who decided to adopt an Indian Street Puppy/Indies/Desi Puppy, opening up the space in their hearts and homes for the pups!

One of the 5 adopted pups is adopted by a person who hails from USA, but currently working in India. It’s so heartwarming to see, that in India, despite so much awareness created against breeders and their methods, people still prefer to go for foreign bred puppy and disregard Indian/Desi Dogs.. On the other hand, there are foreigners who value the life of a Indian/Desi Dog :”)

The 5 Adopted Pups who now have a permanent loving family!

The adorable, “Robin”!
“Don” and his new big brother!

The naughty and handsome, “Swayze”!

First Pup who got adopted – “Pluto”
Playful and Beautiful, “Bella”!

If this isn’t perfect, I don’t know what is! God had planned loving families for them and they found it..
However, Its been almost six months now, that we haven’t found a permanent home for 2 of the 7 pups, but they have the perfect foster family taking care of them right now.

UP FOR ADOPTION – Delhi/NCR
Meet Momo and Beesa,
2 out of the 7 pups, who are still waiting for a forever home, currently living with a loving foster family.

Vaccinations, Deworming is done.
Female pups, 6 months old.

Now as winters are here, Along with one of my friend, who is also fostering the two pups, arranged gunny sack bags for our street, stray dogs. Two NGOs namely, Lotus Indie Foundation and We Exist Foundation who work Animal Welfare in India, were providing dog beds to ground volunteers like us. We are fortunate that they selected us and helped us to make winters less harsh for the stray dogs. However, the cold is such, I also purchased dog coats as well and put in on a few furries. After hours of running behind dogs and convincing them to wear coats, we managed to put it on a few.

Brownie, the mom dog!

This is a puppy!
She did not let us put coat on her 😀

Another topic that needs more sensitivity is Sterilization/Neutering of dogs, that’s a viable method cited by the Government to control dog population and end suffering of puppies who die under the wheels of cars. However, the implementation of this is worrisome, as it’s not as robust. In a span of 6 months, I managed to get 5 dog sterilized in my area, number is not big but I feel it’s a good start as an individual 😊

When I see them now, healthier, happier, running around, with no fear of getting attacked by dogs, I can’t express the content I feel. For 2 of the dogs (Brownie and her friend) I got it done personally from a private vet. For the other 3, I got it done via Municipal Corporations who do it for free for dogs, you’d have to call them and they come with dog catchers and a van and drop them back in the same area, but volunteers like us have to be very vigilant and active during the whole process to follow up with them.

Dogs getting dropped off after sterilization.

My 2020 ended with this, I am not sure why I am I even writing this in my blog where mostly I focused on my technical work and experiences, but this pandemic was challenging was everybody and what we planned couldn’t happen, but because of 2020, because of the pandemic, I was on WFH in my city and I was able to help a few dogs in my area have a healthy life ahead! 😊

What I learned during this entire adventure was, there are a lot of sweet, sensitive, caring people that we are just yet to meet. Along the way, we will also meet insensitive and discouraging people, who are unwilling to change or listen, ignore them and continue your good.

Have 1 person by your side, it’s so much stronger than 10 against you.

Silver lining! Hope you all had some positive experiences despite the adversity faced by every single one of us.

06 January, 2021 05:52PM by urvikagola

hackergotchi for Jonathan Dowland

Jonathan Dowland

PaperWM

My PaperWM desktop, as I write this post.

My PaperWM desktop, as I write this post.

Just before Christmas I decided to try out a GNOME extension I'd read about, PaperWM. It looked promising, but I was a little nervous about breaking my existing workflow, which was heavily reliant on the Put Windows extension.

It's great! I have had to carefully un-train some of my muscle memory but it seems to be worth it. It seems to strike a great balance between the rigidity of a tile-based window manager and a more traditional floating-windows one.

I'm always wary of coming to rely upon large extensions or plugins. The parent software is often quite hands-off about caring about supporting users of them, or breaking them by making API changes. Certainly those Firefox users who were heavily dependent on plugins prior to the Quantum fire-break are still very, very angry. (I actually returned to Firefox at that point, so I avoided the pain, and enjoy the advantages of the re-architecture). PaperWM hopefully is large enough and popular enough to avoid that fate.

06 January, 2021 04:44PM

January 05, 2021

Russell Coker

Planet Linux Australia

Linux Australia have decided to cease running the Planet installation on planet.linux.org.au. I believe that blogging is still useful and a web page with a feed of Australian Linux blogs is a useful service. So I have started running a new Planet Linux Australia on https://planet.luv.asn.au/. There has been discussion about getting some sort of redirection from the old Linux Australia page, but they don’t seem able to do that.

If you have a blog that has a reasonable portion of Linux and FOSS content and is based in or connected to Australia then email me on russell at coker.com.au to get it added.

When I started running this I took the old list of feeds from planet.linux.org.au, deleted all blogs that didn’t have posts for 5 years and all blogs that were broken and had no recent posts. I emailed people who had recently broken blogs so they could fix them. It seems that many people who run personal blogs aren’t bothered by a bit of downtime.

As an aside I would be happy to setup the monitoring system I use to monitor any personal web site of a Linux person and notify them by Jabber or email of an outage. I could set it to not alert for a specified period (10 mins, 1 hour, whatever you like) so it doesn’t alert needlessly on routine sysadmin work and I could have it check SSL certificate validity as well as the basic page header.

05 January, 2021 11:45PM by etbe

Weather and Boinc

I just wrote a Perl script to look at the Australian Bureau of Meteorology pages to find the current temperature in an area and then adjust BOINC settings accordingly. The Perl script (in this post after the break, which shouldn’t be in the RSS feed) takes the URL of a Bureau of Meteorology observation point as ARGV[0] and parses that to find the current (within the last hour) temperature. Then successive command line arguments are of the form “24:100” and “30:50” which indicate that at below 24C 100% of CPU cores should be used and below 30C 50% of CPU cores should be used. In warm weather having a couple of workstations in a room running BOINC (or any other CPU intensive task) will increase the temperature and also make excessive noise from cooling fans.

To change the number of CPU cores used the script changes /etc/boinc-client/global_prefs_override.xml and then tells BOINC to reload that config file. This code is a little ugly (it doesn’t properly parse XML, it just replaces a line of text) and could fail on a valid configuration file that wasn’t produced by the current BOINC code.

The parsing of the BoM page is a little ugly too, it relies on the HTML code in the BoM page – they could make a page that looks identical which breaks the parsing or even a page that contains the same data that looks different. It would be nice if the BoM published some APIs for getting the weather. One thing that would be good is TXT records in the DNS. DNS supports caching with specified lifetime and is designed for high throughput in aggregate. If you had a million IOT devices polling the current temperature and forecasts every minute via DNS the people running the servers wouldn’t even notice the load, while a million devices polling a web based API would be a significant load. As an aside I recommend playing nice and only running such a script every 30 minutes, the BoM page seems to be updated on the half hour so I have my cron jobs running at 5 and 35 minutes past the hour.

If this code works for you then that’s great. If it merely acts as an inspiration for developing your own code then that’s great too! BOINC users outside Australia could replace the code for getting meteorological data (or even interface to a digital thermometer). Australians who use other CPU intensive batch jobs could take the BoM parsing code and replace the BOINC related code. If you write scripts inspired by this please blog about it and comment here with a link to your blog post.

#!/usr/bin/perl
use strict;
use Sys::Syslog;

# St Kilda Harbour RMYS
# http://www.bom.gov.au/products/IDV60901/IDV60901.95864.shtml

my $URL = $ARGV[0];

open(IN, "wget -o /dev/null -O - $URL|") or die "Can't get $URL";
while(<IN>)
{
  if($_ =~ /tr class=.rowleftcolumn/)
  {
    last;
  }
}

sub get_data
{
  if(not $_[0] =~ /headers=.t1-$_[1]/)
  {
    return undef;
  }
  $_[0] =~ s/^.*headers=.t1-$_[1]..//;
  $_[0] =~ s/<.td.*$//;
  return $_[0];
}

my @datetime;
my $cur_temp -100;

while(<IN>)
{
  chomp;
  if($_ =~ /^<.tr>$/)
  {
    last;
  }
  my $res;
  if($res = get_data($_, "datetime"))
  {
    @datetime = split(/\//, $res)
  }
  elsif($res = get_data($_, "tmp"))
  {
    $cur_temp = $res;
  }
}
close(IN);
if($#datetime != 1 or $cur_temp == -100)
{
  die "Can't parse BOM data";
}

my ($sec,$min,$hour,$mday,$mon,$year,$wday,$yday,$isdst) = localtime();

if($mday - $datetime[0] > 1 or ($datetime[0] > $mday and $mday != 1))
{
  die "Date wrong\n";
}

my $mins;
my @timearr = split(/:/, $datetime[1]);
$mins = $timearr[0] * 60 + $timearr [1];
if($timearr[1] =~ /pm/)
{
  $mins += 720;
}
if($mday != $datetime[0])
{
  $mins += 1440;
}

if($mins + 60 < $hour * 60 + $min)
{
  die "page outdated\n";
}

my %temp_hash;
foreach ( @ARGV[1..$#ARGV] )
{
  my @tmparr = split(/:/, $_);
  $temp_hash{$tmparr[0]} = $tmparr[1];
}
my @temp_list = sort(keys(%temp_hash));
my $percent = 0;
my $i;
for($i = $#temp_list; $i >= 0 and $temp_list[$i] > $cur_temp; $i--)
{
  $percent = $temp_hash{$temp_list[$i]}
}

my $prefs = "/etc/boinc-client/global_prefs_override.xml";
open(IN, "<$prefs") or die "Can't read $prefs";
my @prefs_contents;
while(<IN>)
{
  push(@prefs_contents, $_);
}
close(IN);

openlog("boincmgr-cron", "", "daemon");

my @cpus_pct = grep(/max_ncpus_pct/, @prefs_contents);
my $cpus_line = $cpus_pct[0];
$cpus_line =~ s/..max_ncpus_pct.$//;
$cpus_line =~ s/^.*max_ncpus_pct.//;
if($cpus_line == $percent)
{
  syslog("info", "Temp $cur_temp" . "C, already set to $percent");
  exit 0;
}
open(OUT, ">$prefs.new") or die "Can't read $prefs.new";
for($i = 0; $i <= $#prefs_contents; $i++)
{
  if($prefs_contents[$i] =~ /max_ncpus_pct/)
  {
    print OUT "   <max_ncpus_pct>$percent.000000</max_ncpus_pct>\n";
  }
  else
  {
    print OUT $prefs_contents[$i];
  }
}
close(OUT);
rename "$prefs.new", "$prefs" or die "can't rename";
system("boinccmd --read_global_prefs_override");
syslog("info", "Temp $cur_temp" . "C, set percentage to $percent");

05 January, 2021 11:28PM by etbe

hackergotchi for Ben Hutchings

Ben Hutchings

Debian LTS work, December 2020

I was assigned 16 hours of work by Freexian's Debian LTS initiative and carried over 9 hours from earlier months. I worked 16.5 hours this month, so I will carry over 8.5 hours to January. (Updated: corrected number of hours worked.)

I updated linux-4.19 to include the changes in the Debian 10.7 point release, uploaded the package, and issued DLA-2483-1 for this.

I picked some regression fixes from the Linux 4.9 stable branch to the linux package, and uploaded the package. This unfortunately failed to build on arm64 due to some upstream changes uncovering an old bug, so I made a second upload fixing that. I issued DLA-2494-1 for this.

I updated the linux packaging branch for stretch to Linux 4.9.249, but haven't made another package upload yet.

05 January, 2021 10:32PM

hackergotchi for Bernd Zeimetz

Bernd Zeimetz

Building reverse build dependencies in salsa CI

For the next library soname bump of gpsd I needed to rebuild all reverse dependencies. As this is a task I have to do very often, I came up with some code to generate (and keep uptodate) an include for the gitlab CI. Right now it is rather uncommented, undocumented, but works well. If you like it, MRs are very welcome.

https://salsa.debian.org/bzed/reverse-dependency-ci/

The generated files are here:

https://bzed.pages.debian.net/reverse-dependency-ci/

Usage:


include:
 - https://salsa.debian.org/salsa-ci-team/pipeline/raw/master/salsa-ci.yml
 - https://salsa.debian.org/salsa-ci-team/pipeline/raw/master/pipeline-jobs.yml
 - https://bzed.pages.debian.net/reverse-dependency-ci/gpsd.yml

variables:
 SALSA_CI_ENABLE_REVERSE_DEPENDENCY_BUILD: 1

Please do no abuse the salsa CI. Don’t build all of your 100 reverse dependencies with every commit!

05 January, 2021 03:33PM

hackergotchi for Steve Kemp

Steve Kemp

Brexit has come

Nothing too much has happened recently, largely as a result of the pandemic killing a lot of daily interests and habits.

However as a result of Brexit I'm having to do some paperwork, apparently I now need to register for permanent residency under the terms of the withdrawal agreement, and that will supersede the permanent residency I previously obtained.

Of course as a UK citizen I've now lost the previously-available freedom of movement. I can continue to reside here in Helsinki, Finland, indefinitely, but I cannot now move to any other random EU country.

It has crossed my mind, more than a few times, that I should attempt to achieve Finnish citizenship. As a legal resident of Finland the process is pretty simple, I just need two things:

  • Prove I've lived here for the requisite number of years.
  • Pass a language test.

Of course the latter requirement is hard, I can understand a lot of spoken and written Finnish, but writing myself, and speaking a lot is currently beyond me. I need to sit down and make the required effort to increase my fluency. There is the alternative option of learning Swedish, which is a hack a lot of immigrants use:

  • Learning Swedish is significantly easier for a native English-speaker.
  • But the downside is that it would be learning a language solely to "cheat" the test, it wouldn't actually be useful in my daily life.

Finland has two official languages, and so the banks, the medical world, the tax-office, etc, are obliged to provide service in both. However daily life, ordering food at restaurants, talking to parents in the local neighborhood? Finnish, or English are the only real options. So if I went this route I'd end up in a weird situation where I had to learn a language to pass a test, but then would continue to need to learn more Finnish to live my life. That seems crazy, unless I were desperate for a second citizenship which I don't think I am.

Learning Finnish has not yet been a priority, largely because I work in English in the IT-world, and of course when I first moved here I was working (remotely) for a UK company, and didn't have the time to attend lessons (because they were scheduled during daytime, on the basis that many immigrants are unemployed). Later we had a child, which meant that early-evening classes weren't a realistic option either.

(Of course I learned a lot of the obvious things immediately upon moving, things like numbers, names for food, days of the week were essential. Without those I couldn't have bought stuff in shops and would have starved!)

On the topic of languages a lot of people talk about how easy it is for children to pick up new languages, and while that is broadly true it is also worth remembering just how many years of correction and repetition they have to endure as part of the process.

For example we have a child, as noted already, he is spoken to by everybody in Finnish. I speak to him in English, and he hears his mother and myself speaking English. But basically he's 100% Finnish with the exception of:

  • Me, speaking English to him.
  • His mother and I speaking English in his hearing.
  • Watching Paw Patrol.

If he speaks Finnish to me I pretend to not understand him, even when I do, just for consistency. As a result of that I've heard him tell strangers "Daddy doesn't speak Finnish" (in Finnish) when we've been stopped and asked for directions. He also translates what some other children have said into English for my benefit which is adorable

Anyway he's four, and he's pretty amazing at speaking to everybody in the correct language - he's outgrown the phase where he'd mix different languages in the same sentence ("more leipä", "saisinko milk") - when I took him to the UK he surprised and impressed me by being able to understand a lot of the heavy/thick accents he'd never heard before. (I'll still need to train him on Rab C. Nesbitt when he's a wee bit older, but so far no worries.)

So children learn languages, easily and happily? Yes and no. I've spent nearly two years correcting his English and he still makes the same mistake with gender. It's not a big deal, at all, but it's a reminder that while children learn this stuff, they still don't do it as easily as people imagine. I'm trying to learn and if I'd been corrected for two years over the same basic point you'd rightly think I was "slow", but actually that's just how it works. Learning languages requires a hell of a lot of practice, a lot of effort, and a lot of feedback/corrections.

Specifically Finnish doesn't have gendered pronouns, the same word is used for "he" and "she". This leads to a lot of Finnish people, adults and children, getting the pronouns wrong in English. In the case of our child he'll say "Mommy is sleeping, when he wake up?" In the case of adults I've heard people say "My girlfriend is a doctor, he works in a hospital", or "My dad is an accountant, she works for a big firm". As I say I've spent around two years making this correction to the child, and he's still nowhere near getting it right. Kinda adorable actually:

  • "Mommy is a woman we say "when she wakes up"..."
  • "Adriana is a girl we say "her bike".."

05 January, 2021 08:45AM

Russ Allbery

New year haul

For once, I've already read and reviewed quite a few of these books.

Elizabeth Bear — Machine (sff)
Timothy Caulfield — Your Day, Your Way (non-fiction)
S.A. Chakraborty — The City of Brass (sff)
John Dickerson — The Hardest Job in the World (non-fiction)
Tracy Deonn — Legendborn (sff)
Lindsay Ellis — Axiom's End (sff)
Alix E. Harrow — The Once and Future Witches (sff)
TJ Klune — The House in the Cerulean Sea (sff)
Maria Konnikova — The Biggest Bluff (non-fiction)
Talia Levin — Culture Warlords (non-fiction)
Yoon Ha Lee — Phoenix Extravagent (sff)
Yoon Ha Lee, et al. — The Vela (sff)
Michael Lewis — Flash Boys (non-fiction)
Michael Lewis — Losers (non-fiction)
Michael Lewis — The Undoing Project (non-fiction)
Megan Lindholm — Wizard of the Pigeons (sff)
Nathan Lowell — Quarter Share (sff)
Adrienne Martini — Somebody's Gotta Do It (non-fiction)
Tamsyn Muir — Princess Florinda and the Forty-Flight Tower (sff)
Naomi Novik — A Deadly Education (sff)
Margaret Owen — The Merciful Crow (sff)
Anne Helen Peterson — Can't Even (non-fiction)
Devon Price — Laziness Does Not Exist (non-fiction)
The Secret Barrister — The Secret Barrister (non-fiction)
Studs Terkel — Working (non-fiction)
Kathi Weeks — The Problem with Work (non-fiction)
Reeves Wiedeman — Billion Dollar Loser (non-fiction)

Rather a lot of non-fiction in this batch, much more than usual. I've been in a non-fiction mood lately.

So many good things to read!

05 January, 2021 06:03AM

January 04, 2021

Iustin Pop

Year 2020 review

Year 2020. What a year! Sure, already around early January there were rumours/noise about Covid-19, but who would have thought where it will end up! Thankfully, none of my close or extended family was directly (medically) affected by Covid, so I/we had a privileged year compared to so many other people.

I thought how to write a mini-summary, but prose is too difficult, so let’s just go month-by-month. Please note that my memory is fuzzy after 9 months cooked up in the apartment, so things could ±1 month compared to what I wrote.

Timeline

January

Ski weekend. Skiing is awesome! Cancelling a US work trip since there will be more opportunities soon (har har!).

February

Ski vacation. Yep, skiing is awesome. Can’t wait for next season (har har!). Discussions about Covid start in the office, but more “is this scary or just interesting?” (yes, this was before casualties). Then things start escalating, work-from-home at least partially, etc. etc. Definitely not just “intersting” anymore.

In Garmin-speak, I got ~700+ “intensity minutes” in February (correlates with activity time, but depends on intensity of the effort whether 1:1 or 2 intensity minutes for one wall-clock minute).

March

Sometimes during the month, my workplace introduces mandatory WFH. I remember being the last person in our team in the office, on the last day we were allowed to work, and cleaning my desk/etc., thinking “all this, and we’ll be back in 3 weeks or so”. Har har!

I buy a webcam, just in case WFH gets extended. And start to increase my sports - getting double the intensity minutes (1500+).

April

Switzerland enters the first, hard, lockdown. Or was it late March? Not entirely sure, but in my mind March was the opening, and April was the first main course.

It is challenging, having to juggle family and work and stressed schedule, but also interesting. Looking back, I think I liked April the most, as people were actually careful at that time.

I continue upgrading my “home office” - new sound system, so that I don’t have to plug in/plug out cables.

1700+ intensity minutes this month.

May

Continued WFH, somewhat routine now. My then internet provider started sucking hard, so I upgrade with good results. I’m still happy, half a year later (quite happy, even).

Still going strong otherwise, but waiting for summer vacation, whatever it will be. A tiny bit more effort, so 1800 intensity minutes in May.

June

Switzerland relaxes the lock down, but not my company, so as the rest of the family goes out and about, I start feeling alone in the apartment. And somewhat angry at it, which impacts my sports (counter-intuitively), so I only get 1500 intensity minutes.

I go and buy a coffee machine—a real one, that takes beans and grinds them, so I get to enjoy the smell of freshly-ground coffee and the fun of learning about coffee beans, etc. But it occupies the time.

On the work/job front, I think at this time I finally got a workstation for home, instead of a laptop (which was ultra-portable too), so together with the coffee machine, it feels like a normal work environment. Well, modulo all the people. At least I’m not crying anymore every time I open a new tab in Chrome…

July

Situation is slowly going better, but no, not my company. Still mandatory WFH, with (if I recall correctly) one day per week allowed, and no meeting other people. I get angrier, but manage to channel my energy into sports, almost doubly my efforts in July - 2937 intensity minutes, not quite reaching the 3000 magic number.

I buy more stuff to clean and take care of my bicycles, which I don’t really use. So shopping therapy too.

August

The month starts with a one week family vacation, but I take a bike too, so I manage to put in some effort (it was quite nice riding TBH). A bit of changes in the personal life (nothing unexpected), which complicates things a bit, but at this moment I really thought Switzerland is going to continue to decrease in infections/R-factor/etc. so things will get back to normal, right? My company expands a bit the work-from-office part, so I’m optimistic.

Sports wise, still going strong, 2500 intensity minutes, preparing for the single race this year.

September

The personal life changes from August start to stabilise, so things become again routine, and I finally get to do a race. Life was good for an extended weekend (well, modulo race angst, but that’s part of the fun), and I feel justified to take it slow the week after the race. And the week after that too.

I end up the month with close, but not quite, 1900 intensity minutes.

October

October starts with school holidays and a one week family vacation, but I feel demotivated. Everything is closing down again (well, modulo schools), and I actually have difficulty getting re-adjusted to no longer being alone in the apartment during the work hours.

I only get ~1000 intensity minutes in October, mainly thanks to good late autumn weather and outside rides. And I start playing way more computer games. I also sell my PS4, hoping to get a PS5 next month.

November

November continues to suck. I think my vacation in October was actually detrimental - it broke my rhythm, I don’t really do sport anymore, not consistently at least, so I only get 700+ intensity minutes. And I keep playing computer games, even if I missed the PS5 ordering window; so I switch to PC gaming.

My home office feels very crowded, so as kind of anti-shopping therapy, I sell tons of smallish stuff; can’t believe how much crap I kept around while not really using it.

I also manage to update/refresh all my Debian packages, since next freeze approaches. Better than for previous releases, so it feels good.

December

December comes, end of the year, the much awaited vacation - which we decide to cancel due to the situation in whole of Switzerland (and neighbouring countries). I basically only play computer games, and get grand total of 345 activity minutes this month.

And since my weight is inversely correlated to my training, I’m basically back at my February weight, having lost all the gains I made during the year. I mean, having gained back all the fat I lost. Err, you know what I mean; I’m back close to my high-watermark, which is not good.

Conclusion

I was somehow hoping that the end of the year will allow me to reset and restart, but somehow - a few days into January - it doesn’t really feel so. My sleep schedule is totally ruined, my motivation is so-so, and I think the way I crashed in October was much harder/worse than I realised at the time, but in a way expected for this crazy year.

I have some projects for 2021 - or at least, I’m trying to make up a project list - in order to get a bit more structure in my continued “stuck inside the house” part, which is especially terrible when on-call. I don’t know how the next 3-6 months will evolve, but I’m thankful that so far, we are all healthy. Actually, me personally I’ve been healthier physically than in other years, due to less contact with other people.

On the other side, thinking of all the health-care workers, or even service workers, my IT job is comfy and all I am is a spoiled person (I could write many posts on specifically this topic). I really need to up my willpower and lower my spoil level. Hints are welcome :(

Wish everybody has a better year in 2021.

04 January, 2021 10:36PM

hackergotchi for Jan Wagner

Jan Wagner

Backing up Windows (the hard way)

Sometimes you need to do things you don't like and you don't know where you will end up.
In our household there exists one (production) system running Windows. Don't ask why and please no recommandations how to substitute it. Some things are hard to (ex)change, for example your love partner.

Looking into Backup with rsync on Windows (WSL) I needed to start a privileged powershell, so I first started an unprivileged one:

powershell

Just to start a privileged:

Start-Process powershell -Verb runAs

Now you can follow the Instructions from Microsoft to install OpenSSH. Or just install the OpenSSH Server:

Add-WindowsCapability -Online -Name OpenSSH.Server~~~~0.0.1.0

Check if a firewall rule was created (maybe you want to adjust it):

Get-NetFirewallRule -Name *ssh*

Start the OpenSSH server:

Start-Service sshd

Running OpenSSH server as service:

Set-Service -Name sshd -StartupType 'Automatic'

You can create the .ssh directory with the correct permissions by connecting to localhost and creating the known_hosts file.

ssh user@127.0.0.1

When you intend to use public key authentication for users in the administrators group, have a look into How to Login Windows Using SSH Key Under Local Admin.

Indeed you can get rsync running via WSL. But why load tons of dependencies on your system? With the installation of rsync I cheated a bit and used chocolately by running choco install rsync, but there is also an issue requesting rsync support for the OpenSSH server which includes an archive with a rsync.exe and libraries which may also fit. You can place those files for example into C:\Windows\System32\OpenSSH so they are in the PATH.

So here we are. Now I can solve all my other issues with BackupPC, Windows firewall and the network challenges to get access to the isolated dedicated network of the windows system.

04 January, 2021 05:35PM by Jan Wagner

John Goerzen

More Topics on Store-And-Forward (Possibly Airgapped) ZFS and Non-ZFS Backups with NNCP

Note: this is another article in my series on asynchronous communication in Linux with UUCP and NNCP.

In my previous post, I introduced a way to use ZFS backups over NNCP. In this post, I’ll expand on that and also explore non-ZFS backups.

Use of nncp-file instead of nncp-exec

The previous example used nncp-exec (like UUCP’s uux), which lets you pipe stdin in, then queues up a request to run a given command with that input on a remote. I discussed that NNCP doesn’t guarantee order of execution, but that for the ZFS use case, that was fine since zfs receive would just fail (causing NNCP to try again later).

At present, nncp-exec stores the data piped to it in RAM before generating the outbound packet (the author plans to fix this shortly). That made it unusable for some of my backups, so I set it up another way: with nncp-file, the tool to transfer files to a remote machine. A cron job then picks them up and processes them.

On the machine being backed up, we have to find a way to encode the dataset to be received. I chose to do that as part of the filename, so the updated simplesnap-queue could look like this:

#!/bin/bash

set -e
set -o pipefail

DEST="`echo $1 | sed 's,^tank/simplesnap/,,'`"
FILE="bakfsfmt2-`date "+%s.%N".$$`_`echo "$DEST" | sed 's,/,@,g'`"

echo "Processing $DEST to $FILE" >&2
# stdin piped to this
zstd -8 - \
  | gpg --compress-algo none --cipher-algo AES256 -e -r 012345...  \
  | su nncp -c "/usr/local/nncp/bin/nncp-file -nice B -noprogress - 'backupsvr:$FILE'" >&2

echo "Queued $DEST to $FILE" >&2

I’ve added compression and encryption here as well; more on that below.

On the backup server, we would define a different incoming directory for each node in nncp.hjson. For instance:

host1: {
...
   incoming: "/var/local/nncp-bakcups-incoming/host1"
}

host2: {
...
   incoming: "/var/local/nncp-backups-incoming/host2"
}

I’ll present the scanning script in a bit.

Offsite Backup Rotation

Most of the time, you don’t want just a single drive to store the backups. You’d like to have a set. At minimum, one wouldn’t be plugged in so lightning wouldn’t ruin all your backups. But maybe you’d store a second drive at some other location you have access to (friend’s house, bank box, etc.)

There are several ways you could solve this:

  • If the remote machine is at a location with network access and you trust its physical security (remember that although it will store data encrypted at rest and will transport it encrypted, it will — in most cases — handle un-encrypted data during processing), you could of course send NNCP packets to it over the network at the same time you send them to your local backup system.
  • Alternatively, if the remote location doesn’t have network access or you want to keep it airgapped, you could transport the NNCP packets by USB drive to the remote end.
  • Or, if you don’t want to have any kind of processing capability remotely — probably a wise move — you could rotate the hard drives themselves, keeping one plugged in locally and unplugging the other to take it offsite.

The third option can be helped with NNCP, too. One way is to create separate NNCP installations for each of the drives that you store data on. Then, whenever one is plugged in, the appropriate NNCP config will be loaded and appropriate packets received and processed. The neighbor machine — the spooler — would just store up packets for the offsite drive until it comes back onsite (or, perhaps, your airgapped USB transport would do this). Then when it’s back onsite, all the queued up ZFS sends get replayed and the backups replicated.

Now, how might you handle this with NNCP?

The simple way would be to have each system generating backups send them to two destinations. For instance:

zstd -8 - | gpg --compress-algo none --cipher-algo AES256 -e -r 07D5794CD900FAF1D30B03AC3D13151E5039C9D5 \
  | tee >(su nncp -c "/usr/local/nncp/bin/nncp-file -nice B+5 -noprogress - 'backupdisk1:$FILE'") \
        >(su nncp -c "/usr/local/nncp/bin/nncp-file -nice B+5 -noprogress - 'backupdisk2:$FILE'") \
   > /dev/null

You could probably also more safely use pee(1) (from moreutils) to do this.

This has an unfortunate result of doubling the network traffic from every machine being backed up. So an alternative option would be to queue the packets to the spooling machine, and run a distribution script from it; something like this, in part:

INCOMINGDIR="/var/local/nncp-bakfs-incoming"
LOCKFILE="$INCOMINGDIR/.lock"
printf -v EVAL_SAFE_LOCKFILE '%q' "$LOCKFILE"
if dotlockfile -r 0 -l -p "${LOCKFILE}"; then
  logit "Lock obtained at ${LOCKFILE} with dotlockfile"
  trap 'ECODE=$?; dotlockfile -u '"${EVAL_SAFE_LOCKFILE}"'; exit $ECODE' EXIT INT TERM
else
  logit "Could not obtain lock at $LOCKFILE; $0 likely already running."
  exit 0
fi


logit "Scanning queue directory..."
cd "$INCOMINGDIR"
for HOST in *; do
   cd "$INCOMINGDIR/$HOST"
   for FILE in bakfsfmt2-*; do
           if [ -f "$FILE" ]; then
                   for BAKFS in backupdisk1 backupdisk2; do
                           runcommand nncp-file -nice B+5 -noprogress "$FILE" "$BAKFS:$HOST/$FILE"
                   done
                   runcommand rm "$FILE"
           else
                   logit "$HOST: Skipping $FILE since it doesn't exist"
           fi
   done

done
logit "Scan complete."

Security Considerations

You’ll notice that in my example above, the encryption happens as the root user, but nncp is called under su. This means that even if there is a vulnerability in NNCP, the data would still be protected by GPG. I’ll also note here that many sites run ssh as root unnecessarily; the same principles should apply there. (ssh has had vulnerabilities in the past as well). I could have used gpg’s built-in compression, but zstd is faster and better, so we can get good performance by using fast compression and piping that to an algorithm that can use hardware acceleration for encryption.

I strongly encourage considering transport, whether ssh or NNCP or UUCP, to be untrusted. Don’t run it as root if you can avoid it. In my example, the nncp user, which all NNCP commands are run as, has no access to the backup data at all. So even if NNCP were compromised, my backup data wouldn’t be. For even more security, I could also sign the backup stream with gpg and validate that on the receiving end.

I should note, however, that this conversation assumes that a network- or USB-facing ssh or NNCP is more likely to have an exploitable vulnerability than is gpg (which here is just processing a stream). This is probably a safe assumption in general. If you believe gpg is more likely to have an exploitable vulnerability than ssh or NNCP, then obviously you wouldn’t take this particular approach.

On the zfs side, the use of -F with zfs receive is avoided; this could lead to a compromised backed-up machine generating a malicious rollback on the destination. Backup zpools should be imported with -R or -N to ensure that a malicious mountpoint property couldn’t be used to cause an attack. I choose to use “zfs receive -u -o readonly=on” which is compatible with both unmounted backup datasets and zpools imported with -R (or both). To access the data in a backup dataset, you would normally clone it and access it there.

The processing script

So, put this all together and look at an example of a processing script that would run from cron as root and process the incoming ZFS data.

#!/bin/bash
set -e
set -o pipefail

# Log a message
logit () {
   logger -p info -t "`basename "$0"`[$$]" "$1"
}

# Log an error message
logerror () {
   logger -p err -t "`basename "$0"`[$$]" "$1"
}

# Log stdin with the given code.  Used normally to log stderr.
logstdin () {
   logger -p info -t "`basename "$0"`[$$/$1]"
}

# Run command, logging stderr and exit code
runcommand () {
   logit "Running $*"
   if "$@" 2> >(logstdin "$1") ; then
      logit "$1 exited successfully"
      return 0
   else
       RETVAL="$?"
       logerror "$1 exited with error $RETVAL"
       return "$RETVAL"
   fi
}

STORE=backups/simplesnap
INCOMINGDIR=/backups/nncp/incoming

if ! [ -d "$INCOMINGDIR" ]; then
        logerror "$INCOMINGDIR doesn't exist"
        exit 0
fi

LOCKFILE="/backups/nncp/.nncp-backups-zfs-scan.lock"
printf -v EVAL_SAFE_LOCKFILE '%q' "$LOCKFILE"
if dotlockfile -r 0 -l -p "${LOCKFILE}"; then
  logit "Lock obtained at ${LOCKFILE} with dotlockfile"
  trap 'ECODE=$?; dotlockfile -u '"${EVAL_SAFE_LOCKFILE}"'; exit $ECODE' EXIT INT TERM
else
  logit "Could not obtain lock at $LOCKFILE; $0 likely already running."
  exit 0
fi

EXITCODE=0


cd "$INCOMINGDIR"
logit "Scanning queue directory..."
for HOST in *; do
    HOSTPATH="$INCOMINGDIR/$HOST"
    # files like backupsfmt2-134.13134_dest
    for FILE in "$HOSTPATH"/backupsfmt2-[0-9]*_?*; do
        if [ ! -f "$FILE" ]; then
            logit "Skipping non-existent $FILE"
            continue
        fi

        # Now, $DEST will be HOST/DEST.  Strip off the @ also.
        DEST="`echo "$FILE" | sed -e 's/^.*backupsfmt2[^_]*_//' -e 's,@,/,g'`"

        if [ -z "$DEST" ]; then
            logerror "Malformed dest in $FILE"
            continue
        fi
        HOST2="`echo "$DEST" | sed 's,/.*,,g'`"
        if [ -z "$HOST2" ]; then
            logerror "Malformed DEST $DEST in $FILE"
            continue
        fi

        if [ ! "$HOST" = "$HOST2" ]; then
            logerror "$DIR: $HOST doesn't match $HOST2"
            continue
        fi

        logit "Processing $FILE to $STORE/$DEST"
            if runcommand gpg -q -d < "$FILE" | runcommand zstdcat | runcommand zfs receive -u -o readonly=on "$STORE/$DEST"; then
                logit "Successfully processed $FILE to $STORE/$DEST"
                runcommand rm "$FILE"
        else
                logerror "FAILED to process $FILE to $STORE/$DEST"
                EXITCODE=15
        fi

Applying These Ideas to Non-ZFS Backups

ZFS backups made our job easier in a lot of ways:

  • ZFS can calculate a diff based on an efficiently-stored previous local state (snapshot or bookmark), rather than a comparison to a remote state (rsync)
  • ZFS "incremental" sends, while less efficient than rsync, are reasonably efficient, sending only changed blocks
  • ZFS receive detects and enforces that the incremental source on the local machine must match the incremental source of the original stream, enforcing ordering
  • Datasets using ZFS encryption can be sent in their encrypted state
  • Incrementals can be done without a full scan of the filesystem

Some of these benefits you just won't get without ZFS (or something similar like btrfs), but let's see how we could apply these ideas to non-ZFS backups. I will explore the implementation of them in a future post.

When I say "non ZFS", I am being a bit vague as to whether the source, the destination, or both systems are running a non-ZFS filesystem. In general I'll assume that neither are ZFS.

The first and most obvious answer is to just tar up the whole system and send that every day. This is, of course, only suitable for small datasets on a fast network. These tarballs could be unpacked on the destination and stored more efficiently via any number of methods (hardlink trees, a block-level deduplicator like borg or rdedup, or even just simply compressed tarballs).

To make the network trip more efficient, something like rdiff or xdelta could be used. A signature file could be stored on the machine being backed up (generated via tee/pee at stream time), and the next run could simply send an rdiff delta over NNCP. This would be quite network-efficient, but still would require reading every byte of every file on every backup, and would also require quite a bit of temporary space on the receiving end (to apply the delta to the previous tarball and generate a new one).

Alternatively, a program that generates incremental backup files such as rdup could be used. These could be transmitted over NNCP to the backup server, and unpacked there. While perhaps less efficient on the network -- every file with at least one modified byte would be retransmitted in its entirety -- it avoids the need to read every byte of unmodified files or to have enormous temporary space. I should note here that GNU tar claims to have an incremental mode, but it has a potential data loss bug.

There are also some tools with algorithms that may apply well in this use care: syrep and fssync being the two most prominent examples, though rdedup (mentioned above) and the nascent asuran project may also be combinable with other tools to achieve this effect.

I should, of course, conclude this section by mentioning btrfs. Every time I've tried it, I've run into serious bugs, and its status page indicates that only some of them have been resolved. I would not consider using it for something as important as backups. However, if you are comfortable with it, it is likely to be able to run in more constrained environments than ZFS and could probably be processed in much the same way as zfs streams.

04 January, 2021 05:18PM by John Goerzen

Russ Allbery

Review: The Once and Future Witches

Review: The Once and Future Witches, by Alix E. Harrow

Publisher: Redhook Books
Copyright: October 2020
ISBN: 0-316-42202-9
Format: Kindle
Pages: 515

Once upon a time there were three sisters.

They were born in a forgotten kingdom that smelled of honeysuckle and mud, where the Big Sandy ran wide and the sycamores shone white as knuckle-bones on the banks. The sisters had no mother and a no-good father, but they had each other; it might have been enough.

But the sisters were banished from their kingdom, broken and scattered.

The Once and Future Witches opens with Juniper, the youngest, arriving in the city of New Salem. The year is 1893, but not in our world, not quite; Juniper has witch-ways in her pocket and a few words of power. That's lucky for her because the wanted posters arrived before she did.

Unbeknownst to her or to each other, her sisters, Agnes and Bella, are already in New Salem. Agnes works in a cotton mill after having her heart broken one too many times; the mill is safer because you can't love a cotton mill. Bella is a junior librarian, meek and nervous and uncertain but still fascinated by witch-tales and magic. It's Bella who casts the spell, partly by accident, partly out of wild hope, but it was Juniper arriving in the city who provided the final component that made it almost work. Not quite, not completely, but briefly the lost tower of Avalon appears in St. George's Square. And, more importantly, the three sisters are reunited.

The world of the Eastwood sisters has magic, but the people in charge of that world aren't happy about it. Magic is a female thing, contrary to science and, more importantly, God. History has followed a similar course to our world in part because magic has been ruthlessly suppressed. Inquisitors are a recent memory and the cemetery has a witch-yard, where witches are buried unnamed and their ashes sown with salt. The city of New Salem is called New Salem because Old Salem, that stronghold of witchcraft, was burned to the ground and left abandoned, fit only for tourists to gawk at the supposedly haunted ruins. The women's suffrage movement is very careful to separate itself from any hint of witchcraft or scandal, making its appeals solely within the acceptable bounds of the church.

Juniper is the one who starts to up-end all of that in New Salem. Juniper was never good at doing what she was told.

This is an angry book that feels like something out of another era, closer in tone to a Sheri S. Tepper or Joanna Russ novel than the way feminism is handled in recent work. Some of that is the era of the setting, before women even had the right to vote. But primarily it's because Harrow, like those earlier works, is entirely uninterested in making excuses or apologies for male behavior. She takes an already-heated societal conflict and gives the underdogs magic, which turns it into a war. There is likely a better direct analogy from the suffrage movement, but the comparison that came to my mind was if Martin Luther King, Jr. proved ineffective or had not existed, and instead Malcolm X or the Black Panthers became the face of the Civil Rights movement.

It's also an emotionally exhausting book. The protagonists are hurt and lost and shattered. Their moments of victory are viciously destroyed. There is torture and a lot of despair. It works thematically; all the external solutions and mythical saviors fail, but in the process the sisters build their own strength and their own community and rescue themselves. But it's hard reading at times if you're emotionally invested in the characters (and I was very invested). Harrow does try to balance the losses with triumphs and that becomes more effective and easier to read in the back half of the book, but I struggled with the grimness at the start.

One particular problem for me was that the sisters start the book suspicious and distrustful of each other because of lies and misunderstandings. This is obvious to the reader, but they don't work through it until halfway through the book. I can't argue with this as a piece of characterization — it made sense to me that they would have reacted to their past the way that they did. But it was still immensely frustrating to read, since in the meantime awful things were happening and I wanted them to band together to fight. They also worry over the moral implications of the fate of their father, whereas I thought the only problem was that the man couldn't die more than once. There too, it makes sense given the moral framework the sisters were coerced into, but it is not my moral framework and it was infuriating to see them stay trapped in it for so long.

The other thing that I found troubling thematically is that Harrow personalizes evil. I thought the more interesting moral challenge posed in this book is a society that systematically abuses women and suppresses their power, but Harrow gradually supplants that systemic conflict with a villain who has an identity and a backstory. It provides a more straightforward and satisfying climax, and she does avoid the trap of letting triumph over one character solve all the broader social problems, but it still felt too easy. Worse, the motives of the villain turn out to be at right angles to the structure of the social oppression. It's just a tool he's using, and while that's also believable, it means the transfer of the narrative conflict from the societal to the personal feels like a shying away from a sharper political point. Harrow lets the inhabitants of New Salem off too easily by giving them the excuse of being manipulated by an evil mastermind.

What I thought Harrow did handle well was race, and it feels rare to be able to say this about a book written by and about white women. There are black women in New Salem as well, and they have their own ways and their own fight. They are suspicious of the Eastwood sisters because they're worried white women will stir up trouble and then run away and leave the consequences to fall on black women... and they're right. An alliance only forms once the white women show willingness to stay for the hard parts. Black women are essential to the eventual success of the protagonists, but the opposite is not necessarily true; they have their own networks, power, and protections, and would have survived no matter what the Eastwoods did. The book is the Eastwoods' story, so it's mostly concerned with white society, but I thought Harrow avoided both making black women too magical or making white women too central. They instead operate in parallel worlds that can form the occasional alliance of mutual understanding.

It helps that Cleopatra Quinn is one of the best characters of the book.

This was hard, emotional reading. It's the sort of book where everything has a price, even the ending. But I'm very glad I read it. Each of the three sisters gets their own, very different character arc, and all three of those arcs are wonderful. Even Agnes, who was the hardest character for me to like at the start of the book and who I think has the trickiest story to tell, becomes so much stronger and more vivid by the end of the book. Sometimes the descriptions are trying a bit too hard and sometimes the writing is not quite up to the intended goal, but some of the descriptions are beautiful and memorable, and Harrow's way of weaving the mythic and the personal together worked for me.

This is a more ambitious book than The Ten Thousand Doors of January, and while I think the ambition exceeded Harrow's grasp in a few places and she took a few thematic short-cuts, most of it works. The characters felt like living and changing people, which is not easy given how heavily the story structure leans on maiden, mother, and crone archetypes. It's an uncompromising and furious book that turns the anger of 1970s feminist SF onto themes that are very relevant in 2021. You will have to brace yourself for heartbreak and loss, but I think it's fantasy worth reading. Recommended.

Rating: 8 out of 10

04 January, 2021 02:49AM

January 03, 2021

Enrico Zini

COVID-19 vaccines

COVID-19 vaccination has started, and this site tracks progress in Italy. This site, world-wide.

Reverse Engineering the source code of the BioNTech/Pfizer SARS-CoV-2 Vaccine has a pretty good description of the BioNTech/Pfizer SARS-CoV-2 Vaccine, codon by codon, broken down in a way that I managed to follow.

From the same author, DNA seen through the eyes of a coder

03 January, 2021 11:00PM

hackergotchi for Emmanuel Kasper

Emmanuel Kasper

How to move a single VM between cloud providers

I am running since a decade a small Debian VM, that I use for basic web and mail hosting. Since most of the VM setup is done manually and not following the Infrastructure As Code pattern, it is faster to simply copy the filesystem when switching providers instead of reconfiguring everything.
The steps involved are:

1. create a backup of the filesystem using tar of rsync, excluding dynamic content
rsync  --archive \
    --one-file-system --numeric-ids \
    --rsh "ssh -i private_key root@server:/ /local_dir

or
tar -cvpzf backup.tar.gz \
--numeric-owner \
--exclude=/backup.tar.gz \
--one-file-system /


Notice here the --one-file-system switch which avoids back'ing up the content of mount points like /proc, /dev.
If you have extra partitions with a mounted filesystem, like /boot or home you need do add a separate backup for those.

2. create a new VM on the new cloud provider, verify you have a working console access, and power it off.
3. boot on the new cloud provider a rescue image
4. partition the disk image on the new provider.
5. mount the new root partition, and untar your backup on it. You could for instance push the local backup via rsync, or download the tar archive using https.
6. update network configuration and /etc/fstab
7. chroot into the target system, and reinstall grub

This works surprisingly well, and you if made your backup locally, you can test the whole procedure by building a test VM with your backup. Just replace the deboostrap step with a command like tar -xvpzf /path/to/backup.tar.gz -C /mount_point --numeric-owner

Using this procedure, I moved from Hetzner (link in French language) to Digital Ocean, from Digital Ocean to Vultr, and now back at Hetzner.

03 January, 2021 12:58PM by Emmanuel Kasper (noreply@blogger.com)

January 02, 2021

Jonathan Wiltshire

RCBW 21.1

Does software-properties-common really depend on gnupg, as described in #970124, or could it be python3-software-properties? Should it be Depends, or Recommends? And do you accept the challenge of finding out and preparing a patch (and even an upload) to fix the bug?

02 January, 2021 04:36PM by Jonathan

hackergotchi for Martin-&#201;ric Racine

Martin-Éric Racine

Help needed: clean up and submit KMS driver for Geode LX to LKML

Ever since X.org switched to rootless operation, the days of the Geode X.org driver have been numbered. The old codebase dates back from Geode's early days at Cyrix, was then updated by NSC to add support for their new GX2 architecture, from which AMD dropped GX1 support and added support for their new LX architecture. To put it mildly, that codebase is a serious mess.

However, at least the LX code comes with plenty of niceties, such as being able to detect when it runs on an OLPC XO-1 and to probe DCC pins to determine the optimal display resolution. This still doesn't make the codebase cruft-free.

Anyhow, most Linux distributions have dropped support for anything older than i686 with PAE, which essentially means that the GX2 code is just for show. Debian is one of very few distributions whose x86-32 port still ships with i686 without PAE. In fact, the lowest common denominator kernel on i386 is configured for Geode (LX).

A while back, someone had started working on a KMS driver for the Geode LX. Through word of mouth, I got my hands on a copy of their Git tree. The driver worked reasonably well, but the codebase needs some polishing before it could be included in the Linux kernel tree.

Hence this call for help:

Is there anyone with good experience of the LKML coding standards who would be willing to clean up the driver's code and submit the patch to the LKML?

02 January, 2021 10:41AM by Martin-Éric (noreply@blogger.com)

hackergotchi for Jonathan Carter

Jonathan Carter

Free Software Activities for 2020-12

Here’s a list of some Debian packaging work for December 2020.

2020-12-01: Sponsor package mangohud (0.6.1-1) for Debian unstable (mentors.debian.net request).

2020-12-01: Sponsor package spyne (2.13.16-1) for Debian unstable (Python team request).

2020-12-01: Sponsor package python-xlrd (1.2.0-1) for Debian unstable (Python team request).

2020-12-01: Sponsor package buildbot for Debian unstable (Python team request).

2020-12-08: Upload package calamares (3.2.35.1-1) to Debian unstable.

2020-12-09: Upload package btfs (2.23-1) to Debian unstable.

2020-12-09: Upload package feed2toot (0.15-1) to Debian unstable.

2020-12-09: Upload package gnome-shell-extension-harddisk-led (23-1) to Debian unstable.

2020-12-10: Upload package feed2toot (0.16-1) to Debian unstable.

2020-12-10: Upload package gnome-shell-extension-harddisk-led (24-1) to Debian unstable.

2020-12-13: Upload package xabacus (8.3.1-1) to Debian unstable.

2020-12-14: Upload package python-aniso8601 (8.1.0-1) to Debian unstable.

2020-12-19: Upload package rootskel-gtk (1.42) to Debian unstable.

2020-12-21: Sponsor package goverlay (0.4.3-1) for Debian unstable (mentors.debian.net request).

2020-12-21: Sponsor package pastel (0.2.1-1) for Debian unstable (Python team request).

2020-12-22: Sponsor package python-requests-toolbelt (0.9.1-1) for Debian unstable (Python team request).

2020-12-22: Upload kpmcore (20.12.0-1) to Debian unstable.

2020-12-26: Upload package bundlewrap (4.3.0-1) to Debian unstable.

2020-12-26: Review package python-strictyaml (1.1.1-1) (Needs some more work) (Python team request).

2020-12-26: Review package buildbot (2.9.3-1) (Needs some more work) (Python team request).

2020-12-26: Review package python-vttlib (0.9.1+dfsg-1) (Needs some more work) (Python team request).

2020-12-26: Sponsor package python-formencode (2.0.0-1) for Debian unstable (Python team request).

2020-12-26: Sponsor package pylev (1.2.0-1) for Debian unstable (Python team request).

2020-12-26: Review package python-absl (Needs some more work) (Python team request).

2020-12-26: Sponsor package python-moreorless (0.3.0-2) for Debian unstable (Python team request).

2020-12-26: Sponsor package peewee (3.14.0+dfsg-1) for Debian unstable (Python team request).

2020-12-28: Sponsor package pympler (0.9+dfsg1-1) for Debian unstable (Python team request).

2020-12-28: Sponsor package bidict (0.21.2-1) for Debian unstable (Python team request).

02 January, 2021 07:19AM by jonathan

Paul Wise

FLOSS Activities December 2020

Focus

This month I didn't have any particular focus. I just worked on issues in my info bubble.

Changes

Issues

Review

Administration

  • Debian: restart bacula director, ping some people about disk usage
  • Debian wiki: unblock IP addresses, approve accounts, update email for accounts with bouncing email

Communication

  • Respond to queries from Debian users and contributors on the mailing lists and IRC

Sponsors

All work was done on a volunteer basis.

02 January, 2021 01:26AM

January 01, 2021

Jonathan Wiltshire

WordPress in a subdirectory

For many years now I’ve had WordPress installed as a subdirectory to my site but appearing to at the domain level, i.e. /wordpress/index.php is transparently presented as the homepage. This is done by setting the “WordPress Address” and “Site Address” settings and then mapping requests which do not match an existing file or directory through as a PHP pathinfo using Apache’s mod_rewrite rules in a .htaccess file or server configuration.

In this way most of the site is WordPress’s dynamic pages and posts, but WordPress itself is neatly contained and random static resources such as /screentest/1024.GIF work as expected.

Those .htaccess rules were originally hand-crafted but didn’t take account of changing recommendations, e.g. the Authorization header. When I rearranged matters recently I decided to take advantage of WordPress’s own generated rules and ditch my old rules.

They look like this:

# BEGIN WordPress
# The directives (lines) between "BEGIN WordPress" and "END WordPress" are
# dynamically generated, and should only be modified via WordPress filters.
# Any changes to the directives between these markers will be overwritten.
<IfModule mod_rewrite.c>
RewriteEngine On
RewriteRule .* - [E=HTTP_AUTHORIZATION:%{HTTP:Authorization}]
RewriteBase /
RewriteRule ^index\.php$ - [L]
RewriteCond %{REQUEST_FILENAME} !-f
RewriteCond %{REQUEST_FILENAME} !-d
RewriteRule . /index.php [L]
</IfModule>

# END WordPress

If .htaccess is writable, WordPress adds these rules automatically. Unfortunately, when I switched to them and not my own, the site broke.

Crucially the generated rules do not cater for WordPress being in a subdirectory. No matter what combination of settings I tried, local changes to the rules (like RewriteRule . /wordpress/index.php [L]) were rewritten back to the original values at random times and clearly couldn’t be relied upon to stay as I need them.

This feels like a bug: my intuition is that if the “WordPress Address” and “Site Address” settings are different, but within the same domain, the rules should be generated to take care of that.

One option is to make the .htaccess file read-only, which WordPress detects and avoids trying to make changes at all. But this rather defeats the object of letting it take care of future changes as the software and specifications change.

The second option, which I stuck with, is to add an index.php file with the following contents:

<?php
include('wordpress/index.php');

Now the generated rules, though incorrect, result in the right behaviour and won’t break on every setting change.

01 January, 2021 07:12PM by Jonathan

Russ Allbery

2020 Book Reading in Review

In 2020, I finished and reviewed 42 books, two more than 2019 (although I had finished but not reviewed two books at the end of 2019, so the total is even more similar than that). This is the best year for reading in terms of book count since 2012, despite being seriously distracted by a new job, a pandemic, and US political meltdowns. Those distractions do show up in the drop in page count.

If it weren't for the pandemic, the count would have been higher. Just as I got into a rhythm of reading while I exercised, gyms became a bad idea for the rest of the year. Treadmills are good for reading; long walks around the neighborhood not so much. That time went to podcasts instead, which I'm not too sad about but which don't prompt reviews.

Finding the mental space and energy to write reviews proved as much of a challenge as finding time to read this year, and I once again had to do some catch-up at the end of the year. To the extent that I have goals for 2021, it's to tighten up the elapsed time between finishing a book and writing a review so that the reviews don't pile up.

I read one book this year that I rated 10 out of 10: Michael Lewis's The Fifth Risk, which is partly about the US presidential transition and is really about what the US government does and what sort of people make careers in civil service. This book is brilliant, fascinating, and surprisingly touching, and I wish it were four times as long. If anything, it's even more relevant today as we enter another transition than it was when Lewis wrote it or when I read it.

There were so many 9 out of 10 ratings this year that it's hard to know where to start. I read the last Murderbot novella by Martha Wells (Exit Strategy) and then the first Murderbot novel (Network Effect), both of which were everything I was hoping for. Murderbot's sarcastic first-person voice continues to be a delight, and I expect Network Effect to take home several 2021 awards. I'm eagerly awaiting the next novel, Fugitive Telemetry, currently scheduled for the end of April, 2021.

Also on the fiction side were Alix E. Harrow's wonderful debut novel The Ten Thousand Doors of January, a fierce novel about family and claiming power that will hopefully win the 2020 Mythopoeic Award (which was delayed by the pandemic), and TJ Klune's heart-warming The House in the Cerulean Sea, my feel-good novel of the year. Finally, Tamsyn Muir's Gideon the Ninth and Harrow the Ninth were a glorious mess in places, but I had more fun reading and discussing those books than I've had with any novel in a very long time.

On the non-fiction side, Tressie McMillan Cottom's Thick is the best collection of sociology that I've read. It's not easy reading, but that book gave me a new-found appreciation and understanding of sociology and what it's trying to accomplish. Gretchen McCulloch's Because Internet is much easier reading but similarly perspective-enhancing, helping me understand (among other things) how choice of punctuation and capitalization expands the dynamic range of meaning in informal text conversation. Finally, Nick Pettigrew's Anti-Social is a funny, enlightening, and sobering look at the process of addressing low-level unwanted behavior that's directly relevant to the current conflicts over the role of policing in society.

The full analysis includes some additional personal reading statistics, probably only of interest to me.

01 January, 2021 07:11PM

hackergotchi for Chris Lamb

Chris Lamb

OpenUK Influencer 2021

After a turbulent 2020, I am very grateful to have been chosen in OpenUK's 2021 honours as one of the 100 top influencers in the UK's open technology community, which recognises contributions to open source software, open data and open hardware.

Congratulations to all of the other open source heroes and heroines who were also listed — am looking forward to an exciting year together.

01 January, 2021 04:58PM

Andrej Shadura

Transitioning to a new OpenPGP key

Following dkg’s example, I decided to finally transition to my new ed25519/cv25519 key.

Unlike Daniel, I’m not yet trying to split identities, but I’m using this chance to drop old identities I no longer use. My new key only has my main email address and the Debian one, and only those versions of my name I still want around.

My old PGP key (at the moment in the Debian keyring) is:

pub   rsa4096/0x6EA4D2311A2D268D 2010-10-13 [SC] [expires: 2021-11-11]
      782130B4C9944247977B82FD6EA4D2311A2D268D
uid                   [ultimate] Andrej Shadura <andrew@shadura.me>
uid                   [ultimate] Andrew Shadura <andrew@shadura.me>
uid                   [ultimate] Andrew Shadura <andrewsh@debian.org>
uid                   [ultimate] Andrew O. Shadoura <Andrew.Shadoura@gmail.com>
uid                   [ultimate] Andrej Shadura <andrewsh@debian.org>
sub   rsa4096/0xB2C0FE967C940749 2010-10-13 [E]
sub   rsa3072/0xC8C5F253DD61FECD 2018-03-02 [S] [expires: 2021-11-11]
sub   rsa2048/0x5E408CD91CD839D2 2018-03-10 [S] [expires: 2021-11-11]

The is the key I’ve been using in Debian from the very beginning, and its copies at the SKS keyserver network still have my first DD signature from angdraug:

sig  sig   85EE3E0E 2010-12-03 __________ __________ Dmitry Borodaenko <angdraug@mail.ru>
sig  sig   CB4D38A9 2010-12-03 __________ __________ Dmitry Borodaenko <angdraug@debian.org>

My new PGP key is:

pub   ed25519/0xE8446B4AC8C77261 2016-06-13 [SC] [expires: 2022-06-25]
      83DCD17F44B22CC83656EDA1E8446B4AC8C77261
uid                   [ultimate] Andrej Shadura <andrew@shadura.me>
uid                   [ultimate] Andrew Shadura <andrew@shadura.me>
uid                   [ultimate] Andrej Shadura <andrewsh@debian.org>
uid                   [ultimate] Andrew Shadura <andrewsh@debian.org>
uid                   [ultimate] Andrei Shadura <andrew@shadura.me>
sub   cv25519/0xD5A55606B6539A87 2016-06-13 [E] [expires: 2022-06-25]
sub   ed25519/0x52E0EA6F91F1DB8A 2016-06-13 [A] [expires: 2022-06-25]

If you signed my old key and are reading this, please consider signing my new key; if you feel you need to re-confirm this, feel free to contact me; otherwise, a copy of this statement signed by both the old and new keys is available here.

I have uploaded this new key to keys.openpgp.org, and also published it through WKD and the SKS network. Both keys can also be downloaded from my website.

01 January, 2021 03:08PM by Andrej Shadura

Utkarsh Gupta

FOSS Activites in December 2020

Here’s my (fifteenth) monthly update about the activities I’ve done in the F/L/OSS world.

Debian

This was my 24th month of contributing to Debian. I became a DM in late March last year and a DD last Christmas! \o/

Amongs a lot of things, this was month was crazy, hectic, adventerous, and the last of 2020 – more on some parts later this month.
I finally finished my 7th semester (FTW!) and moved onto my last one! That said, I had been busy with other things™ but still did a bunch of Debian stuff

Here are the following things I did this month:

Uploads and bug fixes:

Other $things:

  • Attended the Debian Ruby team meeting.
  • Mentoring for newcomers.
  • FTP Trainee reviewing.
  • Moderation of -project mailing list.
  • Sponsored golang-github-gorilla-css for Fedrico.

Debian (E)LTS

Debian Long Term Support (LTS) is a project to extend the lifetime of all Debian stable releases to (at least) 5 years. Debian LTS is not handled by the Debian security team, but by a separate group of volunteers and companies interested in making it a success.

And Debian Extended LTS (ELTS) is its sister project, extending support to the Jessie release (+2 years after LTS support).

This was my fifteenth month as a Debian LTS and sixth month as a Debian ELTS paid contributor.
I was assigned 26.00 hours for LTS and 38.25 hours for ELTS and worked on the following things:

LTS CVE Fixes and Announcements:

  • Issued DLA 2474-1, fixing CVE-2020-28928, for musl.
    For Debian 9 Stretch, these problems have been fixed in version 1.1.16-3+deb9u1.
  • Issued DLA 2481-1, fixing CVE-2020-25709 and CVE-2020-25710, for openldap.
    For Debian 9 Stretch, these problems have been fixed in version 2.4.44+dfsg-5+deb9u6.
  • Issued DLA 2484-1, fixing #969126, for python-certbot.
    For Debian 9 Stretch, these problems have been fixed in version 0.28.0-1~deb9u3.
  • Issued DLA 2487-1, fixing CVE-2020-27350, for apt.
    For Debian 9 Stretch, these problems have been fixed in version 1.4.11. The update was prepared by the maintainer, Julian.
  • Issued DLA 2488-1, fixing CVE-2020-27351, for python-apt.
    For Debian 9 Stretch, these problems have been fixed in version 1.4.2. The update was prepared by the maintainer, Julian.
  • Issued DLA 2495-1, fixing CVE-2020-17527, for tomcat8.
    For Debian 9 Stretch, these problems have been fixed in version 8.5.54-0+deb9u5.
  • Issued DLA 2488-2, for python-apt.
    For Debian 9 Stretch, these problems have been fixed in version 1.4.3. The update was prepared by the maintainer, Julian.
  • Issued DLA 2508-1, fixing CVE-2020-35730, for roundcube.
    For Debian 9 Stretch, these problems have been fixed in version 1.2.3+dfsg.1-4+deb9u8. The update was prepared by the maintainer, Guilhem.

ELTS CVE Fixes and Announcements:

Other (E)LTS Work:

  • Front-desk duty from 21-12 until 27-12 and from 28-12 until 03-01 for both LTS and ELTS.
  • Triaged openldap, python-certbot, lemonldap-ng, qemu, gdm3, open-iscsi, gobby, jackson-databind, wavpack, cairo, nsd, tomcat8, and bountycastle.
  • Marked CVE-2020-17527/tomcat8 as not-affected for jessie.
  • Marked CVE-2020-28052/bountycastle as not-affected for jessie.
  • Marked CVE-2020-14394/qemu as postponed for jessie.
  • Marked CVE-2020-35738/wavpack as not-affected for jessie.
  • Marked CVE-2020-3550{3-6}/qemu as postponed for jessie.
  • Marked CVE-2020-3550{3-6}/qemu as postponed for stretch.
  • Marked CVE-2020-16093/lemonldap-ng as no-dsa for stretch.
  • Marked CVE-2020-27837/gdm3 as no-dsa for stretch.
  • Marked CVE-2020-{13987, 13988, 17437}/open-iscsi as no-dsa for stretch.
  • Marked CVE-2020-35450/gobby as no-dsa for stretch.
  • Marked CVE-2020-35728/jackson-databind as no-dsa for stretch.
  • Marked CVE-2020-28935/nsd as no-dsa for stretch.
  • Auto EOL’ed libpam-tacplus, open-iscsi, wireshark, gdm3, golang-go.crypto, jackson-databind, spotweb, python-autobahn, asterisk, nsd, ruby-nokogiri, linux, and motion for jessie.
  • General discussion on LTS private and public mailing list.

Other $things! \o/

Bugs and Patches

Well, I did report some bugs and issues and also sent some patches:

  • Issue #44 for github-activity-readme, asking for a feature request to set custom committer’s email address.
  • Issue #711 for git2go, reporting build failure for the library.
  • PR #89 for rubocop-rails_config, bumping RuboCop::Packaging to v0.5.
  • Issue #36 for rubocop-packaging, asking to try out mutant :)
  • PR #212 for cucumber-ruby-core, bumping RuboCop::Packaging to v0.5.
  • PR #213 for cucumber-ruby-core, enabling RuboCop::Packaging.
  • Issue #19 for behance, asking to relax constraints on faraday and faraday_middleware.
  • PR #37 for rubocop-packaging, enabling tests against ruby3.0! \o/
  • PR #489 for cucumber-rails, bumping RuboCop::Packaging to v0.5.
  • Issue #362 for nheko, reporting a crash when opening the application.
  • PR #1282 for paper_trail, adding RuboCop::Packaging amongst other used extensions.
  • Bug #978640 for nheko Debian package, reporting a crash, as a result of libfmt7 regression.

Misc and Fun

Besides squashing bugs and submitting patches, I did some other things as well!

  • Participated in my first Advent of Code event! :)
    Whilst it was indeed fun, I didn’t really complete it. No reason, really. But I’ll definitely come back stronger next year, heh! :)
    All the solutions thus far could be found here.
  • Did a couple of reviews for some PRs and triaged some bugs here and there, meh.
  • Also did some cloud debugging, not so fun if you ask me, but cool enough to make me want to do it again! ^_^
  • Worked along with pollo, zigo, ehashman, rlb, et al for puppet and puppetserver in Debian. OMG, they’re so lovely! <3
  • Ordered some interesting books to read January onward. New year resolution? Meh, not really. Or maybe. But nah.
  • Also did some interesting stuff this month but can’t really talk about it now. Hopefully sooooon.

Until next time.
:wq for today.

01 January, 2021 12:30PM

Dmitry Shachnev

A review of endianness bugs in Qt, and how they were fixed

As you may know, I am Qt 5 maintainer in Debian. Maintaning Qt means not only bumping the version each time a new version is released, but also making sure Qt builds successfully on all architectures that are supported in Debian (and for some submodules, the automatic tests pass).

An important sort of build failures are endianness specific failures. Most widely used architectures (x86_64, aarch64) are little endian. However, Debian officially supports one big endian architecture (s390x), and unofficially a few more ports are provided, such as ppc64 and sparc64.

Unfortunately, Qt upstream does not have any big endian machine in their CI system, so endianness issues get noticed only when the packages fail to build on our build daemons. In the last years I have discovered and fixed some such issues in various parts of Qt, so I decided to write a post to illustrate how to write really cross-platform C/C++ code.

Issue 1: the WebP image format handler (code review)

The relevant code snippet is:

if (srcImage.format() != QImage::Format_ARGB32)
    srcImage = srcImage.convertToFormat(QImage::Format_ARGB32);
// ...
if (!WebPPictureImportBGRA(&picture, srcImage.bits(), srcImage.bytesPerLine())) {
    // ...
}

The code here is serializing the images into QImage::Format_ARGB32 format, and then passing the bytes into WebP’s import function. With this format, the image is stored using a 32-bit ARGB format (0xAARRGGBB). This means that the bytes will be 0xBB, 0xGG, 0xRR, 0xAA or little endian and 0xAA, 0xRR, 0xGG, 0xBB on big endian. However, WebPPictureImportBGRA expects the first format on all architectures.

The fix was to use QImage::Format_RGBA8888. As the QImage documentation says, with this format the order of the colors is the same on any architecture if read as bytes 0xRR, 0xGG, 0xBB, 0xAA.

Issue 2: qimage_converter_map structure (code review)

The code seems to already support big endian. But maybe you can spot the error?

#if Q_BYTE_ORDER == Q_LITTLE_ENDIAN
        0,
        convert_ARGB_to_ARGB_PM,
#else
        0,
        0
#endif

It is the missing comma! It is present in the little endian block, but not in the big endian one. This was fixed trivially.

Issue 3: QHandle, part of Qt 3D module (code review)

QHandle class uses a union that is declared as follows:

struct Data {
    quint32 m_index : IndexBits;
    quint32 m_counter : CounterBits;
    quint32 m_unused : 2;
};
union {
    Data d;
    quint32 m_handle;
};

The sizes are declared such as IndexBits + CounterBits + 2 is always equal to 32 (four bytes).

Then we have a constructor that sets the values of Data struct:

QHandle(quint32 i, quint32 count)
{
    d.m_index = i;
    d.m_counter = count;
    d.m_unused = 0;
}

The value of m_handle will be different depending on endianness! So the test that was expecting a particular value with given constructor arguments was failing. I fixed it by using the following macro:

#if Q_BYTE_ORDER == Q_BIG_ENDIAN
#define GET_EXPECTED_HANDLE(qHandle) ((qHandle.index() << (qHandle.CounterBits + 2)) + (qHandle.counter() << 2))
#else /* Q_LITTLE_ENDIAN */
#define GET_EXPECTED_HANDLE(qHandle) (qHandle.index() + (qHandle.counter() << qHandle.IndexBits))
#endif

Issue 4: QML compiler (code review)

The QML compiler used a helper class named LEUInt32 (based on QLEInteger) that always stored the numbers in little endian internally. This class can be safely mixed with native quint32 on little endian systems, but not on big endian.

Usually the compiler would warn about type mismatch, but here the code used reinterpret_cast, such as:

quint32 *objectTable = reinterpret_cast<quint32*>(data + qmlUnit->offsetToObjects);

So this was not noticed on build time, but the compiler was crashing. The fix was trivial again, replacing quint32 with QLEUInt32.

Issue 5: QModbusPdu, part of Qt Serial Bus module (code review)

The code snippet is simple:

QModbusPdu::FunctionCode code = QModbusPdu::Invalid;
if (stream.readRawData((char *) (&code), sizeof(quint8)) != sizeof(quint8))
    return stream;

QModbusPdu::FunctionCode is an enum, so code is a multi-byte value (even if only one byte is significant). However, (char *) (&code) returns a pointer to the first byte of it. It is the needed byte on little endian systems, but it is the wrong byte on big endian ones!

The correct fix was using a temporary one-byte variable:

quint8 codeByte = 0;
if (stream.readRawData((char *) (&codeByte), sizeof(quint8)) != sizeof(quint8))
    return stream;
QModbusPdu::FunctionCode code = (QModbusPdu::FunctionCode) codeByte;

Issue 6: qt_is_ascii (code review)

This function, as the name says, checks whether a string is ASCII. It does that by splitting the string into 4-byte chunks:

while (ptr + 4 <= end) {
    quint32 data = qFromUnaligned<quint32>(ptr);
    if (data &= 0x80808080U) {
        uint idx = qCountTrailingZeroBits(data);
        ptr += idx / 8;
        return false;
    }
    ptr += 4;
}

idx / 8 is the number of trailing zero bytes. However, the bytes which are trailing on little endian are actually leading on big endian! So we can use qCountLeadingZeroBits there.

Issue 7: the bundled copy of tinycbor (upstream pull request)

Similar to issue 5, the code was reading into the wrong byte:

if (bytesNeeded <= 2) {
    read_bytes_unchecked(it, &it->extra, 1, bytesNeeded);
    if (bytesNeeded == 2)
        it->extra = cbor_ntohs(it->extra);
}

extra has type uint16_t, so it has two bytes. When we need only one byte, we read into the wrong byte, so the resulting number is 256 times higher on big endian than it should be. Adding a temporary one-byte variable fixed it.

Issue 8: perfparser, part of Qt Creator (code review)

Here it is not trivial to find the issue just looking at the code:

qint32 dataStreamVersion = qToLittleEndian(QDataStream::Qt_DefaultCompiledVersion);

However the linker was producing an error:

undefined reference to `QDataStream::Version qbswap(QDataStream::Version)'

On little endian systems, qToLittleEndian is a no-op, but on big endian systems, it is a template function defined for some known types. But it turns out we need to explicitly convert enum values to a simple type, so the fix was passing qint32(QDataStream::Qt_DefaultCompiledVersion) to that function.

Issue 9: Qt Personal Information Management (code review)

The code in test was trying to represent a number as a sequence of bytes, using reinterpret_cast:

static inline QContactId makeId(const QString &managerName, uint id)
{
    return QContactId(QStringLiteral("qtcontacts:basic%1:").arg(managerName), QByteArray(reinterpret_cast<const char *>(&id), sizeof(uint)));
}

The order of bytes will be different on little endian and big endian systems! The fix was adding this line to the beginning of the function:

id = qToLittleEndian(id);

This will cause the bytes to be reversed on big endian systems.

What remains unfixed

There are still some bugs, which require deeper investigation, for example:

P.S. We are looking for new people to help with maintaining Qt 6. Join our team if you want to do some fun work like described above!

01 January, 2021 09:35AM by Dmitry Shachnev

Russ Allbery

Review: A Wizard's Guide to Defensive Baking

Review: A Wizard's Guide to Defensive Baking, by T. Kingfisher

Publisher: Red Wombat Studio
Copyright: July 2020
ASIN: B08CJ86Y1W
Format: Kindle
Pages: 287

Mona is fourteen, an orphan, and works in the bakery owned by her Aunt Tabitha. She's also a magicker, although a very minor one. She can tell bread to do things, like bake properly or slice itself, and can make gingerbread men dance. Also, there's Bob, her sourdough starter, into whom she put a bit too much panicked magic when she was ten. Her magic is a useful small talent for a baker, but nothing all that exceptional.

The dead body she finds in the kitchen when opening the bakery is certainly exceptional.

A Wizard's Guide to Defensive Baking starts as a minor murder mystery. There are constables and sweet buns and Mona is accused of murder by an officious inquisitor, which was rather terrifying, but everything seems like it will work out and get back to normal. Except it won't, because someone is murdering magickers and the authorities who are supposed to be helping Mona don't appear to be on her side.

This is a very Ursula Vernon sort of book. (T. Kingfisher is the pseudonym that Vernon uses when not writing children's books.) The protagonist is brave, scared, and stubborn, the first-person narration has a stream-of-consciousness feel and an undercurrent of constant curiosity, and the story has a strong "this is awful but moping about it won't help so we may as well get on with it" energy. It is (as you might guess from the age of the protagonist) pitched a bit younger than Vernon's other recent Kingfisher work. If I had to classify it, I'd call it young adult possibly edging into middle grade.

It's also a book about creative use of magic powers. If you're the sort of person who liked analyzing an apparently unimpressive superpower and thinking up all the creative ways in which it could be quite powerful, well, that's a lot of the plot here. Vernon sticks to the rules of the game: Mona can only affect bread and dough and maybe icing in tiny ways if she tries very hard (but it gives her a headache). But there are a lot of creative ways that one can use dough and cookies, particularly once you get some help.

The other interesting and unusual thing about this book is its attitude towards authorities and heroism. Authorities who try to do good and authorities who are evil and corrupt are both common in fantasy. Authorities who let themselves get caught up in a tangle of small decisions and minor fears until they become ineffective are not quite as common, and usually fantasy dispenses with that sort of power by calling it weak and inviting the heroes to take over. This book does not go in that direction at all. It's startling and thought-provoking to see a fantasy novel treat Mona's elevation into unwanted heroism as an indefensible societal failure that is not excused by gratitude. This more than any other story element made this feel like a 2020 book.

I won't spoil the ending, but it caught me by surprise, was extremely moving, and further broadens that questioning of what heroism is and why we celebrate it.

This is not a book full of complex plotting or moral ambiguity. The villain is awful, the threat he provokes is rather thoroughly dehumanized, and the plot is a reasonably straightforward sequence of events. One reads this book for Mona's flustered first-person voice, Vernon's humor and no-nonsense morality, and the creative exploration of the limits of what one can do with bread magic and a seriously irritated sourdough starter. And the thoughtfulness about heroism, and the recognition that heroism comes with an impact that the rest of society doesn't want to think about.

To be clear, that thoughtfulness doesn't go beyond questioning here. This is still a heroism book. But it made me want to read the fantasy novel in which people work collectively to remake a society so that it won't need to have heroes again.

I think whether you like Vernon's previous Kingfisher books, particularly the more young adult ones, will strongly predict whether you like A Wizard's Guide to Defensive Baking. If you like her protagonists, recommended.

Rating: 7 out of 10

01 January, 2021 04:37AM

hackergotchi for Keith Packard

Keith Packard

kgames

Reviving Very Old X Code

I've taken the week between Christmas and New Year's off this year. I didn't really have anything serious planned, just taking a break from the usual routine. As often happens, I got sucked into doing a project when I received this simple bug report Debian Bug #974011

I have been researching old terminal and X games recently, and realized
that much of the code from 'xmille' originated from the terminal game
'mille', which is part of bsdgames.

...

[The copyright and license information] has been stripped out of all
code in the xmille distribution.  Also, none of the included materials
give credit to the original author, Ken Arnold.

The reason the 'xmille' source is missing copyright and license information from the 'mille' files is that they were copied in before that information was added upstream. Xmille forked from Mille around 1987 or so. I wrote the UI parts for the system I had at the time, which was running X10R4. A very basic port to X11 was done at some point, and that's what Debian has in the archive today.

At some point in the 90s, I ported Xmille to the Athena widget set, including several custom widgets in an Xaw extension library, Xkw. It's a lot better than the version in Debian, including displaying the cards correctly (the Debian version has some pretty bad color issues).

Here's what the current Debian version looks like:

Fixing The Bug

To fix the missing copyright and license information, I imported the mille source code into the "latest" Xaw-based version. The updated mille code had a number of bug fixes and improvements, along with the copyright information.

That should have been sufficient to resolve the issue and I could have constructed a suitable source package from whatever bits were needed and and uploaded that as a replacement 'xmille' package.

However, at some later point, I had actually merged xmille into a larger package, 'kgames', which also included a number of other games, including Reversi, Dominoes, Cribbage and ten Solitaire/Patience variants. (as an aside, those last ten games formed the basis for my Patience Palm Pilot application, which seems to have inspired an Android App of the same name...)

So began my yak shaving holiday.

Building Kgames in 2020

Ok, so getting this old source code running should be easy, right? It's just a bunch of C code designed in the 80s and 90s to work on VAXen and their kin. How hard could it be?

  1. Everything was a 32-bit computer back then; pointers and ints were both 32 bits, so you could cast them with wild abandon and cause no problems. Today, testing revealed segfaults in some corners of the code.

  2. It's K&R C code. Remember that the first version of ANSI-C didn't come out until 1989, and it was years later that we could reliably expect to find an ANSI compiler with a random Unix box.

  3. It's X11 code. Fortunately (?), X11 hasn't changed since these applications were written, so at least that part still works just fine. Imagine trying to build Windows or Mac OS code from the early 90's on a modern OS...

I decided to dig in and add prototypes everywhere; that found a lot of pointer/int casting issues, as well as several lurking bugs where the code was just plain broken.

After a day or so, I had things building and running and was no longer hitting crashes.

Kgames 1.0 uploaded to Debian New Queue

With that done, I decided I could at least upload the working bits to the Debian archive and close the bug reported above. kgames 1.0-2 may eventually get into unstable, presumably once the Debian FTP team realizes just how important fixing this bug is. Or something.

Here's what xmille looks like in this version:

And here's my favorite solitaire variant too:

But They Look So Old

Yeah, Xaw applications have a rustic appearance which may appeal to some, but for people with higher resolution monitors and “well seasoned” eyesight, squinting at the tiny images and text makes it difficult to enjoy these games today.

How hard could it be to update them to use larger cards and scalable fonts?

Xkw version 2.0

I decided to dig in and start hacking the code, starting by adding new widgets to the Xkw library that used cairo for drawing instead of core X calls. Fortunately, the needs of the games were pretty limited, so I only needed to implement a handful of widgets:

  • KLabel. Shows a text string. It allows the string to be left, center or right justified. And that's about it.

  • KCommand. A push button, which uses KLabel for the underlying presentation.

  • KToggle. A push-on/push-off button, which uses KCommand for most of the implementation. Also supports 'radio groups' where pushing one on makes the others in the group turn off.

  • KMenuButton. A button for bringing up a menu widget; this is some pretty simple behavior built on top of KCommand.

  • KSimpleMenu, KSmeBSB, KSmeLine. These three create pop-up menus; KSimpleMenu creates a container which can hold any number of KSmeBSB (string) and KSmeLine (separator lines) objects).

  • KTextLine. A single line text entry widget.

The other Xkw widgets all got their rendering switched to using cairo, plus using double buffering to make updates look better.

SVG Playing Cards

Looking on wikimedia, I found a page referencing a large number of playing cards in SVG form That led me to Adrian Kennard's playing card web site that let me customize and download a deck of cards, licensed using the CC0 Public Domain license.

With these cards, I set about rewriting the Xkw playing card widget, stripping out three different versions of bitmap playing cards and replacing them with just these new SVG versions.

SVG Xmille Cards

Ok, so getting regular playing cards was good, but the original goal was to update Xmille, and that has cards hand drawn by me. I could just use those images, import them into cairo and let it scale them to suit on the screen. I decided to experiment with inkscape's bitmap tracing code to see what it could do with them.

First, I had to get them into a format that inkscape could parse. That turned out to be a bit tricky; the original format is as a set of X bitmap layers; each layer painting a single color. I ended up hacking the Xmille source code to generate the images using X, then fetching them with XGetImage and walking them to construct XPM format files which could then be fed into the portable bitmap tools to create PNG files that inkscape could handle.

The resulting images have a certain charm:

I did replace the text in the images to make it readable, otherwise these are untouched from what inkscape generated.

The Results

Remember that all of these are applications built using the venerable X toolkit; there are still some non-antialiased graphics visible as the shaped buttons use the X Shape extension. But, all rendering is now done with cairo, so it's all anti-aliased and all scalable.

Here's what Xmille looks like after the upgrades:

And here's spider:

Once kgames 1.0 reaches Debian unstable, I'll upload these new versions.

01 January, 2021 03:57AM

hackergotchi for Junichi Uekawa

Junichi Uekawa

A happy new year.

A happy new year. It was a challenging year last year; hopefully the new one will be less of a stay-home.

01 January, 2021 12:32AM by Junichi Uekawa

December 31, 2020

hackergotchi for Chris Lamb

Chris Lamb

Free software activities in December 2020

Here is my monthly update covering what I have been doing in the free software world during December 2020 (previous month):

  • Reviewed and merged a contribution from Peter Law to my django-cache-toolbox library for Django-based web applications, including explicitly requiring that cached relations are primary keys (#23) and improving the example in the README (#25).
  • I took part in an interview with Vladimir Bejdo, an intern at the Software Freedom Conservancy, in order to talk about the Reproducible Builds project, my participation in software freedom, the importance of reproducibility in software development, and to have a brief discussion on the issues facing free software as a whole. The full interview can be found on Conservancy's webpages.
  • As part of my duties of being on the board of directors of the Open Source Initiative, I attended its monthly meeting and participated in various licensing and other related discussions occurring on the internet. Unfortunately, I could not attend the parallel meeting for Software in the Public Interest this month.

§


Reproducible Builds

One of the original promises of open source software is that distributed peer review and transparency of process results in enhanced end-user security. However, whilst anyone may inspect the source code of free and open source software for malicious flaws, almost all software today is distributed as pre-compiled binaries. This allows nefarious third-parties to compromise systems by injecting malicious code into ostensibly secure software during the various compilation and distribution processes.

The motivation behind the Reproducible Builds effort is to ensure no flaws have been introduced during this compilation process by promising identical results are always generated from a given source, thus allowing multiple third-parties to come to a consensus on whether a build was compromised.

This month, I:

  • Submitted a draft academic paper to IEEE Software. The article (co-written by Stefano Zacchiroli) is aimed a fairly general audience. It first defines the overal problem and then provides insight into the challenges of actually making real-world software reproducible. It then outlines the experiences of the Reproducible Builds project in making large-scale software collections/supply-chains/ecosystems reproducible and concludes by describing the affinity between reproducibility efforts and quality assurance.

  • Kept isdebianreproducibleyet.com up to date. [...]

  • Submitted 11 patches in Debian to fix specific reproducibility issues in circlator, dvbstreamer, eric, jbbp, knot-resolver, libjs-qunit, mail-expire, osmo-mgw, python-pyramid, pyvows & sayonara.

  • Categorised a huge number of packages and issues in the Reproducible Builds 'notes' repository.

  • For disorderfs (our FUSE-based filesystem that deliberately introduces non-determinism into directory system calls in order to flush out reproducibility issues), I made the following changes:

  • Drafted, published and publicised our monthly report, as well managed the project's various social media accounts.

  • Contributed to a discussion about the recent 'SolarWinds' attack. [...]

I also made the following changes to diffoscope, our in-depth and content-aware diff utility that can locate and diagnose reproducibility issues, including releasing version 163:

  • New features & bug fixes:

    • Normalise ret to retq in objdump output in order to support multiple versions of GNU binutils. (#976760)
    • Don't show any progress indicators when running zstd. (#226)
    • Correct the grammatical tense in the --debug log output. [...]
  • Codebase improvements:

    • Update the debian/copyright file to match the copyright notices in the source tree. (#224)
    • Update various years across the codebase in .py copyright headers. [...]
    • Rewrite the filter routine that post-processes the output from readelf(1). [...]
    • Remove unnecessary PEP 263 encoding header lines; unnecessary after PEP 3120. [...]
    • Use minimal instead of basic as a variable name to match the underlying package name. [...]
    • Use pprint.pformat in the JSON comparator to serialise the differences from jsondiff. [...]

§


Debian

Uploads

  • python-django:

  • redis:

    • 6.2~rc1-1 — New upstream RC release.
    • 6.2~rc1-2 — Enable systemd support by compiling against the libsystemd-dev package. (#977852)
    • 6.2~rc1-3 — Specify --supervised systemd now that we specify Type=notify to prevent service startup failure under systemd.
  • mtools (4.0.26-1) — New upstream release.

I also sponsored an upload of adminer (4.7.8-2) on behalf of Alexandre Rossi and performed two QA uploads of sendfile (2.1b.20080616-7 and 2.1b.20080616-8) to make the build the build reproducible (#776938) and to fix a number of other unrelated issues.

Debian LTS

This month I have worked 18 hours on Debian Long Term Support (LTS) and 12 hours on its sister Extended LTS project.

You can find out more about the Debian LTS project via the following video:

31 December, 2020 06:19PM

hackergotchi for Shirish Agarwal

Shirish Agarwal

Pandemic, Informal economy and Khau Galli.

Formal sector issues

Just today it was published that output from eight formal sectors of the economy who make the bulk of the Indian economy were down on a month to month basis . This means all those apologists for the Government who said that it was ok that the Govt. didn’t give the 20 lakh crore package which was announced. In fact, a businessman from my own city, a certain Prafull Sarda had asked via RTI what happened to the 20 lakh crore package which was announced? The answers were in all media as well as newspapers but on the inside pages. You can see one of the article sharing the details here. No wonder Vivek Kaul also shared his take on the way things will hopefully go for the next quarter which seems to be a tad optimistic from where we are atm.

Eight Sectors declining in Indian Economy month-on-month CNBC TV 18

The Informal economy

The informal economy has been strangulated by the current party in power since it came into power. And this has resulted many small businesses which informal are and were part of culture of cities and towns. I share an article from 2018 which only shows how good and on the mark it has aged in the last two years. The damage is all to real to ignore as I would share more of an anecdotes and experiences because sadly there never has been any interest shown especially by GOI to seek any stats. about informal economy. Although CMIE has done some good work in that even though they majorly look at formal, usually blue-collar work where again there is not good data. Sharing an anecdote and a learning from these small businesses which probably an MBA guy wouldn’t know and in all honesty wouldn’t even care.

Khau galli

Few years back, circa 2014 and on wards, when the present Govt. came into power, it did come with lot of promises. One of which was that lot of informal businesses would be encouraged to grow their businesses and in time hopefully, they become part of the formal economy. Or at least that was the story that all of us were told. Due to that they did lot of promises and also named quite a few places where street food was in abundance. Such lanes were named ‘Khau galli’ for those who are from North India, it was be easily known and understood. This was just saying that here are some places where you could get a variety of food without paying obscene prices as you would have to vis-a-vis a restaurant. Slowly, they raised the rates of inputs (food grains), gas cylinder etc. which we know of as food inflation and via ‘GST’ made sure that the restaurants were able to absorb some of the increased inputs (input credit) while still being more than competitive to the street food person/s. The restaurant F&B model is pretty well known so not going there at all. It is however, important to point out that they didn’t make any new ‘khau gallis’ or such, most or all the places existed for years and even decades before that. They also didn’t take any extra effort either in marketing the khau gallis or get them with chefs or marketing folks so that the traditional can marry to the anew. They just named them, that was the only ‘gain’ to be seen on the ground.

In its heyday, the khau galli near my home used to have anywhere between 20-30 Thelas or food carts. Most of the food carts would be of wood and having very limited steel. Such food carts would cost anywhere around INR 15-20k instead of the food cart you see here. The only reason I shared that link is to show how a somewhat typical thela or food cart looks in India. Of course YouTube or any other media platform would show many. On top of it, you need and needed permission from the municipality a license for the same which would be auctioned. Now that license could well run from thousands into lakhs depending on various factors or you gave something to the Municipal worker when he did his rounds/beat much like a constable every day or week. Apart from those, you also have raw material expenditure which could easily run into few thousands depending upon what sort of food you are vending. You also would typically have 2-3 workers so a typical Thela would feed not only its customers but also 2-3 families who are the laborer families as well as surrounding businesses.

As I used to be loyal and usually go to few whom I found to be either tasty or somehow they were good for me. In either case, a relationship was formed. As I have been never fond of crowds, I usually used to in their off-beat hours either when they are close to packing up of when I know they usually have a lull. That way I knew I would get complete attention of the vendor/s. Many a times I used to see money change hands between the vendors themselves and used to see both camaraderie as well as competition between them. This is years ago, once while sharing a chai (tea) with one of the street vendors I casually asked I have often seen you guys exchanging money with each other and most of the time quite a bit of the money is also given to the guy who didn’t make that much sales or any sales at all.

The vendor replied sharing practical symbiotic knowledge. All of us are bound by a single thing called poverty. All of us are struggle. Do you know why so many people come here, because they know that there would be a variety of food to be had. Now if we stopped helping each other, the number of people who would make the effort would be also less, we know we are not the only game in town. Also whatever we give, sooner or later it gets adjusted. Also if one of us has good days, he knows hardship days are not far. Why, simply because people change tastes or want variety. So irrespective of good or bad the skills of the vendor is, she or he is bound to make some sales. The vendor either shares the food with us or whatever. Somehow these things just work out. And that doesn’t mean we don’t have our fights, we do have our fights, but we also understand this. Now you see this and you understand that these guys have and had a community. Even if they changed places due to one reason or the other, they kept themselves connected. Unlike many of us, who even find a hard time keeping up with friends let alone relatives.

Now cut to 2020, and where there used to be 20-30 thelas near my home, there are only 4-5. Of course, multiple reasons, but one of the biggest was of course demonetization. That was a huge shock to which many of thela walas succumbed. Their entire savings and capital were turned to dust. Many of their customers will turn up with either a INR 500 or INR 2000/- Re note where at the most a dish costed INR 100/- most times half or even 1/3rd of that amount. How and from where the thela walas could get that kind of cash. These are people who only if they earn, they and their family will have bread at night. Most of the loose change was tied up at middle to higher tier restaurants where they were giving between INR 20/- 30/- for every INR 100/- change of rupees and coins. Quite a few bankers made money by that as well as other means where the thela walas just could not compete. These guys also didn’t have any black money even though they were and are part of the black/informal economy. Sadly, till date no economist or even sociologist as far as I know has attempted or done any work from what I know on this industry. If you want to formalize such businesses then at the very least understand their problems and devise solutions. And I suspect, what is and has happened near my house has also happened everywhere else, at least within the geographical confines of the Indian state. Whether it was the 2016 demonetization or the pandemic, the results and effects have been similar the same all over. Some states did do well and still do, the suffering still continues.

With the hope that the new year brings cheer to you as well some more ideas to remain in business by the thela walas, I bid you adieu and see you in new year 🙂

31 December, 2020 05:39PM by shirishag75

hackergotchi for Holger Levsen

Holger Levsen

20201231-no-source-change-source-uploads

On doing 540 no-source-change source-only uploads in two weeks

So I've been doing 540 no-source-change source-only uploads in the last two weeks and am planning to do 3000 more in January 2021. We'll see how that goes ;)

Let me explain what I have been doing and why.

So, starting with the Bullseye release cycle the Release Team changed policy: only packages which were build on buildds are allowed to migrate to testing.

Which is pretty nice for reproducible builds as this also ensures that a .buildinfo file is available for anyone wanting to reproduce the binaries of that package.

However, there are many binary (and source) packages in Debian which were uploaded before 2016 (which is when .buildinfo files were introduced) or were uploaded with binaries until that change in release policy July 2019.

Then Ivo De Decker scheduled binNMUs for all the affected packages but due to the way binNMUs work, he couldn't do anything about arch:all packages as they currently cannot be rebuilt with binNMUs.

Ivo and myself discussed what could be done about the remaining packages and (besides long complicated changes to Debian's workflows) the only thing deemed possible was doing many many source uploads with just a new changelog entry:

  * Non maintainer upload by the Reproducible Builds team.
  * No source change upload to rebuild on buildd with .buildinfo files.

These packages are all inherently buggy, because Debian policy mandates that packages should be reproducible and without .buildinfo files one cannot reproducibly rebuild packages. So instead of filing many many bugs we've decided to just fix these bugs by doing a no-source-change source uploads. One nice aspect of these uploads is that there's no follow-up work imposed on the maintainer: whether they keep that changelog entry or whether they discard it, it does not matter.

So Ivo had developed an SQL query which showed 570 packages needing an update roughly two weeks ago, on December 18 and so I started slowly. This is the amount of NMUs I did in the last days:

for i in $(seq 18 30) ; do echo -n "Dec $i: " ; ls -lart1 done/*upload|grep -c "Dec $i" ; done
Dec 18: 12
Dec 19: 0
Dec 20: 3
Dec 21: 13
Dec 22: 13
Dec 23: 16
Dec 24: 4
Dec 25: 28
Dec 26: 0
Dec 27: 38
Dec 28: 198
Dec 29: 206
Dec 30: 9

About ten packages had FTBFS bugs preventing an upload and seven packages were uploaded by the maintainer before me. I've seen two cases of sudden maintainer uploads after 8 and 10 years of no activity!

So what did I do for each upload?

  • pre upload work:
    • test build with pbuilder in sid
    • check PTS for last upload date (and having arch:all binaries) and open RC bugs
    • modify d/changelog
    • check debdiff between two sources (should be only the changelog entry!)
    • upload
    • (some times filing bugs or modifying bug meta data etc)
  • post upload:
    • check PTS for testing migration, so for this I've had >500 browser tabs open and I'll keep each of them open until the packages migrates...

Much to my surprise I didn't get much feedback, there were like 6 people on the #debian-reproducible channel cheering and one on #debian-qa, though that person is a Release Team member so that was kind of important cheering. And I've seen some maintainer uploads to packages which haven't seen uploads since some years. And really nice: no-one complained so far. I hope this will stay this way with the plan to do 3000 more uploads of this kind:

Those 570 packages were only key packages but there are 3000 more source packages which have a binary in bullseye for which no .buildinfo file exists. So I plan to upload them all in January 2021 and you can help me doing so, by uploading your packages before me - and maybe fixing some other bugs in the process!

I've posted the list of packages (sorted by ddlist) to debian-devel@lists.d.o, see https://lists.debian.org/debian-devel/2020/12/msg00419.html.

Many thanks to Ivo and the whole Release Team for their support of Reproducible Builds and generally speaking for the many many enhancements to the release process we've seen over the years. Debian in 2021 will rock'n'roll more than ever! So thank you all, once again, for making Debian what it is and what it will be!

31 December, 2020 12:05PM

hackergotchi for Daniel Kahn Gillmor

Daniel Kahn Gillmor

New OpenPGP certificate for dkg, 2021

dkg's 2021 OpenPGP transition

As 2021 begins, I'm changing to a new OpenPGP certificate.

I did a similar transition two years ago, and a fair amount has changed since then.

You might know my old OpenPGP certificate as:

pub   ed25519 2019-01-19 [C] [expires: 2021-01-18]
      C4BC2DDB38CCE96485EBE9C2F20691179038E5C6
uid          Daniel Kahn Gillmor <dkg@fifthhorseman.net>
uid          Daniel Kahn Gillmor <dkg@debian.org>

My new OpenPGP certificate is:

pub   ed25519 2020-12-27 [C] [expires: 2023-12-24]
      C29F8A0C01F35E34D816AA5CE092EB3A5CA10DBA
uid           [ unknown] Daniel Kahn Gillmor
uid           [ unknown] <dkg@debian.org>
uid           [ unknown] <dkg@fifthhorseman.net>

You can find a signed transition statement if you're into that sort of thing.

If you're interested in the rationale for why I'm making this transition, read on.

Dangers of Offline Primary Secret Keys

There are several reasons for transitioning, but one i simply couldn't argue with was my own technical failure. I put the primary secret key into offline storage some time ago for "safety", and used ext4's filesystem-level encryption layered on top of dm-crypt for additional security.

But either the tools changed out from under me, or there were failures on the storage medium, or I've failed to remember my passphrase correctly, because I am unable to regain access to the cleartext of the secret key. In particular, I find myself unable to use e4crypt add_key with the passphrase I know to get a usable working directory.

I confess I still find e4crypt pretty difficult to use and I don't use it often, so the problem may entirely be user error (either now, or two years ago when I did the initial setup).

Anyway, lesson learned: don't use cryptosystems that you're not comfortable with to encrypt data that you care about recovering. This is a lesson I'm pretty sure I've learned before, sigh, but it's a good reminder.

Split User IDs

I'm trying to split out my User IDs again -- this way if you know me by e-mail address, you don't have to think/worry about certifying my name, and if you know me by name, you don't have to think/worry about certifying my e-mail address. I think that's simpler and more sensible.

It's also nice because e-mail address-only User IDs can be used effectively in contexts like Autocrypt, which I think are increasingly important if we want to have usable encrypted e-mail.

Last time around I initially tried split User IDs but rolled them back and I think most of the bugs I discovered then have been fixed.

Certificate Flooding

Another reason for making a transition to a new certificate is that my older certificate is one of the ones that was "flooded" on the SKS keyserver network last year, which was one of the final straws for that teetering project. Transitioning to a new certificate lets that old flooded cert expire and people can just simply move on from it, ideally deleting it from their local keyrings.

Hopefully as a community we can move on from SKS to key distribution mechanisms like WKD, Autocrypt, DANE, and keys.openpgp.org, all of which address some of the known problems with keyserver abuse.

Trying New Tools

Finally, I'm also interested in thinking about how key and certificate management might be handled in different ways. While I'm reasonably competent in handling GnuPG, the larger OpenPGP community (which I'm a part of) has done a lot of thinking and a lot of work about how people can use OpenPGP.

I'm particularly happy with the collaborative work that has gone into the Stateless OpenPGP CLI (aka sop), which helps to generate a powerful interoperability test suite. While sop doesn't offer the level of certificate management I'd need to use it to manage this new certificate in full, I wish something like it would! Starting from a fresh certificate and actually using it helps me to think through what I might actually need from a tool that is roughly as straightforward and opinionated as sop is.

If you're a software developer who might use or implement OpenPGP, or a protocol designer, and you haven't played around with any of the various implementations of sop yet, I recommend taking a look. And feedback on the specification is always welcome, too, including ideas for new functionality (maybe even like certificate management).

Next Steps

If you're the kind of person who's into making OpenPGP certifications, feel free to check in with me via whatever channels you're used to using to verify that this transition is legit. If you think it is, and you're comfortable, please send me (e-mail is probably best) your certifications over the new certficate.

I'll keep on working to make OpenPGP more usable and acceptable. Hopefully, 2021 will be a better year ahead for all of us.

31 December, 2020 05:00AM by Daniel Kahn Gillmor

Russ Allbery

Review: Billion Dollar Loser

Review: Billion Dollar Loser, by Reeves Wiedeman

Publisher: Little, Brown and Company
Copyright: October 2020
ISBN: 0-316-46134-2
Format: Kindle
Pages: 315

WeWork was founded in 2010 by Adam Neumann and Miguel McKelvey as a successor company to their similar 2008 GreenDesk business. (Adam's wife Rebekah is now presented as a co-founder. This seems dubious in Wiedeman's account, although Rebekah's role in the company is murky, ever-changing, and hard to pin down.) Its business model in reality was to provide turn-key, pre-furnished and stocked co-working and small office space to individuals and businesses on flexible, short-term leases. Its business model in Neumann's speeches and dreams, and represented by the later renaming of the company to the We Corporation, was nothing less than to transform the way people worked, learned, and lived.

Through aggressive, money-losing expansion, WeWork grew rapidly to over 500 locations in 29 countries and became the largest office tenant in New York City. Based primarily on massive financial support from Masayoshi Son, CEO of Japanese holding company SoftBank, WeWork's private valuation rose to $47 billion. In 2019, the company attempted to go public, but its IPO collapsed, in part due to deeper analysis of the company's books. Neumann was forced out of the company (with an individual payout valued at $1.7 billion), the IPO was withdrawn, SoftBank wrote down 90% of their investment in the company and took control of it, and WeWork laid off more than 20% of its workforce.

This book is a detailed history of WeWork's rise and fall, joining a genre that includes The Smartest Guys in the Room (Enron), Bad Blood (Theranos), and Super Pumped (Uber). I believe it's the first full book on WeWork, although it was preceded by several long-form stories, including "The I In We" by Wiedeman for New York magazine. As the first history, it's a somewhat incomplete cut: litigation between Neumann and WeWork is still pending, WeWork staggered into 2020 and a world-wide pandemic that made cramped open-plan offices an epidemiological disaster, and there will doubtless be new revelations yet to come. The discovery process of lawsuits tends to be good for journalists. But despite being the first out of the gate, Billion Dollar Loser reaches a satisfying conclusion with the ouster of Neumann, who had defined WeWork both internally and externally.

I'm fascinated by stories of failed venture capital start-ups in general, but the specific question about WeWork that interested me, and to which Wiedeman provides a partial answer, is why so many people gave Neumann money in the first place. Explaining that question requires a digression into why I thought WeWork's valuation was absurd.

The basic problem WeWork had when justifying its sky-high valuation is competition. WeWork didn't own real estate; it rented properties from landlords with long-term leases and then re-rented them with short-term leases. If its business was so successful, why wouldn't the landlords cut out the middle man, do what WeWork was doing directly, and pocket all the profit? Or why wouldn't some other company simply copy WeWork and drive the profit margins down? Normally with startups the answer revolves around something proprietary: an app, a server technology, patents, a secret manufacturing technique, etc. But nothing WeWork was doing was different from what innumerable tech companies and partner landlords had been doing with their office space for a decade, and none of it was secret.

There are two decent answers to that question. One is simple outsourcing: landlords like being passive rent collectors, so an opportunity to pay someone else to do the market research on office layouts, arrange all the remodeling, adapt to changing desires for how office space should be equipped and stocked, advertise for short-term tenants, and deal with the tenant churn is attractive. The landlord can sit back and pocket the stable long-term rent.

The second answer is related: WeWork is essentially doing rental arbitrage between long-term and short-term rents and thus is taking on most of the risk of a commercial real estate downturn. If no one is renting office space, WeWork is still on the hook for the long-term rent. The landlord is outsourcing risk, at least unless WeWork goes bankrupt. (One infuriating tidbit from this book is that Neumann's explicit and often-stated goal was to make WeWork so large that its bankruptcy would be sufficiently devastating to the real estate industry that it would get a bailout.)

There's a legitimate business here. But that business looks like a quietly profitable real estate company that builds very efficient systems for managing short-term leases, remodeling buildings, and handling the supply chain of stocking an office. That looks nothing like WeWork's business, has nothing to do with transforming the world of work, and certainly doesn't warrant sky-high valuations. WeWork didn't build an efficient anything. It relied on weekend labor from underpaid employees and an IT person who was still in high school. And WeWork actively resisted being called a real estate company and claimed it was a tech company or a lifestyle company on the basis of essentially nothing.

Wiedeman seems almost as baffled by this as I am, but it's clear from the history he tells that part of the funding answer is the Ponzi scheme of start-up investing. People gave Neumann money because other people had previously given Neumann money, and the earlier investors cashed out at the expense of the later ones. Like any Ponzi scheme, it looks like a great investment until it doesn't, and then the last sucker is left holding the bag. That sucker was Masayoshi Son, who in Wiedeman's telling is an astonishingly casual and undisciplined investor who trusted knee-jerk personal reactions to founders over business model analysis and historically (mostly) got away with it by getting extremely lucky.

(I now want to read one of these books about SoftBank, since both this book and Super Pumped make it look like a company that makes numerous wild gambles for the flimsiest of reasons, pushes for completely unsustainable growth, and relies on the sheer volume of investments catching some lucky jackpots and cashing out in IPOs. Unfortunately, the only book currently available seems to be a fawning hagiography of Son.)

On one hand, the IPO process worked properly this time. The sheer madness of WeWork's valuation scared off enough institutional investors that it collapsed. On the other hand, it's startling how close it came to success. If WeWork had kept the Ponzi scheme going a bit longer, the last sucker could have been the general investing public.

Another interesting question that Billion Dollar Loser answers is how Neumann got enough money to start his rapid growth strategy. The answer appears to be the oldest and most obvious explanation: He made friends with rich people. The initial connections appear to have been through his sister, Adi Neumann, who is a model and hosted parties in her New York apartment (and also started dating a Rothschild heir). Adam met his wealthy wife Rebekah, cousin to actress and "wellness" scam marketer Gwyneth Paltrow, via a connection at a party. He built social connections with other parts of the New York real estate scene and tapped them for investment money.

The strong impression one gets from the book is that all of these people have way more money than sense and we should raise their taxes. It won't come as a surprise that Adam and Rebekah Neumann are good friends of Jared Kushner and Ivanka Trump.

Those are the questions I was the most curious about, but there's much more here. Wiedeman's style is nearly straight chronological reporting with little analysis, but the story is so wild and absurd that it doesn't need much embellishment. Neumann is obviously a megalomaniac whose delusions of grandeur got worse and worse as WeWork was apparently succeeding. Rebekah Neumann is if anything even less in touch with reality than he is, although in her case that appears to stem from having so much money that reality is an inconvenient speed bump. Miguel McKelvey, Neumann's co-founder, is an odd and interesting side note to the story; he appears to have balanced Adam out a bit in the early going but then wisely started to cash out and pocket his winnings while letting Adam dominate the stage.

There are some places where I don't think Wiedeman pushed hard enough, and which cut against the view of Neumann as a true believer in his impossible growth vision. Neumann took several investment opportunities to cash out large amounts of his stock even while WeWork employees were being underpaid and told their stock options would make up for it. He clearly used WeWork as a personal piggy bank on multiple occasions. And Wiedeman documents but doesn't, at least in my opinion, make nearly enough of Neumann's self-dealing: buying real estate that WeWork then rented as a tenant, or paying himself for a license for the name We Holdings (although there at least he later returned the money). I think a good argument could be made that Neumann was embezzling from WeWork, at least morally if not legally, and I wish Wiedeman would have pressed harder on that point.

But that aside, this is a great first history of the company, told in a clean, readable, and engaging style, and with a lot more detail here than I've touched on (such as Rebekah Neumann's WeGrow school). It's not as good as Bad Blood (what is), but it's a respectable entry in the corporate collapse genre. If you like this sort of thing, recommended.

Rating: 7 out of 10

31 December, 2020 04:04AM

December 30, 2020

Andrej Shadura

Making the blog part of the Fediverse and IndieWeb

I’ve just made my blog available on the Fediverse, at least partially.

Yesterday while browsing Hacker News, I saw Carl Schwan’s post Adding comments to your static blog with Mastodon(m) about him replacing Disqus with replies posted at Mastodon. Just on Monday I was thinking, why can’t blogs participate in Fediverse? I tried to use WriteFreely as a replacement for Pelican, only to find it very limited, so I thought I might write a gateway to expose the Atom feed using ActivityPub. Turns out, someone already did that: Bridgy, a service connecting websites to Twitter, Mastodon and other social media, also has a Fediverse counterpart, Fed.brid.gy — just what I was looking for!

I won’t go deeply into the details, but just outline issues I had to deal with:

  • Bridgy relies on microformats. While I had some microformats and Schema.org microdata already at both blog and the homepage, they were not sufficiently good for Bridgy to use.
  • Webmention: I had to set up https://webmention.io to act as Webmention host for me.
  • I ran into an issue with Mastodon being more picky about signatures — which the author Ryan Barrett promptly fixed.

The end result is not fully working yet, since posts don’t appear yet, but I can be found and followed as @shadura.me@shadura.me.

As a bonus I enabled IndieAuth at my website, so I can log into compatible services with just the URL of my website.

30 December, 2020 07:07PM by Andrej Shadura

John Goerzen

Airgapped / Asynchronous Backups with ZFS over NNCP

In my previous articles in the series on asynchronous communication with the modern NNCP tool, I talked about its use for asynchronous, potentially airgapped, backups. The first article, How & Why To Use Airgapped Backups laid out the foundations for this. Now let’s dig into the details.

Today’s post will cover ZFS, because it has a lot of features that make it very easy to support in this setup. Non-ZFS backups will be covered later.

The setup is actually about as simple as it is for SSH, but since people are less familiar with this kind of communication, I’m going to try to go into more detail here.

Assumptions

I am assuming a setup where:

  • The machines being backed up run ZFS
  • The disk(s) that hold the backups are also running ZFS
  • zfs send / receive is desired as an efficient way to transport the backups
  • The machine that holds the backups may have no network connection whatsoever
  • Backups will be sent encrypted over some sort of network to a spooling machine, which temporarily holds them until they are transported to the destination backup system and ingested there. This system will be unable to decrypt the data streams it temporarily stores.

Hardware

Let’s start with hardware for the machine to hold the backups. I initially considered a Raspberry Pi 4 with 8GB of RAM. That would probably have been a suitable machine, at least for smaller backup sets. However, none of the Raspberry Pi machines support hardware AES encryption acceleration, and my Pi4 benchmarks as about 60MB/s for AES encryption. I want my backups to be encrypted, and decided this would just be too slow for my purposes. Again, if you don’t need encrypted backups or don’t care that much about performance — may people probably fall into this category — you can have a fully-functional Raspberry Pi 4 system for under $100 that would make a fantastic backup server.

I wound up purchasing a Qotom-Q355G4 micro PC with a Core i5 for about $315. It has USB 3 ports and is designed as a rugged, long-lasting system. I have been using one of their older Celeron-based models as my router/firewall for a number of years now and it’s been quite reliable.

For backup storage, you can get a USB 3 external drive. My own preference is to get a USB 3 “toaster” (device that lets me plug in SATA drives) so that I have more control over the underlying medium and can save the expense and hassle of a bunch of power supplies. In a future post, I will discuss drive rotation so you always have an offline drive.

Then, there is the question of transport to the backup machine. A simple solution would be to have a heavily-firewalled backup system that has no incoming ports open but makes occasional outgoing connections to one specific NNCP daemon on the spooling machine. However, for airgapped operation, it would also be very simple to use nncp-xfer to transport the data across on a USB stick or some such. You could set up automounting for a specific USB stick – plug it in, all the spooled data is moved over, then plug it in to the backup system and it’s processed, and any outbound email traffic or whatever is copied to the USB stick at that point too. The NNCP page has some more commentary about this kind of setup.

Both are fairly easy to set up, and NNCP is designed to be transport-agnostic, so in this article I’m going to focus on how to integrate ZFS with NNCP.

Operating System

Of course, it should be no surprise that I set this up on Debian.

As an added step, I did all the configuration in Ansible stored in a local git repo. This adds a lot of work, but it means that it is trivial to periodically wipe and reinstall if any security issue is suspected. The git repo can be copied off to another system for storage and takes the system from freshly-installed to ready-to-use state.

Security

There is, of course, nothing preventing you from running NNCP as root. The zfs commands, obviously, need to be run as root. However, from a privilege separation standpoint, I have chosen to run everything relating to NNCP as a nncp user. NNCP already does encryption, but if you prefer to have zero knowledge of the data even to NNCP, it’s trivial to add gpg to the pipeline as well, and in fact I’ll be demonstrating that in a future post for other reasons.

Software

Besides NNCP, there needs to be a system that generates the zfs send streams. For this project, I looked at quite a few. Most were designed to inspect the list of snapshots on a remote end, compare it to a list on the local end, and calculate a difference from there. This, of course, won’t work for this situation.

I realized my own simplesnap project was very close to being able to do this. It already used an algorithm of using specially-named snapshots on the machine being backed up, so never needed any communication about what snapshots were present where. All it needed was a few more options to permit sending to a stream instead of zfs receive. I made those changes and they are available in simplesnap 2.0.0 or above. That version has also been uploaded to sid, and will work fine as-is on buster as well.

Preparing NNCP

I’m going to assume three hosts in this setup:

  • laptop is the machine being backed up. Of course, you may have quite a few of these.
  • spooler holds the backup data until the backup system picks it up
  • backupsvr holds the backups

The basic NNCP workflow documentation covers the basic steps. You’ll need to run nncp-cfgnew on each machine. This generates a basic configuration, along with public and private keys for that machine. You’ll copy the public key sets to the configurations of the other machines as usual. On the laptop, you’ll add a via line like this:

backupsvr: {
  id: ....
  exchpub: ...
  signpub: ...
  noisepub: ...
  via: ["spooler"]

This tells NNCP that data destined for backupsvr should always be sent via spooler first.

You can then arrange for the nncp-daemon to run on the spooler, and nncp-caller or nncp-call on the backupsvr. Or, alternatively, airgapped between the two with nncp-xfer.

Generating Backup Data

Now, on the laptop, install simplesnap (2.0.0 or above). Although you won’t be backing up to the local system, simplesnap still maintains a hostlock in ZFS. Prepate a dataset for it:

zfs create tank/simplesnap
zfs set org.complete.simplesnap:exclude=on tank/simplesnap

Then, create a script /usr/local/bin/runsimplesnap like this:

#!/bin/bash

set -e

simplesnap --store tank/simplesnap --setname backups --local --host `hostname` \
   --receivecmd /usr/local/bin/simplesnap-queue \
   --noreap

su nncp -c '/usr/local/nncp/bin/nncp-toss -noprogress -quiet'

if ip addr | grep -q 192.168.65.64; then
  su nncp -c '/usr/local/nncp/bin/nncp-call -noprogress -quiet -onlinedeadline 1 spooler'
fi

The call to simplesnap sets it up to send the data to simplesnap-queue, which we’ll create in a moment. The –receivmd, plus –noreap, sets it up to run without ZFS on the local system.

The call to nncp-toss will process any previously-received inbound NNCP packets, if there are any. Then, in this example, we do a very basic check to see if we’re on the LAN (checking 192.168.65.64), and if so, will establish a connection to the spooler to transmit the data. If course, you could also do this over the Internet, with tor, or whatever, but in my case, I don’t want to automatically do this in case I’m tethered to mobile. I figure if I want to send backups in that case, I can fire up nncp-call myself. You can also use nncp-caller to set up automated connections on other schedules; there are a lot of options.

Now, here’s what /usr/local/bin/simplesnap-queue looks like:

#!/bin/bash

set -e
set -o pipefail

DEST="`echo $1 | sed 's,^tank/simplesnap/,,'`"

echo "Processing $DEST" >&2
# stdin piped to this
su nncp -c "/usr/local/nncp/bin/nncp-exec -nice B -noprogress backupsvr zfsreceive '$DEST'" >&2
echo "Queued for $DEST" >&2

This is a pretty simple script. simplesnap will call it with a path based on the –store, with the hostname after; so, for instance, tank/simplesnap/laptop/root or some such. This script strips off the leading tank/simplesnap (which is a local fragment), leaving the host and dataset paths. Then it just pipes it to nncp-exec. -nice B classifies it as low-priority bulk data (so if you have some more important interactive data, it would be sent first), then passes it to whatever the backupsvr defines as zfsreceive.

Receiving ZFS backups

In the NNCP configuration on the recipient’s side, in the laptop section, we define what command it’s allowed to run as zfsreceive:

      exec: {
        zfsreceive: ["/usr/bin/sudo", "-H", "/usr/local/bin/nncp-zfs-receive"]
      }

We authorize the nncp user to run this under sudo in /etc/sudoers.d/local–nncp:

Defaults env_keep += "NNCP_SENDER"
nncp ALL=(root) NOPASSWD: /usr/local/bin/nncp-zfs-receive

The NNCP_SENDER is the public key ID of the sending node when nncp-toss processes the incoming data. We can use that for sanity checking later.

Now, here’s a basic nncp-zfs-receive script:

#!/bin/bash
set -e
set -o pipefail

STORE=backups/simplesnap
DEST="$1"

# now process stdin
runcommand zfs receive -o readonly=on -x mountpoint "$STORE/$DEST"

And there you have it — all the basics are in place.

Update 2020-12-30: An earlier version of this article had “zfs receive -F” instead of “zfs receive -o readonly=on -x mountpoint”. These changed arguments are more robust.
Update 2021-01-04: I am now recommending “zfs receive -u -o readonly=on”; see my successor article for more.

Enhancements

You could enhance the nncp-zfs-receive script to improve logging and error handling. For instance:

#!/bin/bash

set -e
set -o pipefail

STORE=backups/simplesnap
# $1 will be the host/dataset

DEST="$1"
HOST="`echo "$1" | sed 's,/.*,,g'`"
if [ -z "$HOST" ]; then
   echo "Malformed command line"
   exit 5
fi

# Log a message
logit () {
   logger -p info -t "`basename "$0"`[$$]" "$1"
}

# Log an error message
logerror () {
   logger -p err -t "`basename "$0"`[$$]" "$1"
}

# Log stdin with the given code.  Used normally to log stderr.
logstdin () {
   logger -p info -t "`basename "$0"`[$$/$1]"
}

# Run command, logging stderr and exit code
runcommand () {
   logit "Running $*"
   if "$@" 2> >(logstdin "$1") ; then
      logit "$1 exited successfully"
      return 0
   else
       RETVAL="$?"
       logerror "$1 exited with error $RETVAL"
       return "$RETVAL"
   fi
}
exiterror () {
   logerror "$1"
   echo "$1" 1>&2
   exit 10
}

# Sanity check

if [ "$HOST" = "laptop" ]; then
  if [ "$NNCP_SENDER" != "12345678" ]; then
    exiterror "Host $HOST doesn't match sender $NNCP_SENDER"
  fi
else
  exiterror "Unknown host $HOST"
fi

runcommand zfs receive -F "$STORE/$DEST"

Now you’ll capture the ZFS receive output in syslog in a friendly way, so you can look back later why things failed if they did.

Further notes on NNCP

nncp-toss will examine the exit code from an invocation. If it is nonzero, it will keep the command (and associated stdin) in the queue and retry it on the next invocation. NNCP does not guarantee order of execution, so it is possible in some cases that ZFS streams may be received in the wrong order. That is fine here; zfs receive will exit with an error, and nncp-toss will just run it again after the dependent snapshots have been received. For non-ZFS backups, a simple sequence number can handle this issue.

30 December, 2020 07:00AM by John Goerzen