03 April, 2025 01:29AM by Junichi Uekawa
Another minor update 0.3.12 for our nanotime
package is now on CRAN. nanotime
relies on the RcppCCTZ
package (as well as the RcppDate
package for additional C++ operations) and offers efficient high(er)
resolution time parsing and formatting up to nanosecond resolution,
using the bit64
package for the actual integer64 arithmetic. Initially
implemented using the S3 system, it has benefitted greatly from a
rigorous refactoring by Leonardo who not only rejigged
nanotime internals in S4 but also added new S4 types for
periods, intervals and durations.
This release responds to recent CRAN nag of also suggesting packages used in demos, makes a small update to the CI script, and improves one version comparison for conditional code.
The NEWS snippet below has the fuller details.
Changes in version 0.3.12 (2025-04-02)
Update continuous integration to use r-ci action with bootstrap
Add ggplot2 to Suggests due to use in demo/ which is now scanned
Refine a version comparison for R 4.5.0 to be greater or equal
Thanks to my CRANberries, there is a diffstat report for this release. More details and examples are at the nanotime page; code, issue tickets etc at the GitHub repository – and all documentation is provided at the nanotime documentation site.
This post by Dirk Eddelbuettel originated on his Thinking inside the box blog. If you like this or other open-source work I do, you can now sponsor me at GitHub.
The SWH work was sponsored. All other work was done on a volunteer basis.
Most of my Debian contributions this month were sponsored by Freexian.
You can also support my work directly via Liberapay.
Changes in dropbear 2025.87 broke OpenSSH’s regression tests. I cherry-picked the fix.
I reviewed and merged patches from Luca
Boccassi to
send and accept the COLORTERM and NO_COLOR environment variables.
Following up on last month, I fixed some more uscan errors:
I upgraded these packages to new upstream versions:
In bookworm-backports, I updated python-django to 3:4.2.19-1.
Although Debian’s upgrade to python-click 8.2.0 was reverted for the time being, I fixed a number of related problems anyway since we’re going to have to deal with it eventually:
dh-python dropped its dependency on python3-setuptools in 6.20250306, which was long overdue, but it had quite a bit of fallout; in most cases this was simply a question of adding build-dependencies on python3-setuptools, but in a few cases there was a missing build-dependency on python3-typing-extensions which had previously been pulled in as a dependency of python3-setuptools. I fixed these bugs resulting from this:
We agreed to remove python-pytest-flake8. In support of this, I removed unnecessary build-dependencies from pytest-pylint, python-proton-core, python-pyzipper, python-tatsu, python-tatsu-lts, and python-tinycss, and filed #1101178 on eccodes-python and #1101179 on rpmlint.
There was a dnspython autopkgtest regression on s390x. I independently tracked that down to a pylsqpack bug and came up with a reduced test case before realizing that Pranav P had already been working on it; we then worked together on it and I uploaded their patch to Debian.
I fixed various other build/test failures:
I enabled more tests in python-moto and contributed a supporting fix upstream.
I sponsored Maximilian Engelhardt to reintroduce zope.sqlalchemy.
I fixed various odds and ends of bugs:
I contributed a small documentation improvement to pybuild-autopkgtest(1).
I upgraded rust-asn1 to 0.20.0.
I finally gave in and joined the Debian Science Team this month, since it often has a lot of overlap with the Python team, and Freexian maintains several packages under it.
I fixed a uscan error in hdf5-blosc (maintained by Freexian), and upgraded it to a new upstream version.
I fixed python-vispy: missing dependency on numpy abi.
I fixed debconf should automatically be noninteractive if input is /dev/null.
I fixed a build failure with GCC 15 in yubihsm-shell (maintained by Freexian).
Prompted by a CI failure in debusine, I submitted a large batch of spelling fixes and some improved static analysis to incus (#1777, #1778) and distrobuilder.
After regaining access to the repository, I fixed telegnome: missing app icon in ‘About’ dialogue and made a new 0.3.7 release.
01 April, 2025 12:17PM by Colin Watson
Another short status update of what happened on my side last month. Some more ModemManager bits landed, Phosh 0.46 is out, haptic feedback is now better tunable plus some more. See below for details (no April 1st joke in there, I promise):
meson dist job and work around meson not applying patches in meson dist (MR, MR)XdgSurface to XdgToplevel to prevent errors like the above (MR)[0.0, 1.0] as ranges for magnitude (MR)_NET_WM_WINDOW_OPACITY (which is around since ages) (MR)This is not code by me but reviews on other peoples code. The list is (as usual) slightly incomplete. Thanks for the contributions!
If you want to support my work see donations.
Join the Fediverse thread
I’ve released qmpbackup 0.46 which now utilizes the image fleecing technique for backup.
Usually, during backup, Qemu will use a so called copy-before-write filter so
that data for new guest writes is sent to the backup target first, the guest
write blocks until this operation is finished.
If the backup target is flaky, or becomes unavailable during backup operation, this could lead to high I/O wait times or even complete VM lockups.
To fix this, a so called “fleecing” image is introduced during backup being used as temporary cache for write operations by the guest. This image can be placed on the same storage as the virtual machine disks, so is independent from the backup target performance.
The documentation on which steps are required to get this going, using the Qemu QMP protocol is, lets say.. lacking..
The following examples show the general functionality, but should be enhanced
to use transactions where possible. All commands are in qmp-shell command
format.
Lets start with a full backup:
# create a new bitmap
block-dirty-bitmap-add node=disk1 name=bitmap persistent=true
# add the fleece image to the virtual machine (same size as original disk required)
blockdev-add driver=qcow2 node-name=fleecie file={"driver":"file","filename":"/tmp/fleece.qcow2"}
# add the backup target file to the virtual machine
blockdev-add driver=qcow2 node-name=backup-target-file file={"driver":"file","filename":"/tmp/backup.qcow2"}
# enable the copy-before-writer for the first disk attached, utilizing the fleece image
blockdev-add driver=copy-before-write node-name=cbw file=disk1 target=fleecie
# "blockdev-replace": make the copy-before-writer filter the major device (use "query-block" to get path parameter value, qdev node)
qom-set path=/machine/unattached/device[20] property=drive value=cbw
# add the snapshot-access filter backing the copy-before-writer
blockdev-add driver=snapshot-access file=cbw node-name=snapshot-backup-source
# create a full backup
blockdev-backup device=snapshot-backup-source target=backup-target-file sync=full job-id=test
[ wait until block job finishes]
# remove the snapshot access filter from the virtual machine
blockdev-del node-name=snapshot-backup-source
# switch back to the regular disk
qom-set path=/machine/unattached/device[20] property=drive value=node-disk1
# remove the copy-before-writer
blockdev-del node-name=cbw
# remove the backup-target-file
blockdev-del node-name=backup-target-file
# detach the fleecing image
blockdev-del node-name=fleecieAfter this process, the temporary fleecing image can be deleted/recreated. Now lets go for a incremental backup:
# add the fleecing and backup target image, like before
blockdev-add driver=qcow2 node-name=fleecie file={"driver":"file","filename":"/tmp/fleece.qcow2"}
blockdev-add driver=qcow2 node-name=backup-target-file file={"driver":"file","filename":"/tmp/backup-incremental.qcow2"}
# add the copy-before-write filter, but utilize the bitmap created during full backup
blockdev-add driver=copy-before-write node-name=cbw file=disk1 target=fleecie bitmap={"node":"disk1","name":"bitmap"}
# switch device to the copy-before-write filter
qom-set path=/machine/unattached/device[20] property=drive value=cbw
# add the snapshot-access filter
blockdev-add driver=snapshot-access file=cbw node-name=snapshot-backup-source
# merge the bitmap created during full backup to the snapshot-access device so
# the backup operation can access it. (you better use an transaction here)
block-dirty-bitmap-add node=snapshot-backup-source name=bitmap
block-dirty-bitmap-merge node=snapshot-backup-source target=bitmap bitmaps=[{"node":"disk1","name":"bitmap"}]
# create incremental backup (you better use an transaction here)
blockdev-backup device=snapshot-backup-source target=backup-target-file job-id=test sync=incremental bitmap=bitmap
[ wait until backup has finished ]
[ cleanup like before ]
# clear the dirty bitmap (you better use an transaction here)
block-dirty-bitmap-clear node=disk1 name=bitmapOr, use a simple reproducer by directly passing qmp commands via stdio:
#!/usr/bin/bash
qemu-img create -f raw disk 1M
qemu-img create -f raw fleece 1M
qemu-img create -f raw backup 1M
qemu-system-x86_64 -drive node-name=disk,file=disk,format=file -qmp stdio -nographic -nodefaults <<EOF
{"execute": "qmp_capabilities"}
{"execute": "block-dirty-bitmap-add", "arguments": {"node": "disk", "name": "bitmap"}}
{"execute": "blockdev-add", "arguments": {"node-name": "fleece", "driver": "file", "filename": "fleece"}}
{"execute": "blockdev-add", "arguments": {"node-name": "backup", "driver": "file", "filename": "backup"}}
{"execute": "blockdev-add", "arguments": {"node-name": "cbw", "driver": "copy-before-write", "file": "disk", "target": "fleece", "bitmap": {"node": "disk", "name": "bitmap"}}}
{"execute": "query-block"}
{"execute": "qom-set", "arguments": {"path": "/machine/unattached/device[4]", "property": "drive", "value": "cbw"}}
{"execute": "blockdev-add", "arguments": {"node-name": "snapshot", "driver": "snapshot-access", "file": "cbw"}}
{"execute": "block-dirty-bitmap-add", "arguments": {"node": "snapshot", "name": "tbitmap"}}
{"execute": "block-dirty-bitmap-merge", "arguments": {"node": "snapshot", "target": "tbitmap", "bitmaps": [{"node": "disk", "name": "bitmap"}]}}
[..]
{"execute": "quit"}
EOF
In July 2024, NASA posted an article titled “NASA Has Pride Across the Universe” featuring a pride flag by Rachel Lense where each color band is made up of images from across NASA. Today is the annual International Transgender Day of Visibility. The original NASA article from last year has since been taken offline. But the heroes from archive.org still carry a copy which I now archived myself together with the other source images. Here is NASA’s pride flag in all its glory:
Southern Fried Science has an article about the flag. The original 4000x2547 TIFF image was stored by archive.org but the PNG version in the same resolution can be downloaded here or by clicking on the image.
Version 0.3.16 of the Rblpapi package arrived on CRAN today. Rblpapi provides a direct interface between R and the Bloomberg Terminal via the C++ API provided by Bloomberg (but note that a valid Bloomberg license and installation is required).
This is the sixteenth release since the package first appeared on CRAN in 2016. It contains several enhancements. Two contributed PRs improve an error message, and extended connection options. We cleaned up a bit of internal code. And this release also makes the build conditional on having a valid build environment. This has been driven by the fact CRAN continues to builder under macOS 13 for x86_64, but Bloomberg no longer supplies a library and headers. And our repeated requests to be able to opt out of the build were, well, roundly ignored. So now the builds will succeed, but on unviable platforms such as that one we will only offer ‘empty’ functions. But no more build ERRORS yelling at us for three configurations.
The detailed list of changes follow below.
Changes in Rblpapi version 0.3.16 (2025-03-31)
A quota error message is now improved (Rodolphe Duge in #400)
Convert remaining
throwintoRcpp::stop(Dirk in #402 fixing #401)Add optional
appIdentityKeyargument toblpConnect(Kai Lin in #404)Rework build as function of Blp library availability (Dirk and John in #406, #409, #410 fixing #407, #408)
Courtesy of my CRANberries, there is also a diffstat report for the this release. As always, more detailed information is at the Rblpapi repo or the Rblpapi page. Questions, comments etc should go to the issue tickets system at the GitHub repo.
This post by Dirk Eddelbuettel originated on his Thinking inside the box blog. If you like this or other open-source work I do, you can sponsor me at GitHub.
A new maintenance release 0.4.24 of RProtoBuf arrived on CRAN today. RProtoBuf provides R with bindings for the Google Protocol Buffers (“ProtoBuf”) data encoding and serialization library used and released by Google, and deployed very widely in numerous projects as a language and operating-system agnostic protocol.
This release brings an both an upstream API update affecting one function, and an update to our use of the C API of R, also in one function. Nothing user-facing, and no surprises expected.
The following section from the NEWS.Rd file has full details.
Changes in RProtoBuf version 0.4.24 (2025-03-31)
Add bindings to EnumValueDescriptor::name (Mike Kruskal in #108)
Replace EXTPTR_PTR with R_ExternalPtrAddr (Dirk)
Thanks to my CRANberries, there is a diff to the previous release. The RProtoBuf page has copies of the (older) package vignette, the ‘quick’ overview vignette, and the pre-print of our JSS paper. Questions, comments etc should go to the GitHub issue tracker off the GitHub repo.
This post by Dirk Eddelbuettel originated on his Thinking inside the box blog. If you like this or other open-source work I do, you can sponsor me at GitHub.
Today I was looking for a way on how to best publish my OpenPGP key on my webserver. Surely, somebody came up with some sort of standard way for where to place that key, right? Turns out, they did: https://datatracker.ietf.org/doc/draft-koch-openpgp-webkey-service/
The TLDR summary is, that my key can now be found here:
https://mister-muffin.de/.well-known/openpgpkey/hu/8yxgr5jjfok88r9um56kb44x9h4dyj7f
Or be downloadable by just running:
$ gpg --locate-key josch@mister-muffin.de
Where does the hash come from? It’s the local part of my email (josch) hashed with sha1 and encoded in z-base32. That computation can be done by gpg:
$ gpg --with-wkd-hash -k josch@mister-muffin.de | grep mister-muffin.de
[...]
8yxgr5jjfok88r9um56kb44x9h4dyj7f@mister-muffin.de
I exported the key that I put there using the following command:
$ gpg --no-options --export --export-options export-minimal,export-clean \
--export-filter keep-uid="uid = Johannes Schauer Marin Rodrigues <josch@mister-muffin.de>" \
F83356BBE112B7462A41552F7D5D8C60CF4D3EB4
There is a handy validator for such setups that can be found here: https://www.webkeydirectory.com
I had an interesting debugging experience when I tried to verify my setup in a fresh Debian chroot because I got this error message when I ran above command:
gpg: directory '/root/.gnupg' created
gpg: keybox '/root/.gnupg/pubring.kbx' created
gpg: /root/.gnupg/trustdb.gpg: trustdb created
gpg: error retrieving 'josch@mister-muffin.de' via WKD: General error
gpg: error reading key: General error
That’s not very descriptive… Turns out, that I was missing the
ca-certificates package. After installing it, everything worked as expected:
$ gpg --locate-key josch@mister-muffin.de
gpg: key 7D5D8C60CF4D3EB4: public key "Johannes Schauer Marin Rodrigues <josch@mister-muffin.de>" imported
gpg: Total number processed: 1
gpg: imported: 1
pub rsa4096 2013-07-04 [SC]
F83356BBE112B7462A41552F7D5D8C60CF4D3EB4
uid [ unknown] Johannes Schauer Marin Rodrigues <josch@mister-muffin.de>
sub rsa4096 2013-07-04 [E]
sub rsa4096 2013-07-04 [S]
sub rsa4096 2023-07-08 [S]
Longnow has an insightful article about religion and magic mushrooms [2].
Interesting story about people working for free at Apple to develop a graphing calculator [6]. We need ways for FOSS people to associate to do such projects.
Interesting YouTube video about a wiki for building a cheap road legal car [7].
Interesting video about powering spacecraft with Plutonion 238 and how they are running out [8].
Interesting information about the search for mh370 [9]. I previously hadn’t been convinced that it was hijacked but I am now.
31 March, 2025 11:23AM by etbe
I rebuilt (the top-50 popcon) Debian and Ubuntu packages, on amd and arm64, and compared the results a couple of months ago. Since then the Reproduce.Debian.net effort has been launched. Unlike my small experiment, that effort is a full-scale rebuild with more architectures. Their goal is to reproduce what is published in the Debian archive.
One differences between these two approaches are the build inputs: The Reproduce Debian effort use the same build inputs which were used to build the published packages. I’m using the latest version of published packages for the rebuild.
What does that difference imply? I believe reproduce.debian.net will be able to reproduce more of the packages in the archive. If you build a C program using one version of GCC you will get some binary output; and if you use a later GCC version you are likely to end up with a different binary output. This is a good thing: we want GCC to evolve and produce better output over time. However it means in order to reproduce the binaries we publish and use, we need to rebuild them using whatever build dependencies were used to prepare those binaries. The conclusion is that we need to use the old GCC to rebuild the program, and this appears to be the Reproduce.Debian.Net approach.
It would be a huge success if the Reproduce.Debian.net effort were to reach 100% reproducibility, and this seems to be within reach.
However I argue that we need go further than that. Being able to rebuild the packages reproducible using older binary packages only begs the question: can we rebuild those older packages? I fear attempting to do so ultimately leads to a need to rebuild 20+ year old packages, with a non-negligible amount of them being illegal to distribute or are unable to build anymore due to bit-rot. We won’t solve the Trusting Trust concern if our rebuild effort assumes some initial binary blob that we can no longer build from source code.
I’ve made an illustration of the effort I’m thinking of, to reach something that is stronger than reproducible rebuilds. I am calling this concept a Idempotent Rebuild, which is an old concept that I believe is the same as John Gilmore has described many years ago.
The illustration shows how the Debian main archive is used as input to rebuild another “stage #0” archive. This stage #0 archive can be compared with diffoscope to the main archive, and all differences are things that would be nice to resolve. The packages in the stage #0 archive is used to prepare a new container image with build tools, and the stage #0 archive is used as input to rebuild another version of itself, called the “stage #1” archive. The differences between stage #0 and stage #1 are also useful to analyse and resolve. This process can be repeated many times. I believe it would be a useful property if this process terminated at some point, where the stage #N archive was identical to the stage #N-1 archive. If this would happen, I label the output archive as an Idempotent Rebuild of the distribution.
How big is N today? The simplest assumption is that it is infinity. Any build timestamp embedded into binary packages will change on every iteration. This will cause the process to never terminate. Fixing embedded timestamps is something that the Reproduce.Debian.Net effort will also run into, and will have to resolve.
What other causes for differences could there be? It is easy to see that generally if some output is not deterministic, such as the sort order of assembler object code in binaries, then the output will be different. Trivial instances of this problem will be caught by the reproduce.debian.net effort as well.
Could there be higher order chains that lead to infinite N? It is easy to imagine the existence of these, but I don’t know how they would look like in practice.
An ideal would be if we could get down to N=1. Is that technically possible? Compare building GCC, it performs an initial stage 0 build using the system compiler to produce a stage 1 intermediate, which is used to build itself again to stage 2. Stage 1 and 2 is compared, and on success (identical binaries), the compilation succeeds. Here N=2. But this is performed using some unknown system compiler that is normally different from the GCC version being built. When rebuilding a binary distribution, you start with the same source versions. So it seems N=1 could be possible.
I’m unhappy to not be able to report any further technical progress now. The next step in this effort is to publish the stage #0 build artifacts in a repository, so they can be used to build stage #1. I already showed that stage #0 was around ~30% reproducible compared to the official binaries, but I didn’t save the artifacts in a reusable repository. Since the official binaries were not built using the latest versions, it is to be expected that the reproducibility number is low. But what happens at stage #1? The percentage should go up: we are now compare the rebuilds with an earlier rebuild, using the same build inputs. I’m eager to see this materialize, and hope to eventually make progress on this. However to build stage #1 I believe I need to rebuild a much larger number packages in stage #0, it could be roughly similar to the “build-essentials-depends” package set.
I believe the ultimate end goal of Idempotent Rebuilds is to be able to re-bootstrap a binary distribution like Debian from some other bootstrappable environment like Guix. In parallel to working on a achieving the 100% Idempotent Rebuild of Debian, we can setup a Guix environment that build Debian packages using Guix binaries. These builds ought to eventually converge to the same Debian binary packages, or there is something deeply problematic happening. This approach to re-bootstrap a binary distribution like Debian seems simpler than rebuilding all binaries going back to the beginning of time for that distribution.
What do you think?
PS. I fear that Debian main may have already went into a state where it is not able to rebuild itself at all anymore: the presence and assumption of non-free firmware and non-Debian signed binaries may have already corrupted the ability for Debian main to rebuild itself. To be able to complete the idempotent and bootstrapped rebuild of Debian, this needs to be worked out.
31 March, 2025 08:21AM by simon
Review: Ghostdrift, by Suzanne Palmer
| Series: | Finder Chronicles #4 |
| Publisher: | DAW |
| Copyright: | May 2024 |
| ISBN: | 0-7564-1888-7 |
| Format: | Kindle |
| Pages: | 378 |
Ghostdrift is a science fiction adventure and the fourth (and possibly final) book of the Finder Chronicles. You should definitely read this series in order and not start here, even though the plot of this book would stand alone.
Following The Scavenger Door, in which he made enemies even more dramatically than he had in the previous books, Fergus Ferguson has retired to the beach on Coralla to become a tea master and take care of his cat. It's a relaxing, idyllic life and a much-needed total reset. Also, he's bored. The arrival of his alien friend Qai, in some kind of trouble and searching for him, is a complex balance between relief and disappointment.
Bas Belos is one of the most notorious pirates of the Barrens. He has someone he wants Fergus to find: his twin sister, who disappeared ten years ago. Fergus has an unmatched reputation for finding things, so Belos kidnapped Qai's partner to coerce her into finding Fergus. It's not an auspicious beginning to a relationship, and Qai was ready to fight once they got her partner back, but Belos makes Fergus an offer of payment that, startlingly, is enough for him to take the job mostly voluntarily.
Ghostdrift feels a bit like a return to Finder. Fergus is once again alone among strangers, on an assignment that he's mostly not discussing with others, piecing together clues and navigating tricky social dynamics. I missed his friends, particularly Ignatio, and while there are a few moments with AI ships, they play less of a role.
But Fergus is so very good at what he does, and Palmer is so very good at writing it. This continues to be competence porn at its best. Belos's crew thinks Fergus is a pirate recruited from a prison colony, and he quietly sets out to win their trust with a careful balance of self-deprecation and unflappable skill, helped considerably by the hidden gift he acquired in Finder. The character development is subtle, but this feels like a Fergus who understands friendship and other people at a deeper and more satisfying level than the Fergus we first met three books ago.
Palmer has a real talent for supporting characters and Ghostdrift is no exception. Belos's crew are criminals and murderers, and Palmer does remind the reader of that occasionally, but they're also humans with complex goals and relationships. Belos has earned their loyalty by being loyal and competent in a rough world where those attributes are rare. The morality of this story reminds me of infiltrating a gang: the existence of the gang is not a good thing, and the things they do are often indefensible, but they are an understandable reaction to a corrupt social system. The cops (in this case, the Alliance) are nearly as bad, as we've learned over the past couple of books, and considerably more insufferable. Fergus balances the ethical complexity in a way that I found satisfyingly nuanced, while quietly insisting on his own moral lines.
There is a deep science fiction plot here, possibly the most complex of the series so far. The disappearance of Belos's sister is the tip of an iceberg that leads to novel astrophysics, dangerous aliens, mysterious ruins, and an extended period on a remote and wreck-strewn planet. I groaned a bit when the characters ended up on the planet, since treks across primitive alien terrain with jury-rigged technology are one of my least favorite science fiction tropes, but I need not have worried. Palmer knows what she's doing; the pace of the plot does slow a bit at first, but it quickly picks up again, adding enough new setting and plot complications that I never had a chance to be bored by alien plants. It helps that we get another batch of excellent supporting characters for Fergus to observe and win over.
This series is such great science fiction. Each book becomes my new favorite, and Ghostdrift is no exception. The skeleton of its plot is a satisfying science fiction mystery with multiple competing factions, hints of fascinating galactic politics, complicated technological puzzles, and a sense of wonder that reminds me of reading Larry Niven's Known Space series. But the characters are so much better and more memorable than classic SF; compared to Fergus, Niven's Louis Wu barely exists and is readily forgotten as soon as the story is over. Fergus starts as a quiet problem-solver, but so much character depth unfolds over the course of this series. The ending of this book was delightfully consistent with everything we've learned about Fergus, but also the sort of ending that it's hard to imagine the Fergus from Finder knowing how to want.
Ghostdrift, like each of the books in this series, reaches a satisfying stand-alone conclusion, but there is no reason within the story for this to be the last of the series. The author's acknowledgments, however, says that this the end. I admit to being disappointed, since I want to read more about Fergus and there are numerous loose ends that could be explored. More importantly, though, I hope Palmer will write more novels in any universe of her choosing so that I can buy and read them.
This is fantastic stuff. This review comes too late for the Hugo nominating deadline, but I hope Palmer gets a Best Series nomination for the Finder Chronicles as a whole. She deserves it.
Rating: 9 out of 10
Berge Schwebs Bjørlo, aged 40, died on March 4th in an avalanche together with his friend Ulf, while on winter holiday.
When writing about someone who recently died, it is common to make lists. Lists of education, of where they worked, on projects they did.
But Berge wasn't common. Berge was an outlier. A paradox, even.
Berge was one of my closest friends; someone who always listened, someone you could always argue with (“I'm a pacifist, but I'm aware that this is an extreme position”) but could rarely be angry at. But if you ask around, you'll see many who say similar things; how could someone be so close to so many at the same time?
Berge had running jokes going on 20 years or more. Many of them would be related to his background from Bergen; he'd often talk about “the un-central east” (aka Oslo), yet had to admit at some point that actually started liking the city. Or about his innate positivity (“I'm in on everything but suicide and marriage!”). I know a lot of people have described his humor as dry, but I found him anything but. Just a free flow of living.
He lived his life in free software, but rarely in actually writing code; I don't think I've seen a patch from him, and only the occasional bug report. Instead, he would spend his time guiding others; he spent a lot of time in PostgreSQL circles, helping people with installation or writing queries or chiding them for using an ORM (“I don't understand why people love to make life so hard for themselves”) or just discussing life, love and everything. Somehow, some people's legacy is just the number of others they touched, and Berge touched everyone he met. Kindness is not something we do well in the free software community, but somehow, it came natural to him. I didn't understand until after he died why he was so chronically bad at reading backlog and hard to get hold of; he was interacting with so many people, always in the present and never caring much about the past.
I remember that Berge once visited my parents' house, and was greeted by our dog, who after a pat promptly went back to relaxing lazily on the floor. “Awh! If I were a dog, that's the kind of dog I'd be.” In retrospect, for someone who lived a lot of his life in 300 km/h (at times quite literally), it was an odd thing to say, but it was just one of those paradoxes.
Berge loved music. He'd argue for intensely political punk, but would really consume everything with great enthuisasm and interest. One of the last albums I know he listened to was Thomas Dybdahl's “… that great October sound”:
Tear us in different ways but leave a thread throughout the maze
In case I need to find my way back home
All these decisions make for people living without faith
Fumbling in the dark nowhere to roam
Dreamweaver
I'll be needing you tomorrow and for days to come
Cause I'm no daydreamer
But I'll need a place to go if memory fails me & let you slip away
Berge wasn't found by a lazy dog. He was found by Shane, a very good dog.
Somehow, I think he would have approved of that, too.
Version 0.0.21 of RcppSpdlog arrived on CRAN today and has been uploaded to Debian. RcppSpdlog bundles spdlog, a wonderful header-only C++ logging library with all the bells and whistles you would want that was written by Gabi Melman, and also includes fmt by Victor Zverovich. You can learn more at the nice package documention site.
This release updates the code to the version 1.15.2 of spdlog which was released this weekend as well.
The NEWS entry for this release follows.
Changes in RcppSpdlog version 0.0.21 (2025-03-30)
- Upgraded to upstream release spdlog 1.15.2 (including fmt 11.1.4)
Courtesy of my CRANberries, there is also a diffstat report. More detailed information is on the RcppSpdlog page, or the package documention site.
This post by Dirk Eddelbuettel originated on his Thinking inside the box blog. If you like this or other open-source work I do, you can sponsor me at GitHub.
A new release 0.1.8 of RcppZiggurat is now on the CRAN network for R, following up on the 0.1.7 release last week which was the first release in four and a half years.
The RcppZiggurat package updates the code for the Ziggurat generator by Marsaglia and others which provides very fast draws from a Normal (or Exponential) distribution. The package provides a simple C++ wrapper class for the generator improving on the very basic macros, and permits comparison among several existing Ziggurat implementations. This can be seen in the figure where Ziggurat from this package dominates accessing the implementations from the GSL, QuantLib and Gretl—all of which are still way faster than the default Normal generator in R (which is of course of higher code complexity).
This release switches the vignette to the standard trick of premaking
it as a pdf and including it in a short Sweave document that imports it
via pdfpages; this minimizes build-time dependencies on
other TeXLive components. It also incorporates a change contributed by
Tomas to rely on the system build of the GSL on Windows as well if
Rtools 42 or later is found. No other changes.
The NEWS file entry below lists all changes.
Changes in version 0.1.8 (2025-03-30)
The vignette is now premade and rendered as Rnw via pdfpage to minimize the need for TeXLive package at build / install time (Dirk)
Windows builds now use the GNU GSL when Rtools is 42 or later (Tomas Kalibera in #25)
Courtesy of my CRANberries, there is a diffstat report relative to previous release. More detailed information is on the Rcppziggurat page or the GitHub repository.
This post by Dirk Eddelbuettel originated on his Thinking inside the box blog. If you like this or other open-source work I do, you can sponsor me at GitHub.
Review: Cascade Failure, by L.M. Sagas
| Series: | Ambit's Run #1 |
| Publisher: | Tor |
| Copyright: | 2024 |
| ISBN: | 1-250-87126-3 |
| Format: | Kindle |
| Pages: | 407 |
Cascade Failure is a far-future science fiction adventure with a small helping of cyberpunk vibes. It is the first of a (so far) two-book series, and was the author's first novel.
The Ambit is an old and small Guild ship, not much to look at, but it holds a couple of surprises. One is its captain, Eoan, who is an AI with a deep and insatiable curiosity that has driven them and their ship farther and farther out into the Spiral. The other is its surprisingly competent crew: a battle-scarred veteran named Saint who handles the fighting, and a talented engineer named Nash who does literally everything else. The novel opens with them taking on supplies at Aron Outpost. A supposed Guild deserter named Jalsen wanders into the ship looking for work.
An AI ship with a found-family crew is normally my catnip, so I wanted to love this book. Alas, I did not.
There were parts I liked. Nash is great: snarky, competent, and direct. Eoan is a bit distant and slightly more simplistic of a character than I was expecting, but I appreciated the way Sagas put them firmly in charge of the ship and departed from the conventional AI character presentation. Once the plot starts in earnest (more on that in a moment), we meet Anke, the computer hacker, whose charming anxiety reaction is a complete inability to stop talking and who adds some needed depth to the character interactions. There's plenty of action, a plot that makes at least some sense, and a few moments that almost achieved the emotional payoff the author was attempting.
Unfortunately, most of the story focuses on Saint and Jal, and both of them are irritatingly dense cliches.
The moment Jal wanders onto the Ambit in the first chapter, the reader is informed that Jal, Saint, and Eoan have a history. The crew of the Ambit spent a year looking for Jal and aren't letting go of him now that they've found him. Jal, on the other hand, clearly blames Saint for something and is not inclined to trust him. Okay, fine, a bit generic of a setup but the writing moved right along and I was curious enough.
It then takes a full 180 pages before the reader finds out what the hell is going on with Saint and Jal. Predictably, it's a stupid misunderstanding that could have been cleared up with one conversation in the second chapter.
Cascade Failure does not contain a romance (and to the extent that it hints at one, it's a sapphic romance), but I swear Saint and Jal are both the male protagonist from a certain type of stereotypical heterosexual romance novel. They're both the brooding man with the past, who is too hurt to trust anyone and assumes the worst because he's unable to use his words or ask an open question and then listen to the answer. The first half of this book is them being sullen at each other at great length while both of them feel miserable. Jal keeps doing weird and suspicious things to resolve a problem that would have been far more easily resolved by the rest of the crew if he would offer any explanation at all. It's not even suspenseful; we've read about this character enough times to know that he'll turn out to have a heart of gold and everything will be a misunderstanding. I found it tedious. Maybe people who like slow burn romances with this character type will have a less negative reaction.
The real plot starts at about the time Saint and Jal finally get their shit sorted out. It turns out to have almost nothing to do with either of them. The environmental control systems of worlds are suddenly failing (hence the book title), and Anke, the late-arriving computer programmer and terraforming specialist, has a rather wild theory about what's happening. This leads to a lot of action, some decent twists, and a plot that felt very cyberpunk to me, although unfortunately it culminates in an absurdly-cliched action climax.
This book is an action movie that desperately wants to make you feel all the feels, and it worked about as well as that typically works in action movies for me. Jaded cynicism and an inability to communicate are not the ways to get me to have an emotional reaction to a book, and Jal (once he finally starts talking) is so ridiculously earnest that it's like reading the adventures of a Labrador puppy. There was enough going on that it kept me reading, but not enough for the story to feel satisfying. I needed a twist, some depth, way more Nash and Anke and way less of the men, something.
Everyone is going to compare this book to Firefly, but Firefly had better banter, created more complex character interactions due to the larger and more varied crew, and played the cynical mercenary for laughs instead of straight, all of which suited me better. This is not a bad book, particularly once it gets past the halfway point, but it's not that memorable either, at least for me. If you're looking for a space adventure with heavy action hero and military SF vibes that wants to be about Big Feelings but gets there in mostly obvious ways, you could do worse. If you're looking for a found-family starship crew story more like Becky Chambers, I think you'll find this one a bit too shallow and obvious.
Not really recommended, although there's nothing that wrong with it and I'm sure other people's experience will differ.
Followed by Gravity Lost, which I'm unlikely to read.
Rating: 6 out of 10
Following the 1.2.0beta1 release two weeks ago, a final 1.2.0 release of theora was wrapped up today. This new release is tagged in the Xiph gitlab theora instance and you can fetch it from the Theora home page as soon as someone with access find time to update the web pages. In the mean time (automatically removed after 14 days) the release tarball is also available as a git build artifact from CI build of the release tag.
The list of changes since The 1.2.0beta release from the CHANGES file in the tarball look like this:
libtheora 1.2.0 (2025 March 29)
- Bumped minor SONAME versions as oc_comment_unpack() implementation changed.
- Added example wrapper script encoder_example_ffmpeg (#1601 #2336).
- Improve comment handling on platforms where malloc(0) return NULL (#2304).
- Added pragma in example code to quiet clang op precedenca warnings.
- Adjusted encoder_example help text.
- Adjusted README, CHANGES, pkg-config and spec files to better reflect current release (#2331 #2328).
- Corrected english typos in source and build system.
- Switched http links to https in doc and comments where relevant. Did not touch RFC drafts.
As usual, if you use Bitcoin and want to show your support of my activities, please send Bitcoin donations to my address 15oWEoG9dUPovwmUL9KWAnYRtNJEkP1u1b.
The diffoscope maintainers are pleased to announce the release of diffoscope
version 293. This version includes the following changes:
[ Chris Lamb ]
* Correct import masking issue.
You find out more by visiting the project homepage.
The diffoscope maintainers are pleased to announce the release of diffoscope
version 292. This version includes the following changes:
[ Ivan Trubach ]
* Ignore st_size entry for directories to avoid spurious diffs as this value
is essentially filesystem dependent.
[ Chris Lamb ]
* Update copyright years.
You find out more by visiting the project homepage.
Rust, and resistance to it in some parts of the Linux community, has been in my feed recently. One undercurrent seems to be the notion that Rust is woke (and should therefore be rejected as part of culture wars).
I’m going to argue that Rust, the language, is woke. So the opponents are right, in that sense. Of course, as ever, dissing something for being woke is nasty and fascist-adjacent.
The obvious way that Rust may seem woke is that it has the trappings, and many of the attitudes and outcomes, of a modern, nice, FLOSS community. Rust certainly does better than toxic environments like the Linux kernel, or Debian. This is reflected in a higher proportion of contributors from various kinds of minoritised groups. But Rust is not outstanding in this respect. It certainly has its problems. Many other projects do as well or better.
And this is well-trodden ground. I have something more interesting to say:
Rust is woke technology that embodies a woke understanding of what it means to be a programming language.
Let’s start with Rust’s strapline:
A language empowering everyone to build reliable and efficient software.
Surprisingly, this motto is not mere marketing puff. For Rustaceans, it is a key goal which strongly influences day-to-day decisions (big and small).
Empowering everyone is a key aspect of this, which aligns with my own personal values. In the Rust community, we care about empowerment. We are trying to help liberate our users. And we want to empower everyone because everyone is entitled to technological autonomy. (For a programming language, empowering individuals means empowering their communities, of course.)
This is all very airy-fairy, but it has concrete consequences:
In Rust we consider it a key part of our job to help the programmer avoid mistakes; to limit the consequences of mistakes; and to guide programmers in useful directions.
If you write a bug in your Rust program, Rust doesn’t blame you. Rust asks “how could the compiler have spotted that bug”.
This is in sharp contrast to C (and C++). C nowadays is an insanely hostile programming environment. A C compiler relentlessly scours your program for any place where you may have violated C’s almost incomprehensible rules, so that it can compile your apparently-correct program into a buggy executable. And then the bug is considered your fault.
These aren’t just attitudes implicitly embodied in the software. They are concrete opinions expressed by compiler authors, and also by language proponents. In other words:
Rust sees programmers writing bugs as a systemic problem, which must be addressed by improvements to the environment and the system. The toxic parts of the C and C++ community see bugs as moral failings by individual programmers.
Sound familiar?
Programming has long suffered from the myth of the “rockstar”. Silicon Valley techbro culture loves this notion.
In reality, though, modern information systems are far too complicated for a single person. Developing systems is a team sport. Nontechnical, and technical-adjacent, skills are vital: clear but friendly communication; obtaining and incorporating the insights of every member of your team; willingness to be challenged. Community building. Collaboration. Governance.
The hardcore C community embraces the rockstar myth: they imagine that a few super-programmers (or super-reviewers) are able to spot bugs, just by being so brilliant. Of course this doesn’t actually work at all, as we can see from the atrocious bugfest that is the Linux kernel.
These “rockstars” want us to believe that there is a steep hierarchy in programmming; that they are at the top of this hierarchy; and that being nice isn’t important.
Sound familiar?
Much of the modern crisis of software reliability arises from memory-unsafe programming languages, mostly C and C++.
Addressing this is a big job, requiring many changes. This threatens powerful interests; notably, corporations who want to keep shipping junk. (See also, conniptions over the EU Product Liability Directive.)
The harms of this serious problem mostly fall on society at large, but the convenience of carrying on as before benefits existing powerful interests.
Sound familiar?
Addressing this problem via Rust is a direct threat to the power of established C programmers such as gatekeepers in the Linux kernel. Supplanting C means they will have to learn new things, and jostle for status against better Rustaceans, or be replaced. More broadly, Rust shows that it is practical to write fast, reliable, software, and that this does not need (mythical) “rockstars”.
So established C programmer “experts” are existing vested interests, whose power is undermined by (this approach to) tackling this serious problem.
Sound familiar?
I’m not saying we should rewrite all the world’s C in Rust. We should not try to do that.
Rust is often a good choice for new code, or when a rewrite or substantial overhaul is needed anyway. But we’re going to need other techniques to deal with all of our existing C. CHERI is a very promising approach. Sandboxing, emulation and automatic translation are other possibilities. The problem is a big one and we need a toolkit, not a magic bullet.
But as for Linux: it is a scandal that substantial new drivers and subsystems are still being written in C. We could have been using Rust for new code throughout Linux years ago, and avoided very many bugs. Those bugs are doing real harm. This is not OK.
I first learned C from K&R I in 1989. I spent the first three decades of my life as a working programmer writing lots and lots of C. I’ve written C++ too. I used to consider myself an expert C programmer, but nowadays my C is a bit rusty and out of date. Why is my C rusty? Because I found Rust, and immediately liked and adopted it (despite its many faults).
I like Rust because I care that the software I write actually works: I care that my code doesn’t do harm in the world.
The original meaning of “woke” is something much more specific, to do with racism. For the avoidance of doubt, I don’t think Rust is particularly antiracist.
I’m using “woke” (like Rust’s opponents are) in the much broader, and now much more prevalent, culture wars sense.
If you’re a senior developer who knows only C/C++, doesn’t want their authority challenged, and doesn’t want to have to learn how to write better software, you should hate Rust.
Also you should be fired.
As I write this in March 2025, there is a lot of confusion about Signal messenger due to the recent news of people using Signal in government, and subsequent leaks.
The short version is: there was no problem with Signal here. People were using it because they understood it to be secure, not the other way around.
Both the government and the Electronic Frontier Foundation recommend people use Signal. This is an unusual alliance, and in the case of the government, was prompted because it understood other countries had a persistent attack against American telephone companies and SMS traffic.
So let’s dive in. I’ll cover some basics of what security is, what happened in this situation, and why Signal is a good idea.
This post isn’t for programmers that work with cryptography every day. Rather, I hope it can make some of these concepts accessible to everyone else.
When most people are talking about secure communications, they mean some combination of these properties:
If you think about it, most people care the most about the first two. In fact, authentication is a key part of privacy. There is an attack known as man in the middle in which somebody pretends to be the intended recipient. The interceptor reads the messages, and then passes them on to the real intended recipient. So we can’t really have privacy without authentication.
I’ll have more to say about these later. For now, let’s discuss attack scenarios.
There are a number of ways that security can be compromised. Let’s think through some of them:
Let’s say you used no encryption at all, and connected to public WiFi in a coffee shop to send your message. Who all could potentially see it?
Back in the early days of the Internet, most traffic had no encryption. People were careful about putting their credit cards into webpages and emails because they knew it was easy to intercept them. We have been on a decades-long evolution towards more pervasive encryption, which is a good thing.
Text messages (SMS) follow a similar path to the above scenario, and are unencrypted. We know that all of the above are ways people’s texts can be compromised; for instance, governments can issue search warrants to obtain copies of texts, and China is believed to have a persistent hack into western telcos. SMS fails all four of our attributes of secure communication above (privacy, authentication, ephemerality, and anonymity).
Also, think about what information is collected from SMS and by who. Texts you send could be retained in your phone, the recipient’s phone, your phone company, their phone company, and so forth. They might also live in cloud backups of your devices. You only have control over your own phone’s retention.
So defenses against this involve things like:
You may see some other apps saying they use strong encryption or use the Signal protocol. But while they may do that for some or all of your message content, they may still upload your contact list, history, location, etc. to a central location where it is still vulnerable to these kinds of attacks.
When you think about anonymity, think about it like this: if you send a letter to a friend every week, every postal carrier that transports it – even if they never open it or attempt to peak inside – will be able to read the envelope and know that you communicate on a certain schedule with that friend. The same can be said of SMS, email, or most encrypted chat operators. Signal’s design prevents it from retaining even this information, though nation-states or ISPs might still be able to notice patterns (every time you send something via Signal, your contact receives something from Signal a few milliseconds later). It is very difficult to provide perfect anonymity from well-funded adversaries, even if you can provide very good privacy.
Let’s say you use an app with strong end-to-end encryption. This takes away some of the easiest ways someone could get to your messages. But it doesn’t take away all of them.
What if somebody stole your phone? Perhaps the phone has a password, but if an attacker pulled out the storage unit, could they access your messages without a password? Or maybe they somehow trick or compel you into revealing your password. Now what?
An even simpler attack doesn’t require them to steal your device at all. All they need is a few minutes with it to steal your SIM card. Now they can receive any texts sent to your number - whether from your bank or your friend. Yikes, right?
Signal stores your data in an encrypted form on your device. It can protect it in various ways. One of the most important protections is ephemerality - it can automatically delete your old texts. A text that is securely erased can never fall into the wrong hands if the device is compromised later.
An actively-compromised phone, though, could still give up secrets. For instance, what if a malicious keyboard app sent every keypress to an adversary? Signal is only as secure as the phone it runs on – but still, it protects against a wide variety of attacks.
Perhaps you are sending sensitive information to a contact, but that person doesn’t want to keep it in confidence. There is very little you can do about that technologically; with pretty much any tool out there, nothing stops them from taking a picture of your messages and handing the picture off.
Perhaps your device is secure, but a hidden camera still captures what’s on your screen. You can take some steps against things like this, of course.
Sometimes humans make mistakes. For instance, the reason a reporter got copies of messages recently was because a participant in a group chat accidentally added him (presumably that participant meant to add someone else and just selected the wrong name). Phishing attacks can trick people into revealing passwords or other sensitive data. Humans are, quite often, the weakest link in the chain.
So how can you protect yourself against these attacks? Let’s consider:
There are other methods besides Signal. For instance, you could install GnuPG (GPG) on a laptop that has no WiFi card or any other way to connect it to the Internet. You could always type your messages on that laptop, encrypt them, copy the encrypted text to a floppy disk (or USB device), take that USB drive to your Internet computer, and send the encrypted message by email or something. It would be exceptionally difficult to break the privacy of messages in that case (though anonymity would be mostly lost). Even if someone got the password to your “secure” laptop, it wouldn’t do them any good unless they physically broke into your house or something. In some ways, it is probably safer than Signal. (For more on this, see my article How gapped is your air?)
But, that approach is hard to use. Many people aren’t familiar with GnuPG. You don’t have the convenience of sending a quick text message from anywhere. Security that is hard to use most often simply isn’t used. That is, you and your friends will probably just revert back to using insecure SMS instead of this GnuPG approach because SMS is so much easier.
Signal strikes a unique balance of providing very good security while also being practical, easy, and useful. For most people, it is the most secure option available.
Signal is also open source; you don’t have to trust that it is as secure as it says, because you can inspect it for yourself. Also, while it’s not federated, I previously addressed that.
If you are a government, particularly one that is highly consequential to the world, you can imagine that you are a huge target. Other nations are likely spending billions of dollars to compromise your communications. Signal itself might be secure, but if some other government can add spyware to your phones, or conduct a successful phishing attack, you can still have your communications compromised.
I have no direct knowledge, but I think it is generally understood that the US government maintains communications networks that are entirely separate from the Internet and can only be accessed from secure physical locations and secure rooms. These can be even more secure than the average person using Signal because they can protect against things like environmental compromise, human error, and so forth. The scandal in March of 2025 happened because government employees were using Signal rather than official government tools for sensitive information, had taken advantage of Signal’s ephemerality (laws require records to be kept), and through apparent human error had directly shared this information with a reporter. Presumably a reporter would have lacked access to the restricted communications networks in the first place, so that wouldn’t have been possible.
This doesn’t mean that Signal is bad. It just means that somebody that can spend billions of dollars on security can be more secure than you. Signal is still a great tool for people, and in many cases defeats even those that can spend lots of dollars trying to defeat it.
And remember - to use those restricted networks, you have to go to specific rooms in specific buildings. They are still not as convenient as what you carry around in your pocket.
Signal is practical security. Do you want phone companies reading your messages? How about Facebook or X? Have those companies demonstrated that they are completely trustworthy throughout their entire history?
I say no. So, go install Signal. It’s the best, most practical tool we have.
This post is also available on my website, where it may be periodically updated.
28 March, 2025 02:51AM by John Goerzen
Like each month, have a look at the work funded by Freexian’s Debian LTS offering.
In February, 18 contributors have been paid to work on Debian LTS, their reports are available:
In February, we have released 38 DLAs.
LTS contributors Emilio Pozuelo Monfort and Santiago Ruano Rincón coordinated the administrative aspects of LTS updates of postgresql-13 and pam-u2f, which were prepared by the respective maintainers, to whom we are most grateful.
As has become the custom of the LTS team, work is under way on a number of package updates targeting Debian 12 (codename “bookworm”) with fixes for a variety of vulnerabilities. In February, Guilhem Moulin prepared an upload of sssd, while several other updates are still in progress. Bastien Roucariès prepared an upload of krb5 for unstable as well.
Given the importance of the Debian Security Tracker to the work of the LTS Team, we regularly contribute improvements to it. LTS contributor Emilio Pozuelo Monfort reviewed and merged a change to improve performance, and then dealt with unexpected issues that arose as a result. He also made improvements in the processing of CVEs which are not applicable to Debian.
Looking to the future (the release of Debian 13, codename “trixie”, and beyond), LTS contributor Santiago Ruano Rincón has initiated a conversation among the broader community involved in the development of Debian. The purpose of the discussion is to explore ways to improve the long term supportability of packages in Debian, specifically by focusing effort on ensuring that each Debian release contains the “best” supported upstream version of packages with a history of security issues.
Sponsors that joined recently are in bold.
28 March, 2025 12:00AM by Roberto C. Sánchez
Help us Beta test Kubuntu Plucky Puffin!
Kubuntu work:
Fixed an issue in apparmor preventing QT6 webengine applications from starting.
Beta testing!
KDE Snaps:
Updated Qt6 to 6.8.2
Updated Kf6 6.11.0
Rolling out 25.04 RC applications! You can find them in the –candidate channel!
Life:
I have decided to strike out on my own. I can’t take any more rejections! Honestly, I don’t blame them, I wouldn’t want a one armed engineer either. However, I have persevered and accomplished quite a bit with my one arm! So I have decided to take a leap of faith and with your support for open source work and a resurrected side gig of web development I will survive. If you can help sponsor my work, anything at all, even a dollar! I would be eternally grateful. I have several methods to do so:
If you want your cool application packaged in a variety of formats please contact me!
If you want focused help with an annoying bug, please contact me!
Contact me for any and all kinds of help, if I can’t do it, I will say so.
Do you need web work? Someone to maintain your website? I can do that too!
Thank you all for your support in this new adventure!
27 March, 2025 07:20PM by sgmoore
We are pleased to announce that Viridien has committed to sponsor DebConf25 as a Platinum Sponsor.
Viridien is an advanced technology, digital and Earth data company that pushes the boundaries of science for a more prosperous and sustainable future.
Viridien has been using Debian-based systems to power most of its HPC infrastructure and its cloud platform since 2009 and currently employs two active Debian Project Members.
As a Platinum Sponsor, Viridien is contributing to the Debian annual Developers' conference, directly supporting the progress of Debian and Free Software. Viridien contributes to strengthen the community that collaborates on the Debian project from all around the world throughout all of the year.
Thank you very much, Viridien, for your support of DebConf25!
DebConf25 will take place from 14 to 20 July 2025 in Brest, France, and will be preceded by DebCamp, from 7 to 13 July 2025.
DebConf25 is accepting sponsors! Interested companies and organizations may contact the DebConf team through sponsors@debconf.org, and visit the DebConf25 website at https://debconf25.debconf.org/sponsors /become-a-sponsor/.
27 March, 2025 10:50AM by Sahil Dhiman
Are you a student aspiring to participate in the Google Summer of Code 2025? Would you like to improve the continuous integration pipeline used at salsa.debian.org, the Debian GitLab instance, to help improve the quality of tens of thousands of software packages in Debian?
This summer 2025, I and Emmanuel Arias will be participating as mentors in the GSoC program. We are available to mentor students who propose and develop improvements to the Salsa CI pipeline, as we are members of the Debian team that maintains it.
A post by Santiago Ruano Rincón in the GitLab blog explains what Salsa CI is and its short history since inception in 2018. At the time of the article in fall 2023 there were 9000+ source packages in Debian using Salsa CI. Now in 2025 there are over 27,000 source packages in Debian using it, and since summer 2024 some Ubuntu developers have started using it for enhanced quality assurance of packaging changes before uploading new package revisions to Ubuntu. Personally, I have been using Salsa CI since its inception, and contributing as a team member since 2019. See my blog post about GitLab CI for MariaDB in Debian for a description of an advanced and extensive use case.
Helping Salsa CI is a great way to make a global impact, as it will help avoid regressions and improve the quality of Debian packages. The benefits reach far beyond just Debian, as it will also help hundreds of Debian derivatives, such as Ubuntu, Linux Mint, Tails, Purism PureOS, Pop!_OS, Zorin OS, Raspberry Pi OS, a large portion of Docker containers, and even the Windows Subsystem for Linux.
While Salsa CI with contributions from 71 people is already quite mature and capable, there are many ideas floating around about how it could be further extended. For example, Salsa CI issue #147 describes various static analyzers and linters that may be generally useful. Issue #411 proposes using libfaketime to run autopkgtest on arbitrary future dates to test for failures caused by date assumptions, such as the Y2038 issue.
There are also ideas about making Salsa CI more robust and code easier to reuse by refactoring some of the yaml scripts into independent scripts in #230, which could make it easier to run Salsa CI locally as suggested in #169. There are also ideas about improving the Salsa CI’s own CI to avoid regressions from pipeline changes in #318.
The CI system is also better when it’s faster, and some speed improvement ideas have been noted in #412.
Improvements don’t have to be limited to changes in the pipeline itself. A useful project would also be to update more Debian packages to use Salsa CI, and ensure they adopt it in an optimal way as noted in #416. It would also be nice to have a dashboard with statistics about all public Salsa CI pipeline runs as suggested in #413.
These and more ideas can be found in the issue list by filtering for tags Newcomer, Nice-To-Have or Accepting MRs. A Google Summer of Code proposal does not have to be limited to these existing ideas. Participants are also welcome to propose completely novel ideas!
Anyone working with Debian team should also take the opportunity to learn Debian packaging, and contribute to the packaging or maintenance of 1-2 packages in parallel to improving the Salsa CI. All Salsa CI team members are also Debian Developers who can mentor and sponsor uploads to Debian.
Maintaining a few packages is a great way to eat your own cooking and experience Salsa CI from the user perspective, and likely to make you better at Salsa CI development.
The contributor applications opened yesterday on March 24, so to participate act now! If you are an eligible student and want to attend, head over to summerofcode.withgoogle.com to learn more.
There are over a thousand participating organizations, with Debian, GitLab and MariaDB being some examples. Within these organizations there may be multiple subteams and projects to choose from. The full list of participating Debian projects can be found in the Debian wiki.
If you are interested in GSoC for Salsa CI specifically, feel free to
Note that you don’t have to wait for GSoC to officially start to contribute. In fact, it may be useful to start immediately by submitting a Merge Request to do some small contribution, just to learn the process and to get more familiar with how everything works, and the team maintaining Salsa CI. Looking forward to seeing new contributors!
I am told, by friends who have spent time at Google, about the reason Google Reader finally disappeared. Apparently it had become a 20% Project for those who still cared about it internally, and there was some major change happening to one of it upstream dependencies that was either going to cause a significant amount of work rearchitecting Reader to cope, or create additional ongoing maintenance burden. It was no longer viable to support it as a side project, so it had to go. This was a consequence of an internal culture at Google where service owners are able to make changes that can break downstream users, and the downstream users are the ones who have to adapt.
My experience at Meta goes the other way. If you own a service or other dependency and you want to make a change that will break things for the users, it’s on you to do the migration, or at the very least provide significant assistance to those who own the code. You don’t just get to drop your new release and expect others to clean up; doing that tends to lead to changes being reverted. The culture flows the other way; if you break it, you fix it (nothing is someone else’s problem).
There are pluses and minuses to both approaches. Users having to drive the changes to things they own stops them from blocking progress. Service/code owners having to drive the changes avoids the situation where a wildly used component drops a new release that causes a lot of high priority work for folk in order to adapt.
I started thinking about this in the context of Debian a while back, and a few incidents since have resulted in my feeling that we’re closer to the Google model than the Meta model. Anyone can upload a new version of their package to unstable, and that might end up breaking all the users of it. It’s not quite as extreme as rolling out a new service, because it’s unstable that gets affected (the clue is in the name, I really wish more people would realise that), but it can still result in release critical bugs for lots other Debian contributors.
A good example of this are toolchain changes. Major updates to GCC and friends regularly result in FTBFS issues in lots of packages. Now in this instance the maintainer is usually diligent about a heads up before the default changes, but it’s still a whole bunch of work for other maintainers to adapt (see the list of FTBFS bugs for GCC 15 for instance - these are important, but not serious yet). Worse is when a dependency changes and hasn’t managed to catch everyone who might be affected, so by the time it’s discovered it’s release critical, because at least one package no longer builds in unstable.
Commercial organisations try to avoid this with a decent CI/CD setup that either vendors all dependencies, or tracks changes to them and tries rebuilds before allowing things to land. This is one of the instances where a monorepo can really shine; if everything you need is in there, it’s easier to track the interconnections between different components. Debian doesn’t have a CI/CD system that runs for every upload, allowing us to track exact causes of regressions. Instead we have Lucas, who does a tremendous job of running archive wide rebuilds to make sure we can still build everything. Unfortunately that means I am often unfairly grumpy at him; my heart sinks when I see a bug come in with his name attached, because it often means one of my packages has a new RC bug where I’m going to have to figure out what changed elsewhere to cause it. However he’s just (very usefully) surfacing an issue someone else created, rather than actually being the cause of the problem.
I don’t know if I have a point to this post. I think it’s probably that I wish folk in Free Software would try and be mindful of the incompatible changes they might introducing, and the toil they create for other volunteer developers, often not directly visible to the person making the change. The approach done by the Debian toolchain maintainers strikes me as a good balance; they do a bunch of work up front to try and flag all the places that might need to make changes, far enough in advance of the breaking change actually landing. However they don’t then allow a tardy developer to block progress.
The following contributors got their Debian Developer accounts in the last two months:
The following contributors were added as Debian Maintainers in the last two months:
Congratulations!
24 March, 2025 03:00PM by Jean-Pierre Giraud
Around a year ago I discussed two concerns with software release archives (tarball artifacts) that could be improved to increase confidence in the supply-chain security of software releases. Repeating the goals for simplicity:
While implementing these ideas for a small project was accomplished within weeks – see my announcement of Libntlm version 1.8 – adressing this in complex projects uncovered concerns with tools that had to be addressed, and things stalled for many months pending that work.
I had the notion that these two goals were easy and shouldn’t be hard to accomplish. I still believe that, but have had to realize that improving tooling to support these goals takes time. It seems clear that these concepts are not universally agreed on and implemented generally.
I’m now happy to recap some of the work that led to releases of libtasn1 v4.20.0, inetutils v2.6, libidn2 v2.3.8, libidn v1.43. These releases all achieve these goals. I am working on a bunch of more projects to support these ideas too.
What have the obstacles so far been to make this happen? It may help others who are in the same process of addressing these concerns to have a high-level introduction to the issues I encountered. Source code for projects above are available and anyone can look at the solutions to learn how the problems are addressed.
First let’s look at the problems we need to solve to make “git-archive” style tarballs usable:
To build usable binaries from a minimal tarballs, it need to know which version number it is. Traditionally this information was stored inside configure.ac in git. However I use gnulib’s git-version-gen to infer the version number from the git tag or git commit instead. The git tag information is not available in a git-archive tarball. My solution to this was to make use of the export-subst feature of the .gitattributes file. I store the file .tarball-version-git in git containing the magic cookie like this:
$Format:%(describe)$With this, git-archive will replace with a useful version identifier on export, see the libtasn1 patch to achieve this. To make use of this information, the git-version-gen script was enhanced to read this information, see the gnulib patch. This is invoked by ./configure to figure out which version number the package is for.
We want translations to be included in the minimal source tarball for it to be buildable. Traditionally these files are retrieved by the maintainer from the Translation project when running ./bootstrap, however there are two problems with this. The first one is that there is no strong authentication or versioning information on this data, the tools just download and place whatever wget downloaded into your source tree (printf-style injection attack anyone?). We could improve this (e.g., publish GnuPG signed translations messages with clear versioning), however I did not work on that further. The reason is that I want to support offline builds of packages. Downloading random things from the Internet during builds does not work when building a Debian package, for example. The translation project could solve this by making a monthly tarball with their translations available, for distributors to pick up and provide as a separate package that could be used as a build dependency. However that is not how these tools and projects are designed. Instead I reverted back to storing translations in git, something that I did for most projects back when I was using CVS 20 years ago. Hooking this into ./bootstrap and gettext workflow can be tricky (ideas for improvement most welcome!), but I used a simple approach to store all directly downloaded po/*.po files directly as po/*.po.in and make the ./bootstrap tool move them in place, see the libidn2 commit followed by the actual ‘make update-po’ commit with all the translations where one essential step is:
# Prime po/*.po from fall-back copy stored in git.
for poin in po/*.po.in; do
po=$(echo $poin | sed 's/.in//')
test -f $po || cp -v $poin $po
done
ls po/*.po | sed 's|.*/||; s|\.po$||' > po/LINGUASMost build dependencies are in the shape of “You need a C compiler”. However some come in the shape of “source-code files intended to be vendored”, and gnulib is a huge repository of such files. The latter is a problem when building from a minimal git archive. It is possible to consider translation files as a class of vendor files, since they need to be copied verbatim into the project build directory for things to work. The same goes for *.m4 macros from the GNU Autoconf Archive. However I’m not confident that the solution for all vendor files must be the same. For translation files and for Autoconf Archive macros, I have decided to put these files into git and merge them manually occasionally. For gnulib files, in some projects like OATH Toolkit I also store all gnulib files in git which effectively resolve this concern. (Incidentally, the reason for doing so was originally that running ./bootstrap took forever since there is five gnulib instances used, which is no longer the case since gnulib-tool was rewritten in Python.) For most projects, however, I rely on ./bootstrap to fetch a gnulib git clone when building. I like this model, however it doesn’t work offline. One way to resolve this is to make the gnulib git repository available for offline use, and I’ve made some effort to make this happen via a Gnulib Git Bundle and have explained how to implement this approach for Debian packaging. I don’t think that is sufficient as a generic solution though, it is mostly applicable to building old releases that uses old gnulib files. It won’t work when building from CI/CD pipelines, for example, where I have settled to use a crude way of fetching and unpacking a particular gnulib snapshot, see this Libntlm patch. This is much faster than working with git submodules and cloning gnulib during ./bootstrap. Essentially this is doing:
GNULIB_REVISION=$(. bootstrap.conf >&2; echo $GNULIB_REVISION)
wget -nv https://gitlab.com/libidn/gnulib-mirror/-/archive/$GNULIB_REVISION/gnulib-mirror-$GNULIB_REVISION.tar.gz
gzip -cd gnulib-mirror-$GNULIB_REVISION.tar.gz | tar xf -
rm -fv gnulib-mirror-$GNULIB_REVISION.tar.gz
export GNULIB_SRCDIR=$PWD/gnulib-mirror-$GNULIB_REVISION
./bootstrap --no-git
./configure
makeThis goes without saying, but if you don’t test that building from a git-archive style tarball works, you are likely to regress at some point. Use CI/CD techniques to continuously test that a minimal git-archive tarball leads to a usable build.
So that wasn’t hard, was it? You should now be able to publish a minimal git-archive tarball and users should be able to build your project from it.
I recommend naming these archives as PROJECT-vX.Y.Z-src.tar.gz replacing PROJECT with your project name and X.Y.Z with your version number. The archive should have only one sub-directory named PROJECT-vX.Y.Z/ containing all the source-code files. This differentiate it against traditional PROJECT-X.Y.Z.tar.gz tarballs in that it embeds the git tag (which typically starts with v) and contains a wildcard-friendly -src substring. Alas there is no consistency around this naming pattern, and GitLab, GitHub, Codeberg etc all seem to use their own slightly incompatible variant.
Let’s go on to see what is needed to achieve reproducible “make dist” source tarballs. This is the release artifact that most users use, and they often contain lots of generated files and vendor files. These files are included to make it easy to build for the user. What are the challenges to make these reproducible?
The first part is to realize that if you use tool X with version A to generate a file that goes into the tarball, version B of that tool may produce different outputs. This is a generic concern and it cannot be solved. We want our build tools to evolve and produce better outputs over time. What can be addressed is to avoid needless differences. For example, many tools store timestamps and versioning information in the generated files. This causes needless differences, which makes audits harder. I have worked on some of these, like Autoconf Archive timestamps but solving all of these examples will take a long time, and some upstream are reluctant to incorporate these changes. My approach meanwhile is to build things using similar environments, and compare the outputs for differences. I’ve found that the various closely related forks of GNU/Linux distributions are useful for this. Trisquel 11 is based on Ubuntu 22.04, and building my projects using both and comparing the differences only give me the relevant differences to improve. This can be extended to compare AlmaLinux with RockyLinux (for both versions 8 and 9), Devuan 5 against Debian 12, PureOS 10 with Debian 11, and so on.
Sometimes tools store timestamps in files in a way that is harder to fix. Two notable examples of this are *.po translation files and Texinfo manuals. For translation files, I have resolved this by making sure the files use a predictable POT-Creation-Date timestamp, and I set it to the modification timestamps of the NEWS file in the repository (which I set to the git commit of the latest commit elsewhere) like this:
dist-hook: po-CreationDate-to-mtime-NEWS
.PHONY: po-CreationDate-to-mtime-NEWS
po-CreationDate-to-mtime-NEWS: mtime-NEWS-to-git-HEAD
$(AM_V_GEN)for p in $(distdir)/po/*.po $(distdir)/po/$(PACKAGE).pot; do \
if test -f "$$p"; then \
$(SED) -e 's,POT-Creation-Date: .*\\n",POT-Creation-Date: '"$$(env LC_ALL=C TZ=UTC0 stat --format=%y $(srcdir)/NEWS | cut -c1-16,31-)"'\\n",' < $$p > $$p.tmp && \
if cmp $$p $$p.tmp > /dev/null; then \
rm -f $$p.tmp; \
else \
mv $$p.tmp $$p; \
fi \
fi \
doneSimilarily, I set a predictable modification time of the texinfo source file like this:
dist-hook: mtime-NEWS-to-git-HEAD
.PHONY: mtime-NEWS-to-git-HEAD
mtime-NEWS-to-git-HEAD:
$(AM_V_GEN)if test -e $(srcdir)/.git \
&& command -v git > /dev/null; then \
touch -m -t "$$(git log -1 --format=%cd \
--date=format-local:%Y%m%d%H%M.%S)" $(srcdir)/NEWS; \
fiHowever I’ve realized that this needs to happen earlier and probably has to be run during ./configure time, because the doc/version.texi file is generated on first build before running ‘make dist‘ and for some reason the file is not rebuilt at release time. The Automake texinfo integration is a bit inflexible about providing hooks to extend the dependency tracking.
The method to address these differences isn’t really important, and they change over time depending on preferences. What is important is that the differences are eliminated.
Traditionally ChangeLog files were manually prepared, and still is for some projects. I maintain git2cl but recently I’ve settled with gnulib’s gitlog-to-changelog because doing so avoids another build dependency (although the output formatting is different and arguable worse for my git commit style). So the ChangeLog files are generated from git history. This means a shallow clone will not produce the same ChangeLog file depending on how deep it was cloned. For Libntlm I simply disabled use of generated ChangeLog because I wanted to support an even more extreme form of reproducibility: I wanted to be able to reproduce the full “make dist” source archives from a minimal “git-archive” source archive. However for other projects I’ve settled with a middle ground. I realized that for ‘git describe‘ to produce reproducible outputs, the shallow clone needs to include the last release tag. So it felt acceptable to assume that the clone is not minimal, but instead has some but not all of the history. I settled with the following recipe to produce ChangeLog's covering all changes since the last release.
dist-hook: gen-ChangeLog
.PHONY: gen-ChangeLog
gen-ChangeLog:
$(AM_V_GEN)if test -e $(srcdir)/.git; then \
LC_ALL=en_US.UTF-8 TZ=UTC0 \
$(top_srcdir)/build-aux/gitlog-to-changelog \
--srcdir=$(srcdir) -- \
v$(PREV_VERSION)~.. > $(distdir)/cl-t && \
{ printf '\n\nSee the source repo for older entries\n' \
>> $(distdir)/cl-t && \
rm -f $(distdir)/ChangeLog && \
mv $(distdir)/cl-t $(distdir)/ChangeLog; } \
fiI’m undecided about the usefulness of generated ChangeLog files within ‘make dist‘ archives. Before we have stable and secure archival of git repositories widely implemented, I can see some utility of this in case we lose all copies of the upstream git repositories. I can sympathize with the concept of ChangeLog files died when we started to generate them from git logs: the files no longer serve any purpose, and we can ask people to go look at the git log instead of reading these generated non-source files.
Distributions comes and goes, and old releases of them goes out of support and often stops working. Which build environment should I chose to build the official release archives? To my knowledge only Guix offers a reliable way to re-create an older build environment (guix gime-machine) that have bootstrappable properties for additional confidence. However I had two difficult problems here. The first one was that I needed Guix container images that were usable in GitLab CI/CD Pipelines, and this side-tracked me for a while. The second one delayed my effort for many months, and I was inclined to give up. Libidn distribute a C# implementation. Some of the C# source code files included in the release tarball are generated. By what? You guess it, by a C# program, with the source code included in the distribution. This means nobody could reproduce the source tarball of Libidn without trusting someone elses C# compiler binaries, which were built from binaries of earlier releases, chaining back into something that nobody ever attempts to build any more and likely fail to build due to bit-rot. I had two basic choices, either remove the C# implementation from Libidn (which may be a good idea for other reasons, since the C and C# are unrelated implementations) or build the source tarball on some binary-only distribution like Trisquel. Neither felt appealing to me, but a late christmas gift of a reproducible Mono came to Guix that resolve this.
For Libidn one section of the manual has an image illustrating some concepts. The PNG, PDF and EPS outputs were generated via fig2dev from a *.fig file (hello 1985!) that I had stored in git. Over time, I had also started to store the generated outputs because of build issues. At some point, it was possible to post-process the PDF outputs with grep to remove some timestamps, however with compression this is no longer possible and actually the grep command I used resulted in a 0-byte output file. So my embedded binaries in git was no longer reproducible. I first set out to fix this by post-processing things properly, however I then realized that the *.fig file is not really easy to work with in a modern world. I wanted to create an image from some text-file description of the image. Eventually, via the Guix manual on guix graph, I came to re-discover the graphviz language and tool called dot (hello 1993!). All well then? Oh no, the PDF output embeds timestamps. Binary editing of PDF’s no longer work through simple grep, remember? I was back where I started, and after some (soul- and web-) searching I discovered that Ghostscript (hello 1988!) pdfmarks could be used to modify things here. Cooperating with automake’s texinfo rules related to make dist proved once again a worthy challenge, and eventually I ended up with a Makefile.am snippet to build images that could be condensed into:
info_TEXINFOS = libidn.texi
libidn_TEXINFOS += libidn-components.png
imagesdir = $(infodir)
images_DATA = libidn-components.png
EXTRA_DIST += components.dot
DISTCLEANFILES = \
libidn-components.eps libidn-components.png libidn-components.pdf
libidn-components.eps: $(srcdir)/components.dot
$(AM_V_GEN)$(DOT) -Nfontsize=9 -Teps < $< > $@.tmp
$(AM_V_at)! grep %%CreationDate $@.tmp
$(AM_V_at)mv $@.tmp $@
libidn-components.pdf: $(srcdir)/components.dot
$(AM_V_GEN)$(DOT) -Nfontsize=9 -Tpdf < $< > $@.tmp
# A simple sed on CreationDate is no longer possible due to compression.
# 'exiftool -CreateDate' is alternative to 'gs', but adds ~4kb to file.
# Ghostscript add <1kb. Why can't 'dot' avoid setting CreationDate?
$(AM_V_at)printf '[ /ModDate ()\n /CreationDate ()\n /DOCINFO pdfmark\n' > pdfmarks
$(AM_V_at)$(GS) -q -dBATCH -dNOPAUSE -sDEVICE=pdfwrite -sOutputFile=$@.tmp2 $@.tmp pdfmarks
$(AM_V_at)rm -f $@.tmp pdfmarks
$(AM_V_at)mv $@.tmp2 $@
libidn-components.png: $(srcdir)/components.dot
$(AM_V_GEN)$(DOT) -Nfontsize=9 -Tpng < $< > $@.tmp
$(AM_V_at)mv $@.tmp $@
pdf-recursive: libidn-components.pdf
dvi-recursive: libidn-components.eps
ps-recursive: libidn-components.eps
info-recursive: $(top_srcdir)/.version libidn-components.pngSurely this can be improved, but I’m not yet certain in what way is the best one forward. I like having a text representation as the source of the image. I’m sad that the new image size is ~48kb compared to the old image size of ~1kb. I tried using exiftool -CreateDate as an alternative to GhostScript, but using it to remove the timestamp added ~4kb to the file size and naturally I was appalled by this ignorance of impending doom.
Again, you need to continuously test the properties you desire. This means building your project twice using different environments and comparing the results. I’ve settled with a small GitLab CI/CD pipeline job that perform bit-by-bit comparison of generated ‘make dist’ archives. It also perform bit-by-bit comparison of generated ‘git-archive’ artifacts. See the Libidn2 .gitlab-ci.yml 0-compare job which essentially is:
0-compare:
image: alpine:latest
stage: repro
needs: [ B-AlmaLinux8, B-AlmaLinux9, B-RockyLinux8, B-RockyLinux9, B-Trisquel11, B-Ubuntu2204, B-PureOS10, B-Debian11, B-Devuan5, B-Debian12, B-gcc, B-clang, B-Guix, R-Guix, R-Debian12, R-Ubuntu2404, S-Trisquel10, S-Ubuntu2004 ]
script:
- cd out
- sha256sum */*.tar.* */*/*.tar.* | sort | grep -- -src.tar.
- sha256sum */*.tar.* */*/*.tar.* | sort | grep -v -- -src.tar.
- sha256sum */*.tar.* */*/*.tar.* | sort | uniq -c -w64 | sort -rn
- sha256sum */*.tar.* */*/*.tar.* | grep -- -src.tar. | sort | uniq -c -w64 | grep -v '^ 1 '
- sha256sum */*.tar.* */*/*.tar.* | grep -v -- -src.tar. | sort | uniq -c -w64 | grep -v '^ 1 '
# Confirm modern git-archive tarball reproducibility
- cmp b-almalinux8/src/*.tar.gz b-almalinux9/src/*.tar.gz
- cmp b-almalinux8/src/*.tar.gz b-rockylinux8/src/*.tar.gz
- cmp b-almalinux8/src/*.tar.gz b-rockylinux9/src/*.tar.gz
- cmp b-almalinux8/src/*.tar.gz b-debian12/src/*.tar.gz
- cmp b-almalinux8/src/*.tar.gz b-devuan5/src/*.tar.gz
- cmp b-almalinux8/src/*.tar.gz r-guix/src/*.tar.gz
- cmp b-almalinux8/src/*.tar.gz r-debian12/src/*.tar.gz
- cmp b-almalinux8/src/*.tar.gz r-ubuntu2404/src/*v2.*.tar.gz
# Confirm old git-archive (export-subst but long git describe) tarball reproducibility
- cmp b-trisquel11/src/*.tar.gz b-ubuntu2204/src/*.tar.gz
# Confirm really old git-archive (no export-subst) tarball reproducibility
- cmp b-debian11/src/*.tar.gz b-pureos10/src/*.tar.gz
# Confirm 'make dist' generated tarball reproducibility
- cmp b-almalinux8/*.tar.gz b-rockylinux8/*.tar.gz
- cmp b-almalinux9/*.tar.gz b-rockylinux9/*.tar.gz
- cmp b-pureos10/*.tar.gz b-debian11/*.tar.gz
- cmp b-devuan5/*.tar.gz b-debian12/*.tar.gz
- cmp b-trisquel11/*.tar.gz b-ubuntu2204/*.tar.gz
- cmp b-guix/*.tar.gz r-guix/*.tar.gz
# Confirm 'make dist' from git-archive tarball reproducibility
- cmp s-trisquel10/*.tar.gz s-ubuntu2004/*.tar.gzNotice that I discovered that ‘git archive’ outputs differ over time too, which is natural but a bit of a nuisance. The output of the job is illuminating in the way that all SHA256 checksums of generated tarballs are included, for example the libidn2 v2.3.8 job log:
$ sha256sum */*.tar.* */*/*.tar.* | sort | grep -v -- -src.tar.
368488b6cc8697a0a937b9eb307a014396dd17d3feba3881e6911d549732a293 b-trisquel11/libidn2-2.3.8.tar.gz
368488b6cc8697a0a937b9eb307a014396dd17d3feba3881e6911d549732a293 b-ubuntu2204/libidn2-2.3.8.tar.gz
59db2d045fdc5639c98592d236403daa24d33d7c8db0986686b2a3056dfe0ded b-debian11/libidn2-2.3.8.tar.gz
59db2d045fdc5639c98592d236403daa24d33d7c8db0986686b2a3056dfe0ded b-pureos10/libidn2-2.3.8.tar.gz
5bd521d5ecd75f4b0ab0fc6d95d444944ef44a84cad859c9fb01363d3ce48bb8 s-trisquel10/libidn2-2.3.8.tar.gz
5bd521d5ecd75f4b0ab0fc6d95d444944ef44a84cad859c9fb01363d3ce48bb8 s-ubuntu2004/libidn2-2.3.8.tar.gz
7f1dcdea3772a34b7a9f22d6ae6361cdcbe5513e3b6485d40100b8565c9b961a b-almalinux8/libidn2-2.3.8.tar.gz
7f1dcdea3772a34b7a9f22d6ae6361cdcbe5513e3b6485d40100b8565c9b961a b-rockylinux8/libidn2-2.3.8.tar.gz
8031278157ce43b5813f36cf8dd6baf0d9a7f88324ced796765dcd5cd96ccc06 b-clang/libidn2-2.3.8.tar.gz
8031278157ce43b5813f36cf8dd6baf0d9a7f88324ced796765dcd5cd96ccc06 b-debian12/libidn2-2.3.8.tar.gz
8031278157ce43b5813f36cf8dd6baf0d9a7f88324ced796765dcd5cd96ccc06 b-devuan5/libidn2-2.3.8.tar.gz
8031278157ce43b5813f36cf8dd6baf0d9a7f88324ced796765dcd5cd96ccc06 b-gcc/libidn2-2.3.8.tar.gz
8031278157ce43b5813f36cf8dd6baf0d9a7f88324ced796765dcd5cd96ccc06 r-debian12/libidn2-2.3.8.tar.gz
acf5cbb295e0693e4394a56c71600421059f9c9bf45ccf8a7e305c995630b32b r-ubuntu2404/libidn2-2.3.8.tar.gz
cbdb75c38100e9267670b916f41878b6dbc35f9c6cbe60d50f458b40df64fcf1 b-almalinux9/libidn2-2.3.8.tar.gz
cbdb75c38100e9267670b916f41878b6dbc35f9c6cbe60d50f458b40df64fcf1 b-rockylinux9/libidn2-2.3.8.tar.gz
f557911bf6171621e1f72ff35f5b1825bb35b52ed45325dcdee931e5d3c0787a b-guix/libidn2-2.3.8.tar.gz
f557911bf6171621e1f72ff35f5b1825bb35b52ed45325dcdee931e5d3c0787a r-guix/libidn2-2.3.8.tar.gzI’m sure I have forgotten or suppressed some challenges (sprinkling LANG=C TZ=UTC0 helps) related to these goals, but my hope is that this discussion of solutions will inspire you to implement these concepts for your software project too. Please share your thoughts and additional insights in a comment below. Enjoy Happy Hacking in the course of practicing this!
24 March, 2025 11:09AM by simon
Oh no, it broke again!
Today, this .gitlab-ci.yml file no longer works in GitLab CI:
build-container-image:
stage: build
image: debian:testing
before_script:
- apt-get update
- apt-get install -y buildah ca-certificates
script:
- buildah build -t $CI_REGISTRY_IMAGE .
The command buildah build ... fails with this error message:
STEP 2/3: RUN apt-get update
internal:0:0-0: Error: Could not process rule: No such file or directory
internal:0:0-0: Error: Could not process rule: No such file or directory
error running container: did not get container start message from parent: EOF
Error: building at STEP "RUN apt-get update": setup network: netavark: nftables error: nft did not return successfully while applying ruleset
After some investigation, it's caused by the recent upload of netavark 1.14.0-2. In this version, netavark switched from iptables to nftables as the default firewall driver. That doesn't really fly on GitLab Saas shared runners.
For the complete background, refer to https://discussion.fedoraproject.org/t/125528. Note that the issue with GitLab was reported back in November, but at this point the conversation had died out.
Fortunately, it's easy to workaround, we can tell netavark to keep using
iptables via the environment variables NETAVARK_FW. The .gitlab-ci.yml file
above becomes:
build-container-image:
stage: build
image: debian:testing
variables:
# Cf. https://discussion.fedoraproject.org/t/125528/7
NETAVARK_FW: iptables
before_script:
- apt-get update
- apt-get install -y buildah ca-certificates
script:
- buildah build -t $CI_REGISTRY_IMAGE .
And everything works again!
If you're interested in this issue, feel free to fork https://gitlab.com/arnaudr/gitlab-build-container-image and try it by yourself.
24 March, 2025 12:00AM by Arnaud Rebillout
Recent initial releases of [Ringlet software][r-site] (a fancy name for my pet projects):
ps or lsof installed.shlex.join()][python-shlex-join] that also accepts [pathlib.Path][python-pathlib-path] objects.23 March, 2025 04:54PM by Peter Pentchev
The Open Source Initiative has two classes of board seats: Affiliate seats, and Individual Member seats.
In the upcoming election, each affiliate can nominate a candidate, and each affiliate can cast a vote for the Affiliate candidates, but there's only 1 Affiliate seat available. I initially expressed interest in being nominated as an Affiliate candidate via Debian. But since Bradley Kuhn is also running for an Affiliate seat with a similar platform to me, especially with regards to the OSAID, I decided to run as part of an aligned "ticket" as an Individual Member to avoid contention for the 1 Affiliate seat.
Bradley and I discussed running on a similar ticket around 8/9pm Pacific, and I submitted my candidacy around 9pm PT on 17 February.
I was dismayed when I received the following mail from Nick Vidal:
Dear Luke,
Thank you for your interest in the OSI Board of Directors election. Unfortunately, we are unable to accept your application as it was submitted after the official deadline of Monday Feb 17 at 11:59 pm UTC. To ensure a fair process, we must adhere to the deadline for all candidates.
We appreciate your enthusiasm and encourage you to stay engaged with OSI’s mission. We hope you’ll consider applying in the future or contributing in other meaningful ways.
Best regards,
OSI Election Teams
Nowhere on the "OSI’s board of directors in 2025: details about the elections" page do they list a timezone for closure of nominations; they simply list Monday 17 February.
The OSI's contact address is in California, so it seems arbitrary and capricious to retroactively define all of these processes as being governed by UTC.
I was not able to participate in the "potential board director" info sessions accordingly, but people who attended heard that the importance of accommodating differing TZ's was discussed during the info session, and that OSI representatives mentioned they try to accommodate TZ's of everyone. This seems in sharp contrast with the above policy.
I urge the OSI to reconsider this policy and allow me to stand for an Individual seat in the current cycle.
Upd, N.B.: to people writing about this, I use they/them pronouns
22 March, 2025 04:30PM by Luke Faraone (noreply@blogger.com)
Warning: this is a long ramble I wrote after an outage of my home internet. You'll get your regular scheduled programming shortly.
I didn't realize this until relatively recently, but we're at war.
Fascists and capitalists are trying to take over the world, and it's bringing utter chaos.
We're more numerous than them, of course: this is only a handful of people screwing everyone else over, but they've accumulated so much wealth and media control that it's getting really, really hard to move around.
Everything is surveilled: people are carrying tracking and recording devices in their pockets at all time, or they drive around in surveillance machines. Payments are all turning digital. There's cameras everywhere, including in cars. Personal data leaks are so common people kind of assume their personal address, email address, and other personal information has already been leaked.
The internet itself is collapsing: most people are using the network only as a channel to reach a "small" set of "hyperscalers": mind-boggingly large datacenters that don't really operate like the old internet. Once you reach the local endpoint, you're not on the internet anymore. Netflix, Google, Facebook (Instagram, Whatsapp, Messenger), Apple, Amazon, Microsoft (Outlook, Hotmail, etc), all those things are not really the internet anymore.
Those companies operate over the "internet" (as in the TCP/IP network), but they are not an "interconnected network" as much as their own, gigantic silos so much bigger than everything else that they essentially dictate how the network operates, regardless of standards. You access it over "the web" (as in "HTTP") but the fabric is not made of interconnected links that cross sites: all those sites are trying really hard to keep you captive on their platforms.
Besides, you think you're writing an email to the state department, for example, but you're really writing to Microsoft Outlook. That app your university or border agency tells you to install, the backend is not hosted by those institutions, it's on Amazon. Heck, even Netflix is on Amazon.
Meanwhile I've been operating my own mail server first under my bed (yes, really) and then in a cupboard or the basement for almost three decades now. And what for?
So I can tell people I can? Maybe!
I guess the reason I'm doing this is the same reason people are suddenly asking me about the (dead) mesh again. People are worried and scared that the world has been taken over, and they're right: we have gotten seriously screwed.
It's the same reason I keep doing radio, minimally know how to grow food, ride a bike, build a shed, paddle a canoe, archive and document things, talk with people, host an assembly. Because, when push comes to shove, there's no one else who's going to do it for you, at least not the way that benefits the people.
The Internet is one of humanity's greatest accomplishments. Obviously, oligarchs and fascists are trying to destroy it. I just didn't expect the tech bros to be flipping to that side so easily. I thought we were friends, but I guess we are, after all, enemies.
That said, that old internet is still around. It's getting harder to host your own stuff at home, but it's not impossible. Mail is tricky because of reputation, but it's also tricky in the cloud (don't get fooled!), so it's not that much easier (or cheaper) there.
So there's things you can do, if you're into tech.
Share your wifi with your neighbours.
Build a LAN. Throw a wire over to your neighbour too, it works better than wireless.
Use Tor. Run a relay, a snowflake, a webtunnel.
Host a web server. Build a site with a static site generator and throw it in the wind.
Download and share torrents, and why not a tracker.
Run an IRC server (or Matrix, if you want to federate and lose high availability).
At least use Signal, not Whatsapp or Messenger.
And yes, why not, run a mail server, join a mesh.
Don't write new software, there's plenty of that around already.
(Just kidding, you can write code, cypherpunk.)
You can do many of those things just by setting up a FreedomBox.
That is, after all, the internet: people doing their own thing for their own people.
Otherwise, it's just like sitting in front of the television and watching the ads. Opium of the people, like the good old time.
Let a billion droplets build the biggest multitude of clouds that will storm over this world and rip apart this fascist conspiracy.
Disobey. Revolt. Build.
We are more than them.
This morning, internet was down at home. The last time I had such an issue was in February 2023, when my provider was Oricom. Now I'm with a business service at Teksavvy Internet (TSI), in which I pay 100$ per month for a 250/50 mbps business package, with a static IP address, on which I run, well, everything: email services, this website, etc.
The main problem when the service goes down like this for prolonged outages is email. Mail is pretty resilient to failures like this but after some delay (which varies according to the other end), mail starts to drop. I am actually not sure what the various settings are among different providers, but I would assume mail is typically kept for about 24h, so that's our mark.
Last time, I setup VMs at Linode and Digital Ocean to deal better with this. I have actually kept those VMs running as DNS servers until now, so that part is already done.
I had fantasized about Puppetizing the mail server configuration so
that I could quickly spin up mail exchangers on those machines. But
now I am realizing that my Puppet server is one of the service that's
down, so this would not work, at least not unless the manifests can be
applied without a Puppet server (say with puppet apply).
Thankfully, my colleague groente did amazing work to refactor our Postfix configuration in Puppet at Tor, and that gave me the motivation to reproduce the setup in the lab. So I have finally Puppetized part of my mail setup at home. That used to be hand-crafted experimental stuff documented in a couple of pages in this wiki, but is now being deployed by Puppet.
It's not complete yet: spam filtering (including DKIM checks and
graylisting) are not implemented yet, but that's the next step,
presumably to do during the next outage. The setup should be
deployable with puppet apply, however, and I have refined that
mechanism a little bit, with the run script.
Heck, it's not even deployed yet. But the hard part / grunt work is done.
The outage was "short" enough (5 hours) that I didn't take time to deploy the other mitigations I had deployed in the previous incident.
But I'm starting to seriously consider deploying a web (and caching) reverse proxy so that I endure such problems more gracefully.
Typically, I tend to think of a properly functioning service as having four things:
Yes, I miscounted. This is why you have high availability.
Duh. If data is maliciously or accidentally destroyed, you need a copy somewhere. Preferably in a way that malicious joe can't get to.
This is harder than you think.
I have an entire template for this. Essentially, it boils down to using https://diataxis.fr/ and this "audit" guide. For me, the most important parts are:
You probably know this is hard, and this is why you're not doing it. Do it anyways, you'll think it sucks, but you'll be really grateful for whatever scraps you wrote when you're in trouble.
If you don't have monitoring, you'll know it fails too late, and you won't know it recovers. Consider high availability, work hard to reduce noise, and don't have machine wake people up, that's literally torture and is against the Geneva convention.
Consider predictive algorithm to prevent failures, like "add storage within 2 weeks before this disk fills up".
This is harder than you think.
Make it easy to redeploy the service elsewhere.
Yes, I know you have backups. That is not enough: that typically restores data and while it can also include configuration, you're going to need to change things when you restore, which is what automation (or call it "configuration management" if you will) will do for you anyways.
This also means you can do unit tests on your configuration, otherwise you're building legacy.
This is probably as hard as you think.
Make it not fail when one part goes down.
Eliminate single points of failures.
This is easier than you think, except for storage and DNS (which, I guess, means it's harder than you think too).
In the above 5 items, I check two:
And barely: I'm not happy about the offsite backups, and my documentation is much better at work than at home (and even there, I have a 15 year backlog to catchup on).
I barely have monitoring: Prometheus is scraping parts of the infra, but I don't have any sort of alerting -- by which I don't mean "electrocute myself when something goes wrong", I mean "there's a set of thresholds and conditions that define an outage and I can look at it".
Automation is wildly incomplete. My home server is a random collection of old experiments and technologies, ranging from Apache with Perl and CGI scripts to Docker containers running Golang applications. Most of it is not Puppetized (but the ratio is growing). Puppet itself introduces a huge attack vector with kind of catastrophic lateral movement if the Puppet server gets compromised.
And, fundamentally, I am not sure I can provide high availability in the lab. I'm just this one guy running my home network, and I'm growing older. I'm thinking more about winding things down than building things now, and that's just really sad, because I feel we're losing (well that escalated quickly).
In the end, I didn't need any mitigation and the problem fixed itself. I did do quite a bit of cleanup so that feels somewhat good, although I despaired quite a bit at the amount of technical debt I've accumulated in the lab.
Times are in UTC-4.
Two years after OpenAI launched ChatGPT 3.5, humanity is not on the cusp of extinction and Elon Musk seems more responsible for job loss than any AI agent.
However, ask any web administrator and you will learn that large language models are having a significant impact on the world wide web (or, for a less technical account, see Forbes articles on bots). At May First, a membership organization that has been supporting thousands of web site for over 20 years, we have never seen anything like this before.
It started in 2023. Web sites that performed quite well with a steady viewership started having traffic spikes. These were relatively easy to diagnose, since most of the spikes came from visitors that properly identified themselves as bots, allowing us to see that the big players - OpenAI, Bing, Google, Facebook - were increasing their efforts to scrape as much content from web sites as possible.
Small brochure sites were mostly unaffected because they could be scraped in a matter of minutes. But large sites with an archive of high quality human written content were getting hammered. Any web site with a search feature or a calendar or any interface that generated exponential hits that could be followed were particularly vulnerable.
But hey, that’s what robots.txt is for, right? To tell robots to back off if you don’t want them scraping your site?
Eventually, the cracks began to show. Bots were ignoring robots.txt (did they ever pay that much attention to it in the first place?). Furthermore, rate limiting requests by user agent also began to fail. When you post a link on Facebook, a bot identifying itself as “facebooketernalhit” is invoked to preview the page so it can show a picture and other meta data. We don’t want to rate limit that bot, right? Except, Facebook is also using this bot to scrape your site, often bringing your site to its knees. And don’t get me started on TwitterBot.
Eventually, it became clear that the majority of the armies of bots scraping our sites have completely given up on identifying themselves as bots and are instead using user agents indistinguishable from regular browsers. By using thousands of different IP addresses, it has become really hard to separate the real humans from the bots.
So, no, unfortunately, your web site is not suddenly getting really popular. And, you are blessed with a whole new set of strategic decisions.
Fortunately, May First has undergone a major infrastructure transition, resulting in centralized logging of all web sites and a fleet of web proxy servers that intercept all web traffic. Centralized logging means we can analyze traffic and identify bots more easily, and a web proxy fleet allows us to more easily implement rules across all web sites.
However, even with all of our latest changes and hours upon hours of work to keep out the bots, our members are facing some hard decisions about maintaining an open web.
One member of May First provides Google translations of their web site to every language available. But wow, that is now a disaster because instead of having every bot under the sun scrapping all 843 (a made up number) pieces of unique content on their site, the same bots are scraping 843 * (number of available languages) pieces of content on their site. Should they stop providing this translation service in order to ensure people can access their site in the site’s primary language?
Should web sites turn off their search features that include drop down options of categories to prevent bots from systematically refreshing the search page with every possible combination of search terms?
Do we need to alter our calendar software to avoid providing endless links into the future (ok, that is an easy one)?
Something has to change.
Lock down web 2.0. Web 2.0 brought us wonderful dynamic web sites, which Drupal and WordPress and many other pieces of amazing software have supported for over a decade. This is the software that is getting bogged down by bots. Maybe we need to figure out a way to lock down the dynamic aspects of this software to logged in users and provide static content for everyone else?
Paywalls and accounts everywhere. There’s always been an amazing non-financial reward to providing a web site with high quality movement oriented content for free. It populates the search engines, provides links to inspiring and useful content in moments of crises, and can galvanize movements. But these moments of triumph happen between long periods of hard labor that now seems to mostly feed capitalist AI scumbags. If we add a new set of expenses and labor to keep the sites running for this purpose, how sustainable is that? Will our treasure of free movement content have to move behind paywalls or logins? If we provide logins, will that keep the bots out or just create a small hurdle for them to automate the account creation process? What happens when we can’t search for this kind of content via search engines?
Cutting deals. What if our movement content providers are forced to cut deals with the AI entrepreneurs to allow the paying scumbags to fund the content creation. Eww. Enough said.
Bot detection. Maybe we just need to get better at bot detection? This will surely be an arms race, but would have some good benefits. Bots have also been filling out our forms and populating our databases with spam, testing credit cards against our donation pages, conducting denial of service attacks and all kinds of other irritating acts of vandalism. If we were better at stopping bots automatically it would have a lot of benefits. But what impact would it have on our web sites and the experience of using them? What about “good” bots (RSS feed readers, payment processors, web hooks, uptime detectors)? Will we cut the legs off any developer trying to automate something?
I’m not really sure where this is going, but it seems that the world wide web is about to head in a new direction.
The diffoscope maintainers are pleased to announce the release of diffoscope
version 291. This version includes the following changes:
[ Chris Lamb ]
* Make two required adjustments for the new version of the src:file package:
- file(1) version 5.46 now emits "XHTML document" for .xhtml files, such as
those found nested within our .epub tests. Therefore, match this string
when detecting XML files. This was causing an FTBFS due to inconsistent
indentation in diffoscope's output.
- Require the new, upcoming, version of file(1) for a quine-related
testcase after adjusting the expected output. Previous versions of
file(1) had a duplicated "last modified, last modified" string for some
Zip archives that has now been removed.
* Add a missing subprocess import.
* Bump Standards-Version to 4.7.2.
You find out more by visiting the project homepage.
The diffoscope maintainers are pleased to announce the release of diffoscope
version 290. This version includes the following changes:
[ Chris Lamb ]
* Also consider .aar files as APK files for the sake of not falling back to a
binary diff. (Closes: #1099632)
* Ensure all calls to out_check_output in the ELF comparator have the
potential CalledProcessError exception caught. (Re: #398)
* Ensure a potential CalledProcessError is caught in the OpenSSL comparator
as well.
* Update copyright years.
[ Eli Schwartz ]
* Drop deprecated and no longer functional "setup.py test" command.
You find out more by visiting the project homepage.
`dmidecode | grep -A8 ‘^System Information’`
tells me that the Manufacturer is HP and Product Name is OMEN Transcend Gaming Laptop 14-fb0xxx
I’m provisioning a new piece of hardware for my eng consultant and it’s proving more difficult than I expected. I must admit guilt for some of this difficulty. Instead of installing using the debian installer on my keychain, I dd’d the pv block device of the 16 inch 2023 version onto the partition set aside from it. I then rebooted into rescue mode and cleaned up the grub config, corrected the EFI boot partition’s path in /etc/fstab, ran the grub installer from the rescue menu, and rebooted.
On the initial boot of the system, X or Wayland or whatever is supposed to be talking to this vast array of GPU hardware in this device, it’s unable to do more than create a black screen on vt1. It’s easy enough to switch to vt2 and get a shell on the installed system. So I’m doing that and investigating what’s changed in Trixie. It seems like it’s pretty significant. Did they just throw out Keith Packard’s and Behdad Esfahbod’s work on font rendering? I don’t understand what’s happening in this effort to abstract to a simpler interface. I’ll probably end up reading more about it.
In an effort to have Debian re-configure the system for Desktop use, I have uninstalled as many packages as I could find that were in the display and human interface category, or were firmware/drivers for devices not present in this Laptop’s SoC. Some commands I used to clear these packages and re-install connamon follow:
```
dpkg -S /etc/X11
dpkg -S /usr/lib/firmware
apt-get purge $(dpkg -l | grep -i \
-e gnome -e gtk -e x11-common -e xfonts- -e libvdpau -e dbus-user-session -e gpg-agent \
-e bluez -e colord -e cups -e fonts -e drm -e xf86 -e mesa -e nouveau -e cinnamon \
-e avahi -e gdk -e pixel -e desktop -e libreoffice -e x11 -e wayland -e xorg \
-e firmware-nvidia-graphics -e firmware-amd-graphics -e firmware-mediatek -e firmware-realtek \
| awk '{print $2}')
apt-get autoremove
apt-get purge $(dpkg -l | grep '^r' | awk '{print $2}')
tasksel install cinnamon-desktop
```
And then I rebooted. When it came back up, I was greeted with a login prompt, and Trixie looks to be fully functional on this device, including the attached wifi radio, tethering to my android, and the thunderbolt-attached Marvell SFP+ enclosure.
I’m also installing libvirt and fetched the DVD iso material for Debian, Ubuntu and Rocky in case we have a need of building VMs during the development process. These are the platforms that I target at work with gcp Dataproc, so I’m pretty good at performing maintenance operation on them at this point.
20 March, 2025 11:06PM by C.J. Collier
The film is centered around the idea of establishing an alternative to the GDP as the metric to measure success of a country/society. The film follows mostly Katherine Trebeck on her journey of convincing countries to look beyond the GDP. I very much enjoyed watching this documentary to get a first impression of the idea itself and the effort involved. I had the chance to watch the german version of it online. But there is now another virtual screening offered by the Permacultur Film Club on the 29th and 30th of March 2025. This screening is on a pay-as-you-like-and-can basis and includes a Q&A session with Kathrine Trebeck.
Trailer 1 and Trailer 2 are available on Youtube if you like to get a first impression.
Seems in the k8s world there are sufficient enough race conditions in shutting down pods and removing those from endpoint slices in time. Thus people started to do all kind of workarounds like adding a statically linked sleep binary to otherwise "distroless" and rather empty OCI images to just run a sleep command on shutdown before really shutting down. Or even base64 encoding the sleep binary and shipping it via configMap. Or whatever else. Eventually the situation was so severe that upstream decided to implement a sleep feature in the deployment resource directly.
In short it looks like this:
apiVersion: apps/v1
kind: Deployment
metadata:
name: foo
spec:
template:
spec:
lifecycle:
preStop:
sleep:
seconds: 10
Maybe highlighting that "feature" helps some more people to get rid of their own preStop sleep commands and make some deployments a tiny bit simpler.
I regularly visit Seoul, and for the last couple of years I&aposve been doing segments from the Seoul Trail, a series of walks that add up to a 150km circuit around the outskirts of Seoul. If you like hiking I recommend it, it&aposs mostly through the hills and wooded areas surrounding the city or parks within the city and the bits I&aposve done thus far have mostly been very enjoyable. Everything is generally well signposted and easy to follow, with varying degrees of difficulty from completely flat paved roads to very hilly trails.
The trail had been divided into eight segments but just after I last visited the trail was reorganised into 21 smaller ones. This was very sensible, the original segments mostly being about 10-20km and taking 3-6 hours (with the notable exception of section 8, which was 36km) which can be a bit much (especially that section 8, or section 1 which had about 1km of ascent in it overall). It does complicate matters if you&aposre trying to keep track of what you&aposve done already though so I&aposve put together a quick table:
| Original | Revised |
|---|---|
| 1 | 1-3 |
| 2 | 4-5 |
| 3 | 6-8 |
| 4 | 9-10 |
| 5 | 11-12 |
| 6 | 13-14 |
| 7 | 15-16 |
| 8 | 17-21 |
This is all straightforward, the original segments had all been arranged to start and stop at metro stations (which I think explains the length of 8, the metro network is thin around Bukhansan what with it being an actual mountain) and the new segments are all straight subdivisions, but it&aposs handy to have it written down and I figured other people might find it useful.
19 March, 2025 12:18AM by Mark Brown
Time flies! 15 years ago, on 2010-03-18, my first upload to the Debian archive was accepted. Debian had replaced Windows as my primary OS in 2005, but it was only when I saw that package zd1211-firmware had been orphaned that I thought of becoming a contributor. I owned a Zyxel G-202 USB WiFi fob that needed said firmware, and as is so often is with open-source software, I was going to scratch my own itch. Bart Martens thankfully helped me adopt the package, and sponsored my upload.
I then joined Javier Fernández-Sanguino Peña as a cron maintainer and upstream, and also worked within the Debian Python Applications, Debian Python Modules, and Debian Science Teams, where Jakub Wilk and Yaroslav Halchenko were kind enough to mentor me and eventually support my application to become a Debian Maintainer.
Life intervened, and I was mostly inactive in Debian for the next two years. Upon my return in 2014, I had Vincent Cheng to thank for sponsoring most of my newer work, and for eventually supporting my application to become a Debian Developer. It was around that time that I also attended my first DebConf, in Portland, which remains one of my fondest memories. I had never been to an open-source software conference before, and DebConf14 really knocked it out of the park in so many ways.
After another break, I returned in 2019 to work mostly on Python and machine learning libraries. In 2020, I finally completed a process that I had first started in 2012 but had never managed to finish before: converting cron from source format 1.0 (one big diff) to source format 3.0 (quilt) (a series of patches). This was a process where I converted 25 years worth of organic growth into a minimal series of logically grouped changes (more here). This was my white whale.
In early 2023, shortly after the launch of ChatGPT which triggered an unprecedented AI boom, I started contributing to the Debian ROCm Team, where over the following year, I bootstrapped our CI at ci.rocm.debian.net. Debian's current tooling lack a way to express dependencies on specific hardware other than CPU ISA, nor does it have the means to run autopkgtests using such hardware. To get autopkgtests to make use of AMD GPUs in QEMU VMs and in containers, I had to fork autopkgtest, debci, and a few other components, as well as create a fair share of new tooling for ourselves. This worked out pretty well, and the CI has grown to support 17 different AMD GPU architectures. I will share more on this in upcoming posts.
I have mentioned a few contributors by name, but I have countless others to thank for collaborations over the years. It has been a wonderful experience, and I look forward to many years more.
18 March, 2025 11:16PM by Christian Kastner
As promised on my previous post, on this entry I’ll explain how I’ve set up forgejo actions on the source repository of this site to build it using a runner instead of doing it on the public server using a webhook to trigger the operation.
The first thing I’ve done is to disable the forgejo webhook call that was used to publish the site, as I don’t want to run it anymore.
After that I added a new workflow to the repository that does the following things:
curl to send a notification to an instance of the webhook server installed on the
remote server that triggers a script that updates the site using the git branch.webhook serviceOn the server machine we have installed and configured the webhook service to run a script that updates the site.
To install the application and setup the configuration we have used the following script:
#!/bin/sh
set -e
# ---------
# VARIABLES
# ---------
ARCH="$(dpkg --print-architecture)"
WEBHOOK_VERSION="2.8.2"
DOWNLOAD_URL="https://github.com/adnanh/webhook/releases/download"
WEBHOOK_TGZ_URL="$DOWNLOAD_URL/$WEBHOOK_VERSION/webhook-linux-$ARCH.tar.gz"
WEBHOOK_SERVICE_NAME="webhook"
# Files
WEBHOOK_SERVICE_FILE="/etc/systemd/system/$WEBHOOK_SERVICE_NAME.service"
WEBHOOK_SOCKET_FILE="/etc/systemd/system/$WEBHOOK_SERVICE_NAME.socket"
WEBHOOK_TML_TEMPLATE="/srv/blogops/action/webhook.yml.envsubst"
WEBHOOK_YML="/etc/webhook.yml"
# Config file values
WEBHOOK_USER="$(id -u)"
WEBHOOK_GROUP="$(id -g)"
WEBHOOK_LISTEN_STREAM="172.31.31.1:4444"
# ----
# MAIN
# ----
# Install binary from releases (on Debian only version 2.8.0 is available, but
# I need the 2.8.2 version to support the systemd activation mode).
curl -fsSL -o "/tmp/webhook.tgz" "$WEBHOOK_TGZ_URL"
tar -C /tmp -xzf /tmp/webhook.tgz
sudo install -m 755 "/tmp/webhook-linux-$ARCH/webhook" /usr/local/bin/webhook
rm -rf "/tmp/webhook-linux-$ARCH" /tmp/webhook.tgz
# Service file
sudo sh -c "cat >'$WEBHOOK_SERVICE_FILE'" <<EOF
[Unit]
Description=Webhook server
[Service]
Type=exec
ExecStart=webhook -nopanic -hooks $WEBHOOK_YML
User=$WEBHOOK_USER
Group=$WEBHOOK_GROUP
EOF
# Socket config
sudo sh -c "cat >'$WEBHOOK_SOCKET_FILE'" <<EOF
[Unit]
Description=Webhook server socket
[Socket]
# Set FreeBind to listen on missing addresses (the VPN can be down sometimes)
FreeBind=true
# Set ListenStream to the IP and port you want to listen on
ListenStream=$WEBHOOK_LISTEN_STREAM
[Install]
WantedBy=multi-user.target
EOF
# Config file
BLOGOPS_TOKEN="$(uuid)" \
envsubst <"$WEBHOOK_TML_TEMPLATE" | sudo sh -c "cat >$WEBHOOK_YML"
chmod 0640 "$WEBHOOK_YML"
chwon "$WEBHOOK_USER:$WEBHOOK_GROUP" "$WEBHOOK_YML"
# Restart and enable service
sudo systemctl daemon-reload
sudo systemctl stop "$WEBHOOK_SERVICE_NAME.socket"
sudo systemctl start "$WEBHOOK_SERVICE_NAME.socket"
sudo systemctl enable "$WEBHOOK_SERVICE_NAME.socket"
# ----
# vim: ts=2:sw=2:et:ai:sts=2As seen on the code, we’ve installed the application using a binary from the project repository instead of a package
because we needed the latest version of the application to use systemd with socket activation.
The configuration file template is the following one:
- id: "update-blogops"
execute-command: "/srv/blogops/action/bin/update-blogops.sh"
command-working-directory: "/srv/blogops"
trigger-rule:
match:
type: "value"
value: "$BLOGOPS_TOKEN"
parameter:
source: "header"
name: "X-Blogops-Token"The version on /etc/webhook.yml has the BLOGOPS_TOKEN adjusted to a random value that has to exported as a secret on
the forgejo project (see later).
Once the service is started each time the action is executed the webhook daemon will get a notification and will run
the following update-blogops.sh script to publish the updated version of the site:
#!/bin/sh
set -e
# ---------
# VARIABLES
# ---------
# Values
REPO_URL="ssh://git@forgejo.mixinet.net/mixinet/blogops.git"
REPO_BRANCH="html"
REPO_DIR="public"
MAIL_PREFIX="[BLOGOPS-UPDATE-ACTION] "
# Address that gets all messages, leave it empty if not wanted
MAIL_TO_ADDR="blogops@mixinet.net"
# Directories
BASE_DIR="/srv/blogops"
PUBLIC_DIR="$BASE_DIR/$REPO_DIR"
NGINX_BASE_DIR="$BASE_DIR/nginx"
PUBLIC_HTML_DIR="$NGINX_BASE_DIR/public_html"
ACTION_BASE_DIR="$BASE_DIR/action"
ACTION_LOG_DIR="$ACTION_BASE_DIR/log"
# Files
OUTPUT_BASENAME="$(date +%Y%m%d-%H%M%S.%N)"
ACTION_LOGFILE_PATH="$ACTION_LOG_DIR/$OUTPUT_BASENAME.log"
# ---------
# Functions
# ---------
action_log() {
echo "$(date -R) $*" >>"$ACTION_LOGFILE_PATH"
}
action_check_directories() {
for _d in "$ACTION_BASE_DIR" "$ACTION_LOG_DIR"; do
[ -d "$_d" ] || mkdir "$_d"
done
}
action_clean_directories() {
# Try to remove empty dirs
for _d in "$ACTION_LOG_DIR" "$ACTION_BASE_DIR"; do
if [ -d "$_d" ]; then
rmdir "$_d" 2>/dev/null || true
fi
done
}
mail_success() {
to_addr="$MAIL_TO_ADDR"
if [ "$to_addr" ]; then
subject="OK - updated blogops site"
mail -s "${MAIL_PREFIX}${subject}" "$to_addr" <"$ACTION_LOGFILE_PATH"
fi
}
mail_failure() {
to_addr="$MAIL_TO_ADDR"
if [ "$to_addr" ]; then
subject="KO - failed to update blogops site"
mail -s "${MAIL_PREFIX}${subject}" "$to_addr" <"$ACTION_LOGFILE_PATH"
fi
exit 1
}
# ----
# MAIN
# ----
ret="0"
# Check directories
action_check_directories
# Go to the base directory
cd "$BASE_DIR"
# Remove the old build dir if present
if [ -d "$PUBLIC_DIR" ]; then
rm -rf "$PUBLIC_DIR"
fi
# Update the repository checkout
action_log "Updating the repository checkout"
git fetch --all >>"$ACTION_LOGFILE_PATH" 2>&1 || ret="$?"
if [ "$ret" -ne "0" ]; then
action_log "Failed to update the repository checkout"
mail_failure
fi
# Get it from the repo branch & extract it
action_log "Downloading and extracting last site version using 'git archive'"
git archive --remote="$REPO_URL" "$REPO_BRANCH" "$REPO_DIR" \
| tar xf - >>"$ACTION_LOGFILE_PATH" 2>&1 || ret="$?"
# Fail if public dir was missing
if [ "$ret" -ne "0" ] || [ ! -d "$PUBLIC_DIR" ]; then
action_log "Failed to download or extract site"
mail_failure
fi
# Remove old public_html copies
action_log 'Removing old site versions, if present'
find $NGINX_BASE_DIR -mindepth 1 -maxdepth 1 -name 'public_html-*' -type d \
-exec rm -rf {} \; >>"$ACTION_LOGFILE_PATH" 2>&1 || ret="$?"
if [ "$ret" -ne "0" ]; then
action_log "Removal of old site versions failed"
mail_failure
fi
# Switch site directory
TS="$(date +%Y%m%d-%H%M%S)"
if [ -d "$PUBLIC_HTML_DIR" ]; then
action_log "Moving '$PUBLIC_HTML_DIR' to '$PUBLIC_HTML_DIR-$TS'"
mv "$PUBLIC_HTML_DIR" "$PUBLIC_HTML_DIR-$TS" >>"$ACTION_LOGFILE_PATH" 2>&1 ||
ret="$?"
fi
if [ "$ret" -eq "0" ]; then
action_log "Moving '$PUBLIC_DIR' to '$PUBLIC_HTML_DIR'"
mv "$PUBLIC_DIR" "$PUBLIC_HTML_DIR" >>"$ACTION_LOGFILE_PATH" 2>&1 ||
ret="$?"
fi
if [ "$ret" -ne "0" ]; then
action_log "Site switch failed"
mail_failure
else
action_log "Site updated successfully"
mail_success
fi
# ----
# vim: ts=2:sw=2:et:ai:sts=2hugo-adoc workflowThe workflow is defined in the .forgejo/workflows/hugo-adoc.yml file and looks like this:
name: hugo-adoc
# Run this job on push events to the main branch
on:
push:
branches:
- 'main'
jobs:
build-and-push:
if: ${{ vars.BLOGOPS_WEBHOOK_URL != '' && secrets.BLOGOPS_TOKEN != '' }}
runs-on: docker
container:
image: forgejo.mixinet.net/oci/hugo-adoc:latest
# Allow the job to write to the repository (not really needed on forgejo)
permissions:
contents: write
steps:
- name: Checkout the repo
uses: actions/checkout@v4
with:
submodules: 'true'
- name: Build the site
shell: sh
run: |
rm -rf public
hugo
- name: Push compiled site to html branch
shell: sh
run: |
# Set the git user
git config --global user.email "blogops@mixinet.net"
git config --global user.name "BlogOps"
# Create a new orphan branch called html (it was not pulled by the
# checkout step)
git switch --orphan html
# Add the public directory to the branch
git add public
# Commit the changes
git commit --quiet -m "Updated site @ $(date -R)" public
# Push the changes to the html branch
git push origin html --force
# Switch back to the main branch
git switch main
- name: Call the blogops update webhook endpoint
shell: sh
run: |
HEADER="X-Blogops-Token: ${{ secrets.BLOGOPS_TOKEN }}"
curl --fail -k -H "$HEADER" ${{ vars.BLOGOPS_WEBHOOK_URL }}The only relevant thing is that we have to add the BLOGOPS_TOKEN variable to the project secrets (its value is the one
included on the /etc/webhook.yml file created when installing the webhook service) and the BLOGOPS_WEBHOOK_URL
project variable (its value is the URL of the webhook server, in my case
http://172.31.31.1:4444/hooks/update-blogops); note that the job includes the -k flag on the curl command just in
case I end up using TLS on the webhook server in the future, as discussed previously.
Now that I have forgejo actions on my server I no longer need to build the site on the public server as I did initially, a good thing when the server is a small OVH VPS that only runs a couple of containers and a web server directly on the host.
I’m still using a notification system to make the server run a script to update the site because that way the forgejo
server does not need access to the remote machine shell, only the webhook server which, IMHO, is a more secure setup.
When I a few days ago discovered that a security problem reported against the theora library last year was still not fixed, and because I was already up to speed on Xiph development, I decided it was time to wrap up a new theora release. This new release was tagged in the Xiph gitlab theora instance Saturday. You can fetch the new release from the Theora home page.
The list of changes since The 1.2.0alpha1 release from the CHANGES file in the tarball look like this:
libteora 1.2.0beta1 (2025 March 15)
- Bumped minor SONAME versions as methods changed constness of arguments.
- Updated libogg dependency to version 1.3.4 for ogg_uint64_t.
- Updated doxygen setup.
- Updated autotools setup and support scripts (#1467 #1800 #1987 #2318 #2320).
- Added support for RISC OS.
- Fixed mingw build (#2141).
- Improved ARM support.
- Converted SCons setup to work with Python 3.
- Introduced new configure options --enable-mem-constraint and --enable-gcc-sanitizers.
- Fixed all known compiler warnings and errors from gcc and clang.
- Improved examples for stability and correctness.
- Variuos speed, bug fixes and code quality improvements.
- Fixed build problem with Visual Studio (#2317).
- Avoids undefined bit shift of signed numbers (#2321, #2322).
- Avoids example encoder crash on bogus audio input (#2305).
- Fixed musl linking issue with asm enabled (#2287).
- Fixed some broken clamping in rate control (#2229).
- Added NULL check _tc and _setup even for data packets (#2279).
- Fixed mismatched oc_mb_fill_cmapping11 signature (#2068).
- Updated the documentation for theora_encode_comment() (#726).
- Adjusted build to Only link libcompat with dump_video (#1587).
- Corrected an operator precedence error in the visualization code (#1751).
- Fixed two spelling errors in the comments (#1804).
- Avoid negative bit shift operation in huffdec.c (CVE-2024-56431).
- Improved library documentation and specification text.
- Adjusted library dependencies so libtheoraenc do not depend on libtheoradec.
- Handle fallout from CVE-2017-14633 in libvorbis, check return value in encoder_example and transcoder_example.
There are a few bugs still being investigated, and my plan is to wrap up a final 1.2.0 release two weekends from now.
As usual, if you use Bitcoin and want to show your support of my activities, please send Bitcoin donations to my address 15oWEoG9dUPovwmUL9KWAnYRtNJEkP1u1b.
There's an issue in the Eigen linear algebra library where linking together objects compiled with different flags causes the resulting binary to crash. Some details are written-up in this mailing list thread.
I just encountered a situation where a large application sometimes crashes for
unknown reasons, and needed a method to determine whether this Eigen issue could
be the cause. I ended up doing this by using the DWARF data to see if the linked
binary contains the different incompatible flavors of malloc / free or not.
I downloaded the small demo program showing the problem. I built it:
CCXXXFLAGS=-g make
Here if you run ./main, the bug is triggered, and a crash occurs. I looked at
the debug info for the code in question:
for o (main lib.so) { echo "======== $o"; readelf --debug-dump=decodedline $o \ | awk \ '$1 ~ /^Memory.h/ { if(180 <= $2 && $2 <= 186) { have["malloc_glibc"]=1 } if(188 == $2) { have["malloc_handmade"]=1 } if(201 <= $2 && $2 <= 204) { have["free_glibc"]=1 } if(206 == $2) { have["free_handmade"]=1 } } END { for (var in have) { print(var); } }' }
It says:
======== main free_handmade ======== lib.so malloc_glibc free_glibc
Here I looked at main and lib.so (the build products from this little demo).
In a real case you'd look at every shared library linked into the binary and the
binary itself. On my machine /usr/include/eigen3/Eigen/src/Core/util/Memory.h
looks like this, starting on line 174:
174 EIGEN_DEVICE_FUNC inline void* aligned_malloc(std::size_t size)
175 {
176 check_that_malloc_is_allowed();
177
178 void *result;
179 #if (EIGEN_DEFAULT_ALIGN_BYTES==0) || EIGEN_MALLOC_ALREADY_ALIGNED
180
181 EIGEN_USING_STD(malloc)
182 result = malloc(size);
183
184 #if EIGEN_DEFAULT_ALIGN_BYTES==16
185 eigen_assert((size<16 || (std::size_t(result)%16)==0) && "System's malloc returned an unaligned pointer. Compile with EIGEN_MALLOC_ALREADY_ALIGNED=0 to fallback to handmade aligned memory allocator.");
186 #endif
187 #else
188 result = handmade_aligned_malloc(size);
189 #endif
190
191 if(!result && size)
192 throw_std_bad_alloc();
193
194 return result;
195 }
196
197 /** \internal Frees memory allocated with aligned_malloc. */
198 EIGEN_DEVICE_FUNC inline void aligned_free(void *ptr)
199 {
200 #if (EIGEN_DEFAULT_ALIGN_BYTES==0) || EIGEN_MALLOC_ALREADY_ALIGNED
201
202 EIGEN_USING_STD(free)
203 free(ptr);
204
205 #else
206 handmade_aligned_free(ptr);
207 #endif
208 }
The above awk script looks at the two malloc paths and the two free paths, and
we can clearly see that it only ever calls malloc_glibc(), but has both
flavors of free(). So this can crash. We want to see that the whole executable
(shared libraries and all) should only have one type of malloc() and free(),
and that would guarantee no crashing.
There are a more functions in that header that should be instrumented
(realloc() for instance) and the different alignment paths should be
instrumented similarly (as described in the mailing list thread above), but here
we see that this technique works.
18 March, 2025 03:52AM by Dima Kogan
Last week I decided I wanted to try out forgejo actions to build this blog instead of using webhooks, so I looked the documentation and started playing with it until I had it working as I wanted.
This post is to describe how I’ve installed and configured a forgejo runner, how I’ve added an oci organization to my instance to build, publish and mirror container images and added a couple of additional organizations (actions and docker for now) to mirror interesting actions.
The changes made to build the site using actions will be documented on a separate post, as I’ll be using this entry to test the new setup on the blog project.
The first thing I’ve done is to install a runner on my server, I decided to use the OCI image installation method, as it seemed to be the easiest and fastest one.
The commands I’ve used to setup the runner are the following:
$ cd /srv
$ git clone https://forgejo.mixinet.net/blogops/forgejo-runner.git
$ cd forgejo-runner
$ sh ./bin/setup-runner.shThe setup-runner.sh script does multiple things:
forgejo-runner user and group.runner file with a predefined secret and the docker labelThe setup-runner.sh code is available here.
After running the script the runner has to be registered with the forgejo server, it can be done using the following command:
$ forgejo forgejo-cli actions register --name "$RUNNER_NAME" \
--secret "$FORGEJO_SECRET"The RUNNER_NAME variable is defined on the setup-runner.sh script and the FORGEJO_SECRET must match the value used
on the .runner file.
docker-composeTo launch the runner I’m going to use a docker-compose.yml file that starts two containers, a docker in docker
service to run the containers used by the workflow jobs and another one that runs the forgejo-runner itself.
The initial version used a TCP port to communicate with the dockerd server from the runner, but when I tried to build
images from a workflow I noticed that the containers launched by the runner were not going to be able to execute
another dockerd inside the dind one and, even if they were, it was going to be expensive computationally.
To avoid the issue I modified the dind service to use a unix socket on a shared volume that can be used by the
runner service to communicate with the daemon and also re-shared with the job containers so the dockerd server can
be used from them to build images.
The use of the same docker server that runs the jobs from them has security implications, but this instance is
for a home server where I am the only user, so I am not worried about it and this way I can save some resources (in
fact, I could use the host docker server directly instead of using a dind service, but just in case I want to run
other containers on the host I prefer to keep the one used for the runner isolated from it).
For those concerned about sharing the same server an alternative would be to launch a second dockerd only for the jobs
(i.e. actions-dind) using the same approach (the volume with its socket will have to be shared with the runner
service so it can be re-shared, but the runner does not need to use it).
The final docker-compose.yaml file is as follows:
services:
dind:
image: docker:dind
container_name: 'dind'
privileged: 'true'
command: ['dockerd', '-H', 'unix:///dind/docker.sock', '-G', '$RUNNER_GID']
restart: 'unless-stopped'
volumes:
- ./dind:/dind
runner:
image: 'data.forgejo.org/forgejo/runner:6.2.2'
links:
- dind
depends_on:
dind:
condition: service_started
container_name: 'runner'
environment:
DOCKER_HOST: 'unix:///dind/docker.sock'
user: $RUNNER_UID:$RUNNER_GID
volumes:
- ./config.yaml:/config.yaml
- ./data:/data
- ./dind:/dind
restart: 'unless-stopped'
command: '/bin/sh -c "sleep 5; forgejo-runner daemon -c /config.yaml"'There are multiple things to comment about this file:
dockerd server is started with the -H unix:///dind/docker.sock flag to use the unix socket to communicate
with the daemon instead of using a TCP port (as said, it is faster and allows us to share the socket with the
containers started by the runner).dockerd daemon with the RUNNER_GID group so the runner can communicate with it (the socket
gets that group which is the same used by the runner).data directory, the dind folder where docker creates the unix
socket and a config.yaml file used by us to change the default runner configuration.The config.yaml file was originally created using the forgejo-runner:
$ docker run --rm data.forgejo.org/forgejo/runner:6.2.2 \
forgejo-runner generate-config > config.yamlThe changes to it are minimal, the runner capacity has been increased to 2 (that allows it to run two jobs at the
same time) and the /dind/docker.sock value has been added to the valid_volumes key to allow the containers launched
by the runner to mount it when needed; the diff against the default version is as follows:
@@ -13,7 +13,8 @@
# Where to store the registration result.
file: .runner
# Execute how many tasks concurrently at the same time.
- capacity: 1
+ # STO: Allow 2 concurrent tasks
+ capacity: 2
# Extra environment variables to run jobs.
envs:
A_TEST_ENV_NAME_1: a_test_env_value_1
@@ -87,7 +88,9 @@
# If you want to allow any volume, please use the following configuration:
# valid_volumes:
# - '**'
- valid_volumes: []
+ # STO: Allow to mount the /dind/docker.sock on the containers
+ valid_volumes:
+ - /dind/docker.sock
# overrides the docker client host with the specified one.
# If "-" or "", an available docker host will automatically be found.
# If "automount", an available docker host will automatically be found and ...To start the runner we export the RUNNER_UID and RUNNER_GID variables and call docker-compose up to start the
containers on the background:
$ RUNNER_UID="$(id -u forgejo-runner)" RUNNER_GID="$(id -g forgejo-runner)" \
docker compose up -dIf the server was configured right we are now able to start using actions with this runner.
To avoid unnecessary network traffic we are going to create a multiple organizations in our forgejo instance to maintain our own actions and container images and mirror remote ones.
The rationale behind the mirror use is that we reduce a lot the need to connect to remote servers to download the actions and images, which is good for performance and security reasons.
In fact, we are going to build our own images for some things to install the tools we want without needing to do it over and over again on the workflow jobs.
The actions we are mirroring are on the actions and docker organizations, we have
created the following ones for now (the mirrors were created using the forgejo web interface and we have disabled
manually all the forgejo modules except the code one for them):
To use our actions by default (i.e., without needing to add the server URL on the uses keyword) we have added the
following section to the app.ini file of our forgejo server:
[actions]
ENABLED = true
DEFAULT_ACTIONS_URL = https://forgejo.mixinet.netTo be able to push images to the oci organization I’ve created a token with package:write permission for my own
user because I’m a member of the organization and I’m authorized to publish packages on it (a different user could be
created, but as I said this is for personal use, so there is no need to complicate things for now).
To allow the use of those credentials on the actions I have added a secret (REGISTRY_PASS) and a variable
(REGISTRY_USER) to the oci organization to allow the actions to use them.
I’ve also logged myself on my local docker client to be able to push images to the oci group by hand, as I it is
needed for bootstrapping the system (as I’m using local images on the worflows I need to push them to the server before
running the ones that are used to build the images).
Our images will be stored on the packages section of a new organization called oci, inside it we have created two projects that use forgejo actions to keep things in shape:
On the next sections we are going to describe the actions and images we have created and mirrored from those projects.
oci/images projectThe images project is a monorepo that contains the source files for the images we are going to build and a couple of
actions.
The image sources are on sub directories of the repository, to be considered an image the folder has to contain a
Dockerfile that will be used to build the image.
The repository has two workflows:
build-image-from-tag: Workflow to build, tag and push an image to the oci organizationmulti-semantic-release: Workflow to create tags for the images using the multi-semantic-release tool.As the workflows are already configured to use some of our images we pushed some of them from a checkout of the repository using the following commands:
registry="forgejo.mixinet.net/oci"
for img in alpine-mixinet node-mixinet multi-semantic-release; do
docker build -t $registry/$img:1.0.0 $img
docker tag $registry/$img:1.0.0 $registry/$img:latest
docker push $registry/$img:1.0.0
docker push $registry/$img:latest
doneOn the next sub sections we will describe what the workflows do and will show their source code.
build-image-from-tag workflowThis workflow uses a docker client to build an image from a tag on the repository with the format
image-name-v[0-9].[0-9].[0-9]+.
As the runner is executed on a container (instead of using lxc) it seemed unreasonable to run another dind
container from that one, that is why, after some tests, I decided to share the dind service server socket with the
runner container and enabled the option to mount it also on the containers launched by the runner when needed (I only
do it on the build-image-from-tag action for now).
The action was configured to run using a trigger or when new tags with the right format were created, but when the tag
is created by multi-semantic-release the trigger does not work for some reason, so now it only runs the job on
triggers and checks if it is launched for a tag with the right format on the job itself.
The source code of the action is as follows:
name: build-image-from-tag
on:
workflow_dispatch:
jobs:
build:
# Don't build the image if the registry credentials are not set, the ref is not a tag or it doesn't contain '-v'
if: ${{ vars.REGISTRY_USER != '' && secrets.REGISTRY_PASS != '' && startsWith(github.ref, 'refs/tags/') && contains(github.ref, '-v') }}
runs-on: docker
container:
image: forgejo.mixinet.net/oci/node-mixinet:latest
# Mount the dind socket on the container at the default location
options: -v /dind/docker.sock:/var/run/docker.sock
steps:
- name: Extract image name and tag from git and get registry name from env
id: job_data
run: |
echo "::set-output name=img_name::${GITHUB_REF_NAME%%-v*}"
echo "::set-output name=img_tag::${GITHUB_REF_NAME##*-v}"
echo "::set-output name=registry::$(
echo "${{ github.server_url }}" | sed -e 's%https://%%'
)"
echo "::set-output name=oci_registry_prefix::$(
echo "${{ github.server_url }}/oci" | sed -e 's%https://%%'
)"
- name: Checkout the repo
uses: actions/checkout@v4
- name: Export build dir and Dockerfile
id: build_data
run: |
img="${{ steps.job_data.outputs.img_name }}"
build_dir="$(pwd)/${img}"
dockerfile="${build_dir}/Dockerfile"
if [ -f "$dockerfile" ]; then
echo "::set-output name=build_dir::$build_dir"
echo "::set-output name=dockerfile::$dockerfile"
else
echo "Couldn't find the Dockerfile for the '$img' image"
exit 1
fi
- name: Login to the Container Registry
uses: docker/login-action@v3
with:
registry: ${{ steps.job_data.outputs.registry }}
username: ${{ vars.REGISTRY_USER }}
password: ${{ secrets.REGISTRY_PASS }}
- name: Set up Docker Buildx
uses: docker/setup-buildx-action@v3
- name: Build and Push
uses: docker/build-push-action@v6
with:
push: true
tags: |
${{ steps.job_data.outputs.oci_registry_prefix }}/${{ steps.job_data.outputs.img_name }}:${{ steps.job_data.outputs.img_tag }}
${{ steps.job_data.outputs.oci_registry_prefix }}/${{ steps.job_data.outputs.img_name }}:latest
context: ${{ steps.build_data.outputs.build_dir }}
file: ${{ steps.build_data.outputs.dockerfile }}
build-args: |
OCI_REGISTRY_PREFIX=${{ steps.job_data.outputs.oci_registry_prefix }}/Some notes about this code:
if condition of the build job is not perfect, but it is good enough to avoid wrong uses as long as nobody
uses manual tags with the wrong format and expects things to work (it checks if the REGISTRY_USER and
REGISTRY_PASS variables are set, if the ref is a tag and if it contains the -v string).dind socket we mount it on the container using the options key on the container section
of the job (this only works if supported by the runner configuration as explained before).job_data step to get information about the image from the tag and the registry URL from the environment
variables, it is executed first because all the information is available without checking out the repository.job_data step to get the build dir and Dockerfile paths from the repository (right now we are
assuming fixed paths and checking if the Dockerfile exists, but in the future we could use a configuration file to
get them, if needed).OCI_REGISTRY_PREFIX build argument to the Dockerfile to be able to use it
on the FROM instruction (we are using it in our images).multi-semantic-release workflowThis workflow is used to run the multi-semantic-release tool on pushes to the main branch.
It is configured to create the configuration files on the fly (it prepares things to tag the folders that contain a
Dockerfile using a couple of template files available on the repository’s .forgejo directory) and run the
multi-semantic-release tool to create tags and push them to the repository if new versions are to be built.
Initially we assumed that the tag creation pushed by multi-semantic-release would be enough to run the
build-tagged-image-task action, but as it didn’t work we removed the rule to run the action on tag creation and added
code to trigger the action using an api call for the newly created tags (we get them from the output of the
multi-semantic-release execution).
The source code of the action is as follows:
name: multi-semantic-release
on:
push:
branches:
- 'main'
jobs:
multi-semantic-release:
runs-on: docker
container:
image: forgejo.mixinet.net/oci/multi-semantic-release:latest
steps:
- name: Checkout the repo
uses: actions/checkout@v4
- name: Generate multi-semantic-release configuration
shell: sh
run: |
# Get the list of images to work with (the folders that have a Dockerfile)
images="$(for img in */Dockerfile; do dirname "$img"; done)"
# Generate a values.yaml file for the main packages.json file
package_json_values_yaml=".package.json-values.yaml"
echo "images:" >"$package_json_values_yaml"
for img in $images; do
echo " - $img" >>"$package_json_values_yaml"
done
echo "::group::Generated values.yaml for the project"
cat "$package_json_values_yaml"
echo "::endgroup::"
# Generate the package.json file validating that is a good json file with jq
tmpl -f "$package_json_values_yaml" ".forgejo/package.json.tmpl" | jq . > "package.json"
echo "::group::Generated package.json for the project"
cat "package.json"
echo "::endgroup::"
# Remove the temporary values file
rm -f "$package_json_values_yaml"
# Generate the package.json file for each image
for img in $images; do
tmpl -v "img_name=$img" -v "img_path=$img" ".forgejo/ws-package.json.tmpl" | jq . > "$img/package.json"
echo "::group::Generated package.json for the '$img' image"
cat "$img/package.json"
echo "::endgroup::"
done
- name: Run multi-semantic-release
shell: sh
run: |
multi-semantic-release | tee .multi-semantic-release.log
- name: Trigger builds
shell: sh
run: |
# Get the list of tags published on the previous steps
tags="$(
sed -n -e 's/^\[.*\] \[\(.*\)\] .* Published release \([0-9]\+\.[0-9]\+\.[0-9]\+\) on .*$/\1-v\2/p' \
.multi-semantic-release.log
)"
rm -f .multi-semantic-release.log
if [ "$tags" ]; then
# Prepare the url for building the images
workflow="build-image-from-tag.yaml"
dispatch_url="${{ github.api_url }}/repos/${{ github.repository }}/actions/workflows/$workflow/dispatches"
echo "$tags" | while read -r tag; do
echo "Triggering build for tag '$tag'"
curl \
-H "Content-Type:application/json" \
-H "Authorization: token ${{ secrets.GITHUB_TOKEN }}" \
-d "{\"ref\":\"$tag\"}" "$dispatch_url"
done
fiNotes about this code:
tmpl tool to process the multi-semantic-release configuration templates comes from previous uses,
but on this case we could use a different approach (i.e. envsubst could be used) but we left it because it keeps
things simple and can be useful in the future if we want to do more complex things with the template files.tee to show and dump to a file the output of the multi-semantic-release execution.tags using sed against the output of the multi-semantic-release execution and for
each one found we use curl to call the forgejo API to trigger the build job; as the call is against the same
project we can use the GITHUB_TOKEN generated for the workflow to do it, without creating a user token that has to
be shared as a secret.The .forgejo/package.json.tmpl file is the following one:
{
"name": "multi-semantic-release",
"version": "0.0.0-semantically-released",
"private": true,
"multi-release": {
"tagFormat": "${name}-v${version}"
},
"workspaces": {{ .images | toJson }}
}As can be seen it only needs a list of paths to the images as argument (the file we generate contains the names and paths, but it could be simplified).
And the .forgejo/ws-package.json.tmpl file is the following one:
{
"name": "{{ .img_name }}",
"license": "UNLICENSED",
"release": {
"plugins": [
[
"@semantic-release/commit-analyzer",
{
"preset": "conventionalcommits",
"releaseRules": [
{ "breaking": true, "release": "major" },
{ "revert": true, "release": "patch" },
{ "type": "feat", "release": "minor" },
{ "type": "fix", "release": "patch" },
{ "type": "perf", "release": "patch" }
]
}
],
[
"semantic-release-replace-plugin",
{
"replacements": [
{
"files": [ "{{ .img_path }}/msr.yaml" ],
"from": "^version:.*$",
"to": "version: ${nextRelease.version}",
"allowEmptyPaths": true
}
]
}
],
[
"@semantic-release/git",
{
"assets": [ "msr.yaml" ],
"message": "ci(release): {{ .img_name }}-v${nextRelease.version}\n\n${nextRelease.notes}"
}
]
],
"branches": [ "main" ]
}
}oci/mirrors projectThe repository contains a template for the configuration file we are going to use with regsync
(regsync.envsubst.yml) to mirror images from remote registries using a workflow that generates a configuration file
from the template and runs the tool.
The initial version of the regsync.envsubst.yml file is prepared to mirror alpine containers from version 3.21 to
3.29 (we explicitly remove version 3.20) and needs the forgejo.mixinet.net/oci/node-mixinet:latest image to run
(as explained before it was pushed manually to the server):
version: 1
creds:
- registry: "$REGISTRY"
user: "$REGISTRY_USER"
pass: "$REGISTRY_PASS"
sync:
- source: alpine
target: $REGISTRY/oci/alpine
type: repository
tags:
allow:
- "latest"
- "3\\.2\\d+"
- "3\\.2\\d+.\\d+"
deny:
- "3\\.20"
- "3\\.20.\\d+"mirror workflowThe mirror workflow creates a configuration file replacing the value of the REGISTRY environment variable (computed
by removing the protocol from the server_url), the REGISTRY_USER organization value and the REGISTRY_PASS secret
using the envsubst command and running the regsync tool to mirror the images using the configuration file.
The action is configured to run daily, on push events when the regsync.envsubst.yml file is modified on the main
branch and can also be triggered manually.
The source code of the action is as follows:
name: mirror
on:
schedule:
- cron: '@daily'
push:
branches:
- main
paths:
- 'regsync.envsubst.yml'
workflow_dispatch:
jobs:
mirror:
if: ${{ vars.REGISTRY_USER != '' && secrets.REGISTRY_PASS != '' }}
runs-on: docker
container:
image: forgejo.mixinet.net/oci/node-mixinet:latest
steps:
- name: Checkout
uses: actions/checkout@v4
- name: Sync images
run: |
REGISTRY="$(echo "${{ github.server_url }}" | sed -e 's%https://%%')" \
REGISTRY_USER="${{ vars.REGISTRY_USER }}" \
REGISTRY_PASS="${{ secrets.REGISTRY_PASS }}" \
envsubst <regsync.envsubst.yml >.regsync.yml
regsync --config .regsync.yml once
rm -f .regsync.ymlWe have installed a forgejo-runner and configured it to run actions for our own server and things are working fine.
This approach allows us to have a powerful CI/CD system on a modest home server, something very useful for maintaining personal projects and playing with things without needing SaaS platforms like github or gitlab.
FreedomBox is a Debian blend that makes it easier to run your own server. Approximately every two years, there is a new stable release of Debian. This year’s release will be called Debian 13 "trixie".
This post will provide an overview of changes between FreedomBox 23.6 (the version that shipped in Debian 12 "bookworm") and 25.5 (the latest release). Note: Debian 13 "trixie" is not yet released, so things may still change, be added or removed, before the official release.
Over the past two years, FreedomBox has been increasing the number of features and applications available to its users. We have also focused on improving the reliability of the system, detecting unexpected situations, and providing means to return to a known good state. With these improvements, FreedomBox has become a good solution for people with limited time or energy to set up and start running a personal server, at home or in the cloud.
Looking forward, we would like to focus on making more powerful hardware available with FreedomBox pre-installed and ready to be used. This hardware would also support larger storage devices, making it suitable as a NAS or media server. We are also very interested in exploring new features such as atomic updates, which will further enhance the reliability of the system.
17 March, 2025 12:40PM by James Valleroy
An offline PKI enhances security by physically isolating the certificate authority from network threats. A YubiKey is a low-cost solution to store a root certificate. You also need an air-gapped environment to operate the root CA.
This post describes an offline PKI system using the following components:
It is possible to add more YubiKeys as a backup of the root CA if needed. This is not needed for the intermediate CA as you can generate a new one if the current one gets destroyed.
offline-pki is a small Python application to manage an offline PKI.
It relies on yubikey-manager to manage YubiKeys and cryptography for
cryptographic operations not executed on the YubiKeys. The application has some
opinionated design choices. Notably, the cryptography is hard-coded to use NIST
P-384 elliptic curve.
The first step is to reset all your YubiKeys:
$ offline-pki yubikey reset This will reset the connected YubiKey. Are you sure? [y/N]: y New PIN code: Repeat for confirmation: New PUK code: Repeat for confirmation: New management key ('.' to generate a random one): WARNING[pki-yubikey] Using random management key: e8ffdce07a4e3bd5c0d803aa3948a9c36cfb86ed5a2d5cf533e97b088ae9e629 INFO[pki-yubikey] 0: Yubico YubiKey OTP+FIDO+CCID 00 00 INFO[pki-yubikey] SN: 23854514 INFO[yubikit.management] Device config written INFO[yubikit.piv] PIV application data reset performed INFO[yubikit.piv] Management key set INFO[yubikit.piv] New PUK set INFO[yubikit.piv] New PIN set INFO[pki-yubikey] YubiKey reset successful!
Then, generate the root CA and create as many copies as you want:
$ offline-pki certificate root --permitted example.com Management key for Root X: Plug YubiKey "Root X"... INFO[pki-yubikey] 0: Yubico YubiKey CCID 00 00 INFO[pki-yubikey] SN: 23854514 INFO[yubikit.piv] Data written to object slot 0x5fc10a INFO[yubikit.piv] Certificate written to slot 9C (SIGNATURE), compression=True INFO[yubikit.piv] Private key imported in slot 9C (SIGNATURE) of type ECCP384 Copy root certificate to another YubiKey? [y/N]: y Plug YubiKey "Root X"... INFO[pki-yubikey] 0: Yubico YubiKey CCID 00 00 INFO[pki-yubikey] SN: 23854514 INFO[yubikit.piv] Data written to object slot 0x5fc10a INFO[yubikit.piv] Certificate written to slot 9C (SIGNATURE), compression=True INFO[yubikit.piv] Private key imported in slot 9C (SIGNATURE) of type ECCP384 Copy root certificate to another YubiKey? [y/N]: n
You can inspect the result:
$ offline-pki yubikey info INFO[pki-yubikey] 0: Yubico YubiKey CCID 00 00 INFO[pki-yubikey] SN: 23854514 INFO[pki-yubikey] Slot 9C (SIGNATURE): INFO[pki-yubikey] Private key type: ECCP384 INFO[pki-yubikey] Public key: INFO[pki-yubikey] Algorithm: secp384r1 INFO[pki-yubikey] Issuer: CN=Root CA INFO[pki-yubikey] Subject: CN=Root CA INFO[pki-yubikey] Serial: 1 INFO[pki-yubikey] Not before: 2024-07-05T18:17:19+00:00 INFO[pki-yubikey] Not after: 2044-06-30T18:17:19+00:00 INFO[pki-yubikey] PEM: -----BEGIN CERTIFICATE----- MIIBcjCB+aADAgECAgEBMAoGCCqGSM49BAMDMBIxEDAOBgNVBAMMB1Jvb3QgQ0Ew HhcNMjQwNzA1MTgxNzE5WhcNNDQwNjMwMTgxNzE5WjASMRAwDgYDVQQDDAdSb290 IENBMHYwEAYHKoZIzj0CAQYFK4EEACIDYgAERg3Vir6cpEtB8Vgo5cAyBTkku/4w kXvhWlYZysz7+YzTcxIInZV6mpw61o8W+XbxZV6H6+3YHsr/IeigkK04/HJPi6+i zU5WJHeBJMqjj2No54Nsx6ep4OtNBMa/7T9foyMwITAPBgNVHRMBAf8EBTADAQH/ MA4GA1UdDwEB/wQEAwIBhjAKBggqhkjOPQQDAwNoADBlAjEAwYKy/L8leJyiZSnn xrY8xv8wkB9HL2TEAI6fC7gNc2bsISKFwMkyAwg+mKFKN2w7AjBRCtZKg4DZ2iUo 6c0BTXC9a3/28V5aydZj6rvx0JqbF/Ln5+RQL6wFMLoPIvCIiCU= -----END CERTIFICATE-----
Then, you can create an intermediate certificate with offline-pki yubikey
intermediate and use it to sign certificates by providing a CSR to offline-pki
certificate sign. Be careful and inspect the CSR before signing it, as only the
subject name can be overridden. Check the documentation for more details.
Get the available options using the --help flag.
To ensure the operations on the root and intermediate CAs are air-gapped, a cost-efficient solution is to use an ARM64 single board computer. The Libre Computer Sweet Potato SBC is a more open alternative to the well-known Raspberry Pi.1
I interact with it through an USB to TTL UART converter:
$ tio /dev/ttyUSB0 [16:40:44.546] tio v3.7 [16:40:44.546] Press ctrl-t q to quit [16:40:44.555] Connected to /dev/ttyUSB0 GXL:BL1:9ac50e:bb16dc;FEAT:ADFC318C:0;POC:1;RCY:0;SPI:0;0.0;CHK:0; TE: 36574 BL2 Built : 15:21:18, Aug 28 2019. gxl g1bf2b53 - luan.yuan@droid15-sz set vcck to 1120 mv set vddee to 1000 mv Board ID = 4 CPU clk: 1200MHz […]
To bring everything together, I am using Nix with a Flake providing:
offline-pki application, with shell completion,offline-pki application,# Execute the application locally nix run github:vincentbernat/offline-pki -- --help # Run the application inside a QEMU VM nix run github:vincentbernat/offline-pki\#qemu # Build a SD card for the Sweet Potato or for the Raspberry Pi nix build --system aarch64-linux github:vincentbernat/offline-pki\#sdcard.potato nix build --system aarch64-linux github:vincentbernat/offline-pki\#sdcard.generic # Get a development shell with the application nix develop github:vincentbernat/offline-pki
The key for the root CA is not generated by the YubiKey. Using an air-gapped computer is all the more important. Put it in a safe with the YubiKeys when done! ↩︎
17 March, 2025 08:12AM by Vincent Bernat
Google tracking everything we read is bad, particularly since Google abandoned the “don’t be evil” plan and are presumably open to being somewhat evil.
The article recommendations on Chrome on Android are useful and I’d like to be able to get the same quality of recommendations without Google knowing about everything I read. Ideally without anything other than the device I use knowing what interests me.
A ML system to map between sources of news that are of interest should be easy to develop and run on end user devices. The model could be published and when given inputs of articles you like give an output of sites that contain other articles you like. Then an agent on the end user system could spider the sites in question and run a local model to determine which articles to present to the user.
Mapping for hate following is possible for such a system (Google doesn’t do that), the user could have 2 separate model runs for regular reading and hate-following and determine how much of each content to recommend. It could also give negative weight to entries that match the hate criteria.
Some sites with articles (like Medium) give an estimate of reading time. An article recommendation system should have a fixed limit of articles (both in articles and in reading time) to support the “I spend half an hour reading during lunch” model not doom scrolling.
For getting news using only FOSS it seems that the best option at the moment is to use the Lemmy FOSS social network which is like Reddit [1] to recommend articles etc.
The Lemoa client for Lemmy uses GTK [2] but it’s no longer maintained. The Lemonade client for Lemmy is written in Rust [3]. It would be good if one of those was packaged for Debian, preferably one that’s maintained.
16 March, 2025 04:19AM by etbe
Debian Med sprint in Berlin on 15 and 16 February
The Debian Med team works on software packages that are associated with medicine, pre-clinical research, and life sciences, and makes them available for the Debian distribution. Seven Debian developers and contributors to the team gathered for their annual Sprint, in Berlin, Germany on 15 and 16 February 2025. The purpose of the meeting was to tackle bugs in Debian-Med packages, enhance the quality of the team's packages, and coordinate the efforts of team members overall.
This sprint allowed participants to fix dozens of bugs, including release-critical ones. New upstream versions were uploaded, and the participants took some time to modernize some packages. Additionally, they discussed the long-term goals of the team, prepared a forthcoming invited talk for a conference, and enjoyed working together.
More details on the event and individual agendas/reports can be found at https://wiki.debian.org/Sprints/2025/DebianMed.
15 March, 2025 11:00PM by Pierre Gruet, Jean-Pierre Giraud, Joost van Baal-Ilić
KDE MascotThank you everyone for keeping the lights on for a bit longer. KDE snaps have been restored. I also released 24.12.3! In addition, I have moved “most” snaps to core24. The remaining snaps need newer qt6/kf6, which is a WIP. “The Bad luck girl” has been hit once again with another loss, so with that, I will be reducing my hours on snaps while I consider my options for my future. I am still around, just a bit less.
Thanks again everyone, if you can get me through one more ( lingering broken arm ) surgery I would be forever grateful! https://gofund.me/d5d59582
15 March, 2025 12:52PM by sgmoore
Last year, I attended the annual LibreOffice Conference in Luxembourg with the help of a generous travel grant by The Document Foundation (TDF). It was a three-day event from the 10th to the 12th of October 2024, with an additional day for community meetup on the 9th.
Luxembourg is a small (twice as big as Delhi) country in Western Europe. After going through an arduous visa process, I reached Luxembourg on the 8th of October. Upon arriving in Luxembourg, I took a bus to the city center, where my hotel — Park Inn — was located. All the public transport in Luxembourg was free of cost. It was as if I stepped in another world. There were separate tracks for cycling and a separate lane for buses, along with good pedestrian infrastructure. In addition, the streets were pretty neat and clean.
My hotel was 20 km from the conference venue in Belval. However, the commute was convenient due to a free of cost train connection, which were comfortable, smooth, and scenic, covering the distance in half an hour. The hotel included a breakfast buffet, recharging us before the conference.
Pre-conference, a day was reserved for the community meetup on the 9th of October. On that day, the community members introduced themselves and their contributions to the LibreOffice project. It acted as a brainstorming session. I got a lovely conference bag, which contained a T-Shirt, a pen and a few stickers. I also met my long time collaborators Mike, Sophie and Italo from the TDF, whom I had interacted only remotely till then. Likewise, I also met TDF’s sysadmin Guilhem, who I interacted before regarding setting up my LibreOffice mirror.
The conference started on the 10th. There were 5 attendees from India, including me, while most of the attendees were from Europe. The talks were in English. One of the talks that stood out for me was about Luxchat — a chat service run by the Luxembourg government based on the Matrix protocol for the citizens of Luxembourg. I also liked Italo’s talk on why document formats must be freedom-respecting. On the first night, the conference took us to a nice dinner in a restaurant. It offered one more way to socialize with other attendees and explore food at the same time.
On the 11th of October, I went for a walk in the morning with Biswadeep for some sightseeing around our hotel area. As a consequence, I missed the group photo of the conference, which I wanted to be in. Anyway, we enjoyed roaming around the picturesque Luxembourg city. We also sampled a tram ride to return to our hotel.
The conference ended on the 12th with a couple of talks. This conference gave me an opportunity to meet the global LibreOffice community, connect and share ideas. It also gave me a peek into the country of Luxembourg and its people, where I had good experience. English was widely known, and I had no issues getting by.
Thanks to all the organizers and sponsors of the conference!
For uninteresting reasons I need very regular 58Hz pulses coming out of an
RS-232 Tx line: the time between each pulse should be as close to 1/58s as
possible. I produce each pulse by writing an \xFF byte to the device. The
start bit is the only active-voltage bit being sent, and that produces my pulse.
I wrote this obvious C program:
#include <stdio.h> #include <stdlib.h> #include <stdbool.h> #include <sys/ioctl.h> #include <unistd.h> #include <fcntl.h> #include <termios.h> #include <stdint.h> #include <sys/time.h> static uint64_t gettimeofday_uint64() { struct timeval tv; gettimeofday(&tv, NULL); return (uint64_t) tv.tv_sec * 1000000ULL + (uint64_t) tv.tv_usec; } int main(int argc, char* argv[]) { // open the serial device, and make it as raw as possible const char* device = "/dev/ttyS0"; const speed_t baud = B9600; int fd = open(device, O_WRONLY|O_NOCTTY); tcflush(fd, TCIOFLUSH); struct termios options = {.c_iflag = IGNBRK, .c_cflag = CS8 | CREAD | CLOCAL}; cfsetspeed(&options, baud); tcsetattr(fd, TCSANOW, &options); const uint64_t T_us = (uint64_t)(1e6 / 58.); const uint64_t t0 = gettimeofday_uint64(); for(int i=0; ; i++) { const uint64_t t_target = t0 + T_us*i; const uint64_t t1 = gettimeofday_uint64(); if(t_target > t1) usleep(t_target - t1); write(fd, &((char){'\xff'}), 1); } return 0; }
This tries to make sure that each write() call happens at 58Hz. I need these
pulses to be regular, so I need to also make sure that the time between each
userspace write() and when the edge actually hits the line is as short as
possible or, at least, stable.
Potential reasons for timing errors:
usleep() doesn't wake up exactly when it should. This is subject to the
Linux scheduler waking up the trigger process
write() almost certainly ends up scheduling a helper task to actually
write the \xFF to the hardware. This helper task is also subject to the
Linux scheduler waking it up.
The scheduler-related questions are observable without any extra hardware, so let's do that first.
I run the ./trigger program, and look at diagnostics while that's running.
I look at some device details:
# ls -lh /dev/ttyS0 crw-rw---- 1 root dialout 4, 64 Mar 6 18:11 /dev/ttyS0 # ls -lh /sys/dev/char/4:64/ total 0 -r--r--r-- 1 root root 4.0K Mar 6 16:51 close_delay -r--r--r-- 1 root root 4.0K Mar 6 16:51 closing_wait -rw-r--r-- 1 root root 4.0K Mar 6 16:51 console -r--r--r-- 1 root root 4.0K Mar 6 16:51 custom_divisor -r--r--r-- 1 root root 4.0K Mar 6 16:51 dev lrwxrwxrwx 1 root root 0 Mar 6 16:51 device -> ../../../0000:00:16.3:0.0 -r--r--r-- 1 root root 4.0K Mar 6 16:51 flags -r--r--r-- 1 root root 4.0K Mar 6 16:51 iomem_base -r--r--r-- 1 root root 4.0K Mar 6 16:51 iomem_reg_shift -r--r--r-- 1 root root 4.0K Mar 6 16:51 io_type -r--r--r-- 1 root root 4.0K Mar 6 16:51 irq -r--r--r-- 1 root root 4.0K Mar 6 16:51 line -r--r--r-- 1 root root 4.0K Mar 6 16:51 port drwxr-xr-x 2 root root 0 Mar 6 16:51 power -rw-r--r-- 1 root root 4.0K Mar 6 16:51 rx_trig_bytes lrwxrwxrwx 1 root root 0 Mar 6 16:51 subsystem -> ../../../../../../../class/tty -r--r--r-- 1 root root 4.0K Mar 6 16:51 type -r--r--r-- 1 root root 4.0K Mar 6 16:51 uartclk -rw-r--r-- 1 root root 4.0K Mar 6 16:51 uevent -r--r--r-- 1 root root 4.0K Mar 6 16:51 xmit_fifo_size
Unsurprisingly, this is a part of the tty subsystem. I don't want to spend the
time to really figure out how this works, so let me look at all the tty
kernel calls and also at all the kernel tasks scheduled by the trigger
process, since I suspect that the actual hardware poke is happening in a helper
task. I see this:
# bpftrace -e 'k:*tty* /comm=="trigger"/
{ printf("%d %d %s\n",pid,tid,probe); }
t:sched:sched_wakeup /comm=="trigger"/
{ printf("switching to %s(%d); current backtrace:", args.comm, args.pid); print(kstack()); }'
...
3397345 3397345 kprobe:tty_ioctl
3397345 3397345 kprobe:tty_check_change
3397345 3397345 kprobe:__tty_check_change
3397345 3397345 kprobe:tty_wait_until_sent
3397345 3397345 kprobe:tty_write
3397345 3397345 kprobe:file_tty_write.isra.0
3397345 3397345 kprobe:tty_ldisc_ref_wait
3397345 3397345 kprobe:n_tty_write
3397345 3397345 kprobe:tty_hung_up_p
switching to kworker/0:1(3400169); current backtrace:
ttwu_do_activate+268
ttwu_do_activate+268
try_to_wake_up+605
kick_pool+92
__queue_work.part.0+582
queue_work_on+101
rpm_resume+1398
__pm_runtime_resume+75
__uart_start+85
uart_write+150
n_tty_write+1012
file_tty_write.isra.0+373
vfs_write+656
ksys_write+109
do_syscall_64+130
entry_SYSCALL_64_after_hwframe+118
3397345 3397345 kprobe:tty_update_time
3397345 3397345 kprobe:tty_ldisc_deref
... repeated with each pulse ...
Looking at the sources I see that uart_write() calls __uart_start(), which
schedules a task to call serial_port_runtime_resume() which eventually calls
serial8250_tx_chars(), which calls some low-level functions to actually send
the bits.
I look at the time between two of those calls to quantify the scheduler latency:
pulserate=58 sudo zsh -c \ '( echo "# dt_write_ns dt_task_latency_ns"; bpftrace -q -e "k:vfs_write /comm==\"trigger\" && arg2==1/ {\$t=nsecs(); if(@t0) { @dt_write = \$t-@t0; } @t0=\$t;} k:serial8250_tx_chars /@dt_write/ {\$t=nsecs(); printf(\"%d %d\\n\", @dt_write, \$t-@t0);}" )' \ | vnl-filter \ --stream -p dt_write_ms="dt_write_ns/1e6 - 1e3/$pulserate",dt_task_latency_ms=dt_task_latency_ns/1e6 \ | feedgnuplot \ --stream \ --lines \ --points \ --xlen 200 \ --vnl \ --autolegend \ --xlabel 'Pulse index' \ --ylabel 'Latency (ms)'
Here I'm making a realtime plot showing
write() call happens. This shows effect #1
from above: how promptly the trigger process wakes up
The raw data as I tweak things lives here. Initially I see big latency spikes:
These can be fixed by adjusting the priority of the trigger task. This tells
the scheduler to wake that task up first, even if something else is currently
using the CPU. I do this:
sudo chrt -p 90 `pidof trigger`
And I get better-looking latencies:
During some experiments (not in this dataset) I would see high helper-task
timing instabilities as well. These could be fixed by prioritizing the helper
task. In this kernel (6.12) the helper task is called kworker/N where N is
the CPU index. I tie the trigger process to cpu 0, and priorities all the
relevant helpers:
taskset -c 0 ./trigger 58 pgrep -f kworker/0 | while { read pid } { sudo chrt -p 90 $pid }
This fixes the helper-task latency spikes.
OK, so it looks like on the software side we're good to within 0.1ms of the true
period. This is in the ballpark of the precision I need; even this might be too
high. It's possible to try to push the software to do better: one could look at
the kernel sources a bit more, to do smarter things with priorities or to try an
-rt kernel. But all this doesn't matter if the serial hardware adds
unacceptable delays. Let's look.
Let's look at it with a logic analyzer. I use a saleae logic analyzer with
sigrok. The tool spits out the samples as it gets them, and an awk script
finds the edges and reports the timings to give me a realtime plot.
samplerate=500000; pulserate=58.; sigrok-cli -c samplerate=$samplerate -O csv --continuous -C D1 \ | mawk -Winteractive \ "prev_logic==0 && \$0==1 \ { iedge = NR; if(prev_iedge) { di = iedge -prev_iedge; dt = di/$samplerate; print(dt*1000); } prev_iedge = iedge; } { prev_logic=\$0; } " | feedgnuplot --stream --ylabel 'Period (ms)' --equation "1000./$pulserate title \"True ${pulserate}Hz period\""
On the server I was using (physical RS-232 port, ancient 3.something kernel):
OK… This is very discrete for some reason, and generally worse than 0.1ms. What about my laptop (physical RS-232 port, recent 6.12 kernel)?
Not discrete anymore, but not really any more precise. What about using a usb-serial converter? I expect this to be worse.
Yeah, looks worse. For my purposes, an accuracy of 0.1ms is marginal, and the hardware adds non-negligible errors. So I cut my losses, and use an external signal generator:
Yeah. That's better, so that's what I use.
14 March, 2025 12:47PM by Dima Kogan
Contributing to Debian is part of Freexian’s mission. This article covers the latest achievements of Freexian and their collaborators. All of this is made possible by organizations subscribing to our Long Term Support contracts and consulting services.
Over the last year, the Debian.social services outgrew the infrastructure that was supporting them. The matrix bridge in particular was hosted on a cloud instance backed by a large expensive storage volume. Debian.CH rented a new large physical server to host all these instances, earlier this year. Stefano set up Incus on the new physical machine and migrated all the old debian.social LXC Containers, libvirt VMs, and cloud instances into Incus-managed LXC containers.
Stefano set up Prometheus monitoring and alerts for the new infrastructure and a Grafana dashboard. The current stack of debian.social services seem to comfortably fit on the new machine, with good room to grow.
DebConf 25 preparations continue. The team is currently finalizing a budget. Stefano helped to review the current budget proposals and suggest approaches for balancing it.
Stefano installed a Zammad instance to organize queries from attendees, for the registration and visa teams.
Santiago continued discussions with possible caterers so we can have options for the different diet requirements and that could fit into the DebConf budget. Also, in collaboration with Anupa, Santiago pushed the first draft changes to document the venue information in the DebConf 25 website and how to get to Brest.
Colin fixed a fun bug in the Python requests package. Santiago Vila has been running tests of what happens when Debian packages are built on a system in which time has been artificially set to somewhere around the end of the support period for the next Debian release, in order to make it easier to do things like issuing security updates for the lifetime of that release. In this case, the failure indicated an expired test certificate, and since the repository already helpfully included scripts to regenerate those certificates, it seemed natural to try regenerating them just before running tests. However, this failed for more obscure reasons and Colin spent some time investigating. This turned out to be because the test CA was missing the CA constraint and so recent versions of OpenSSL reject it; Colin sent a pull request to fix this.
Santiago started a discussion on debian-devel about packages that have a history of security issues and that are outdated regarding new upstream releases. The goal of the mentioned effort is to have a prioritized list of packages needing some work, from a security point of view. Moreover, the aim of publicly sharing the list of packages with the Debian Developers community is to make it easier to look at the packages maintained by teams, or even other maintainers’ where help could be welcome. Santiago is planning to take into account the feedback provided in debian-devel and to propose a tooling that could help to regularly bring collective awareness of these packages.
Pre-Depends as a conflict mechanism.security-tracker that would pre-compute
/data/json during database updates rather than on demand resulting in a
reduced response time.11 March, 2025 12:00AM by Anupa Ann Joseph
A few months ago I explained that one reason why this blog has become more quiet is that all my work on Lean is covered elsewhere.
This post is an exception, because it is an observation that is (arguably) interesting, but does not lead anywhere, so where else to put it than my own blog…
Want to share your thoughts about this? Please join the discussion on the Lean community zulip!
When defining a function recursively in Lean that has nested recursion, e.g. a recusive call that is in the argument to a higher-order function like List.map, then extra attention used to be necessary so that Lean can see that xs.map applies its argument only elements of the list xs. The usual idiom is to write xs.attach.map instead, where List.attach attaches to the list elements a proof that they are in that list. You can read more about this my Lean blog post on recursive definitions and our new shiny reference manual, look for Example “Nested Recursion in Higher-order Functions”.
To make this step less tedious I taught Lean to automatically rewrite xs.map to xs.attach.map (where suitable) within the construction of well-founded recursion, so that nested recursion just works (issue #5471). We already do such a rewriting to change if c then … else … to the dependent if h : c then … else …, but the attach-introduction is much more ambitious (the rewrites are not definitionally equal, there are higher-order arguments etc.) Rewriting the terms in a way that we can still prove the connection later when creating the equational lemmas is hairy at best. Also, we want the whole machinery to be extensible by the user, setting up their own higher order functions to add more facts to the context of the termination proof.
I implemented it like this (PR #6744) and it ships with 4.18.0, but in the course of this work I thought about a quite different and maybe better™ way to do this, and well-founded recursion in general:
fixRecall that to use WellFounded.fix
WellFounded.fix : (hwf : WellFounded r) (F : (x : α) → ((y : α) → r y x → C y) → C x) (x : α) : C xwe have to rewrite the functorial of the recursive function, which naturally has type
F : ((y : α) → C y) → ((x : α) → C x)to the one above, where all recursive calls take the termination proof r y x. This is a fairly hairy operation, mangling the type of matcher’s motives and whatnot.
Things are simpler for recursive definitions using the new partial_fixpoint machinery, where we use Lean.Order.fix
Lean.Order.fix : [CCPO α] (F : β → β) (hmono : monotone F) : βso the functorial’s type is unmodified (here β will be ((x : α) → C x)), and everything else is in the propositional side-condition montone F. For this predicate we have a syntax-guided compositional tactic, and it’s easily extensible, e.g. by
theorem monotone_mapM (f : γ → α → m β) (xs : List α) (hmono : monotone f) :
monotone (fun x => xs.mapM (f x)) Once given, we don’t care about the content of that proof. In particular proving the unfolding theorem only deals with the unmodified F that closely matches the function definition as written by the user. Much simpler!
Isabelle also supports well-founded recursion, and has great support for nested recursion. And it’s much simpler!
There, all you have to do to make nested recursion work is to define a congruence lemma of the form, for List.map something like our List.map_congr_left
List.map_congr_left : (h : ∀ a ∈ l, f a = g a) :
List.map f l = List.map g lThis is because in Isabelle, too, the termination proofs is a side-condition that essentially states “the functorial F calls its argument f only on smaller arguments”.
I had wished we could do the same in Lean for a while, but that form of congruence lemma just isn’t strong enough for us.
But maybe there is a way to do it, using an existential to give a witness that F can alternatively implemented using the more restrictive argument. The following callsOn P F predicate can express that F calls its higher-order argument only on arguments that satisfy the predicate P:
section setup
variable {α : Sort u}
variable {β : α → Sort v}
variable {γ : Sort w}
def callsOn (P : α → Prop) (F : (∀ y, β y) → γ) :=
∃ (F': (∀ y, P y → β y) → γ), ∀ f, F' (fun y _ => f y) = F f
variable (R : α → α → Prop)
variable (F : (∀ y, β y) → (∀ x, β x))
local infix:50 " ≺ " => R
def recursesVia : Prop := ∀ x, callsOn (· ≺ x) (fun f => F f x)
noncomputable def fix (wf : WellFounded R) (h : recursesVia R F) : (∀ x, β x) :=
wf.fix (fun x => (h x).choose)
def fix_eq (wf : WellFounded R) h x :
fix R F wf h x = F (fix R F wf h) x := by
unfold fix
rw [wf.fix_eq]
apply (h x).choose_specThis allows nice compositional lemmas to discharge callsOn predicates:
theorem callsOn_base (y : α) (hy : P y) :
callsOn P (fun (f : ∀ x, β x) => f y) := by
exists fun f => f y hy
intros; rfl
@[simp]
theorem callsOn_const (x : γ) :
callsOn P (fun (_ : ∀ x, β x) => x) :=
⟨fun _ => x, fun _ => rfl⟩
theorem callsOn_app
{γ₁ : Sort uu} {γ₂ : Sort ww}
(F₁ : (∀ y, β y) → γ₂ → γ₁) -- can this also support dependent types?
(F₂ : (∀ y, β y) → γ₂)
(h₁ : callsOn P F₁)
(h₂ : callsOn P F₂) :
callsOn P (fun f => F₁ f (F₂ f)) := by
obtain ⟨F₁', h₁⟩ := h₁
obtain ⟨F₂', h₂⟩ := h₂
exists (fun f => F₁' f (F₂' f))
intros; simp_all
theorem callsOn_lam
{γ₁ : Sort uu}
(F : γ₁ → (∀ y, β y) → γ) -- can this also support dependent types?
(h : ∀ x, callsOn P (F x)) :
callsOn P (fun f x => F x f) := by
exists (fun f x => (h x).choose f)
intro f
ext x
apply (h x).choose_spec
theorem callsOn_app2
{γ₁ : Sort uu} {γ₂ : Sort ww}
(g : γ₁ → γ₂ → γ)
(F₁ : (∀ y, β y) → γ₁) -- can this also support dependent types?
(F₂ : (∀ y, β y) → γ₂)
(h₁ : callsOn P F₁)
(h₂ : callsOn P F₂) :
callsOn P (fun f => g (F₁ f) (F₂ f)) := by
apply_rules [callsOn_app, callsOn_const]With this setup, we can have the following, possibly user-defined, lemma expressing that List.map calls its arguments only on elements of the list:
theorem callsOn_map (δ : Type uu) (γ : Type ww)
(P : α → Prop) (F : (∀ y, β y) → δ → γ) (xs : List δ)
(h : ∀ x, x ∈ xs → callsOn P (fun f => F f x)) :
callsOn P (fun f => xs.map (fun x => F f x)) := by
suffices callsOn P (fun f => xs.attach.map (fun ⟨x, h⟩ => F f x)) by
simpa
apply callsOn_app
· apply callsOn_app
· apply callsOn_const
· apply callsOn_lam
intro ⟨x', hx'⟩
dsimp
exact (h x' hx')
· apply callsOn_const
end setupSo here is the (manual) construction of a nested map for trees:
section examples
structure Tree (α : Type u) where
val : α
cs : List (Tree α)
-- essentially
-- def Tree.map (f : α → β) : Tree α → Tree β :=
-- fun t => ⟨f t.val, t.cs.map Tree.map⟩)
noncomputable def Tree.map (f : α → β) : Tree α → Tree β :=
fix (sizeOf · < sizeOf ·) (fun map t => ⟨f t.val, t.cs.map map⟩)
(InvImage.wf (sizeOf ·) WellFoundedRelation.wf) <| by
intro ⟨v, cs⟩
dsimp only
apply callsOn_app2
· apply callsOn_const
· apply callsOn_map
intro t' ht'
apply callsOn_base
-- ht' : t' ∈ cs -- !
-- ⊢ sizeOf t' < sizeOf { val := v, cs := cs }
decreasing_trivial
end examplesThis makes me happy!
All details of the construction are now contained in a proof that can proceed by a syntax-driven tactic and that’s easily and (likely robustly) extensible by the user. It also means that we can share a lot of code paths (e.g. everything related to equational theorems) between well-founded recursion and partial_fixpoint.
I wonder if this construction is really as powerful as our current one, or if there are certain (likely dependently typed) functions where this doesn’t fit, but the β above is dependent, so it looks good.
With this construction, functions defined by well-founded recursion will reduce even worse in the kernel, I assume. This may be a good thing.
What unfortunately kills this idea, though, is the generation of the functional induction principles, which I believe is not (easily) possible with this construction: The functional induction principle is proved by massaging F to return a proof, but since the extra assumptions (e.g. for ite or List.map) only exist in the termination proof, they are not available in F.
Oh wey, how anticlimactic.
Curiously, if we didn’t have functional induction at this point yet, then very likely I’d change Lean to use this construction, and then we’d either not get functional induction, or it would be implemented very differently, maybe a more syntactic approach that would re-prove termination. I guess that’s called path dependence.
10 March, 2025 05:47PM by Joachim Breitner (mail@joachim-breitner.de)
This was my hundred-twenty-eighth month that I did some work for the Debian LTS initiative, started by Raphael Hertzog at Freexian. During my allocated time I uploaded or worked on:
Last but not least I did some days of FD this month and attended the monthly LTS/ELTS meeting.
This month was the seventy-ninth ELTS month. During my allocated time I uploaded or worked on:
Last but not least I did some days of FD this month and attended the monthly LTS/ELTS meeting.
This month I uploaded new packages or new upstream or bugfix versions of:
This work is generously funded by Freexian!
This month I uploaded new packages or new upstream or bugfix versions of:
Finally matomo was uploaded. Thanks a lot to Utkarsh Gupta and William Desportes for doing most of the work to make this happen.
This work is generously funded by Freexian!
Unfortunately I didn’t found any time to upload packages.
Have you ever heard of poliastro? It was a package to do calculations related to astrodynamics and orbital mechanics? It was archived by upstream end of 2023. I am now trying to revive it under the new name boinor and hope to get it back into Debian over the next months.
This is almost the last month that Patrick, our Outreachy intern for the Debian Astro project, is handling his tasks. He is working on automatic updates of the indi 3rd-party driver.
Unfortunately I didn’t found any time to work on this topic.
This month I uploaded new packages or new upstream or bugfix versions of:
Unfortunately I didn’t found any time to work on this topic.
This month I accepted 437 and rejected 64 packages. The overall number of packages that got accepted was 445.
10 March, 2025 03:33PM by alteholz
A friend of mine living in the USA sent me this link to help the flood victims: Support Bahía Blanca (Argentina) Flood Victims
These are not good news. In fact, much the contrary. Compared to the real issue, the fact that I'm not able to attend Embedded World at Nuremberg is, well, a detail. Or at least that's what I'm forcing myself to believe, as I REALLY wanted to be there. But mother nature said otherwise.
Park "D. Alberto Martinelli", Las Cañitas, Bahía Blanca (Google Maps)
Bahía Blanca , the city I live, has received a lot on rainfall. Really, a lot. Let me introduce the number like this: the previous highest recorded measurement was 170mm (6.69 inch)... in a month. Yesterday Friday 07 we had more than 400mm (15.75 inch) in 9 hours.
But those are just numbers. Some things are better seen in images.
I'll start with some soft ones.
Sink in Fournier street near Cambaceres (Google Maps)
I also happen to do figure skating in the same school of the 4 times world champions (where "world" means the whole world) Roller Dreams precision skating team - Instagram, from Club El Nacional. Our skating rink got severely damaged with the hail we had like 3 weeks ago (yes, we had hail too!!!). Now it's just impossible:
Let's get to the heavy, heartbreaker part. I did go to downtown Bahía Blanca, but during night, so let me share some links, most of them in Spanish, but images are images:
My alma matter, Universidad Nacional del Sur, lost its main library, great part of the Physics department and a lot of labs :-(
A nearby town, General Cerri, had even worst luck. In Bahía Blanca, a city of 300k+ people, has around 400 evacuated people. General Cerri, a town of 3000? people, had at least 800.
Every place has its legends. We do too. This land was called "Huecuvú Mapú", something like "Devil's land" by the original inhabitants of the zone, due to its harsh climate: string winters and hot summers, couple with fierce wind. But back in 1855 the Cacique (chief) José María Bulnes Yanquetruz had a peace agreement with commander Nicanor Otamendi. But a battle ensued, which Yanquetruz won. At this point history defers depending upon who tells it. Some say Yanquetruz was assigned a military grade as Captain of the indigenous auxiliary forces and provided a military suit, some say he stole it, some say this was a setup of another chief wanting to disrupt peace. But what is known is that Yanquetruz was killed, and his wife, the "machi" (sorceress), issued a curse over the land that would last 1000 years, and the curse was on the climate.
No, we are not there yet. This has just happened. The third violent climate occurrence in 15 months. The city needs to mourn and start healing itself. Time will say.
09 March, 2025 06:35PM by Lisandro Damián Nicanor Pérez Meyer
The other day, I noted that the emacs integration with debputy stopped working. After debugging for a while, I realized that emacs no longer sent the didOpen notification that is expected of it, which confused debputy. At this point, I was already several hours into the debugging and I noted there was some discussions on debian-devel about emacs and byte compilation not working. So I figured I would shelve the emacs problem for now.
But I needed an LSP capable editor and with my vi skills leaving much to be desired, I skipped out on vim-youcompleteme. Instead, I pulled out kate, which I had not been using for years. It had LSP support, so it would fine, right?
Well, no. Turns out that debputy LSP support had some assumptions that worked for emacs but not kate. Plus once you start down the rabbit hole, you stumble on things you missed previously.
First order of business was to tell kate about debputy. Conveniently, kate has a configuration tab for adding language servers in a JSON format right next to the tab where you can see its configuration for built-in LSP (also in JSON format9. So a quick bit of copy-paste magic and that was done.
Yesterday, I opened an MR against upstream to have the configuration added (https://invent.kde.org/utilities/kate/-/merge_requests/1748) and they already merged it. Today, I then filed a wishlist against kate in Debian to have the Debian maintainers cherry-pick it, so it works out of the box for Trixie (https://bugs.debian.org/1099876).
So far so good.
Since July (2024), debputy has support for Inlay hints. They are basically small bits of text that the LSP server can ask the editor to inject into the text to provide hints to the reader.
Typically, you see them used to provide typing hints, where the editor or the underlying LSP server has figured out the type of a variable or expression that you did not explicitly type. Another common use case is to inject the parameter name for positional arguments when calling a function, so the user do not have to count the position to figure out which value is passed as which parameter.
In debputy, I have been using the Inlay hints to show inherited fields in debian/control. As an example, if you have a definition like:
Source: foo-src
Section: devel
Priority: optional
Package: foo-bin
Architecture: any
Then foo-bin inherits the Section and Priority field since it does not supply its own. Previously, debputy would that by injecting the fields themselves and their value just below the Package field as if you had typed them out directly. The editor always renders Inlay hints distinctly from regular text, so there was no risk of confusion and it made the text look like a valid debian/control file end to end. The result looked something like:
Source: foo-src
Section: devel
Priority: optional
Package: foo-bin
Section: devel
Priority: optional
Architecture: any
With the second instances of Section and Priority being rendered differently than its surrendering (usually faded or colorlessly).
Unfortunately, kate did not like injecting Inlay hints with a newline in them, which was needed for this trick. Reading into the LSP specs, it says nothing about multi-line Inlay hints being a thing and I figured I would see this problem again with other editors if I left it be.
I ended up changing the Inlay hints to be placed at the end of the Package field and then included surrounding () for better visuals. So now, it looks like:
Source: foo-src
Section: devel
Priority: optional
Package: foo-bin (Section: devel) (Priority: optional)
Architecture: any
Unfortunately, it is no longer 1:1 with the underlying syntax which I liked about the previous one. But it works in more editors and is still explicit. I also removed the Inlay hint for the Homepage field. It takes too much space and I have yet to meet someone missing it in the binary stanza.
If you have any better ideas for how to render it, feel free to reach out to me.
As I was debugging the Inlay hints, I wanted to do a quick restart of debputy after each fix. Then I would trigger a small change to the document to ensure kate would request an update from debputy to render the Inlay hints with the new code.
The full outgoing payloads are sent via the logs to the client, so it was really about minimizing which LSP requests are sent to debputy. Notably, two cases would flood the log:
- Completion requests. These are triggered by typing anything at all and since I wanted to a change, I could not avoid this. So here it was about making sure there would be nothing to complete, so the result was a small as possible.
- Hover doc requests. These are triggered by mouse hovering over field, so this was mostly about ensuring my mouse movement did not linger over any field on the way between restarting the LSP server and scrolling the log in kate.
In my infinite wisdom, I chose to make a comment line where I would do the change. I figured it would neuter the completion requests completely and it should not matter if my cursor landed on the comment as there would be no hover docs for comments either.
Unfortunately for me, debputy would ignore the fact that it was on a comment line. Instead, it would find the next field after the comment line and try to complete based on that. Normally you do not see this, because the editor correctly identifies that none of the completion suggestions start with a \#, so they are all discarded.
But it was pretty annoying for the debugging, so now debputy has been told to explicitly stop these requests early on comment lines.
I added a feature in debputy where you can hover over package names in your relationship fields (such as Depends) and debputy will render a small snippet about it based on data from your local APT cache.
This doc is then handed to the editor and tagged as markdown provided the editor supports markdown rendering. Both emacs and kate support markdown. However, not all markdown renderings are equal. Notably, emacs's rendering does not reformat the text into paragraphs. In a sense, emacs rendering works a bit like <pre>...</pre> except it does a bit of fancy rendering inside the <pre>...</pre>.
On the other hand, kate seems to convert the markdown to HTML and then throw the result into an HTML render engine. Here it is important to remember that not all newlines are equal in markdown. A Foo<newline>Bar is treated as one "paragraph" (<p>...</p>) and the HTML render happily renders this as single line Foo Bar provided there is sufficient width to do so.
A couple of extra newlines made wonders for the kate rendering, but I have a feeling this is not going to be the last time the hover docs will need some tweaking for prettification. Feel free to reach out if you spot a weirdly rendered hover doc somewhere.
Quickfixes are treated as generic code actions in the LSP specs. Each code action has a "type" (kind in the LSP lingo), which enables the editor to group the actions accordingly or filter by certain types of code actions.
The design in the specs leads to the following flow:
- The LSP server provides the editor with diagnostics (there are multiple ways to trigger this, so we will keep this part simple).
- The editor renders them to the user and the user chooses to interact with one of them.
- The interaction makes the editor asks the LSP server, which code actions are available at that location (optionally with filter to only see quickfixes).
- The LSP server looks at the provided range and is expected to return the relevant quickfixes here.
This flow is really annoying from a LSP server writer point of view. When you do the diagnostics (in step 1), you tend to already know what the possible quickfixes would be. The LSP spec authors realized this at some point, so there are two features the editor provides to simplify this.
- In the editor request for code actions, the editor is expected to provide the diagnostics that they received from the server. Side note: I cannot quite tell if this is optional or required from the spec.
- The editor can provide support for remembering a data member in each diagnostic. The server can then store arbitrary information in that member, which they will see again in the code actions request. Again, provided that the editor supports this optional feature.
All the quickfix logic in debputy so far has hinged on both of these two features.
As life would have it, kate provides neither of them.
Which meant I had to teach debputy to keep track of its diagnostics on its own. The plus side is that makes it easier to support "pull diagnostics" down the line, since it requires a similar feature. Additionally, it also means that quickfixes are now available in more editors. For consistency, debputy logic is now always used rather than relying on the editor support when present.
The downside is that I had to spend hours coming up with and debugging a way to find the diagnostics that overlap with the range provided by the editor. The most difficult part was keeping the logic straight and getting the runes correct for it.
With all of that, kate would show the quickfixes for diagnostics from debputy and you could use them too. However, they would always apply twice with suboptimal outcome as a result.
The LSP spec has multiple ways of defining what need to be changed in response to activating a code action. In debputy, all edits are currently done via the WorkspaceEdit type. It has two ways of defining the changes. Either via changes or documentChanges with documentChanges being the preferred one if both parties support this.
I originally read that as I was allowed to provide both and the editor would pick the one it preferred. However, after seeing kate blindly use both when they are present, I reviewed the spec and it does say "The edit should either provide changes or documentChanges", so I think that one is on me.
None of the changes in debputy currently require documentChanges, so I went with just using changes for now despite it not being preferred. I cannot figure out the logic of whether an editor supports documentChanges. As I read the notes for this part of the spec, my understanding is that kate does not announce its support for documentChanges but it clearly uses them when present. Therefore, I decided to keep it simple for now until I have time to dig deeper.
There is one remaining limitation with kate that I have not yet solved. The kate program uses KSyntaxHighlighting for its language detection, which in turn is the basis for which LSP server is assigned to a given document.
This engine does not seem to support as complex detection logic as I hoped from it. Concretely, it either works on matching on an extension / a basename (same field for both cases) or mime type. This combined with our habit in Debian to use extension less files like debian/control vs. debian/tests/control or debian/rules or debian/upstream/metadata makes things awkward a best.
Concretely, the syntax engine cannot tell debian/control from debian/tests/control as they use the same basename. Fortunately, the syntax is close enough to work for both and debputy is set to use filename based lookups, so this case works well enough.
However, for debian/rules and debian/upstream/metadata, my understanding is that if I assign these in the syntax engine as Debian files, these rules will also trigger for any file named foo.rules or bar.metadata. That seems a bit too broad for me, so I have opted out of that for now. The down side is that these files will not work out of the box with kate for now.
The current LSP configuration in kate does not recognize makefiles or YAML either. Ideally, we would assign custom languages for the affected Debian files, so we do not steal the ID from other language servers. Notably, kate has a built-in language server for YAML and debputy does nothing for a generic YAML document. However, adding YAML as a supported language for debputy would cause conflict and regressions for users that are already happy with their generic YAML language server from kate.
So there are certainly still work to be done. If you are good with KSyntaxHighlighting and know how to solve some of this, I hope you will help me out.
I am glad I tested with kate to weed out most of these issues in time before the freeze. The Debian freeze will start within a week from now. Since debputy is a part of the toolchain packages it will be frozen from there except for important bug fixes.
09 March, 2025 12:05PM by Niels Thykier
Almost exactly four years after I started with this project, yesterday I presented my PhD defense.
My thesis was what I’ve been presenting advances of all around since ≈2022: «A certificate-poisoning-resistant protocol for the synchronization of Web of Trust networks»
Lots of paperwork is still on the road for me. But at least in the immediate future, I can finally use this keyring my friend Raúl Gómez 3D-printed for me:
A complete feed is available in any of your favourite syndication formats linked by the buttons below.
![[RSS 1.0 Feed]](common/rss10.png){kind=small}
![[RSS 2.0 Feed]](common/rss20.png){kind=small}
![[Atom Feed]](common/atom.png){kind=small}
![[FOAF Subscriptions]](common/foaf.png){kind=small}
![[OPML Subscriptions]](common/opml.png){kind=small}
![[Hacker]](common/hacker.png){kind=small}
![[Planet]](common/planet.png){kind=small}
Last updated: 03 Apr 2025 09:00
All times are UTC.
Contact: Debian Planet Maintainers