Updating old Debian Printing software to meet C23 requirements, by Thorsten Alteholz
The work of Thorsten fell under the motto “gcc15”. Due to the introduction of
gcc15 in Debian, the default language version was changed to C23. This means
that for example, function declarations without parameters are no longer allowed.
As old software, which was created with ANSI C (or C89) syntax, made use of such
function declarations, it was a busy month. One could have used something like
-std=c17 as compile flags, but this would have just postponed the tasks. As a
result Thorsten uploaded modernized versions of ink, nm2ppa and rlpr for the
Debian printing team.
Work done to decommission packages.qa.debian.org, by Raphaël Hertzog
Raphaël worked to decommission the old package tracking system
(packages.qa.debian.org). After figuring out
that it was still receiving emails from the bug tracking system
(bugs.debian.org), from multiple debian lists and from
some release team tools, he reached out to the respective teams to either drop
those emails or adjust them so that they are sent to the current Debian Package
Tracker (tracker.debian.org).
rebootstrap uses *-for-host, by Helmut Grohne
Architecture cross bootstrapping is an ongoing effort that has shaped Debian in
various ways over the years. A longereffort to express
toolchain dependencies now bears fruit. When cross compiling, it becomes
important to express what architecture one is compiling for in Build-Depends.
As these packages have become available in “trixie”, more and more packages add
this extra information and in August, the libtool package
gained
a gfortran-for-host dependency. It was the first package in the essential
build closure to adopt this and required putting the pieces together in
rebootstrap that now has to
build gcc-defaults early on. There still are
hundreds of packages whose dependencies need to be updated
though.
Miscellaneous contributions
Raphaël dropped the “Build Log Scan” integration in tracker.debian.org
since it was showing stale data for a while as the underlying service has been
discontinued.
Emilio updated pixman to 0.46.4.
Emilio coordinated several transitions, and NMUed guestfs-tools to unblock one.
Stefano uploaded Python 3.14rc3 to Debian unstable. It’s not yet used by any
packages, but it allows testing the level of support in packages to begin.
Stefano upgraded almost all of the debian-social infrastructure to Debian “trixie”.
Stefano attended the Debian Technical Committee meeting.
Stefano uploaded routine upstream updates for a handful of Python packages
(pycparser, beautifulsoup4, platformdirs, pycparser, python-authlib,
python-cffi, python-mitogen, python-resolvelib, python-super-collections,
twine).
Stefano reviewed and responded to DebConf 25 feedback.
Stefano investigated and fixed a request visibility bug in debian-reimbursements
(for admin-altered requests).
Lucas reviewed a couple of merge requests from external contributors for Go
and Ruby packages.
Lucas updated some ruby packages to its latest upstream version (thin,
passenger, and puma is still WIP).
Lucas set up the build environment to run rebuilds of reverse dependencies of
ruby using ruby3.4. As an alternative, he is looking for personal repositories
provided by Debusine to perform this task more easily. This is the preparation
for the transition to ruby3.4 as the default in Debian.
Lucas helped on the next round of the Outreachy internship program.
Helmut sent patches for 30 cross build failures and responded to cross
building support questions on the mailing list.
Helmut continued to maintain rebootstrap.
As gcc version 15 became the default, test jobs for version 14 had to be dropped.
A fair number of patches were applied to packages and could be dropped.
Helmut resumed removing RC-buggy packages from unstable and sponsored a
termrec upload to avoid its deletion. This work was paused to give packages
some time to migrate to “forky”.
While doing some new upstream release updates, thanks to Debusine’s
reverse dependencies autopkgtest
checks, Santiago discovered that paramiko 4.0 will introduce a
regression in libcloud by the drop of support
for the obsolete DSA keys. Santiago finally uploaded to unstable both
paramiko 4.0,
and a regression fix for libcloud.
Santiago has taken part in different discussions and meetings for the
preparation of DebConf 26. The DebConf 26 local team aims to prepare for the
conference with enough time in advance.
Carles kept working on the missing-package-relations and reporting missing
Recommends. He improved the tooling to detect and report bugs creating
269 bugs
and followed up comments. 37 bugs have been resolved, others acknowledged.
The missing Recommends are a mixture of packages that are gone from Debian,
packages that changed name, typos and also packages that were recommended but
are not packaged in Debian.
Carles improved the missing-package-relations to report broken Suggests only
for packages that used to be in Debian but are removed from it now. No bugs have
been created yet for this case but identified 1320 of them.
Colin spent much of the month chasing down build/test regressions in various
Python packages due to other upgrades, particularly relating to pydantic,
python-pytest-asyncio, and rust-pyo3.
A bit delayed in terms of an announcement, but last month I tagged a new version of onak, my OpenPGP compatible keyserver. It’s been 2 years since the last release, and this is largely a bunch of minor fixes to make compilation under Debian trixie with more recent CMake + GCC versions happy.
OpenPGP v6 support, RFC9580, hasn’t made it. I’ve got a branch which adds it, but a lack of keys to do any proper testing with, and no X448 support implemented, mean I’m not ready to include it in a release yet. The plan is that’ll land for 0.7.0 (along with some backend work), but no idea when that might be.
At work, we are trying to rotate the GPG signing keys for the Linux packages of
the eID middleware
We created new keys, and they will be installed on all Linux machines that have
the eid-archive package installed soon (they were already supposed to be, but
we made a
mistake).
Running some tests, however, I have a bit of a problem:
The only difference between the old keys and the new one, apart of
course from the fact that the old one is, well, old, is that the old one
uses the RSA algorithm whereas the new one uses ECDSA on the NIST P-384
curve (the same algorithm as the one used by the eID card).
Does RPM not support ECDSA keys? Does anyone know where this is
documented?
(Yes, I probably should have tested this before publishing the new
key, but this is where we are)
Just over a year ago I configured my blog to only allow signed in users to comment to reduce spam [1]. This has stopped all spam comments, it was even more successful than expected but spammers keep registering accounts. I’ve now got almost 5000 spam accounts, an average of more than 10 per day. I don’t know why they keep creating them without trying to enter comments. At first I thought that they were trying to assemble a lot of accounts for a deluge of comment spam but that hasn’t happened.
There are some WordPress plugins for bulk deletion of users but I couldn’t find one with support for “delete all users who haven’t submitted a comment”. So I do it a page at a time, but of course I don’t want to do it 100 at a time so I used the below SQL to change it to 400 at a time. I initially tried larger numbers like 2000 but got Chrome timeouts when trying to click the check-box to select all users. From experimenting it seems that the time taken to check that is worse than linear. Doing it for 2000 users is obviously much more than 5* the duration of doing it for 400. 800 users was one attempt which resulted in it being possible to select them all but then it gave an error about the URL being too long when it came to actually delete them. After a binary search I found that 450 was too many but 400 worked. So now it’s 12 operations to delete all the spam accounts. Each bulk delete operation is 5 GUI operations so it’s 60 operations to delete 15 months of spam users. This is annoying, but less than the other problems of spam.
UPDATE `wp_usermeta` SET `meta_value` = 400 WHERE `user_id` = 2 AND `meta_key` = 'users_per_page';
Deleting the spam users reduced the size of the backup (zstd -9 of a mysql dump) for my blog by 6.5%. Then changing from zstd -9 to -19 reduced it by another 13%. After realising this difference I configured all my mysql backups to be compressed with zstd -19, this will make a difference on the system with over 30G of zstd compressed mysql backups.
Version 0.0.23 of RcppSpdlog arrived
on CRAN today (after a slight
delay) and has been uploaded to Debian. RcppSpdlog
bundles spdlog, a
wonderful header-only C++ logging library with all the bells and
whistles you would want that was written by Gabi Melman, and also includes fmt by Victor Zverovich. You can learn
more at the nice package
documention site.
This release updates the code to the version 1.16.0 of spdlog which was released
yesterday morning, and includes version 1.12.0 of fmt. We also converted the documentation site
to now using mkdocs-material
to altdoc (plus local
style and production tweaks) rather than directly.
I updated the package yesterday morning when spdlog was updated. But the
passage was delayed for a day at CRAN as their machines still times
out hitting the GPL-2 URL from the README.md badge, leading to a human
to manually check the log assert the nothingburgerness of it. This
timeout does not happen to me locally using the corresponding URL
checker package. I pondered this in a r-package-devel
thread and may just have to switch to using the R Project URL for
the GPL-2 as this is in fact recurrning.
The NEWS entry for this release follows.
Changes in
RcppSpdlog version 0.0.23 (2025-10-11)
Upgraded to upstream release spdlog 1.16.0 (including fmt
12.0)
The mkdocs-material documentation site is now generated via altdoc
On mail.quux, a node of NNCPNET (the NNCP-based peer-to-peer email network), I started noticing emails not being delivered. They were all in the queue, frozen, and Exim’s log had entries like:
unable to set gid=5001 or uid=5001 (euid=100): local delivery to [redacted] transport=nncp
Weird.
Stranger still, when I manually ran the queue with sendmail -qff -v, they all delivered fine.
Huh.
Well, I thought, it was a one-off weird thing. But then it happened again.
Upon investigating, I observed that this issue was happening only on messages submitted by SMTP. Which, on these systems, aren’t that many.
While trying different things, I tried submitting a message to myself using SMTP. Nothing to do with NNCP at all. But look at this:
jgoerzen@[redacted] R=userforward defer (-1): require_files: error for /home/jgoerzen/.forward: Permission denied
Strraaannnge….
All the information I could find about this, even a FAQ entry, said that the problem is that Exim isn’t setuid root. But it is:
-rwsr-xr-x 1 root root 1533496 Mar 29 2025 /usr/sbin/exim4
This problem started when I upgraded to Debian Trixie. So what changed there?
There are a lot of possibilities; this is running in Docker using my docker-debian-base system, which runs a regular Debian in Docker, including systemd.
I eventually tracked it down to Exim migrating from init.d to systemd in trixie, and putting a bunch of lockdowns in its service file. After a bunch of trial and error, I determined that I needed to override this set of lockdowns to make it work. These overrides did the trick:
I don’t know for sure if the issue is related to setuid. But if it is, there’s nothing that immediately jumps out at me about any of these that would indicate a problem with setuid.
I also don’t know if running in Docker makes any difference.
test-drove said instance and suggested improvements to the
new documentation based on a Trixie VM
Pictures
This time again, we were hosted at La Balise (formely ATSÉ).
It's nice to see this community project continuing to improve: the social
housing apartments on the top floors should be opening this month! Lots of
construction work was also ongoing to make the Espace des Possibles more
accessible from the street level.
Some of us ended up grabbing a drink after the event at l'Isle de Garde,
a pub right next to the venue, but I didn't take any pictures.
Welcome to the very latest report from the Reproducible Builds project. Our monthly reports outline what we’ve been up to over the past month, and highlight items of news from elsewhere in the increasingly-important area of software supply-chain security. As ever, if you are interested in contributing to the Reproducible Builds project, please see the Contribute page on our website.
We are thrilled to host the eighth edition of this exciting event, following the success of previous summits in various iconic locations around the world, including Venice, Marrakesh, Paris, Berlin, Hamburg and Athens. Our summits are a unique gathering that brings together attendees from diverse projects, united by a shared vision of advancing the Reproducible Builds effort.
During this enriching event, participants will have the opportunity to engage in discussions, establish connections and exchange ideas to drive progress in this vital field. Our aim is to create an inclusive space that fosters collaboration, innovation and problem-solving.
If you’re interesting in joining us this year, please make sure to read the event page which has more details about the event and location. Registration is open until 20th September 2025, and we are very much looking forward to seeing many readers of these reports there!
Debian Developer Gunnar Wolf blogged that George V. Neville-Neil’s “Kode Vicious” column in Communications of the ACM in which reproducible builds “is mentioned without needing to introduce it (assuming familiarity across the computing industry and academia)”. Titled, Can’t we have nice things?, the article mentions:
Once the proper measurement points are known, we want to constrain the system such that what it does is simple enough to understand and easy to repeat. It is quite telling that the push for software that enables reproducible builds only really took off after an embarrassing widespread security issue ended up affecting the entire Internet. That there had already been 50 years of software development before anyone thought that introducing a few constraints might be a good idea is, well, let’s just say it generates many emotions, none of them happy, fuzzy ones. […]
Distribution work
In Debian this month, Johannes Starosta filed a bug against the debian-repro-status package, reporting that it does not work on Debian trixie. (An upstream bug report was also filed.) Furthermore, 17 reviews of Debian packages were added, 10 were updated and 14 were removed this month adding to our knowledge about identified issues.
strip-nondeterminism version 1.15.0-1 was uploaded to Debian unstable by Chris Lamb. It included a contribution by Matwey Kornilov to add support for inline archive files for Erlang’s escript […].
kpcyrd has released a new version of rebuilderd. As a quick recap, rebuilderd is an automatic build scheduler that tracks binary packages available in a Linux distribution and attempts to compile the official binary packages from their (purported) source code and dependencies. The code for in-toto attestations has been reworked, and the instances now feature a new endpoint that can be queried to fetch the list of public-keys an instance currently identifies itself by. […]
Lastly, Holger Levsen bumped the Standards-Version field of disorderfs, with no changes needed. […][…]
Reproducibility testing framework
The Reproducible Builds project operates a comprehensive testing framework running primarily at tests.reproducible-builds.org in order to check packages and other artifacts for reproducibility. In August, however, a number of changes were made by Holger Levsen, including:
Setting up six new rebuilderd workers with 16 cores and 16 GB RAM each.
Lastly, Roland Clobus did some maintenance relating to the reproducibility testing of the Debian Live images. […][…][…][…]
Upstream patches
The Reproducible Builds project detects, dissects and attempts to fix as many currently-unreproducible packages as possible. We endeavour to send all of our patches upstream where appropriate. This month, we wrote a large number of such patches, including:
Finally, if you are interested in contributing to the Reproducible Builds project, please visit our Contribute page on our website. However, you can get in touch with us via:
Avoiding
5XX errors by adjusting Load Balancer Idle Timeout
Recently I faced a problem in production where a client was running a
RabbitMQ server behind the Load Balancers we provisioned and the TCP
connections were closed every minute.
My team is responsible for the LBaaS (Load Balancer as a Service)
product and this Load Balancer was an Envoy proxy provisioned by our
control plane.
At first glance, the issue is simple: the Load Balancer's idle
timeout is shorter than the RabbitMQ heartbeat interval.
The idle timeout is the time at which a downstream or upstream
connection will be terminated if there are no active streams. Heartbeats
generate periodic network traffic to prevent idle TCP connections from
closing prematurely.
Adjusting these timeout settings to align properly solved the
issue.
However, what I want to explore in this post are other similar
scenarios where it's not so obvious that the idle timeout is the
problem. Introducing an extra network layer, such as an Envoy proxy, can
introduce unpredictable behavior across your services, like intermittent
5XX errors.
To make this issue more concrete, let's look at a minimal,
reproducible setup that demonstrates how adding an Envoy proxy can lead
to sporadic errors.
I'll be running experiments with two different
envoy.yaml configurations: one that uses Envoy's TCP proxy,
and another that uses Envoy's HTTP connection manager.
Here's the simplest Envoy TCP proxy setup: a listener on port 8000
forwarding traffic to a backend running on port 8080.
The default idle timeout if not otherwise specified is 1 hour, which
is the case here.
The backend setup is simple as well:
package mainimport("fmt""net/http""time")func helloHandler(w http.ResponseWriter, r *http.Request){ w.Write([]byte("Hello from Go!"))}func main(){ http.HandleFunc("/", helloHandler) server := http.Server{ Addr:":8080", IdleTimeout:3* time.Second,} fmt.Println("Starting server on :8080")panic(server.ListenAndServe())}
The IdleTimeout is set to 3 seconds to make it easier to test.
Now, oha is the perfect tool to generate the HTTP
requests for this test. The Load test is not meant to stress this setup,
the idea is to wait long enough so that some requests are closed. The
burst-delay feature will help with that:
I'm running the Load test for 30 seconds, sending 100 requests at
three-second intervals. I also use the -w option to wait
for ongoing requests when the duration is reached. The result looks like
this:
We had 886 responses with status code 200 and 64 connections closed.
The backend terminated 64 connections while the load balancer still had
active requests directed to it.
Let's change the Load Balancer idle_timeout to two
seconds.
filter_chains:-filters:-name: envoy.filters.network.tcp_proxytyped_config:"@type": type.googleapis.com/envoy.extensions.filters.network.tcp_proxy.v3.TcpProxystat_prefix: go_server_tcpcluster: go_server_clusteridle_timeout: 2s # <--- NEW LINE
Run the same test again.
Great! Now all the requests worked.
This is a common issue, not specific to Envoy Proxy or the setup
shown earlier. Major cloud providers have all documented it.
The target closed the connection with a TCP RST or a TCP FIN while the load balancer had an outstanding request to the target. Check whether the keep-alive duration of the target is shorter than the idle timeout value of the load balancer.
Verify that the keepalive configuration parameter for the HTTP server software running on the backend instance is not less than the keepalive timeout of the load balancer, whose value is fixed at 10 minutes (600 seconds) and is not configurable.
The load balancer generates an HTTP 5XX response code when the connection to the backend has unexpectedly closed while sending the HTTP request or before the complete HTTP response has been received. This can happen because the keepalive configuration parameter for the web server software running on the backend instance is less than the fixed keepalive timeout of the load balancer. Ensure that the keepalive timeout configuration for HTTP server software on each backend is set to slightly greater than 10 minutes (the recommended value is 620 seconds).
Certain networking tools (HAproxy, AWS ELB) and equipment (hardware load balancers) may terminate "idle" TCP connections when there is no activity on them for a certain period of time. Most of the time it is not desirable.
Most of them are talking about Application Load Balancers and the
test I did was using a Network Load Balancer. For the sake of
completeness, I will do the same test but using Envoy's HTTP connection
manager.
The yaml above is an example of a service proxying HTTP from
0.0.0.0:8000 to 0.0.0.0:8080. The only difference from a minimal
configuration is that I enabled access logs.
Let's run the same tests with oha.
Even thought the success rate is 100%, the status code distribution
show some responses with status code 503. This is the case where is not
that obvious that the problem is related to idle timeout.
However, it's clear when we look the Envoy access logs:
Armadillo is a powerful
and expressive C++ template library for linear algebra and scientific
computing. It aims towards a good balance between speed and ease of use,
has a syntax deliberately close to Matlab, and is useful for algorithm
development directly in C++, or quick conversion of research code into
production environments. RcppArmadillo
integrates this library with the R environment and language–and is
widely used by (currently) 1273 other packages on CRAN, downloaded 41.8 million
times (per the partial logs from the cloud mirrors of CRAN), and the CSDA paper (preprint
/ vignette) by Conrad and myself has been cited 651 times according
to Google Scholar.
Armadillo 15 brought changes. We mentioned these in the 15.0.2-1
and 15.0.2-1
release blog posts:
Minimum C++ standard of C++14
No more suppression of deprecation notes
(The second point is a consequence of the first. Prior to C++14,
deprecation notes were issue via a macro, and the macro was set up by Conrad in the common way of
allowing an override, which we took advantage of in RcppArmadillo
effectively shielding downstream package. In C++14 this is now an
attribute, and those cannot be suppressed.)
We tested this then-upcoming change extensively: Thirteen reverse
dependency runs expoloring different settings and leading to the current
package setup where an automatic fallback to the last Armadillo 14
release offers fallback for hardwired C++11 use and Armadillo 15 others.
Given the 1200+ reverse deoendencies, this took considerable time. All
this was also quite extensively discussed with CRAN (especially Kurt
Hornik) and documented / controlled via a series of issue tickets
starting with overall issue #475
covering the subissues:
open issue
#475 describes the version selection between Armadillo 14 and 15 via
#define
open issue
#476 illustrates how package without deprecation notes are already
suitable for Armadillo 15 and C++14
open issue
#477 demonstrates how a package with a simple deprecation note can
be adjusted for Armadillo 15 and C++14
closed issue
#479 documents a small bug we created in the initial transition
package RcppArmadillo 15.0.1-1 and fixed in the 15.0.2-1
closed issue
#481 discusses removal of the check for insufficient LAPACK routines
which has been removed given that R 4.5.0 or later has sufficient code
in its fallback LAPACK (used e.g. on Windows)
open issue
#484 offering help to the (then 226) packages needing help
transitioning from (enforced) C++11
open issue
#485 offering help to the (then 135) packages needing help with
deprecations
open issue
#489 coordinating pull requests and patches to 35 packages for the
C++11 transition
open issue
#491 coordinating pull requests and patches to 25 packages for
deprecation transition
The sixty pull requests (or emailed patches) followed a suggestion by
CRAN to rank-order packages affected by their reverse dependencies
sorted in descending package count. Now, while this change from
Armadillo 14 to 15 was happening, CRAN also tightened the C++11
requirement for packages and imposed a deadline for changes. In
discussion, CRAN also convinced me that a deadline for the deprecation
warning, now unmasked, was viable (and is in fairness commensurate with
similar, earlier changes triggered by changes in the behaviour of either
gcc/g++ or clang/clang++). So we now have two larger deadline campaigns
affecting the package (and as always there are some others).
These deadlines are coming close: October 17 for the C++11
transition, and October 23 for the deprecation warning. Now, as became
clear preparing the sixty pull requests and patches, these changes are
often relatively straightforward. For the former, remove the C++11
enforcement and the package will likely build without changes. For the
latter, make the often simple (e.g. swith from
arma::is_finite to std::isfinite) change. I
did not encounter anything much more complicated yet.
The number of affected packages—approximated by looking at all
packages with a reverse dependency on RcppArmadillo and having
a deadline–can be computed as
and has been declining steadily from over 350 to now under 200. For
that a big and heartfelt Thank You! to all the
maintainers who already addressed their package and uploaded updated
packages to CRAN. That rocks, and is truly appreciated.
Yet the number is still large. And while issues #489 and
#491
show a number of ‘pending’ packages that have merged but not uploaded
(yet?) there are also all the other packages I have not been able to
look at in detail. While preparing sixty PRs / patches was viable over a
period of a good week, I cannot create these for all packages. So with
that said, here is a different suggestion for help: All of next week, I
will be holding open door ‘open source’ office hours online two times
each day (11:00h to 13:00h Central, 16:00h to 18:00h Central) which can
be booked via this booking
link for Monday to Friday next week in either fifteen or thirty
minutes slots you can book. This should offer Google Meet video
conferencing (with jitsi as an
alternate, you should be able to control that) which should allow for
screen sharing. (I cannot hookup Zoom as my default account has
organization settings with a different calendar integration.)
If you are reading this and have a package that still needs helps, I
hope to see you in the Open Source Office Hours to aid in the
RcppArmadillo package updates for your package. Please book a slot!
“Help us get our next thousand (or million) followers!”
I was using Linux before it was popular. Back in the day where you had to write Modelines for your XF86Config file — and do it properly, or else you might ruin your monitor. Back when there wasn’t a word processor (thankfully; that forced me to learn LaTeX, which I used to write my papers in college).
I then ran Linux on an Alpha, a difficult proposition in an era when web browsers were either closed-source or too old to be useful; all sorts of workarounds, including emulating Digital UNIX.
Nobody can monetize things like this. I am one of maybe a dozen or two people globally that care about that sort of thing. That’s fine.
Today, I’m interested in things like asynchronous communication, NNCP, and Gopher. Heck, I’m posting these words on a blog. Social media displaced those, right?
Some of the things I write about here have maybe a few dozen people on the planet interested in them. That’s fine.
I have no idea how many people read my blog. I have no idea where people hear about my posts from. I guess I can check my Mastodon profile to see how many followers I have, but it’s not something I tend to do. I don’t know if the number is going up or down, or if it is all that much in Mastodon terms (probably not).
Thank goodness.
Since I don’t have to care about what’s popular, or spend hours editing video, or thousands of dollars on video equipment, I can just sit down and write about what interests me. If that also interests you, then great. If not, you can find what interests you — also fine.
I once had a colleague that was one of these “plugged into Silicon Valley” types. He would periodically tell me, with a mixture of excitement and awe, that one of my posts had made Hacker News.
This was always news to me, because I never paid a lot of attention over there. Occasionally that would bring in some excellent discussion, but more often than not, it was comments from people that hadn’t read or understood the article trying to appear smart by arguing with what it — or rather, what they imagined it said, I guess.
The thing I value isn’t subscriber count. It’s discussion. A little discussion in the comments or on Mastodon – that’s perfect, even if only 10 people read the article. I have the most fun in a community.
And I’ll go on writing about NNCP and Gopher and non-square DOS pixels, with audiences of dozens globally. I have no advertisers to keep happy, and I enjoy it, so why not?
This was my hundred-thirty-fifth month that I did some work for the Debian LTS initiative, started by Raphael Hertzog at Freexian. During my allocated time I uploaded or worked on:
[DLA 4168-2] openafs regression update to fix an incomplete patch in the previous upload.
[DSA 5998-1] cups security update to fix two CVEs related to a authentication bypass and a denial of service.
[DLA 4298-1] cups security update to fix two CVEs related to a authentication bypass and a denial of service.
[DLA 4304-1] cjson security update to fix one CVE related to an out-of-bounds memory access.
[DLA 4307-1] jq security update to fix one CVE related to a heap buffer overflow.
[DLA 4308-1] corosync security update to fix one CVE related to a stack-based buffer overflow.
An upload of spim was not needed, as the corresponding CVE could be marked as ignored.
I also started to work on an open-vm-tools and attended the monthly LTS/ELTS meeting.
Debian ELTS
This month was the eighty-sixth ELTS month. During my allocated time I uploaded or worked on:
[ELA-1512-1] cups security update to fix two CVEs in Buster and Stretch, related to a authentication bypass and a denial of service.
[ELA-1520-1] jq security update to fix one CVE in Buster and Stretch, related to a heap buffer overflow.
[ELA-1524-1] corosync security update to fix one CVE in Buster and Stretch, related to a stack-based buffer overflow.
[ELA-1527-1] mplayer security update to fix ten CVEs in Stretch, distributed all over the code.
The CVEs for open-vm-tools could be marked as not-affeceted as the corresponding plugin was not yet available. I also attended the monthly LTS/ELTS meeting.
Debian Printing
This month I uploaded a new upstream version or a bugfix version of:
The main topic of this month has been gcc15 and cmake4, so my upload rate was extra high. This month I uploaded a new upstream version or a bugfix version of:
I wonder what MBF will happen next, I guess the /var/lock-issue will be a good candidate.
On my fight against outdated RFPs, I closed 30 of them in September. Meanwhile only 3397 are still open, so don’t hesitate to help closing one or another.
FTP master
This month I accepted 294 and rejected 28 packages. The overall number of packages that got accepted was 294.
Excited to share that xptr is back on
CRAN! The xptr package
helps to create, check, modify, use, share, … external pointer
objects.
External pointers are used quite extensively throughout R to manage external ‘resources’
such as datanbase connection objects and alike, and can be very useful
to pass pointers to just about any C / C++ data structure around. While
described in Writing
R Extensions (notably Section
5.13), they can be a little bare-bones—and so this package can be
useful. It had been created by Randy
Lai and maintained by him during 2017 to 2020, but then fell off
CRAN. In work with nanoarrow and
its clean and minimal Arrow interface xptr came in
handy so I adopted it.
Several extensions and updates have been added: (compiled) function
registration, continuous integration, tests, refreshed and extended
documentation as well as a print format extension useful for PyCapsule
objects when passing via reticulate. The
package documentation
site was switched to altdoc driving the
most excellent Material for
MkDocs framework (providing my first test case of altdoc replacing my
older local scripts; I should post some more about that …).
The first NEWS entry follow.
Changes in version 1.2.0
(2025-10-03)
New maintainer
Compiled functions are now registered, .Call()
adjusted
README.md and DESCRIPTION edited and update
Simple unit tests and continunous integration have been
added
The package documentation site has been recreated using altdoc
All manual pages for functions now contain \value{}
sections
It’s always nice to have container images of Debian releases to test things,
run applications or explore a bit without polluting your host machine. From
some Brazilian friends (you know who you are ;-), I’ve learned the best way to
debug a problem or test a fix is spinning up an incus container, getting to it
and finding the minimum reproducer. So the combination incus + Debian is
something that I’m very used to, but the problem is there are no images for
Debian ELTS and testing security fixes to see if they actually fix the
vulnerability and don’t break anything else is very important.
Well, the regular images don’t materialize out of thin air, right? So we can
learn how they are made and try to generate ELTS images in the same way -
shouldn’t be that difficult, right? Well, kinda ;-)
The images available by default in incus come from
images.linuxcontainers.org and are built
by Jenkins using distrobuilder. If you follow the links, you will find the
repository containing the yaml image definitions used by distrobuilder at
github.com/lxc/lxc-ci. With a bit of
investigation work, a fork, an incus
VM with distrobuilder installed and some magic (also called trial and error) I
was able to build a buster image! Whooray, but VM and stretch images are still
work in progress.
Anyway, I wanted to share how you can build your images and document this
process so I don’t forget, so here we are…
Building Instructions
We will use an incus trixie VM to perform the build so we don’t clutter our own
machine.
Then let’s hop into the machine and install the dependencies.
incus shell <instance-name>
And…
apt install git distrobuilder
Let’s clone the repository with the yaml definition to build a buster
container.
git clone --branch support-debian-buster https://github.com/charles2910/lxc-ci.git
# and cd into itcd lxc-ci
Then all we need is to pass the correct arguments to distrobuilder so it can build
the image. It can output the image in the current directory or in a pre-defined
place, so let’s create an easy place for the images.
mkdir-p /tmp/images/buster/container
# and perform the build
distrobuilder build-incus images/debian.yaml /tmp/images/buster/container/ \-o image.architecture=amd64 \-o image.release=buster \-o image.variant=default \-o source.url="http://archive.debian.org/debian"
It requires a build definition written in yaml format to perform the build. If
you are curious, check the images/ subdir.
If all worked correctly, you should have two files in your pre-defined target
directory. In our case, /tmp/images/buster/container/ contains:
incus.tar.xz rootfs.squashfs
Let’s copy it to our host so we can add the image to our incus server.
incus file pull <instance-name>/tmp/images/buster/container/incus.tar.xz .
incus file pull <instance-name>/tmp/images/buster/container/rootfs.squashfs .# and import it as debian/10
incus image import incus.tar.xz rootfs.squashfs --alias debian/10
If we are lucky, we can run our Debian buster container now!
I had to spend a fair bit of time this month chasing down build/test
regressions in various packages due to some other upgrades, particularly to
pydantic, python-pytest-asyncio, and rust-pyo3:
After some upstream discussion I requested removal of
pydantic-compat, since it was more trouble
than it was worth to keep it working with the latest pydantic version.
pkg_resources is
deprecated. In most cases
replacing it is a relatively simple matter of porting to
importlib.resources,
but packages that used its old namespace package support need more
complicated work to port them to implicit namespace
packages. We had quite a few bugs about
this on zope.* packages, but fortunately upstream did the hard part of
this recently. I went
round and cleaned up most of the remaining loose ends, with some help from
Alexandre Detiste. Some of these aren’t completely done yet as they’re
awaiting new upstream releases:
A new version 0.3.5 of the RPushbullet
package arrived on CRAN. It
marks the first release in 4 1/2 years for this mature and
feature-stable package. RPushbullet
interfaces the neat Pushbullet
service for inter-device messaging, communication, and more. It lets you
easily send (programmatic) alerts like the one to the left to your
browser, phone, tablet, … – or all at once.
This releases reflects mostly internal maintenance and updates to the
documentation
site, to continuous integration, to package metadata, … and one code
robustitication. See below for more details.
Changes in version 0.3.5
(2025-10-08)
URL and BugReports fields have been added to DESCRIPTION
The pbPost function deals more robustly with the
case of multiple target emails
The continuous integration and the README badge have been
updated
The DESCRIPTION file now use Authors@R
The (encrypted) unit test configuration has been adjusted to
reflect the current set of active devices
The mkdocs-material documentation site is now generated via altdoc
Courtesy of my CRANberries, there
is a diffstat
report relative to previous release. More detailed information is on
the repo where
comments and suggestions are welcome.
Brief note to maybe spare someone else the trouble. If you want to
hide e.g. a huge table in Backstage (techdocs/mkdocs) behind a
collapsible element you need the md_in_html extension and use the
markdown attribute for it to kick in on the <details> html tag.
Add the extension to your mkdocs.yaml:
markdown_extensions:
- md_in_html
Hide the table in your markdown document in a collapsible element
like this:
It's also required to have an empty line between the html tag and starting
the markdown part. Rendered for me that way in VSCode, GitHub and Backstage.
Continuing with posts #51
from Tuesday and #52
from Wednesday and their stated intent of posting some more … here
is another quick one. Earlier today I helped another package developer
who came to the r-package-devel list asking for help with a build error
on the Fedora machine at CRAN running recent / development
clang. In such situations, the best first step is often to
replicate the issue. As I pointed out on the list, the LLVM team behind
clang maintains an apt repo
at apt.llvm.org/ making it a good resource to add to Debian-based
container such as Rocker r-base or the offical r-base (the two
are in fact interchangeable, and I take care of both).
A small pothole, however, is that the documentation at the top of apt.llvm.org site is a bit stale and
behind two aspects that changed on current Debian systems
(i.e. unstable/testing as used for r-base). First,
apt now prefers files ending in .sources (in a
nicer format) and second, it now really requires a key (which is good
practice). As it took me a few minutes to regather how to meet both
requirements, I reckoned I might as well script this.
Et voilà the following script does that:
it can update and upgrade the container (currently
commented-out)
it fetches the repository key in ascii form from the llvm.org
site
it creates the sources entry for, here tagged for llvm ‘current’ (22
at time of writing)
it sets up the required ~/.R/Makevars to use that
compiler
it installs clang-22 (and clang++-22)
(still using the g++ C++ library)
#!/bin/sh## Update does not hurt but is not strictly needed#apt update --quiet --quiet#apt upgrade --yes## wget -qO- https://apt.llvm.org/llvm-snapshot.gpg.key | sudo tee /etc/apt/trusted.gpg.d/apt.llvm.org.asc## or as we are root in containerwget-qO- https://apt.llvm.org/llvm-snapshot.gpg.key > /etc/apt/trusted.gpg.d/apt.llvm.org.asccat<<EOF>/etc/apt/sources.list.d/llvm-dev.sourcesTypes: debURIs: http://apt.llvm.org/unstable/# for clang-21 # Suites: llvm-toolchain-21# for current clangSuites: llvm-toolchainComponents: mainSigned-By: /etc/apt/trusted.gpg.d/apt.llvm.org.ascEOFtest-d ~/.R ||mkdir ~/.Rcat<<EOF>~/.R/MakevarsCLANGVER=-22# CLANGLIB=-stdlib=libc++CXX=clang++\$(CLANGVER) \$(CLANGLIB)CXX11=clang++\$(CLANGVER) \$(CLANGLIB)CXX14=clang++\$(CLANGVER) \$(CLANGLIB)CXX17=clang++\$(CLANGVER) \$(CLANGLIB)CXX20=clang++\$(CLANGVER) \$(CLANGLIB)CC=clang\$(CLANGVER)SHLIB_CXXLD=clang++\$(CLANGVER) \$(CLANGLIB)EOFapt updateapt install --yes clang-22
Once the script is run, one can test a package (or set of packages)
against clang-22 and clang++-22. This may help
R package developers. The script is also generic enough for other
development communities who can ignore (or comment-out / delete) the bit
about ~/.R/Makevars and deploy the compiler differently.
Updating the softlink as apt-preferences does is one way
and done in many GitHub Actions recipes. As we only need
wget here a basic Debian container should work, possibly
with the addition of wget. For R users r-base
hits a decent sweet spot.
There's a new Nine Inch Nails album! That
doesn't happen very often.
There's a new Trent Reznor & Atticus Ross soundtrack! That happens
all the time!
For the first time, they're the same thing.
The new one, Tron: Ares, is very deliberately presented as a Nine Inch Nails
album, and not a TR&AR soundtrack. But is it neither fish nor fowl? 24 tracks,
four with lyrics. Singing is not unheard of on TR&AR soundtracks,
but it's rare (A Minute to Breathe from the excellent Before the Flood
is another). Instrumentals are not rare on NIN albums, either, but this ratio
is very unusual, and has disappointed some fans who were hoping for a more
traditional NIN album.
What does it mean to label something a NIN album anyway? For me, the lines are
now further blurred. One thing for sure is it means a lot of media attention,
and this release, as well as the film it's promoting, are all over the media
at the moment. Posters, trailers, promotional tie-in items, Disney logos
everywhere. The album is hitched to the Disney marketing and promotion machine.
It's a bit weird seeing the NIN logo all over the place advertising the movie.
On to the music. I love TR&AR soundtracks, and some of my favourite NIN tracks
are instrumentals. Despite that, three highlights for me are songs: As Alive
As You Need Me To Be, I Know You Can Feel It and closer Shadow Over Me.
The other stand-out is Building Better Worlds, a short instrumental and
clear nod to Wendy Carlos.
My main complaint here applies to some of the more recent soundtracks as well:
the tracks are too short. They're scored to scenes in the movie, which makes a
lot of sense in that presentation, but less so for independent listening. It's
not a problem that their earlier, lauded soundtracks suffered (The Social Network,
Before the Flood, Bird Box Extended). Perhaps a future remix album will address that.
Another short status update of what happened on my side last
month. Nothing stands out too much, I enjoyed doing the OSK changes
the most as that helped to improve the typing experience further. Also
doing a small bit of kernel work again was fun (still need to figure out
the 6mq's touch controller repsonsiveness though).
Like many, bot traffic has been causing significant issues for my hosted server recently. I’ve been noticing a dramatic increase in bots that do not respect robots.txt, especially the crawl-delay I have set there. Not only that, but many of them are sending user-agent strings that are quite precisely matching what desktop browsers send. That is, they don’t identify themselves.
I had already added a crawl-delay line to robots.txt. It helped a bit, but many bots these days aren’t well-behaved. Next, I added WP Super Cache to my WordPress installation. I also enabled APCu in PHP and installed APCu Manager. Again, each step helped. Again, not quite enough.
Finally, I added Anubis. Installing it (especially if using the Docker container) was under-documented, but I figured it out. By default, it is designed to block AI bots and try to challenge everything with “Mozilla” in its user-agent (which is most things) with a Javascript challenge.
That’s not quite what I want. If a bot is well-behaved, AI or otherwise, it will respect my robots.txt and I can more precisely control it there. Also, I intentionally support non-Javascript browsers on many of the sites I host, so I wanted to be judicious. Eventually I configured Anubis to only challenge things that present a user-agent that looks fully like a real browser. In other words, real browsers should pass right through, and bad bots pretending to be real browsers will fail.
That was quite effective. It reduced load further to the point where things are ordinarily fairly snappy.
I had previously been using mod_security to block some bots, but it seemed to be getting in the way of the Fediverse at times. When I disabled it, I observed another increase in speed. Anubis was likely going to get rid of those annoying bots itself anyhow.
As a final step, I migrated to a faster hosting option. This post will show me how well it survives the Mastodon thundering herd!
Regarding Debian packaging this was a rather quiet month. I uploaded version
1.24.0-1 of foot and version 2.8.0-1 of
git-quick-stats. I took
the opportunity and started migrating my packages to the new version 5 watch
file
format,
which I think is much more readable than the previous format.
I also uploaded version 0.1.1-1 of
libscfg to NEW. libscfg is a C
implementation of the scfg configuration
file format and it is a dependency of recent version of
kanshi. kanshi is a tool
similar to autorandr which allows you define output profiles and kanshi
switches to the correct output profile on hotplug events. Once libscfg is in
unstable I can finally update kanshi to the latest version.
A lot of time this month in finalizing a redesign of the output rendering of
carl. carl is a small rust program I wrote
that provides a calendar view similar to cal, but it comes with colors and
ical file integration. That means that you can not only display a simple
calendar, but also colorize/highlight dates based on various attributes or
based on events on that day. In the initial versions of carl the output
rendering was simply hardcoded into the app.
This was a bit cumbersome to maintain and not configurable for users. I am
using templating languages on a daily basis, so I decided I would reimplement
the output generation of carl to use templates. I chose the
minijinja Rust library which is
“based on the syntax and behavior of the Jinja2 template engine for Python”.
There are others out there, like tera, but
minijinja seems to be more active in development currently. I worked on this
implementation on and off for the last year and finally had the time to finish
it up and write some additional tests for the outputs. It is easier to maintain
templates than Rust code that uses write!() to format the output. I also
implemented a configuration option for users to override the templates.
Additional to the output refactoring I also fixed couple of bugs and finally
released v0.4.0 of carl.
In my dayjob I released version 0.53 of apis-core-rdf which contains the place
lookup field which I implemented in August. A couple of weeks later we released
version 0.54 which comes with a middleware to show pass on messages from the
Django messages
framework via
response header to HTMX to trigger message popups. This implementation is based
on the blog post Using the Django messages framework with
HTMX. Version
0.55 was the last release in September. It contained preparations for
refactoring the import logic as well as a couple of UX improvements.
The last (software) piece of sorting out a local voice assistant is tying the openWakeWord piece to a local microphone + speaker, and thus back to Home Assistant. For that we use wyoming-satellite.
That starts us listening for connections from Home Assistant on port 10700, uses the openWakeWord instance on localhost port 10400, uses aplay/arecord to talk to the local microphone and speaker, and gives us some debug output so we can see what’s going on.
And it turns out we need the debug. This setup is a bit too flaky for it to have ended up in regular use in our household. I’ve had some problems with reliable audio setup; you’ll note the Python is calling out to other tooling to grab audio, which feels a bit clunky to me and I don’t think is the actual problem, but the main audio for this host is hooked up to the TV (it’s a media box), so the setup for the voice assistant needs to be entirely separate. That means not plugging into Pipewire or similar, and instead giving direct access to wyoming-satellite. And sometimes having to deal with how to make the mixer happy + non-muted manually.
I’ve also had some issues with the USB microphone + speaker; I suspect a powered USB hub would help, and that’s on my list to try out.
When it does work I have sometimes found it necessary to speak more slowly, or enunciate my words more clearly. That’s probably something I could improve by switching from the base.en to small.en whisper.cpp model, but I’m waiting until I sort out the audio hardware issue before poking more.
Finally, the wake word detection is a little bit sensitive sometimes, as I mentioned in the previous post. To be honest I think it’s possible to deal with that, if I got the rest of the pieces working smoothly.
This has ended up sounding like a more negative post than I meant it to. Part of the issue in a resolution is finding enough free time to poke things (especially as it involves taking over the living room and saying “Hey Jarvis” a lot), part of it is no doubt my desire to actually hook up the pieces myself and understand what’s going on. Stay tuned and see if I ever manage to resolve it all!
During 2025-03-21-another-home-outage, I reflected upon what's a
properly ran service and blurted out what turned out to be something
important I want to outline more. So here it is, again, on its own
for my own future reference.
Typically, I tend to think of a properly functioning service as having
four things:
backups
documentation
monitoring
automation
high availability (HA)
Yes, I miscounted. This is why you need high availability.
A service doesn't properly exist if it doesn't at least have the first
3 of those. It will be harder to maintain without automation, and
inevitably suffer prolonged outages without HA.
The five components of a proper service
Backups
Duh. If data is maliciously or accidentally destroyed, you need a copy
somewhere. Preferably in a way that malicious Joe can't get to.
You probably know this is hard, and this is why you're not doing
it. Do it anyways, you'll think it sucks, it will grow out of sync
with reality, but you'll be really grateful for whatever scraps you
wrote when you're in trouble.
Any docs, in other words, is better than no docs, but are no excuse
for doing the work correctly.
Monitoring
If you don't have monitoring, you'll know it fails too late, and you
won't know it recovers. Consider high availability, work hard to
reduce noise, and don't have machine wake people up, that's literally
torture and is against the Geneva convention.
Consider predictive algorithm to prevent failures, like "add storage
within 2 weeks before this disk fills up".
This is also harder than you think.
Automation
Make it easy to redeploy the service elsewhere.
Yes, I know you have backups. That is not enough: that typically
restores data and while it can also include configuration, you're
going to need to change things when you restore, which is what
automation (or call it "configuration management" if you will) will do
for you anyways.
This also means you can do unit tests on your configuration, otherwise
you're building legacy.
This is probably as hard as you think.
High availability
Make it not fail when one part goes down.
Eliminate single points of failures.
This is easier than you think, except for storage and DNS ("naming
things" not "HA DNS", that is easy), which, I guess, means it's
harder than you think too.
Assessment
In the above 5 items, I currently check two in my lab:
backups
documentation
And barely: I'm not happy about the offsite backups, and my
documentation is much better at work than at home (and even there, I
have a 15 year backlog to catchup on).
I barely have monitoring: Prometheus is scraping parts of the infra,
but I don't have any sort of alerting -- by which I don't mean
"electrocute myself when something goes wrong", I mean "there's a set
of thresholds and conditions that define an outage and I can look at
it".
Automation is wildly incomplete. My home server is a random collection
of old experiments and technologies, ranging from Apache with Perl and
CGI scripts to Docker containers running Golang applications. Most of
it is not Puppetized (but the ratio is growing). Puppet itself
introduces a huge attack vector with kind of catastrophic lateral
movement if the Puppet server gets compromised.
And, fundamentally, I am not sure I can provide high availability in
the lab. I'm just this one guy running my home network, and I'm
growing older. I'm thinking more about winding things down than
building things now, and that's just really sad, because I feel
we're losing (well that escalated
quickly).
Side note about Tor
The above applies to my personal home lab, not work!
At work, of course, it's another (much better) story:
all services have backups
lots of services are well documented, but not all
most services have at least basic monitoring
most services are Puppetized, but not crucial parts (DNS, LDAP,
Puppet itself), and there are important chunks of legacy coupling
between various services that make the whole system brittle
most websites, DNS and large parts of email are highly available,
but key services like the the Forum, GitLab and similar
applications are not HA, although most services run under
replicated VMs that can trivially survive a total, single-node
hardware failure (through Ganeti and DRBD)
Also circled back on previously opened ticket for supported packages for ELTS.
Partially reviewed and added comment on Emilio’s MP.
Re-visited an old thread (in order to fully close it) about issues being fixed in buster & bookworm but not in bullseye. And brought it up in the LTS meeting, too.
[LTS] Partook in some internal discussions about introducing support for handling severity of CVEs, et al.
Santiago had asked for an input from people doing FD so spent some time reflecting on his proposal and getting back with thoughts and suggestions.
[LTS] Helped Lee with testing gitk and git-gui aspects of his git update.
[LTS] Attended the monthly LTS meeting on IRC. Summary here.
It was also followed by a 40-minute discussion of technical questions/reviews/discussions - which in my opinion was pretty helpful. :)
[LTS] Prepared the LTS update for wordpress, bumping the package from 5.7.11 to 5.7.13.
Prepared an update for stable, Craig approved. Was waiting on the Security team’s +1 to upload.
Now we’ve waited enough that we have new CVEs. Oh well.
[ELTS] Finally setup debusine for ELTS uploads.
Since I use Ubuntu, this required installing debusine* from bookworm-backport but that required Python 3.11.
So I had to upgrade from Jammy (22.04) to Noble (24.04) - which was anyway pending.. :)
Deep Black is a far-future science fiction novel and the direct
sequel to Artifact Space. You do not
want to start here. I regretted not reading the novels closer together and
had to refresh my memory of what happened in the first book.
The shorter fiction in Beyond the Fringe
takes place between the two series novels and leads into some of the
events in this book, although reading it is optional.
Artifact Space left Marca Nbaro at the farthest point of the voyage
of the Greatship Athens, an unexpected heroine and now
well-integrated into the crew. On a merchant ship, however, there's always
more work to be done after a heroic performance. Deep Black opens
with that work: repairs from the events of the first book, the
never-ending litany of tasks required to keep the ship running smoothly,
and of course the trade with aliens that drew them so far out into the
Deep Black.
We knew early in the first book that this wouldn't be the simple, if long,
trading voyage that most of the crew of the Athens was expecting,
but now they have to worry about an unsettling second group of aliens on
top of a potential major war between human factions. They don't yet have
the cargo they came for, they have to reconstruct their trading post, and
they're a very long way from home. Marca also knows, at this point in the
story, that this voyage had additional goals from the start. She will
slowly gain a more complete picture of those goals during this novel.
Artifact Space was built around one of the most satisfying plots in
military science fiction (at least to me): a protagonist who benefits
immensely from the leveling effect and institutional inclusiveness of the
military slowly discovering that, when working at its best, the military
can be a true meritocracy. (The merchant marine of the Athens is
not military, precisely, since it's modeled on the trading ships of
Venice, but it's close enough for the purposes of this plot.) That's not a
plot that lasts into a sequel, though, so Cameron had to find a new spine
for the second half of the story. He chose first contact (of a sort) and
space battle.
The space battle parts are fine. I read a ton of children's World War II
military fiction when I was a boy, and I always preferred the naval
battles to the land battles. This part of Deep Black reminded me of
those naval battles, particularly a book whose title escapes me about the
Arctic convoys to the Soviet Union. I'm more interested in character than
military adventure these days, but every once in a while I enjoy reading
about a good space battle. This was not an exemplary specimen of the
genre, but it delivered on all the required elements.
The first contact part was more original, in part because Cameron chose an
interesting medium ground between total incomprehensibility and universal
translators. He stuck with the frustrations of communication for
considerably longer than most SF authors are willing to write, and it
worked for me. This is the first book I've read in a while where
superficial alien fluency with the mere words of a human language masks
continuing profound mutual incomprehension. The communication difficulties
are neither malicious nor a setup for catastrophic misunderstanding, but
an intrinsic part of learning about a truly alien species. I liked this,
even though it makes for slower and more frustrating progress. It felt
more believable than a lot of first contact, and it forced the characters
to take risks and act on hunches and then live with the consequences.
One of the other things that Cameron does well is maintain the steady
rhythm of life on a working ship as a background anchor to the story. I've
read a lot of science fiction that shows the day-to-day routine only until
something more interesting and plot-focused starts happening and then
seems to forget about it entirely. Not here. Marca goes through intense
and adrenaline-filled moments requiring risk and fast reactions, and then
has to handle promotion write-ups, routine watches, and studying for
advancement. Cameron knows that real battles involve long periods of
stressful waiting and incorporates them into the book without making them
too boring, which requires a lot of writing skill.
I prefer the emotional magic of finding a place where one belongs, so I
was not as taken with Deep Black as I was with Artifact
Space, but that's the inevitable result of plot progression and not
really a problem with this book. Marca is absurdly central to the story in
ways that have a whiff of "chosen one" dynamics, but if one can suspend
one's disbelief about that, the rest of the book is solid. This is,
fundamentally, a book about large space battles, so save it when you're in
the mood for that sort of story, but it was a satisfying continuation of
the series. I will definitely keep reading.
Recommended if you enjoyed Artifact Space. If you didn't,
Deep Black isn't going to change your mind.
Followed by Whalesong, which is not yet released (and is currently
in some sort of limbo for pre-orders in the US, which I hope will clear
up).
Updates on FAIme service: Linux Mint 22.2 and trixie backports available
The FAIme service [1] now offers to
build customized installation images for Xfce edition of Linux Mint 22.2 'Zara'.
For Debian 13 installations, you can select the kernel from backports for
the trixie release, which is currently version 6.16. This will support
newer hardware.
The Incandescent is a stand-alone magical boarding school fantasy.
Your students forgot you. It was natural for them to forget you. You
were a brief cameo in their lives, a walk-on character from the
prologue. For every sentimental my teacher changed my life
story you heard, there were dozens of my teacher made me
moderately bored a few times a week and then I got through the year
and moved on with my life and never thought about them again.
They forgot you. But you did not forget them.
Doctor Saffy Walden is Director of Magic at Chetwood, an elite boarding
school for prospective British magicians. She has a collection of
impressive degrees in academic magic, a specialization in demonic
invocation, and a history of vague but lucrative government job offers
that go with that specialty. She turned them down to be a teacher, and
although she's now in a mostly administrative position, she's a good
teacher, with the usual crop of promising, lazy, irritating, and nervous
students.
As the story opens, Walden's primary problem is Nikki Conway. Or, rather,
Walden's primary problem is protecting Nikki Conway from the Marshals, and
the infuriating Laura Kenning in particular.
When Nikki was seven, she summoned a demon who killed her entire family
and left her a ward of the school. To Laura Kenning, that makes her a risk
who should ideally be kept far away from invocation. To Walden, that makes
Nikki a prodigious natural talent who is developing into a brilliant
student and who needs careful, professional training before she's tempted
into trying to learn on her own.
Most novels with this setup would become Nikki's story. This one does not.
The Incandescent is Walden's story.
There have been a lot of young-adult magical boarding school novels since
Harry Potter became a mass phenomenon, but most of them focus on
the students and the inevitable coming-of-age story. This is a story about
the teachers: the paperwork, the faculty meetings, the funding challenges,
the students who repeat in endless variations, and the frustrations and
joys of attempting to grab the interest of a young mind. It's also about
the temptation of higher-paying, higher-status, and less ethical work,
which however firmly dismissed still nibbles around the edges.
Even if you didn't know Emily Tesh is herself a teacher, you would guess
that before you get far into this novel. There is a vividness and a depth
of characterization that comes from being deeply immersed in the nuance
and tedium of the life that your characters are living. Walden's
exasperated fondness for her students was the emotional backbone of this
book for me. She likes teenagers without idealizing the process of
being a teenager, which I think is harder to pull off in a novel than it
sounds.
It was hard to quantify the difference between a merely very
intelligent student and a brilliant one. It didn't show up in a list
of exam results. Sometimes, in fact, brilliance could be a
disadvantage — when all you needed to do was neatly jump the hoop of
an examiner's grading rubric without ever asking why. It was the
teachers who knew, the teachers who could feel the difference. A few
times in your career, you would have the privilege of teaching someone
truly remarkable; someone who was hard work to teach because they made
you work harder, who asked you questions that had never
occurred to you before, who stretched you to the very edge of your own
abilities. If you were lucky — as Walden, this time, had been lucky —
your remarkable student's chief interest was in your discipline: and
then you could have the extraordinary, humbling experience of teaching
a child whom you knew would one day totally surpass you.
I also loved the world-building, and I say this as someone who is
generally not a fan of demons. The demons themselves are a bit of a
disappointment and mostly hew to one of the stock demon conventions, but
the rest of the magic system is deep enough to have practitioners who
approach it from different angles and meaty enough to have some satisfying
layered complexity. This is magic, not magical science, so don't expect a
fully fleshed-out set of laws, but the magical system felt substantial and
satisfying to me.
Tesh's first novel, Some Desperate
Glory, was by far my favorite science fiction novel of 2023. This is a
much different book, which says good things about Tesh's range and the
potential of her work yet to come: adult rather than YA, fantasy rather
than science fiction, restrained and subtle in places where Some
Desperate Glory was forceful and pointed. One thing the books do have in
common, though, is some structure, particularly the false climax near the
midpoint of the book. I like the feeling of uncertainty and possibility
that gives both books, but in the case of The Incandescent, I was
not quite in the mood for the second half of the story.
My problem with this book is more of a reader preference than an objective
critique: I was in the mood for a story about a confident, capable
protagonist who was being underestimated, and Tesh was writing a novel
with a more complicated and fraught emotional arc. (I'm being
intentionally vague to avoid spoilers.) There's nothing wrong with the
story that Tesh wanted to tell, and I admire the skill with which she did
it, but I got a tight feeling in my stomach when I realized where she was
going. There is a satisfying ending, and I'm still very happy I read this
book, but be warned that this might not be the novel to read if you're in
the mood for a purer competence porn experience.
Recommended, and I am once again eagerly awaiting the next thing Emily
Tesh writes (and reminding myself to go back and read her novellas).
Content warnings: Grievous physical harm, mind control, and some body
horror.
Review: Echoes of the Imperium, by Nicholas & Olivia Atwater
Series:
Tales of the Iron Rose #1
Publisher:
Starwatch Press
Copyright:
2024
ISBN:
1-998257-04-5
Format:
Kindle
Pages:
547
Echoes of the Imperium is a steampunk fantasy adventure novel, the
first of a projected series. There is another novella in the series,
A Matter of Execution, that takes place chronologically before this
novel, but which I am told that you should read afterwards. (I have not
yet read it.) If Olivia Atwater's name sounds familiar, it's probably for
the romantic fantasy Half a Soul. Nicholas Atwater is her husband.
William Blair, a goblin, was a child sailor on the airship HMS
Caliban during the final battle that ended the Imperium, and an
eyewitness to the destruction of the capital. Like every imperial solider,
that loss made him an Oathbreaker; the fae Oath that he swore to defend
the Imperium did not care that nothing a twelve-year-old boy could have
done would have changed the result of the battle. He failed to kill
himself with most of the rest of the crew, and thus was taken captive by
the Coalition.
Twenty years later, William Blair is the goblin captain of the airship
Iron Rose. It's an independent transport ship that takes various
somewhat-dodgy contracts and has to avoid or fight through pirates. The
crew comes from both sides of the war and has built their own working
truce. Blair himself is a somewhat manic but earnest captain who doesn't
entirely believe he deserves that role, one who tends more towards wildly
risky plans and improvisation than considered and sober decisions. The
rest of the crew are the sort of wild mix of larger-than-life personality
quirks that populate swashbuckling adventure books but leave me dubious
that stuffing that many high-maintenance people into one ship would go as
well as it does.
I did appreciate the gunnery knitting circle, though.
Echoes of the Imperium is told in the first person from Blair's
perspective in two timelines. One follows Blair in the immediate aftermath
of the war, tracing his path to becoming an airship captain and meeting
some of the people who will later be part of his crew. The other is the
current timeline, in which Blair gets deeper and deeper into danger by
accepting a risky contract with unexpected complications.
Neither of these timelines are in any great hurry to arrive at some
destination, and that's the largest problem with this book. Echoes
of the Imperium is long, sprawling, and unwilling to get anywhere near
any sort of a point until the reader is deeply familiar with the horrific
aftermath of the war, the mountains guilt and trauma many of the
characters carry around, and Blair's impostor syndrome and feelings of
inadequacy. For the first half of this book, I was so bored. I almost
bailed out; only a few flashes of interesting character interactions and
hints of world-building helped me drag myself through all of the tedious
setup.
What saves this book is that the world-building is a delight. Once the
characters finally started engaging with it in earnest, I could not
put it down. Present-time Blair is no longer an Oathbreaker because he was
forgiven by a fairy; this will become important later. The sites of great
battles are haunted by ghostly echoes of the last moments of the lives of
those who died (hence the title); this will become very important later.
Blair has a policy of asking no questions about people's pasts if they're
willing to commit to working with the rest of the crew; this, also, will
become important later. All of these tidbits the authors drop into the
story and then ignore for hundreds of pages do have a payoff if you're
willing to wait for it.
As the reader (too) slowly discovers, the Atwaters' world is set in a war
of containment by light fae against dark fae. Instead of being inscrutable
and separate, the fae use humans and human empires as tools in that war.
The fallen Imperium was a bastion of fae defense, and the war that led to
the fall of that Imperium was triggered by the price its citizens paid for
that defense, one that the fae could not possibly care less about. The
creatures may be out of epic fantasy and the technology from the imagined
future of Victorian steampunk, but the politics are that of the Cold War
and containment strategies. This book has a lot to say about colonialism
and empire, but it says those things subtly and from a fantasy slant, in
a world with magical Oaths and direct contact with powers that are both
far beyond the capabilities of the main characters and woefully deficient
in in humanity and empathy. It has a bit of the feel of Greek mythology if
the gods believed in an icy realpolitik rather than embodying the
excesses of human emotion.
The second half of this book was fantastic. The found-family vibe among a
crew of high-maintenance misfits that completely failed to cohere for me
in the first half of the book, while Blair was wallowing in his feelings
and none of the events seemed to matter, came together brilliantly as soon
as the crew had a real problem and some meaty world-building and plot to
sink their teeth into. There is a delightfully competent teenager, some
satisfying competence porn that Blair finally stops undermining, and a
sharp political conflict that felt emotionally satisfying, if perhaps not
that intellectually profound. In short, it turns into the fun, adventurous
romp of larger-than-life characters that the setting promises. Even the
somewhat predictable mid-book reveal worked for me, in part because the
emotions of the characters around that reveal sold its impact.
If you're going to write a book with a bad half and a good half, it's
always better to put the good half second. I came away with very positive
feelings about Echoes of the Imperium and a tentative willingness
to watch for the sequel. (It reaches a fairly satisfying conclusion, but
there are a lot of unresolved plot hooks.) I'm a bit hesitant to recommend
it, though, because the first half was not very fun. I want to say that
about 75% of the first half of the book could have been cut and the book
would have been stronger for it. I'm not completely sure I'm right, since
the Atwaters were laying the groundwork for a lot of payoff, but I wish
that groundwork hadn't been as much of a slog.
Tentatively recommended, particularly if you're in the mood for steampunk
fae mythology, but know that this book requires some investment.
Technically, A Matter of Execution comes first, but I plan to read
it as a sequel.
As I was shopping groceries I had a shocking realization: The active dependencies
of packages in a solver actually form a trie (a dependency A|B - “A or B” - of
a package X is considered active if we marked X for install).
Consider the dependencies A|B|C, A|B, B|X.
In most package managers these just express alternatives, that is, the “or” relationship,
but in Debian packages, it also expresses a preference relationship between its operands,
so in A|B|C, A is preferred over B and B over C (and A transitively over C).
This means that we can convert the three dependencies into a trie as follows:
Solving the dependency here becomes a matter of trying to install the package
referenced by the first edge of the root, and seeing if that sticks. In this
case, that would be ‘a’. Let’s assume that ‘a’ failed to install, the next
step is to remove the empty node of a, and merging its children into the
root.
For ease of visualisation, we remove “a” from the dependency nodes as well,
leading us to a trie of the dependencies “b”, “b|c”, and “b|x”.
Presenting the Debian dependency problem, or the positive part of it as a
trie allows us for a great visualization of the problem but it may not proof
to be an effective implementation choice.
In the real world we may actually store this as a priority queue that we
can delete from. Since we don’t actually want to delete from the queue
for real, our queue items are pairs of a pointer to dependency and an
activitity level, say A|B@1.
Whenever a variable is assigned false, we look at its reverse dependencies
and bump their activity, and reinsert them (the priority of the item being
determined by the leftmost solution still possible, it has now changed).
When we iterate the queue, we remove items with a lower activity level:
Our queue is A|B@1, A|B|C@1, B|X@1
Rejecting A bump the activity for its reverse dependencies and reinset them:
Our queue is A|B@1, A|B|C@1, (A|)B@2, (A|)B|C@2, B|X@1
We visit A|B@1 but see the activity of the underlying dependency is now 2 and remove it
Our queue is A|B|C@1, (A|)B@2, (A|)B|C@2, B|X@1
We visit A|B|C@1 but see the activity of the underlying dependency is now 2 and remove it
Our queue is (A|)B@2, (A|)B|C@2, B|X@1
We visit A|B@2, see the activity matches and find B is the solution.
Sometimes, it's nice to write up something that was a solution
to an interesting problem but that didn't work; perhaps
someone else can figure out a crucial missing piece, or perhaps
their needs are subtly different. Or perhaps they'll just find it
interesting. This is such a post.
The problem in question is that I have a medium-sized sparse
bitset (say, 1024 bits) and some of those bits (say, 20–50,
but may be more and may be less) are set. I want to iterate
over those bits, spending as little time as possible on the
rest.
The standard formulation (as far as I know, anyway?),
given modern CPUs, is to treat them as a series of 64-bit
unsigned integers, and then use a double for loop like this
(C++20, but should be easily adaptable to any low-level enough
language):
// Assume we have: uint64_t arr[1024 / 64];
for (unsigned block = 0; block < 1024 / 64; ++block) {
for (unsigned bits = arr[block]; bits != 0; bits &= bits - 1) {
unsigned idx = 64 * block + std::countr_zero(bits);
// do something with idx here
}
}
The building blocks are simple enough if you're familiar with
bit manipulation; std::countr_zero() invokes a bit-scan instruction,
and
bits &= bits - 1
clears the lowest set bit.
This is roughly proportional to the number of set bits in the bit
set, except that if you have lots of zeros, you'll spend time
skipping over empty blocks. That's fine. What's not fine is that
this is a disaster for the branch predictor, and my code was (is!)
spending something like 20% of its time in the CPU handling
mispredicted branches. The structure of the two loops is just so
irregular; what we'd like is a branch-free way of iterating.
Now, we can of course never be fully branch-free; in particular,
we need to end the loop at some point, and that branch needs to
be predicted. So call it branch…less? Less branchy. Perhaps.
(As an aside; of course you could just test the bits one by one,
but that means you always get work proportional to the number of
total bits, and you still get really difficult branch prediction,
so I'm not going to discuss that option.)
Now, here are a bunch of things I tried to make this work that
didn't.
First, there's a way to splat the bit set into uint8_t indexes
using AVX512 (after which you can iterate over them using a normal
for loop); it's based on setting up a full adder-like structure
and then using compressed writes. I tried it, and it was just
way too slow. Geoff Langdale has the code
(in a bunch of different formulations) if you'd like to look at it
yourself.
So, the next natural attempt is to try to make larger blocks.
If we had an uint128_t and could use that just like we did with
uint64_t, we'd make life easier for the branch predictor since
there would be, simply put, fewer times the inner loop would end.
You can do it branch-free by means of conditional moves and such
(e.g., do two bit scans, switch between them based on whether the
lowest word is zero or not—similar for the other operations),
and there is some support from the compiler (__uint128_t
on GCC-like platforms), but in the end, going to 128 was just
not enough to end up net positive.
Going to 256 or 512 wasn't easily workable; you don't have
bit-scan instructions over the entire word, nor really anything
like whole word subtraction. And moving data between the SIMD
and integer pipes typically has a cost in itself.
So I started thinking; isn't this much of what a decompressor
does? We don't really care about the higher bits of the word;
as long as we can get the position of the lowest one, we
don't care whether we have few or many left. So perhaps we
can look at the input more like a bit stream (or byte stream)
than a series of blocks; have a loop where we find the lowest bit,
shift everything we just skipped or processed out, and then
refill bits from the top. As always, Fabian Giesen has a thorough treatise
on the subject. I wasn't concerned with squeezing every last
drop out, and my data order was largely fixed anyway, so I
only tried two different ways, really:
The first option is what a typical decompressor would do,
except byte-based; once I've got a sufficient number of zero
bits at the bottom, shift them out and reload bytes at the top.
This can be done largely branch-free, so in a sense, you only
have a single loop, you just keep reloading and reloading until
the end. (There are at least two ways to do this reloading;
you can reload only at the top, or you can reload the entire
64-bit word and mask out the bits you just read. They seemed
fairly equivalent in my case.) There is a problem with the ending,
though; you can read past the end. This may or may not be a problem;
it was for me, but it wasn't the biggest problem (see below),
so I let it be.
The other variant is somewhat more radical; I always read exactly
the next 64 bits (after the previously found bit). This is done
by going back to the block idea; a 64-bit word will overlap exactly
one or two blocks, so we read 128 bits (two consecutive blocks)
and shift the right number of bits to the right. x86 has 128-bit
shifts (although they're not that fast), so this makes it fairly
natural, and you can use conditional moves to make sure the
second read never goes past the end of the buffer, so this feels
overall like a slightly saner option.
However: None of them were faster than the normal double-loop.
And I think (but never found the energy to try to positively
prove) that comes down to an edge case: If there's not a single
bit set in the 64-bit window, we need to handle that specially.
So there we get back a fairly unpredictable branch after all—or
at least, in my data set, this seems to happen fairly often.
If you've got a fairly dense bit set, this won't be an issue,
but then you probably have more friendly branch behavior in the
loop, too. (For the reference, I have something like 3% branch
misprediction overall, which is really bad when most of the
stuff that I do involves ANDing bit vectors with each other!)
So, that's where I ended up. It's back to the double-loop.
But perhaps someone will be able to find a magic trick
that I missed. Email is welcome if you ever got this to work. :-)
rebuilderd v0.25.0 was recently released, this version has improved in-toto support for cryptographic attestations that this blog post briefly outlines. 😺
As a quick recap, rebuilderd is an automatic build scheduler that emerged in 2019/2020 from the Reproducible Builds project doing the following:
Track binary packages available in a Linux distribution
Attempt to compile the official binary packages from their (alleged) source code
Check if the package we compiled is bit-for-bit identical
If so, mark it GOOD, issue an attestation
In every other case, mark it BAD, generate a diffoscope
The binary packages in question are explicitly the packages users would also fetch and install.
The original in-toto integration was added 4 years ago by Joy Liu during GSoC 2021, with help from Santiago Torres and Aditya Sirish (shoutout to the real ones!). Each rebuilderd-worker had its own cryptographic key and included a signed attestation along with the build result that could then be fetched from /api/v0/builds/{id}/attestation.
Since these workers are potentially ephemeral, and the list of worker public keys wasn’t publicly known, it was difficult to make use of those signatures.
Since this version
This version introduces the following:
The rebuilderd daemon itself generates a long-term signing key
All attestations signed by a trusted worker also get signed by the rebuilderd daemon
The rebuilderd daemon gets a new endpoint that can be used to query the public-key this instance identifies with: /api/v0/public-keys
An example of this new endpoint can be found here:
The response looks something like this (this is the real long-term signing key used by reproducible.archlinux.org):
{"current":["-----BEGIN PUBLIC KEY-----\r\nMCwwBwYDK2VwBQADIQBLNcEcgErQ1rZz9oIkUnzc3fPuqJEALr22rNbrBK7iqQ==\r\n-----END PUBLIC KEY-----\r\n"]}
It’s a list so keys can potentially be rolled over time, and in future versions it should also list the public keys the instance has used in the past.
I haven’t develop any integrations for this yet (partially also to allow deployments to catch up with the new release), but I’m planning to do so using the in-toto crate.
Closing words
To give credit where credit is due (and because people pointed out I tend to end my blog posts too abruptly), rebuilderd is only the scheduler software, the actual build in the correct build-environment is outsourced to external tooling like archlinux-repro and debrebuild.
Also, there are currently efforts by the European Commission to outlaw unregulated end-to-end encrypted chat, so this may be a good time to prepare for (potential) impact and check what tools you have available to reduce unchecked trust in (open source) software authorities, to keep them operating honest and accountable.
I had to rent a house for a couple of months recently, which is long enough in California that it pushes you into proper tenant protection law. As landlords tend to do, they failed to return my security deposit within the 21 days required by law, having already failed to provide the required notification that I was entitled to an inspection before moving out. Cue some tedious argumentation with the letting agency, and eventually me threatening to take them to small claims court.
This post is not about that.
Now, under Californian law, the onus is on the landlord to hold and return the security deposit - the agency has no role in this. The only reason I was talking to them is that my lease didn't mention the name or address of the landlord (another legal violation, but the outcome is just that you get to serve the landlord via the agency). So it was a bit surprising when I received an email from the owner of the agency informing me that they did not hold the deposit and so were not liable - I already knew this.
The odd bit about this, though, is that they sent me another copy of the contract, asserting that it made it clear that the landlord held the deposit. I read it, and instead found a clause reading SECURITY: The security deposit will secure the performance of Tenant’s obligations. IER may, but will not be obligated to, apply all portions of said deposit on account of Tenant’s obligations. Any balance remaining upon termination will be returned to Tenant. Tenant will not have the right to apply the security deposit in payment of the last month’s rent. Security deposit held at IER Trust Account., where IER is International Executive Rentals, the agency in question. Why send me a contract that says you hold the money while you're telling me you don't? And then I read further down and found this: Ok, fair enough, there's an addendum that says the landlord has it (I've removed the landlord's name, it's present in the original).
Except. I had no recollection of that addendum. I went back to the copy of the contract I had and discovered: Huh! But obviously I could just have edited that to remove it (there's no obvious reason for me to, but whatever), and then it'd be my word against theirs. However, I'd been sent the document via RightSignature, an online document signing platform, and they'd added a certification page that looked like this: Interestingly, the certificate page was identical in both documents, including the checksums, despite the content being different. So, how do I show which one is legitimate? You'd think given this certificate page this would be trivial, but RightSignature provides no documented mechanism whatsoever for anyone to verify any of the fields in the certificate, which is annoying but let's see what we can do anyway.
First up, let's look at the PDF metadata. pdftk has a dump_data command that dumps the metadata in the document, including the creation date and the modification date. My file had both set to identical timestamps in June, both listed in UTC, corresponding to the time I'd signed the document. The file containing the addendum? The same creation time, but a modification time of this Monday, shortly before it was sent to me. This time, the modification timestamp was in Pacific Daylight Time, the timezone currently observed in California. In addition, the data included two ID fields, ID0 and ID1. In my document both were identical, in the one with the addendum ID0 matched mine but ID1 was different.
These ID tags are intended to be some form of representation (such as a hash) of the document. ID0 is set when the document is created and should not be modified afterwards - ID1 initially identical to ID0, but changes when the document is modified. This is intended to allow tooling to identify whether two documents are modified versions of the same document. The identical ID0 indicated that the document with the addendum was originally identical to mine, and the different ID1 that it had been modified.
Well, ok, that seems like a pretty strong demonstration. I had the "I have a very particular set of skills" conversation with the agency and pointed these facts out, that they were an extremely strong indication that my copy was authentic and their one wasn't, and they responded that the document was "re-sealed" every time it was downloaded from RightSignature and that would explain the modifications. This doesn't seem plausible, but it's an argument. Let's go further.
My next move was pdfalyzer, which allows you to pull a PDF apart into its component pieces. This revealed that the documents were identical, other than page 3, the one with the addendum. This page included tags entitled "touchUp_TextEdit", evidence that the page had been modified using Acrobat. But in itself, that doesn't prove anything - obviously it had been edited at some point to insert the landlord's name, it doesn't prove whether it happened before or after the signing.
But in the process of editing, Acrobat appeared to have renamed all the font references on that page into a different format. Every other page had a consistent naming scheme for the fonts, and they matched the scheme in the page 3 I had. Again, that doesn't tell us whether the renaming happened before or after the signing. Or does it?
You see, when I completed my signing, RightSignature inserted my name into the document, and did so using a font that wasn't otherwise present in the document (Courier, in this case). That font was named identically throughout the document, except on page 3, where it was named in the same manner as every other font that Acrobat had renamed. Given the font wasn't present in the document until after I'd signed it, this is proof that the page was edited after signing.
But eh this is all very convoluted. Surely there's an easier way? Thankfully yes, although I hate it. RightSignature had sent me a link to view my signed copy of the document. When I went there it presented it to me as the original PDF with my signature overlaid on top. Hitting F12 gave me the network tab, and I could see a reference to a base.pdf. Downloading that gave me the original PDF, pre-signature. Running sha256sum on it gave me an identical hash to the "Original checksum" field. Needless to say, it did not contain the addendum.
Why do this? The only explanation I can come up with (and I am obviously guessing here, I may be incorrect!) is that International Executive Rentals realised that they'd sent me a contract which could mean that they were liable for the return of my deposit, even though they'd already given it to my landlord, and after realising this added the addendum, sent it to me, and assumed that I just wouldn't notice (or that, if I did, I wouldn't be able to prove anything). In the process they went from an extremely unlikely possibility of having civil liability for a few thousand dollars (even if they were holding the deposit it's still the landlord's legal duty to return it, as far as I can tell) to doing something that looks extremely like forgery.
There's a hilarious followup. After this happened, the agency offered to do a screenshare with me showing them logging into RightSignature and showing the signed file with the addendum, and then proceeded to do so. One minor problem - the "Send for signature" button was still there, just below a field saying "Uploaded: 09/22/25". I asked them to search for my name, and it popped up two hits - one marked draft, one marked completed. The one marked completed? Didn't contain the addendum.
If you are like me that you are installing machines with testing and then go and flip them over to the current stable for a while using APT::Default-Release, you might not be receiving all relevant updates. In fact this setting is kind of discouraged in favor of more extensive pinning configuration.
However, the field does support regexps, so instead of just specifying, say, "trixie", you can put this in place:
In December 2024, I went on a trip through four countries - Singapore, Malaysia, Brunei, and Vietnam - with my friend Badri. This post covers our experiences in Singapore.
I took an IndiGo flight from Delhi to Singapore, with a layover in Chennai. At the Chennai airport, I was joined by Badri. We had an early morning flight from Chennai that would land in Singapore in the afternoon. Within 48 hours of our scheduled arrival in Singapore, we submitted an arrival card online. At immigration, we simply needed to scan our passports at the gates, which opened automatically to let us through, and then give our address to an official nearby. The process was quick and smooth, but it unfortunately meant that we didn’t get our passports stamped by Singapore.
Before I left the airport, I wanted to visit the nature-themed park with a fountain I saw in pictures online. It is called Jewel Changi, and it took quite some walking to get there. After reaching the park, we saw a fountain that could be seen from all the levels. We roamed around for a couple of hours, then proceeded to the airport metro station to get to our hotel.
A shot of Jewel Changi. Photo by Ravi Dwivedi. Released under the CC-BY-SA 4.0.
There were four ATMs on the way to the metro station, but none of them provided us with any cash. This was the first country (outside India, of course!) where my card didn’t work at ATMs.
To use the metro, one can tap the EZ-Link card or bank cards at the AFC gates to get in. You cannot buy tickets using cash. Before boarding the metro, I used my credit card to get Badri an EZ-Link card from a vending machine. It was 10 Singapore dollars (₹630) - 5 for the card, and 5 for the balance. I had planned to use my Visa credit card to pay for my own fare. I was relieved to see that my card worked, and I passed through the AFC gates.
We had booked our stay at a hostel named Campbell’s Inn, which was the cheapest we could find in Singapore. It was ₹1500 per night for dorm beds. The hostel was located in Little India. While Little India has an eponymous metro station, the one closest to our hostel was Rochor.
On the way to the hostel, we found out that our booking had been canceled.
We had booked from the Hostelworld website, opting to pay the deposit in advance and to pay the balance amount in person upon reaching. However, Hostelworld still tried to charge Badri’s card again before our arrival. When the unauthorized charge failed, they sent an automatic message saying “we tried to charge” and to contact them soon to avoid cancellation, which we couldn’t do as we were in the plane.
Despite this, we went to the hostel to check the status of our booking.
The trip from the airport to Rochor required a couple of transfers. It was 2 Singapore dollars (approx. ₹130) and took approximately an hour.
Upon reaching the hostel, we were informed that our booking had indeed been canceled, and were not given any reason for the cancelation. Furthermore, no beds were available at the hostel for us to book on the spot.
We decided to roam around and look for accommodation at other hostels in the area. Soon, we found a hostel by the name of Snooze Inn, which had two beds available. It was 36 Singapore dollars per person (around ₹2300) for a dormitory bed. Snooze Inn advertised supporting RuPay cards and UPI. Some other places in that area did the same. We paid using my card. We checked in and slept for a couple of hours after taking a shower.
By the time we woke up, it was dark. We met Praveen’s friend Sabeel to get my FLX1 phone. We also went to Mustafa Center nearby to exchange Indian rupees for Singapore dollars. Mustafa Center also had a shopping center with shops selling electronic items and souvenirs, among other things. When we were dropping off Sabeel at a bus stop, we discovered that the bus stops in Singapore had a digital board mentioning the bus routes for the stop and the number of minutes each bus was going to take.
In addition to an organized bus system, Singapore had good pedestrian infrastructure. There were traffic lights and zebra crossings for pedestrians to cross the roads. Unlike in Indian cities, rules were being followed. Cars would stop for pedestrians at unmanaged zebra crossings; pedestrians would in turn wait for their crossing signal to turn green before attempting to walk across. Therefore, walking in Singapore was easy.
Traffic rules were taken so seriously in Singapore I (as a pedestrian) was afraid of unintentionally breaking them, which could get me in trouble, as breaking rules is dealt with heavy fines in the country. For example, crossing roads without using a marked crossing (while being within 50 meters of it) - also known as jaywalking - is an offence in Singapore.
Moreover, the streets were litter-free, and cleanliness seemed like an obsession.
After exploring Mustafa Center, we went to a nearby 7-Eleven to top up Badri’s EZ-Link card. He gave 20 Singapore dollars for the recharge, which credited the card by 19.40 Singapore dollars (0.6 dollars being the recharge fee).
When I was planning this trip, I discovered that the World Chess Championship match was being held in Singapore. I seized the opportunity and bought a ticket in advance. The next day - the 5th of December - I went to watch the 9th game between Gukesh Dommaraju of India and Ding Liren of China. The venue was a hotel on Sentosa Island, and the ticket was 70 Singapore dollars, which was around ₹4000 at the time.
We checked out from our hostel in the morning, as we were planning to stay with Badri’s aunt that night. We had breakfast at a place in Little India. Then we took a couple of buses, followed by a walk to Sentosa Island. Paying the fare for the buses was similar to the metro - I tapped my credit card in the bus, while Badri tapped his EZ-Link card. We also had to tap it while getting off.
If you are tapping your credit card to use public transport in Singapore, keep in mind that the total amount of all the trips taken on a day is deducted at the end. This makes it hard to determine the cost of individual trips. For example, I could take a bus and get off after tapping my card, but I would have no way to determine how much this journey cost.
When you tap in, the maximum fare amount gets deducted. When you tap out, the balance amount gets refunded (if it’s a shorter journey than the maximum fare one). So, there is incentive for passengers not to get off without tapping out. Going by your card statement, it looks like all that happens virtually, and only one statement comes in at the end. Maybe this combining only happens for international cards.
We got off the bus a kilometer away from Sentosa Island and walked the rest of the way. We went on the Sentosa Boardwalk, which is itself a tourist attraction. I was using Organic Maps to navigate to the hotel Resorts World Sentosa, but Organic Maps’ route led us through an amusement park. I tried asking the locals (people working in shops) for directions, but it was a Chinese-speaking region, and they didn’t understand English. Fortunately, we managed to find a local who helped us with the directions.
A shot of Sentosa Boardwalk. Photo by Ravi Dwivedi. Released under the CC-BY-SA 4.0.
Following the directions, we somehow ended up having to walk on a road which did not have pedestrian paths. Singapore is a country with strict laws, so we did not want to walk on that road. Avoiding that road led us to the Michael Hotel. There was a person standing at the entrance, and I asked him for directions to Resorts World Sentosa. The person told me that the bus (which was standing at the entrance) would drop me there! The bus was a free service for getting to Resorts World Sentosa. Here I parted ways with Badri, who went to his aunt’s place.
I got to the Resorts Sentosa and showed my ticket to get in. There were two zones inside - the first was a room with a glass wall separating the audience and the players. This was the room to watch the game physically, and resembled a zoo or an aquarium. :) The room was also a silent room, which means talking or making noise was prohibited. Audiences were only allowed to have mobile phones for the first 30 minutes of the game - since I arrived late, I could not bring my phone inside that room.
The other zone was outside this room. It had a big TV on which the game was being broadcast along with commentary by David Howell and Jovanka Houska - the official FIDE commentators for the event. If you don’t already know, FIDE is the authoritative international chess body.
I spent most of the time outside that silent room, giving me an opportunity to socialize. A lot of people were from Singapore. I saw there were many Indians there as well. Moreover, I had a good time with Vasudevan, a journalist from Tamil Nadu who was covering the match. He also asked questions to Gukesh during the post-match conference. His questions were in Tamil to lift Gukesh’s spirits, as Gukesh is a Tamil speaker.
Tea and coffee were free for the audience. I also bought a T-shirt from their stall as a souvenir.
After the game, I took a shuttle bus from Resorts World Sentosa to a metro station, then travelled to Pasir Ris by metro, where Badri was staying with his aunt. I thought of getting something to eat, but could not find any cafés or restaurants while I was walking from the Pasir Ris metro station to my destination, and was positively starving when I got there.
Badri’s aunt’s place was an apartment in a gated community. On the gate was a security guard who asked me the address of the apartment. Upon entering, there were many buildings. To enter the building, you need to dial the number of the apartment you want to go to and speak to them. I had seen that in the TV show Seinfeld, where Jerry’s friends used to dial Jerry to get into his building.
I was afraid they might not have anything to eat because I told them I was planning to get something on the way. This was fortunately not the case, and I was relieved to not have to sleep with an empty stomach.
Badri’s uncle gave us an idea of how safe Singapore is. He said that even if you forget your laptop in a public space, you can go back the next day to find it right there in the same spot. I also learned that owning cars was discouraged in Singapore - the government imposes a high registration fee on them, while also making public transport easy to use and affordable. I also found out that 7-Eleven was not that popular among residents in Singapore, unlike in Malaysia or Thailand.
The next day was our third and final day in Singapore. We had a bus in the evening to Johor Bahru in Malaysia. We got up early, had breakfast, and checked out from Badri’s aunt’s home. A store by the name of Cat Socrates was our first stop for the day, as Badri wanted to buy some stationery. The plan was to take the metro, followed by the bus. So we got to Pasir Ris metro station. Next to the metro station was a mall. In the mall, Badri found an ATM where our cards worked, and we got some Singapore dollars.
It was noon when we reached the stationery shop mentioned above. We had to walk a kilometer from the place where the bus dropped us. It was a hot, sunny day in Singapore, so walking was not comfortable. We had to go through residential areas in Singapore. We saw some non-touristy parts of Singapore.
After we were done with the stationery shop, we went to a hawker center to get lunch. Hawker centers are unique to Singapore. They have a lot of shops that sell local food at cheap prices. It is similar to a food court. However, unlike the food courts in malls, hawker centers are open-air and can get quite hot.
This is the hawker center we went to. Photo by Ravi Dwivedi. Released under the CC-BY-SA 4.0.
To have something, you just need to buy it from one of the shops and find a table. After you are done, you need to put your tray in the tray-collecting spots. I had a kaya toast with chai, since there weren’t many vegetarian options. I also bought a persimmon from a nearby fruit vendor. On the other hand, Badri sampled some local non-vegetarian dishes.
Table littering at the hawker center was prohibited by law. Photo by Ravi Dwivedi. Released under the CC-BY-SA 4.0.
Next, we took a metro to Raffles Place, as we wanted to visit Merlion, the icon of Singapore. It is a statue having the head of a lion and the body of a fish. While getting through the AFC gates, my card was declined. Therefore, I had to buy an EZ-Link card, which I had been avoiding because the card itself costs 5 Singapore dollars.
From the Raffles Place metro station, we walked to Merlion. The place also gave a nice view of Marina Bay Sands. It was filled with tourists clicking pictures, and we also did the same.
Merlion from behind, giving a good view of Marina Bay Sands. Photo by Ravi Dwivedi. Released under the CC-BY-SA 4.0.
After this, we went to the bus stop to catch our bus to the border city of Johor Bahru, Malaysia. The bus was more than an hour late, and we worried that we had missed the bus. I asked an Indian woman at the stop who also planned to take the same bus, and she told us that the bus was late. Finally, our bus arrived, and we set off for Johor Bahru.
Before I finish, let me give you an idea of my expenditure. Singapore is an expensive country, and I realized that expenses could go up pretty quickly. Overall, my stay in Singapore for 3 days and 2 nights was approx. 5500 rupees. That too, when we stayed one night at Badri’s aunt’s place (so we didn’t have to pay for accomodation for one of the nights) and didn’t have to pay for a couple of meals. This amount doesn’t include the ticket for the chess game, but includes the costs of getting there. If you are in Singapore, it is likely you will pay a visit to Sentosa Island anyway.
Stay tuned for our experiences in Malaysia!
Credits: Thanks to Dione, Sahil, Badri and Contrapunctus for reviewing the draft. Thanks to Bhe for spotting a duplicate sentence.
If you're still using Vagrant (I am) and try to boot a box that uses UEFI (like boxen/debian-13),
a simple vagrant init boxen/debian-13 and vagrant up will entertain you with a nice traceback:
% vagrantup
Bringing machine 'default' up with 'libvirt' provider...==> default: Checking if box 'boxen/debian-13' version '2025.08.20.12' is up to date...==> default: Creating image (snapshot of base box volume).==> default: Creating domain with the following settings...==> default: -- Name: tmp.JV8X48n30U_default==> default: -- Description: Source: /tmp/tmp.JV8X48n30U/Vagrantfile==> default: -- Domain type: kvm==> default: -- Cpus: 1==> default: -- Feature: acpi==> default: -- Feature: apic==> default: -- Feature: pae==> default: -- Clock offset: utc==> default: -- Memory: 2048M==> default: -- Loader: /home/evgeni/.vagrant.d/boxes/boxen-VAGRANTSLASH-debian-13/2025.08.20.12/libvirt/OVMF_CODE.fd==> default: -- Nvram: /home/evgeni/.vagrant.d/boxes/boxen-VAGRANTSLASH-debian-13/2025.08.20.12/libvirt/efivars.fd==> default: -- Base box: boxen/debian-13==> default: -- Storage pool: default==> default: -- Image(vda): /home/evgeni/.local/share/libvirt/images/tmp.JV8X48n30U_default.img, virtio, 20G==> default: -- Disk driver opts: cache='default'==> default: -- Graphics Type: vnc==> default: -- Video Type: cirrus==> default: -- Video VRAM: 16384==> default: -- Video 3D accel: false==> default: -- Keymap: en-us==> default: -- TPM Backend: passthrough==> default: -- INPUT: type=mouse, bus=ps2==> default: -- CHANNEL: type=unix, mode===> default: -- CHANNEL: target_type=virtio, target_name=org.qemu.guest_agent.0==> default: Creating shared folders metadata...==> default: Starting domain.==> default: Removing domain...==> default: Deleting the machine folder/usr/share/gems/gems/fog-libvirt-0.13.1/lib/fog/libvirt/requests/compute/vm_action.rb:7:in 'Libvirt::Domain#create': Call to virDomainCreate failed: internal error: process exited while connecting to monitor: 2025-09-22T10:07:55.081081Z qemu-system-x86_64: -blockdev {"driver":"file","filename":"/home/evgeni/.vagrant.d/boxes/boxen-VAGRANTSLASH-debian-13/2025.08.20.12/libvirt/OVMF_CODE.fd","node-name":"libvirt-pflash0-storage","auto-read-only":true,"discard":"unmap"}: Could not open '/home/evgeni/.vagrant.d/boxes/boxen-VAGRANTSLASH-debian-13/2025.08.20.12/libvirt/OVMF_CODE.fd': Permission denied (Libvirt::Error) from /usr/share/gems/gems/fog-libvirt-0.13.1/lib/fog/libvirt/requests/compute/vm_action.rb:7:in 'Fog::Libvirt::Compute::Shared#vm_action' from /usr/share/gems/gems/fog-libvirt-0.13.1/lib/fog/libvirt/models/compute/server.rb:81:in 'Fog::Libvirt::Compute::Server#start' from /usr/share/vagrant/gems/gems/vagrant-libvirt-0.11.2/lib/vagrant-libvirt/action/start_domain.rb:546:in 'VagrantPlugins::ProviderLibvirt::Action::StartDomain#call' from /usr/share/vagrant/gems/gems/vagrant-2.3.4/lib/vagrant/action/warden.rb:48:in 'Vagrant::Action::Warden#call' from /usr/share/vagrant/gems/gems/vagrant-libvirt-0.11.2/lib/vagrant-libvirt/action/set_boot_order.rb:22:in 'VagrantPlugins::ProviderLibvirt::Action::SetBootOrder#call' from /usr/share/vagrant/gems/gems/vagrant-2.3.4/lib/vagrant/action/warden.rb:48:in 'Vagrant::Action::Warden#call' from /usr/share/vagrant/gems/gems/vagrant-libvirt-0.11.2/lib/vagrant-libvirt/action/share_folders.rb:22:in 'VagrantPlugins::ProviderLibvirt::Action::ShareFolders#call' from /usr/share/vagrant/gems/gems/vagrant-2.3.4/lib/vagrant/action/warden.rb:48:in 'Vagrant::Action::Warden#call' from /usr/share/vagrant/gems/gems/vagrant-libvirt-0.11.2/lib/vagrant-libvirt/action/prepare_nfs_settings.rb:21:in 'VagrantPlugins::ProviderLibvirt::Action::PrepareNFSSettings#call' from /usr/share/vagrant/gems/gems/vagrant-2.3.4/lib/vagrant/action/warden.rb:48:in 'Vagrant::Action::Warden#call' from /usr/share/vagrant/gems/gems/vagrant-2.3.4/lib/vagrant/action/builtin/synced_folders.rb:87:in 'Vagrant::Action::Builtin::SyncedFolders#call' from /usr/share/vagrant/gems/gems/vagrant-2.3.4/lib/vagrant/action/warden.rb:48:in 'Vagrant::Action::Warden#call' from /usr/share/vagrant/gems/gems/vagrant-2.3.4/lib/vagrant/action/builtin/delayed.rb:19:in 'Vagrant::Action::Builtin::Delayed#call' from /usr/share/vagrant/gems/gems/vagrant-2.3.4/lib/vagrant/action/warden.rb:48:in 'Vagrant::Action::Warden#call' from /usr/share/vagrant/gems/gems/vagrant-2.3.4/lib/vagrant/action/builtin/synced_folder_cleanup.rb:28:in 'Vagrant::Action::Builtin::SyncedFolderCleanup#call' from /usr/share/vagrant/gems/gems/vagrant-2.3.4/lib/vagrant/action/warden.rb:48:in 'Vagrant::Action::Warden#call' from /usr/share/vagrant/gems/gems/vagrant-2.3.4/plugins/synced_folders/nfs/action_cleanup.rb:25:in 'VagrantPlugins::SyncedFolderNFS::ActionCleanup#call' from /usr/share/vagrant/gems/gems/vagrant-2.3.4/lib/vagrant/action/warden.rb:48:in 'Vagrant::Action::Warden#call' from /usr/share/vagrant/gems/gems/vagrant-libvirt-0.11.2/lib/vagrant-libvirt/action/prepare_nfs_valid_ids.rb:14:in 'VagrantPlugins::ProviderLibvirt::Action::PrepareNFSValidIds#call' from /usr/share/vagrant/gems/gems/vagrant-2.3.4/lib/vagrant/action/warden.rb:48:in 'Vagrant::Action::Warden#call' from /usr/share/vagrant/gems/gems/vagrant-2.3.4/lib/vagrant/action/warden.rb:127:in 'block in Vagrant::Action::Warden#finalize_action' from /usr/share/vagrant/gems/gems/vagrant-2.3.4/lib/vagrant/action/warden.rb:48:in 'Vagrant::Action::Warden#call' from /usr/share/vagrant/gems/gems/vagrant-2.3.4/lib/vagrant/action/builder.rb:180:in 'Vagrant::Action::Builder#call' from /usr/share/vagrant/gems/gems/vagrant-2.3.4/lib/vagrant/action/runner.rb:101:in 'block in Vagrant::Action::Runner#run' from /usr/share/vagrant/gems/gems/vagrant-2.3.4/lib/vagrant/util/busy.rb:19:in 'Vagrant::Util::Busy.busy' from /usr/share/vagrant/gems/gems/vagrant-2.3.4/lib/vagrant/action/runner.rb:101:in 'Vagrant::Action::Runner#run' from /usr/share/vagrant/gems/gems/vagrant-2.3.4/lib/vagrant/action/builtin/call.rb:53:in 'Vagrant::Action::Builtin::Call#call' from /usr/share/vagrant/gems/gems/vagrant-2.3.4/lib/vagrant/action/warden.rb:48:in 'Vagrant::Action::Warden#call' from /usr/share/vagrant/gems/gems/vagrant-2.3.4/lib/vagrant/action/warden.rb:127:in 'block in Vagrant::Action::Warden#finalize_action' from /usr/share/vagrant/gems/gems/vagrant-2.3.4/lib/vagrant/action/warden.rb:48:in 'Vagrant::Action::Warden#call' from /usr/share/vagrant/gems/gems/vagrant-2.3.4/lib/vagrant/action/builder.rb:180:in 'Vagrant::Action::Builder#call' from /usr/share/vagrant/gems/gems/vagrant-2.3.4/lib/vagrant/action/runner.rb:101:in 'block in Vagrant::Action::Runner#run' from /usr/share/vagrant/gems/gems/vagrant-2.3.4/lib/vagrant/util/busy.rb:19:in 'Vagrant::Util::Busy.busy' from /usr/share/vagrant/gems/gems/vagrant-2.3.4/lib/vagrant/action/runner.rb:101:in 'Vagrant::Action::Runner#run' from /usr/share/vagrant/gems/gems/vagrant-2.3.4/lib/vagrant/action/builtin/call.rb:53:in 'Vagrant::Action::Builtin::Call#call' from /usr/share/vagrant/gems/gems/vagrant-2.3.4/lib/vagrant/action/warden.rb:48:in 'Vagrant::Action::Warden#call' from /usr/share/vagrant/gems/gems/vagrant-libvirt-0.11.2/lib/vagrant-libvirt/action/create_network_interfaces.rb:197:in 'VagrantPlugins::ProviderLibvirt::Action::CreateNetworkInterfaces#call' from /usr/share/vagrant/gems/gems/vagrant-2.3.4/lib/vagrant/action/warden.rb:48:in 'Vagrant::Action::Warden#call' from /usr/share/vagrant/gems/gems/vagrant-libvirt-0.11.2/lib/vagrant-libvirt/action/create_networks.rb:40:in 'VagrantPlugins::ProviderLibvirt::Action::CreateNetworks#call' from /usr/share/vagrant/gems/gems/vagrant-2.3.4/lib/vagrant/action/warden.rb:48:in 'Vagrant::Action::Warden#call' from /usr/share/vagrant/gems/gems/vagrant-libvirt-0.11.2/lib/vagrant-libvirt/action/create_domain.rb:452:in 'VagrantPlugins::ProviderLibvirt::Action::CreateDomain#call' from /usr/share/vagrant/gems/gems/vagrant-2.3.4/lib/vagrant/action/warden.rb:48:in 'Vagrant::Action::Warden#call' from /usr/share/vagrant/gems/gems/vagrant-libvirt-0.11.2/lib/vagrant-libvirt/action/resolve_disk_settings.rb:143:in 'VagrantPlugins::ProviderLibvirt::Action::ResolveDiskSettings#call' from /usr/share/vagrant/gems/gems/vagrant-2.3.4/lib/vagrant/action/warden.rb:48:in 'Vagrant::Action::Warden#call' from /usr/share/vagrant/gems/gems/vagrant-libvirt-0.11.2/lib/vagrant-libvirt/action/create_domain_volume.rb:97:in 'VagrantPlugins::ProviderLibvirt::Action::CreateDomainVolume#call' from /usr/share/vagrant/gems/gems/vagrant-2.3.4/lib/vagrant/action/warden.rb:48:in 'Vagrant::Action::Warden#call' from /usr/share/vagrant/gems/gems/vagrant-libvirt-0.11.2/lib/vagrant-libvirt/action/handle_box_image.rb:127:in 'VagrantPlugins::ProviderLibvirt::Action::HandleBoxImage#call' from /usr/share/vagrant/gems/gems/vagrant-2.3.4/lib/vagrant/action/warden.rb:48:in 'Vagrant::Action::Warden#call' from /usr/share/vagrant/gems/gems/vagrant-2.3.4/lib/vagrant/action/builtin/handle_box.rb:56:in 'Vagrant::Action::Builtin::HandleBox#call' from /usr/share/vagrant/gems/gems/vagrant-2.3.4/lib/vagrant/action/warden.rb:48:in 'Vagrant::Action::Warden#call' from /usr/share/vagrant/gems/gems/vagrant-libvirt-0.11.2/lib/vagrant-libvirt/action/handle_storage_pool.rb:63:in 'VagrantPlugins::ProviderLibvirt::Action::HandleStoragePool#call' from /usr/share/vagrant/gems/gems/vagrant-2.3.4/lib/vagrant/action/warden.rb:48:in 'Vagrant::Action::Warden#call' from /usr/share/vagrant/gems/gems/vagrant-libvirt-0.11.2/lib/vagrant-libvirt/action/set_name_of_domain.rb:34:in 'VagrantPlugins::ProviderLibvirt::Action::SetNameOfDomain#call' from /usr/share/vagrant/gems/gems/vagrant-2.3.4/lib/vagrant/action/warden.rb:48:in 'Vagrant::Action::Warden#call' from /usr/share/vagrant/gems/gems/vagrant-2.3.4/lib/vagrant/action/builtin/provision.rb:80:in 'Vagrant::Action::Builtin::Provision#call' from /usr/share/vagrant/gems/gems/vagrant-2.3.4/lib/vagrant/action/warden.rb:48:in 'Vagrant::Action::Warden#call' from /usr/share/vagrant/gems/gems/vagrant-libvirt-0.11.2/lib/vagrant-libvirt/action/cleanup_on_failure.rb:21:in 'VagrantPlugins::ProviderLibvirt::Action::CleanupOnFailure#call' from /usr/share/vagrant/gems/gems/vagrant-2.3.4/lib/vagrant/action/warden.rb:48:in 'Vagrant::Action::Warden#call' from /usr/share/vagrant/gems/gems/vagrant-2.3.4/lib/vagrant/action/warden.rb:127:in 'block in Vagrant::Action::Warden#finalize_action' from /usr/share/vagrant/gems/gems/vagrant-2.3.4/lib/vagrant/action/warden.rb:48:in 'Vagrant::Action::Warden#call' from /usr/share/vagrant/gems/gems/vagrant-2.3.4/lib/vagrant/action/builder.rb:180:in 'Vagrant::Action::Builder#call' from /usr/share/vagrant/gems/gems/vagrant-2.3.4/lib/vagrant/action/runner.rb:101:in 'block in Vagrant::Action::Runner#run' from /usr/share/vagrant/gems/gems/vagrant-2.3.4/lib/vagrant/util/busy.rb:19:in 'Vagrant::Util::Busy.busy' from /usr/share/vagrant/gems/gems/vagrant-2.3.4/lib/vagrant/action/runner.rb:101:in 'Vagrant::Action::Runner#run' from /usr/share/vagrant/gems/gems/vagrant-2.3.4/lib/vagrant/action/builtin/call.rb:53:in 'Vagrant::Action::Builtin::Call#call' from /usr/share/vagrant/gems/gems/vagrant-2.3.4/lib/vagrant/action/warden.rb:48:in 'Vagrant::Action::Warden#call' from /usr/share/vagrant/gems/gems/vagrant-2.3.4/lib/vagrant/action/builtin/box_check_outdated.rb:93:in 'Vagrant::Action::Builtin::BoxCheckOutdated#call' from /usr/share/vagrant/gems/gems/vagrant-2.3.4/lib/vagrant/action/warden.rb:48:in 'Vagrant::Action::Warden#call' from /usr/share/vagrant/gems/gems/vagrant-2.3.4/lib/vagrant/action/builtin/config_validate.rb:25:in 'Vagrant::Action::Builtin::ConfigValidate#call' from /usr/share/vagrant/gems/gems/vagrant-2.3.4/lib/vagrant/action/warden.rb:48:in 'Vagrant::Action::Warden#call' from /usr/share/vagrant/gems/gems/vagrant-2.3.4/lib/vagrant/action/builder.rb:180:in 'Vagrant::Action::Builder#call' from /usr/share/vagrant/gems/gems/vagrant-2.3.4/lib/vagrant/action/runner.rb:101:in 'block in Vagrant::Action::Runner#run' from /usr/share/vagrant/gems/gems/vagrant-2.3.4/lib/vagrant/util/busy.rb:19:in 'Vagrant::Util::Busy.busy' from /usr/share/vagrant/gems/gems/vagrant-2.3.4/lib/vagrant/action/runner.rb:101:in 'Vagrant::Action::Runner#run' from /usr/share/vagrant/gems/gems/vagrant-2.3.4/lib/vagrant/machine.rb:248:in 'Vagrant::Machine#action_raw' from /usr/share/vagrant/gems/gems/vagrant-2.3.4/lib/vagrant/machine.rb:217:in 'block in Vagrant::Machine#action' from /usr/share/vagrant/gems/gems/vagrant-2.3.4/lib/vagrant/environment.rb:631:in 'Vagrant::Environment#lock' from /usr/share/vagrant/gems/gems/vagrant-2.3.4/lib/vagrant/machine.rb:203:in 'Method#call' from /usr/share/vagrant/gems/gems/vagrant-2.3.4/lib/vagrant/machine.rb:203:in 'Vagrant::Machine#action' from /usr/share/vagrant/gems/gems/vagrant-2.3.4/lib/vagrant/batch_action.rb:86:in 'block (2 levels) in Vagrant::BatchAction#run'
The important part here is
Call to virDomainCreate failed: internal error: process exited while connecting to monitor:
2025-09-22T10:07:55.081081Z qemu-system-x86_64: -blockdev {"driver":"file","filename":"/home/evgeni/.vagrant.d/boxes/boxen-VAGRANTSLASH-debian-13/2025.08.20.12/libvirt/OVMF_CODE.fd","node-name":"libvirt-pflash0-storage","auto-read-only":true,"discard":"unmap"}:
Could not open '/home/evgeni/.vagrant.d/boxes/boxen-VAGRANTSLASH-debian-13/2025.08.20.12/libvirt/OVMF_CODE.fd': Permission denied (Libvirt::Error)
Of course we checked that the file permissions on this file are correct (I'll save you the ls output), so what's next?
Yes, of course, SELinux!
A process in the svirt_t domain tries to access something in the user_home_t domain and is denied by the kernel.
So far, SELinux is both working as designed and preventing us from doing our work, nice.
For "normal" (non-UEFI) boxes, Vagrant uploads the image to libvirt, which stores it in ~/.local/share/libvirt/images/ and boots fine from there.
For UEFI boxen, one also needs loader and nvram files, which Vagrant keeps in ~/.vagrant.d/boxes/<box_name> and that's what explodes in our face here.
As ~/.local/share/libvirt/images/ works well, and is labeled svirt_home_t let's see what other folders use that label:
# semanagefcontext-l|grepsvirt_home_t
/home/[^/]+/\.cache/libvirt/qemu(/.*)? all files unconfined_u:object_r:svirt_home_t:s0/home/[^/]+/\.config/libvirt/qemu(/.*)? all files unconfined_u:object_r:svirt_home_t:s0/home/[^/]+/\.libvirt/qemu(/.*)? all files unconfined_u:object_r:svirt_home_t:s0/home/[^/]+/\.local/share/gnome-boxes/images(/.*)? all files unconfined_u:object_r:svirt_home_t:s0/home/[^/]+/\.local/share/libvirt/boot(/.*)? all files unconfined_u:object_r:svirt_home_t:s0/home/[^/]+/\.local/share/libvirt/images(/.*)? all files unconfined_u:object_r:svirt_home_t:s0
Okay, that all makes sense, and it's just missing the Vagrant-specific folders!
% restorecon-rv~/.vagrant.d/boxes/boxen-VAGRANTSLASH-debian-13
Relabeled /home/evgeni/.vagrant.d/boxes/boxen-VAGRANTSLASH-debian-13 from unconfined_u:object_r:user_home_t:s0 to unconfined_u:object_r:svirt_home_t:s0Relabeled /home/evgeni/.vagrant.d/boxes/boxen-VAGRANTSLASH-debian-13/metadata_url from unconfined_u:object_r:user_home_t:s0 to unconfined_u:object_r:svirt_home_t:s0Relabeled /home/evgeni/.vagrant.d/boxes/boxen-VAGRANTSLASH-debian-13/2025.08.20.12 from unconfined_u:object_r:user_home_t:s0 to unconfined_u:object_r:svirt_home_t:s0Relabeled /home/evgeni/.vagrant.d/boxes/boxen-VAGRANTSLASH-debian-13/2025.08.20.12/libvirt from unconfined_u:object_r:user_home_t:s0 to unconfined_u:object_r:svirt_home_t:s0Relabeled /home/evgeni/.vagrant.d/boxes/boxen-VAGRANTSLASH-debian-13/2025.08.20.12/libvirt/box_0.img from unconfined_u:object_r:user_home_t:s0 to unconfined_u:object_r:svirt_home_t:s0Relabeled /home/evgeni/.vagrant.d/boxes/boxen-VAGRANTSLASH-debian-13/2025.08.20.12/libvirt/metadata.json from unconfined_u:object_r:user_home_t:s0 to unconfined_u:object_r:svirt_home_t:s0Relabeled /home/evgeni/.vagrant.d/boxes/boxen-VAGRANTSLASH-debian-13/2025.08.20.12/libvirt/Vagrantfile from unconfined_u:object_r:user_home_t:s0 to unconfined_u:object_r:svirt_home_t:s0Relabeled /home/evgeni/.vagrant.d/boxes/boxen-VAGRANTSLASH-debian-13/2025.08.20.12/libvirt/OVMF_CODE.fd from unconfined_u:object_r:user_home_t:s0 to unconfined_u:object_r:svirt_home_t:s0Relabeled /home/evgeni/.vagrant.d/boxes/boxen-VAGRANTSLASH-debian-13/2025.08.20.12/libvirt/OVMF_VARS.fd from unconfined_u:object_r:user_home_t:s0 to unconfined_u:object_r:svirt_home_t:s0Relabeled /home/evgeni/.vagrant.d/boxes/boxen-VAGRANTSLASH-debian-13/2025.08.20.12/libvirt/box_update_check from unconfined_u:object_r:user_home_t:s0 to unconfined_u:object_r:svirt_home_t:s0Relabeled /home/evgeni/.vagrant.d/boxes/boxen-VAGRANTSLASH-debian-13/2025.08.20.12/libvirt/efivars.fd from unconfined_u:object_r:user_home_t:s0 to unconfined_u:object_r:svirt_home_t:s0
And it works!
% vagrantup
Bringing machine 'default' up with 'libvirt' provider...==> default: Checking if box 'boxen/debian-13' version '2025.08.20.12' is up to date...==> default: Creating image (snapshot of base box volume).==> default: Creating domain with the following settings...==> default: -- Name: tmp.JV8X48n30U_default==> default: -- Description: Source: /tmp/tmp.JV8X48n30U/Vagrantfile==> default: -- Domain type: kvm==> default: -- Cpus: 1==> default: -- Feature: acpi==> default: -- Feature: apic==> default: -- Feature: pae==> default: -- Clock offset: utc==> default: -- Memory: 2048M==> default: -- Loader: /home/evgeni/.vagrant.d/boxes/boxen-VAGRANTSLASH-debian-13/2025.08.20.12/libvirt/OVMF_CODE.fd==> default: -- Nvram: /home/evgeni/.vagrant.d/boxes/boxen-VAGRANTSLASH-debian-13/2025.08.20.12/libvirt/efivars.fd==> default: -- Base box: boxen/debian-13==> default: -- Storage pool: default==> default: -- Image(vda): /home/evgeni/.local/share/libvirt/images/tmp.JV8X48n30U_default.img, virtio, 20G==> default: -- Disk driver opts: cache='default'==> default: -- Graphics Type: vnc==> default: -- Video Type: cirrus==> default: -- Video VRAM: 16384==> default: -- Video 3D accel: false==> default: -- Keymap: en-us==> default: -- TPM Backend: passthrough==> default: -- INPUT: type=mouse, bus=ps2==> default: -- CHANNEL: type=unix, mode===> default: -- CHANNEL: target_type=virtio, target_name=org.qemu.guest_agent.0==> default: Creating shared folders metadata...==> default: Starting domain.==> default: Domain launching with graphics connection settings...==> default: -- Graphics Port: 5900==> default: -- Graphics IP: 127.0.0.1==> default: -- Graphics Password: Not defined==> default: -- Graphics Websocket: 5700==> default: Waiting for domain to get an IP address...==> default: Waiting for machine to boot. This may take a few minutes... default: SSH address: 192.168.124.157:22 default: SSH username: vagrant default: SSH auth method: private key default: default: Vagrant insecure key detected. Vagrant will automatically replace default: this with a newly generated keypair for better security. default: default: Inserting generated public key within guest... default: Removing insecure key from the guest if it's present... default: Key inserted! Disconnecting and reconnecting using new SSH key...==> default: Machine booted and ready!
I first blogged about the Colmi P80 just over a month ago [2]. Now I have a couple of relatives using it happily on Android with the proprietary app. I couldn’t use it myself because I require more control over which apps have their notifications go to the watch than the Colmi app offers. Also I’m trying to move away from non-free software.
Yesterday the f-droid repository informed me that there was a new version of Gadget Bridge and the changelog indicated support for the Colmi P80 so I connected the P80 and disconnected the PineTime.
The first problem I noticed is that the vibrator on the P80 when on it’s maximum setting is much weaker than that on the PineTime, so weak that I often didn’t notice it. Maybe if I wore it for a few weeks I would teach myself to notice it but it should just be able to work with me on this. If it could be set to have multiple bursts of vibrating then that would work.
The next problem is that the P80 by default does not turn the screen on when there’s a notification and there seems to be no way to configure it to do so. I configured it to turn on when I raise my arm which can mostly work but that still relies on me noticing the vibration. Vibration and the screen light turning on would be harder to miss than vibration on it’s own.
I don’t recall seeing any review of smart watches ever that stated whether the screen would turn on when there’s a notification or whether the vibration was easy to notice.
One problem with both the PineTime (running InfiniTime) and the P80 is that when the screen is turned on (through gesture, pushing the button, or a notification in the case of the Pinetime) it is active for swiping to change the settings. I would like to have some other action required before settings can be changed so that if the screen turns on when I’m asleep my watch won’t brush against something and change it’s settings (which has happened).
It’s neat how Gadget Bridge supports talking to multiple smart watches at the same time. One useful feature for that would be to have different notification settings for each watch. I can imagine someone changing between a watch for jogging and a watch for work and wanting different settings.
Colmi P80 Problems
No authentication for Bluetooth connections.
Runs non-free software so no chance to fix things.
Battery life worse than PineTime (but not really bad).
Vibration weak.
Screen doesn’t turn on when notification is sent.
Conclusion
I’m using the PineTime as my daily driver again. While it works well enough for some people (even with the Colmi proprietary app) it doesn’t do what I want. It is however a good test device for FOSS work on the phone side, it has a decent feature set and is cheap.
Apart from lack of authentication and running non-free software the problems are mostly a matter of taste. Some people might think it’s great the way it works.
Akvorado 2.0 was released today! Akvorado collects network flows with
IPFIX and sFlow. It enriches flows and stores them in a ClickHouse database.
Users can browse the data through a web console. This release introduces an
important architectural change and other smaller improvements. Let’s dive in! 🤿
The major change in Akvorado 2.0 is splitting the inlet service into two
parts: the inlet and the outlet. Previously, the inlet handled all flow
processing: receiving, decoding, and enrichment. Flows were then sent to Kafka
for storage in ClickHouse:
Akvorado flow processing before the introduction of the outlet service
Network flows reach the inlet service using UDP, an unreliable protocol. The
inlet must process them fast enough to avoid losing packets. To handle a
high number of flows, the inlet spawns several sets of workers to receive flows,
fetch metadata, and assemble enriched flows for Kafka. Many configuration
options existed for scaling, which increased complexity for users. The code
needed to avoid blocking at any cost, making the processing pipeline complex
and sometimes unreliable, particularly the BMP receiver.1 Adding new
features became difficult without making the problem worse.2
In Akvorado 2.0, the inlet receives flows and pushes them to Kafka without
decoding them. The new outlet service handles the remaining tasks:
Akvorado flow processing after the introduction of the outlet service
This change goes beyond a simple split:3 the outlet now reads flows from
Kafka and pushes them to ClickHouse, two tasks that Akvorado did not handle
before. Flows are heavily batched to increase efficiency and reduce the load
on ClickHouse using ch-go, a low-level Go client for ClickHouse. When
batches are too small, asynchronous inserts are used (e20645). The number of
outlet workers scales dynamically (e5a625) based on the target batch
size and latency (50,000 flows and 5 seconds by default).
This new architecture also allows us to simplify and optimize the code. The
outlet fetches metadata synchronously (e20645). The BMP component becomes
simpler by removing cooperative multitasking (3b9486). Reusing the same
RawFlow object to decode protobuf-encoded flows from Kafka reduces pressure on
the garbage collector (8b580f).
The effect on Akvorado’s overall performance was somewhat uncertain, but a
user reported35% lower CPU usage after migrating from the previous
version, plus resolution of the long-standing BMP component issue. 🥳
Other changes
This new version includes many miscellaneous changes, such as completion for
source and destination ports (f92d2e), and automatic restart of the
orchestrator service (0f72ff) when configuration changes to avoid a common
pitfall for newcomers.
Akvorado exposes metrics to provide visibility into the processing pipeline and
help troubleshoot issues. These are available through Prometheus HTTP metrics
endpoints, such as /api/v0/inlet/metrics. With the introduction
of the outlet, many metrics moved. Some were also renamed (4c0b15) to match
Prometheus best practices. Kafka consumer lag was added as a new metric
(e3a778).
If you do not have your own observability stack, the Docker Compose setup
shipped with Akvorado provides one. You can enable it by activating the profiles
introduced for this purpose (529a8f).
The prometheus profile ships Prometheus to store metrics and Alloy
to collect them (2b3c46, f81299, and 8eb7cd). Redis and Kafka
metrics are collected through the exporter bundled with Alloy (560113).
Other metrics are exposed using Prometheus metrics endpoints and are
automatically fetched by Alloy with the help of some Docker labels, similar to
what is done to configure Traefik. cAdvisor was also added (83d855) to
provide some container-related metrics.
The loki profile ships Loki to store logs (45c684). While Alloy
can collect and ship logs to Loki, its parsing abilities are limited: I could
not find a way to preserve all metadata associated with structured logs produced
by many applications, including Akvorado. Vector replaces Alloy (95e201)
and features a domain-specific language, VRL, to transform logs. Annoyingly,
Vector currently cannot retrieve Docker logs from before it was
started.
Finally, the grafana profile ships Grafana, but the shipped dashboards are
broken. This is planned for a future version.
Documentation
The Docker Compose setup provided by Akvorado makes it easy to get the web
interface up and running quickly. However, Akvorado requires a few mandatory
steps to be functional. It ships with comprehensive documentation, including
a chapter about troubleshooting problems. I hoped this documentation would
reduce the support burden. It is difficult to know if it works. Happy users
rarely report their success, while some users open discussions asking for help
without reading much of the documentation.
In this release, the documentation was significantly improved.
The documentation was updated (fc1028) to match Akvorado’s new architecture.
The troubleshooting section was rewritten (17a272). Instructions on how to
improve ClickHouse performance when upgrading from versions earlier than 1.10.0
was added (5f1e9a). An LLM proofread the entire content (06e3f3).
Developer-focused documentation was also improved (548bbb, e41bae, and
871fc5).
From a usability perspective, table of content sections are now collapsable
(c142e5). Admonitions help draw user attention to important points
(8ac894).
Example of use of admonitions in Akvorado's documentation
Continuous integration
This release includes efforts to speed up continuous integration on GitHub.
Coverage and race tests run in parallel (6af216 and fa9e48). The Docker
image builds during the tests but gets tagged only after they succeed
(8b0dce).
GitHub workflow to test and build Akvorado
End-to-end tests (883e19) ensure the shipped Docker Compose setup works as
expected. Hurl runs tests on various HTTP endpoints, particularly to verify
metrics (42679b and 169fa9). For example:
## Test inlet has received NetFlow flowsGEThttp://127.0.0.1:8080/prometheus/api/v1/query[Query]query:sum(akvorado_inlet_flow_input_udp_packets_total{job="akvorado-inlet",listener=":2055"})
HTTP200[Captures]inlet_receivedflows:jsonpath "$.data.result[0].value[1]" toInt
[Asserts]variable"inlet_receivedflows">10## Test inlet has sent them to KafkaGEThttp://127.0.0.1:8080/prometheus/api/v1/query[Query]query:sum(akvorado_inlet_kafka_sent_messages_total{job="akvorado-inlet"})
HTTP200[Captures]inlet_sentflows:jsonpath "$.data.result[0].value[1]" toInt
[Asserts]variable"inlet_sentflows">={{inlet_receivedflows}}
Docker
Akvorado ships with a comprehensive Docker Compose setup to help users get
started quickly. It ensures a consistent deployment, eliminating many
configuration-related issues. It also serves as a living documentation of the
complete architecture.
This release brings some small enhancements around Docker:
Previously, many Docker images were pulled from the Bitnami Containers
library. However, VMWare acquired Bitnami in 2019 and Broadcom acquired
VMWare in 2023. As a result, Bitnami images were deprecated in less than a
month. This was not really a surprise4. Previous versions of Akvorado
had already started moving away from them. In this release, the Apache project’s
Kafka image replaces the Bitnami one (1eb382). Thanks to the switch to KRaft
mode, Zookeeper is no longer needed (0a2ea1, 8a49ca, and f65d20).
Akvorado’s Docker images were previously compiled with Nix. However, building
AArch64 images on x86-64 is slow because it relies on QEMU userland emulation.
The updated Dockerfile uses multi-stage and multi-platform builds: one
stage builds the JavaScript part on the host platform, one stage builds the Go
part cross-compiled on the host platform, and the final stage assembles the
image on top of a slim distroless image (268e95 and d526ca).
# This is a simplified versionFROM--platform=$BUILDPLATFORMnode:20-alpineASbuild-js
RUNapkadd--no-cachemake
WORKDIR/buildCOPYconsole/frontendconsole/frontend
COPYMakefile.
RUNmakeconsole/data/frontend
FROM--platform=$BUILDPLATFORMgolang:alpineASbuild-go
RUNapkadd--no-cachemakecurlzip
WORKDIR/buildCOPY..
COPY--from=build-js/build/console/data/frontendconsole/data/frontend
RUNgomoddownload
RUNmakeall-indep
ARGTARGETOSTARGETARCHTARGETVARIANTVERSION
RUNmake
FROMgcr.io/distroless/static:latestCOPY--from=build-go/build/bin/akvorado/usr/local/bin/akvorado
ENTRYPOINT["/usr/local/bin/akvorado"]
When building for multiple platforms with --platform
linux/amd64,linux/arm64,linux/arm/v7, the build steps until the highlighted
line execute only once for all platforms. This significantly speeds up the
build. 🚅
Akvorado now ships Docker images for these platforms: linux/amd64,
linux/amd64/v3, linux/arm64, and linux/arm/v7. When requesting
ghcr.io/akvorado/akvorado, Docker selects the best image for the current CPU.
On x86-64, there are two choices. If your CPU is recent enough, Docker
downloads linux/amd64/v3. This version contains additional optimizations and
should run faster than the linux/amd64 version. It would be interesting to
ship an image for linux/arm64/v8.2, but Docker does not support the same
mechanism for AArch64 yet (792808).
Go
This release includes many changes related to Go but not visible to the users.
Toolchain
In the past, Akvorado supported the two latest Go versions, preventing immediate
use of the latest enhancements. The goal was to allow users of stable
distributions to use Go versions shipped with their distribution to compile
Akvorado. However, this became frustrating when interesting features, like go
tool, were released. Akvorado 2.0 requires Go 1.25 (77306d) but can be
compiled with older toolchains by automatically downloading a newer one
(94fb1c).5 Users can still override GOTOOLCHAIN to revert this
decision. The recommended toolchain updates weekly through CI to ensure we get
the latest minor release (5b11ec). This change also simplifies updates to
newer versions: only go.mod needs updating.
Thanks to this change, Akvorado now uses wg.Go() (77306d) and I have
started converting some unit tests to the new test/synctest package
(bd787e, 7016d8, and 159085).
Testing
When testing equality, I use a helper function Diff() to display the
differences when it fails:
This function uses kylelemons/godebug. This package is
no longer maintained and has some shortcomings: for example, by default, it does
not compare struct private fields, which may cause unexpectedly successful
tests. I replaced it with google/go-cmp, which is stricter
and has better output (e2f1df).
Another package for Kafka
Another change is the switch from Sarama to franz-go to interact with
Kafka (756e4a and 2d26c5). The main motivation for this change is to
get a better concurrency model. Sarama heavily relies on channels and it is
difficult to understand the lifecycle of an object handed to this package.
franz-go uses a more modern approach with callbacks6 that is both more
performant and easier to understand. It also ships with a package to spawn fake
Kafka broker clusters, which is more convenient than the mocking functions
provided by Sarama.
Improved routing table for BMP
To store its routing table, the BMP component used
kentik/patricia, an implementation of a patricia tree
focused on reducing garbage collection pressure.
gaissmai/bart is a more recent alternative using an
adaptation of [Donald Knuth’s ART algorithm][] that promises better
performance and delivers it: 90% faster lookups and 27% faster
insertions (92ee2e and fdb65c).
Unlike kentik/patricia, gaissmai/bart does not help efficiently store values
attached to each prefix. I adapted the same approach as kentik/patricia to
store route lists for each prefix: store a 32-bit index for each prefix, and use
it to build a 64-bit index for looking up routes in a map. This leverages Go’s
efficient map structure.
gaissmai/bart also supports a lockless routing table version, but this is not
simple because we would need to extend this to the map storing the routes and to
the interning mechanism. I also attempted to use Go’s new unique package to
replace the intern package included in Akvorado, but performance was
worse.7
Miscellaneous
Previous versions of Akvorado were using a custom Protobuf encoder for
performance and flexibility. With the introduction of the outlet service,
Akvorado only needs a simple static schema, so this code was removed. However,
it is possible to enhance performance with
planetscale/vtprotobuf (e49a74, and 8b580f).
Moreover, the dependency on protoc, a C++ program, was somewhat annoying.
Therefore, Akvorado now uses buf, written in Go, to convert a Protobuf
schema into Go code (f4c879).
Another small optimization to reduce the size of the Akvorado binary by
10 MB was to compress the static assets embedded in Akvorado in a ZIP file. It
includes the ASN database, as well as the SVG images for the documentation. A
small layer of code makes this change transparent (b1d638 and e69b91).
npm-run-all was removed (3424e8, 132 dependencies). patch-package was
removed (625805 and e85ff0, 69 dependencies) by moving missing
TypeScript definitions to env.d.ts. eslint was replaced with oxlint, a
linter written in Rust (97fd8c, 125 dependencies, including the plugins).
I switched from npm to Pnpm, an alternative package manager (fce383).
Pnpm does not run install scripts by default8 and prevents installing
packages that are too recent. It is also significantly faster.9 Node.js
does not ship Pnpm but it ships Corepack, which allows us to use Pnpm
without installing it. Pnpm can also list licenses used by each dependency,
removing the need for license-compliance (a35ca8, 42 dependencies).
For additional speed improvements, beyond switching to Pnpm and Oxlint, Vite
was replaced with its faster Rolldown version (463827).
After these changes, Akvorado “only” pulls 225 dependencies. 😱
Next steps
I would like to land three features in the next version of Akvorado:
Add Grafana dashboards to complete the observability stack. See issue
#1906 for details.
Integrate OVH’s Grafana plugin by providing a stable API for such
integrations. Akvorado’s web console would still be useful for browsing
results, but if you want to build and share dashboards, you should switch to
Grafana. See issue #1895.
Move some work currently done in ClickHouse (custom dictionaries, GeoIP and IP
enrichment) back into the outlet service. This should give more flexibility
for adding features like the one requested in issue #1030. See issue #2006.
I started working on splitting the inlet into two parts more than one year ago.
I found more motivation in recent months, partly thanks to Claude Code,
which I used as a rubber duck. Almost none of the produced code was
kept:10 it is like an intern who does not learn. 🦆
Many attempts were made to make the BMP component both performant and
not blocking. See for example PR #254, PR #255, and PR #278.
Despite these efforts, this component remained problematic for most users.
See issue #1461 as an example. ↩︎
Some features have been pushed to ClickHouse to avoid the
processing cost in the inlet. See for example PR #1059. ↩︎
Broadcom is known for its user-hostile moves. Look at what happened
with VMWare. ↩︎
As a Debian developer, I dislike these mechanisms that circumvent
the distribution package manager. The final straw came when Go 1.25 spent one month in the Debian NEW queue, an arbitrary mechanism I
don’t like at all. ↩︎
In the early years of Go, channels were heavily promoted. Sarama
was designed during this period. A few years later, a more nuanced approach
emerged. See notably “Go channels are bad and you should feel bad.” ↩︎
This should be investigated further, but my theory is that the
intern package uses 32-bit integers, while unique uses 64-bit pointers.
See commit 74e5ac. ↩︎
This is also possible with npm. See commit dab2f7. ↩︎
An even faster alternative is Bun, but it is less available. ↩︎
The exceptions are part of the code for the admonition blocks,
the code for collapsing the table of content, and part of the documentation. ↩︎
When this book was presented as available for review, I jumped on it. After
all, who doesn’t love reading a nice bit of computing history, as told by a
well-known author (affectionaly known as “Uncle Bob”), one who has been
immersed in computing since forever? What’s not to like there?
Reading on, the book does not disappoint. Much to the contrary, it digs
into details absent in most computer history books that, being an operating
systems and computer architecture geek, I absolutely enjoyed. But let me
first address the book’s organization.
The book is split into four parts. Part 1, “Setting the Stage,” is a short
introduction, answering the question “Who are we?” (“we” being the
programmers, of course). It describes the fascination many of us felt when
we realized that the computer was there to obey us, to do our bidding, and
we could absolutely control it.
Part 2 talks about “the giants” of the computing world, on whose shoulders
we stand. It digs in with a level of detail I have never seen before,
discussing their personal lives and technical contributions (as well as the
hoops they had to jump through to get their work done). Nine chapters cover
these giants, ranging chronologically from Charles Babbage and Ada Lovelace
to Ken Thompson, Dennis Richie, and Brian Kernighan (understandably, giants
who worked together are grouped in the same chapter). This is the part with
the most historically overlooked technical details. For example, what was
the word size in the first computers, before even the concept of a “byte”
had been brought into regular use? What was the register structure of early
central processing units (CPUs), and why did it lead to requiring
self-modifying code to be able to execute loops?
Then, just as Unix and C get invented, Part 3 skips to computer history as
seen through the eyes of Uncle Bob. I must admit, while the change of
rhythm initially startled me, it ends up working quite well. The focus is
no longer on the giants of the field, but on one particular person (who
casts a very long shadow). The narrative follows the author’s life: a boy
with access to electronics due to his father’s line of work; a computing
industry leader, in the early 2000s, with extreme programming; one of the
first producers of training materials in video format–a role that today
might be recognized as an influencer. This first-person narrative reaches
year 2023.
But the book is not just a historical overview of the computing world, of
course. Uncle Bob includes a final section with his thoughts on the future
of computing. As this is a book for programmers, it is fitting to start
with the changes in programming languages that we should expect to see and
where such changes are likely to take place. The unavoidable topic of
artificial intelligence is presented next: What is it and what does it
spell for computing, and in particular for programming? Interesting (and
sometimes surprising) questions follow: What does the future of hardware
development look like? What is prone to be the evolution of the World Wide
Web? What is the future of programming–and programmers?
At just under 500 pages, the book is a volume to be taken seriously. But
space is very well used with this text. The material is easy to read, often
funny and always informative. If you enjoy computer history and
understanding the little details in the implementations, it might very well
be the book you want.
The solar fence and some other ground and pole mount solar panels, seen through leaves.
Solar fencing manufacturers have some good simple designs, but it's hard
to buy for a small installation. They are selling to utility scale solar
mostly. And those are installed by driving metal beams into the ground,
which requires heavy machinery.
Since I have experience with Ironridge rails for roof mount solar, I
decided to adapt that system for a vertical mount. Which is something it
was not designed for. I combined the Ironridge hardware with regular parts
from the hardware store.
The cost of mounting solar panels nowadays is often higher than the cost of
the panels. I hoped to match the cost, and I nearly did. The solar panels cost
$100 each, and the fence cost $110 per solar panel. This fence was
significantly cheaper than conventional ground mount arrays that I
considered as alternatives, and made a better use of a difficult hillside
location.
I used 7 foot long Ironridge XR-10 rails, which fit 2 solar panels per rail.
(Longer rails would need a center post anyway, and the 7 foot long rails
have cheaper shipping, since they do not need to be shipped freight.)
For the fence posts, I used regular 4x4" treated posts. 12 foot long, set
in 3 foot deep post holes, with 3x 50 lb bags of concrete per hole and 6
inches of gravel on the bottom.
detail of how the rails are mounted to the posts, and the panels to the rails
To connect the Ironridge rails to the fence posts, I used the Ironridge
LFT-03-M1 slotted L-foot bracket. Screwed into the post with a 5/8” x 3
inch hot-dipped galvanized lag screw. Since a treated post can react badly
with an aluminum bracket, there needs to be some flashing between the post
and bracket. I used Shurtape PW-100 tape for that. I see no sign of
corrosion after 1 year.
The rest of the Ironridge system is a T-bolt that connects the rail to the
L-foot (part BHW-SQ-02-A1), and Ironridge solar panel fasteners
(UFO-CL-01-A1 and UFO-STP-40MM-M1). Also XR-10 end caps and wire clips.
Since the Ironridge hardware is not designed to hold a solar panel at a 90
degree angle, I was concerned that the panels might slide downward over
time. To help prevent that, I added some additional support brackets under
the bottom of the panels. So far, that does not seem to have been a problem
though.
I installed Aptos 370 watt solar panels on the fence. They are bifacial,
and while the posts block the back partially, there is still bifacial
gain on cloudy days. I left enough space under the solar panels to be able
to run a push mower under them.
Me standing in front of the solar fence at end of construction
I put pairs of posts next to one-another, so each 7 foot segment of fence
had its own 2 posts. This is the least elegant part of this design, but
fitting 2 brackets next to one-another on a single post isn't feasible.
I bolted the pairs of posts together with some spacers. A side benefit of
doing it this way is that treated lumber can warp as it dries, and this
prevented much twisting of the posts.
Using separate posts for each segment also means that the fence can
traverse a hill easily. And it does not need to be perfectly straight. In
fact, my fence has a 30 degree bend in the middle. This means it has both
south facing and south-west facing panels, so can catch the light for
longer during the day.
After building the fence, I noticed there was a slight bit of sway at the
top, since 9 feet of wooden post is not entirely rigid. My worry was that a
gusty wind could rattle the solar panels. While I did not actually observe
that happening, I added some diagonal back bracing for peace of mind.
view of rear upper corner of solar fence, showing back bracing connection
Inspecting the fence today, I find no problems after the first year. I hope
it will last 30 years, with the lifespan of the treated lumber
being the likely determining factor.
As part of my larger (and still ongoing) ground mount solar install, the
solar fence has consistently provided great power. The vertical orientation
works well at latitude 36. It also turned out that the back of the fence was
useful to hang conduit and wiring and solar equipment, and so it turned into
the electrical backbone of my whole solar field. But that's another story..
Life can sometimes be tricky, and it's useful to know that there are
some simple things to take pleasure from. Amongst them for me are
lava lamps.
At some point in the late 90s, my brother and I somehow had 6
lavalamps between us. I'm not sure what happened to them (and
the gallery of photos I had of them has long disappeared from
my site.)
DebConf26 is already in the air in Argentina. Organizing DebConf26 give us the
opportunity to talk about Debian in our country again. This is not the first
time that Debian has come here, previously Argentina has hosted DebConf 8 in Mar
del Plata.
In August, Nattie Mayer-Hutchings and Stefano Rivera from DebConf Committee
visited the venue where the next DebConf will take place. They came to Argentina
in order to see what it is like to travel from Buenos Aires to Santa Fe (the
venue of the next DebConf). In addition, they were able to observe the layout
and size of the classrooms and halls, as well as the infrastructure available at
the venue, which will be useful for the Video Team.
But before going to Santa Fe, on the August 27th, we organized a meetup in
Buenos Aires at GCoop, where we hosted some talks:
¿Qué es Debian? - Pablo Gonzalez (sultanovich) / Emmanuel Arias
On August 28th, we had the opportunity to get to know the Venue. We walked around
the city and, obviously, sampled some of the beers from Santa Fe.
On August 29th we met with representatives of the University and local government
who were all very supportive. We are very grateful to them for opening
their doors to DebConf.
In the afternoon we met some of the local free software community at an event we
held in ATE Santa Fe. The event included several talks:
¿Qué es Debian? - Pablo (sultanovich) / Emmanuel Arias
Ciberrestauradores: Gestores de basura electrónica - Programa RAEES Acutis
Debian and DebConf (Stefano Rivera/Nattie Mayer-Hutchings)
Thanks to Debian Argentina, and all the people who will make DebConf26
possible.
Thanks to Nattie Mayer-Hutchings and Stefano Rivera for reviewing an earlier
version of this article.
tl;dr: https://osbpo.debian.net/deb-status is now real-time updated and much better than it used to, helping the OpenStack packaging team be a way more efficient.
How it used to be
For years, the Debian OpenStack team has relied on automated tools to track the status of OpenStack packages across Debian releases. Our goal has always been simple: transparency, efficiency, accuracy.
We used to use a tool called os-version-checker, written by Michal Arbet, which generated a static status page at https://osbpo.debian.net/deb-status. It was functional and served us well — but it had limitations:
It ran on a cron job, not on demand
It processed all OpenStack releases at once, making it slow
The rsync from Jenkins hosts to osbpo.debian.net was also cron-driven
No immediate feedback after a package build
This meant that when a developer pushed a new package to salsa (the Debian GitLab instance) in the team’s repository, the following would happen:
Jenkins would build the backport
Store it in a local repository
Wait up to 30 minutes (or more) for the cron job to run rsync + status update
Only then would the status page reflect the new version
For maintainers actively working on a new release, this delay was frustrating. You’d fix a bug, push, build — and still see your package marked as “missing” or “out of date” for minutes. You had no real-time feedback. This was also an annoyance for testing, because when fixing a bug, I often had to trigger the rsync manually in order to not wait for it, so I could do my tests. Now, osbpo is always up-to-date a few seconds after the build of the package.
The New Way: Event-Driven, Real-Time Updates
We’ve rebuilt the system from the ground up to be fast, responsive, and event-driven. Now, the workflow is:
Developer git push → triggers Jenkins
Jenkins builds the package → publishes to local repo
Jenkins immediately triggers a webhook on osbpo.debian.net
The webhook on osbpo does:
rsyncs the new package to the central Debian repo
Pulls the latest OpenStack releases from git and use its YAML data (instead of parsing the release HTML pages)
Regenerates the status page, comparing what upstream released and what’s in Debian
No more cron. No more waiting…
How it works
The central osbpo.debian.net server runs:
webhook — to receive secure, HMAC-verified triggers that it processes in an async way
Apache — to serve the status pages and the Debian OpenStack repositories
Custom scripts — to rsync packages, validate, and generate reports
Jenkins instances are configured to curl the webhook on successful build. The status page is generated by openstack-debian-release-manager, a new tool I’ve packaged and deployed. The dashboard uses AJAX to load content dynamically (like when browsing from one release to another), with sorting, metadata, and real-time auto-refresh every 10 seconds.
openstack-debian-release-manager is easy to deploy and configure, and will do most (if not all) of the needed configuration. Uploading it to Debian is probably not needed, and a bit over-kill, so I believe I’ll just keep it in Salsa for the moment, unless there’s a way to make it more generic so it can help someone else (another team?) in Debian.
Room for improvement
There’s still things I want to add. Namely:
Add status for Debian stable (ie: without the osbpo.debian.net add-on repository), which we used to have with os-version-checker.
Add a per-release config file option to be able to mask not packaged project on a per OpenStack release granularity
Special thanks to Michal Arbet for the original os-version-checker that served me for years, helping me to never forget a missing OpenStack package release.
Many people that were once enthusiast Twitter users have dropped as a
direct or indirect effect of its ownership change and the following policy
changes. Given Twitter X is getting each time more irrelevant, it is
less interesting and enciting for more and more people… But also, its
current core users (mostly, hate-apologists of the right-wing mindset that
finds conspiration theories everywhere) are becoming more commonplace, and
by sheer probability (if not for algorithmic bias), every time it becomes
more likely a given piece of content will be linked to what their authors
would classify as crap.
So, there has been in effect an X exodus. This has been reported in media
outlets as important as Reuters, or The
Guardianresearch
institutes such as
Berkeley,
even media that no matter how hard you push cannot be identified as the
radical left Mr. Trump is so happy to blame for everything, such as
Forbes…
Today I read a short note in a magazine I very much enjoy, Communications
of the ACM, where SIGDOC (the ACM’s Special Interest
Group on Design of Communication) is officially closing their X
account. The reasoning is crystal clear. They have the mission to create
and study User Experience (UX) implementations and report on it, «focused
on making communication clearer and more human centered». That is no
longer, for many reasons, a goal that can be furthered by the means of an X
account.
(BTW, and… How many people are actually angry that Mr. Musk took the X11
old logo and made it his? I am sure it is now protected under too many
layers of legalese, even though I am aware of it since at least 30 years
ago…)
Here, in classic Goerzen deep dive fashion, is more information than you knew you wanted about a topic you’ve probably never thought of. I found it pretty interesting, because it took me down a rabbit hole of subsystems I’ve never worked with much and a mishmash of 1980s and 2020s tech.
I had previously tried and failed to get an actual 80x25 Linux console, but I’ve since figured it out!
This post is about the Linux text console – not X or Wayland. We’re going to get the console right without using those systems. These instructions are for Debian trixie, but should be broadly applicable elsewhere also. The end result can look like this:
(That’s a Wifi Retromodem that I got at VCFMW last year in the Hayes modem case)
What’s a pixel?
How would you define a “pixel” these days? Probably something like “a uniquely-addressable square dot in a two-dimensional grid”.
In the world of VGA and CRTs, that was just a logical abstraction. We got an API centered around that because it was convenient. But, down the VGA cable and on the device, that’s not what a pixel was.
A pixel, back then, was a time interval. On a multisync monitor, which were common except in the very early days of VGA, the timings could be adjusted which produced logical pixels of different sizes. Those screens often had a maximum resolution but not necessarily a “native resolution” in the sense that an LCD panel does. Different timings produced different-sized pixels with equal clarity (or, on cheaper monitors, equal fuzziness).
A side effect of this was that pixels need not be square. And, in fact, in the standard DOS VGA 80x25 text mode, they weren’t.
You might be seeing why DVI, DisplayPort, and HDMI replaced VGA for LCD monitors: with a VGA cable, you did a pixel-to-analog-timings conversion, then the display did a timings-to-pixels conversion, and this process could be a bit lossy. (Hence why you sometimes needed to fill the screen with an image and push the “center” button on those older LCD screens)
(Note to the pedantically-inclined: yes I am aware that I have simplified several things here; for instance, a color LCD pixel is made up of approximately 3 sub-dots of varying colors, and that things like color eInk displays have two pixel grids with different sizes of pixels layered atop each other, and printers are another confusing thing altogether, and and and…. MOST PEOPLE THINK OF A PIXEL AS A DOT THESE DAYS, OK?)
What was DOS text mode?
We think of this as the “standard” display: 80 columns wide and 25 rows tall. 80x25. By the time Linux came along, the standard Linux console was VGA text mode – something like the 4th incarnation of text modes on PCs (after CGA, MDA, and EGA). VGA also supported certain other sizes of characters giving certain other text dimensions, but if I cover all of those, this will explode into a ridiculously more massive page than it already is.
So to display text on an 80x25 DOS VGA system, ultimately characters and attributes were written into the text buffer in memory. The VGA system then rendered it to the display as a 720x400 image (at 70Hz) with non-square pixels such that the result was approximately a 4:3 aspect ratio.
The font used for this rendering was a bitmapped one using 8x16 cells. You might do some math here and point out that 8 * 80 is only 640, and you’d be correct. The fonts were 8x16 but the rendered cells were 9x16. The extra pixel was normally used for spacing between characters. However, in line graphics mode, characters 0xC0 through 0xDF repeated the 8th column in the position of the 9th, allowing the continuous line-drawing characters we’re used to from TUIs.
Problems rendering DOS fonts on modern systems
By now, you’re probably seeing some of the issues we have rendering DOS screens on more modern systems. These aren’t new at all; I remember some of these from back in the days when I ran OS/2, and I think also saw them on various terminals and consoles in OS/2 and Windows.
Some issues you’d encounter would be:
Incorrect aspect ratio caused by using the original font and rendering it using 1:1 square pixels (resulting in a squashed appearance)
Incorrect aspect ratio for ANOTHER reason, caused by failing to render column 9, resulting in text that is overall too narrow
Characters appearing to be touching each other when they shouldn’t (failing to render column 9; looking at you, dosbox)
Gaps between line drawing characters that should be continuous, caused by rendering column 9 as empty space in all cases
Character set issues
DOS was around long before Unicode was. In the DOS world, there were codepages that selected the glyphs for roughly the high half of the 256 possible characters. CP437 was the standard for the USA; others existed for other locations that needed different characters. On Unix, the USA pre-Unicode standard was Latin-1. Same concept, but with different character mappings.
Nowadays, just about everything is based on UTF-8. So, we need some way to map our CP437 glyphs into Unicode space. If we are displaying DOS-based content, we’ll also need a way to map CP437 characters to Unicode for display later, and we need these maps to match so that everything comes out right. Whew.
So, let’s get on with setting this up!
Selecting the proper video mode
As explained in my previous post, proper hardware support for DOS text mode is limited to x86 machines that do not use UEFI. Non-x86 machines, or x86 machines with UEFI, simply do not contain the necessary support for it. As these are now standard, most of the time, the text console you see on Linux is actually the kernel driving the video hardware in graphics mode, and doing the text rendering in software.
That’s all well and good, but it makes it quite difficult to actually get an 80x25 console.
First, we need to be running at 720x400. This is where I ran into difficulty last time. I realized that my laptop’s LCD didn’t advertise any video modes other than its own native resolution. However, almost all external monitors will, and 720x400@70 is a standard VGA mode from way back, so it should be well-supported.
You need to find the Linux device name for your device. You can look at the possible devices with ls -l /sys/class/drm. If you also have a GUI, xrandr may help too. But in any case, each directory under /sys/class/drm has a file named modes, and if you cat them all, you will eventually come across one with a bunch of modes defined. Drop the leading “card0” or whatever from the directory name, and that’s your device. (Verify that 720x400 is in modes while you’re at it.)
Now, you’re going to edit /etc/default/grub and add something like this to GRUB_CMDLINE_LINUX_DEFAULT:
video=DP-1:720x400@70
Of course, replace DP-1 with whatever your device is.
Now you can run update-grub and reboot. You should have a 720x400 display.
At first, I thought I had succeeded by using Linux’s built-in VGA font with that mode. But it looked too tall. After noticing that repeated 0s were touching, I got suspicious about the missing 9th column in the cells. stty -a showed that my screen was 90x25, which is exactly what it would show if I was using 8x16 instead of 9x16 cells. Sooo…. I need to prepare a 9x16 font.
Preparing a font
Here’s where it gets complicated.
I’ll give you the simple version and the hard mode.
The file we’re interested in is otb - Bm (linux bitmap)/Bm437_IBM_VGA_9x16.otb. Open it in fontforge by running fontforge BmPlus_IBM_VGA_9x16.otb. When it asks if you will load the bitmap fonts, hit select all, then yes. Go to File -> generate fonts. Save in a BDF, no need for outlines, and use “guess” for resolution.
Now you have a file such as Bm437_IBM_VGA_9x16-16.bdf. Excellent.
Now we need to generate a Unicode map file. We will make sure this matches the system’s by enumerating every character from 0x00 to 0xFF, converting it from CP437 to Unicode, and writing the appropriate map.
By convention, we normally store these files gzipped, so gzip CP437-VGA.psf.
You can test it on the console with setfont CP437-VGA.psf.gz.
Now copy this file into /usr/local/etc.
Activating the font
Now, edit /etc/default/console-setup. It should look like this:
# CONFIGURATION FILE FOR SETUPCON
# Consult the console-setup(5) manual page.
ACTIVE_CONSOLES="/dev/tty[1-6]"
CHARMAP="UTF-8"
CODESET="Lat15"
FONTFACE="VGA"
FONTSIZE="8x16"
FONT=/usr/local/etc/CP437-VGA.psf.gz
VIDEOMODE=
# The following is an example how to use a braille font
# FONT='lat9w-08.psf.gz brl-8x8.psf'
At this point, you should be able to reboot. You should have a proper 80x25 display! Log in and run stty -a to verify it is indeed 80x25.
Using and testing CP437
Part of the point of CP437 is to be able to access BBSs, ANSI art, and similar.
Now, remember, the Linux console is still in UTF-8 mode, so we have to translate CP437 to UTF-8, then let our font map translate it back to CP437. A weird trip, but it works.
Let’s test it using the Textfiles ANSI art collection. In the artworks section, I randomly grabbed a file near the top: borgman.ans. Download that, and display with:
clear; iconv -f CP437 -t UTF-8 < borgman.ans
You should see something similar to – but actually more accurate than – the textfiles PNG rendering of it, which you’ll note has an incorrect aspect ratio and some rendering issues. I spot-checked with a few others and they seemed to look good. belinda.ans in particular tries quite a few characters and should give you a good sense if it is working.
Use with interactive programs
That’s all well and good, but you’re probably going to want to actually use this with some interactive program that expects CP437. Maybe Minicom, Kermit, or even just telnet?
For this, you’ll want to apt-get install luit. luit maps CP437 (or any other encoding) to UTF-8 for display, and then of course the Linux console maps UTF-8 back to the CP437 font.
Here’s a way you can repeat the earlier experiment using luit to run the cat program:
clear; luit -encoding CP437 cat borgman.ans
You can run any command under luit. You can even run luit -encoding CP437 bash if you like. If you do this, it is probably a good idea to follow my instructions on generating locales on my post on serial terminals, and then within luit, set LANG=en_us.IBM437. But note especially that you can run programs like minicom and others for accessing BBSs under luit.
Final words
This gave you a nice DOS-type console. Although it doesn’t have glyphs for many codepoints, it does run in UTF-8 mode and therefore is compatible with modern software.
You can achieve greater compatibility with more UTF-8 codepoints with the DOS font, at the expense of accuracy of character rendering (especially for the double-line drawing characters) by using /usr/share/bdf2psf/standard.equivalents instead of /dev/null in the bdf2psf command.
Or you could go for another challenge, such as using the DEC vt-series fonts for coverage of ISO-8859-1. But just using fonts extracted from DEC ROM won’t work properly, because DEC terminals had even more strangeness going on than DOS fonts.
Six years ago, I was inspired to buy a DEC serial terminal. Since then, my collection has grown to include several DEC models, an IBM 3151, a Wyse WY-55, a Televideo 990, and a few others.
When you are running a terminal program on Linux or MacOS, what you are really running is a terminal emulator. In almost all cases, the terminal emulator is emulating one of the DEC terminals in the vt100 through vt520 line, which themselves use a command set based on an ANSI standard.
In short, you spend all day using a program designed to pretend to be the exact kind of physical machine I’m using for this experiment!
While I have used a terminal with the Pi, I’ve never before used it as a serial console all the way from early boot, and I have never installed Debian using the terminal to run the installer. A serial terminal gives you a login prompt. A serial console gives you access to kernel messages, the initrd environment, and sometimes even the bootloader.
This might be fun, I thought.
I selected one of my vt510 terminals for this. It is one of my newer ones, having been built in 1993. But it has a key feature: I can remap Ctrl to be at the caps lock position, something I do on every other system I use anyhow. I could have easily selected an older one from the 1980s.
Kernel configuration
To enable a serial console for Linux, you need to pass a parameter on the kernel command line. See the kernel documentaiton for more. I very frequently see instructions that are incomplete; they particularly omit flow control, which is most definitely needed for these real serial terminals.
I run my terminal at 57600 bps, so the parameter I need is console=ttyS0,57600n8r. The “r” means to use hardware flow control (ttyS0 corresponds to the first serial port on the system; use ttyS1 or something else as appropriate for your situation). While booting the Debian installer, according to Debian’s instructions, it may be useful to also add TERM=vt102 (the installer doesn’t support the vt510 terminal type directly). The TERM parameter should not be specified on a running system after instlalation.
Booting the Debian installer
When you start the Debian installer, to get it into serial mode, you have a couple of options:
You can use a traditional display and keyboard just long enough to input the kernel parameters described above
You can edit the bootloader configuration on the installer’s filesystem prior to booting from it
Option 1 is pretty easy. Option 2 is hard mode, but not that bad.
On x86, the Debian installer boots in at least two different ways: it uses GRUB if you’re booting under UEFI (which is most systems these days), or ISOLINUX if you are booting from the BIOS.
If using GRUB, the file to edit on the installer image is boot/grub/grub.cfg.
Near the top, add these lines:
serial --unit=0 --speed=57600 --word=8 --parity=no --stop=1
terminal_input console serial
terminal_output console serial
Unit 0 corresponds to ttyS0 as above.
GRUB’s serial command does not support flow control. If your terminal gets corrupted during the GRUB stage, you may need to configure it to a slower speed.
Then, find the “linux” line under the “Install” menuentry. Edit it to insert console=ttyS0,57600n8r TERM=vt102 right after the vga=788.
Save, unmount, and boot. You should see the GRUB screen displayed on your serial terminal. Select the Install option and the installer begins.
If you are using BIOS boot, I’m sure you can do something similar with the files in the isolinux directory, but haven’t researched it.
Now, you can install Debian like usual!
Configuring the System
I was pleasantly surprised to find that Debian’s installer took care of many, but not all, of the things I want to do in order to make the system work nicely with a serial terminal. You can perform these steps from a chroot under the installer environment before a reboot, or later in the running system.
First, while Debian does set up a getty (the program that displays the login prompt) on the serial console by default, it doesn’t enable hardware flow control. So let’s do that.
Configuring the System: agetty with systemd
Run systemctl edit serial-getty@ttyS0.service. This opens an editor that lets you customize the systemd configuration for a given service without having to edit the file directly. All you really need to do is modify the agetty command, so we just override it. At the top, in the designated area, write:
The empty ExecStart= line is necessary to tell systemd to remove the existing ExecStart command (otherwise, it will logically contain two ExecStart lines, which is an error).
These arguments say:
–wait-cr means to wait for the user to press Return at the terminal before attempting to display the login prompt
-8 tells it to assume 8-bit mode on the serial line
-h enables hardware flow control
-L=always enables local line mode, disabling monitoring of modem control lines
%I substitutes the name of the port from systemd
57600 gives the desired speed, and vt510 gives the desired setting for the TERM environment variable
The systemd documentation refers to this page about serial consoles, which gives more background. However, I think it is better to use the systemctl edit method described here, rather than just copying the config file, since this lets things like new configurations with new Debian versions take effect.
Configuring the System: Kernel and GRUB
Your next stop is the /etc/default/grub file. Debian’s installer automatically makes some changes here. There are three lines you want to change. First, near the top, edit GRUB_CMDLINE_LINUX_DEFAULT and add console=tty0 console=ttyS0,57600n8r. By specifying console twice, you allow output to go both to the standard display and to the serial console. By specifying the serial console last, you make it be the preferred one for things like entering the root filesystem password.
Next, towards the bottom, make sure these two lines look like this:
Finally, near the top, you may want to raise the GRUB_TIMEOUT to somewhere around 10 to 20 seconds since things may be a bit slower than you’re used to.
Save the file and run update-grub.
Now, GRUB will display on both your standard display and the serial console. You can edit the boot command from either. If you have a VGA or HDMI monitor attached, for instance, and need to not use the serial console, you can just edit the Linux command line in GRUB and remove the reference to ttyS0 for one boot. Easy!
That’s it. You now have a system that is fully operational from a serial terminal.
My original article from 2019 has some additional hints, including on how to convert from UTF-8 for these terminals.
Update 2025-09-17: It is also useful to set up proper locales. To do this, first edit /etc/locale.gen. Make sure to add, or uncomment:
Then run locale-gen. Normally, your LANG will be set to en_us.UTF-8, which will select the appropriate encoding. Plain en_US will select ISO-8859-1, which you need for the vt510. Then, add something like this to your ~/.bashrc:
if [ `tty` = "/dev/ttyS0" -o "$TERM" = "vt510" ]; then
stty -iutf8
# might add ixon ixoff
export LANG=en_US
export MANOPT="-E ascii"
stty rows 25
fi
if [ "$TERM" = "screen" -o "$TERM" = "vt100" ]; then
export LANG=en_US.utf8
fi
Finally, in my ~/.screenrc, I have this. It lets screen convert between UTF-8 and ISO-8859-1:
I sometimes use Vagrant to deploy my VM&aposs and recently when I tried to deploy one for Trixie, I could see one available. So I checked the official Debian images on Vagrant cloud at https://portal.cloud.hashicorp.com/vagrant/discover/debian and could not find an image for trixie on Vagrant cloud.
Also looked at other cloud image sources like Docker hub, and I could see an image their for Trixie. So I looked into how I can generate a Vagrant image locally for Debian to use.
this will install some dependency packages, will ask for sudo password if need to install something not already installed.
Let&aposs call make help
$ make help
To run this makefile, run:
make <DIST>-<CLOUD>-<ARCH>
WHERE <DIST> is bullseye, buster, stretch, sid or testing
And <CLOUD> is azure, ec2, gce, generic, genericcloud, nocloud, vagrant, vagrantcontrib
And <ARCH> is amd64, arm64, ppc64el
Set DESTDIR= to write images to given directory.
$ make trixie-vagrant-amd64
umask 022; \
./bin/debian-cloud-images build \
trixie vagrant amd64 \
--build-id vagrant-cloud-images-master \
--build-type official
usage: debian-cloud-images build
debian-cloud-images build: error: argument RELEASE: invalid value: trixie
make: *** [Makefile:22: trixie-vagrant-amd64] Error 2
As you can see, trixie is not even in the available options and it is not building as well. Before trying to look at updating the codebase, I looked at the pending MR&aposs on Salsa and found Michael Ablassmeier&aposs pending merge request at https://salsa.debian.org/cloud-team/debian-vagrant-images/-/merge_requests/18
So let me test that commit and see if I can build trixie locally from Michael&aposs MR
$ make help
To run this makefile, run:
make <DIST>-<CLOUD>-<ARCH>
WHERE <DIST> is bullseye, buster, stretch, sid or testing
And <CLOUD> is azure, ec2, gce, generic, genericcloud, nocloud, vagrant, vagrantcontrib
And <ARCH> is amd64, arm64, ppc64el
Set DESTDIR= to write images to given directory.
$ make trixie-vagrant-amd64
umask 022; \
./bin/debian-cloud-images build \
trixie vagrant amd64 \
--build-id vagrant-cloud-images-master \
--build-type official
2025-09-17 00:36:25,919 INFO Adding class DEBIAN
2025-09-17 00:36:25,919 INFO Adding class CLOUD
2025-09-17 00:36:25,919 INFO Adding class TRIXIE
2025-09-17 00:36:25,920 INFO Adding class VAGRANT
2025-09-17 00:36:25,920 INFO Adding class AMD64
2025-09-17 00:36:25,920 INFO Adding class LINUX_IMAGE_BASE
2025-09-17 00:36:25,920 INFO Adding class GRUB_PC
2025-09-17 00:36:25,920 INFO Adding class LAST
2025-09-17 00:36:25,921 INFO Running FAI: sudo env PYTHONPATH=/home/rajudev/dev/salsa/michael/debian-vagrant-images/src/debian_cloud_images/build/../.. CLOUD_BUILD_DATA=/home/rajudev/dev/salsa/michael/debian-vagrant-images/src/debian_cloud_images/data CLOUD_BUILD_INFO={"type": "official", "release": "trixie", "release_id": "13", "release_baseid": "13", "vendor": "vagrant", "arch": "amd64", "build_id": "vagrant-cloud-images-master", "version": "20250917-1"} CLOUD_BUILD_NAME=debian-trixie-vagrant-amd64-official-20250917-1 CLOUD_BUILD_OUTPUT_DIR=/home/rajudev/dev/salsa/michael/debian-vagrant-images CLOUD_RELEASE_ID=vagrant CLOUD_RELEASE_VERSION=20250917-1 fai-diskimage --verbose --hostname debian --class DEBIAN,CLOUD,TRIXIE,VAGRANT,AMD64,LINUX_IMAGE_BASE,GRUB_PC,LAST --size 100G --cspace /home/rajudev/dev/salsa/michael/debian-vagrant-images/src/debian_cloud_images/build/fai_config debian-trixie-vagrant-amd64-official-20250917-1.raw
..... continued
Although we can now build the images, we just don&apost see an option for it in the help text, not even for bookworm. Just the text in Makefile is outdated, but I can build and trixie Vagrant box now. Thanks to Michael for the fix.
If you use HaProxy to e.g. terminate TLS on the frontend and connect via
TLS to a backend, one has to take care of sending the SNI (server name
indication) extension in the TLS handshake sort of manually.
Even if you use host names to address the backend server, e.g.
server foobar foobar.example:2342 ssl verify required ca-file /etc/haproxy/ca/foo.crt
HaProxy will
try to establish the connection without SNI. You manually have to enforce
SNI here, e.g.
server foobar foobar.example:2342 ssl verify required ca-file /etc/haproxy/ca/foo.crt sni str(foobar.example)
The surprising thing here was that it requires an expression, so you can not just write sni foobar.example, you've to wrap it in an expression. The simplest one is making sure it's a string.
Update: Might be noteworthy that you've to configure SNI for the health check separately, and
in that case it's a string not an expression. E.g.
If someone hands you an IP:Port of a Google Cloud load balancer, and tells you to
connect there with TLS, but all you receive in return is an F (and a few other
bytes with none printable characters) on running openssl s_client -connect ...,
you might be missing SNI (server name indication). Sadly the other side
was not transparent enough to explain in detail which exact type of Google Cloud
load balancer they used, but the conversation got more detailed and up to a working
TLS connection when the missing -servername foobar.host.name was added. I could not
find any sort of official documentation on the responses of the GFE (the frontend part)
when TLS parameters do not match the expectations. Also you won't have anything in the
logs, because logging at Google Cloud is a backend function, and as long as your requests
do not reach the backend, there are no logs. That makes it rather unpleasant to debug such cases,
when one end says "I do not see anything in the logs", and the other one says "you reject
my connection and just reply F".
We announced tag2upload’s open beta in mid-July. That was in the middle of the the freeze for trixie, so usage was fairly light until the forky floodgates opened.
Since then the service has successfully performed 637 uploads, of which 420 were in the last 32 days. That’s an average of about 13 per day. For comparison, during the first half of September up to today there have been 2475 uploads to unstable. That’s about 176/day.
So, tag2upload is already handling around 7.5% of uploads. This is very gratifying for a service which is advertised as still being in beta!
Sean and I are very pleased both with the uptake, and with the way the system has been performing.
During this open beta period we have been hard at work. We have made many improvements to the user experience.
Current git-debpush in forky, or trixie-backports, is much better at detecting various problems ahead of time.
When uploads do fail on the service the emailed error reports are now more informative. For example, anomalies involving orig tarballs, which by definition can’t be detected locally (since one point of tag2upload is not to have tarballs locally) now generally result in failure reports containing a diffstat, and instructions for a local repro.
The biggest of these is that the service should be able to retry when Salsa fails. Sadly, Salsa isn’t wholly reliable, and right now if it breaks when the service is trying to handle your tag, your upload can fail.
We think most of these failures could be avoided. Implementing retries is a fairly substantial task, but doesn’t pose any fundamental difficulties. We’re working on this right now.
We want to support pristine-tar, so that pristine-tar users can do a new upstream release. Andrea Pappacoda is working on that with us. See #1106071. (Note that we would generally recommend against use of pristine-tar within Debian. But we want to support it.)
We have been having conversations with Debusine folks about what integration between tag2upload and Debusine would look like. We’re making some progress there, but a lot is still up in the air.
We are considering how best to provide tag2upload pre-checks as part of Salsa CI. There are several problems detected by the tag2upload service that could be detected by Salsa CI too, but which can’t be detected by git-debpush.
We’ve been monitoring the service and until very recently we have investigated every service-side failure, to understand the root causes. This has given us insight into the kinds of things our users want, and the kinds of packaging and git practices that are common. We’ve been able to improve the system’s handling of various anomalies and also improved the documentation.
Right now our failure rate is still rather high, at around 7%. Partly this is because people are trying out the system on packages that haven’t ever seen git tooling with such a level of rigour.
There are two classes of problem that are responsible for the vast majority of the failures that we’re still seeing:
tag2upload, like git (and like dgit), hates it when you reuse a version number, or try to pretend that a (perhaps busted) release never happened.
git tags aren’t namespaced, and tend to spread about promiscuously. So replacing a signed git tag, with a different tag of the same name, is a bad idea. More generally, reusing the same version number for a different (signed!) package is poor practice. Likewise, it’s usually a bad idea to remove changelog entries for versions which were actually released, just because they were later deemed improper.
We understand that many Debian contributors have gotten used to this kind of thing. Indeed, tools like dcut encourage it. It does allow you to make things neat-looking, even if you’ve made mistakes - but really it does so by covering up those mistakes!
The bottom line is that tag2upload can’t support such history-rewriting. If you discover a mistake after you’ve signed the tag, please just burn the version number and add a new changelog stanza.
One bonus of tag2upload’s approach is that it will discover if you are accidentally overwriting an NMU, and report that as an error.
tag2upload promises that the source package that it generates corresponds precisely to the git tree you tag and sign.
Orig tarballs make this complicated. They aren’t present on your laptop when you git-debpush. When you’re not uploading a new upstream version, the tag2upload service reuses existing orig tarballs from the archive. If your git and the archive’s orig don’t agree, the tag2upload service will report an error, rather than upload a package with contents that differ from your git tag.
With the most common Debian workflows, everything is fine:
If you base everything on upstream git, and make your orig tarballs with git archive (or git deborig), your orig tarballs are the same as the git, by construction. We recommend usually ignoring upstream tarballs: most upstreams work in git, and their tarballs can contain weirdness that we don’t want. (At worst, the tarball can contain an attack that isn’t visible in git, as with xz!)
Alternatively, if you use gbp import-orig, the differences (including an attack like Jia Tan’s) are imported into git for you. Then, once again, your git and the orig tarball will correspond.
But there are other workflows where this correspondence may not hold. Those workflows are hazardous, because the thing you’re probably working with locally for your routine development is the git view. Then, when you upload, your work is transplanted onto the orig tarball, which might be quite different - so what you upload isn’t what you’ve been working on!
This situation is detected by tag2upload, precisely because tag2upload checks that it’s keeping its promise: the source package is identical to the git view. (dgit push makes the same promise.)
We would love to have more contributors. There are some easy tasks to get started with, in bugs we’ve tagged “newcomer” — mostly UX improvements such as detecting certain problems earlier, in git-debpush.
More substantially, we are looking for help with sbuild: we’d like it to be able to work directly from git, rather than needing to build source packages: #868527.
Locking down database access is probably the single most important thing for a system administrator or software developer to prevent their application from leaking its data. As MariaDB 11.8 is the first long-term supported version with a few new key security features, let’s recap what the most important things are every DBA should know about MariaDB in 2025.
Back in the old days, MySQL administrators had a habit of running the clumsy mysql-secure-installation script, but it has long been obsolete. A modern MariaDB database server is already secure by default and locked down out of the box, and no such extra scripts are needed. On the contrary, the database administrator is expected to open up access to MariaDB according to the specific needs of each server. Therefore, it is important that the DBA can understand and correctly configure three things:
Separate application-specific users with granular permissions allowing only necessary access and no more.
Distributing and storing passwords and credentials securely
Ensuring all remote connections are properly encrypted
For holistic security, one should also consider proper auditing, logging, backups, regular security updates and more, but in this post we will focus only on the above aspects related to securing database access.
How encrypting database connections with TLS differs from web server HTTP(S)
Even though MariaDB (and other databases) use the same SSL/TLS protocol for encrypting remote connections as web servers and HTTPS, the way it is implemented is significantly different, and the different security assumptions are important for a database administrator to grasp.
Firstly, most HTTP requests to a web server are unauthenticated, meaning the web server serves public web pages and does not require users to log in. Traditionally, when a user logs in over a HTTP connection, the username and password were transmitted in plaintext as a HTTP POST request. Modern TLS, which was previously called SSL, does not change how HTTP works but simply encapsulates it. When using HTTPS, a web browser and a web server will start an encrypted TLS connection as the very first thing, and only once established, do they send HTTP requests and responses inside it. There are no passwords or other shared secrets needed to form the TLS connection. Instead, the web server relies on a trusted third party, a Certificate Authority (CA), to vet that the TLS certificate offered by the web server can be trusted by the web browser.
For a database server like MariaDB, the situation is quite different. All users need to first authenticate and log in to the server before getting being allowed to run any SQL and getting any data out of the server. The database server and client programs have built-in authentication methods, and passwords are not, and have never been, sent in plaintext. Over the years, MySQL and its successor, MariaDB, have had multiple password authentication methods: the original SHA-1-based hashing, later double SHA-1-based mysql_native_password, followed by sha256_password and caching_sha256_password in MySQL and ed25519 in MariaDB. The MariaDB.org blog post by Sergei Golubchik recaps the history of these well.
Even though most modern MariaDB installations should be using TLS to encrypt all remote connections in 2025, having the authentication method be as secure as possible still matters, because authentication is done before the TLS connection is fully established.
To further harden the authentication agains man-in-the-middle attacks, a new password the authentication method PARSEC was introduced in MariaDB 11.8, which builds upon the previous ed25519 public-key-based verification (similar to how modern SSH does), and also combines key derivation with PBKDF2 with hash functions (SHA512,SHA256) and a high iteration count.
At first it may seem like a disadvantage to not wrap all connections in a TLS tunnel like HTTPS does, but actually not having the authentication done in a MitM resistant way regardless of the connection encryption status allows a clever extra capability that is now available in MariaDB: as the database server and client already have a shared secret that is being used by the server to authenticate the user, it can also be used by the client to validate the server’s TLS certificate and no third parties like CAs or root certificates are needed. MariaDB 11.8 was the first LTS version to ship with this capability for zero-configuration TLS.
Note that the zero-configuration TLS also works on older password authentication methods and does not require users to have PARSEC enabled. As PARSEC is not yet the default authentication method in MariaDB, it is recommended to enable it in installations that use zero-configuration TLS encryption to maximize the security of the TLS certificate validation.
Why the ‘root’ user in MariaDB has no password and how it makes the database more secure
Relying on passwords for security is problematic, as there is always a risk that they could leak, and a malicious user could access the system using the leaked password. It is unfortunately far too common for database passwords to be stored in plaintext in configuration files that are accidentally committed into version control and published on GitHub and similar platforms. Every application or administrative password that exists should be tracked to ensure only people who need it know it, and rotated at regular intervals to ensure old employees etc won’t be able to use old passwords. This password management is complex and error-prone.
Replacing passwords with other authentication methods is always advisable when possible. On a database server, whoever installed the database by running e.g. apt install mariadb-server, and configured it with e.g. nano /etc/mysql/mariadb.cnf, already has full root access to the operating system, and asking them for a password to access the MariaDB database shell is moot, since they could circumvent any checks by directly accessing the files on the system anyway. Therefore, MariaDB, since version 10.4 stopped requiring the root user to enter a password when connecting locally, and instead checks using socket authentication whether the user is the operating-system root user or equivalent (e.g. running sudo). This is an elegant way to get rid of a password that was actually unnecessary to begin with. As there is no root password anymore, the risk of an external user accessing the database as root with a leaked password is fully eliminated.
Note that socket authentication only works for local connections on the same server. If you want to access a MariaDB server remotely as the root user, you would need to configure a password for it first. This is not generally recommended, as explained in the next section.
Create separate database users for normal use and keep ‘root’ for administrative use only
Out of the box a MariaDB installation is already secure by default, and only the local root user can connect to it. This account is intended for administrative use only, and for regular daily use you should create separate database users with access limited to the databases they need and the permissions required.
The most typical commands needed to create a new database for an app and a user the app can use to connect to the database would be the following:
sqlCREATE DATABASE app_db;
CREATE USER 'app_user'@'%' IDENTIFIED BY 'your_secure_password';
GRANT ALL PRIVILEGES ON app_db.* TO 'app_user'@'%';
FLUSH PRIVILEGES;
Alternatively, if you want to use the parsec authentication method, run this to create the user:
sqlCREATE OR REPLACE USER 'app_user'@'%'
IDENTIFIED VIA parsec
USING PASSWORD('your_secure_password');
CREATEORREPLACEUSER'app_user'@'%' IDENTIFIED VIA parsec
USING PASSWORD('your_secure_password');
Note that the plugin auth_parsec is not enabled by default. If you see the error message ERROR 1524 (HY000): Plugin 'parsec' is not loaded fix this by running INSTALL SONAME 'auth_parsec';.
In the CREATE USER statements, the @'%' means that the user is allowed to connect from any host. This needs to be defined, as MariaDB always checks permissions based on both the username and the remote IP address or hostname of the user, combined with the authentication method. Note that it is possible to have multiple user@remote combinations, and they can have different authentication methods. A user could, for example, be allowed to log in locally using the socket authentication and over the network using a password.
If you are running a custom application and you know exactly what permissions are sufficient for the database users, replace the ALL PRIVILEGES with a subset of privileges listed in the MariaDB documentation.
For new permissions to take effect, restart the database or run FLUSH PRIVILEGES.
Allow MariaDB to accept remote connections and enforce TLS
Using the above 'app_user'@'%' is not enough on its own to allow remote connections to MariaDB. The MariaDB server also needs to be configured to listen on a network interface to accept remote connections. As MariaDB is secure by default, it only accepts connections from localhost until the administrator updates its configuration. On a typical Debian/Ubuntu system, the recommended way is to drop a new custom config in e.g. /etc/mysql/mariadb.conf.d/99-server-customizations.cnf, with the contents:
[mariadbd]
# Listen for connections from anywhere
bind-address = 0.0.0.0
# Only allow TLS encrypted connections
require-secure-transport = on
[mariadbd]
# Listen for connections from anywhere
bind-address = 0.0.0.0
# Only allow TLS encrypted connections
require-secure-transport = on
For settings to take effect, restart the server with systemctl restart mariadb. After this, the server will accept connections on any network interface. If the system is using a firewall, the port 3306 would additionally need to be allow-listed.
To confirm that the settings took effect, run e.g. mariadb -e "SHOW VARIABLES LIKE 'bind_address';" , which should now show 0.0.0.0.
When allowing remote connections, it is important to also always define require-secure-transport = on to enforce that only TLS-encrypted connections are allowed. If the server is running MariaDB 11.8 and the clients are also MariaDB 11.8 or newer, no additional configuration is needed thanks to MariaDB automatically providing TLS certificates and appropriate certificate validation in recent versions.
On older long-term-supported versions of the MariaDB server one would have had to manually create the certificates and configure the ssl_key, ssl_cert and ssl_ca values on the server, and distribute the certificate to the clients as well, which was cumbersome, so good it is not required anymore. In MariaDB 11.8 the only additional related config that might still be worth setting is tls_version = TLSv1.3 to ensure only the latest TLS protocol version is used.
Finally, test connections to ensure they work and to confirm that TLS is used by running e.g.:
--------------
mariadb from 11.8.3-MariaDB, client 15.2 for debian-linux-gnu (x86_64)
...
Current user: app_user@192.168.1.66
SSL: Cipher in use is TLS_AES_256_GCM_SHA384, cert is OK
...
--------------
mariadb from 11.8.3-MariaDB, client 15.2 for debian-linux-gnu (x86_64)
...
Current user: app_user@192.168.1.66
SSL: Cipher in use is TLS_AES_256_GCM_SHA384, cert is OK
...
If running a Debian/Ubuntu system, see the bundled README with zcat /usr/share/doc/mariadb-server/README.Debian.gz to read more configuration tips.
Should TLS encryption be used also on internal networks?
If a database server and app are running on the same private network, the chances that the connection gets eavesdropped on or man-in-the-middle attacked by a malicious user are low. However, it is not zero, and if it happens, it can be difficult to detect or prove that it didn’t happen. The benefit of using end-to-end encryption is that both the database server and the client can validate the certificates and keys used, log it, and later have logs audited to prove that connections were indeed encrypted and show how they were encrypted.
If all the computers on an internal network already have centralized user account management and centralized log collection that includes all database sessions, reusing existing SSH connections, SOCKS proxies, dedicated HTTPS tunnels, point-to-point VPNs, or similar solutions might also be a practical option. Note that the zero-configuration TLS only works with password validation methods. This means that systems configured to use PAM or Kerberos/GSSAPI can’t use it, but again those systems are typically part of a centrally configured network anyway and are likely to have certificate authorities and key distribution or network encryption facilities already set up.
In a typical software app stac however, the simplest solution is often the best and I recommend DBAs use the end-to-end TLS encryption in MariaDB 11.8 in most cases.
Hopefully with these tips you can enjoy having your MariaDB deployments both simpler and more secure than before!
It's been a while since the last performance check of Transparent Data Encryption (TDE) in Cybertec's PGEE distribution - that was in 2016. Of course, the question is still interesting, so I did some benchmarks.
Since the difference is really small between running without any extras, with data checksums turned on, and with both encryption and checksums turned on, we need to pick a configuration that will stress-test these features the most. So in the spirit of making PostgreSQL deliberately run slow, I went with only 1MB of shared_buffers with a pgbench workload of scale factor 50. The 770MB of database size will easily fit into RAM. However, having such a small buffer cache setting will cause a lot of cache misses with pages re-read from the OS disk cache, checksums checked, and the page decrypted again. To further increase the effect, I ran pgbench --skip-some-updates so the smaller, in-cache-anyway pgbench tables are not touched. Overall, this yields a pretty consistent buffer cache hit rate of only 82.8%.
Here are the PGEE 17.6 tps (transactions per second) numbers averaged over a few 1-minute 3-client pgbench runs for different combinations of data checksums on/off, TDE off, and the various supported key bit lengths:
no checksums
data checksums
no TDE
2455,6
100,00 %
2449,7
99,76 %
128 bits
2440,9
99,40 %
2443,3
99,50 %
192 bits
2439,6
99,35 %
2446,1
99,61 %
256 bits
2450,3
99,78 %
2443,1
99,49 %
There is a lot of noise in the individual runtimes before averaging, so the numbers must be viewed with some care (192-bit TDE is certainly not faster with checksums than without), but if we dare to interpret these tiny differences, we can conclude the following:
The cost of enabling data checksums on this bad-cache-ratio workload is about 0.25 %.
The cost of enabling both TDE encryption and data checksums on this workload is about 0.5%.
Any workload with a better shared_buffers cache hit rate would see a lower penalty of enabling checksums and TDE than that.
Debusine is a tool designed
for Debian developers and Operating System developers in general. This posts
describes our approach to the use of JavaScript, and some practical designs we
came up with to integrate it with Django with minimal effort.
Debusine web UI and JavaScript
Debusine currently has 3 user interfaces: a client on the command line, a
RESTful API, and a Django-based Web UI.
Debusine’s web UI is a tool to interact with the system, and we want to spend
most of our efforts in creating a system that works and works well, rather than
chasing the latest and hippest of the frontend frameworks for the web.
Also, Debian as a community has an aversion to having parts of the JavaScript
ecosystem in the critical path of its core infrastructure, and in our
professional experience this aversion is not at all unreasonable.
This leads to having some interesting requirements for the web UI, that (rather
surprisingly, one would think) one doesn’t usually find advertised in many
projects:
Straightforward to create and maintain.
Well integrated with Django.
Easy to package in Debian, with as little vendoring as possible, which helps mitigate supply chain attacks
Usable without JavaScript whenever possible, for progressive
enhancement rather
than core functionality.
The idea is to avoid growing the technical complexity and requirements of the
web UI, both server-side and client-side, for functionality that is not needed
for this kind of project, with tools that do not fit well in our ecosystem.
Also, to limit the complexity of the JavaScript portions that we do develop, we
choose to limit our JavaScript browser supports to the main browser versions
packaged in Debian Stable, plus recent oldstable.
This makes JavaScript easier to write and maintain, and it also makes it less
needed, as modern HTML plus modern CSS interfaces can go a long way with less
scripting interventions.
From the start we built the UI using Bootstrap,
which helps in having responsive layouts that can also work on mobile devices.
When we started having large select fields, we introduced
Select2 to make interaction more efficient, and which
degrades gracefully to working HTML.
Both Bootstrap and Select2 are packaged in Debian.
Form validation is done server-side by Django, and we do not reimplement it
client-side in JavaScript, as we prefer the extra round trip through a form
submission to the risk of mismatches between the two validations.
In those cases where a UI task is not at all possible without JavaScript, we
can make its support mandatory as long as the same goal can be otherwise
achieved using the debusine client command.
Django messages as Bootstrap toasts
Django has a Messages framework
that allows different parts of a view to push messages to the user, and it is
useful to signal things like a successful form submission, or warnings on
unexpected conditions.
Django messages integrate well with Bootstrap
toasts, which use a
recognisable notification language, are nicely dismissible and do not invade
the rest of the page layout.
Doing so was surprisingly simple: we handle the toasts as usual, and also render the
plain alerts inside a <noscript> tag.
This is precisely the intended usage of the <noscript> tag, and it works
perfectly: toasts are displayed by JavaScript when it’s available, or rendered
as alerts when not.
The resulting Django template is something like this:
Debusine is built around
workspaces,
which are, among other things, containers for resources.
Workspaces can inherit from other workspaces, which act as fallback lookups for
resources. This allows, for example, to maintain an experimental package to be
built on Debian Unstable, without the need to copy the whole Debian Unstable
workspace. A workspace can inherit from multiple others, which are looked up in
order.
When adding UI to configure workspace
inheritance, we
faced the issue that plain HTML forms do not have a convenient way to perform
data entry of an ordered list.
We initially built the data entry around Django formsets, which support
ordering
using an extra integer input field to enter the ordering position. This works,
and it’s good as a fallback, but we wanted something more appropriate, like
dragging and dropping items to reorder them, as the main method of interaction.
Our final approach
renders the plain formset inside a <noscript> tag, and the JavaScript widget
inside a display: none element, which is later shown by JavaScript code.
As the workspace inheritance is edited, JavaScript serializes its state into
<form type='hidden'> fields that match the structure used by the formset, so
that when the form is submitted, the view performs validation and updates the
server state as usual without any extra maintenance burden.
Serializing state as hidden form fields looks a bit vintage, but it is an
effective way of preserving the established data entry protocol between the
server and the browser, allowing us to do incremental improvement of the UI
while minimizing the maintenance effort.
More to come
Debusine is now gaining significant adoption and is still under active
development, with new features like personal archives coming soon.
This will likely mean more user stories for the UI, so this is a design space
that we are going to explore again and again in the coming future.
Please consider supporting my work in Debian and elsewhere through Liberapay.
Debian stable updates work through three main channels: point releases, security repositories, and the updates repository. Understanding these ensures your system stays secure and current.
A note about suite names
Every Debian release, or suite, has a codename — the most recent major release was trixie, or Debian 13. The codename uniquely identifies that suite.
We also use changeable aliases to add meaning to the suite’s lifecycle. For example, trixie currently has the alias stable, but when forky becomes stable instead, trixie will become known as oldstable.
This post uses either codenames or aliases depending on context. In source lists, codenames are generally preferred since that avoids surprise major upgrades right after a release is made.
The stable suites (point releases)
stable and oldstable (currently trixie and bookworm) are only updated during a “point release.” This is a minor update released to a major version. For example, 13.1 is the first minor update to trixie. It’s not possible to install older minor versions of a suite except via the snapshots mechanism (not covered here). It’s possible to view past versions via snapshot.debian.org, which preserves historical Debian archives.
There are also the testing and unstable aliases for the development suites. However, these are not relevant for users who want to run officially released versions.
Almost every stable installation of Debian will be opted into a stable or oldstable base suite. An example APT source might look like:
Type: deb
URIs: http://deb.debian.org/debian
Suites: trixie
Components: main
Signed-By: /usr/share/keyrings/debian-archive-keyring.pgp
Or, in legacy sources.list style:
deb https://deb.debian.org/debian trixie main
The security suites (DSAs explained)
For urgent security-related updates, the Security Team maintains a counterpart suite for each stable suite. These are called stable-security and oldstable-security when maintained by Debian’s security team, and oldstable-security, oldoldstable-security, etc when maintained by the LTS team.
Example APT source:
Type: deb
URIs: https://deb.debian.org/debian-security
Suites: trixie-security
Components: main
Signed-By: /usr/share/keyrings/debian-archive-keyring.pgp
Or, in legacy sources.list style:
deb https://deb.debian.org/debian-security trixie-security main
For urgent non-security updates, the final recommended suites are stable-updates and oldstable-updates. This is where updates staged for a point release, but needed sooner, are published. Examples include virus database updates, timezone changes, urgent bug fixes for specific problems and corrections to errors in the release process itself.
Example APT source:
Type: deb
URIs: https://deb.debian.org/debian
Suites: trixie-updates
Components: main
Signed-By: /usr/share/keyrings/debian-archive-keyring.pgp
Or, in legacy sources.list style:
deb https://deb.debian.org/debian trixie-updates main
Debian enables the updates suites by default. Stable Update Announcements (SUAs) are published to debian-stable-announce@lists.debian.org. This is also where announcements of forthcoming point releases are published.
Summary
These are the recommended suites for all production Debian systems:
Suite
Example codename
Purpose
Announcements
stable
trixie
Base suite containing all the available software for a release. Point releases every 2–4 months including lower-severity security fixes that do not require immediate release.
After a release moves from oldstable to unsupported status, Long Term Support (LTS) takes over for several more years. LTS provides urgent security updates for selected architectures. For details, see wiki.debian.org/LTS.
If you’d like to stay informed, the official Debian announcement lists and release.debian.org share the latest schedules and updates.
![[RSS 1.0 Feed]](common/rss10.png){kind=small}
![[RSS 2.0 Feed]](common/rss20.png){kind=small}
![[Atom Feed]](common/atom.png){kind=small}
![[FOAF Subscriptions]](common/foaf.png){kind=small}
![[OPML Subscriptions]](common/opml.png){kind=small}
![[Hacker]](common/hacker.png){kind=small}
![[Planet]](common/planet.png){kind=small}
