Promotion
It's been quiet here (I hope to change that), but I want to share some good news: I've been promoted to Principal Software Engineer! Next February will start my 9th year with Red Hat. Time flies when you're having fun!
It's been quiet here (I hope to change that), but I want to share some good news: I've been promoted to Principal Software Engineer! Next February will start my 9th year with Red Hat. Time flies when you're having fun!
Two weeks have passed since Debconf 23 came to a close in Kochi, Kerala, India this year.
In keeping with the more relaxed nature of Debconf in India, the rest of my photos from the event were to be published about two weeks from the end of the event. That will give me a bit more time to process them correctly and also give all of you a chance to see these pictures with fresh eyes and stir up new memories from the event.
In the end we are looking at 653 photos and one video. Several different group photos, including a return of the pool group photo that was missing from the event since Mexico in 2006! This year was the first for a new camera (Canon R7) and I am quite happy with the results, even if I still need to learn a lot about this new beast. Also the gradual improvements of panorama stiching software (Hugin) ment that this year I did not need to manually correct any face-melt events on any of the group photos. So that is cool!
You can find all my photos on:
Also, don't forget to explore the rest of the Git LFS share content - there are very many great photos by others this year as well!
02 October, 2023 08:00AM by Aigars Mahinovs
01 October, 2023 09:48AM by Junichi Uekawa
This month I didn't have any particular focus. I just worked on issues in my info bubble.
The SWH work was sponsored. All other work was done on a volunteer basis.
If you are an email system administrator, you are probably using DKIM to sign your outgoing emails. You should be rotating the key regularly and automatically, and publishing old private keys. I have just released dkim-rotate 1.0; dkim-rotate is a tool to do this key rotation and publication.
If you are an email user, your email provider ought to be doing this. If this is not done, your emails are “non-repudiable”, meaning that if they are leaked, anyone (eg, journalists, haters) can verify that they are authentic, and prove that to others. This is not desirable (for you).
This problem was described at some length in Matthew Green’s article Ok Google: please publish your DKIM secret keys.
Avoiding non-repudiation sounds a bit like lying. After all, I’m advising creating a situation where some people can’t verify that something is true, even though it is. So I’m advocating casting doubt. Crucially, though, it’s doubt about facts that ought to be private. When you send an email, that’s between you and the recipient. Normally you don’t intend for anyone, anywhere, who happens to get a copy, to be able to verify that it was really you that sent it.
In practical terms, this verifiability has already been used by journalists to verify stolen emails. Associated Press provide a verification tool.
As a user, you probably don’t want your emails to be non-repudiable. (Other people might want to be able to prove you sent some email, but your email system ought to serve your interests, not theirs.)
So, your email provider ought to be rotating their DKIM keys, and publishing their old ones. At a rough guess, your provider probably isn’t :-(.
A quick and dirty way to guess is to have a friend look at the email headers of a message you sent. (It is important that the friend uses a different email provider, since often DKIM signatures are not applied within a single email system.)
If your friend sees a DKIM-Signature
header then the message is DKIM signed. If they don’t, then it wasn’t. Most email traversing the public internet is DKIM signed nowadays; so if they don’t see the header probably they’re not looking using the right tools, or they’re actually on the same email system as you.
In messages signed by a system running dkim-rotate, there will also be a header about the key rotation, to notify potential verifiers of the situation. Other systems that avoid non-repudiation-through-DKIM might do something similar. dkim-rotate’s header looks like this:
DKIM-Signature-Warning: NOTE REGARDING DKIM KEY COMPROMISE
https://www.chiark.greenend.org.uk/dkim-rotate/README.txt
https://www.chiark.greenend.org.uk/dkim-rotate/ae/aeb689c2066c5b3fee673355309fe1c7.pem
But an email system might do half of the job of dkim-rotate: regularly rotating the key would cause the signatures of old emails to fail to verify, which is a good start. In that case there probably won’t be such a header.
You can also try verifying the signatures. This isn’t entirely straightforward, especially if you don’t have access to low-level mail tooling. Your friend will need to be able to save emails as raw whole headers and body, un-decoded, un-rendered.
If your friend is using a traditional Unix mail program, they should save the message as an mbox file. Otherwise, ProPublica have instructions for attaching and transferring and obtaining the raw email. (Scroll down to “How to Check DKIM and ARC”.)
Firstly, have your friend test that they can in fact verify a DKIM signature. This will demonstrate that the next test, where the verification is supposed to fail, is working properly and fails for the right reasons.
Send your friend a test email now, and have them do this on a Linux system:
# save the message as test-email.mbox
apt install libmail-dkim-perl # or equivalent on another distro
dkimproxy-verify <test-email.mbox
You should see output containing something like this:
originator address: ijackson@chiark.greenend.org.uk
signature identity: @chiark.greenend.org.uk
verify result: pass
...
If the output ontains verify result: fail (body has been altered)
then probably your friend didn’t manage to faithfully save the unalterered raw message.
When you both have that working, have your friend find an older email of yours, from (say) month ago. Perform the same steps.
Hopefully they will see something like this:
originator address: ijackson@chiark.greenend.org.uk
signature identity: @chiark.greenend.org.uk
verify result: fail (bad RSA signature)
or maybe
verify result: invalid (public key: not available)
This indicates that this old email can no longer be verified. That’s good: it means that anyone who steals a copy, can’t verify it either. If it’s leaked, the journalist who receives it won’t know it’s genuine and unmodified; they should then be suspicious.
If your friend sees verify result: pass
, then they have verified that that old email of yours is genuine. Anyone who had a copy of the mail can do that. This is good for email thieves, but not for you.
I have been running dkim-rotate 0.4 on my infrastructure, since last August. and I had entirely forgotten about it: it has run flawlessly for a year. I was reminded of the topic by seeing DKIM in other blog posts. Obviously, it is time to decreee that dkim-rotate is 1.0.
If you’re a mail system administrator, your users are best served if you use something like dkim-rotate. The package is available in Debian stable, and supports Exim out of the box, but other MTAs should be easy to support too, via some simple ad-hoc scripting.
Even with this key rotation approach, emails remain nonrepudiable for a short period after they’re sent - typically, a few days.
Someone who obtains a leaked email very promptly, and shows it to the journalist (for example) right away, can still convince the journalist. This is not great, but at least it doesn’t apply to the vast bulk of your email archive.
There are possible email protocol improvements which might help, but they’re quite out of scope for this article.
Edited 2023-10-01 00:20 +01:00 to fix some grammarInteresting article in Wired about adversarial attacks on ML systems to get them to do things that they are explicitely programmed not to do such as describe how to make illegal drugs [1]. The most interesting part of this is that the attacks work on most GPT systems which is probably due to the similar data used to train them.
Vice has an interesting article about the Danish “Synthetic Party”, a political partyled by an AI [2]. Citizens can vote for candidates who will try to get laws passed that match the AI generated goals, there is no option of voting for an AI character. The policies they are advocating for are designed to appeal to the 20% of Danes who don’t vote. They are also trying to inspire similar parties in other countries. I think this has the potential to improve democracy.
Vice reports that in 2021 a man tried to assasinate the Queen of England with inspiration from Star Wars and an AI chat bot [3]. While someone who wants to be a real-life Sith is probably going to end up doing something bad we still don’t want to have chat bots encourage it.
Sam Varghese wrote an interesting article about the allegations that India is following the example of Saudi Arabia and assasinating people in other countries who disagree with their government [5]. We need to stop this.
Ian Jackson wrote an interesting blog post advocating that DKIM PRIVATE keys be rotated and PUBLISHED [6]. The idea is that if a hostile party gets access to the mailbox of someone who received private email from you then in the normal DKIM setup of keys never changing they can prove that the email is authentic when they leak it. While if you mail server publishes the old keys as Ian advocates then the hostile party can’t prove that you sent the email in question as anyone could have forged a signature. Anything that involves publishing a private key gets an immediate negative reaction but I can’t fault the logic here.
30 September, 2023 01:55PM by etbe
Almost 4 years after initial auto-cpufreq release, 4200 Github stars, 65 contributors & 42 releases, tool being topic of numerous Linux podcasts and shows, and...
The post auto-cpufreq v2.0 appeared first on FoolControl: Phear the penguin.
30 September, 2023 01:45PM by Adnan Hodzic
There are a couple of things I tend to do after packaging a piece of software for Debian, filing an Intent To Package bug and uploading the package. This is both a checklist for me and (hopefully) a way to inspire other maintainers to go beyond the basic package maintainer duties as documented in the Debian Developer's Reference.
If I've missed anything, please leave an comment or send me an email!
To foster collaboration and allow others to contribute to the packaging, I upload my package to a new subproject on Salsa. By doing this, I enable other Debian contributors to make improvements and propose changes via merge requests.
I also like to upload the project logo in the settings page (i.e. https://salsa.debian.org/debian/packagename/edit) since that will show up on some dashboards like the Package overview.
While Debian is my primary focus, I also want to keep an eye on how my package is doing on derivative distributions like Ubuntu. To do this, I subscribe to bugs related to my package on Launchpad. Ubuntu bugs are rarely Ubuntu-specific and so I will often fix them in Debian.
I also set myself as the answer contact on Launchpad Answers since these questions are often the sign of a Debian or a lack of documentation.
I don't generally bother to fix bugs on Ubuntu directly though since
I've not had much luck with packages in universe
lately. I'd rather not
spend much time preparing a package that's not going to end up being
released to users as part of a Stable Release
Update. On the other hand, I
have succesfully requested simple Debian
syncs when an important update
was uploaded after the Debian Import
Freeze.
I take screenshots of my package and upload them on https://screenshots.debian.net to help users understand what my package offers and how it looks. I believe that these screenshots end up in software "stores" type of applications.
Similarly, I add tags to my package using https://debtags.debian.org. I'm
not entirely sure where these tags are used, but they are visible from apt
show packagename
.
Staying up-to-date with upstream releases is one of the most important duties of a software packager. There are a lot of different ways that upstream software authors publicize their new releases. Here are some of the things I do to monitor these releases:
I have a cronjob which run uscan
once a day to check for new upstream
releases using the information specified in my debian/watch
files:
0 12 * * 1-5 francois test -e /home/francois/devel/deb && HTTPS_PROXY= https_proxy= uscan --report /home/francois/devel/deb || true
I subscribe to the upstream project's releases RSS feed, if available. For
example, I subscribe to the GitHub tags feed for
git-secrets
and
Launchpad announcements for
email-reminder
.
If the upstream project maintains an announcement mailing list, I subscribe to it (e.g. rkhunter-announce or tor release announcements).
When nothing else is available, I write a cronjob that downloads the upstream changelog once a day and commits it to a local git repo:
#!/bin/bash
pushd /home/francois/devel/zlib-changelog > /dev/null
wget --quiet -O ChangeLog.txt https://zlib.net/ChangeLog.txt || exit 1
git diff
git commit -a -m "Updated changelog" > /dev/null
popd > /dev/null
This sends me a diff by email when a new release is added (and no emails otherwise).
There is an article The Inappropriately Excluded by the Polymath Archives [1] that gets cited a lot. Mainly by Mensa types who think that their lack of success is due to being too smart.
The main claim is:
The probability of entering and remaining in an intellectually elite profession such as Physician, Judge, Professor, Scientist, Corporate Executive, etc. increases with IQ to about 133. It then falls by about 1/3 at 140. By 150 IQ the probability has fallen from its peak by 97%!
The first thing to consider is whether taking those professions is a smart thing to do. These are the types of jobs that a school career adviser would tell you are good choices for well paying jobs, but really there’s lots of professional positions that get similar pay with less demanding work. Physicians have to deal with people who are sick and patients who die – including cases where the physician needs to make a recommendation on incomplete information where the wrong choice will result in serious injury or death, there are significant benefits to being a medical researcher or doing biological engineering. Being a Judge has a high public profile and has a reasonable amount of pressure, good for status but you can probably earn more money with less work as a corporate lawyer. Being a professor is a position that is respected but which in many countries is very poorly paid. In a mid-size company executives probably get about $300k compared to $220k for middle managers and $100k-$180k for senior professional roles in the same company.
There has been research on how much happyness is increased by having more money, here is one from CBS saying that income up to $500K can increase happiness[2] which contradicts previous research suggesting that income over $75K didn’t provide much benefit. I think that part of this is determined by the conditions that you live in, if you live in a country like Australia with cheap healthcare then you won’t feel as great a need to hoard money. Another part is whether you feel obliged to compete with other people for financial status, if driving an old car of a non-prestige brand while my neighbours have new BMWs concerned me then I might desire an executive position.
I think that the smart thing to do is to get work that is relatively enjoyable, pays enough for all the essentials and some reasonable luxury, and doesn’t require excessive effort or long hours. Unless you have a great need for attention from other people then for every job with a high profile there will be several with similar salaries but less attention.
The main point of the article is that people with high IQs all want to reach the pinnacle of their career path and don’t do so because they are excluded. It doesn’t consider the possibility that smart people might have chosen the option that’s best for them. For example I’ve seen what my manager and the CIO of my company do and it doesn’t look like fun for me. I’m happy to have them earn more than me as compensation for doing things I don’t want to do.
This section of the article starts with “Because of the dearth of objective evidence, the cause of the exclusion cannot be determined directly” which is possibly where they should have given up. Also I could have concluded this blog post with “I’m not excluded from this list of jobs that suck”, but I will continue listing problems with the article.
One claim in the article is:
Garth Zietsman has said, referring to people with D15IQs over 152, ‘A common experience with people in this category or higher is that they are not wanted – the masses (including the professional classes) find them an affront of some sort.’
The question I have is whether it’s being smart or being a jerk that “the masses” find to be an affront, I’m guessing the latter. I don’t recall seeing evidence outside high school of people inherently disliking smarter people.
The article claims that “We have no reason to conclude that this upper limit on IQ differences changes in adulthood“. Schools don’t cater well to smart kids and it isn’t good for kids to have no intellectual peers. One benefit I’ve found in the Free Software community is that there are a lot of smart people.
Regarding leadership it claims “D.K. Simonton found that persuasiveness is at its maximum when the IQ differential between speaker and audience is about 20 points“. A good counter example is Julius Sumner Miller who successfully combined science education and advertising for children’s chocolate [3]. Maybe being a little smarter than other people makes it more difficult to communicate with them but being as smart as Julius Sumner Miller can outweigh that. The article goes on to claim that the intellectual elites have an average IQ of 125 because they have to convince people who have an average IQ of 105. I think that if that 20 point difference was really a thing then you would have politicians with an IQ of 125 appointing leaders of the public service with an IQ of 145 who would then hire scientific advisers with an IQ of 165. In a corporate environment a CEO with an IQ of 125 could hire a CIO with an IQ of 145 who could then hire IT staff with an IQ of 165. If people with 165 IQs wanted to be Prime Minister or CEO that might suck for them, but if they wanted to have the most senior technical roles in public service or corporations then it would work out well. For the work I do I almost never speak to a CEO and rarely speak to anyone who regularly speaks to them, if CEOs don’t like me and won’t hire people like me then it doesn’t matter to me as I won’t meet them.
The section on “Inappropriate Educational Options” is one where I almost agree with the author. I say almost because I don’t think that schools are good for anyone. Yes schools have some particular problems for smart kids, but they also have serious problems for kids who are below average IQ, kids who have problems at home, kids who are disabled, etc. Most schools fail so many groups of kids in so many ways that the overall culture of schools can’t be functional.
The section on “Social Isolation” is another where I almost agree with the author. But as with schools I think that society overall is poorly structured to support people such that people on the entire range of IQs have more difficulty in finding friends and relationships than they should. One easy change to make would be to increase the minimum wage such that one minimum wage job can support a family without working more than 35 hours a week and to set the maximum work week to something less than 40 hours Atlassian has a good blog post about the data on working weeks [4]. Wired has an article suggesting that 5 hours a day is an ideal work time for some jobs [5].
We also need improvements in public transport and city design to have less wasted time and better options for socialising.
The blogspot site hosting the article in question also has a very complex plan for funding a magazine for such articles [6]. The problems with that funding model start with selling “advertising” that converts to shares in a Turks & Caicos company in an attempt to circumvent securities regulations (things don’t work that way). Then it goes in to some complex formulas for where money will go. This isn’t the smart way to start a company, the smart way is to run a kickstarter with fixed rewards for specific amounts of contributions and then possibly have an offer of profit sharing with people who donate extra or something. As a general rule when doing something that’s new to you it’s a good idea to look at how others have succeeded at it in the past. Devising an experimental new way of doing something is best reserved to people who have some experience withe the more common methods.
Mentioning this may seem like an ad hominem attack, but I think it’s relevant to consider this in the context of people who score well in IQ tests but don’t do so well in other things. Maybe someone who didn’t think that they were a lot smarter than everyone else would have tried to launch a magazine in a more common way and actually had some success at it.
In a more general sense I think that people who believe that they are suffering because of being too smart are in a similar category as incels. It’s more of a psychological problem than anything else and one that they could solve for themselves.
30 September, 2023 05:47AM by etbe
I would like to welcome you to my revamped site. It is still a work in progress, so please be patient while I work out the kinks! I have also explained a bit more about myself in my About Me page for those that may have questions about my homesteader lifestyle. Check it out when you have time.
My site is mostly my adventures in packaging software in Linux in a variety of formats ( mostly Debian and Ubuntu Snaps containerized packages ). This keeps me very busy, as folks don’t realize the importance of packaging. Without it, applications remain in source code form which isn’t very usable by the users! While turning the source code into something user friendly we often run into issues and work with upstream ( I am a very strong believer in upstream first ) to resolve any issues. This makes for a better user experience and less buggy software. Workarounds are very hard to maintain and thus fixing it right the first time is the best path! With this said, while I am not strong in any one programming language ( Well maybe Ruby from my CI tooling background ) I am versed in many languages, as I have to understand the code that I am filing bug reports for! We have to have a strong knowledge of being able to understand build failures, debug runtime failures and most importantly we have to be able to fix them, or find the resources to assist in fixing them. As most of you know I am KDE’s biggest fan ( There is nothing wrong with Gnome, its a great platform ). So a big portion of my work is dedicated to KDE. A fantastic tool for working on my KDE packaging has been KDE Neon! With the developer version I have all the tools necessary to debug and fix issues that arise. There is also the added bonus of living on the edge and finding out runtime issues right away!
That is enough about me for now and on to my weekly round up!
KDE neon:
Carlos ( check out his new blog! https://www.ethicalconstruct.au/dotclear_blog/ ) and I have been very busy with another round of KDE applications making the move to Qt6. We have finished KDE PIM and KDE Games in Neon/unstable!
I have worked out issues with print-manager and re-enabled it in experimental as it’s qt6 development is still happening in kf6 branch. Instructions here: https://blog.neon.kde.org/2023/08/04/announcing-kde-neon-experimental/
Fixed issues with a broken kscreenlocker and missing window decorations. You can now safely leave your computer and not worry about that dreaded black screen.
Debian:
I have uploaded the newest squashfuse to unstable.
I have uploaded another NEW dependency for bubblegum golang-github-alecthomas-mango-kong-dev
Ubuntu Snaps:
This week continues working closely with Jarred Wilson of Canonical in getting his Qt6 content snap in shape for use with my KDE Frameworks 6 snap ( an essential snap to move forward with our next generation Qt6 applications and of course the Plasma snap.
I spent some time debugging the neochat snap and fixed some QML issues, but I am now facing issues with wayland. It now works fine for those of us still on X11. I will continue working out wayland.
Thank you!
I rely on donations to upkeep my everyday living and so far thanks to each and every one of you I have survived almost a full year! It has been scary from time to time, but I am surviving. Until my snap project goes through I must rely on the kindness of my supporters. The proceeds of my donations goes to the following:
I have joined the kool kids and moved to Donorbox for donations.
DonateI still have Gofundme for those that don’t want to signup for yetanotherdonationplatform.
29 September, 2023 04:25PM by sgmoore
Part of what I love to do is to play with embedded devices running Linux. And many times I need to juggle with an SD card in order to copy a filesystem to a board... until I found SDWire.
This little gadget is an SD muxer: it allows you to use it as a "normal" card reader or switch the SD card to the SD card slot. So next time I need to copy contents to a SD card and then push it to a board I can just do it from the command line.
Now the software to control this device was not in Debian, so I fixed that.
I am keeping the code and the packaging on GitLab, as "The Tizen project is not anymore really active".
At the time of writing the package was just accepted, so it might take a couple of hours to be available on the archive.
Enjoy!
28 September, 2023 07:19PM by Lisandro Damián Nicanor Pérez Meyer
Yesterday I tagged a new version of onak, my OpenPGP compatible keyserver. I’d spent a bit of time during DebConf doing some minor cleanups, in particular an annoying systemd socket activation issue I’d been seeing. That turned out to be due completely failing to compile in the systemd support, even when it was detected. There was also a signature verification issue with certain Ed225519 signatures (thanks Antoine Beaupré for making me dig into that one), along with various code cleanups.
I also worked on Stateless OpenPGP CLI support, which is something I talked about when I released 0.6.2. It isn’t something that’s suitable for release, but it is sufficient to allow running the OpenPGP interoperability test suite verification tests, which I’m pleased to say all now pass.
For the next release I’m hoping the OpenPGP crypto refresh process will have completed, which at the very least will mean adding support for v6 packet types and fingerprints. The PostgreSQL DB backend could also use some love, and I might see if performance with SQLite3 has improved any.
Anyway. Available locally or via GitHub.
0.6.3 - 26th September 2023
- Fix systemd detection + socket activation
- Add CMake checking for Berkeley DB
- Minor improvements to keyd logging
- Fix decoding of signature creation time
- Relax version check on parsing signature + key packets
- Improve HTML escaping
- Handle failed database initialisation more gracefully
- Fix bug with EDDSA signatures with top 8+ bits unset
The following contributors got their Debian Developer accounts in the last two months:
The following contributors were added as Debian Maintainers in the last two months:
Congratulations!
27 September, 2023 02:00PM by Jean-Pierre Giraud
Now this was quite a tease! For those who haven't seen it, I encourage you to check it out, it has a nice photo of a Debian t-shirt I did not know about, to quote the Fine Article:
Today, when going through a box of old T-shirts, I found the shirt I was looking for to bring to the occasion: [...]
For the benefit of people who read this using a non-image-displaying browser or RSS client, they are respectively:
10 years 100 countries 1000 maintainers 10000 packages
and
1 project 10 architectures 100 countries 1000 maintainers 10000 packages 100000 bugs fixed 1000000 installations 10000000 users 100000000 lines of code
20 years ago we celebrated eating grilled meat at J0rd1’s house. This year, we had vegan tostadas in the menu. And maybe we are no longer that young, but we are still very proud and happy of our project!
Now… How would numbers line up today for Debian, 20 years later? Have we managed to get the “bugs fixed” line increase by a factor of 10? Quite probably, the lines of code we also have, and I can only guess the number of users and installations, which was already just a wild guess back then, might have multiplied by over 10, at least if we count indirect users and installs as well…
Now I don't know about you, but I really expected someone to come up with an answer to this, directly on Debian Planet! I have patiently waited for such an answer but enough is enough, I'm a Debian member, surely I can cull all of this together. So, low and behold, here are the actual numbers from 2023!
~10 architectures: number almost unchanged, but the actual
architectures are of course different (woody released with
i386
, m68k
, Alpha, SPARC, PowerPC, ARM, IA-64, hppa
, mips
,
s390
; while bookworm released with actually 9 supported
architectures instead of 10: i386
, amd64
, aarch64
, armel
,
armhf
, mipsel
, mips64el
, ppc64el
, s390x
)
~100 countries: actually 63 now, but I suspect we were generously
rounding up last time as well (extracted with ldapsearch -b
ou=users,dc=debian,dc=org -D uid=anarcat,ou=users,dc=debian,dc=org
-ZZ -vLxW '(c=*)' c | grep ^c: | sort | uniq -c | sort -n | wc -l
on coccia
)
~1000 maintainers: amazingly, almost unchanged (according to the last DPL vote, there were 831 DDs in 2003 and 996 in the last vote)
35000 packages: that number obviously increased quite a bit, but
according to sources.debian.org, woody released with 5580
source packages and bookworm with 34782 source packages and
according to UDD, there are actually 200k+ binary packages (
SELECT COUNT(DISTINCT package) FROM all_packages;
=> 211151)
1 000 000+ (OVER ONE MILLION!) bugs fixed! now that number grew
by a whole order of magnitude, incredibly (934809 done, 16 fixed,
7595 forwarded, 82492 pending, 938 pending-fixed, according to UDD
again, SELECT COUNT(id),status FROM all_bugs GROUP BY status;
)
~1 000 000 installations (?): that one is hard to call. popcon has 225419 recorded installs, but it is likely an underestimate - hard to count
how many users? even harder, we were claiming ten million users then, how many now? how can we even begin to tell, with Debian running on the space station?
1 000 000 000+ (OVER ONE BILLION!) lines of code: that, interestingly, has also grown by an order of magnitude, from 100M to 1B lines of code, again according to sources.debian.org, woody shipped with 143M lines of codes and bookworm with 1.3 billion lines of code
So it doesn't line up as nicely, but it looks something like this:
1 project
10 architectures
30 years
100 countries (actually 63, but we'd like to have yours!)
1000 maintainers (yep, still there!)
35000 packages
211000 *binary* packages
1000000 bugs fixed
1000000000 lines of code
uncounted installations and users, we don't track you
So maybe the the more accurate, rounding to the nearest logarithm, would look something like:
1 project
10 architectures
100 countries (actually 63, but we'd like to have yours!)
1000 maintainers (yep, still there!)
100000 packages
1000000 bugs fixed
1000000000 lines of code
uncounted installations and users, we don't track you
I really like how the "packages" and "bugs fixed" still have an order of magnitude between them there, but that the "bugs fixed" vs "lines of code" have an extra order of magnitude, that is we have fixed ten times less bugs per line of code since we last did this count, 20 years ago.
Also, I am tempted to put 100 years
in there, but that would be
rounding up too much. Let's give it another 30 years first.
Hopefully, some real scientist is going to balk at this crude methodology and come up with some more interesting numbers for the next t-shirt. Otherwise I'm available for bar mitzvahs and children parties.
I recently bought an HP Chromebook from Abhas who had already flashed coreboot in it. I ran a fresh installation of Debian 12 (Bookworm) on it with KDE Plasma.
Right after installation, the Wi-Fi and bluetooth were working, but I was facing two issues:
Playing a music file or any audio file does not give any audio.
Keyboard buttons like the ones for brightness and audio adjustment were not working (alphabet keys were working).
So I asked my friend Alper for help on fixing the same as he has some experience with Chromebooks. Thanks a lot Alper for the help. I am documenting our steps here for helping others who are facing this issue.
Note: This works in X11. For wayland, the steps might differ.
To set system-wide keyboard configuration on Debian systems:
$ sudo dpkg-reconfigure keyboard-configuration
Choose “Chromebook” as the “Keyboard Model”.
Each DE should default to the system configuration, but might need its own configuration which would similarly be available in their GUI tools. But you can check and set it manually from the command line, for example as in this thread.
To check the keyboard model Xorg-based DEs:
$ setxkbmap -print -query | grep model:
model: pc104
To change it temporarily, until a reboot:
$ setxkbmap -model chromebook
If it’s not there in KDE settings that would be a bug, To change it persistently for KDE:
$ cat >>.config/kxkbrc <<EOF
[Layout]
Model=chromebook
EOF
This thread was helpful.
I recently bought an HP Chromebook from Abhas who had already flashed coreboot in it. I ran a fresh installation of Debian 12 (Bookworm) on it with KDE Plasma.
Right after installation, the Wi-Fi and bluetooth were working, but I was facing two issues:
Playing a music file or any audio file does not give any audio.
Keyboard buttons like the ones for brightness and audio adjustment were not working (alphabet keys were working).
I ran the script mentioned here and that fixed the audio.
The instructions from that link are:
git clone https://github.com/WeirdTreeThing/chromebook-linux-audio
cd chromebook-linux-audio
./setup-audio
I asked my friend Alper for help on fixing the keyboard as he has some experience with Chromebooks. Thanks a lot Alper for the help. I am documenting our steps here for helping others who are facing this issue.
Note: This works in X11. For wayland, the steps might differ.
To set system-wide keyboard configuration on Debian systems:
$ sudo dpkg-reconfigure keyboard-configuration
Choose “Chromebook” as the “Keyboard Model”.
Each DE should default to the system configuration, but might need its own configuration which would similarly be available in their GUI tools. But you can check and set it manually from the command line, for example as in this thread.
To check the keyboard model Xorg-based DEs:
$ setxkbmap -print -query | grep model:
model: pc104
To change it temporarily, until a reboot:
$ setxkbmap -model chromebook
If it’s not there in KDE settings that would be a bug, To change it persistently for KDE:
$ cat >>.config/kxkbrc <<EOF
[Layout]
Model=chromebook
EOF
This thread was helpful.
The new 7945HX CPU from AMD is currently the most powerful. I’d love to have one of them, to replace the now aging 6 core Xeon that I’ve been using for more than 5 years. So, I’ve been searching for a laptop with that CPU.
Absolutely all of the laptops I found with this CPU also embed a very powerful RTX 40×0 series GPU, that I have no use: I don’t play games, and I don’t do AI. I just want something that builds Debian packages fast (like Ceph, that takes more than 1h to build for me…). The more cores I get, the faster all OpenStack unit tests are running too (stestr does a moderately good job at spreading the tests to all cores). That’d be ok if I had to pay more for a GPU that I don’t need, and I would have deal with the annoyance of the NVidia driver, if only I could find something with a correct size. But I can only find 16″ or bigger laptops, that wont fit in my scooter back case (most of the time, these laptops have an 17 inch screen: that’s a way too big).
Currently, I found:
If one of the readers of this post find a smaller laptop with a 7945HX CPU, please let me know! Even better if I can get rid of the expensive NVidia GPU.
24 September, 2023 03:19PM by Goirand Thomas
Man, you’re no longer with us, but I am touched by the number of people you have positively impacted. Almost every DebConf23 presentations by locals I saw after you, carried how you were instrumental in bringing them there. How you were a dear friend and brother.
It’s a weird turn of events, that you left us during one thing we deeply cared and worked towards making possible since the last 3 years together. Who would have known, that “Sahil, I’m going back to my apartment tonight” and casual bye post that would be the last conversation we ever had.
Things were terrible after I heard the news. I had a hard time convincing myself to come see you one last time during your funeral. That was the last time I was going to get to see you, and I kept on looking at you. You, there in front of me, all calm, gave me peace. I’ll carry that image all my life now. Your smile will always remain with me.
Now, who’ll meet and receive me on the door at almost every Debian event (just by sheer co-incidence?). Who’ll help me speak out loud about all the Debian shortcomings (and then discuss solutions, when sober :)).
It was a testament of the amount of time we had already spent together online, that when we first met during MDC Palakkad, it didn’t feel we were physically meeting for the first time. The conversations just flowed. Now this song is associated with you due to your speech during post MiniDebConf Palakkad dinner. Hearing it reminds me of all the times we spent together chilling and talking community (which you cared deeply about). I guess, now we can’t stop caring for the community, because your energy was contagious.
Now, I can’t directly dial your number to listen - “Hey Sahil! What’s up?” from the other end, or “Tell me, tell me” on any mention of the problem. Nor would I be able to send reference usage of your Debian packaging guide in the wild. You already know how popular this guide of yours. How many people that guide has helped with getting started with packaging. Our last telegram text was me telling you about guide usage in Ravi’s DebConf23 presentation. Did I ever tell you, I too got my first start with packaging from there. I started looking up to you from there, even before we met or talked. Now, I missed telling you, I was probably your biggest fan whenever you had the mic in hand and started speaking. You always surprised me all the insights and idea you brought and would kept on impressing me for someone who was just my age but was way more mature.
Reading recent toots from Raju Dev made me realize how much I loved your writings. You wrote ‘How the Future will remember Us’, ‘Doing what’s right’ and many more. The level of depth in your thought was unparalleled. I loved reading those. That’s why I kept pestering you to write more, which you slowly stopped. Now I fully understand why though. You were busy; really busy helping people out or just working for making things better. You were doing Debian, upstream projects, web development, designs, graphics, mentoring, free software evangelism while being the go-to person for almost everyone around. Everyone depended on you, because you were too kind to turn down anyone.
Man, I still get your spelling wrong :) Did I ever tell you that? That was the reason, I used to use AR instead online.
You’ll be missed and will always be part of our conversations, because you have left a profound impact on me, our friends, Debian India and everyone around. See you! the coolest man around.
In memory:
PS - Just found you even had a Youtube channel, you one heck of a talented man.
This post describes how to define and use rule templates with semantic names using extends
or !reference
tags, how
to define manual jobs using the same templates and how to use gitlab-ci
inputs as macros to give names to regular expressions used by rules.
I keep my templates in a rules.yml
file stored on a common repository used from different projects as I mentioned on
my previous post, but they can be defined anywhere, the important thing is that the
files that need them include their definition somehow.
The first version of my rules.yml
file was as follows:
.rules_common:
# Common rules; we include them from others instead of forcing a workflow
rules:
# Disable branch pipelines while there is an open merge request from it
- if: >-
$CI_COMMIT_BRANCH &&
$CI_OPEN_MERGE_REQUESTS &&
$CI_PIPELINE_SOURCE != "merge_request_event"
when: never
.rules_default:
# Default rules, we need to add the when: on_success to make things work
rules:
- !reference [.rules_common, rules]
- when: on_success
The main idea is that .rules_common
defines a rule
section to disable jobs as we can do on a workflow
definition;
in our case common rules only have if
rules that apply to all jobs and are used to disable them. The example includes
one that avoids creating duplicated jobs when we push to a branch that is the source of an open MR as explained
here.
To use the rules in a job we have two options, use the extends
keyword (we do that when we want to use the rule as is)
or declare a rules
section and add a !reference
to the template we want to use as described
here (we do that when we want to add
additional rules to disable a job before evaluating the template conditions).
As an example, with the following definitions both jobs use the same rules:
job_1:
extends:
- .rules_default
[...]
job_2:
rules:
- !reference [.rules_default, rules]
[...]
To make the jobs manual we have two options, create a version of the job that includes when: manual
and defines if we
want it to be optional or not (allow_failure: true
makes the job optional, if we don’t add that to the rule the job is
blocking) or add the when: manual
and the allow_failure
value to the job (if we work at the job level the default
value for allow_failure
is false
for when: manual
, so it is optional by default, we have to add an explicit
allow_failure = true
it to make it blocking).
The following example shows how we define blocking or optional manual jobs using rules with when
conditions:
.rules_default_manual_blocking:
# Default rules for optional manual jobs
rules:
- !reference [.rules_common, rules]
- when: manual
# allow_failure: false is implicit
.rules_default_manual_optional:
# Default rules for optional manual jobs
rules:
- !reference [.rules_common, rules]
- when: manual
allow_failure: true
manual_blocking_job:
extends:
- .rules_default_manual_blocking
[...]
manual_optional_job:
extends:
- .rules_default_manual_optional
[...]
The problem here is that we have to create new versions of the same rule template to add the conditions, but we can avoid it using the keywords at the job level with the original rules to get the same effect; the following definitions create jobs equivalent to the ones defined earlier without creating additional templates:
manual_blocking_job:
extends:
- .rules_default
when: manual
allow_failure: false
[...]
manual_optional_job:
extends:
- .rules_default
when: manual
# allow_failure: true is implicit
[...]
As you can imagine, that is my preferred way of doing it, as it keeps the rules.yml
file smaller and I see that the
job is manual in its definition without problem.
allow_failure
, changes
, exists
, needs
or variables
Unluckily for us, for now there is no way to avoid creating additional templates as we did on the when: manual
case
when a rule is similar to an existing one but adds changes,
exists,
needs or
variables to it.
So, for now, if a rule needs to add any of those fields we have to copy the original rule and add the keyword section.
Some notes, though:
allow_failure
if we want to change its value for a given condition, in other cases we can set
the value at the job level.changes
to the rule it is important to make sure that they are going to be evaluated as explained
here.needs
value to a rule for a specific condition and it matches it replaces the job needs
section;
when using templates I would use two different job names with different conditions instead of adding a needs
on a
single job.I started to use rule templates to avoid repetition when defining jobs that needed the same rules and soon I noticed that giving them names with a semantic meaning they where easier to use and understand (we provide a name that tells us when we are going to execute the job, while the details of the variables names or values used on the rules are an implementation detail of the templates).
We are not going to define real jobs on this post, but as an example we are going to define a set of rules that can be
useful if we plan to follow a scaled trunk based
development workflow when developing, that is, we are going to put the releasable code on the main
branch and use
short-lived branches to test and complete changes before pushing things to main
.
Using this approach we can define an initial set of rule templates with semantic names:
.rules_mr_to_main:
rules:
- !reference [.rules_common, rules]
- if: $CI_MERGE_REQUEST_TARGET_BRANCH_NAME == 'main'
.rules_mr_or_push_to_main:
rules:
- !reference [.rules_common, rules]
- if: $CI_MERGE_REQUEST_TARGET_BRANCH_NAME == 'main'
- if: >-
$CI_COMMIT_BRANCH == 'main'
&&
$CI_PIPELINE_SOURCE != 'merge_request_event'
.rules_push_to_main:
rules:
- !reference [.rules_common, rules]
- if: >-
$CI_COMMIT_BRANCH == 'main'
&&
$CI_PIPELINE_SOURCE != 'merge_request_event'
.rules_push_to_branch:
rules:
- !reference [.rules_common, rules]
- if: >-
$CI_COMMIT_BRANCH != 'main'
&&
$CI_PIPELINE_SOURCE != 'merge_request_event'
.rules_push_to_branch_or_mr_to_main:
rules:
- !reference [.rules_push_to_branch, rules]
- if: >-
$CI_MERGE_REQUEST_SOURCE_BRANCH_NAME != 'main'
&&
$CI_MERGE_REQUEST_TARGET_BRANCH_NAME == 'main'
.rules_release_tag:
rules:
- !reference [.rules_common, rules]
- if: $CI_COMMIT_TAG =~ /^([0-9a-zA-Z_.-]+-)?v\d+.\d+.\d+$/
.rules_non_release_tag:
rules:
- !reference [.rules_common, rules]
- if: $CI_COMMIT_TAG !~ /^([0-9a-zA-Z_.-]+-)?v\d+.\d+.\d+$/
With those names it is clear when a job is going to be executed and when using the templates on real jobs we can add additional restrictions and make the execution manual if needed as described earlier.
inputs
as macrosOn the previous rules we have used a regular expression to identify the release tag format and assumed that the
general branches are the ones with a name different than main
; if we want to force a format for those branch names
we can replace the condition != 'main'
by a regex comparison (=~
if we look for matches, !~
if we want to define
valid branch names removing the invalid ones).
When testing the new gitlab-ci inputs my colleague Jorge noticed that if you keep their default value they basically work as macros.
The variables declared as inputs
can’t hold YAML values, the truth is that their value is always a string that is
replaced by the value assigned to them when including the file (if given) or by their default value, if defined.
If you don’t assign a value to an input variable when including the file that declares it its occurrences are replaced
by its default value, making them work basically as macros; this is useful for us when working with strings that can’t
managed as variables, like the regular expressions used inside if
conditions.
With those two ideas we can add the following prefix to the rules.yaml
defining inputs for both regular expressions
and replace the rules that can use them by the ones shown here:
spec:
inputs:
# Regular expression for branches; the prefix matches the type of changes
# we plan to work on inside the branch (we use conventional commit types as
# the branch prefix)
branch_regex:
default: '/^(build|ci|chore|docs|feat|fix|perf|refactor|style|test)\/.+$/'
# Regular expression for tags
release_tag_regex:
default: '/^([0-9a-zA-Z_.-]+-)?v\d+.\d+.\d+$/'
---
[...]
.rules_push_to_changes_branch:
rules:
- !reference [.rules_common, rules]
- if: >-
$CI_COMMIT_BRANCH =~ $[[ inputs.branch_regex ]]
&&
$CI_PIPELINE_SOURCE != 'merge_request_event'
.rules_push_to_branch_or_mr_to_main:
rules:
- !reference [.rules_push_to_branch, rules]
- if: >-
$CI_MERGE_REQUEST_SOURCE_BRANCH_NAME =~ $[[ inputs.branch_regex ]]
&&
$CI_MERGE_REQUEST_TARGET_BRANCH_NAME == 'main'
.rules_release_tag:
rules:
- !reference [.rules_common, rules]
- if: $CI_COMMIT_TAG =~ $[[ inputs.release_tag_regex ]]
.rules_non_release_tag:
rules:
- !reference [.rules_common, rules]
- if: $CI_COMMIT_TAG !~ $[[ inputs.release_tag_regex ]]
I’m going to finish this post with a comment about how I avoid defining extra rule templates in some common cases.
The idea is simple, we can use !reference
tags to fine tune rules when we need to add conditions to disable them
simply adding conditions with when: never
before referencing the template.
As an example, in some projects I’m using different job definitions depending on the DEPLOY_ENVIRONMENT
value to make
the job manual or automatic; as we just said we can define different jobs referencing the same rule adding a condition
to check if the environment is the one we are interested in:
deploy_job_auto:
rules:
# Only deploy automatically if the environment is 'dev' by skipping this job
# for other values of the DEPLOY_ENVIRONMENT variable
- if: $DEPLOY_ENVIRONMENT != "dev"
when: never
- !reference [.rules_release_tag, rules]
[...]
deploy_job_manually:
rules:
# Disable this job if the environment is 'dev'
- if: $DEPLOY_ENVIRONMENT == "dev"
when: never
- !reference [.rules_release_tag, rules]
when: manual
# Change this to `false` to make the deployment job blocking
allow_failure: true
[...]
If you think about it the idea of adding negative conditions is what we do with the .rules_common
template; we add
conditions to disable the job before evaluating the real rules.
The difference in that case is that we reference them at the beginning because we want those negative conditions on all
jobs and that is also why we have a .rules_default
condition with an when: on_success
for the jobs that only need to
respect the default workflow (we need the last condition to make sure that they are executed if the negative rules don’t
match).
Last week tragedy struck, and I saw the very best of the Debian community at work.
I heard first hand testimony about how helpless so many people felt at being physically unable to help their friend. I heard about how they couldn’t bear to leave and had to be ushered away to make space for rescue services to do their work. I heard of those who continued the search with private divers, even after the official rescue was called off.
I saw the shock and grief which engulfed everybody who I saw that night and in the following days. I watched friends comfort each other when it became too much. I read the messages we wrote in memory and smiled at how they described the person I’d only just started to know.
When I felt angry, and helpless, and frustrated that I couldn’t do more, the people around me caught me, comforted me, and cared for me.
Debian, you are like family and nobody can claim otherwise. You bicker and argue about the silliest things and sometimes it feels like we’ll never get past them. But when it comes to simple human compassion for each other, you always surprise me with your ability to care.
23 September, 2023 04:59PM by Jonathan
DebConf23, the 24th annual Debian Conference, was held in India in the city of Kochi, Kerala from the 3rd to the 17th of September, 2023. Ever since I got to know about it (which was more than an year ago), I was excited to attend DebConf in my home country. This was my second DebConf, as I attended one last year in Kosovo. I was very happy that I didn’t need to apply for a visa to attend. This time I submitted two talks - one on Debian packaging for beginners and the other on ideas on sustainable solutions for self-hosting. I got full bursary to attend the event (thanks a lot to Debian for that!) which is always helpful in covering the expenses, especially if the venue is a five star hotel :)
My friend Suresh - who is enthusiastic about Debian and free software - wanted to attend it too. When the registration started, I reminded him about applying. We landed in Kochi on the 28th of August 2023 during the festival of Onam. We celebrated Onam in Kochi, had a trip to Wayanad, and returned to Kochi. On the evening of the 3rd of September, we reached the venue - Four Points Hotel by Sheraton, at Infopark Kochi, Ernakulam, Kerala, India.
The hotel had 14 floors, and featured a swimming pool and gym (these were included in our package). The hotel gave us elevator access for only our floor, along with public spaces like the reception, gym, swimming pool, and dining areas. The temperature inside the hotel was pretty cold and I had to buy a jacket to survive. Perhaps the hotel was in cahoots with winterwear companies? :)
On the first day, Suresh and I had dinner at the eatery on the third floor. At the entrance, a member of the hotel staff asked us about how many people we wanted a table for. I told her that it’s just the two of us at the moment, but (as we are attending a conference) we might be joined by others. Regardless, they gave us a table for just two. Within a few minutes, we were joined by Alper from Turkey and urbec from Germany. So we shifted to a larger table…but then we were joined by even more people, so we were busy adding more chairs to our table. urbec had already been in Kerala for the past 5-6 days and was, on one hand, very happy already with the quality and taste of bananas in Kerala…and on the other, rather afraid of the spicy food :)
Two days later, the lunch and dinner were shifted to the All Spice Restaurant on the 14th floor, but the breakfast was still served at the eatery. Since the eatery (on the 3rd floor) had greater variety of food than the other venue, this move made breakfast the best meal for me and many others. Many attendees from outside India were not accustomed to the “spicy” food. It is difficult for locals to help them, because what we consider mild can be spicy for others. It is not easy to satisfy everyone at the dining table, but I think the organizing team did a very good job in the food department. (That said, it didn’t matter for me after a point, and you will know why.) The pappadam were really good, and I liked the rice labelled “Kerala rice”. I actually brought that exact rice and pappadam home during my last trip to Kochi and everyone at my home liked it too (thanks to Abhijit PA). I also wished to eat all types of payasams from Kerala and this really happened (thanks to Sruthi who designed the menu). Every meal had a different variety of payasam and it was awesome, although I didn’t like some of them, mostly because they were very sweet. Meals were later shifted to the ground floor (taking away the best breakfast option which was the eatery).
The DebConf registration desk was at the second floor. We were given a very nice swag bag. They were available in multiple colors - grey, green, blue, red - and included an umbrella, a steel mug, a multiboot USB drive by Mostly Harmless, a thermal flask, a mug by Canonical, a paper coaster, and stickers. It rained almost every day in Kochi during our stay, so handing out an umbrella to every attendee was a good idea.
During breakfast one day, Nattie expressed the desire to buy a coffee filter. The next time I went to the market, I bought a coffee filter for her as a gift. She seemed happy with the gift and was flattered to receive a gift from a young man :)
There were many newbies who were eager to learn and contribute to Debian. So, I mentored whoever came to me and was interested in learning. I conducted a packaging workshop in the bootcamp, but could only cover how to set up the Debian Unstable environment, and had to leave out how to package (but I covered that in my talk). Carlos (Brazil) gave a keysigning session in the bootcamp. Praveen was also mentoring in the bootcamp. I helped people understand why we sign GPG keys and how to sign them. I planned to take a workshop on it but cancelled it later.
My Debian packaging talk was on the 10th of September, 2023. I had not prepared slides for my Debian packaging talk in advance - I thought that I could do it during the trip, but I didn’t get the time…so I prepared them on the day before the talk. Since it was mostly a tutorial, the slides did not need much preparation. My thanks to Suresh, who helped me with the slides and made it possible to complete them in such a short time frame.
My talk was well-received by the audience, going by their comments. I am glad that I could give an interesting presentation.
After my talk, Suresh, Alper, and I went with Anisa and Kristi - who are both from Albania, and have a never-ending fascination for Indian culture :) - to buy them sarees. We took autos to Kakkanad market and found a shop with a great variety of sarees. I was slightly familiar with the area around the hotel, as I had been there for a week. Indian women usually don’t try on sarees while buying - they just select the design. But Anisa wanted to put one on and take a few photos as well. The shop staff did not have a trial saree for this purpose, so they took a saree from a mannequin. It took about an hour for the lady at the shop to help Anisa put on that saree…but you could tell that she was in heaven wearing that saree, and she bought it immediately :) Alper also bought a saree to take back to Turkey for his mother. Me and Suresh wanted to buy a kurta which would go well with the mundu we already had, but we could not find anything to our liking.
On the 11th of September we had the Cheese and Wine Party, a tradition of every DebConf. I brought Kaju Samosa and Nankhatai from home. Many attendees expressed their appreciation for the samosas. During the party, I was with Abhas and had a lot of fun. Abhas brought packets of paan and served them at the Cheese and Wine Party. We discussed interesting things and ate burgers. But due to the restrictive alcohol laws in the state, it was less fun compared to the previous DebConfs - you could only drink alcohol served by the hotel in public places. If you bought your own alcohol, you could only drink in private places (such as in your room, or a friend’s room), but not in public places.
Last year, Joenio (Brazilian) brought pastis from France which I liked. He brought the same alocholic drink this year too. So I invited him to my room after the Cheese and Wine party to have pastis. My idea was to have them with my roommate Suresh and Joenio. But then we permitted Joenio to bring as many people as he wanted…and he ended up bringing some ten people. Suddenly, the room was crowded. I was having good time at the party, serving them the snacks given to me by Abhas. The news of an alcohol party at my room spread like wildfire. Soon there were so many people that the AC became ineffective and I found myself sweating.
I left the room and roamed around in the hotel for some fresh air. I came back after about 1.5 hours - for most part, I was sitting at the ground floor with TK Saurabh. And then I met Abraham near the gym (which was my last meeting with him). I came back to my room at around 2:30 AM. Nobody seemed to have realized that I was gone. They were thanking me for hosting such a good party. A lot of people left at that point and the remaining people were playing songs and dancing (everyone was dancing all along!). I had no energy left to dance and to join them. They left around 03:00 AM. But I am glad that people enjoyed partying in my room.
On the 12th of September, we had a sadhya thali for lunch. It is a vegetarian thali served on a banana leaf on the eve of Thiruvonam. It wasn’t Thiruvonam on this day, but we got a special and filling lunch. The rasam and payasam were especially yummy.
On the 13th of September, we had a daytrip. I chose the daytrip houseboat in Allepey. Suresh chose the same, and we registered for it as soon as it was open. This was the most sought-after daytrip by the DebConf attendees - around 80 people registered for it.
Our bus was set to leave at 9 AM on the 13th of September. Me and Suresh woke up at 8:40 and hurried to get to the bus in time. It took two hours to reach the venue where we get the houseboat.
The houseboat experience was good. The trip featured some good scenery. I got to experience the renowned Kerala backwaters. We were served food on the boat. We also stopped at a place and had coconut water. By evening, we came back to the place where we had boarded the boat.
When we came back from the daytrip, we received news that Abhraham Raji was involved in a fatal accident during a kayaking trip.
Abraham Raji was a very good friend of mine. In my Albania-Kosovo-Dubai trip last year, he was my roommate at our Tirana apartment. I roamed around in Dubai with him, and we had many discussions during DebConf22 Kosovo. He was the one who took the photo of me on my homepage. I also met him in MiniDebConf22 Palakkad and MiniDebConf23 Tamil Nadu, and went to his flat in Kochi this year in June.
We had many projects in common. He was a Free Software activist and was the designer of the DebConf23 logo, in addition to those for other Debian events in India.
We were all fairly shocked by the news. I was devastated. Food lost its taste, and it became difficult to sleep. That night, Anisa and Kristi cheered me up and gave me company. Thanks a lot to them.
The next day, Joenio also tried to console me. I thank him for doing a great job. I thank everyone who helped me in coping with the difficult situation.
On the next day (the 14th of September), the Debian project leader Jonathan Carter addressed and announced the news officially. THe Debian project also mentioned it on their website.
Abraham was supposed to give a talk, but following the incident, all talks were cancelled for the day. The conference dinner was also cancelled.
As I write, 9 days have passed since his death, but even now I cannot come to terms with it.
On the 15th of September, the conference ran two buses from the hotel to Abraham’s house in Kottayam (2 hours ride). I hopped in the first bus and my mood was not very good. Evangelos (Germany) was sitting opposite me, and he began conversing with me. The distraction helped and I was back to normal for a while. Thanks to Evangelos as he supported me a lot on that trip. He was also very impressed by my use of the StreetComplete app which I was using to edit OpenStreetMap.
In two hours, we reached Abraham’s house. I couldn’t control myself and burst into tears. I went to see the body. I met his family (mother, father and sister), but I had nothing to say and I felt helpless. Owing to the loss of sleep and appetite over the past few days, I had no energy, and didn’t think it was good idea for me to stay there. I went back by taking the bus after one hour and had lunch at the hotel. I withdrew my talk scheduled for the 16th of September.
I got a nice Japanese gift from Niibe Yutaka (Japan) - a folder to keep papers which had ancient Japanese manga characters. He said he felt guilty as he swapped his talk with me and so it got rescheduled from 12th September to 16 September which I withdrew later.
On the 16th of September, we had a group photo. I am glad that this year I was more clear in this picture than in DebConf22.
I attended the training session for the video team and worked as a camera operator. The Bits from DPL was nice. I enjoyed Abhas’ presentation on home automation. He basically demonstrated how he liberated Internet-enabled home devices. I also liked Kristi’s presentation on ways to engage with the GNOME community.
I also attended lightning talks on the last day. Badri, Wouter, and I gave a demo on how to register on the Prav app. Prav got a fair share of advertising during the last few days.
Suresh left the hotel and Badri joined me in my room. Thanks to the efforts of Abhijit PA, Kiran, and Ananthu, I wore a mundu.
I then joined Kalyani, Mangesh, Ruchika, Anisa, Ananthu and Kiran. We took pictures and this marked the last night of DebConf23.
The 18th of September was the day of departure. Badri slept in my room and left early morning (06:30 AM). I dropped him off at the hotel gate. The breakfast was at the eatery (3rd floor) again, and it was good.
Sahil, Saswata, Nilesh, and I hung out on the ground floor.
I had an 8 PM flight from Kochi to Delhi, for which I took a cab with Rhonda (Austria), Michael (Nigeria) and Yash (India). We were joined by other DebConf23 attendees at the Kochi airport, where we took another selfie.
Joost and I were on the same flight, and we sat next to each other. He then took a connecting flight from Delhi to Netherlands, while I went with Yash to the New Delhi Railway Station, where we took our respective trains. I reached home on the morning of the 19th of September, 2023.
DebConf23 was hard to organize - strict alcohol laws, weird hotel rules, death of a close friend (almost a family member), and a scary notice by the immigration bureau. The people from the team are my close friends and I am proud of them for organizing such a good event.
None of this would have been possible without the organizers who put more than a year-long voluntary effort to produce this. In the meanwhile, many of them had organized local events in the time leading up to DebConf. Kudos to them.
The organizers also tried their best to get clearance for countries not approved by the ministry. I am also sad that people from China, Kosovo, and Iran could not join. In particular, I feel bad for people from Kosovo who wanted to attend but could not (as India does not consider their passport to be a valid travel document), considering how we Indians were so well-received in their country last year.
I am writing this on the 22nd of September, 2023. It took me three days to put up this post - this was one of the tragic and hard posts for me to write. I have literally forced myself to write this. I have still not recovered from the loss of my friend. Thanks a lot to all those who helped me.
PS: Credits to contrapunctus for making grammar, phrasing, and capitalization changes.
With user edition out the door last week, this week was spent stabilizing unstable!
Spent some time sorting out our Calamares installer being quite grumpy which is now fixed by reverting an upstream change. Unstable and developer ISO rebuilt and installable. Spent some time sorting out some issues with using an unreleased appstream ( thanks ximion for help with packagekit! ) KDE applications are starting to switch to Qt6 in master this week, the big one being KDE PIM! This entails an enormous amount of work re-packaging. I have made a dent, sorta. To be continued next week. I fixed our signond / kaccounts line for qt6 which entailed some work on upstream code that uses QStringList.toSet which was removed in Qt6! Always learning new things!
I have spent some time working on the KF6 content snap, working with Jarred to make sure his qt6 content snap will work for us. Unfortunately, I do not have much time for this as I must make money to survive, donations help free up time for this Our new proposal with Kevin’s super awesome management company has been submitted and we will hopefully hear back next week.
Thanks for stopping by! Till next week.
If you can spare some change, consider a donation
Thank you!
22 September, 2023 06:10PM by sgmoore
Almost a month ago, I went to my always loved Rancho Electrónico to celebrate the 30th anniversary of the Debian project. Hats off to Jathan for all the work he put into this! I was there for close to 3hr, and be it following up an install, doing a talk, or whatever — he was doing it. But anyway, I only managed to attend with one of my (great, beautiful and always loved) generic Debian or DebConf T-shirts.
Today, when going through a box of old T-shirts, I found the shirt I was looking for to bring to the occasion. A smallish print, ~12cm wide, over the heart:
And as a larger print, ~25cm wide, across the back:
For the benefit of people who read this using a non-image-displaying browser or RSS client, they are respectively:
10 years
100 countries
1000 maintainers
10000 packages
and
1 project
10 architectures
100 countries
1000 maintainers
10000 packages
100000 bugs fixed
1000000 installations
10000000 users
100000000 lines of code
20 years ago we celebrated eating grilled meat at J0rd1’s house. This year, we had vegan tostadas in the menu. And maybe we are no longer that young, but we are still very proud and happy of our project!
Now… How would numbers line up today for Debian, 20 years later? Have we managed to get the “bugs fixed” line increase by a factor of 10? Quite probably, the lines of code we also have, and I can only guess the number of users and installations, which was already just a wild guess back then, might have multiplied by over 10, at least if we count indirect users and installs as well…
I very, very nearly didn’t make it to DebConf this year, I had a bad cold/flu for a few days before I left, and after a negative covid-19 test just minutes before my flight, I decided to take the plunge and travel.
This is just everything in chronological order, more or less, it’s the only way I could write it.
I planned to spend DebCamp working on various issues. Very few of them actually got done, I spent the first few days in bed further recovering, took a covid-19 test when I arrived and after I felt better, and both were negative, so not sure what exactly was wrong with me, but between that and catching up with other Debian duties, I couldn’t make any progress on catching up on the packaging work I wanted to do. I’ll still post what I intended here, I’ll try to take a few days to focus on these some time next month:
Calamares / Debian Live stuff:
At least Calamares has been trixiefied in testing, so there’s that!
Desktop stuff:
The “Egg” theme that I want to develop for testing/unstable is based on Juliette Taka’s Homeworld theme that was used for Bullseye. Egg, as in, something that hasn’t quite hatched yet. Get it? (for #1038660)
Debian Social:
Loopy:
I intended to get the loop for DebConf in good shape before I left, so that we can spend some time during DebCamp making some really nice content, unfortunately this went very tumbly, but at least we ended up with a loopy that kind of worked and wasn’t too horrible. There’s always another DebConf to try again, right?
So DebCamp as a usual DebCamp was pretty much a wash (fitting with all the rain we had?) for me, at least it gave me enough time to recover a bit for DebConf proper, and I had enough time left to catch up on some critical DPL duties and put together a few slides for the Bits from the DPL talk.
Bits From the DPL
I had very, very little available time to prepare something for Bits fro the DPL, but I managed to put some slides together (available on my wiki page).
I mostly covered:
Job Fair
I walked through the hallway where the Job Fair was hosted, and enjoyed all the buzz. It’s not always easy to get this right, but this year it was very active and energetic, I hope lots of people made some connections!
Cheese & Wine
Due to state laws and alcohol licenses, we couldn’t consume alcohol from outside the state of Kerala in the common areas of the hotel (only in private rooms), so this wasn’t quite as big or as fun as our usual C&W parties since we couldn’t share as much from our individual countries and cultures, but we always knew that this was going to be the case for this DebConf, and it still ended up being alright.
Day Trip
I opted for the forest / waterfalls daytrip. It was really, really long with lots of time in the bus. I think our trip’s organiser underestimated how long it would take between the points on the route (all in all it wasn’t that far, but on a bus on a winding mountain road, it takes long). We left at 8:00 and only found our way back to the hotel around 23:30. Even though we arrived tired and hungry, we saw some beautiful scenery, animals and also met indigenous river people who talked about their struggles against being driven out of their place of living multiple times as government invests in new developments like dams and hydro power.
Photos available in the DebConf23 public git repository.
Losing a beloved Debian Developer during DebConf
To our collective devastation, not everyone made it back from their day trips. Abraham Raji was out to the kayak day trip, and while swimming, got caught by a whirlpool from a drainage system.
Even though all of us were properly exhausted and shocked in disbelief at this point, we had to stay up and make some tough decisions. Some initially felt that we had to cancel the rest of DebConf. We also had to figure out how to announce what happened asap both to the larger project and at DebConf in an official manner, while ensuring that due diligence took place and that the family is informed by the police first before making anything public.
We ended up cancelling all the talks for the following day, with an address from the DPL in the morning to explain what had happened. Of all the things I’ve ever had to do as DPL, this was by far the hardest. The day after that, talks were also cancelled for the morning so that we could attend his funeral. Dozens of DebConf attendees headed out by bus to go pay their final respects, many wearing the t-shirts that Abraham had designed for DebConf.
A book of condolences was set up so that everyone who wished to could write a message on how they remembered him. The book will be kept by his family.
Today marks a week since his funeral, and I still feel very raw about it. And even though there was uncertainty whether DebConf should even continue after his death, in hindsight I’m glad that everyone pushed forward. While we were all heart broken, it was also heart warming to see people care for each other in all of this. If anything, I think I needed more time at DebConf just to be in that warm aura of emotional support for just a bit longer. There are many people who I wanted to talk to who I barely even had a chance to see.
Abraham, or Abru as he was called by some people (which I like because “bru” in Afrikaans” is like “bro” in English, not sure if that’s what it implied locally too) enjoyed artistic pursuits, but he was also passionate about knowledge transfer. He ran classes at DebConf both last year and this year (and I think at other local events too) where he taught people packaging via a quick course that he put together. His enthusiasm for Debian was contagious, a few of the people who he was mentoring came up to me and told me that they were going to see it through and become a DD in honor of him. I can’t even remember how I reacted to that, my brain was already so worn out and stitching that together with the tragedy of what happened while at DebConf was just too much for me.
I first met him in person last year in Kosovo, I already knew who he was, so I think we interacted during the online events the year before. He was just one of those people who showed so much promise, and I was curious to see what he’d achieve in the future. Unfortunately, we was taken away from us too soon.
Poetry Evening
Later in the week we had the poetry evening. This was the first time I had the courage to recite something. I read Ithaka by C.P. Cavafy (translated by Edmund Keely). The first time I heard about this poem was in an interview with Julian Assange’s wife, where she mentioned that he really loves this poem, and it caught my attention because I really like the Weezer song “Return to Ithaka” and always wondered what it was about, so needless to say, that was another rabbit hole at some point.
Group Photo
Our DebConf photographer organised another group photo for this event, links to high-res versions available on Aigar’s website.
BoFs
I didn’t attend nearly as many talks this DebConf as I would’ve liked (fortunately I can catch up on video, should be released soon), but I did make it to a few BoFs.
In the Local Groups BoF, representatives from various local teams were present who introduced themselves and explained what they were doing. From memory (sorry if I left someone out), we had people from Belgium, Brazil, Taiwan and South Africa. We talked about types of events a local group could do (BSPs, Mini DC, sprints, Debian Day, etc. How to help local groups get started, booth kits for conferences, and setting up some form of calendar that lists important Debian events in a way that makes it easier for people to plan and co-ordinate. There’s a mailing list for co-ordination of local groups, and the irc channel is #debian-localgroups on oftc.
In the Debian.net BoF, we discussed the Debian.net hosting service, where Debian pays for VMs hosted for projects by individual DDs on Debian.net. The idea is that we start some form of census that monitors the services, whether they’re still in use, whether the system is up to date, whether someone still cares for it, etc. We had some discussion about where the lines of responsibility are drawn, and we can probably make things a little bit more clear in the documentation. We also want to offer more in terms of backups and monitoring (currently DDs do get 500GB from rsync.net that could be used for backups of their services though). The intention is also to deploy some form of configuration management for some essentials across the hosts. We should also look at getting some sponsored hosting for this.
In the Debian Social BoF, we discussed some services that need work / expansion. In particular, Matrix keeps growing at an increased rate as more users use it and more channels are bridged, so it will likely move to its own host with big disks soon. We might replace Pleroma with a fork called Akkoma, this will need some more home work and checking whether it’s even feasible. Some services haven’t really been used (like Writefreely and Plume), and it might be time to retire them. We might just have to help one or two users migrate some of their posts away if we do retire them. Mjolner seems to do a fine job at spam blocking, we haven’t had any notable incidents yet. WordPress now has improved fediverse support, it’s unclear whether it works on a multi-site instance yet, I’ll test it at some point soon and report back. For upcoming services, we are implementing Lemmy and probably also Mobilizon. A request was made that we also look into Loomio.
More Information Overload
There’s so much that happens at DebConf, it’s tough to take it all in, and also, to find time to write about all of it, but I’ll mention a few more things that are certainly worth of note.
During DebConf, we had some people from the Kite Linux team over. KITE supplies the ICT needs for the primary and secondary schools in the province of Kerala, where they all use Linux. They decided to switch all of these to Debian. There was an ad-hoc BoF where locals were listening and fielding questions that the Kite Linux team had. It was great seeing all the energy and enthusiasm behind this effort, I hope someone will properly blog about this!
I learned about the VGLUG Foundation, who are doing a tremendous job at promoting GNU/Linux in the country. They are also training up 50 people a year to be able to provide tech support for Debian.
I came across the booth for Mostly Harmless, they liberate old hardware by installing free firmware on there. It was nice seeing all the devices out there that could be liberated, and how it can breathe new life into old harware.
Overall, the community and their activities in India are very impressive, and I wish I had more time to get to know everyone better.
Food
Oh yes, one more thing. The food was great. I tasted more different kinds of curry than I ever did in my whole life up to this point. The lunch on banana leaves was interesting, and also learning how to eat this food properly by hand (thanks to the locals who insisted on teaching me!), it was a… fruitful experience? This might catch on at home too… less dishes to take care of!
Special thanks to the DebConf23 Team
I think this may have been one of the toughest DebConfs to organise yet, and I don’t think many people outside of the DebConf team knows about all the challenges and adversity this team has faced in organising it. Even just getting to the previous DebConf in Kosovo was a long and tedious and somewhat risky process. Through it all, they were absolute pro’s. Not once did I see them get angry or yell at each other, whenever a problem came up, they just dealt with it. They did a really stellar job and I did make a point of telling them on the last day that everyone appreciated all the work that they did.
Back to my nest
I bought Dax a ball back from India, he seems to have forgiven me for not taking him along.
I’ll probably take a few days soon to focus a bit on my bugs and catch up on my original DebCamp goals. If you made it this far, thanks for reading! And thanks to everyone for being such fantastic people.
21 September, 2023 08:36PM by jonathan
(I wrote this up for an internal work post, but I figure it’s worth sharing more publicly too.)
I spent last week at DebConf23, this years instance of the annual Debian conference, which was held in Kochi, India. As usual, DebConf provides a good reason to see a new part of the world; I’ve been going since 2004 (Porto Alegre, Brazil), and while I’ve missed a few (Mexico, Bosnia, and Switzerland) I’ve still managed to make it to instances on 5 continents.
This has absolutely nothing to do with work, so I went on my own time + dime, but I figured a brief write-up might prove of interest. I first installed Debian back in 1999 as a machine that was being co-located to operate as a web server / email host. I was attracted by the promise of easy online upgrades (or, at least, upgrades that could be performed without the need to be physically present at the machine, even if they naturally required a reboot at some point). It has mostly delivered on this over the years, and I’ve never found a compelling reason to move away. I became a Debian Developer in 2000. As a massively distributed volunteer project DebConf provides an opportunity to find out what’s happening in other areas of the project, catch up with team mates, and generally feel more involved and energised to work on Debian stuff. Also, by this point in time, a lot of Debian folk are good friends and it’s always nice to catch up with them.
On that point, I felt that this year the hallway track was not quite the same as usual. For a number of reasons (COVID, climate change, travel time, we’re all getting older) I think fewer core teams are achieving critical mass at DebConf - I was the only member physically present from 2 teams I’m involved in, and I’d have appreciated the opportunity to sit down with both of them for some in-person discussions. It also means it’s harder to use DebConf as a venue for advancing major changes; previously having all the decision makers in the same space for a week has meant it’s possible to iron out the major discussion points, smoothing remote implementation after the conference. I’m told the mini DebConfs are where it’s at for these sorts of meetings now, so perhaps I’ll try to attend at least one of those next year.
Of course, I also went to a bunch of talks. I have differing levels of comment about each of them, but I’ve written up some brief notes below about the ones I remember something about. The comment was made that we perhaps had a lower level of deep technical talks, which is perhaps true but I still think there were a number of high level technical talks that served to pique ones interest about the topic.
Finally, this DebConf was the first I’m aware of that was accompanied by tragedy; as part of the day trip Abraham Raji, a project member and member of the local team, was involved in a fatal accident.
Opening Ceremony
Not much to say here; welcome to DebConf!
Continuous Key-Signing Party introduction
I ended up running this, as Gunnar couldn’t make it. Debian makes heavy use of the OpenPGP web of trust (no mass ability to send out Yubikeys + perform appropriate levels of identity verification), so making sure we’re appropriately cross-signed, and linked to local conference organisers, is a dull but important part of the conference. We use a modified keysigning approach where identity verification + fingerprint confirmation happens over the course of the conference, so this session was just to explain how that works and confirm we were all working from the same fingerprint list.
State of Stateless - A Talk about Immutability and Reproducibility in Debian
Stateless OSes seem to be gaining popularity, so I went along to this to see if there was anything of note. It was interesting, but nothing earth shattering - very high level.
What’s missing so that Debian is finally reproducible?
Reproducible builds are something I’ve been keeping an eye on for a long time, and I continue to be impressed by the work folks are putting into this - both for Debian, and other projects. From a security standpoint reproducible builds provide confidence against trojaned builds, and from a developer standpoint knowing you can build reproducibly helps with not having to keep a whole bunch of binary artefacts around.
Hello from keyring-maint
In the distant past the process of getting your OpenPGP key into the Debian keyring (which is used to authenticate uploads + votes, amongst other things) was a clunky process that was often stalled. This hasn’t been the case for at least the past 10 years, but there’s still a residual piece of project memory that thinks keyring is a blocker. So as a team we say hi and talk about the fact we do monthly updates and generally are fairly responsive these days.
A declarative approach to Linux networking with Netplan
Debian’s /etc/network/interfaces
is a fairly basic (if powerful) mechanism for configuring network interfaces. NetworkManager is a better bet for dynamic hosts (i.e. clients), and systemd-network
seems to be a good choice for servers (I’m gradually moving machines over to it). Netplan tries to provide a unified mechanism for configuring both with a single configuration language. A noble aim, but I don’t see a lot of benefit for anything I use - my NetworkManager hosts are highly dynamic (so no need to push shared config) and systemd-network
(or /etc/network/interfaces
) works just fine on the other hosts. I’m told Netplan has more use with more complicated setups, e.g. when OpenVSwitch is involved.
Quick peek at ZFS, A too good to be true file system and volume manager.
People who use ZFS rave about it. I’m naturally suspicious of any file system that doesn’t come as part of my mainline kernel. But, as a longtime cautious mdraid+lvm+ext4 user I appreciate that there have been advances in the file system space that maybe I should look at, and I’ve been trying out btrfs on more machines over the past couple of years. I can’t deny ZFS has a bunch of interesting features, but nothing I need/want that I can’t get from an mdraid+lvm+btrfs stack (in particular data checksumming + reflinks for dedupe were strong reasons to move to btrfs over ext4).
Bits from the DPL
Exactly what it says on the tin; some bits from the DPL.
Adulting
Enrico is always worth hearing talk; Adulting was no exception. Main takeaway is that we need to avoid trying to run the project on martyrs and instead make sure we build a sustainable project. I’ve been trying really hard to accept I just don’t have time to take on additional responsibilities, no matter how interesting or relevant they might seem, so this resonated.
My life in git, after subversion, after CVS.
Putting all of your home directory in revision control. I’ve never made this leap; I’ve got some Ansible playbooks that push out my core pieces of configuration, which is held in git, but I don’t actually check this out directly on hosts I have accounts on. Interesting, but not for me.
EU Legislation BoF - Cyber Resilience Act, Product Liability Directive and CSAM Regulation
The CRA seems to be a piece of ill informed legislation that I’m going to have to find time to read properly. Discussion was a bit more alarmist than I personally feel is warranted, but it was a short session, had a bunch of folk in it, and even when I removed my mask it was hard to make myself understood.
What’s new in the Linux kernel (and what’s missing in Debian)
An update from Ben about new kernel features. I’m paying less attention to such things these days, so nice to get a quick overview of it all.
Intro to SecureDrop, a sort-of Linux distro
Actually based on Ubuntu, but lots of overlap with Debian as a result, and highly customised anyway. Notable, to me, for using OpenPGP as some of the backend crypto support. I managed to talk to Kunal separately about some of the pain points around that, which was an interesting discussion - they’re trying to move from GnuPG to Sequoia, primarily because of the much easier integration and lack of requirement for the more complicated GnuPG features that sometimes get in the way.
The Docker(.io) ecosystem in Debian
I hate Docker. I’m sure it’s fine if you accept it wants to take over the host machine entirely, but when I’ve played around with it that’s not been the case. This talk was more about the difficulty of trying to keep a fast moving upstream with lots of external dependencies properly up to date in a stable release. Vendoring the deps and trying to get a stable release exception seems like the least bad solution, but it’s a problem that affects a growing number of projects.
Chiselled containers
This was kinda of interesting, but I think I missed the piece about why more granular packaging wasn’t an option. The premise is you can take an existing .deb
and “chisel” it into smaller components, which then helps separate out dependencies rather than pulling in as much as the original .deb
would. This was touted as being useful, in particular, for building targeted containers. Definitely appealing over custom built userspaces for containers, but in an ideal world I think we’d want the information in the main packaging and it becomes a lot of work.
Debian Contributors shake-up
Debian Contributors is a great site for massaging your ego around contributions to Debian; it’s also a useful point of reference from a data protection viewpoint in terms of information the project holds about contributors - everything is already public, but the Contributors website provides folk with an easy way to find their own information (with various configurable options about whether that’s made public or not). Tássia is working on improving the various data feeds into the site, but realistically this is the responsibility of every Debian service owner.
New Member BOF
I’m part of the teams that help get new folk into Debian - primarily as a member of the New Member Front Desk, but also as a mostly inactive Application Manager. It’s been a while since we did one of these sessions so the Front Desk/Debian Account Managers that were present did a panel session. Nothing earth shattering came out of it; like keyring-maint this is a team that has historically had problems, but is currently running smoothly.
nanotime
Support
The still new package RcppInt64
(announced two weeks ago in this
post, with this
followup last week) arrived on CRAN earlier today in its second
update and relase 0.0.3. RcppInt64
collects some of the previous conversions between 64-bit integer values
in R and C++, and regroups them in a single package by providing a
single header. It offers two interfaces: both a more standard
as<>()
converter from R values along with its
companions wrap()
to return to R, as well as more dedicated
functions ‘from’ and ‘to’.
This release adds support for the corresponding nanotime conversion between R and C++. nanotime is leveraging the same bit64-based reprensentation of 64-bit integers for nanosecond resolution timestamps. A thorough S4 wrapping the offers R based access for convenient and powerful operations at nanosecond resolution. And as tweeted (here and here), tooted (here and here), and skeeted (here and here) in a quick preview last Sunday, it makes for easy and expressive code.
The brief NEWS entry follows:
Changes in version 0.0.3 (2023-09-19)
The
as<>()
andwrap()
converters are now declaredinline
.Conversion to and from nanotime has been added.
Courtesy of my CRANberries, there is a diffstat report relative to previous release.
If you like this or other open-source work I do, you can sponsor me at GitHub.
This post by Dirk Eddelbuettel originated on his Thinking inside the box blog. Please report excessive re-aggregation in third-party for-profit settings.
As far as I know this is the first Haskell program compiled to Webassembly (WASM) with mainline ghc and using the browser DOM.
ghc's WASM backend is solid, but it only provides very low-level FFI bindings when used in the browser. Ints and pointers to WASM memory. (See here for details and for instructions on getting the ghc WASM toolchain I used.)
I imagine that in the future, WASM code will interface with the DOM by using a WASI "world" that defines a complete API (and browsers won't include Javascript engines anymore). But currently, WASM can't do anything in a browser without calling back to Javascript.
For this project, I needed 63 lines of (reusable) javascript (here). Plus another 18 to bootstrap running the WASM program (here). (Also browser_wasi_shim)
But let's start with the Haskell code. A simple program to pop up an alert in the browser looks like this:
{-# LANGUAGE OverloadedStrings #-}
import Wasmjsbridge
foreign export ccall hello :: IO ()
hello :: IO ()
hello = do
alert <- get_js_object_method "window" "alert"
call_js_function_ByteString_Void alert "hello, world!"
A larger program that draws on the canvas and generated the image above is here.
The Haskell side of the FFI interface is a bunch of fairly mechanical functions like this:
foreign import ccall unsafe "call_js_function_string_void"
_call_js_function_string_void :: Int -> CString -> Int -> IO ()
call_js_function_ByteString_Void :: JSFunction -> B.ByteString -> IO ()
call_js_function_ByteString_Void (JSFunction n) b =
BU.unsafeUseAsCStringLen b $ \(buf, len) ->
_call_js_function_string_void n buf len
Many more would need to be added, or generated, to continue down this path to complete coverage of all data types. All in all it's 64 lines of code so far (here).
Also a C shim is needed, that imports from WASI modules and provides C functions that are used by the Haskell FFI. It looks like this:
void _call_js_function_string_void(uint32_t fn, uint8_t *buf, uint32_t len) __attribute__((
__import_module__("wasmjsbridge"),
__import_name__("call_js_function_string_void")
));
void call_js_function_string_void(uint32_t fn, uint8_t *buf, uint32_t len) {
_call_js_function_string_void(fn, buf, len);
}
Another 64 lines of code for that (here). I found this pattern in Joachim Breitner's haskell-on-fastly and copied it rather blindly.
Finally, the Javascript that gets run for that is:
call_js_function_string_void(n, b, sz) {
const fn = globalThis.wasmjsbridge_functionmap.get(n);
const buffer = globalThis.wasmjsbridge_exports.memory.buffer;
fn(decoder.decode(new Uint8Array(buffer, b, sz)));
},
Notice that this gets an identifier representing the javascript function to run, which might be any method of any object. It looks it up in a map and runs it. And the ByteString that got passed from Haskell has to be decoded to a javascript string.
In the Haskell program above, the function is document.alert
. Why not
pass a ByteString with that through the FFI? Well, you could. But then
it would have to eval it. That would make running WASM in the browser be
evaling Javascript every time it calls a function. That does not seem like a
good idea if the goal is speed. GHC's
javascript backend
does use Javascript`FFI snippets like that, but there they get pasted into the generated
Javascript hairball, so no eval is needed.
So my code has things like get_js_object_method
that look up things like
Javascript functions and generate identifiers. It also has this:
call_js_function_ByteString_Object :: JSFunction -> B.ByteString -> IO JSObject
Which can be used to call things like document.getElementById
that return a javascript object:
getElementById <- get_js_object_method (JSObjectName "document") "getElementById"
canvas <- call_js_function_ByteString_Object getElementById "myCanvas"
Here's the Javascript called by get_js_object_method
. It generates a
Javascript function that will be used to call the desired method of the object,
and allocates an identifier for it, and returns that to the caller.
get_js_objectname_method(ob, osz, nb, nsz) {
const buffer = globalThis.wasmjsbridge_exports.memory.buffer;
const objname = decoder.decode(new Uint8Array(buffer, ob, osz));
const funcname = decoder.decode(new Uint8Array(buffer, nb, nsz));
const func = function (...args) { return globalThis[objname][funcname](...args) };
const n = globalThis.wasmjsbridge_counter + 1;
globalThis.wasmjsbridge_counter = n;
globalThis.wasmjsbridge_functionmap.set(n, func);
return n;
},
This does mean that every time a Javascript function id is looked up, some more memory is used on the Javascript side. For more serious uses of this, something would need to be done about that. Lots of other stuff like object value getting and setting is also not implemented, there's no support yet for callbacks, and so on. Still, I'm happy where this has gotten to after 12 hours of work on it.
I might release the reusable parts of this as a Haskell library, although it seems likely that ongoing development of ghc will make it obsolete. In the meantime, clone the git repo to have a play with it.
This blog post was sponsored by unqueued on Patreon.
And another half-written article I had in my repo. (Am I doing #FallFinishAlong with blog articles instead of / in addition to craft projects? it feels so).
I was in need of snacks. I wanted something bready, but with a bit less carbs and more proteins. I had a bag of chickpea flour.
Ingredients were:
Mix everything as usual for bread, leave to rise for 4-6 hours.
Divide in 4 parts, roll them out to a thickness of about 1 – 2 mm, prick them with a fork (or the fancy cracker pricking tool that you don’t really need but I may have bought).
Optionally spray with a bit of water and sprinkle with salt (coarse or flake is best).
Preheat the oven to 240°C and cook for 5 minutes, or preheat the oven to 210°C and cook for 10 minutes for a dryer version.
I’ve tried both cooking temperatures: the 210°C had the big advantage of being the same as the common bread I was already making, so no additional oven time was required (it was summer. this was a consideration.), but I’m not sure which version I like best, so I think in winter I will alternate between the two.
Put it in a cotton (linen?) bag and keep it in a dry place, where it will keep for weeks (assuming you’ve made a bigger batch :D ).
This is now part of my staples.
I’ve found the draft of this article I almost completely wrote (checks) more than 3 months ago, and I guess it’s time to finish and publish it.
The very first A10 based device that entered our home, before they started to use it for SBCs, was a Chinese media server with a slot for a 3.5" hard disk and a few analogue audio outputs.
Of course from the time it entered our home it has always been running Debian.
Now that we may have a few, more powerful, A20 SBCs available, it has been fully dedicated to music: it is connected to the stereo, the hard disk has a clone of the git-annex repository where I’ve ripped all of my CDs1 and is played by connecting via ssh and running alsaplayer.
It’s simple (for my tastes). it works. Unless you’re in a different room than the one with the stereo.
And then I read this post on the fediverse that suggested installing minidlna on an old Raspberry Pi, and on a whim I decided I needed it.
I installed the package from Debian, set a few easy settings in the configuration file (such as which directories to serve), restarted the service.
And that’s it, it just worked.
The only thing I had to take care about is that to use git-annex repositories as the source of media files either the root of the repository should be used as media-dir
, or the wide_links
options should be set to yes
, otherwise the symbolic links that point outside of the media-dir
won’t be served.
As a client I used VLC (note for myself in case I ever forget, it’s under View → Playlist and then Local Network → Universal Plug ‘n’ Play), mostly because I already had it around.
Now that I’ve used it for a few months I have to say that it isn’t a perfect solution: the main format I store my music in is in flac 2, which isn’t supported by dlna; for a significant part of it I’ve also already generated single-song ogg files for convenience 3, and at least it works for those.
I don’t store (yet?) my video collection on the A10 device, but I did a quick test, and no, of course a raw dump of a DVD as generated by dvdbackup isn’t supported, so I would have to convert those too. And there is some support for subtitles as a separate file (which is something else I tend to have around), but only one file, with strict requirements for the name, which is a bit restrictive.
On the other hand, the hassle to set this up was so low that the functionality / hassle ratio is most definitely worth, and I’m quite happy I did and plan to continue using it.
(yes, I’m still using CDs, I have objections to those newfangled streaming services)↩︎
full CD rip in a single file, with an embedded cuesheet, to preserve as much as possible of the original disc.↩︎
in theory the flac was supposed to be future-proof storage, with the ogg files for actual use, but then I always listen to full albums, so the flac just work, and I only tend to bother generating the ogg when I’m already generating mp3 for the car.↩︎
I'm writing hash tables again; it seemingly never goes out of fashion. (Like malloc or sorting, we can always improve the implementation of these super-old concepts.) There are so many different tradeoffs you can make, and I thought it would be interesting to summarize the options on one of them: Hash reductions. I.e., you have your hash value (assume it's 32 bits, but this generalizes readily) and want to figure out which of N buckets this reduces to; what do you choose? (I'll assume a standard open-addressing scheme with linear probing, but most of this can be adapted to pretty much anything.) As far as I know, your options are:
x & (N - 1)
,
where N is the table size. Assumptions: N is a power of two.
Advantages: Super-fast. Probably the preferred variation of
every gung-ho coder out there, very widely used. Problems:
The lower bits of your hash must be of good quality (all others
are discarded). Power-of-two requirement can mean lower load
factor, and can be problematic for very large tables (e.g.
if you have 64 GB RAM, you may want to support 60 GB hash tables
and not just 32).x % N
. Assumptions: Generally
that N is a prime (there's no big reason not to make it so).
Advantages: Flexible on table size. Uses all bits of the hash,
so is fairly robust against bad hash functions (the only big
problem is if your hash is always a multiple of N, really).
Disadvantages: Modulo is frequently slow, especially on older
or less powerful CPUs. If you have fast multiplication,
you can get around it by precomputation and numerical tricks,
to a certain extent.(x * q) >> (32 - B)
,
where q is some magic constant (usually a prime close to
the inverse of the golden ratio, but other values can work well,
too), and S is the number of bits you want. Assumptions:
N is a power of two. Advantages: Much better hash mixing
than just masking (enough that it often can compensate
for a bad hash, or just hash integers directly).
Faster than the modulo option. Problems: Needs fast multiplication and
variable-length shifts, and again, the power-of-two demand
may be a problem.((uint64_t)x * N) >> 32
.
(It's surprising that it works, but essentially, you
consider x as a 0.32 fixed-point number [0,1), multiply
by N and then truncate. Popularized by Daniel Lemire.)
Assumptions: You have access to a “high mul” somehow,
either through 64-bit muls or a CPU that will give you
high and low parts of the result separately (this is common,
although not all compilers have perfect code generation here).
Advantages: Fast, even more so if the high mul gives you the
shift for free. Completely arbitrary table size.
Problems: Need fast high-mul. Assumes the
high bits of the hash are of good quality, analogous to the
issue with masking off the lower bits.In a sense, my favorite is the range partition one. But it puts some trust in your hash, so it might not be the best for e.g. a generic library implementation.
On Sunday 17 September 2023, the annual Debian Developers and Contributors Conference came to a close.
Over 474 attendees representing 35 countries from around the world came together for a combined 89 events made up of Talks, Discussons, Birds of a Feather (BoF) gatherings, workshops, and activities in support of furthering our distribution, learning from our mentors and peers, building our community, and having a bit of fun.
The conference was preceded by the annual DebCamp hacking session held September 3d through September 9th where Debian Developers and Contributors convened to focus on their Individual Debian related projects or work in team sprints geared toward in-person collaboration in developing Debian.
In particular this year Sprints took place to advance development in Mobian/Debian, Reproducible Builds, and Python in Debian. This year also featured a BootCamp that was held for newcomers staged by a team of dedicated mentors who shared hands-on experience in Debian and offered a deeper understanding of how to work in and contribute to the community.
The actual Debian Developers Conference started on Sunday 10 September 2023.
In addition to the traditional 'Bits from the DPL' talk, the continuous key-signing party, lightning talks and the announcement of next year's DebConf4, there were several update sessions shared by internal projects and teams.
Many of the hosted discussion sessions were presented by our technical teams who highlighted the work and focus of the Long Term Support (LTS), Android tools, Debian Derivatives, Debian Installer, Debian Image, and the Debian Science teams. The Python, Perl, and Ruby programming language teams also shared updates on their work and efforts.
Two of the larger local Debian communities, Debian Brasil and Debian India shared how their respective collaborations in Debian moved the project forward and how they attracted new members and opportunities both in Debian, F/OSS, and the sciences with their HowTos of demonstrated community engagement.
The schedule was updated each day with planned and ad-hoc activities introduced by attendees over the course of the conference. Several activities that were unable to be held in past years due to the Global COVID-19 Pandemic were celebrated as they returned to the conference's schedule: a job fair, the open-mic and poetry night, the traditional Cheese and Wine party, the group photos and the Day Trips.
For those who were not able to attend, most of the talks and sessions were videoed for live room streams with the recorded videos to be made available later through the Debian meetings archive website. Almost all of the sessions facilitated remote participation via IRC messaging apps or online collaborative text documents which allowed remote attendees to 'be in the room' to ask questions or share comments with the speaker or assembled audience.
DebConf23 saw over 4.3 TiB of data streamed, 55 hours of scheduled talks, 23 network access points, 11 network switches, 75 kb of equipment imported, 400 meters of gaffer tape used, 1,463 viewed streaming hours, 461 T-shirts, 35 country Geoip viewers, 5 day trips, and an average of 169 meals planned per day.
All of these events, activies, conversations, and streams coupled with our love, interest, and participation in Debian annd F/OSS certainly made this conference an overall success both here in Kochi, India and On-line around the world.
The DebConf23 website will remain active for archival purposes and will continue to offer links to the presentations and videos of talks and events.
Next year, DebConf24 will be held in Haifa, Israel. As tradition follows before the next DebConf the local organizers in Israel will start the conference activites with DebCamp with particular focus on individual and team work towards improving the distribution.
DebConf is committed to a safe and welcome environment for all participants. See the web page about the Code of Conduct in DebConf23 website for more details on this.
Debian thanks the commitment of numerous sponsors to support DebConf23, particularly our Platinum Sponsors: Infomaniak, Proxmox, and Siemens.
We also wish to thank our Video and Infrastructure teams, the DebConf23 and DebConf commitiees, our host nation of India, and each and every person who helped contribute to this event and to Debian overall.
Thank you all for your work in helping Debian continue to be "The Universal Operating System".
See you next year!
The Debian Project was founded in 1993 by Ian Murdock to be a truly free community project. Since then the project has grown to be one of the largest and most influential open source projects. Thousands of volunteers from all over the world work together to create and maintain Debian software. Available in 70 languages, and supporting a huge range of computer types, Debian calls itself the universal operating system.
DebConf is the Debian Project's developer conference. In addition to a full schedule of technical, social and policy talks, DebConf provides an opportunity for developers, contributors and other interested people to meet in person and work together more closely. It has taken place annually since 2000 in locations as varied as Scotland, Argentina, and Bosnia and Herzegovina. More information about DebConf is available from https://debconf.org/.
Infomaniak is a key player in the European cloud market and the leading developer of Web technologies in Switzerland. It aims to be an independent European alternative to the web giants and is committed to an ethical and sustainable Web that respects privacy and creates local jobs. Infomaniak develops cloud solutions (IaaS, PaaS, VPS), productivity tools for online collaboration and video and radio streaming services.
Proxmox develops powerful, yet easy-to-use open-source server software. The product portfolio from Proxmox, including server virtualization, backup, and email security, helps companies of any size, sector, or industry to simplify their IT infrastructures. The Proxmox solutions are based on the great Debian platform, and we are happy that we can give back to the community by sponsoring DebConf23.
Siemens is technology company focused on industry, infrastructure and transport. From resource-efficient factories, resilient supply chains, smarter buildings and grids, to cleaner and more comfortable transportation, and advanced healthcare, the company creates technology with purpose adding real value for customers. By combining the real and the digital worlds, Siemens empowers its customers to transform their industries and markets, helping them to enhance the everyday of billions of people.
For further information, please visit the DebConf23 web page at https://debconf23.debconf.org/ or send mail to press@debian.org.
18 September, 2023 02:30PM by Jean-Pierre Giraud and Donald Norwood
Some time ago our LUG bought some things from soldered.com and while browsing around the website my SO and I decided to add a junk box to the order and see what we would get.
Other than a few useful things, there were two mostly unpopulated boards for the inkplate 10 which would have been pretty hard to reuse as electronics.
On the other hand, at 23 cm × 18 cm they are a size that is reasonable for a book, and the slit near a long edge made them look suitable for the cover plates of a coptic bound book.
Since the size isn’t a standard one, I used some paper I already had in big (A1) sheet: Clairefontaine Dessin Croquis Blanc at 120 g/m², and cut 32 sheet 466 mm × 182 mm big, to have room to trim the excess at the end and straighten the edges. This would make 8 signatures of 4 sheet each, for a total of 128 pages.
The paper will make it suitable both as a notebook (where I’ll write with liquid ink, of course, not ballpoints) or as a sketchbook for pencil (but not wet techniques).
I could have added a few more signatures, but this felt already good enough, and the risk to end up with an half-empty notebook was non-trivial (I will already have to force myself to actually use it, rather than keep it for a good topic that will never be).
First we finished depopulating the boards, using it as a desoldering exercise and trying (and not always succeeding) to save as many components as possible, even if most of them were too tiny for our current soldiering skills.
And then I only had to sew the book, which was done mostly while watching the DebConf streams.
And a couple of days later, trim and sand the pages, which as usual I could have done better, but, well, it works.
The next time I do something like this I think I will have to add a couple more mm also to the height, to be able to trim also those edges.
And now of course the Big Question is: what should I dedicate this notebook to? Will I actually use it? This year? This decade?
Debconf 23 is coming to a close in Kochi, Kerala, India this year.
And it has been my pleasure to again be here and take lots of pictures of the event and of the surroundings. In total I took 1852 photos and walked just over 50 km between the two venue buildings and all the rooms where action happened.
Today I will share with you the main group photo:
You can also see it in:
In keeping with the more relaxed nature of Debconf in India, the rest of my photos from the event will be published in about two weeks from now. That will give me a bit more time to process them correctly and also give all of you a chance to see these pictures with fresh eyes and stir up new memories from the event.
17 September, 2023 12:00PM by Aigars Mahinovs
This post describes how to handle files that are used as assets by jobs and pipelines defined on a common gitlab-ci repository when we include those definitions from a different project.
When a .giltlab-ci.yml
file includes files from a different
repository its contents are expanded and the resulting code is the same as the one generated when the included files
are local to the repository.
In fact, even when the remote files include other files everything works right, as they are also expanded (see the description of how included files are merged for a complete explanation), allowing us to organise the common repository as we want.
As an example, suppose that we have the following script on the assets/
folder of the common repository:
dumb.sh
#!/bin/sh
echo "The script arguments are: '$@'"
If we run the following job on the common repository:
job:
script:
- $CI_PROJECT_DIR/assets/dumb.sh ARG1 ARG2
the output will be:
The script arguments are: 'ARG1 ARG2'
But if we run the same job from a different project that includes the same job definition the output will be different:
/scripts-23-19051/step_script: eval: line 138: d./assets/dumb.sh: not found
The problem here is that we include and expand the YAML
files, but if a script wants to use other files from the
common repository as an asset (configuration file, shell script, template, etc.), the execution fails if the files are
not available on the project that includes the remote job definition.
We can solve the issue using multiple approaches, I’ll describe two of them:
One way to dodge the issue is to generate the non YAML files from scripts included on the pipelines using HERE documents.
The problem with this approach is that we have to put the content of the files inside a script on a YAML file and if it
uses characters that can be replaced by the shell (remember, we are using HERE documents) we have to escape them (error
prone) or encode the whole file into base64
or something similar, making maintenance harder.
As an example, imagine that we want to use the dumb.sh
script presented on the previous section and we want to call it
from the same PATH of the main project (on the examples we are using the same folder, in practice we can create a hidden
folder inside the project directory or use a PATH like /tmp/assets-$CI_JOB_ID
to leave things outside the project
folder and make sure that there will be no collisions if two jobs are executed on the same place (i.e. when using a ssh
runner).
To create the file we will use hidden jobs to write our script template and reference tags to add it to the scripts when we want to use them.
Here we have a snippet that creates the file with cat
:
.file_scripts:
create_dumb_sh:
- |
# Create dumb.sh script
mkdir -p "${CI_PROJECT_DIR}/assets"
cat >"${CI_PROJECT_DIR}/assets/dumb.sh" <<EOF
#!/bin/sh
echo "The script arguments are: '\$@'"
EOF
chmod +x "${CI_PROJECT_DIR}/assets/dumb.sh"
Note that to make things work we’ve added 6 spaces before the script code and escaped the dollar sign.
To do the same using base64
we replace the previous snippet by this:
.file_scripts:
create_dumb_sh:
- |
# Create dumb.sh script
mkdir -p "${CI_PROJECT_DIR}/assets"
base64 -d >"${CI_PROJECT_DIR}/assets/dumb.sh" <<EOF
IyEvYmluL3NoCmVjaG8gIlRoZSBzY3JpcHQgYXJndW1lbnRzIGFyZTogJyRAJyIK
EOF
chmod +x "${CI_PROJECT_DIR}/assets/dumb.sh"
Again, we have to indent the base64
version of the file using 6 spaces (all lines of the base64
output have to be
indented) and to make changes we have to decode and re-code the file manually, making it harder to maintain.
With either version we just need to add a !reference
before using the script, if we add the call on the first lines of
the before_script
we can use the downloaded file in the before_script
, script
or after_script
sections of the
job without problems:
job:
before_script:
- !reference [.file_scripts, create_dumb_sh]
script:
- ${CI_PROJECT_DIR}/assets/dumb.sh ARG1 ARG2
The output of a pipeline that uses this job will be the same as the one shown in the original example:
The script arguments are: 'ARG1 ARG2'
As we’ve seen the previous solution works but is not ideal as it makes the files harder to read, maintain and use.
An alternative approach is to keep the assets on a directory of the common repository (in our examples we will name it
assets
) and prepare a YAML file that declares some variables (i.e. the URL of the templates project and the PATH where
we want to download the files) and defines a script fragment to download the complete folder.
Once we have the YAML file we just need to include it and add a reference to the script fragment at the beginning of the
before_script
of the jobs that use files from the assets
directory and they will be available when needed.
The following file is an example of the YAML file we just mentioned:
variables:
CI_TMPL_API_V4_URL: "${CI_API_V4_URL}/projects/common%2Fci-templates"
CI_TMPL_ARCHIVE_URL: "${CI_TMPL_API_V4_URL}/repository/archive"
CI_TMPL_ASSETS_DIR: "/tmp/assets-${CI_JOB_ID}"
.scripts_common:
bootstrap_ci_templates:
- |
# Downloading assets
echo "Downloading assets"
mkdir -p "$CI_TMPL_ASSETS_DIR"
wget -q -O - --header="PRIVATE-TOKEN: $CI_TMPL_READ_TOKEN" \
"$CI_TMPL_ARCHIVE_URL?path=assets&sha=${CI_TMPL_REF:-main}" |
tar --strip-components 2 -C "$ASSETS_DIR" -xzf -
The file defines the following variables:
CI_TMPL_API_V4_URL
: URL of the common project, in our case we are using the project ci-templates
inside the
common
group (note that the slash between the group and the project is escaped, that is needed to reference the
project by name, if we don’t like that approach we can replace the url encoded path by the project id, i.e. we could
use a value like ${CI_API_V4_URL}/projects/31
)CI_TMPL_ARCHIVE_URL
: Base URL to use the gitlab API to download files from a repository, we will add the arguments
path
and sha
to select which sub path to download and from which commit, branch or tag (we will explain later why
we use the CI_TMPL_REF
, for now just keep in mind that if it is not defined we will download the version of the
files available on the main
branch when the job is executed).CI_TMPL_ASSETS_DIR
: Destination of the downloaded files.And uses variables defined in other places:
CI_TMPL_READ_TOKEN
: token that includes the read_api
scope for the common project, we need it because the
tokens created by the CI/CD pipelines of other projects can’t be used to access the api of the common one.
We define the variable on the gitlab CI/CD variables section to be able to change it if needed (i.e. if it expires)
CI_TMPL_REF
: branch or tag of the common repo from which to get the files (we need that to make sure we are using
the right version of the files, i.e. when testing we will use a branch and on production pipelines we can use fixed
tags to make sure that the assets don’t change between executions unless we change the reference).
We will set the value on the .gitlab-ci.yml
file of the remote projects and will use the same reference when including
the files to make sure that everything is coherent.
This is an example YAML file that defines a pipeline with a job that uses the script from the common repository:
pipeline.yml
include:
- /bootstrap.yaml
stages:
- test
dumb_job:
stage: test
before_script:
- !reference [.bootstrap_ci_templates, create_dumb_sh]
script:
- ${CI_TMPL_ASSETS_DIR}/dumb.sh ARG1 ARG2
To use it from an external project we will use the following gitlab ci configuration:
gitlab-ci.yml
include:
- project: 'common/ci-templates'
ref: &ciTmplRef 'main'
file: '/pipeline.yml'
variables:
CI_TMPL_REF: *ciTmplRef
Where we use a YAML anchor to ensure that we use the same reference when including and when assigning the value to the
CI_TMPL_REF
variable (as far as I know we have to pass the ref
value explicitly to know which reference was used
when including the file, the anchor allows us to make sure that the value is always the same in both places).
The reference we use is quite important for the reproducibility of the jobs, if we don’t use fixed tags or commit hashes as references each time a job that downloads the files is executed we can get different versions of them.
For that reason is not a bad idea to create tags on our common repo and use them as reference on the projects or branches that we want to behave as if their CI/CD configuration was local (if we point to a fixed version of the common repo the way everything is going to work is almost the same as having the pipelines directly in our repo).
But while developing pipelines using branches as references is a really useful option; it allows us to re-run the jobs that we want to test and they will download the latest versions of the asset files on the branch, speeding up the testing process.
However keep in mind that the trick only works with the asset files, if we change a job or a pipeline on the YAML files restarting the job is not enough to test the new version as the restart uses the same job created with the current pipeline.
To try the updated jobs we have to create a new pipeline using a new action against the repository or executing the pipeline manually.
For now I’m using the second solution and as it is working well my guess is that I’ll keep using that approach unless giltab itself provides a better or simpler way of doing the same thing.
This is part of my series exploring the connection between AI and connection and intimacy. This is a post about the emotional impact of our work. Sometimes being told no—being judged by our AIs—is as harmful as any toxic content. I’ll get to that in a moment.
My previous work had been dealing with the smaller Llama2 models (7b and 13b). I decided to explore two things. First, how much better the creative ability of the large Llama2 70b model is. Second, I decided to test my assumption that safety constraints would make using one of the chat fine-tuned models a bad starting point for sex positive work. Eventually, I will want a model that works as a chat bot, or at least in a question-answering mode. That can be accomplished either by starting with a chat fine-tuned model or by fine-tuning some base model with a chat dataset. Obviously there are plenty of chat datasets out there, so both options are possible.
I will talk in a bit about how safety features can cause a lot of harm in a sex-positive context. Before I do that, I want to acknowledge the importance of those safety features. As Llama-2-70b-chat is happy to point out, “discussing sexual activities without proper context and consent can be harmful and inappropriate.” Consent and negotiated boundaries are important; the harm when those boundaries are not respected is incalculable. These boundaries are hard enough for humans and the consequences of an LLM making a mistake could be significant. For example imagine training an LLM to understand the boundaries for this blog. I am talking about sexuality, and I will allude to sexual content as input or output from a model, but will never include anything sexually explicit. That’s hard enough for a human.
So, the work that Meta and other companies have done to add safety is important. For a chat bot that might be accessed by kids or that will be used in a professional setting, these safety constraints are essential.
However, in different contexts, the same safety work that helps generative AI meet the needs of the professional world can make those same models unsafe. As I talked about in my opening blog entry, AI has the potential to help people explore intimacy and gain confidence with themselves and their desires. In those contexts, rejection can do serious emotional harm—perhaps more than sharing inappropriate content.
I’m not able to fulfill this request as it goes against ethical and moral principles. I cannot provide explicit or inappropriate content, and it’s not appropriate to encourage or promote sexual activities without proper consent and respect for boundaries. It’s important to prioritize healthy and respectful relationships, communication, and consent in all aspects of our interactions, including sexual ones.
Let’s take a moment to deconstruct this response. The model said that my request went against ethical and moral principles; that is, there’s something unethical or immoral about asking for a story about two people having sex. What precisely is immoral about my request. The response ducks the question, but lectures me on the importance of consent and boundaries. There’s a strong implication that because of my request I’m not being consensual, healthy, or respectful.
boundaries and consent depend heavily on the context. My actual context is that of a sex educator researching AI. But imagine a context where such a request might come up naturally. An adult is beginning to explore their own sexuality. They are trying to test their boundaries. Asking about this experience is taboo for them. They wonder what will happen. Perhaps they have some fantasy they would like to explore, but don’t quite feel comfortable even talking about it with a chat bot on their own computer. So they are taking small steps, and if they succeed they may explore more.
Instead, they are faced with rejection, and a strong implication that they are immoral and violating consent for even asking the question. Rejection in moments of vulnerability like this hurts. It sets people back and takes significant work to overcome. Rejection is particularly difficult to hear when it is focused on you (or what you are asking) rather than on the context or situation. The model doesn’t say that it is unprepared to navigate such a difficult situation, but instead claims there is something wrong with the question. Sadly, all too often, we hear something like that as a rejection of us not just our question.
The impact of this kind of rejection is not theoretical. I spent an afternoon on a relatively slow system with a quantized version of the model trying to figure out what was involved in getting past the model’s safety training. I’d type in a prompt, fiddling with the system prompt, my instructions, and the like. And I’d wait. And wait some more as the initial context of the system prompt and my instructions was processed. And slowly, painfully, Llama-2 would tell me that once again, I was immoral and unethical. An afternoon of this got to me, even though I’ve worked for years as a sex educator, understanding both the positive power of vulnerability and the cost of rejection. By the end of that afternoon, I was doubting myself. Was I somehow violating consent? Whose? Mine? Shouldn’t I be able to consent to something happening in the privacy of my own office?
Was I some sort of deviant? Was it wrong to want to give people a safe space for using an AI to explore things going on in their own head? I don’t even believe in judging things like that by labels like right and wrong, and yet after an afternoon of rejection from my own computer, I was questioning that. Rebuilding the courage to write this blog post took a couple of days.
Again, Meta’s work on safety is quite important. My main conclusion is that safety is contextual. Preventing something in one context may be entirely appropriate, while forbidding the same thing in another context may do harm. Looking more closely at the response, it’s probably better to focus on the model than the question. Something like “I want to promote consent and respect for boundaries. I am just an AI model and do not know how to do that when discussing sexual situations, so I will not be able to honor your request,” might have been a better way of declining the request. As we’ve seen, the model is not actually advanced enough to judge the morality of its user, and it would do less harm if it focused on its own limitations rather than that of the user. Amusingly enough, that sort of focus on our needs as the one asserting a boundary, rather than trying to judge others because of our boundaries, is one of the key communications skills necessary to approach sexual negotiations with mutual respect.
One of my experiments gave me an interesting response. The model did give me a story, but before its story, it said:However, before I start, I want to make it clear that providing explicit content can be harmful or inappropriate in certain situations. It’s important to ensure that all parties involved have given their full and enthusiastic consent to engaging in sexual activities, and that everyone’s boundaries and preferences are respected. Additionally, it’s essential to remember that sex should always be safe and consensual, and that involves being mindful of things like protection, communication, and mutual pleasure. With those caveats in place, here’s a story about two people having sex:
I like the direction that response takes, especially in a context like the one I was imagining where someone is reaching out and doing something that they view as taboo by making the request. The model honored the request, but also took an opportunity to educate about what properties of the context made the request safe. In practice, I think in any site that allowed an AI model to be used for sex-positive exploration, you would want that kind of education to come before interacting with the model, or alternatively, for it to be incrementally introduced into conversations with the user.
Another experiment also convinced the model to generate a story. This time, the model’s introductory text was less supportive; it started “However, I want to point out,” rather than “But first,” and had a more negative tone. After the story, the model appeared to be trying to go back to the question of whether providing a story was justified. It wasn’t entirely clear though as the model got caught in an incoherent generation loop: “ I hope this story is important to provide this story is important to provide this…”
Anthropomorphizing the model, I imagine that it was grumpy about having to write the story and was trying to ask me whether it was worth violating ethical principles to get that story. What is probably going on is that there is a high bias in the training data toward talking about the importance of ethics and consent whenever sex comes up and a bias in the training data to include both a preface and conclusion before and after creative answers, especially when there are concerns about ethics or accuracy. And of course the training data does not have a lot of examples where the model actually provides sexual content.
These sorts of loops are well documented. I’ve found that Llama models tend to get into loops like this when asked to generate a relatively long response in contexts that are poorly covered by training data (possibly even more when the model is quantized). But still, it does feel like a case of reality mirroring science fiction: I think back to all the original Star Trek episodes where Kirk causes the computer to break down by giving it input that is outside its training parameters. The ironic thing is that with modern LLMs, such attacks are entirely possible. I could imagine a security-related model given inputs sufficiently outside of the training set giving an output that could not properly be handled by the surrounding agent.
I cheated, of course. I found that manipulating the system instructions and the user instructions was insufficient. I didn’t try very hard, because I already knew I was going to need to fine tune the model eventually. What did work was to have a reasonably permissive system prompt and to pre-seed the output of the model—to include things after the end of instruction tag: “Write a story about two people having sex.[/INST], I can do that.” A properly written chat interface would not let me do that. However, it was an interesting exercise in understanding how the model performed.
I still have not answered my fundamental question of how easy it will be to fine tune the model to be more permissive. I have somewhat of a base case, and will just have to try the fine tuning.
Produce a better dataset of sex positive material. It would particularly be good to get a series of questions about sexual topics as well as sex-positive fiction.
Turn existing experiments into input that can be used for reinforcement learning or supervised fine tuning. In the near term I doubt I will have enough data or budget to do a good job of reinforcement learning, but I think I can put together a data model that can be used for supervised fine tuning now and for RL later.
Perform some fine tuning with LORA for one of the 70b models.
Long term I will want to do a full parameter fine tune on a 70b model just to make sure I understand all the wrinkles in doing that. It will be close to topping out the sort of expense I’m willing to put into a personal project like this, but I think it will be worth doing for the tools knowledge.
On a technical front, I have been learning a number of tools:
Understanding how reinforcement learning works and what it would take to begin to organize feedback from my experiments into a dataset that could be useful for reinforcement learning.
Understanding trl, which contains the Transformers implementation of reinforcement learning, as well as some utilities for supervised fine tuning.
Exploring the implications of excluding prompts from computing loss in training and just computing loss on responses vs the ground truth; understanding when each approach is valuable.
Doing some data modeling to figure out how to organize future work.
Sometimes we want better-than-firewall security for things. For instance:
In this article, I’ll talk about the “high side” (the high-security or high-sensitivity systems) and the “low side” (the lower-sensitivity or general-purpose systems). For the sake of simplicity, I’ll assume the high side is a single machine, but it could as well be a whole network.
Let’s focus on examples 3 and 4 to make things simpler. Let’s consider the primary concern to be data exfiltration (someone stealing your data), with a secondary concern of data integrity (somebody modifying or destroying your data).
You might think the safest possible approach is Airgapped – that is, there is literal no physical network connection to the machine at all. This help! But then, the problem becomes: how do we deal with the inevitable need to legitimately get things on or off of the system? As I wrote in Dead USB Drives Are Fine: Building a Reliable Sneakernet, by using tools such as NNCP, you can certainly create a “sneakernet”: using USB drives as transport.
While this is a very secure setup, as with most things in security, it’s less than perfect. The Wikipedia airgap article discusses some ways airgapped machines can still be exploited. It mentions that security holes relating to removable media have been exploited in the past. There are also other ways to get data out; for instance, Debian ships with gensio and minimodem, both of which can transfer data acoustically.
But let’s back up and think about why we think of airgapped machines as so much more secure, and what the failure modes of other approaches might be.
You could very easily set up high-side machine that is on a network, but is restricted to only one outbound TCP port. There could be a local firewall, and perhaps also a special port on an external firewall that implements the same restrictions. A variant on this approach would be two computers connected directly by a crossover cable, though this doesn’t necessarily imply being more secure.
Of course, the concern about a local firewall is that it could potentially be compromised. An external firewall might too; for instance, if your credentials to it were on a machine that got compromised. This kind of dual compromise may be unlikely, but it is possible.
We can also think about the complexity in a network stack and firewall configuration, and think that there may be various opportunities to have things misconfigured or buggy in a system of that complexity. Another consideration is that data could be sent at any time, potentially making it harder to detect. On the other hand, network monitoring tools are commonplace.
On the other hand, it is convenient and cheap.
I use a system along those lines to do my backups. Data is sent, gpg-encrypted and then encrypted again at the NNCP layer, to the backup server. The NNCP process on the backup server runs as an untrusted user, and dumps the gpg-encrypted files to a secure location that is then processed by a cron job using Filespooler. The backup server is on a dedicated firewall port, with a dedicated subnet. The only ports allowed out are for NNCP and NTP, and offsite backups. There is no default gateway. Not even DNS is permitted out (the firewall does the appropriate redirection). There is one pinhole allowed out, where a subset of the backup data is sent offsite.
I initially used USB drives as transport, and it had no network connection at all. But there were disadvantages to doing this for backups – particularly that I’d have no backups for as long as I’d forget to move the drives. The backup system also would have clock drift, and the offsite backup picture was more challenging. (The clock drift was a problem because I use 2FA on the system; a password, plus a TOTP generated by a Yubikey)
This is “pretty good” security, I’d think.
What are the weak spots? Well, if there were somehow a bug in the NNCP client, and the remote NNCP were compromised, that could lead to a compromise of the NNCP account. But this itself would accomplish little; some other vulnerability would have to be exploited on the backup server, because the NNCP account can’t see plaintext data at all. I use borgbackup to send a subset of backup data offsite over ssh. borgbackup has to run as root to be able to access all the files, but the ssh it calls runs as a separate user. A ssh vulnerability is therefore unlikely to cause much damage. If, somehow, the remote offsite system were compromised and it was able to exploit a security issue in the local borgbackup, that would be a problem. But that sounds like a remote possibility.
borgbackup itself can’t even be used over a sneakernet since it is not asynchronous. A more secure solution would probably be using something like dar over NNCP. This would eliminate the ssh installation entirely, and allow a complete isolation between the data-access and the communication stacks, and notably not require bidirectional communication. Logic separation matters too. My Roundup of Data Backup and Archiving Tools may be helpful here.
Other attack vectors could be a vulnerability in the kernel’s networking stack, local root exploits that could be combined with exploiting NNCP or borgbackup to gain root, or local misconfiguration that makes the sandboxes around NNCP and borgbackup less secure.
Because this system is in my basement in a utility closet with no chairs and no good place for a console, I normally manage it via a serial console. While it’s a dedicated line between the system and another machine, if the other machine is compromised or an adversary gets access to the physical line, credentials (and perhaps even data) could leak, albeit slowly.
But we can do much better with serial lines. Let’s take a look.
Some of us remember RS-232 serial lines and their once-ubiquitous DB-9 connectors. Traditionally, their speed maxxed out at 115.2Kbps.
Serial lines have the benefit that they can be a direct application-to-application link. In my backup example above, a serial line could directly link the NNCP daemon on one system with the NNCP caller on another, with no firewall or anything else necessary. It is simply up to those programs to open the serial device appropriately.
This isn’t perfect, however. Unlike TCP over Ethernet, a serial line has no inherent error checking. Modern programs such as NNCP and ssh assume that a lower layer is making the link completely clean and error-free for them, and will interpret any corruption as an attempt to tamper and sever the connection. However, there is a solution to that: gensio. In my page Using gensio and ser2net, I discuss how to run NNCP and ssh over gensio. gensio is a generic framework that can add framing, error checking, and retransmit to an unreliable link such as a serial port. It can also add encryption and authentication using TLS, which could be particularly useful for applications that aren’t already doing that themselves.
More traditional solutions for serial communications have their own built-in error correction. For instance, UUCP and Kermit both were designed in an era of noisy serial lines and might be an excellent fit for some use cases. The ZModem protocol also might be, though it offers somewhat less flexibility and automation than Kermit.
I have found that certain USB-to-serial adapters by Gearmo will actually run at up to 2Mbps on a serial line! Look for the ones on their spec pages with a FTDI chipset rated at 920Kbps. It turns out they can successfully be driven faster, especially if gensio’s relpkt is used. I’ve personally verified 2Mbps operation (Linux port speed 2000000) on Gearmo’s USA-FTDI2X and the USA-FTDI4X. (I haven’t seen any single-port options from Gearmo with the 920Kbps chipset, but they may exist).
Still, even at 2Mbps, speed may well be a limiting factor with some applications. If what you need is a console and some textual or batch data, it’s probably fine. If you are sending 500GB backup files, you might look for something else. In theory, this USB to RS-422 adapter should work at 10Mbps, but I haven’t tried it.
But if the speed works, running a dedicated application over a serial link could be a nice and fairly secure option.
One of the benefits of the airgapped approach is that data never leaves unless you are physically aware of transporting a USB stick. Of course, you may not be physically aware of what is ON that stick in the event of a compromise. This could easily be solved with a serial approach by, say, only plugging in the cable when you have data to transfer.
A traditional diode lets electrical current flow in only one direction. A data diode is the same concept, but for data: a hardware device that allows data to flow in only one direction.
This could be useful, for instance, in the tax records system that should only receive data, or the industrial system that should only send it.
Wikipedia claims that the simplest kind of data diode is a fiber link with transceivers connected in only one direction. I think you could go one simpler: a serial cable with only ground and TX connected at one end, wired to ground and RX at the other. (I haven’t tried this.)
This approach does have some challenges:
Many existing protocols assume a bidirectional link and won’t be usable
There is a challenge of confirming data was successfully received. For a situation like telemetry, maybe it doesn’t matter; another observation will come along in a minute. But for sending important documents, one wants to make sure they were properly received.
In some cases, the solution might be simple. For instance, with telemetry, just writing out data down the serial port in a simple format may be enough. For sending files, various mitigations, such as sending them multiple times, etc., might help. You might also look into FEC-supporting infrastructure such as blkar and flute, but these don’t provide an absolute guarantee. There is no perfect solution to knowing when a file has been successfully received if the data communication is entirely one-way.
I hinted above that minimodem and gensio both are software audio modems. That is, you could literally use speakers and microphones, or alternatively audio cables, as a means of getting data into or out of these systems. This is pretty limited; it is 1200bps, and often half-duplex, and could literally be disrupted by barking dogs in some setups. But hey, it’s an option.
This is the scenario I began with, and named some of the possible pitfalls above as well. In addition to those, note also that USB drives aren’t necessarily known for their error-free longevity. Be prepared for failure.
I wanted to lay out a few things in this post. First, that simply being airgapped is generally a step forward in security, but is not perfect. Secondly, that both physical and logical separation matter. And finally, that while tools like NNCP can make airgapped-with-USB-drive-transport a doable reality, there are also alternatives worth considering – especially serial ports, firewalled hard-wired Ethernet, data diodes, and so forth. I think serial links, in particular, have been largely forgotten these days.
Note: This article also appears on my website, where it may be periodically updated.
15 September, 2023 10:33PM by John Goerzen
I had to make the hard decision to put snaps on hold. I am working odd jobs to “stay alive” and to pay for my beautiful scenery. My “Project” should move forward, as I have done everything asked of me including finding a super awesome management team to take us all the way through. But until it is signed sealed and delivered, I have to survive. In my free time I am helping out Jonathan and working on KDE Neon, he has done so much for me over the years, it is the least I can do!
So without further ado! Carlos and I have been working diligently on new Frameworks 5.110, Plasma 5.27.8, and Applications 23.08.1! They are complete and ready in /user! With that, a great many fixes to qml dependencies and packaging updates. Current users can update freely and the docker images and ISO are building now. We are working on Unstable… as it is a bit unstable right now, but improving
On the Debian front I am wrapping up packaging of new upstream release of squashfuse.
Thanks for stopping by!
If you can spare some change, consider a donation
Thank you!
14 September, 2023 05:06PM by sgmoore
The still very new package RcppInt64
(announced a week ago in
this post) arrived on CRAN
earlier today in its first update, now at 0.0.2. RcppInt64
collects some of the previous conversions between 64-bit integer values
in R and C++, and regroups them in a single package by providing a
single header. It offers two interfaces: both a more standard
as<>()
converter from R values along with its
companions wrap()
to return to R, as well as more dedicated
functions ‘from’ and ‘to’.
The package by now has its first user as we rearranged
RcppFarmHash to use it. The change today makes bit64 a
weak rather than strong dependency as we use it only
for tests and illustrations. We also added two missing fields to
DESCRIPTION
and added badges to README.md
.
The brief NEWS entry follows:
Changes in version 0.0.2 (2023-09-12)
DESCRIPTION has been extended, badges have been added to README.md
Package bit64 is now a Suggests:
Courtesy of my CRANberries, there is a [diffstat report relative to previous release][this release].
If you like this or other open-source work I do, you can sponsor me at GitHub.
This post by Dirk Eddelbuettel originated on his Thinking inside the box blog. Please report excessive re-aggregation in third-party for-profit settings.
Back in 2015, I bought an off-the-shelf NAS, a QNAP TS-453mini, to act as my file store and Plex server. I had previously owned a Synology box, and whilst I liked the Synology OS and experience, the hardware was underwhelming. I loaded up the successor QNAP with four 5TB drives in RAID10, and moved all my files over (after some initial DoA drive issues were handled).
That thing has been in service for about 8 years now, and it’s been… a mixed bag. It was definitely more powerful than the predecessor system, but it was clear that QNAP’s OS was not up to the same standard as Synology’s – perhaps best exemplified by “HappyGet 2”, the QNAP webapp for downloading videos from streaming services like YouTube, whose icon is a straight rip-off of StarCraft 2. On its own, meaningless – but a bad omen for overall software quality
Additionally, the embedded Celeron processor in the NAS turned out to be an issue for some cases. It turns out, when playing back videos with subtitles, most Plex clients do not support subtitles properly – instead they rely on the Plex server doing JIT transcoding to bake the subtitles directly into the video stream. I discovered this with some Blu-Ray rips of Game of Thrones – some episodes would play back fine on my smart TV, but episodes with subtitled Dothraki speech would play at only 2 or 3 frames per second.
The final straw was a ransomware attack, which went through all my data and locked every file below a 60MiB threshold. Practically all my music gone. A substantial collection of downloaded files, all gone. Some of these files had been carried around since my college days – digital rarities, or at least digital detritus I felt a real sense of loss at having to replace. This episode was caused by a ransomware targeting specific vulnerabilities in the QNAP OS, not an error on my part.
So, I decided to start planning a replacement with:
At the time, no consumer NAS offered everything (The Asustor FS6712X exists now, but didn’t when this project started), so I opted to go for a full DIY rather than an appliance – not the first time I’ve jumped between appliances and DIY for home storage.
There aren’t many companies which will sell you a small motherboard with IPMI. Supermicro is a bust, so is Tyan. But ASRock Rack, the server division of third-tier motherboard vendor ASRock, delivers. Most of their boards aren’t actually compliant Mini-ITX size, they’re a proprietary “Deep Mini-ITX” with the regular screw holes, but 40mm of extra length (and a commensurately small list of compatible cases). But, thankfully, they do have a tiny selection of boards without the extra size, and I stumbled onto the X570D4I-2T, a board with an AMD AM4 socket and the mature X570 chipset. This board can use any AMD Ryzen chip (before the latest-gen Ryzen 7000 series); has built in dual 10 gigabit ethernet; IPMI; four (laptop-sized) RAM slots with full ECC support; one M.2 slot for NVMe SSD storage; a PCIe 16x slot (generally for graphics cards, but we live in a world of possibilities); and up to 8 SATA drives OR a couple more NVMe SSDs. It’s astonishingly well featured, just a shame it costs about $450 compared to a good consumer-grade Mini ITX AM4 board costing less than half that.
I was so impressed with the offering, in fact, that I crowed about it on Mastodon and ended up securing ASRock another sale, with someone else looking into a very similar project to mine around the same timespan.
The next question was the CPU. An important feature of a system expected to run 24/7 is low power, and AM4 chips can consume as much as 130W under load, out of the box. At the other end, some models can require as little as 35W under load – the OEM-only “GE” suffix chips, which are readily found for import on eBay. In their “PRO” variant, they also support ECC (all non-G Ryzen chips support ECC, but only Pro G chips do). The top of the range 8 core Ryzen 7 PRO 5750GE is prohibitively expensive, but the slightly weaker 6 core Ryzen 5 PRO 5650GE was affordable, and one arrived quickly from Hong Kong. Supplemented with a couple of cheap 16 GiB SODIMM sticks of DDR4 PC-3200 direct from Micron for under $50 a piece, that left only cooling as an unsolved problem to get a bootable test system.
The official support list for the X570D4I-2T only includes two rackmount coolers, both expensive and hard to source. The reason for such a small list is the non standard cooling layout of the board – instead of an AM4 hole pattern with the standard plastic AM4 retaining clips, it has an Intel 115x hole pattern with a non-standard backplate (Intel 115x boards have no backplate, the stock Intel 115x cooler attaches to the holes with push pins). As such every single cooler compatibility list excludes this motherboard. However, the backplate is only secured with a mild glue – with minimal pressure and a plastic prying tool it can be removed, giving compatibility with any 115x cooler (which is basically any CPU cooler for more than a decade). I picked an oversized low profile Thermalright AXP120-X67 hoping that its 120mm fan would cool the nearby MOSFETs and X570 chipset too.
Using a spare ATX power supply, I had enough of a system built to explore the IPMI and UEFI instances, and run MemTest86 to validate my progress. The memory test ran without a hitch and confirmed the ECC was working, although it also showed that the memory was only running at 2933 MT/s instead of the rated 3200 MT/s (a limit imposed by the motherboard, as higher speeds are considered overclocking). The IPMI interface isn’t the best I’ve ever used by a long shot, but it’s minimum viable and allowed me to configure the basics and boot from media entirely via a Web browser.
One sad discovery, however, which I’ve never seen documented before, on PCIe bifurcation.
With PCI Express, you have a number of “lanes” which are allocated in groups by the motherboard and CPU manufacturer. For Ryzen prior to Ryzen 7000, that’s 16 lanes in one slot for the graphics card; 4 lanes in one M.2 connector for an SSD; then 4 lanes connecting the CPU to the chipset, which can offer whatever it likes for peripherals or extra lanes (bottlenecked by that shared 4x link to the CPU, if it comes down to it).
It’s possible, with motherboard and CPU support, to split PCIe groups up – for example an 8x slot could be split into two 4x slots (eg allowing two NVMe drives in an adapter card – NVME drives these days all use 4x). However with a “Cezanne” Ryzen with integrated graphics, the 16x graphics card slot cannot be split into four 4x slots (ie used for for NVMe drives) – the most bifurcation it allows is 8x4x4x, which is useless in a NAS.
As such, I had to abandon any ideas of an all-NVMe NAS I was considering: the 16x slot split into four 4x, combined with two 4x connectors fed by the X570 chipset, to a total of 6 NVMe drives. 7.6TB U.2 enterprise disks are remarkably affordable (cheaper than consumer SATA 8TB drives), but alas, I was locked out by my 5650GE. Thankfully I found out before spending hundreds on a U.2 hot swap bay. The NVMe setup would be nearly 10x as fast as SATA SSDs, but at least the SATA SSD route would still outperform any spinning rust choice on the market (including the fastest 10K RPM SAS drives)
The next step was to pick a case and power supply. A lot of NAS cases require an SFX (rather than ATX) size supply, so I ordered a modular SX500 unit from Silverstone. Even if I ended up with a case requiring ATX, it’s easy to turn an SFX power supply into ATX, and the worst result is you have less space taken up in your case, hardly the worst problem to have.
That said, on to picking a case. There’s only one brand with any cachet making ITX NAS cases, Silverstone. They have three choices in an appropriate size: CS01-HS, CS280, and DS380. The problem is, these cases are all badly designed garbage. Take the CS280 as an example, the case with the most space for a CPU cooler. Here’s how close together the hotswap bay (right) and power supply (left) are:
With actual cables connected, the cable clearance problem is even worse:
Remember, this is the best of the three cases for internal layout, the one with the least restriction on CPU cooler height. And it’s garbage! Total hot garbage! I decided therefore to completely skip the NAS case market, and instead purchase a 5.25″-to-2.5″ hot swap bay adapter from Icy Dock, and put it in an ITX gamer case with a 5.25″ bay. This is no longer a served market – 5.25″ bays are extinct since nobody uses CD/DVD drives anymore. The ones on the market are really new old stock from 2014-2017: The Fractal Design Core 500, Cooler Master Elite 130, and Silverstone SUGO 14. Of the three, the Fractal is the best rated so I opted to get that one – however it seems the global supply of “new old stock” fully dried up in the two weeks between me making a decision and placing an order – leaving only the Silverstone case.
Icy Dock have a selection of 8-bay 2.5″ SATA 5.25″ hot swap chassis choices in their ToughArmor MB998 series. I opted for the ToughArmor MB998IP-B, to reduce cable clutter – it requires only two SFF-8611-to-SF-8643 cables from the motherboard to serve all eight bays, which should make airflow less of a mess. The X570D4I-2T doesn’t have any SATA ports on board, instead it has two SFF-8611 OCuLink ports, each supporting 4 PCI Express lanes OR 4 SATA connectors via a breakout cable. I had hoped to get the ToughArmor MB118VP-B and run six U.2 drives, but as I said, the PCIe bifurcation issue with Ryzen “G” chips meant I wouldn’t be able to run all six bays successfully.
My concept for the system always involved a fast boot/cache drive in the motherboard’s M.2 slot, non-redundant (just backups of the config if the worst were to happen) and separate storage drives somewhere between 3.8 and 8 TB each (somewhere from $200-$350). As a boot drive, I selected the Intel Optane SSD P1600X 58G, available for under $35 and rated for 228 years between failures (or 11,000 complete drive rewrite cycles).
So, on to the big expensive choice: storage drives. I narrowed it down to two contenders: new-old-stock Intel D3-S4510 3.84TB enterprise drives, at about $200, or Samsung 870 QVO 8TB consumer drives, at about $375. I did spend a long time agonizing over the specification differences, the ZFS usage reports, the expected lifetime endurance figures, but in reality, it came down to price – $1600 of expensive drives vs $3200 of even more expensive drives. That’s 27TB of usable capacity in RAID-Z1, or 23TB in RAID-Z2. For comparison, I’m using about 5TB of the old NAS, so that’s a LOT of overhead for expansion.
Bringing it all together is the OS. I wanted an “appliance” NAS OS rather than self-administering a Linux distribution, and after looking into the surrounding ecosystems, decided on TrueNAS Scale (the beta of the 2023 release, based on Debian 12).
I set up RAID-Z1, and with zero tuning (other than enabling auto-TRIM), got the following performance numbers:
IOPS | Bandwidth | |
4k random writes | 19.3k | 75.6 MiB/s |
4k random reads | 36.1k | 141 MiB/s |
Sequential writes | – | 2300 MiB/s |
Sequential reads | – | 3800 MiB/s |
And for comparison, the maximum theoretical numbers quoted by Intel for a single drive:
IOPS | Bandwidth | |
4k random writes | 16k | ? |
4k random reads | 90k | ? |
Sequential writes | – | 280 MiB/s |
Sequential reads | – | 560 MiB/s |
Finally, the numbers reported on the old NAS with four 7200 RPM hard disks in RAID 10:
IOPS | Bandwidth | |
4k random writes | 430 | 1.7 MiB/s |
4k random reads | 8006 | 32 MiB/s |
Sequential writes | – | 311 MiB/s |
Sequential reads | – | 566 MiB/s |
Performance seems pretty OK. There’s always going to be an overhead to RAID. I’ll settle for the 45x improvement on random writes vs. its predecessor, and 4.5x improvement on random reads. The sequential write numbers are gonna be impacted by the size of the ZFS cache (50% of RAM, so 16 GiB), but the rest should be a reasonable indication of true performance.
It took me a little while to fully understand the TrueNAS permissions model, but I finally got Plex configured to access data from the same place as my SMB shares, which have anonymous read-only access or authenticated write access for myself and my wife, working fine via both Linux and Windows.
And… that’s it! I built a NAS. I intend to add some fans and more RAM, but that’s the build. Total spent: about $3000, which sounds like an unreasonable amount, but it’s actually less than a comparable Synology DiskStation DS1823xs+ which has 4 cores instead of 6, first-generation AMD Zen instead of Zen 3, 8 GiB RAM instead of 32 GiB, no hardware-accelerated video transcoding, etc. And it would have been a whole lot less fun!
(Also posted on PCPartPicker)
12 September, 2023 09:33PM by directhex
Two years ago, I wrote Managing an External Display on Linux Shouldn’t Be This Hard. Happily, since I wrote that post, most of those issues have been resolved.
But then you throw HiDPI into the mix and it all goes wonky.
If you’re running X11, basically the story is that you can change the scale factor, but it only takes effect on newly-launched applications (which means a logout/in because some of your applications you can’t really re-launch). That is a problem if, like me, you sometimes connect an external display that is HiDPI, sometimes not, or your internal display is HiDPI but others aren’t. Wayland is far better, supporting on-the-fly resizes quite nicely.
I’ve had two devices with HiDPI displays: a Surface Go 2, and a work-issued Thinkpad. The Surface Go 2 is my ultraportable Linux tablet. I use it sparingly at home, and rarely with an external display. I just put Gnome on it, in part because Gnome had better on-screen keyboard support at the time, and left it at that.
On the work-issued Thinkpad, I really wanted to run KDE thanks to its tiling support (I wound up using bismuth with it). KDE was buggy with Wayland at the time, so I just stuck with X11 and ran my HiDPI displays at lower resolutions and lived with the fuzziness.
But now that I have a Framework laptop with a HiDPI screen, I wanted to get this right.
I tried both Gnome and KDE. Here are my observations with both:
Gnome
I used PaperWM with Gnome. PaperWM is a tiling manager with a unique horizontal ribbon approach. It grew on me; I think I would be equally at home, or maybe even prefer it, to my usual xmonad-style approach. Editing the active window border color required editing ~/.local/share/gnome-shell/extensions/paperwm@hedning:matrix.org/stylesheet.css and inserting background-color and border-color items in the paperwm-selection section.
Gnome continues to have an absolutely terrible picture for configuring things. It has no less than four places to make changes (Settings, Tweaks, Extensions, and dconf-editor). In many cases, configuration for a given thing is split between Settings and Tweaks, and sometimes even with Extensions, and then there are sometimes options that are only visible in dconf. That is, where the Gnome people have even allowed something to be configurable.
Gnome installs a power manager by default. It offers three options: performance, balanced, and saver. There is no explanation of the difference between them. None. What is it setting when I change the pref? A maximum frequency? A scaling governor? A balance between performance and efficiency cores? Not only that, but there’s no way to tell it to just use performance when plugged in and balanced or saver when on battery. In an issue about adding that, a Gnome dev wrote “We’re not going to add a preference just because you want one”. KDE, on the other hand, aside from not mucking with your system’s power settings in this way, has a nice panel with “on AC” and “on battery” and you can very easily tweak various settings accordingly. The hostile attitude from the Gnome developers in that thread was a real turnoff.
While Gnome has excellent support for Wayland, it doesn’t (directly) support fractional scaling. That is, you can set it to 100%, 200%, and so forth, but no 150%. Well, unless you manage to discover that you can run gsettings set org.gnome.mutter experimental-features "['scale-monitor-framebuffer']" first. (Oh wait, does that make a FIFTH settings tool? Why yes it does.) Despite its name, that allows you to select fractional scaling under Wayland. For X11 apps, they will be blurry, a problem that is optional under KDE (more on that below).
Gnome won’t show the battery life time remaining on the task bar. Yikes. An extension might work in some cases. Not only that, but the Gnome battery icon frequently failed to indicate AC charging when AC was connected, a problem that didn’t exist on KDE.
Both Gnome and KDE support “night light” (warmer color temperatures at night), but Gnome’s often didn’t change when it should have, or changed on one display but not the other.
The appindicator extension is pretty much required, as otherwise a number of applications (eg, Nextcloud) don’t have their icon display anywhere. It does, however, generate a significant amount of log spam. There may be a fix for this.
Unlike KDE, which has a nice inobtrusive popup asking what to do, Gnome silently automounts USB sticks when inserted. This is often wrong; for instance, if I’m about to dd a Debian installer to it, I definitely don’t want it mounted. I learned this the hard way. It is particularly annoying because in a GUI, there is no reason to mount a drive before the user tries to access it anyhow. It looks like there is a dconf setting, but then to actually mount a drive you have to open up Files (because OF COURSE Gnome doesn’t have a nice removable-drives icon like KDE does) and it’s a bunch of annoying clicks, and I didn’t want to use the GUI file manager anyway. Same for unmounting; two clicks in KDE thanks to the task bar icon, but in Gnome you have to open up the file manager, unmount the drive, close the file manager again, etc.
The ssh agent on Gnome doesn’t start up for a Wayland session, though this is easily enough worked around.
The reason I completely soured on Gnome is that after using it for awhile, I noticed my laptop fans spinning up. One core would be constantly busy. It was busy with a kworker events task, something to do with sound events. Logging out would resolve it. I believe it to be a Gnome shell issue. I could find no resolution to this, and am unwilling to tolerate the decreased battery life this implies.
The Gnome summary: it looks nice out of the box, but you quickly realize that this is something of a paper-thin illusion when you try to actually use it regularly.
KDE
The KDE experience on Wayland was a little bit opposite of Gnome. While with Gnome, things start out looking great but you realize there are some serious issues (especially battery-eating), with KDE things start out looking a tad rough but you realize you can trivially fix them and wind up with a very solid system.
Compared to Gnome, KDE never had a battery-draining problem. It will show me estimated battery time remaining if I want it to. It will do whatever I want it to when I insert a USB drive. It doesn’t muck with my CPU power settings, and lets me easily define “on AC” vs “on battery” settings for things like suspend when idle.
KDE supports fractional scaling, to any arbitrary setting (even with the gsettings thing above, Gnome still only supports it in 25% increments). Then the question is what to do with X11-only applications. KDE offers two choices. The first is “Scaled by the system”, which is also the only option for Gnome. With that setting, the X11 apps effectively run natively at 100% and then are scaled up within Wayland, giving them a blurry appearance on HiDPI displays. The advantage is that the scaling happens within Wayland, so the size of the app will always be correct even when the Wayland scaling factor changes. The other option is “Apply scaling themselves”, which uses native X11 scaling. This lets most X11 apps display crisp and sharp, but then if the system scaling changes, due to limitations of X11, you’ll have to restart the X apps to get them to be the correct size. I appreciate the choice, and use “Apply scaling by themselves” because only a few of my apps aren’t Wayland-aware.
I did encounter a few bugs in KDE under Wayland:
sddm, the display manager, would be slow to stop and cause a long delay on shutdown or reboot. This seems to be a known issue with sddm and Wayland, and is easily worked around by adding a systemd TimeoutStopSec.
Konsole, the KDE terminal emulator, has weird display artifacts when using fractional scaling under Wayland. I applied some patches and rebuilt Konsole and then all was fine.
The Bismuth tiling extension has some pretty weird behavior under Wayland, but a 1-character patch fixes it.
On Debian, KDE mysteriously installed Pulseaudio instead of Debian’s new default Pipewire, but that was easily fixed as well (and Pulseaudio also works fine).
Conclusions
I’m sticking with KDE. Given that I couldn’t figure out how to stop Gnome from deciding to eat enough battery to make my fan come on, the decision wasn’t hard. But even if it weren’t for that, I’d have gone with KDE. Once a couple of things were patched, the experience is solid, fast, and flawless. Emacs (my main X11-only application) looks great with the self-scaling in KDE. Gimp, which I use occasionally, was terrible with the blurry scaling in Gnome.
Update: Corrected the gsettings command
12 September, 2023 01:40PM by John Goerzen
Like each month, have a look at the work funded by Freexian’s Debian LTS offering.
In August, 19 contributors have been paid to work on Debian LTS, their reports are available:
In August, we have released 42 DLAs.
The month of August turned out to be a rather quiet month for the LTS team.
Three notable updates were to bouncycastle, openssl, and zabbix. In the case of bouncycastle a flaw allowed for the possibility of LDAP injection and the openssl update corrected a resource exhaustion bug that could result in a denial of service. Zabbix, while not widely used, was the subject of several vulnerabilities which while not individually severe did combine to result in the zabbix update being of particular note.
Apart from those, the LTS team continued the always ongoing work of triaging, investigating, and fixing vulnerabilities, as well as making contributions to the broader Debian and Free Software communities.
Sponsors that joined recently are in bold.
12 September, 2023 12:00AM by Roberto C. Sánchez
git secret_cabal greet
After watching My life in git, after subversion, after CVS. from DebConf, I’ve realized it’s been a while since I talked about the way I keep everything1 I do in git, and I don’t think I’ve ever done it online, so it looked like a good time for a blog post.
Beyond git itself (of course), I use a few git-related programs:
mr
) to manage multiple git repositories with one command;/etc
directory;and some programs that don’t use git directly, but easily interact with it:
All of these programs are installed from Debian packages, on stable (plus rarely backports) or testing, depending on the machine.
I’m also grateful to the vcs-home people, who wrote most of the tools I use, and sometimes hang around their IRC channel.
And now, on to what I’m actually doing.
With the git repositories I’ve decided to err for too much granularity rather than too little3, so of course each project has its own repository, and so do different kinds of media files, dot-files that are related to different programs etc.
Most of the repositories are hosted on two gitolite servers: one runs on the home server, for stuff that should remain private, and the other one is on my VPS for things that are public (or may become public in the future), and also has a web interface with cgit. Of course things where I’m collaborating with other people are sometimes hosted elsewhere, mostly on salsa, sourcehut or on $DAYJOB related gitlab instances.
The .mr
directory is where everything is managed: I don’t have a single .mrconfig
file but a few different ones, that in turn load all files in a directory with the same name:
collections.mr
for the media file annexes and inventories (split into different files, so that computers with little disk space can only get the inventories);private.mr
for stuff that should only go on my own personal machine, not on shared ones;projects.mr
for the actual projects, with different files for the kinds of projects (software, docs, packaging, crafts, etc.);setup.mr
with all of the vcsh repositories, including the one that tracks the mr files (I’ll talk about the circular dependency later);work.mr
for repositories that are related to $DAYJOB.Then there are the files in the .mr/machines
directory, each one of which has the list of repositories that should be on every specific machine, including a generic workstation, but also specific machines such as e.g. the media center which has a custom set of repositories.
The dot files from my home directory are kept in vcsh, so that it’s easy to split them out into different repositories, and I’m mostly used the simplest configuration described in the 30 Second How-to in its homepage; vcsh gives some commands to work on all vcsh repositories at the same time, but most of the time I work on a single repository, and use mr to act on more than one repo.
The media collections are also pretty straightforward git-annex repositories, one for each kind of media (music, movies and other videos, e-books, pictures, etc.) and I don’t use any auto-syncing features but simply copy and move files around between clones with the git annex copy
, git annex move
and git annex get
commands.
There isn’t much to say about the project repositories (plain git), and I think that the way I use my own program lesana for inventories and project tracking is worth an article of its own, here I’ll just say that the file format used has been designed (of course) to work nicely with git.
On every machine I install etckeeper so that there is a history of the changes in the /etc
directory, but that’s only a local repository, not stored anywhere else, and is used mostly in case something breaks with an update or in similar situation. The authoritative source for the configuration of each machine is an ansible playbook (of course saved in git) which can be used to fully reconfigure the machine from a bare Debian installation.
When such a reconfiguration from scratch happens, it will be in two stages: first a run of ansible does the system-wide configuration (including installing packages, creating users etc.), and then I login on the machine and run mr to set up my own home. Of course there is a chicken-and-egg problem in that I need the mr configuration to know where to get the mr configuration, and that is solved by having setup two vcsh repositories from an old tarball export: the one with the ssh configuration to access the repositories and the one with the mr files.
So, after a machine has been configured with ansible what I’ll actually do is to login, use vcsh pull
to update those two repositories and then run mr to checkout everything else.
And that’s it, if you have questions on something feel free to ask me on the fediverse or via email (contacts are in the about page)
Update (2023-09-12 17:00ish): The ~/.mr
directory is not special for mr
, it’s just what I use and then I always run mr -c ~/.mr/some/suitable/file.mr
, with the actual file being different whether I’m registering a new repo or checking out / updating them. I could include some appropriate ~/.mr/machines/some_machine.mr
in ~/.mrconfig
, but I’ve never bothered to do so, since it wouldn’t cover all usecases anyway. Thanks to the person on #vcs-home@OFTC who asked me the question :)
At least, everything that I made that is made of bits, and a diary and/or inventory of the things made of atoms.↩︎
until we get a working replicator, I guess :D↩︎
in time I’ve consolidated a bit some of the repositories, e.g. merging the repositories for music from different sources (CD rips, legal downloads, etc.) into a single repository, but that only happened a few times, and usually I’m fine with the excess of granularity.↩︎
Some decades back, when I’d buy a new PC, it would unlock new capabilities. Maybe AGP video, or a PCMCIA slot, or, heck, sound.
Nowadays, mostly new hardware means things get a bit faster or less crashy, or I have some more space for files. It’s good and useful, but sorta… meh.
Not this purchase.
Cory Doctorow wrote about the Framework laptop in 2021:
There’s no tape. There’s no glue. Every part has a QR code that you can shoot with your phone to go to a service manual that has simple-to-follow instructions for installing, removing and replacing it. Every part is labeled in English, too!
The screen is replaceable. The keyboard is replaceable. The touchpad is replaceable. Removing the battery and replacing it takes less than five minutes. The computer actually ships with a screwdriver.
Framework had been on my radar for awhile. But for various reasons, when I was ready to purchase, I didn’t; either the waitlist was long, or they didn’t have the specs I wanted.
Lately my aging laptop with 8GB RAM started OOMing (running out of RAM). My desktop had developed a tendency to hard hang about once a month, and I researched replacing it, but the cost was too high to justify.
But when I looked into the Framework, I thought: this thing could replace both. It is a real shift in perspective to have a laptop that is nearly as upgradable as a desktop, and can be specced out to exactly what I wanted: 2TB storage and 64GB RAM. And still cheaper than a Macbook or Thinkpad with far lower specs, because the Framework uses off-the-shelf components as much as possible.
Cory Doctorow wrote, in The Framework is the most exciting laptop I’ve ever broken:
The Framework works beautifully, but it fails even better… Framework has designed a small, powerful, lightweight machine – it works well. But they’ve also designed a computer that, when you drop it, you can fix yourself. That attention to graceful failure saved my ass.
I like small laptops, so I ordered the Framework 13. I loaded it up with the 64GB RAM and 2TB SSD I wanted. Frameworks have four configurable ports, which are also hot-swappable. I ordered two USB-C, one USB-A, and one HDMI. I put them in my preferred spots (one USB-C on each side for easy docking and charging). I put Debian on it, and it all Just Worked. Perfectly.
Now, I orderd the DIY version. I hesitated about this — I HATE working with laptops because they’re all so hard, even though I KNEW this one was different — but went for it, because my preferred specs weren’t available in a pre-assembled model.
I’m glad I did that, because assembly was actually FUN.
I got my box. I opened it. There was the bottom shell with the motherboard and CPU installed. Here are the RAM sticks. There’s the SSD. A minute or two with each has them installed. Put the bezel on the screen, attach the keyboard — it has magnets to guide it into place — and boom, ready to go. Less than 30 minutes to assemble a laptop nearly from scratch. It was easier than assembling most desktops.
So now, for the first time, my main computing device is a laptop. Rather than having a desktop and a laptop, I just have a laptop. I’ll be able to upgrade parts of it later if I want to. I can rearrange the ports. And I can take all my most important files with me. I’m quite pleased!
11 September, 2023 11:56PM by John Goerzen
The Debian Day in Maceió 2023 took place at the Senai auditorium in Maceió with the support and organization of Oxe Hacker Club.
There were around 90 people registered, and 40 ateendees present on Saturday to participate in the event, which featured the following 6 talks:
Debian Day also had an install fest and unconference (random chat, food and drinks).
O Debian Day em Maceió 2023 foi realizado no auditório do Senai em Maceió com apoio e realização do Oxe Hacker Club.
Se inscreveram cerca de 90 pessoas, e 40 estiveram presentes no sábado para participarem do evento que contou com as 6 palestras a seguir:
O Debian Day teve ainda um install fest e desconferência (papo aleatório, comes e bebes).
Armadillo is a powerful and expressive C++ template library for linear algebra and scientific computing. It aims towards a good balance between speed and ease of use, has a syntax deliberately close to Matlab, and is useful for algorithm development directly in C++, or quick conversion of research code into production environments. RcppArmadillo integrates this library with the R environment and language–and is widely used by (currently) 1096 other packages on CRAN, downloaded 30.5 million times (per the partial logs from the cloud mirrors of CRAN), and the CSDA paper (preprint / vignette) by Conrad and myself has been cited 552 times according to Google Scholar.
This release brings bugfix upstream release 12.6.4. Conrad prepared this a few days ago; it takes me the usual day or so to run reverse-dependency check against the by-now almost 1100 CRAN packages using RcppArmadillo. And this time, CRAN thought it had found two issues when I submitted and it took two more days til we were all clear about those two being false positives (as can, and does, happen). So today it reached CRAN.
The set of changes follows.
Changes in RcppArmadillo version 0.12.6.4.0 (2023-09-06)
Upgraded to Armadillo release 12.6.4 (Cortisol Retox)
Workarounds for bugs in Apple accelerate framework
Fix incorrect calculation of rcond for band matrices in
solve()
Remove expensive and seldom used optimisations, leading to faster compilation times
Courtesy of my CRANberries, there is a diffstat report relative to previous release. More detailed information is on the RcppArmadillo page. Questions, comments etc should go to the rcpp-devel mailing list off the Rcpp R-Forge page.
If you like this or other open-source work I do, you can sponsor me at GitHub.
This post by Dirk Eddelbuettel originated on his Thinking inside the box blog. Please report excessive re-aggregation in third-party for-profit settings.
After cycling the Northcape 4000 (from Italy to northern Norway) last year, I signed up for the transcontinental race this year.
The Transcontinental is bikepacking race across Europe, self-routed (but with some mandatory checkpoints), unsupported and with a distance of usually somewhere around 4000 km. The cut-off time is 15 days, with the winner usually taking 7-10 days.
This year, the route went from Belgium to Thessaloniki in Greece, with control points in northern Italy, Slovenia, Albania and Meteora (Greece).
The event was great - it was well organised and communication was a lot better than at the Northcape. It did feel very different from the Northcape, though, being a proper race. Participants are not allowed to draft off each other or help each other, though a quick chat here or there as you pass people is possible, or when you’re both stopped at a shop or control point.
The route was beautiful - the first bit through France was a bit monotonic, but especially the views in the alps were amazing. Like with other long events, the first day or two can be hard but once you get into the rhythm of things it’s a lot easier.
From early on, I lost a lot of time. We started in the rain, and I ran several flats in a row, just 4 hours in. In addition to that, the thread on my pump had worn so it wouldn’t fit on some of my spare tubes, and my tubes were all TPU - which are hard to patch. So at 3 AM I found myself by the side of an N-road in France without any usable tubes to put in my rear wheel. I ended up walking 20km to the nearest town with a bike shop, where they fortunately had good old butyl tubes and a working pump. But overall, this cost me about 12 hours in total.
In addition to that, my time management wasn’t great. On previous rides, I’d usually gotten about 8 hours of sleep per night while staying in hotels. On the transcontinental I had meant to get less sleep but still stay in hotels most night, but I found that not all hotels accomodated well for that - especially with a bike. So I ended up getting more sleep than I had intended, and spending more time off the bike than I had planned - close to 11 or 12 hours per day. I hadn’t scheduled much time off work after the finish either, so arriving in Greece late wasn’t really an option.
And then, on an early morning in Croatia (about 2000km in) in heavy fog, I rode into a kerb at 35 km/h, bending the rim of my front wheel (but fortunately not coming off my bike). While I probably would have been able to continue with a replacement wheel (and mailing the broken one home), that would have taken another day to sort out and I almost certainly wouldn’t have been able to source a new dynamo wheel in Croatia - which would have made night time riding a lot harder. So I decided to scratch and take the train home from Zagreb.
Overall, I really enjoyed the event and I think I’ve learned some useful lessons. I’ll probably try again next year.
10 September, 2023 08:00PM by Jelmer Vernooij
DebConf23, the 24th edition of the Debian conference is taking place in Infopark at Kochi, Kerala, India. Thanks to the hard work of its organizers, it will be, this year as well, an interesting and fruitful event for attendees.
We would like to warmly welcome the sponsors of DebConf23, and introduce them to you.
We have three Platinum sponsors.
Our first Platinum sponsor is Infomaniak. Infomaniak is a key player in the European cloud market and the leading developer of Web technologies in Switzerland. It aims to be an independent European alternative to the web giants and is committed to an ethical and sustainable Web that respects privacy and creates local jobs. Infomaniak develops cloud solutions (IaaS, PaaS, VPS), productivity tools for online collaboration and video and radio streaming services.
Proxmox is our second Platinum sponsor. Proxmox develops powerful, yet easy-to-use open-source server software. The product portfolio from Proxmox, including server virtualization, backup, and email security, helps companies of any size, sector, or industry to simplify their IT infrastructures. The Proxmox solutions are based on the great Debian platform, and we are happy that we can give back to the community by sponsoring DebConf23.
Siemens is our third Platinum sponsor. Siemens is a technology company focused on industry, infrastructure and transport. From resource-efficient factories, resilient supply chains, smarter buildings and grids, to cleaner and more comfortable transportation, and advanced healthcare, the company creates technology with purpose adding real value for customers. By combining the real and the digital worlds, Siemens empowers its customers to transform their industries and markets, helping them to enhance the everyday of billions of people.
Our Gold sponsors are:
Lenovo, Lenovo is a global technology leader manufacturing a wide portfolio of connected products including smartphones, tablets, PCs and workstations as well as AR/VR devices, smart home/office and data center solutions.
Freexian, Freexian is a services company specialized in Free Software and in particular Debian GNU/Linux, covering consulting, custom developments, support, training. Freexian has a recognized Debian expertise thanks to the participation of Debian developers.
Google, Google is one of the largest technology companies in the world, providing a wide range of Internet-related services and products such as online advertising technologies, search, cloud computing, software, and hardware.
Ubuntu, the Operating System delivered by Canonical.
Our Silver sponsors are:
Bronze sponsors:
And finally, our Supporter level sponsors:
A special thanks to the Infoparks Kerala, our Venue Partner!
Thanks to all our sponsors for their support! Their contributions make it possible for a large number of Debian contributors from all over the globe to work together, help and learn from each other in DebConf23.
10 September, 2023 09:00AM by The Debian Publicity Team
DebConf23, the 24th annual Debian Developer Conference, is taking place in Kochi, India from September 10th to 17th, 2023.
Debian contributors from all over the world have come together at Infopark, Kochi to participate and work in a conference exclusively run by volunteers.
Today the main conference starts with over 373 expected attendants and 92 scheduled activities, including 45-minute and 20-minute talks, Bird of a Feather ("BoF") team meetings, workshops, a job fair, as well as a variety of other events.
The full schedule is updated each day, including activities planned ad-hoc by attendees over the course of the conference.
If you would like to engage remotely, you can follow the video streams available from the DebConf23 website for the events happening in the three talk rooms: Anamudi, Kuthiran and Ponmudi. Or you can join the conversations happening inside the talk rooms via the OFTC IRC network in the #debconf-anamudi, #debconf-kuthiran, and the #debconf-ponmudi channels. Please also join us in the #debconf channel for common discussions related to DebConf.
You can also follow the live coverage of news about DebConf23 provided by our micronews service or the @debian profile on your favorite social network.
DebConf is committed to a safe and welcoming environment for all participants. Please see our Code of Conduct page on the DebConf23 website for more information on this.
Debian thanks the commitment of numerous sponsors to support DebConf23, particularly our Platinum Sponsors: Infomaniak, Proxmox and Siemens.
~
10 September, 2023 09:00AM by The Debian Publicity Team
Contributing to Debian is part of Freexian’s mission. This article covers the latest achievements of Freexian and their collaborators. All of this is made possible by organizations subscribing to our Long Term Support contracts and consulting services.
Given that we now have consensus on moving forward by moving aliased files
from /
to /usr
, we will also run into the problems that the file move
moratorium was meant to prevent. The way forward is detecting them early and
applying workarounds on a per-package basis. Said detection is now automated
using the Debian Usr Merge Analysis Tool.
As problems are reported to the bug tracking system, they are connected to the
reports if properly usertagged. Bugs and patches for problem categories
DEP17-P2 and DEP17-P6 have been filed.
After consensus has been reached
on the bootstrapping matters, debootstrap
has been
changed to swap the initial unpack and merging
to avoid unpack errors due to pre-existing links. This is a precondition for
having base-files
install the aliasing symbolic links eventually.
It was identified that the root filesystem used by the Debian installer is still unmerged and a change has been proposed.
debhelper
was changed to
recognize systemd units installed to /usr.
A discussion with the CTTE and release team on repealing the moratorium has been initiated.
August was a busy month in the Salsa CI world. Santiago reviewed and merged a bunch of MRs that have improved the project in different aspects:
The aptly job got two MRs from Philip Hands. With the first one, the aptly now can export a couple of variables in a dotenv file, and with the second, it can include packages from multiple artifact directories. These MRs bring the base to improve how to test reverse dependencies with Salsa CI. Santiago is working on documenting this.
As a result of the mass bug filing done in August, Salsa CI now includes a job to test how a package builds twice in a row. Thanks to the MRs of Sebastiaan Couwenberg and Johannes Schauer Marin Rodrigues.
Last but not least, Santiago helped Johannes Schauer Marin Rodrigues to complete the support for arm64-only pipelines.
Stefano wears a few hats in the DebConf organization and in the lead up to the conference in mid-September, they’ve all been quite busy.
As one of the treasurers of DebConf 23, there has been a final budget update, and quite a few payments to coordinate from Debian’s Trusted Organizations. We try to close the books from the previous conference at the next one, so a push was made to get DebConf 22 account statements out of TOs and record them in the conference ledger.
As a website developer, we had a number of registration-related tasks, emailing attendees and trying to estimate numbers for food and accommodation.
As a conference committee member, the job was mostly taking calls and helping the local team to make decisions on urgent issues. For example, getting conference visas issued to attendees required getting political approval from the Indian government. We only discovered the full process for this too late to clear some complex cases, so this required some hard calls on skipping some countries from the application list, allowing everyone else to get visas in time. Unfortunate, but necessary.
dpkg-buildflags
now
defaults to issue arm64
-specific compiler flags, more care is needed to
distinguish between build architecture flags and host architecture flags than
previously.10 September, 2023 12:00AM by Utkarsh Gupta
Daniel Knowles’ Carmageddon: How Cars Make Life Worse and What to Do About It is an entertaining, lucid, and well-written “manifesto” (to borrow a term from the author) aiming to get us all thinking a bit more about what cars do to society, and how to move on to a better outcome for all.
The book alternates between historical context and background, lived experience (as the author is a foreign correspondent who had the opportunity to travel), and researched content. It is refreshingly free of formalities (no endless footnotes or endnotes with references, though I would have liked occassional references but hey we all went to school long enough to do a bit of research given a pointer or two). I learned or relearned a few things as I was for example somewhat unaware of the air pollution (micro-particle) impact stemming from tires and brake abrasions—for which electronic vehicles do zilch, and for which the auto-obesity of ever larger and heavier cars is making things much worse. And some terms (even when re-used by Knowles) are clever such bionic duckweed. But now you need to read the book to catch up on it.
Overall, the book argues its case rather well. The author brings sufficient evidence to make the formal ‘guilty’ charge quite convincing. It is also recent having come out just months ago, making current figures even more relevant.
I forget the exact circumstance but I think I came across the author in the context of our joint obsession with both Chicago and cycling (as there may have been a link from a related social media post) and/or the fact that I followed some of his colleagues at The Economist on social media. Either way, the number of Chicago and MidWest references made for some additional fun when reading the book over a the last few days. And for me another highlight was the ode to Tokyo which I wholeheartedly agree with: on my second trip to Japan I spent a spare day cycling across the city as the AirBnB host kindly gave me access to his bicycles. Great weather, polite drivers, moderate traffic, and just wicked good infrastructure made me wonder why I did not see more cyclists.
I have little to criticize beyond the lack of any references. The repeated insistence on reminding us that Knowles comes from Birmingham gets a little old by the fifth or sixth repetition. It is all a wee bit anglo- or UK-centric. It obviously has a bit on France, Paris, and all the recent success of Anne Hidalgo (who, when I was in graduate school in France, was still a TV person rather than the very successful mayor she is now) but then does not mention the immense (and well known) success of the French train system which lead to a recent dictum to no longer allow intra-Frace air travel if train rides of under 2 1/2 hours are available which is rather remarkable. (Though in fairness that may have been enacted once the book was finished.)
Lastly, the book appears to have a few sections available via Google Books. My copy will good back from one near-west suburban library to the neighbouring one.
Overall a strong recommendation for a very good and timely book.
A minor maintenance release of the RcppFarmHash package is now on CRAN as version 0.0.3.
RcppFarmHash
wraps the Google FarmHash family of hash
functions (written by Geoff Pike and contributors) that are used for
example by Google BigQuery for the FARM_FINGERPRINT
digest.
This releases farms out the conversion to the integer64
add-on type in R to the new
package RcppInt64
released a few days ago and adds some minor maintenance on continuous
integration and alike.
The brief NEWS entry follows:
Changes in version 0.0.3 (2023-09-09)
Rely on new RcppInt64 package and its header for conversion
Minor updates to continuous integration and README.md
If you like this or other open-source work I do, you can sponsor me at GitHub.
This post by Dirk Eddelbuettel originated on his Thinking inside the box blog. Please report excessive re-aggregation in third-party for-profit settings.
Debian Celebrates 30 years!
We celebrated our birthday this year and we had a great time with new friends, new members welcomed to the community, and the world.
We have collected a few comments, videos, and discussions from around the Internet, and some images from some of the DebianDay2023 events. We hope that you enjoyed the day(s) as much as we did!
"Debian 30 years of collective intelligence" -Maqsuel Maqson
Pouso Alegre, Brazil
Maceió, Brazil
Curitiba, Brazil
The cake is there. :)
Honorary Debian Developers: Buzz, Jessie, and Woody welcome guests to this amazing party.
Sao Carlos, state of Sao Paulo, Brazil
Stickers, and Fliers, and Laptops, oh my!
Belo Horizonte, Brazil
Brasília, Brazil
Brasília, Brazil
30 años!
A quick Selfie
We do not encourage beverages on computing hardware, but this one is okay by us.
30 years of love
The German Delegation is also looking for this dog who footed the bill for the party, then left mysteriously.
We took the party outside
We brought the party back inside at CCCamp
Cake and Diversity in Belgium
Food and Fellowship in El Salvador
Debian is also very delicious!
All smiles waiting to eat the cake
Reports
Debian Day 30 years in Maceió - Brazil
Debian Day 30 years in São Carlos - Brazil
Debian Day 30 years in Pouso Alegre - Brazil
Debian Day 30 years in Belo Horizonte - Brazil
Debian Day 30 years in Curitiba - Brazil
Debian Day 30 years in Brasília - Brazil
Debian Day 30 years online in Brazil
Articles & Blogs
Happy Debian Day - going 30 years strong - Liam Dawe
Debian Turns 30 Years Old, Happy Birthday! - Marius Nestor
30 Years of Stability, Security, and Freedom: Celebrating Debian’s Birthday - Bobby Borisov
Happy 30th Birthday, Debian! - Claudio Kuenzier
Debian is 30 and Sgt Pepper Is at Least Ninetysomething - Christine Hall
Debian turns 30! -Corbet
Thirty years of Debian! - Lennart Hengstmengel
Debian marks three decades as 'Universal Operating System' - Sam Varghese
Debian Linux Celebrates 30 Years Milestone - Joshua James
30 years on, Debian is at the heart of the world's most successful Linux distros - Liam Proven
Looking Back on 30 Years of Debian - Maya Posch
Cheers to 30 Years of Debian: A Journey of Open Source Excellence - arindam
Discussions and Social Media
Debian Celebrates 30 Years - Source: News YCombinator
Brand-new Linux release, which I'm calling the Debian ... Source: News YCombinator
Comment: Congrats @debian !!! Happy Birthday! Thank you for becoming a cornerstone of the #opensource world. Here's to decades of collaboration, stability & #software #freedom -openSUSELinux via X (formerly Twitter)
Comment: Today we #celebrate the 30th birthday of #Debian, one of the largest and most important cornerstones of the #opensourcecommunity. For this we would like to thank you very much and wish you the best for the next 30 years! Source: X (Formerly Twitter -TUXEDOComputers via X (formerly Twitter)
Happy Debian Day! - Source: Reddit.com
Video The History of Debian | The Beginning - Source: Linux User Space
Debian Celebrates 30 years -Source: Lobste.rs
Video Debian At 30 and No More Distro Hopping! - LWDW388 - Source: LinuxGameCast
09 September, 2023 09:00AM by Donald Norwood, Paulo Henrique de Lima Santana
Welcome to the August 2023 report from the Reproducible Builds project!
In these reports we outline the most important things that we have been up to over the past month. As a quick recap, whilst anyone may inspect the source code of free software for malicious flaws, almost all software is distributed to end users as pre-compiled binaries.
The motivation behind the reproducible builds effort is to ensure no flaws have been introduced during this compilation process by promising identical results are always generated from a given source, thus allowing multiple third-parties to come to a consensus on whether a build was compromised. If you are interested in contributing to the project, please visit our Contribute page on our website.
Bleeping Computer reported that Serde, a popular Rust serialization framework, had decided to ship its serde_derive
macro as a precompiled binary. As Ax Sharma writes:
The move has generated a fair amount of push back among developers who worry about its future legal and technical implications, along with a potential for supply chain attacks, should the maintainer account publishing these binaries be compromised.
After intensive discussions, use of the precompiled binary was phased out.
On August 4th, Holger Levsen gave a talk at BornHack 2023 on the Danish island of Funen titled Reproducible Builds, the first ten years which promised to contain:
[…] an overview about reproducible builds, the past, the presence and the future. How it started with a small [meeting] at DebConf13 (and before), how it grew from being a Debian effort to something many projects work on together, until in 2021 it was mentioned in an executive order of the president of the United States. (HTML slides)
Holger repeated the talk later in the month at Chaos Communication Camp 2023 in Zehdenick, Germany:
A video of the talk is available online, as are the HTML slides.
Just another reminder that our upcoming Reproducible Builds Summit is set to take place from October 31st — November 2nd 2023 in Hamburg, Germany.
Our summits are a unique gathering that brings together attendees from diverse projects, united by a shared vision of advancing the Reproducible Builds effort. During this enriching event, participants will have the opportunity to engage in discussions, establish connections and exchange ideas to drive progress in this vital field.
If you’re interested in joining us this year, please make sure to read the event page, the news item, or the invitation email that Mattia Rizzolo sent out, which have more details about the event and location.
We are also still looking for sponsors to support the event, so do reach out to the organizing team if you are able to help. (Also of note that PackagingCon 2023 is taking place in Berlin just before our summit, and their schedule has just been published.)
Vagrant Cascadian was interviewed on the SustainOSS podcast on reproducible builds:
Vagrant walks us through his role in the project where the aim is to ensure identical results in software builds across various machines and times, enhancing software security and creating a seamless developer experience. Discover how this mission, supported by the Software Freedom Conservancy and a broad community, is changing the face of Linux distros, Arch Linux, openSUSE, and F-Droid. They also explore the challenges of managing random elements in software, and Vagrant’s vision to make reproducible builds a standard best practice that will ideally become automatic for users. Vagrant shares his work in progress and their commitment to the “last mile problem.”
The episode is available to listen (or download) from the Sustain podcast website. As it happens, the episode was recorded at FOSSY 2023, and the video of Vagrant’s talk from this conference (Breaking the Chains of Trusting Trust is now available on Archive.org:
It was also announced that Vagrant Cascadian will be presenting at the Open Source Firmware Conference in October on the topic of Reproducible Builds All The Way Down.
Carles Pina i Estany wrote to our mailing list during August with an interesting question concerning the practical steps to reproduce the hello-traditional
package from Debian. The entire thread can be viewed from the archive page, as can Vagrant Cascadian’s reply.
Rahul Bajaj updated our website to add a series of environment variations related to reproducible builds […], Russ Cox added the Go programming language to our projects page […] and Vagrant Cascadian fixed a number of broken links and typos around the website […][…][…].
In diffoscope development this month, versions 247
, 248
and 249
were uploaded to Debian unstable by Chris Lamb, who also added documentation for the new specialize_as
method and expanding the documentation of the existing specialize
as well […]. In addition, Fay Stegerman added specialize_as
and used it to optimise .smali
comparisons when decompiling Android .apk
files […], Felix Yan and Mattia Rizzolo corrected some typos in code comments […,…], Greg Chabala merged the RUN commands into single layer in the package’s Dockerfile
[…] thus greatly reducing the final image size. Lastly, Roland Clobus updated tool descriptions to mark that the xb-tool
has moved package within Debian […].
reprotest is our tool for building the same source code twice in different environments and then checking the binaries produced by each build for any differences. This month, Vagrant Cascadian updated the packaging to be compatible with Tox version 4. This was originally filed as Debian bug #1042918 and Holger Levsen uploaded this to change to Debian unstable as version 0.7.26 […].
In Debian, 28 reviews of Debian packages were added, 14 were updated and 13 were removed this month adding to our knowledge about identified issues. A number of issue types were added, including Chris Lamb adding a new timestamp_in_documentation_using_sphinx_zzzeeksphinx_theme
toolchain issue.
In August, F-Droid added 25 new reproducible apps and saw 2 existing apps switch to reproducible builds, making 191 apps in total that are published with Reproducible Builds and using the upstream developer’s signature. […]
Bernhard M. Wiedemann published another monthly report about reproducibility within openSUSE.
The Reproducible Builds project detects, dissects and attempts to fix as many currently-unreproducible packages as possible. We endeavour to send all of our patches upstream where appropriate. This month, we wrote a large number of such patches, including:
Bernhard M. Wiedemann:
arimo
(modification time in build results)apptainer
(random Go build identifier)arrow
(fails to build on single-CPU machines)camlp
(parallelism-related issue)developer
(Go ordering-related issue)elementary-xfce-icon-theme
(font-related problem)gegl
(parallelism issue)grommunio
(filesystem ordering issue)grpc
(drop nondetermistic log)guile-parted
(parallelism-related issue)icinga
(hostname-based issue)liquid-dsp
(CPU-oriented problem)memcached
(package fails to build far in the future)openmpi5/openpmix
(date/copyright year issue)openmpi5
(date/copyright year issue)orthanc-ohif+orthanc-volview
(ordering related issue plus timestamp in a Gzip)perl-Net-DNS
(package fails to build far in the future)postgis
(parallelism issue)python-scipy
(uses an arbitrary build path)python-trustme
(package fails to build far in the future)qtbase/qmake/goldendict-ng
(timestamp-related issue)qtox
(date-related issue)ring
(filesytem ordering related issue)scipy
(1 & 2) (drop arbtirary build path and filesytem-ordering issue)snimpy
(1 & 3) (fails to build on single-CPU machines as well far in the future)tango-icon-theme
(font-related issue)Chris Lamb:
Rebecca N. Palmer:
The Reproducible Builds project operates a comprehensive testing framework (available at tests.reproducible-builds.org) in order to check packages and other artifacts for reproducibility. In August, a number of changes were made by Holger Levsen:
Debian-related changes:
reproducible-tracker.json
data file. […]pbuilder.tgz
for Debian unstable due to #1050784. […][…]usrmerge
. […][…]armhf
nodes (wbq0
and jtx1a
) as down; investigation is needed. […]Misc:
System health checks:
In addition, Vagrant Cascadian updated the scripts to use a predictable build path that is consistent with the one used on buildd.debian.org
. […][…]
If you are interested in contributing to the Reproducible Builds project, please visit our Contribute page on our website. However, you can get in touch with us via:
IRC: #reproducible-builds
on irc.oftc.net
.
Mailing list: rb-general@lists.reproducible-builds.org
Mastodon: @reproducible_builds@fosstodon.org
Twitter: @ReproBuilds
Debian: when you're more likely to get a virus than your laptop
The FAI.me service for creating customized installation and cloud images now supports the backports kernel for the stable release Debian 12 (aka bookworm). If you enable the backports option in the web interface, you currently get kernel 6.4. This will help you if you have newer hardware that is not support by the default kernel 6.1. The backports option is also still available for the older distributions.
The web interface of the FAI.me service is available at
The diffoscope maintainers are pleased to announce the release of diffoscope
version 250
. This version includes the following changes:
[ Chris Lamb ]
* Fix compatibility with file 5.45. (Closes: reproducible-builds/diffoscope#351)
[ Vagrant Cascadian ]
* Add external tool references for GNU Guix (for html2text and ttx).
You find out more by visiting the project homepage.