December 24, 2024

Divine Attah-Ohiemi

Seamless Transitions: Mastering Apache Redirects for a Smooth Hugo Migration

This week, I dove into setting up redirects with Apache to make the transition to Hugo's multilingual system smoother. The challenge? Ensuring that all those old links still worked while I migrated to the new URL format.

For instance, I needed to redirect:

/es/distrib to /distrib/index.es.html
/es/social_contract to /social_contract.es.html
/es/intro/about to /intro/about.es.html
/da to /index.da.html

To tackle this, I turned to Apache's mod_rewrite. Here’s the magic I came up with in my .htaccess file:

RewriteCond %{REQUEST_URI} ^/([a-z]{2}(?:-[a-z]{2})?)/(.*)$
RewriteCond %{DOCUMENT_ROOT}/$2/index.%1.html -f
RewriteCond %{DOCUMENT_ROOT}/$1/$2 !-d
RewriteRule ^/([a-z]{2}(?:-[a-z]{2})?)/(.*)$ /$2/index.%1.html [last,redirect]

RewriteCond %{REQUEST_URI} ^/([a-z]{2}(?:-[a-z]{2})?)/(.*)$
RewriteCond %{DOCUMENT_ROOT}/$2.%1.html -f
RewriteCond %{DOCUMENT_ROOT}/$1/$2 !-d
RewriteRule ^/([a-z]{2}(?:-[a-z]{2})?)/(.*)$ /$2.%1.html [last,redirect]

What’s happening here? The rules check if the URL starts with a language code (like /es or /da). Then, they verify whether the corresponding HTML file exists. If it does, and the path isn’t a directory, voilà! The user gets redirected to the new format.

It’s a bit of a dance with conditions and rules, but it’s satisfying to see everything working seamlessly. Now, as I continue migrating content, users clicking on old links won’t end up in a digital dead end. It’s all about keeping the flow smooth and maintaining that user experience.

So, if you’re also juggling multilingual pages and thinking about making the switch to Hugo, don’t underestimate the power of mod_rewrite. It’s your best friend in the world of redirects! Happy coding!

24 December, 2024 03:54PM by Divine Attah-Ohiemi

Russ Allbery

Review: Number Go Up

Review: Number Go Up, by Zeke Faux

Publisher: Crown Currency
Copyright: 2023
Printing: 2024
ISBN: 0-593-44382-9
Format: Kindle
Pages: 373

Number Go Up is a cross between a history and a first-person account of investigative journalism around the cryptocurrency bubble and subsequent collapse in 2022. The edition I read has an afterward from June 2024 that brings the story up to date with Sam Bankman-Fried's trial and a few other events. Zeke Faux is a reporter for Bloomberg News and a fellow of New America.

Last year, I read Michael Lewis's Going Infinite, a somewhat-sympathetic book-length profile of Sam Bankman-Fried that made a lot of people angry. One of the common refrains at the time was that people should read Number Go Up instead, and since I'm happy to read more about the absurdities of the cryptocurrency world, I finally got around to reading the other big crypto book of 2023.

This is a good book, with some caveats that I am about to explain at absurd length. If you want a skeptical history of the cryptocurrency bubble, you should read it. People who think that it's somehow in competition with Michael Lewis's book or who think the two books disagree (including Faux himself) have profoundly missed the point of Going Infinite. I agree with Matt Levine: Both of these books are worth your time if this is the sort of thing you like reading about. But (much) more on Faux's disagreements with Lewis later.

The frame of Number Go Up is Faux's quixotic quest to prove that Tether is a fraud. To review this book, I therefore need to briefly explain what Tether is. This is only the first of many extended digressions.

One natural way to buy cryptocurrency would be to follow the same pattern as a stock brokerage account. You would deposit some amount of money into the account (or connect the brokerage account to your bank account), and then exchange money for cryptocurrency or vice versa, using bank transfers to put money in or take it out. However, there are several problems with this. One is that swapping cryptocurrency for money is awkward and sometimes expensive. Another is that holding people's investment money for them is usually highly regulated, partly for customer safety but also to prevent money laundering. These are often called KYC laws (Know Your Customer), and the regulation-hostile world of cryptocurrency didn't want to comply with them.

Tether is a stablecoin, which means that the company behind Tether attempts to guarantee that one Tether is always worth exactly one US dollar. It is not a speculative investment like Bitcoin; it's a cryptocurrency substitute for dollars. People exchange dollars for Tether to get their money into the system and then settle all of their subsequent trades in Tether, only converting the Tether back to dollars when they want to take their money out of cryptocurrency entirely. In essence, Tether functions like the cash reserve in a brokerage account: Your Tether holdings are supposedly guaranteed to be equivalent to US dollars, you can withdraw them at any time, and because you can do so, you don't bother, instead leaving your money in the reserve account while you contemplate what new coin you want to buy.

As with a bank, this system rests on the assurance that one can always exchange one Tether for one US dollar. The instant people stop believing this is true, people will scramble to get their money out of Tether, creating the equivalent of a bank run. Since Tether is not a regulated bank or broker and has no deposit insurance or strong legal protections, the primary defense against a run on Tether is Tether's promise that they hold enough liquid assets to be able to hand out dollars to everyone who wants to redeem Tether. (A secondary defense that I wish Faux had mentioned is that Tether limits redemptions to registered accounts redeeming more than $100,000, which is a tiny fraction of the people who hold Tether, but for most purposes this doesn't matter because that promise is sufficient to maintain the peg with the dollar.)

Faux's firmly-held belief throughout this book is that Tether is lying. He believes they do not have enough money to redeem all existing Tether coins, and that rather than backing every coin with very safe liquid assets, they are using the dollars deposited in the system to make illiquid and risky investments.

Faux never finds the evidence that he's looking for, which makes this narrative choice feel strange. His theory was tested when there was a run on Tether following the collapse of the Terra stablecoin. Tether passed without apparent difficulty, redeeming $16B or about 20% of the outstanding Tether coins. This doesn't mean Faux is wrong; being able to redeem 20% of the outstanding tokens is very different from being able to redeem 100%, and Tether has been fined for lying about its reserves. But Tether is clearly more stable than Faux thought it was, which makes the main narrative of the book weirdly unsatisfying. If he admitted he might be wrong, I would give him credit for showing his work even if it didn't lead where he expected, but instead he pivots to focusing on Tether's role in money laundering without acknowledging that his original theory took a serious blow.

In Faux's pursuit of Tether, he wanders through most of the other elements of the cryptocurrency bubble, and that's the strength of this book. Rather than write Number Go Up as a traditional history, Faux chooses to closely follow his own thought processes and curiosity. This has the advantage of giving Faux an easy and natural narrative, something that non-fiction books of this type can struggle with, and it lets Faux show how confusing and off-putting the cryptocurrency world is to an outsider.

The best parts of this book were the parts unrelated to Tether. Faux provides an excellent summary of the Axie Infinity speculative bubble and even traveled to the Philippines to interview people who were directly affected. He then wandered through the bizarre world of NFTs, and his first-hand account of purchasing one (specifically a Mutant Ape) to get entrance to a party (which sounded like a miserable experience I would pay money to get out of) really drives home how sketchy and weird cryptocurrency-related software and markets can be. He also went to El Salvador to talk to people directly about the country's supposed embrace of Bitcoin, and there's no substitute for that type of reporting to show how exaggerated and dishonest the claims of cryptocurrency adoption are.

The disadvantage of this personal focus on Faux himself is that it sometimes feels tedious or sensationalized. I was much less interested in his unsuccessful attempts to interview the founder of Tether than Faux was, and while the digression into forced labor compounds in Cambodia devoted to pig butchering scams was informative (and horrific), I think Faux leaned too heavily on an indirect link to Tether. His argument is that cryptocurrency enables a type of money laundering that is particularly well-suited to supporting scams, but both scams and this type of economic slavery existed before cryptocurrency and will exist afterwards. He did not make a very strong case that Tether was uniquely valuable as a money laundering service, as opposed to a currently useful tool that would be replaced with some other tool should it go away.

This part of the book is essentially an argument that money laundering is bad because it enables crime, and sure, to an extent I agree. But if you're going to put this much emphasis on the evils of money laundering, I think you need to at least acknowledge that many people outside the United States do not want to give US government, which is often openly hostile to them, veto power over their financial transactions. Faux does not.

The other big complaint I have with this book, and with a lot of other reporting on cryptocurrency, is that Faux is sloppy with the term "Ponzi scheme." This is going to sound like nit-picking, but I think this sloppiness matters because it may obscure an ongoing a shift in cryptocurrency markets.

A Ponzi scheme is not any speculative bubble. It is a very specific type of fraud in which investors are promised improbably high returns at very low risk and with safe principal. These returns are paid out, not via investment in some underlying enterprise, but by taking the money from new investments and paying it to earlier investors. Ponzi schemes are doomed because satisfying their promises requires a constantly increasing flow of new investors. Since the population of the world is finite, all Ponzi schemes are mathematically guaranteed to eventually fail, often in a sudden death spiral of ever-increasing promises to lure new investors when the investment stream starts to dry up.

There are some Ponzi schemes in cryptocurrency, but most practices that are called Ponzi schemes are not. For example, Faux calls Axie Infinity a Ponzi scheme, but it was missing the critical elements of promised safe returns and fraudulently paying returns from the investments of later investors. It was simply a speculative bubble that people bought into on the assumption that its price would increase, and like any speculative bubble those who sold before the peak made money at the expense of those who bought at the peak.

The reason why this matters is that Ponzi schemes are a self-correcting problem. One can decry the damage caused when they collapse, but one can also feel the reassuring certainty that they will inevitably collapse and prove the skeptics correct. The same is not true of speculative assets in general. You may think that the lack of an underlying economic justification for prices means that a speculative bubble is guaranteed to collapse eventually, but in the famous words of Gary Schilling, "markets can remain irrational a lot longer than you and I can remain solvent."

One of the people Faux interviews explains this distinction to him directly:

Rong explained that in a true Ponzi scheme, the organizer would have to handle the "fraud money." Instead, he gave the sneakers away and then only took a small cut of each trade. "The users are trading between each other. They are not going through me, right?" Rong said. Essentially, he was arguing that by downloading the Stepn app and walking to earn tokens, crypto bros were Ponzi'ing themselves.

Faux is openly contemptuous of this response, but it is technically correct. Stepn is not a Ponzi scheme; it's a speculative bubble. There are no guaranteed returns being paid out of later investments and no promise that your principal is safe. People are buying in at price that you may consider irrational, but Stepn never promised you would get your money back, let alone make a profit, and therefore it doesn't have the exponential progression of a Ponzi scheme. One can argue that this is a distinction without a moral difference, and personally I would agree, but it matters immensely if one is trying to analyze the future of cryptocurrencies.

Schemes as transparently unstable as Stepn (which gives you coins for exercise and then tries to claim those coins have value through some vigorous hand-waving) are nearly as certain as Ponzi schemes to eventually collapse. But it's also possible to create a stable business around allowing large numbers of people to regularly lose money to small numbers of sophisticated players who are collecting all of the winnings. It's called a poker room at a casino, and no one thinks poker rooms are Ponzi schemes or are doomed to collapse, even though nearly everyone who plays poker will lose money.

This is the part of the story that I think Faux largely missed, and which Michael Lewis highlights in Going Infinite. FTX was a legitimate business that made money (a lot of money) off of trading fees, in much the same way that a casino makes money off of poker rooms. Lots of people want to bet on cryptocurrencies, similar to how lots of people want to play poker. Some of those people will win; most of those people will lose. The casino doesn't care. Its profit comes from taking a little bit of each pot, regardless of who wins. Bankman-Fried also speculated with customer funds, and therefore FTX collapsed, but there is no inherent reason why the core exchange business cannot be stable if people continue to want to speculate in cryptocurrencies. Perhaps people will get tired of this method of gambling, but poker has been going strong for 200 years.

It's also important to note that although trading fees are the most obvious way to be a profitable cryptocurrency casino, they're not the only way. Wall Street firms specialize in finding creative ways to take a cut of every financial transaction, and many of those methods are more sophisticated than fees. They are so good at this that buying and selling stock through trading apps like Robinhood is free. The money to run the brokerage platform comes from companies that are delighted to pay for the opportunity to handle stock trades by day traders with a phone app. This is not, as some conspiracy theories would have you believe, due to some sort of fraudulent price manipulation. It is because the average person with a Robinhood phone app is sufficiently unsophisticated that companies that have invested in complex financial modeling will make a steady profit taking the other side of their trades, mostly because of the spread (the difference between offered buy and sell prices).

Faux is so caught up in looking for Ponzi schemes and fraud that I think he misses this aspect of cryptocurrency's transformation. Wall Street trading firms aren't piling into cryptocurrency because they want to do securities fraud. They're entering this market because there seems to be persistent demand for this form of gambling, cryptocurrency markets reward complex financial engineering, and running a legal casino is a profitable business model.

Michael Lewis appears as a character in this book, and Faux portrays him quite negatively. The root of this animosity appears to stem from a cryptocurrency conference in the Bahamas that Faux attended. Lewis interviewed Bankman-Fried on stage, and, from Faux's account, his questions were fawning and he praised cryptocurrencies in ways that Faux is certain he knew were untrue. From that point on, Faux treats Lewis as an apologist for the cryptocurrency industry and for Sam Bankman-Fried specifically.

I think this is a legitimate criticism of Lewis's methods of getting close to the people he wants to write about, but I think Faux also makes the common mistake of assuming Lewis is a muckraking reporter like himself. This has never been what Lewis is interested in. He writes about people he finds interesting and that he thinks a reader will also find interesting. One can legitimately accuse him of being credulous, but that's partly because he's not even trying to do the same thing Faux is doing. He's not trying to judge; he's trying to understand.

This shows when it comes to the parts of this book about Sam Bankman-Fried. Faux's default assumption is that everyone involved in cryptocurrency is knowingly doing fraud, and a lot of his research is looking for evidence to support the conclusion he had already reached. I don't think there's anything inherently wrong with that approach: Faux is largely, although not entirely, correct, and this type of hostile journalism is incredibly valuable for society at large. Upton Sinclair didn't start writing The Jungle with an open mind about the meat-packing industry. But where Faux and Lewis disagree on Bankman-Fried's motivations and intentions, I think Lewis has the much stronger argument.

Faux's position is that Bankman-Fried always intended to steal people's money through fraud, perhaps to fund his effective altruism donations, and his protestations that he made mistakes and misplaced funds are obvious lies. This is an appealing narrative if one is looking for a simple villain, but Faux's evidence in support of this is weak. He mostly argues through stereotype: Bankman-Fried was a physics major and a Jane Street trader and therefore could not possibly be the type of person to misplace large amounts of money or miscalculate risk.

If he wants to understand how that could be possible, he could read Going Infinite? I find it completely credible that someone with what appears to be uncontrolled, severe ADHD could be adept at trading and calculating probabilities and yet also misplace millions of dollars of assets because he wasn't thinking about them and therefore they stopped existing.

Lewis made a lot of people angry by being somewhat sympathetic to someone few people wanted to be sympathetic towards, but Faux (and many others) are also misrepresenting his position. Lewis agrees that Bankman-Fried intentionally intermingled customer funds with his hedge fund and agrees that he lied about doing this. His only contention is that Bankman-Fried didn't do this to steal the money; instead, he invested customer money in risky bets that he thought would pay off. In support of this, Lewis made a prediction that was widely scoffed at, namely that much less of FTX's money was missing than was claimed, and that likely most or all of it would be found.

And, well, Lewis was basically correct? The FTX bankruptcy is now expected to recover considerably more than the amount of money owed to creditors. Faux argues that this is only because the bankruptcy clawed back assets and cryptocurrencies have gone up considerably since the FTX bankruptcy, and therefore that the lost money was just replaced by unexpected windfall profits on other investments, but I don't think this point is as strong as he thinks it is. Bankman-Fried lost money on some of what he did with customer funds, made money on other things, and if he'd been able to freeze withdrawals for the year that the bankruptcy froze them, it does appear most of the money would have been recoverable. This does not make what he did legal or morally right, but no one is arguing that, only that he didn't intentionally steal money for his own personal gain or for effective altruism donations. And on that point, I don't think Faux is giving Lewis's argument enough credit.

I have a lot of complaints about this book because I know way too much about this topic than anyone should probably know. I think Faux missed the plot in a couple of places, and I wish someone would write a book about where cryptocurrency markets are currently going. (Matt Levine's Money Stuff newsletter is quite good, but it's about all sorts of things other than cryptocurrency and isn't designed to tell a coherent story.) But if you know less about cryptocurrency and just want to hear the details of the run-up to the 2022 bubble, this is a great book for that. Faux is writing for people who are already skeptical and is not going to convince people who are cryptocurrency true believers, but that's fine. The details are largely correct (and extensively footnoted) and will satisfy most people's curiosity.

Lewis's Going Infinite is a better book, though. It's not the same type of book at all, and it will not give you the broader overview of the cryptocurrency world. But if you're curious about what was going through the head of someone at the center of all of this chaos, I think Lewis's analysis is much stronger than Faux's. I'm happy I read both books.

Rating: 8 out of 10

24 December, 2024 03:21AM

December 23, 2024

Dropping the git protocol

Ever since I started converting my free software repositories to Git and hosting them on git.eyrie.org, I've made them available via the Git network protocol (the one that uses git:// URLs). This protocol doesn't support TLS or any other form of integrity checking on the wire and has been gently deprecated for some time. Recent changes to git daemon broke the way I was managing permissions for my personal repositories, which meant that the git://git.eyrie.org/ URLs have been broken for several months. Since no one has complained or apparently noticed, this seems like a good excuse to drop support entirely and run one fewer service.

All of my personal repositories will continue to be accessible via Git over HTTPS, which is now the standard for anonymous Git checkouts. If the prior URL was git://git.eyrie.org/area/package.git, the new URL is https://git.eyrie.org/git/area/package.git. I've updated my web pages accordingly. This protocol provides integrity protection on the wire and a moderate amount of authentication of my server via a Let's Encrypt certificate.

23 December, 2024 07:53PM

Sahil Dhiman

Debian Mirrors Hierarchy

After finding AlmaLinux mirror sync capacity at Tier 0 (or Tier 1, however you look at it) is around 140 Gbps, I wanted to find source and hierarchy in Debian mirroring systems.

There are two main types of mirrors in Debian - Debian package mirrors (for package installs and updates) and Debian CD mirrors (for ISO and others medias). Let’s talk about package mirrors (and it’s hierarchy) first.

Package mirror hierarchy

Trace file was a good starting point for checking upstream for a package mirror in Debian. It resides at <URL>/debian/project/trace/_traces and shows flow of data. Sample trace file from jing.rocks’s mirror. It showed, canonical source for packages is ftp-master.debian.org. Checking via https://db.debian.org/machines.cgi, showed it’s fasolo.d.o hosted at Brown University, US. This serves as “Master Archive Server”, making it a Tier 0 mirror. It’s entry mentions that it has 1 Gbps shared LAN connectivity (dated information?) but it only has to push to 3 other machines/sites.

Side note - .d.o is .debian.org

As shown on https://mirror-master.debian.org/status/mirror-hierarchy.html, the three sites are:

  • syncproxy2.eu.debian.org ie smit.d.o hosted by University of Twente, Netherlands with 2x10 Gbps connectivity.
  • syncproxy4.eu.debian.org ie schmelzer.d.o hosted by Conova in Austria with 2x10 Gbps connectivity.
  • syncproxy2.wna.debian.org - https://db.debian.org/machines.cgi entry mentions it being hosted at UBC here, but IP seems to be pointing to OSUOSL IP range as of now. IIRC few months ago, syncproxy2.wna.d.o was made to point to other host due to some issue (?). mirror-osuosl.d.o seems to be serving as syncproxy2.wna.d.o now. Bandwidth isn’t explicitly mentioned but from my experience seeing bandwidths which other free software projects hosted at OSUOSL have, it would be atleast 10 Gbps and maybe more for Debian.

                     syncproxy2.eu.d.o (NL) ---> to the world
                    /
ftp-master.d.o (US) -- syncproxy4.eu.d.o (AT)  --> to the world 
                    \
                     syncproxy2.wna.d.o (US) --> to the world
A visualation of flow of package from ftp-master.d.o

These form the Debian Tier 1 mirror network, as all the mirrors sync from them. So Debian has atleast 50 Gbps+ capacity at Tier 1. A normal Debian user might never directly interact with any of these 3 machines, but every Debian package they run/download/install flows through these machines. Though, I’m unsure what wna stands for in syncproxy2.wna.d.o. NA probably is North America and W is west (coast)? If you know, do let me know.

After Tier 1, there are a few more syncproxies (detailed below). There are atleast 45 mirrors at Tier 2, updates for which are directly pushed from the three Tier 1 sync proxies. Most country mirrors i.e. ftp..debian.org are at Tier 2 too (barring a few like ftp.au.d.o, ftp.nz.do etc).

Coming back to Sync proxies at Tier 2:

  • syncproxy3.wna.debian.org - gretchaninov.d.o which is marked as syncproxy2 on db.d.o (information dated). It’s hosted in University of British Columbia, Canada, where a lot of Debian infrastructure including Salsa is hosted.
  • syncproxy.eu.debian.org - Croatian Academic and Research Network managed machine. CNAME directs to debian.carnet.hr.
  • syncproxy.au.debian.org - mirror-anu.d.o hosted by Australian National University with 100Mbps connectivity. Closest sync proxy for all Australian mirrors.
  • syncproxy4.wna.debian.org - syncproxy-aws-wna-01.d.o hosted in AWS, in US (according to GeoIP). IPv6 only (CNAME to syncproxy-aws-wna-01.debian.org. which only has an AAAA record, no A record). A m6g.2xlarge instance which has speeds upto 10 Gbps.

Coming back to https://mirror-master.debian.org/status/mirror-hierarchy.html, one can see chain extend till Tier 6 like in case of this mirror in AU which should add some latency for the updates from being pushed at ftp-master.d.o to them. Ideally, which shouldn’t be a problem as https://www.debian.org/mirror/ftpmirror#when mentions “The main archive gets updated four times a day”.

In my case, I get my updates from NITC mirror, so my updates flows from US > US > TW > IN > me in IN.

CDNs have to internally manage cache purging too unlike normal mirrors which directly serve static file. Both deb.debian.org (sponsored by Fastly) and cdn-aws.deb.debian.org (sponsored by Amazon Cloudfront) sync from following CDN backends:

See deb.d.o trace file and cdn-aws.deb.d.o trace file.

(Thanks to Philipp Kern for the heads up here.)

CD image mirrors Hierarchy

Till now, I have only talked about Debian package mirrors. When you see /debian directory on various mirrors, they’re usually for package install and updates. If you want to grab the latest (and greatest) Debian ISO, you go to Debian CD (as they’re still called) mirror site.

casulana.d.o is mentioned as CD builder site hosted by Bytemark while pettersson-ng.d.o is mentioned as CD publishing server hosted at Academic Computer Club in Umeå, Sweden. Primary download site for Debian CD when you click download on debian.org homepage is https://cdimage.debian.org/debian-cd/ is hosted here as well. This essentially becomes Tier 0 mirror for Debian CD. All Debian CD mirrors are downstream to it.

pettersson-ng.d.o / cdimage.d.o (SE) ---> to the world
A visualation of flow of Debian CD from cdimage.d.o

Academic Computer Club’s mirror setup uses a combination of multiple machines (called frontends and offloading servers) to load balance requests. Their document setup is a highly recommended read. Also, in that document, they mention , “All machines are reachable via both IPv4 and IPv6 and connected with 10 or 25 gigabit Ethernet, external bandwidth available is 200 gigabit/s.”

For completeness sake, following mirror (or mirror systems) exists too for Debian:

Debian heavily rely on various organizations to donate resources (hosting and hardware) to distribute and update Debian. Compiling above information made me thankful to all these organizations. Many thanks to DSA and mirror team as well for managing these stuffs.

I relied heavily on https://db.debian.org/machines.cgi which seems to be manually updated, so things might have changed along the way. If anything looks amiss, feel free to ping.

23 December, 2024 03:32PM

hackergotchi for Joey Hess

Joey Hess

the twenty-fifth year of my free software career

I've been lucky to be able to spend twenty! five! years! developing free software and making a living on it, and this was a banner year for that career.

To start with, there was the Distribits conference. There's a big ecosystem of tools and projects that are based on git-annex, especially in scientific data management, and this was the first conference focused on that. Basically every talk involved git-annex in some way. It's been a while since I was at a conference where my software was in the center like that -- reminded me of Debconf days.

I gave a talk on how git-annex was probably basically feature complete. I have been very busy ever since adding new features to it, because in mapping out git-annex's feature set, I discovered new possibilities.

Meeting people and getting a better feel for the shape of that ecosytem, both technically and funding wise, led to several big developments in funding later in the year. Going into the year, I had an ongoing source of funding from several projects at Dartmouth that use git-annex, but after 10 years, some of that was winding up.

That all came together in my essentially writing a grant proposal to the OpenNeuro project at Stanford, to spend 6 months building out a whole constellation of features. The summer became a sprint to get it all done. Signficant amounts of very productive design work were done while swimming in the river. That was great.

(Somehow in there, I ended up onstage at FOSSY in Portland, in a keynote panel on Open Source and AI. This required developing a nuanced understanding of the mess of the OSI's Open Source AI definition, but I was mostly on the panel as the unqualified guy.)

Capping off the year, I have a new maintenance contract with Forschungszentrum Jülich. This covers the typical daily grind kind of tasks, like bug triage, keeping on top of security, release preparation, and updating dependencies, which is the kind of thing I've never been able to find dedicated funding for before.

A career in free software is a succession of hurdles. How to do something new and worthwhile? How to make any income while developing it at all? How to maintain your independant vision when working on it for hire? How to deal with burn-out? How to grow a project to be more than a one developer affair? And on and on.

How does a free software project keep paying the bills once it's feature complete? Maybe I am starting to get a glimpse of an answer.

23 December, 2024 02:57PM

hackergotchi for Thomas Lange

Thomas Lange

Happy Birthday FAI!

A Brief History of FAI, Which Began 25 Years Ago

On Dec 21st, 1999 version 1.0 of FAI (Fully Automatic Installation) was announced. That was 25 years ago.

Some months before, the computer science department of the University of Cologne bought a small HPC cluster with 16 nodes (each with dual CPU Pentium II 400Mhz, 256 MB RAM) and I was too lazy to install those nodes manually. That's why I started the FAI project. With FAI you can install computers in a few minutes from scratch to a machine with a custom configuration that is ready to go for their users.

At that time Debian 2.1 aka slink was using kernel 2.0.36 and it was the first release using apt. Many things have happened since then.

In the beginning we wrote the first technical report about FAI and a lot of documentation were added afterwards. I gave more than 45 talks about FAI all over the world. Over the past 25 years, there has been an average of more than one commit per day to the FAI software repository.

Several top500.org HPC clusters were built using FAI and many companies are using FAI for their IT infrastructure or deploying Linux on their products using FAI. An overview of users can be found here.

Some major milestones of FAI are listed in the blog post of the 20th anniversary.

What Happended in the Last 5 Years?

  • Live images can be created
  • Writeable data partition on USB sticks
  • FAIme web service creates custom live ISOs
  • Support for Alpine Linux and Arch Linux package managers
  • Automatic detect a local config space
  • Live and installation images for Debian for new hardware using a backports kernel or using the Debian testing release
  • The FAIme web services created more than 30.000 customized ISOs

Currently, I'm preparing for the next FAI release and I still have ideas for new features.

Thanks for all the feedback from you, which helped a lot in making FAI a successful project.

About FAI

FAI is a tool for unattended mass deployment of Linux. It's a system to install and configure Linux systems and software packages on computers as well as virtual machines, from small labs to large-scale infrastructures like clusters and cloud environments. You can take one or more virgin PC's, turn on the power, and after a few minutes, the systems are installed, and completely configured to your exact needs, without any interaction necessary.

23 December, 2024 11:45AM

Russ Allbery

Review: The House That Walked Between Worlds

Review: The House That Walked Between Worlds, by Jenny Schwartz

Series: Uncertain Sanctuary #1
Publisher: Jenny Schwartz
Copyright: 2020
Printing: September 2024
ASIN: B0DBX6GP8Z
Format: Kindle
Pages: 215

The House That Walked Between Worlds is the first book of a self-published trilogy of... hm. Space fantasy? Pure fantasy with a bit of science fiction thrown in for flavor? Something like that. I read it as part of the Uncertain Sanctuary omnibus, which is reflected in the sidebar metadata.

Kira Aist is a doctor. She's also a witch and a direct descendant of Baba Yaga. Her Russian grandmother warned her to never use magic and never reveal who she was because people would hunt her and her family if she did. She broke the rule to try to save a child, her grandmother was right, and now multiple people are dead, including her parents. As the story opens, she's deep in the wilds of New Zealand in a valley with buried moa bones, summoning her House so that she can flee Earth.

Kira's first surprise is that her House is not the small hut that she was expecting from childhood visits to Baba Yaga. It's larger. A lot larger: an obsidian castle with nine towers and legs that resemble dragons rather than the moas whose magic she drew on. Her magic apparently had a much different idea of what she needs than she did.

Her second surprise is that her magical education is highly incomplete, and she is not the witch that she thought she was. Her ability to create a House means that she's a sorcerer, the top tier of magical power in a hierarchy about which she knows essentially nothing. Thankfully the House has a library, but Kira has a lot to learn about the universe and her place in it.

I picked this up because the premise sounded a little like the Innkeeper novels, and since another novel in that series does not appear to be immediately forthcoming, I went looking elsewhere for my cozy sentient building fix. The House That Walked Between Worlds is nowhere near as well-written (or, frankly, coherent) as the Innkeeper books, but it did deliver some of the same vibes.

You should know going in that there isn't much in the way of a plot. Schwartz invented an elaborate setting involving archetype worlds inhabited by classes of mythological creatures that in some mystical sense surround a central system called Qaysar. These archetype worlds spawn derived worlds, each of which seems to be its own dimension, although the details are a bit murky to me. The world Kira thinks of as Earth is just one of the universes branched off of an archetypal Earth, and is the only one of those branchings where the main population is human. The other Earth-derived worlds are populated by the Dinosaurians and the Neanderthals. Similarly, there is a Fae world that branches into Elves and Goblins, an Epic world that branches into Shifters, Trolls, and Kobolds, and so forth. Travel between these worlds is normally by slow World Walker Caravans, but Houses break the rules of interdimensional travel in ways that no one entirely understands.

If your eyes are already starting to glaze over, be warned there's a lot of this. The House That Walked Between Worlds is infodumping mixed with vibes, and I think you have to enjoy the setting, or at least the sheer enthusiasm of Schwartz's presentation of it, to get along with this book. The rest of the story is essentially Kira picking up strays: first a dangerous-looking elf cyborg, then a juvenile giant cat (because of course there's a pet fantasy space cat; it's that sort of book), and then a charming martial artist who I'm fairly sure is up to no good. Kira is entirely out of her depth and acting on instinct, which luckily plays into stereotypes of sorcerers as mysterious and unpredictable. It also helps that her magic is roughly "anything she wants to happen, happens."

This is, in other words, not a tightly-crafted story with coherent rules and a sense of risk and danger. It's a book that succeeds or fails almost entirely on how much you like the main characters and enjoy the world-building. Thankfully, I thought the characters were fun, if not (so far) all that deep. Kira deals with her trauma without being excessively angsty and leans into her new situation with a chaotic decisiveness that I found charming. The cyborg elf is taciturn and a bit inscrutable at first, but he grew on me, and thankfully this book does not go immediately to romance. Late in the book, Kira picks up a publicity expert, which was not at all the type of character that I was expecting and which I found delightful.

Most importantly, the House was exactly what I was looking for: impish, protective, mysterious, inhuman, and absurdly overpowered. I adore cozy sentient building stories, so I'm an easy audience for this sort of thing, but I'm already eager to read more about the House.

This is not great writing by any stretch, and you will be unsurprised that it's self-published. If you're expecting the polish and plot coherence of the Innkeeper stories, you'll be disappointed. But if you just want to spend some time with a giant sentient space-traveling mansion inhabited by unlikely misfits, and you don't mind large amounts of space fantasy infodumping, consider giving this a shot. I had fun with it and plan on reading the rest of the omnibus.

Followed by House in Hiding.

Rating: 6 out of 10

23 December, 2024 03:33AM

Simon Josefsson

OpenSSH and Git on a Post-Quantum SPHINCS+

Are you aware that Git commits and tags may be signed using OpenSSH? Git signatures may be used to improve integrity and authentication of our software supply-chain. Popular signature algorithms include Ed25519, ECDSA and RSA. Did you consider that these algorithms may not be safe if someone builds a post-quantum computer?

As you may recall, I have earlier blogged about the efficient post-quantum key agreement mechanism called Streamlined NTRU Prime and its use in SSH and I have attempted to promote the conservatively designed Classic McEliece in a similar way, although it remains to be adopted.

What post-quantum signature algorithms are available? There is an effort by NIST to standardize post-quantum algorithms, and they have a category for signature algorithms. According to wikipedia, after round three the selected algorithms are CRYSTALS-Dilithium, FALCON and SPHINCS+. Of these, SPHINCS+ appears to be a conservative choice suitable for long-term digital signatures. Can we get this to work?

Recall that Git uses the ssh-keygen tool from OpenSSH to perform signing and verification. To refresh your memory, let’s study the commands that Git uses under the hood for Ed25519. First generate a Ed25519 private key:

jas@kaka:~$ ssh-keygen -t ed25519 -f my_ed25519_key -P ""
Generating public/private ed25519 key pair.
Your identification has been saved in my_ed25519_key
Your public key has been saved in my_ed25519_key.pub
The key fingerprint is:
SHA256:fDa5+jmC2+/aiLhWeWA3IV8Wj6yMNTSuRzqUZlIGlXQ jas@kaka
The key's randomart image is:
+--[ED25519 256]--+
|    .+=.E ..     |
|     oo=.ooo     |
|    . =o=+o .    |
|     =oO+o .     |
|     .=+S.=      |
|      oo.o o     |
|     . o  .      |
|    ...o.+..     |
|   .o.o.=**.     |
+----[SHA256]-----+
jas@kaka:~$ cat my_ed25519_key
-----BEGIN OPENSSH PRIVATE KEY-----
b3BlbnNzaC1rZXktdjEAAAAABG5vbmUAAAAEbm9uZQAAAAAAAAABAAAAMwAAAAtzc2gtZW
QyNTUxOQAAACAWP/aZ8hzN0WNRMSpjzbgW1tJXNd2v6/dnbKaQt7iIBQAAAJCeDotOng6L
TgAAAAtzc2gtZWQyNTUxOQAAACAWP/aZ8hzN0WNRMSpjzbgW1tJXNd2v6/dnbKaQt7iIBQ
AAAEBFRvzgcD3YItl9AMmVK4xDKj8NTg4h2Sluj0/x7aSPlhY/9pnyHM3RY1ExKmPNuBbW
0lc13a/r92dsppC3uIgFAAAACGphc0BrYWthAQIDBAU=
-----END OPENSSH PRIVATE KEY-----
jas@kaka:~$ cat my_ed25519_key.pub 
ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIBY/9pnyHM3RY1ExKmPNuBbW0lc13a/r92dsppC3uIgF jas@kaka
jas@kaka:~$ 

Then let’s sign something with this key:

jas@kaka:~$ echo "Hello world!" > msg
jas@kaka:~$ ssh-keygen -Y sign -f my_ed25519_key -n my-namespace msg
Signing file msg
Write signature to msg.sig
jas@kaka:~$ cat msg.sig 
-----BEGIN SSH SIGNATURE-----
U1NIU0lHAAAAAQAAADMAAAALc3NoLWVkMjU1MTkAAAAgFj/2mfIczdFjUTEqY824FtbSVz
Xdr+v3Z2ymkLe4iAUAAAAMbXktbmFtZXNwYWNlAAAAAAAAAAZzaGE1MTIAAABTAAAAC3Nz
aC1lZDI1NTE5AAAAQLmWsq05tqOOZIJqjxy5ZP/YRFoaX30lfIllmfyoeM5lpVnxJ3ZxU8
SF0KodDr8Rtukg2N3Xo80NGvZOzbG/9Aw=
-----END SSH SIGNATURE-----
jas@kaka:~$

Now let’s create a list of trusted public-keys and associated identities:

jas@kaka:~$ echo 'my.name@example.org ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIBY/9pnyHM3RY1ExKmPNuBbW0lc13a/r92dsppC3uIgF' > allowed-signers
jas@kaka:~$ 

Then let’s verify the message we just signed:

jas@kaka:~$ cat msg | ssh-keygen -Y verify -f allowed-signers -I my.name@example.org -n my-namespace -s msg.sig
Good "my-namespace" signature for my.name@example.org with ED25519 key SHA256:fDa5+jmC2+/aiLhWeWA3IV8Wj6yMNTSuRzqUZlIGlXQ
jas@kaka:~$ 

I have implemented support for SPHINCS+ in OpenSSH. This is early work, but I wanted to announce it to get discussion of some of the details going and to make people aware of it.

What would a better way to demonstrate SPHINCS+ support in OpenSSH than by validating the Git commit that implements it using itself?

Here is how to proceed, first get a suitable development environment up and running. I’m using a Debian container launched in a protected environment using podman.

jas@kaka:~$ podman run -it --rm debian:stable

Then install the necessary build dependencies for OpenSSH.

# apt-get update 
# apt-get install git build-essential autoconf libz-dev libssl-dev

Now clone my OpenSSH branch with the SPHINCS+ implentation and build it. You may browse the commit on GitHub first if you are curious.

# cd
# git clone https://github.com/jas4711/openssh-portable.git -b sphincsp
# cd openssh-portable
# autoreconf -fvi
# ./configure
# make

Configure a Git allowed signers list with my SPHINCS+ public key (make sure to keep the public key on one line with the whitespace being one ASCII SPC character):

# mkdir -pv ~/.ssh
# echo 'simon@josefsson.org ssh-sphincsplus@openssh.com AAAAG3NzaC1zcGhpbmNzcGx1c0BvcGVuc3NoLmNvbQAAAECI6eacTxjB36xcPtP0ZyxJNIGCN350GluLD5h0KjKDsZLNmNaPSFH2ynWyKZKOF5eRPIMMKSCIV75y+KP9d6w3' > ~/.ssh/allowed_signers
# git config gpg.ssh.allowedSignersFile ~/.ssh/allowed_signers

Then verify the commit using the newly built ssh-keygen binary:

# PATH=$PWD:$PATH
# git log -1 --show-signature
commit ce0b590071e2dc845373734655192241a4ace94b (HEAD -> sphincsp, origin/sphincsp)
Good "git" signature for simon@josefsson.org with SPHINCSPLUS key SHA256:rkAa0fX0lQf/7V7QmuJHSI44L/PAPPsdWpis4nML7EQ
Author: Simon Josefsson <simon@josefsson.org>
Date:   Tue Dec 3 18:44:25 2024 +0100

    Add SPHINCS+.

# git verify-commit ce0b590071e2dc845373734655192241a4ace94b
Good "git" signature for simon@josefsson.org with SPHINCSPLUS key SHA256:rkAa0fX0lQf/7V7QmuJHSI44L/PAPPsdWpis4nML7EQ
# 

Yay!

So what are some considerations?

SPHINCS+ comes in many different variants. First it comes with three security levels approximately matching 128/192/256 bit symmetric key strengths. Second choice is between the SHA2-256, SHAKE256 (SHA-3) and Haraka hash algorithms. Final choice is between a “robust” and a “simple” variant with different security and performance characteristics. To get going, I picked the “sphincss256sha256robust” SPHINCS+ implementation from SUPERCOP 20241022. There is a good size comparison table in the sphincsplus implementation, if you want to consider alternative variants.

SPHINCS+ public-keys are really small, as you can see in the allowed signers file. This is really good because they are handled by humans and often by cut’n’paste.

What about private keys? They are slightly longer than Ed25519 private keys but shorter than typical RSA private keys.

# ssh-keygen -t sphincsplus -f my_sphincsplus_key -P ""
Generating public/private sphincsplus key pair.
Your identification has been saved in my_sphincsplus_key
Your public key has been saved in my_sphincsplus_key.pub
The key fingerprint is:
SHA256:4rNfXdmLo/ySQiWYzsBhZIvgLu9sQQz7upG8clKziBg root@ad600ff56253
The key's randomart image is:
+[SPHINCSPLUS 256-+
| .  .o           |
|o . oo.          |
| = .o.. o        |
|o o  o o . .   o |
|.+    = S o   o .|
|Eo=  . + . . .. .|
|=*.+  o . . oo . |
|B+=    o o.o. .  |
|o*o   ... .oo.   |
+----[SHA256]-----+
# cat my_sphincsplus_key.pub 
ssh-sphincsplus@openssh.com AAAAG3NzaC1zcGhpbmNzcGx1c0BvcGVuc3NoLmNvbQAAAEAltAX1VhZ8pdW9FgC+NdM6QfLxVXVaf1v2yW4v+tk2Oj5lxmVgZftfT37GOMOlK9iBm9SQHZZVYZddkEJ9F1D7 root@ad600ff56253
# cat my_sphincsplus_key 
-----BEGIN OPENSSH PRIVATE KEY-----
b3BlbnNzaC1rZXktdjEAAAAABG5vbmUAAAAEbm9uZQAAAAAAAAABAAAAYwAAABtzc2gtc3
BoaW5jc3BsdXNAb3BlbnNzaC5jb20AAABAJbQF9VYWfKXVvRYAvjXTOkHy8VV1Wn9b9slu
L/rZNjo+ZcZlYGX7X09+xjjDpSvYgZvUkB2WVWGXXZBCfRdQ+wAAAQidiIwanYiMGgAAAB
tzc2gtc3BoaW5jc3BsdXNAb3BlbnNzaC5jb20AAABAJbQF9VYWfKXVvRYAvjXTOkHy8VV1
Wn9b9sluL/rZNjo+ZcZlYGX7X09+xjjDpSvYgZvUkB2WVWGXXZBCfRdQ+wAAAIAbwBxEhA
NYzITN6VeCMqUyvw/59JM+WOLXBlRbu3R8qS7ljc4qFVWUtmhy8B3t9e4jrhdO6w0n5I4l
mnLnBi2hJbQF9VYWfKXVvRYAvjXTOkHy8VV1Wn9b9sluL/rZNjo+ZcZlYGX7X09+xjjDpS
vYgZvUkB2WVWGXXZBCfRdQ+wAAABFyb290QGFkNjAwZmY1NjI1MwECAwQ=
-----END OPENSSH PRIVATE KEY-----
# 

Signature size? Now here is the challenge, for this variant the size is around 29kb or close to 600 lines of base64 data:

# git cat-file -p ce0b590071e2dc845373734655192241a4ace94b | head -10
tree ede42093e7d5acd37fde02065a4a19ac1f418703
parent 826483d51a9fee60703298bbf839d9ce37943474
author Simon Josefsson <simon@josefsson.org> 1733247865 +0100
committer Simon Josefsson <simon@josefsson.org> 1734907869 +0100
gpgsig -----BEGIN SSH SIGNATURE-----
 U1NIU0lHAAAAAQAAAGMAAAAbc3NoLXNwaGluY3NwbHVzQG9wZW5zc2guY29tAAAAQIjp5p
 xPGMHfrFw+0/RnLEk0gYI3fnQaW4sPmHQqMoOxks2Y1o9IUfbKdbIpko4Xl5E8gwwpIIhX
 vnL4o/13rDcAAAADZ2l0AAAAAAAAAAZzaGE1MTIAAHSDAAAAG3NzaC1zcGhpbmNzcGx1c0
 BvcGVuc3NoLmNvbQAAdGDHlobgfgkKKQBo3UHmnEnNXczCMNdzJmeYJau67QM6xZcAU+d+
 2mvhbksm5D34m75DWEngzBb3usJTqWJeeDdplHHRe3BKVCQ05LHqRYzcSdN6eoeZqoOBvR
# git cat-file -p ce0b590071e2dc845373734655192241a4ace94b | tail -5 
 ChvXUk4jfiNp85RDZ1kljVecfdB2/6CHFRtxrKHJRDiIavYjucgHF1bjz0fqaOSGa90UYL
 RZjZ0OhdHOQjNP5QErlIOcZeqcnwi0+RtCJ1D1wH2psuXIQEyr1mCA==
 -----END SSH SIGNATURE-----

Add SPHINCS+.
# git cat-file -p ce0b590071e2dc845373734655192241a4ace94b | wc -l
579
# 

What about performance? Verification is really fast:

# time git verify-commit ce0b590071e2dc845373734655192241a4ace94b
Good "git" signature for simon@josefsson.org with SPHINCSPLUS key SHA256:rkAa0fX0lQf/7V7QmuJHSI44L/PAPPsdWpis4nML7EQ

real	0m0.010s
user	0m0.005s
sys	0m0.005s
# 

On this machine, verifying an Ed25519 signature is a couple of times slower, and needs around 0.07 seconds.

Signing is slower, it takes a bit over 2 seconds on my laptop.

# echo "Hello world!" > msg
# time ssh-keygen -Y sign -f my_sphincsplus_key -n my-namespace msg
Signing file msg
Write signature to msg.sig

real	0m2.226s
user	0m2.226s
sys	0m0.000s
# echo 'my.name@example.org ssh-sphincsplus@openssh.com AAAAG3NzaC1zcGhpbmNzcGx1c0BvcGVuc3NoLmNvbQAAAEAltAX1VhZ8pdW9FgC+NdM6QfLxVXVaf1v2yW4v+tk2Oj5lxmVgZftfT37GOMOlK9iBm9SQHZZVYZddkEJ9F1D7' > allowed-signers
# cat msg | ssh-keygen -Y verify -f allowed-signers -I my.name@example.org -n my-namespace -s msg.sig
Good "my-namespace" signature for my.name@example.org with SPHINCSPLUS key SHA256:4rNfXdmLo/ySQiWYzsBhZIvgLu9sQQz7upG8clKziBg
# 

Welcome to our new world of Post-Quantum safe digital signatures of Git commits, and Happy Hacking!

23 December, 2024 12:44AM by simon

December 22, 2024

hackergotchi for Steinar H. Gunderson

Steinar H. Gunderson

Kernel adventures: When two rights make a wrong

My 3D printer took me on another adventure recently. Or, well, actually someone else's 3D printer did: It turns out that building a realtime system (with high-speed motors controlling to a 300-degree metal rod) by cobbling together a bunch of Python and JavaScript on an anemic Arm SoC with zero resource isolation doesn't always meet those realtime guarantees. So in particular after installing a bunch of plugins, people would report the infamous “MCU timer too close” Klipper error, which essentially means that the microcontroller didn't get new commands in time from the Linux host and shut down as a failsafe. (Understandably, this sucks if it happens in the middle of an eight-hour print. Nobody really invented a way to reliably resume from these things yet.)

I was wondering whether it was possible to provoke this and then look at what was actually going on in the scheduler; perf sched lets you look at scheduling history on the host, so if I could reproduce the error while collecting data, I could go in afterwards and see what was the biggest CPU hog, or at least that was the theory.

However, to my surprise, perf sched record died with an error essentially saying that the kernel was compiled without ftrace support (which is needed for the scheduler hooks; it's somewhat possible to do without by just doing a regular profile, but that's a different story and much more annoying). Not very surprising, these things tend to run stone-age vendor kernels from some long-forgotten branch with zero security support and seemingly no ftrace.

Now, I did not actually run said vendor kernel; at some point, I upgraded to the latest stable kernel (6.6) from Armbian, which is still far from mainline (for one, it needs to carry out-of-tree drivers to make wireless work at all) but which I trust infinitely more to actually provide updated kernels over time. It doesn't support ftrace either, so I thought the logical step would be to upgrade to the latest “edge” kernel (aka 6.11) and then compile with the right stuff on.

After a couple of hours of compiling (almost nostalgic to have such slow kernel compiles; cross-compiling didn't work for me!), I could boot into the new kernel, and:

[   23.775976] platform 5070400.thermal-sensor: deferred probe pending: platform: wait for supplier 

and then Klipper would refuse to start because it couldn't find the host thermal sensors. (I don't know exactly why it is a hard dependency, but seemingly, it is.) A bit of searching shows that this error message is doubly vexing; it should have said “wait for supplier /i2c@fdd40000/pmic@20/regulators/SWITCH_REG1” or something similar, but ends only in a space and then nothing.

So evidently this has to be something about the device tree (DT), and switching out the new DT for the old one didn't work. Bisecting was also pretty much out of the question (especially with 400+ patches that go on top of the git tree), but after a fair bit of printk debugging and some more reading, I figured out what had happened:

First, the sun8i-thermal driver, which had been carried out-of-tree in Armbian, had gone into mainline. But it was in a slightly different version; while the out-of-tree version used previously (in Armbian's 6.6 kernel) had relied on firmware (run as part of U-Boot, as I understand it) to set a special register bit, the mainline version would be stricter and take care to set it itself. I don't really know what the bit does, short of “if you don't set it, all the values you get back are really crazy”, so this is presumably a good change. So the driver would set a bit in a special memory address somewhere (sidenote: MMIO will always feel really weird to me; like, some part of the CPU has to check all memory accesses in case they're really not to RAM at all?), and for that, the thermal driver would need to take on a DT reference to the allwinner,sram (comma is evidently some sort of hierarchical separator) node so that it could get its address. Like, in case it was moved around in future SoCs or something.

Second, there was an Armbian patch that dealt with exactly these allwinner,sram nodes in another way; it would make sure that references to them would cause devlink references between the nodes. I don't know what those are either, but it seems the primary use case is for waiting: If you have a dependency from A to B, then A's initialization will wait until B is ready. The configuration bit in question is always ready, but I guess it's cleaner somehow, and you get a little symlink somewhere in /sys to explain the relationship, so perhaps it's good? But that's what the error message means; “A: deferred probe pending: wait for supplier B” means that we're not probing for A's existence yet, because it wants B to supply something and B isn't ready yet.

But why is the relationship broken? Well, for that, we need to look at how the code in the patch looks:

        sram_node = of_parse_phandle(np, prop_name, 0);
        sram_node = of_get_parent(sram_node);
        sram_node = of_get_parent(sram_node);

        return sram_node;

And how the device tree is set up in this case (lots of irrelevant stuff removed for clarity):

        bus@1000000 {  /* this works */
                reg = <0x1000000 0x400000>;
                allwinner,sram = <&de3_sram 1>;
        };
        ths: thermal-sensor@5070400 {  /* this doesn't */
                allwinner,sram = <&syscon>;
        };
        syscon: syscon@3000000 {
                sram_c: sram@28000 {
                        de3_sram: sram-section@0 {
                                reg = <0x0000 0x1e000>;
                        };
                };
        };

So that explains it; the code expects that all DT references are to a child of a child of syscon to find the supplier, and just goes up two levels to find it. But for the thermal sensor, the reference is directly to the syscon itself, and it goes up past the root of the tree, which is, well, NULL. And then the error message doesn't have a node name to print out, and the dependency just fails forever.

So that's two presumably good changes that just interacted in a really bad way (in particular, due to too little flexibility in the second one). A small patch later, and the kernel boots with thermals again!

Oh, and those scheduling issues I wanted to debug? I never managed to reliably reproduce them; I have seen them, but they're very rare for me. I guess that upstream for the plugins in question just made things a bit less RAM-hungry in the meantime, or that having a newer kernel improves things enough in itself. Shrug. :-)

22 December, 2024 08:50AM

hackergotchi for Junichi Uekawa

Junichi Uekawa

Looking at my private repositories for what language I wrote.

Looking at my private repositories for what language I wrote. I'm counting the number of days I wrote a certain file with specific file extension. Markdown was at the top, because I usually have a doc, that's fine. C++ is my top language. Then lilypond and then js. Rust came much lower. Lilypond was for my band scores it seems.

22 December, 2024 07:08AM by Junichi Uekawa

Russ Allbery

Review: Beyond the Fringe

Review: Beyond the Fringe, by Miles Cameron

Series: Arcana Imperii #1.5
Publisher: Gollancz
Copyright: 2023
ISBN: 1-3996-1537-8
Format: Kindle
Pages: 173

Beyond the Fringe is a military science fiction short story collection set in the same universe as Artifact Space. It is intended as a bridge between that novel and its sequel, Deep Black.

Originally I picked this up for exactly the reason it was published: I was eagerly awaiting Deep Black and thought I'd pass the time with some filler short fiction. Then, somewhat predictably, I didn't get around to reading it until after Deep Black was already out. I still read this collection first, partly because I'm stubborn about reading things in publication order but mostly to remind myself of what was going on in Artifact Space before jumping into the sequel.

My stubbornness was satisfied. My memory was not; there's little to no background information here, and I had to refresh my memory of the previous book anyway to figure out the connections between these stories and the novel.

My own poor decisions aside, these stories are... fine, I guess? They're competent military SF short fiction, mostly more explicitly military than Artifact Space. All of them were reasonably engaging. None of them were that memorable or would have gotten me to read the series on their own. They're series filler, in other words, offering a bit of setup for the next novel but not much in the way of memorable writing or plot.

If you really want more in this universe, this exists, but my guess (not having read Deep Black) is that it's entirely skippable.

"Getting Even": A DHC paratrooper lands on New Shenzen, a planet that New Texas is trying to absorb into the empire it is attempting to build. He gets captured by one group of irregulars and then runs into another force with an odd way of counting battle objectives.

I think this exists because Cameron wanted to tell a version of a World War II story he'd heard, but it's basically a vignette about a weird military unit with no real conclusion, and I am at a loss as to the point of the story. There isn't even much in the way of world-building. I'm probably missing something, but I thought it was a waste of time. (4)

"Partners": The DHC send a planetary exobiologist to New Texas as a negotiator. New Texas is aggressively, abusively capitalist and is breaking DHC regulations on fair treatment of labor. Why send a planetary exobiologist is unclear (although probably ties into the theme of this collection that the reader slowly pieces together); maybe it's because he's originally from New Texas, but more likely it's because of his partner. Regardless, the New Texas government are exploitative assholes with delusions of grandeur, so the negotiations don't go very smoothly.

This was my favorite story of the collection just because I enjoy people returning rudeness and arrogance to sender, but like a lot of stories in this collection it doesn't have much of an ending. I suspect it's mostly setup for Deep Black. (7)

"Dead Reckoning": This is the direct fallout of the previous story and probably has the least characterization of this collection. It covers a few hours of a merchant ship having to make some fast decisions in a changing political situation. The story is framed around a veteran spacer and his new apprentice, although even that frame is mostly dropped once the action starts. It was suspenseful and enjoyable enough while I was reading it, but it's the sort of story that you forget entirely after it's over. (6)

"Trade Craft": Back on a planet for this story, which follows an intelligence agent on a world near but not inside New Texas's area of influence. I thought this was one of the better stories of the collection even though it's mostly action. There are some good snippets of characterization, an interesting mix of characters, and some well-written tense scenes. Unfortunately, I did not enjoy the ending for reasons that would be spoilers. Otherwise, this was good but forgettable. (6)

"One Hour": This is the first story with a protagonist outside of the DHC and its associates. It instead follows a PTX officer (PTX is a competing civilization that features in Artifact Space) who has suspicions about what his captain is planning and recruits his superior officer to help him do something about it.

This is probably the best story in the collection, although I personally enjoyed "Partners" a smidgen more. Shunfu, the first astrogator who is recruited by the protagonist, is a thoroughly enjoyable character, and the story is tense and exciting all the way through. For series readers, it also adds some depth to events in Artifact Space (if the reader remembers them), and I suspect will lead directly into Deep Black. (7)

"The Gifts of the Magi": A kid and his mother, struggling asteroid miners with ancient and malfunctioning equipment, stumble across a DHC ship lurking in the New Texas system for a secret mission. This is a stroke of luck for the miners, since the DHC is happy to treat the serious medical problems of the mother without charging unaffordable fees the way that the hyper-capitalist New Texas doctors would. It also gives the reader a view into DHC's covert monitoring of the activities of New Texas that all the stories in this collection have traced.

As you can tell from the title, this is a Christmas story. The crew of the DHC ship is getting ready to celebrate Alliday, which they claim rolls all of the winter holidays into one. Just like every other effort to do this, no, it does not, it just subsumes them all into Christmas with some lip service to other related holidays. I am begging people to realize that other religions often do not have major holidays in December, and therefore you cannot include everyone by just declaring December to be religious holiday time and thinking that will cover it.

There is the bones of an interesting story here. The covert mission setup has potential, the kid and his mother are charming if cliched, there's a bit of world-building around xenoglas (the magical alien material at the center of the larger series plot), and there's a lot of foreshadowing for Deep Black. Unfortunately, this is too obviously a side story and a setup story: none of this goes anywhere satisfying, and along the way the reader has to endure endless rather gratuitous Christmas references, such as the captain working on a Nutcracker ballet performance for the ship talent show.

This isn't bad, exactly, but it rubbed me the wrong way. If you love Christmas stories, you may find it more agreeable. (5)

Rating: 6 out of 10

22 December, 2024 03:17AM

December 21, 2024

hackergotchi for Benjamin Mako Hill

Benjamin Mako Hill

Thug Life

My current playlist is this diorama of Lulu the Piggy channeling Tupac Shakur in a toy vending machine in the basement of New World Mall in Flushing Chinatown.

21 December, 2024 11:06PM by Benjamin Mako Hill

hackergotchi for Dirk Eddelbuettel

Dirk Eddelbuettel

anytime 0.3.11 on CRAN: Maintenance

A follow-up release 0.3.11 to the recent 0.3.10 release release of the anytime package arrived on CRAN two days ago. The package is fairly feature-complete, and code and functionality remain mature and stable, of course.

anytime is a very focused package aiming to do just one thing really well: to convert anything in integer, numeric, character, factor, ordered, … input format to either POSIXct (when called as anytime) or Date objects (when called as anydate) – and to do so without requiring a format string as well as accomodating different formats in one input vector. See the anytime page, or the GitHub repo for a few examples, and the beautiful documentation site for all documentation.

This release simply skips one test file. CRAN labeled an error ‘M1mac’ yet it did not reproduce on any of the other M1 macOS I can access (macbuilder, GitHub Actions) as this appeared related to a local setting of timezone values I could not reproduce anywwhere. So the only way to get rid of the ‘fail’ is to … not to run the test. Needless to say the upload process was a little tedious as I got the passive-aggressive ‘not responding’ treatment on a first upload and the required email answer it lead to. Anyway, after a few days, and even more deep breaths, it is taken care of and now the package result standing is (at least currently) pristinely clean.

The short list of changes follows.

Changes in anytime version 0.3.11 (2024-12-18)

  • Skip a test file

Courtesy of my CRANberries, there is also a diffstat report of changes relative to the previous release. The issue tracker tracker off the GitHub repo can be use for questions and comments. More information about the package is at the package page, the GitHub repo and the documentation site.

This post by Dirk Eddelbuettel originated on his Thinking inside the box blog. If you like this or other open-source work I do, you can now sponsor me at GitHub.

21 December, 2024 08:35PM

hackergotchi for Joey Hess

Joey Hess

aiming at December

I have been working all year on a solar upgrade aimed at December. Now here it is, midwinter, and my electric car is charging on a cloudy day from my offgrid solar fence.

I lived happily enough with 1 kilowatt of solar that I installed in 2017. Meanwhile, solar panel prices came down massively, incentives increased and everything came together: This was the year.

In the spring I started clearing forest trees that were leaning over the house, making both a firebreak and a solar field.

In June I picked up a pallet of panels in a box truck.

a porch with a a bunch of solar panels, stacked on edge leaning up against the wall. A black and white cat is sprawled in front of them.

In August I bought the EV and was able to charge it offgrid from my old solar system... a few miles per day on the most sunny days.

In September and October I built a solar fence, of my own design.

Me standing in front of the solar fence, which is 10 panels long

For the past several weeks I have been installing additional solar panels on ballasted ground mounts full of gravel. At this point I'm half way through installing my 30 panel upgrade.

The design goal of my 12 kilowatt system is to produce 1 kilowatt of power all day on a cloudy day in midwinter, which allows swapping between major loads (EV charger, hot water heater, etc) on a cloudy day and running everything on a sunny day. So the size of the battery bank doesn't matter much. Batteries are getting cheaper fast too, but they are a wear item, so it's better to oversize the solar system and minimize the battery.

A lot of this is nonstandard and experimental. And that makes sense with the price of solar panels. It costs more to mount solar panels now than the panels are worth. And non-ideal panel orientation isn't a problem when the system is massively overpaneled.

I'm hoping to finish up the install before the end of winter. I have more trees to clear, more ballasted ground mounts to install, and need to come up with something even more experimental for a half dozen or so panels. Using solar panels as mounts for solar panels? Hanging them from trees?

Soon the wan light will fade, time to head off to the solstice party to enjoy the long night, and a bonfire.

Solar fence with some ballasted ground mounts in front of it, late evening light. Old pole mounted solar panels in the foreground are from the 90's.

21 December, 2024 05:00AM

December 20, 2024

hackergotchi for Michael Prokop

Michael Prokop

Grml 2024.12 – codename Adventgrenze

Picture with metrics of three user profiles on GitHub.com, with many contributions especially in the last quarter of the year

We did it again™! Just in time, we’re excited to announce the release of Grml stable version 2024.12, code-named ‘Adventgrenze’! (If you’re not familiar with Grml, it’s a Debian-based live system tailored for system administrators.)

This new release is built on Debian trixie, and for the first time, we’re introducing support for 64-bit ARM CPUs (arm64 architecture)!

I’m incredibly proud of the hard work that went into this release. A significant amount of behind-the-scenes effort went into reworking our infrastructure and redesigning the build process. Special thanks to Chris and Darsha – our Grml developer days in November and December were a blast!

For a detailed overview of the changes between releases 2024.02 and 2024.12, check out our official release announcement. And, as always, after a release comes the next one – exciting improvements are already in the works!

BTW: recently we also celebrated 20(!) years of Grml Releases. If you’re a Grml and or grml-zsh user, please join us in celebrating and send us a postcard!

20 December, 2024 06:05PM by mika

Noah Meyerhans

Local Development VM Management

A coworker asked recently about how people use VMs locally for dev work, so I figured I’d take a few minutes to write up a bit about what I do. There are many use cases for local virtual machines in software development and testing. They’re self-contained, meaning you can make a mess of them without impacting your day-to-day computing environment. They can run different distributions, kernels, and even entirely different operating systems from the one you use regularly. Etc. They’re also cheaper than cloud services and provide finer grained control over the resources.

I figured I’d share a little bit about how I manage different virtual machines in case anybody finds this useful. This is what works for me, but it won’t necessarily work for you, or maybe you’ve already got something better. I’ve found it to be easy to work with, light weight, and is easy to evolve my needs change.

Use short-lived VMs

Rather than keep a long-lived “development” VM around that you customize over time, I recommend automating the common customizations and provisioning new VMs regularly. If I’m working on reproducing a bug or testing a change prior to submitting it upstream, I’ll do this work in a VM and delete the VM when when I’m done. When provisioning VMs this frequently, though, walking through the installation process for every new VM is tedious and a waste of time. Since most of my work is done in Debian, so I start with images generated daily by the cloud team. These images are available for multiple releases and architectures. The ‘nocloud’ variant boots to a root prompt and can be useful directly, or the ‘generic’ images can be used for cloud-init based customization.

Automating image preparation

This makefile lets me do something like make image and get a new qcow2 image with the latest build of a given Debian release (sid by default, with others available by specifying DIST).

DATESTAMP=$(shell date +"%Y-%m-%d")
FLAVOR?=generic
ARCH?=$(shell dpkg --print-architecture)
DIST?=sid
RELEASE=$(DIST)
URL_PATH=https://cloud.debian.org/images/cloud/$(DIST)/daily/latest/
ifeq ($(DIST),trixie)
RELEASE=13
endif
ifeq ($(DIST),bookworm)
RELEASE=12
endif
ifeq ($(DIST),bullseye)
RELEASE=11
endif
debian-$(DIST)-$(FLAVOR)-$(ARCH)-daily.tar.xz:
curl --fail --connect-timeout 20 -LO \
$(URL_PATH)/debian-$(RELEASE)-$(FLAVOR)-$(ARCH)-daily.tar.xz
$(DIST)-$(FLAVOR)-$(DATESTAMP).qcow2: debian-$(RELEASE)-$(FLAVOR)-$(ARCH)-daily.tar.xz
tar xvf debian-$(RELEASE)-$(FLAVOR)-$(ARCH)-daily.tar.xz
qemu-img convert -O qcow2 disk.raw $@
rm -f disk.raw
qemu-img resize $@ 20g
qemu-img snapshot -c untouched $@
image: $(DIST)-$(FLAVOR)-$(DATESTAMP).qcow2
.PHONY: image

Customize the VM environment with cloud-init

While the ‘nocloud’ images can be useful, I typically find that I want to apply the same modifications to each new VM I launch, and they don’t provide facilities for automating this. The ‘generic’ images, on the other hand, run cloud-init by default. Using cloud-init, I can create my user account, point apt at local mirrors, install my preferred tools, ensure the root filesystem is resized to make full use of the backing storage, etc.

The cloud-init configuration on the generic images will read from a local config drive, which can contain an ISO9660 (cdrom) filesystem image. This image can be generated from a subdirectory containing the various cloud-init input files using the following make syntax:

IMDS_FILES=$(shell find seedconfig -path '*/.git/*' \
-prune -o -type f -name '*.in.json' -print) \
seedconfig/openstack/latest/user_data
seed.iso: $(IMDS_FILES)
genisoimage -V config-2 -o $@ -J -R -m '*~' -m '.git' seedconfig

With the image in place, the VM can be created with

 qemu-system-x86_64 -machine q35,accel=kvm
-cpu host -m 4g -drive file=${img},index=0,if=virtio,media=disk
-drive file=seed.iso,media=cdrom,format=raw,index=2,if=virtio
-nic user -nographic

This invokes qemu with the root volume and ISO image attached as disks, uses an emulated “q35” machine with the host’s CPU and KVM acceleration, the userspace network stack, and a serial console. The first time the VM boots, cloud-init will apply the configuration from the cloud-config available in the ISO9660 filesystem.

Alternatives to cloud-init

virt-customize is another tool accomplishing the same type of customization. I use cloud-init because it works directly with cloud providers in addition to local VM images. You could also use something like ansible.

Variations

I have a variant of this that uses a bridged network, which I’ll write more about later. The bridge is nice because it’s more featureful, with full support for IPv6, etc, but it needs a bit more infrastructure in place.

It also can be helpful to use 9p or virtfs to share filesystem state between the host the VM. I don’t tend to rely on these, and will instead use rsync or TRAMP for moving files around.

Containers are also useful, of course, and there are plenty of times when the full isolation of a VM is not worth the overhead.

20 December, 2024 02:40PM by Noah Meyerhans (frodo+blog@morgul.net)

December 19, 2024

hackergotchi for Gregory Colpart

Gregory Colpart

MiniDebConf Toulouse 2024

After the MiniDebConf Marseille 2019, COVID-19 made it impossible or difficult to organize new MiniDebConfs for a few years. With the gradual resumption of in-person events (like FOSDEM, DebConf, etc.), the idea emerged to host another MiniDebConf in France, but with a lighter organizational load. In 2023, we decided to reach out to the organizers of Capitole du Libre to repeat the experience of 2017: hosting a MiniDebConf alongside their annual event in Toulouse in November. However, our request came too late for 2023. After discussions with Capitole du Libre in November 2023 in Toulouse and again in February 2024 in Brussels, we confirmed that a MiniDebConf Toulouse would take place in November 2024!

We then assembled a small organizing team and got to work: a Call for Papers in May 2024, adding a two-day MiniDebCamp, coordinating with the DebConf video team, securing sponsors, creating a logo, ordering T-shirts and stickers, planning the schedule, and managing registrations. Even with lighter logistics (conference rooms, badges, and catering during the weekend were handled by Capitole du Libre), there was still quite a bit of preparation to do.

On Thursday, November 14, and Friday, November 15, 2024, about forty developers arrived from around the world (France, Spain, Italy, Switzerland, Germany, England, Brazil, Uruguay, India, Brest, Marseille…) to spend two days at the MiniDebCamp in the beautiful collaborative spaces of Artilect in Toulouse city center.

Then, on Saturday, November 16, and Sunday, November 17, 2024, the MiniDebConf took place at ENSEEIHT as part of the Capitole du Libre event. The conference kicked off on Saturday morning with an opening session by Jérémy Lecour, which included a tribute to Lunar (Nicolas Dandrimont). This was followed by Reproducible Builds – Rebuilding What is Distributed from ftp.debian.org (Holger Levsen) and Discussion on My Research Work on Sustainability of Debian OS (Eda). After lunch at the Capitole du Libre food trucks, the intense afternoon schedule began: What’s New in the Linux Kernel (and What’s Missing in Debian) (Ben Hutchings), Linux Live Patching in Debian (Santiago Ruano Rincón), Trixie on Mobile: Are We There Yet? (Arnaud Ferraris), PostgreSQL Container Groups, aka cgroups Down the Road (Cédric Villemain), Upgrading a Thousand Debian Hosts in Less Than an Hour (Jérémy Lecour and myself), and Using Debusine to Automate Your QA (Stefano Rivera & co).

Sunday marked the second day, starting with a presentation on DebConf 25 (Benjamin Somers), which will be held in Brest in July 2025. The morning continued with talks: How LTS Goes Beyond LTS (Santiago Ruano Rincón & Roberto C. Sánchez), Cross-Building (Helmut Grohne), and State of JavaScript (Bastien Roucariès). In the afternoon, there were Lightning Talks, PyPI Security: Past, Present & Future (Salvo “LtWorf” Tomaselli), and the classic Bits from DPL (Andreas Tille), before closing with the final session led by Pierre-Elliott Bécue.

All talks are available on video (a huge thanks to the amazing DebConf video team), and many thanks to our sponsors (Viridien, Freexian, Evolix, Collabora, and Data Bene). A big thank-you as well to the entire Capitole du Libre team for hosting and supporting us… see you in Brest in July 2025!

Articles about (or mentioning) MiniDebConf Toulouse:

19 December, 2024 09:18AM by Gregory Colpart

hackergotchi for Benjamin Mako Hill

Benjamin Mako Hill

Being a bread torus

A concerned nutritional epidemiologist in Tokyo realizes that if you are what you eat, that means…

It’s a similar situation in Seoul, albeit with less oil and more confidence.

19 December, 2024 02:49AM by Benjamin Mako Hill

December 18, 2024

Simon Josefsson

Guix Container Images for GitLab CI/CD

I am using GitLab CI/CD pipelines for several upstream projects (libidn, libidn2, gsasl, inetutils, libtasn1, libntlm, …) and a long-time concern for these have been that there is too little testing on GNU Guix. Several attempts have been made, and earlier this year Ludo’ came really close to finish this. My earlier effort to idempotently rebuild Debian recently led me to think about re-bootstrapping Debian. Since Debian is a binary distribution, it re-use earlier binary packages when building new packages. The prospect of re-bootstrapping Debian in a reproducible way by rebuilding all of those packages going back to the beginning of time does not appeal to me. Instead, wouldn’t it be easier to build Debian trixie (or some future release of Debian) from Guix, by creating a small bootstrap sandbox that can start to build Debian packages, and then make sure that the particular Debian release can idempotently rebuild itself in a reproducible way? Then you will eventually end up with a reproducible and re-bootstrapped Debian, which pave the way for a trustworthy release of Trisquel. Fortunately, such an endeavour appears to offer many rabbit holes. Preparing Guix container images for use in GitLab pipelines is one that I jumped into in the last few days, and just came out of.

Let’s go directly to the point of this article: here is a GitLab pipeline job that runs in a native Guix container image that builds libksba after installing the libgpg-error dependency from Guix using the pre-built substitutes.

test-amd64-latest-wget-configure-make-libksba:
  image: registry.gitlab.com/debdistutils/guix/container:latest
  before_script:
  - lndir /gnu/store/*profile/etc/ /etc
  - rm -f /etc/group
  - groupadd --system guixbuild
  - for i in $(seq -w 1 10); do useradd -g guixbuild -G guixbuild -d /var/empty -s $(command -v nologin) -c "Guix build user $i" --system guixbuilder$i; done
  - export HOME=/
  - export LANG=C.UTF-8
  - guix-daemon --disable-chroot --build-users-group=guixbuild &
  - guix archive --authorize < /share/guix/ci.guix.gnu.org.pub
  - guix archive --authorize < /share/guix/bordeaux.guix.gnu.org.pub
  - guix describe
  - guix package -i libgpg-error
  - GUIX_PROFILE="//.guix-profile"
  - . "$GUIX_PROFILE/etc/profile"
  script:
  - wget https://www.gnupg.org/ftp/gcrypt/libksba/libksba-1.6.7.tar.bz2
  - tar xfa libksba-1.6.7.tar.bz2
  - cd libksba-1.6.7
  - ./configure
  - make V=1
  - make check VERBOSE=t V=1

You can put that in a .gitlab-ci.yml and push it to GitLab and you will end up with a nice pipeline job output.

As you may imagine, there are several things that are sub-optimal in the before_script above that ought to be taken care of by the Guix container image, and I hope to be able to remove as much of the ugliness as possible. However that doesn’t change that these images are useful now, and I wanted to announce this work to allow others to start testing them and possibly offer help. I have started to make use of these images in some projects, see for example the libntlm commit for that.

You are welcome to join me in the Guix container images for GitLab CI/CD project! Issues and merge requests are welcome – happy hacking folks!

18 December, 2024 06:43PM by simon

December 17, 2024

hackergotchi for Dirk Eddelbuettel

Dirk Eddelbuettel

BH 1.87.0-1 on CRAN: New Upstream

Boost

Boost is a very large and comprehensive set of (peer-reviewed) libraries for the C++ programming language, containing well over one hundred individual libraries. The BH package provides a sizeable subset of header-only libraries for (easier, no linking required) use by R. It is fairly widely used: the (partial) CRAN mirror logs (aggregated from the cloud mirrors) show over 38.5 million package downloads.

Version 1.87.0 of Boost was released last week following the regular Boost release schedule of April, August and December releases. As before, we packaged it almost immediately and started testing following our annual update cycle which strives to balance being close enough to upstream and not stressing CRAN and the user base too much. The reverse depends check revealed six packages requiring changes or adjustments. We opened issue #103 to coordinate the issue (just as we did in previous years). Our sincere thanks to Matt Fidler who fixed two packages pretty much immediately.

As I had not heard back from the other maintainers since filing the issue, I uploaded the package to CRAN suggesting that the coming winter break may be a good opportunity for the four other packages to catch up. CRAN concurred, and 1.87.0-1 is now available there.

There are no other changes apart from cosmetics in the DESCRIPTION file. For once, we did not add any new Boost libraries. The short NEWS entry follows.

Changes in version 1.87.0-1 (2024-12-17)

  • Upgrade to Boost 1.87.0, patched as usual to comment-out diagnostic suppression messages per the request of CRAN

  • Switched to Authors@R

Via my CRANberries, there is a diffstat report relative to the previous release. Comments and suggestions about BH are welcome via the issue tracker at the GitHub repo.

This post by Dirk Eddelbuettel originated on his Thinking inside the box blog. If you like this or other open-source work I do, you can now sponsor me at GitHub.

17 December, 2024 10:34PM

hackergotchi for Gunnar Wolf

Gunnar Wolf

The science of detecting LLM-generated text

This post is a review for Computing Reviews for The science of detecting LLM-generated text , a article published in Communications of the ACM

While artificial intelligence (AI) applications for natural language processing (NLP) are no longer something new or unexpected, nobody can deny the revolution and hype that started, in late 2022, with the announcement of the first public version of ChatGPT. By then, synthetic translation was well established and regularly used, many chatbots had started attending users’ requests on different websites, voice recognition personal assistants such as Alexa and Siri had been widely deployed, and complaints of news sites filling their space with AI-generated articles were already commonplace. However, the ease of prompting ChatGPT or other large language models (LLMs) and getting extensive answers–its text generation quality is so high that it is often hard to discern whether a given text was written by an LLM or by a human–has sparked significant concern in many different fields. This article was written to present and compare the current approaches to detecting human- or LLM-authorship in texts.

The article presents several different ways LLM-generated text can be detected. The first, and main, taxonomy followed by the authors is whether the detection can be done aided by the LLM’s own functions (“white-box detection”) or only by evaluating the generated text via a public application programming interface (API) (“black-box detection”).

For black-box detection, the authors suggest training a classifier to discern the origin of a given text. Although this works at first, this task is doomed from its onset to be highly vulnerable to new LLMs generating text that will not follow the same patterns, and thus will probably evade recognition. The authors report that human evaluators find human-authored text to be more emotional and less objective, and use grammar to indicate the tone of the sentiment that should be used when reading the text–a trait that has not been picked up by LLMs yet. Human-authored text also tends to have higher sentence-level coherence, with less term repetition in a given paragraph. The frequency distribution for more and less common words is much more homogeneous in LLM-generated texts than in human-written ones.

White-box detection includes strategies whereby the LLMs will cooperate in identifying themselves in ways that are not obvious to the casual reader. This can include watermarking, be it rule based or neural based; in this case, both processes become a case of steganography, as the involvement of a LLM is explicitly hidden and spread through the full generated text, aiming at having a low detectability and high recoverability even when parts of the text are edited.

The article closes by listing the authors’ concerns about all of the above-mentioned technologies. Detecting an LLM, be it with or without the collaboration of the LLM’s designers, is more of an art than a science, and methods deemed as robust today will not last forever. We also cannot assume that LLMs will continue to be dominated by the same core players; LLM technology has been deeply studied, and good LLM engines are available as free/open-source software, so users needing to do so can readily modify their behavior. This article presents itself as merely a survey of methods available today, while also acknowledging the rapid progress in the field. It is timely and interesting, and easy to follow for the informed reader coming from a different subfield.

17 December, 2024 11:23AM

Russ Allbery

Review: Iris Kelly Doesn't Date

Review: Iris Kelly Doesn't Date, by Ashley Herring Blake

Series: Bright Falls #3
Publisher: Berkley Romance
Copyright: October 2023
ISBN: 0-593-55058-7
Format: Kindle
Pages: 381

Iris Kelly Doesn't Date is a sapphic romance novel (probably a romantic comedy, although I'm bad at romance subgenres). It is the third book in the Bright Falls series. In the romance style, it has a new set of protagonists, but the protagonists of the previous books appear as supporting characters and reading this will spoil the previous books.

Among the friend group we were introduced to in Delilah Green Doesn't Care, Iris was the irrepressible loudmouth. She's bad at secrets, good at saying whatever is on her mind, and has zero desire to either get married or have children. After one of the side plots of Astrid Parker Doesn't Fail, she has sworn off dating entirely.

Iris is also now a romance novelist. Her paper store didn't get enough foot traffic to justify staying open, so she switched her planner business to online only and wrote a romance novel that was good enough to get a two-book deal. Now she needs to write a second book and she has absolutely nothing. Her own avoidance of romantic situations is not helping, but neither is her meddling family who are convinced her choices about marriage and family can be overturned with sufficient pestering. She desperately needs to shake up her life, get out of her creative rut, and do something new. Failing that, she'll settle for meeting someone in a bar and having some fun.

Stevie is a barista and actress living in Portland. Six months ago, she broke up with Adri, her creative partner, girlfriend of six years, and the first person with whom she had a serious relationship. More precisely, Adri broke up with her. They're still friends, truly, even though that friendship is being seriously strained by Adri dating Vanessa, another member of their small and close-knit friend group. Stevie has occasionally-crippling anxiety, not much luck in finding real acting roles in Portland, and a desperate desire to not make waves. Ren, the fourth member of their friend group, thinks Stevie needs a new relationship, or at least a fling. That's how Stevie, with Ren as backup and encouragement, ends up at the same bar with Iris.

The resulting dance and conversation was rather fun for both Stevie and Iris. The attempted one-night stand afterwards was a disaster due to Stevie's anxiety, and neither of them expected to see the other again. Stevie therefore felt safe pretending they'd hit it off to get her friends off her back. When Iris's continued restlessness lands her in an audition for Adri's fundraiser play that she also talked Stevie into performing in, this turns into a full-blown fake dating trope.

These books continue to be impossible to put down. I'm not sure what Blake is doing to make the pacing so perfect, but as with the previous books of the series I found this utterly compulsive reading. I started it in the afternoon, took a break in the evening for a few hours, and then finished it at 2am.

I wasn't sure if a book focused on Iris would work as well, but I need not have worried. Iris Kelly Doesn't Date is both more dramatic and more trope-centered than the earlier books, but Blake handles that in a way that fits Iris's personality and wasn't annoying even to a reader like me, who has an aversion to many types of relationship drama. The secret is Stevie, and specifically having the other protagonist be someone with severe anxiety.

No was never a very easy word for Stevie when it came to Adri, when it came to anyone, really. She could handle the little stuff — do you want a soda, have you seen this movie, do you like onions on your pizza — but the big stuff, the stuff that caused disappointed expressions and down-turned mouths... yeah, she sucked at that part. Her anxiety would flare, and she'd spend the next week convinced her friends hated her, she'd die alone and miserable, and wasn't worth a damn to anyone. Then, when said friend or family member eventually got ahold of her to tell her that, no, of course they didn't hate her, why in the world would she think that, her anxiety would crest once again, convincing her that she was terrible at understanding people and could never trust her own brain to make heads or tails of any social situation.

This is a spot-on description of a particular type of anxiety, but also this is the perfect protagonist to pair with Iris. Throughout the series, Iris has always been the ride-or-die friend, the person who may have no idea how to help but who will show up anyway and at least try to distract you. Stevie's anxiety makes Iris feel protective, which reveals one of the best sides of Iris's personality, and then the protectiveness plays off against Iris's own relationship issues and tendency to avoid taking anything too seriously. It's one of those relationships that starts a bit one-sided and then becomes mutually supporting once Stevie gets her feet under her. That's a relationship pattern I really enjoy reading about.

As with the rest of the series, the friendship dynamics are great. Here we get to see two friend groups at work: Iris's, which we've seen in the previous two volumes and which expanded interestingly in Astrid Parker Doesn't Fail, and Stevie's, which is new. I liked all of these people, even Adri in her own way (although she's the hardest to like). The previous happily-ever-afters do get a bit awkward here, but Blake tries to make that part of the plot and also avoids most of the problem of somewhat-boring romantic bliss by spreading the friendship connections a bit wider.

Stevie's friend group formed at orientation at Reed College, and that let me put my finger on another property of this series: essentially all of the characters are from a very specific social class. They're nearly all arts people (bookstore owner, photographer, interior decorator, actress, writer, director), they've mostly gone to college, and while most of them don't have lots of money, there's always at least one person in each friend group with significant wealth. Jordan, from the previous book, is a bit of an exception since she works in a trade (a carpenter), but she still acts like someone from that same social class. It's a bit like reading Jane Austen novels and realizing that the protagonists are drawn from a very specific and very narrow portion of society.

This is not a complaint, to be clear; I have no objections to reading about a very specific social class. But if one has already read lots of books about this class of people, I could see that diminishing the appeal of this series a bit. There are a lot of assumptions baked into the story that aren't really questioned, such as the ubiquity of therapists. (I don't know how Stevie affords one on a barista salary.) There are also some small things in the terminology (therapy speak, for example) and in the specific type of earnestness with which the books attempt to be diverse on most axes other than social class that I suspect may grate a bit for some readers. If that's you, this is your warning.

There is a third-act breakup here, just like the previous volumes. There is also a defense of the emotional punch of third-act breakups in romance novels in the book itself, put into Iris's internal monologue, so I suspect that's the author's answer to critics like myself who don't like the trope. I was less frustrated by this one because it fit the drama level of the protagonists, but I'll also know to expect a third-act breakup in any Blake novel I read in the future.

But, all that said, the summary once again is that I loved this book and could not put it down. Iris is dramatic and occasionally self-destructive but has a core of earnest empathy that makes her easy to like. She's exactly the sort of extrovert who is soothing to introverts rather than draining because she carries the extrovert load of social situations. Stevie is adorably earnest and thoughtful beneath her anxiety. They two of them are wildly different and yet remarkably good together, and I loved reading their story.

Highly recommended, along with the whole series. Start with Delilah Green Doesn't Care; if you like that, you're in for a treat.

Content note: This book is also rather sex-forward and pretty explicit in the sex scenes, maybe a touch more than Astrid Parker Doesn't Fail. If that is or is not your thing in romance novels, be aware going in.

Rating: 9 out of 10

17 December, 2024 05:21AM

December 16, 2024

hackergotchi for Dirk Eddelbuettel

Dirk Eddelbuettel

#45: Some r-ci Updates

market monitor

Welcome to post 45 in the $R^4 series!

We introduced r-ci here in post #32 here nearly four years ago. It has found pretty widespread use and adoption, and we received a few kind words then (in the linked issue) and also more recently (in a follow-up comment) from which we merrily quote:

[…] almost 3 years later on and I have had zero problems with this CI setup. For people who want reliable R software, resources like these are invaluable.

And while we followed up with post #41 about r2u for simple continuous integration, we may not have posted when we based r-ci on r2u (for the obvious Linux usage case). So let’s make time now for a (comparitively smaller) update, and an update usage examples.

We made two changes in the last few days. One is a (obvious in hindsight) simplification. Given that the bootstrap step was always executed, and needed no parameters, we pulled it into a new aggregated setup simply called r-ci that includes it so that it can be omitted as a step in the yaml file. Second, we recently needed Fortran on macOS too, and realized it was not installed by default so we just added that too.

With that a real and used example is now as simple as the screenshot to the left (and hence one ‘paragraph’ shorter). The trained eye will no doubt observe that there is nothing specific to a given repo. And that is basically the key feature: we can simply copy this file around and get fast and easy and reliable CI by taking advantage of the underlying robustness of r2u solving all dependency automagically and reliably. The option to enable macOS is also solid and compelling as the GitHub runners are fast (but more ‘expensive’ in how the count against the limit of minutes—so again a tradeoff to make), as is the option to run coverage if one so desires. Some of my repos do too.

Take a look at the r-ci website which has more examples for the other supported CI servics it can used with, and feel free to ask questions as issue in the repo.

This post by Dirk Eddelbuettel originated on his Thinking inside the box blog. If you like this or other open-source work I do, you can now sponsor me at GitHub. Please report excessive re-aggregation in third-party for-profit settings.

16 December, 2024 10:57PM

December 15, 2024

Russell Coker

Hisense 65U80G 65″ Inch 8K ULED Android TV (2021)

The Aim

I just bought a Hisense 65U80G 65″ Inch 8K ULED Android TV (2021 model) for $1,568 including delivery. I got that deal by googling refurbished 8K TVs and finding the cheapest one I could buy. Amazon and eBay didn’t have any good prices on second hand 8K TVs and new ones start at $3,000 on special. I didn’t assess how Hisense compares to other TVs, as far as I could determine there was only one model of 8K TV on sale in Australia in the price range I was prepared to pay. So I won’t review how this TV compares to other models but how refurbished TVs compare to other display options.

I bought this because the highest resolution monitor in my price range is 5120*2160 [1]. While I could get a 5128*2880 monitor for around $1,500 paying 3* the money for 33% more pixels is bad value for money. Getting 4* the pixels for under 3* the price is good value even when it’s a TV with the lower display quality that involves.

Before buying this TV I read this blog post by Daniel Lawrence about using an 8K TV as a primary monitor [2]. While he has an interesting setup with a 65″ TV on a large desk it’s not what I plan to do at this time.

My Plans for Use

I don’t plan to make it a main monitor. While 5120*2160 isn’t as good as I like on my desk it’s bearable and the quality of the display is high. High resolution isn’t needed for all tasks, for example I’m writing this blog post on my laptop while watching a movie on the 8K TV.

One thing I’d like to do with the 8K TV when I get it working as a monitor is to share the screen for team programming projects. I don’t have any specific plans other than team coding projects at the moment. But it will be interesting to experiment with it when I get it working.

Technical Issues with High Resolution Monitors

Hardware Needed

A lot of the graphic hardware out there don’t support resolutions higher than 5120*2880. It seems that most laptops don’t support resolutions higher than that and higher resolutions than 4K are difficult. Only quite recent and high end video cards will do 8K. Apparently the RTX 2080 is one of the oldest ones that does and that’s $400 on ebay. Strangely the GPU chipset spec pages don’t list the maximum resolution and there’s the additional complication that the other chips might not support the resolutions that the GPU itself can support.

As an aside I don’t use NVidia cards for regular workstations due to reliability problems. But they are good for ML work and for special purpose systems.

Interface Versions

To do 8K video it seems that you need HDMI 2.1 (or maybe 2.0 with 4:2:0 chroma subsampling) or DisplayPort 1.3 for 30Hz with 24bit color and 2.0 for higher refresh rates. But using a particular version of the interface doesn’t require supporting all the resolutions that it might support. This TV has HDMI 2.1 inputs, I’ve bought an adaptor cable that does DisplayPort 1.4 to HDMI 2.1 at 8K resolution. So I need a video card that does DisplayPort 1.4 or HDMI 2.1 output. That doesn’t mean that the card will work, but it could work.

It’s a pity that no-one has made a USB-C video controller that has a basic frame-buffer supporting 8K and the minimal GPU capabilities. The consensus of opinion is that no games will run well at 8K at this time so anyone using 8K resolution doesn’t need GPU power unless it’s for ML stuff.

I’m thinking of making a system that can be used as a ML server and X/Wayland server so a GPU with a decent amount of RAM and compute power would be good. I’m not particularly interested in spending $1,500+ to get a GPU that can drive a $1,568 TV. I’m looking into getting a RTX A2000 with 12G of RAM which should be adequate for ML experiments and can handle 8K@60Hz output.

I’ve ordered a DisplayPort to HDMI converter cable so if I get a DisplayPort card it will work.

Software Support

When I first got started with 4K monitors I had significant problems in adjusting the UI to be usable. The support for scaling software is much better now than it was then and 8K 65″ has a lower DPI than 4K 32″. So I hope this won’t be an issue.

Progress So Far

My first Hisense 8K TV stopped working properly. It would change to a mostly white screen after being used for some time. The screen would change in ways that correlate to changes in what should appear, but not in a way that was usable. It was just a different pattern of white blobs when I changed to a menu view not anything that allowed using it. I presume that this was the problem that drove a need for refurbishment as when I first got the TV it was still signed in to Google accounts for YouTube and to NetFlix.

Best Buy Electrical was good about providing a quick replacement, they took away the old TV and delivered a new one on the same visit and it’s now working well.

I’ve obtained a NVidia card that can allegedly do 8K output and a combination of cables that might be able to carry an 8K signal. Now I just need to get the NVidia drivers to not cause a kernel panic to get things to work.

15 December, 2024 01:40AM by etbe

OnePlus 6 Debian

I recently got a OnePlus 6 for the purpose of running Debian, here’s the Debian wiki page about it [1]. It runs Debian nicely and the basic functions all work, but the problem I’m having now is that AldiMobile (Telstra) and KoganMobile (Vodafone) don’t enable VoLTE for that and all the Australian telcos have turned off 3G. The OnePlus 6 does VoLTE with Chinese SIMs so the phone itself can do it.

The OnePlus 6 was never sold in Australia by the telcos, so they are all gray-market imports which aren’t designed by OnePlus to work in Australia. Until recently that wasn’t a problem, but now that the 3G network has been turned off we need VoLTE and OnePlus didn’t include that in the OS. Reddit has documentation on how to fix this but it has to be done on Android [2]. So I had to go back from Mobian to Android to get VoLTE (and VoWifi) working and then install Mobian again.

For people with similar issues Telstra has a page for checking which phones are supported [3], it’s the only way to determine if it’s the phone or the network that makes VoLTE not work – Android isn’t informative about such things. Telstra lists the OP6 as a suitable phone.

Now after doing this I still can’t get the OP6 working for phone calls on Phosh or PlasmaMobile and I’m not sure why. I’m going to give the PinePhone Pro another go and see if it now works better. In the past I had problems with the PinePhonePro battery discharging too fast, charging too slowly, and having poor call quality [4]. The battery discharge issue should be at least alleviated by some of the changes in the Plasma 6 code that’s now in Debian/Unstable.

I’ve also been lent a PinePhone (non-pro) and been told that it will have better battery life in many situations. I’ll do some tests of that. The PinePhonePro isn’t capable of doing the convergence things I was hoping to do so the greater RAM and CPU power that it has aren’t as relevant as they otherwise would be.

I have a vision for how phones should work. I am not discouraged by the Librem 5, PinePhonePro, Note 9, and OnePlus 6 failing in various ways to do what I hoped for. I will eventually find a phone that I can get working well enough.

15 December, 2024 01:39AM by etbe

December 13, 2024

Emanuele Rocca

Murder Mystery: GCC Builds Failing After sbuild Refactoring

This is the story of an investigation conducted by Jochen Sprickerhof, Helmut Grohne, and myself. It was true teamwork, and we would have not reached the bottom of the issue working individually. We think you will find it as interesting and fun as we did, so here is a brief writeup. A few of the steps mentioned here took several days, others just a few minutes. What is described as a natural progression of events did not always look very obvious at the moment at all.

Let us go through the Six Stages of Debugging together.

Stage 1: That cannot happen

Official Debian GCC builds start failing on multiple architectures in late November.

The build error happens on the build servers when running the testuite, but we know this cannot happen. GCC builds are not meant to fail in case of testsuite failures! Return codes are not making the build fail, make is being called with -k, it just cannot happen.

A lot of the GCC tests are always failing in fact, and an extensive log of the results is posted to the debian-gcc mailing list, but the packages always build fine regardless.

On the build daemons, build failures take several hours.

Stage 2: That does not happen on my machine

Building on my machine running Bookworm is just fine. The Build Daemons run Bookworm and use a Sid chroot for the build environment, just like I am. Same kernel.

The only obvious difference between my setup and the Debian buildds is that I am using sbuild 0.85.0 from bookworm, and the buildds have 0.86.3~bpo12+1 from bookworm-backports. Trying again with 0.86.3~bpo12+1, the build fails on my system too. The build daemons were updated to the bookworm-backports version of sbuild at some point in late November. Ha.

Stage 3: That should not happen

There are quite a few sbuild versions in between 0.85.0 and 0.86.3~bpo12+1, but looking at recent sbuild bugs shows that sbuild 0.86.0 was breaking "quite a number of packages". Indeed, with 0.86.0 the build still fails. Trying the version immediately before, 0.85.11, the build finishes correctly. This took more time than it sounds, one run including the tests takes several hours. We need a way to shorten this somehow.

The Debian packaging of GCC allows to specify which languages you may want to skip, and by default it builds Ada, Go, C, C++, D, Fortran, Objective C, Objective C++, M2, and Rust. When running the tests sequentially, the build logs stop roughly around the tests of a runtime library for D, libphobos. So can we still reproduce the failure by skipping everything except for D? With DEB_BUILD_OPTIONS=nolang=ada,go,c,c++,fortran,objc,obj-c++,m2,rust the build still fails, and it fails faster than before. Several minutes, not hours. This is progress, and time to file a bug. The report contains massive spoilers, so no link. :-)

Stage 4: Why does that happen?

Something is causing the build to end prematurely. It’s not the OOM killer, and the kernel does not have anything useful to say in the logs. Can it be that the D language tests are sending signals to some process, and that is what’s killing make ? We start tracing signals sent with bpftrace by writing the following script, signals.bt:

tracepoint:signal:signal_generate {
    printf("%s PID %d (%s) sent signal %d to PID %d\n", comm, pid, args->sig, args->pid);
}

And executing it with sudo bpftrace signals.bt.

The build takes its sweet time, and it fails. Looking at the trace output there’s a suspicious process.exe terminating stuff.

process.exe (PID: 2868133) sent signal 15 to PID 711826

That looks interesting, but we have no clue what PID 711826 may be. Let’s change the script a bit, and trace signals received as well.

tracepoint:signal:signal_generate {
    printf("PID %d (%s) sent signal %d to %d\n", pid, comm, args->sig, args->pid);
}

tracepoint:signal:signal_deliver {
    printf("PID %d (%s) received signal %d\n", pid, comm, args->sig);
}

The working version of sbuild was using dumb-init, whereas the new one features a little init in perl. We patch the current version of sbuild by making it use dumb-init instead, and trace two builds: one with the perl init, one with dumb-init.

Here are the signals observed when building with dumb-init.

PID 3590011 (process.exe) sent signal 2 to 3590014
PID 3590014 (sleep) received signal 9
PID 3590011 (process.exe) sent signal 15 to 3590063
PID 3590063 (std.process tem) received signal 9
PID 3590011 (process.exe) sent signal 9 to 3590065
PID 3590065 (std.process tem) received signal 9

And this is what happens with the new init in perl:

PID 3589274 (process.exe) sent signal 2 to 3589291
PID 3589291 (sleep) received signal 9
PID 3589274 (process.exe) sent signal 15 to 3589338
PID 3589338 (std.process tem) received signal 9
PID 3589274 (process.exe) sent signal 9 to 3589340
PID 3589340 (std.process tem) received signal 9
PID 3589274 (process.exe) sent signal 15 to 3589341
PID 3589274 (process.exe) sent signal 15 to 3589323
PID 3589274 (process.exe) sent signal 15 to 3589320
PID 3589274 (process.exe) sent signal 15 to 3589274
PID 3589274 (process.exe) received signal 9
PID 3589341 (sleep) received signal 9
PID 3589273 (sbuild-usernsex) sent signal 9 to 3589320
PID 3589273 (sbuild-usernsex) sent signal 9 to 3589323

There are a few additional SIGTERM being sent when using the perl init, that’s helpful. At this point we are fairly convinced that process.exe is worth additional inspection. The source code of process.d shows something interesting:

1221 @system unittest
1222 {
[...]
1247     auto pid = spawnProcess(["sleep", "10000"],
[...]
1260     // kill the spawned process with SIGINT
1261     // and send its return code
1262     spawn((shared Pid pid) {
1263         auto p = cast() pid;
1264         kill(p, SIGINT);

So yes, there’s our sleep and the SIGINT (signal 2) right in the unit tests of process.d, just like we have observed in the bpftrace output.

Can we study the behavior of process.exe in isolation, separatedly from the build? Indeed we can. Let’s take the executable from a failed build, and try running it under /usr/libexec/sbuild-usernsexec.

First, we prepare a chroot inside a suitable user namespace:

unshare --map-auto --setuid 0 --setgid 0 mkdir /tmp/rootfs
cd /tmp/rootfs
cat /home/ema/.cache/sbuild/unstable-arm64.tar | unshare --map-auto --setuid 0 --setgid 0 tar xf  -
unshare --map-auto --setuid 0 --setgid 0 mkdir /tmp/rootfs/whatever
unshare --map-auto --setuid 0 --setgid 0 cp process.exe /tmp/rootfs/

Now we can run process.exe on its own using the perl init, and trace signals at will:

/usr/libexec/sbuild-usernsexec --pivotroot --nonet u:0:100000:65536  g:0:100000:65536 /tmp/rootfs ema /whatever -- /process.exe

We can compare the behavior of the perl init vis-a-vis the one using dumb-init in milliseconds instead of minutes.

Stage 5: Oh, I see.

Why does process.exe send more SIGTERMs when using the perl init is now the big question. We have a simple reproducer, so this is where using strace becomes possible.

sudo strace --user ema --follow-forks -o sbuild-dumb-init.strace ./sbuild-usernsexec-dumb-init --pivotroot --nonet u:0:100000:65536  g:0:100000:65536 /tmp/dumbroot ema /whatever -- /process.exe

We start comparing the strace output of dumb-init with that of perl-init, looking in particular for different calls to kill.

Here is what process.exe does under dumb-init:

3593883 kill(-2, SIGTERM)               = -1 ESRCH (No such process)

No such process. Under perl-init instead:

3593777 kill(-2, SIGTERM <unfinished ...>

The process is there under perl-init!

That is a kill with negative pid. From the kill(2) man page:

If pid is less than -1, then sig is sent to every process in the process group whose ID is -pid.

It would have been very useful to see this kill with negative pid in the output of bpftrace, why didn’t we? The tracepoint used, tracepoint:signal:signal_generate, shows when signals are actually being sent, and not the syscall being called. To confirm, one can trace tracepoint:syscalls:sys_enter_kill and see the negative PIDs, for example:

PID 312719 (bash) sent signal 2 to -312728

The obvious question at this point is: why is there no process group 2 when using dumb-init?

Stage 6: How did that ever work?

We know that process.exe sends a SIGTERM to every process in the process group with ID 2. To find out what this process group may be, we spawn a shell with dumb-init and observe under /proc PIDs 1, 16, and 17. With perl-init we have 1, 2, and 17. When running dumb-init, there are a few forks before launching the program, explaining the difference. Looking at /proc/2/cmdline we see that it’s bash, ie. the program we are running under perl-init. When building a package, that is dpkg-buildpackage itself.

The test is accidentally killing its own process group.

Now where does this -2 come from in the test?

2363     // Special values for _processID.
2364     enum invalid = -1, terminated = -2;

Oh. -2 is used as a special value for PID, meaning "terminated". And there’s a call to kill() later on:

2694     do { s = tryWait(pid); } while (!s.terminated);
[...]
2697     assertThrown!ProcessException(kill(pid));

What sets pid to terminated you ask?

Here is tryWait:

2568 auto tryWait(Pid pid) @safe
2569 {
2570     import std.typecons : Tuple;
2571     assert(pid !is null, "Called tryWait on a null Pid.");
2572     auto code = pid.performWait(false);

And performWait:

2306         _processID = terminated;

The solution, dear reader, is not to kill.

PS: the bug report with spoilers for those interested is #1089007.

13 December, 2024 03:31PM by Emanuele Rocca (ema@linux.it)

hackergotchi for Freexian Collaborators

Freexian Collaborators

Monthly report about Debian Long Term Support, November 2024 (by Roberto C. Sánchez)

Like each month, have a look at the work funded by Freexian’s Debian LTS offering.

Debian LTS contributors

In November, 20 contributors have been paid to work on Debian LTS, their reports are available:

  • Abhijith PA did 14.0h (out of 6.0h assigned and 8.0h from previous period).
  • Adrian Bunk did 53.0h (out of 15.0h assigned and 85.0h from previous period), thus carrying over 47.0h to the next month.
  • Andrej Shadura did 7.0h (out of 7.0h assigned).
  • Arturo Borrero Gonzalez did 1.0h (out of 10.0h assigned), thus carrying over 9.0h to the next month.
  • Bastien Roucariès did 20.0h (out of 20.0h assigned).
  • Ben Hutchings did 0.0h (out of 24.0h assigned), thus carrying over 24.0h to the next month.
  • Chris Lamb did 18.0h (out of 18.0h assigned).
  • Daniel Leidert did 17.0h (out of 26.0h assigned), thus carrying over 9.0h to the next month.
  • Emilio Pozuelo Monfort did 40.5h (out of 60.0h assigned), thus carrying over 19.5h to the next month.
  • Guilhem Moulin did 7.25h (out of 7.5h assigned and 12.5h from previous period), thus carrying over 12.75h to the next month.
  • Jochen Sprickerhof did 3.5h (out of 10.0h assigned), thus carrying over 6.5h to the next month.
  • Lee Garrett did 14.75h (out of 15.25h assigned and 44.75h from previous period), thus carrying over 45.25h to the next month.
  • Lucas Kanashiro did 10.0h (out of 54.0h assigned and 10.0h from previous period), thus carrying over 54.0h to the next month.
  • Markus Koschany did 20.0h (out of 40.0h assigned), thus carrying over 20.0h to the next month.
  • Roberto C. Sánchez did 6.75h (out of 9.75h assigned and 14.25h from previous period), thus carrying over 17.25h to the next month.
  • Santiago Ruano Rincón did 24.75h (out of 23.5h assigned and 1.5h from previous period), thus carrying over 0.25h to the next month.
  • Sean Whitton did 2.0h (out of 6.0h assigned), thus carrying over 4.0h to the next month.
  • Sylvain Beucler did 21.5h (out of 9.5h assigned and 50.5h from previous period), thus carrying over 38.5h to the next month.
  • Thorsten Alteholz did 11.0h (out of 11.0h assigned).
  • Tobias Frost did 12.0h (out of 10.5h assigned and 1.5h from previous period).

Evolution of the situation

In November, we have released 38 DLAs.

The LTS coordinators, Roberto and Santiago, delivered a talk at the Mini-DebConf event in Toulouse, France. The title of the talk was “How LTS goes beyond LTS”. The talk covered work done by the LTS Team during the past year. This included contributions related to individual packages in Debian (such as tomcat, jetty, radius, samba, apache2, ruby, and many others); improvements to tooling and documentation useful to the Debian project as a whole; and contributions to upstream work (apache2, freeimage, node-dompurify, samba, and more). Additionally, several contributors external to the LTS Team were highlighted for their contributions to LTS. Readers are encouraged to watch the video of the presentation for a more detailed review of various ways in which the LTS team has contributed more broadly to the Debian project and to the free software community during the past year.

We wish to specifically thank Salvatore (of the Debian Security Team) for swiftly handling during November the updates of needrestart and libmodule-scandeps-perl, both of which involved arbitrary code execution vulnerabilities. We are happy to see increased involvement in LTS work by contributors from outside the formal LTS Team.

The work of the LTS Team in November was otherwise unremarkable, encompassing the customary triage, development, testing, and release of numerous DLAs, along with some associated contributions to related packages in stable and unstable.

Thanks to our sponsors

Sponsors that joined recently are in bold.

13 December, 2024 12:00AM by Roberto C. Sánchez

December 12, 2024

hackergotchi for Dirk Eddelbuettel

Dirk Eddelbuettel

#44: r2u For ML and MLops Talk

Welcome to the 44th post in the $R^4 series.

A few weeks ago, and following an informal ‘call for talks’ by James Lamb, I had an opportunity to talk about r2u to the Chicago ML and MLops meetup groups. You can find the slides here.

Over the last 2 1/2 years, r2u has become a widely-deployed mechanism in a number of settings, including (but not limited to) software testing via continuous integration, deployment on cloud servers—besides of course to more standard use on local laptops or workstation. 30 million downloads illustrate this. My thesis for the talk was that this extends equally to ML(ops) where no surprises, no hickups automated deployments are key for large-scale model training, evaluation and of course production deployments.

In this context, I introduce r2u while giving credit both to what came before it, the existing alternatives (or ‘competitors’ for mindshare if one prefers that phrasing), and of course what lies underneath it.

The central takeaway, I argue, is that r2u can and does take advantage of a unique situation in that we can ‘join’ the package manager task for the underlying (operating) system and and the application domain, here R and its unique CRAN repository network. Other approaches can, and of course do, provide binaries, but by doing this outside the realm of the system package manager can only arrive at a lesser integration (and I show a common error arising in that case). So where r2u is feasible, it dominates the alternatives (while the alternatives may well provide deployment on more platforms which, even when less integrated, may be of greater importance for some). As always, it all depends.

But the talk, and its slides, motivate and illustrate why we keep calling r2u by its slogan of r2u: Fast. Easy. Reliable. Pick All Three.

This post by Dirk Eddelbuettel originated on his Thinking inside the box blog. If you like this or other open-source work I do, you can now sponsor me at GitHub. Please report excessive re-aggregation in third-party for-profit settings.

12 December, 2024 10:02PM

hackergotchi for Matthew Garrett

Matthew Garrett

When should we require that firmware be free?

The distinction between hardware and software has historically been relatively easy to understand - hardware is the physical object that software runs on. This is made more complicated by the existence of programmable logic like FPGAs, but by and large things tend to fall into fairly neat categories if we're drawing that distinction.

Conversations usually become more complicated when we introduce firmware, but should they? According to Wikipedia, Firmware is software that provides low-level control of computing device hardware, and basically anything that's generally described as firmware certainly fits into the "software" side of the above hardware/software binary. From a software freedom perspective, this seems like something where the obvious answer to "Should this be free" is "yes", but it's worth thinking about why the answer is yes - the goal of free software isn't freedom for freedom's sake, but because the freedoms embodied in the Free Software Definition (and by proxy the DFSG) are grounded in real world practicalities.

How do these line up for firmware? Firmware can fit into two main classes - it can be something that's responsible for initialisation of the hardware (such as, historically, BIOS, which is involved in initialisation and boot and then largely irrelevant for runtime[1]) or it can be something that makes the hardware work at runtime (wifi card firmware being an obvious example). The role of free software in the latter case feels fairly intuitive, since the interface and functionality the hardware offers to the operating system is frequently largely defined by the firmware running on it. Your wifi chipset is, these days, largely a software defined radio, and what you can do with it is determined by what the firmware it's running allows you to do. Sometimes those restrictions may be required by law, but other times they're simply because the people writing the firmware aren't interested in supporting a feature - they may see no reason to allow raw radio packets to be provided to the OS, for instance. We also shouldn't ignore the fact that sufficiently complicated firmware exposed to untrusted input (as is the case in most wifi scenarios) may contain exploitable vulnerabilities allowing attackers to gain arbitrary code execution on the wifi chipset - and potentially use that as a way to gain control of the host OS (see this writeup for an example). Vendors being in a unique position to update that firmware means users may never receive security updates, leaving them with a choice between discarding hardware that otherwise works perfectly or leaving themselves vulnerable to known security issues.

But even the cases where firmware does nothing other than initialise the hardware cause problems. A lot of hardware has functionality controlled by registers that can be locked during the boot process. Vendor firmware may choose to disable (or, rather, never to enable) functionality that may be beneficial to a user, and then lock out the ability to reconfigure the hardware later. Without any ability to modify that firmware, the user lacks the freedom to choose what functionality their hardware makes available to them. Again, the ability to inspect this firmware and modify it has a distinct benefit to the user.

So, from a practical perspective, I think there's a strong argument that users would benefit from most (if not all) firmware being free software, and I don't think that's an especially controversial argument. So I think this is less of a philosophical discussion, and more of a strategic one - is spending time focused on ensuring firmware is free worthwhile, and if so what's an appropriate way of achieving this?

I think there's two consistent ways to view this. One is to view free firmware as desirable but not necessary. This approach basically argues that code that's running on hardware that isn't the main CPU would benefit from being free, in the same way that code running on a remote network service would benefit from being free, but that this is much less important than ensuring that all the code running in the context of the OS on the primary CPU is free. The maximalist position is not to compromise at all - all software on a system, whether it's running at boot or during runtime, and whether it's running on the primary CPU or any other component on the board, should be free.

Personally, I lean towards the former and think there's a reasonably coherent argument here. I think users would benefit from the ability to modify the code running on hardware that their OS talks to, in the same way that I think users would benefit from the ability to modify the code running on hardware the other side of a network link that their browser talks to. I also think that there's enough that remains to be done in terms of what's running on the host CPU that it's not worth having that fight yet. But I think the latter is absolutely intellectually consistent, and while I don't agree with it from a pragmatic perspective I think things would undeniably be better if we lived in that world.

This feels like a thing you'd expect the Free Software Foundation to have opinions on, and it does! There are two primarily relevant things - the Respects your Freedoms campaign focused on ensuring that certified hardware meets certain requirements (including around firmware), and the Free System Distribution Guidelines, which define a baseline for an OS to be considered free by the FSF (including requirements around firmware).

RYF requires that all software on a piece of hardware be free other than under one specific set of circumstances. If software runs on (a) a secondary processor and (b) within which software installation is not intended after the user obtains the product, then the software does not need to be free. (b) effectively means that the firmware has to be in ROM, since any runtime interface that allows the firmware to be loaded or updated is intended to allow software installation after the user obtains the product.

The Free System Distribution Guidelines require that all non-free firmware be removed from the OS before it can be considered free. The recommended mechanism to achieve this is via linux-libre, a project that produces tooling to remove anything that looks plausibly like a non-free firmware blob from the Linux source code, along with any incitement to the user to load firmware - including even removing suggestions to update CPU microcode in order to mitigate CPU vulnerabilities.

For hardware that requires non-free firmware to be loaded at runtime in order to work, linux-libre doesn't do anything to work around this - the hardware will simply not work. In this respect, linux-libre reduces the amount of non-free firmware running on a system in the same way that removing the hardware would. This presumably encourages users to purchase RYF compliant hardware.

But does that actually improve things? RYF doesn't require that a piece of hardware have no non-free firmware, it simply requires that any non-free firmware be hidden from the user. CPU microcode is an instructive example here. At the time of writing, every laptop listed here has an Intel CPU. Every Intel CPU has microcode in ROM, typically an early revision that is known to have many bugs. The expectation is that this microcode is updated in the field by either the firmware or the OS at boot time - the updated version is loaded into RAM on the CPU, and vanishes if power is cut. The combination of RYF and linux-libre doesn't reduce the amount of non-free code running inside the CPU, it just means that the user (a) is more likely to hit since-fixed bugs (including security ones!), and (b) has less guidance on how to avoid them.

As long as RYF permits hardware that makes use of non-free firmware I think it hurts more than it helps. In many cases users aren't guided away from non-free firmware - instead it's hidden away from them, leaving them less aware that their freedom is constrained. Linux-libre goes further, refusing to even inform the user that the non-free firmware that their hardware depends on can be upgraded to improve their security.

Out of sight shouldn't mean out of mind. If non-free firmware is a threat to user freedom then allowing it to exist in ROM doesn't do anything to solve that problem. And if it isn't a threat to user freedom, then what's the point of requiring linux-libre for a Linux distribution to be considered free by the FSF? We seem to have ended up in the worst case scenario, where nothing is being done to actually replace any of the non-free firmware running on people's systems and where users may even end up with a reduced awareness that the non-free firmware even exists.

[1] Yes yes SMM

comment count unavailable comments

12 December, 2024 03:57PM

Android privacy improvements break key attestation

Sometimes you want to restrict access to something to a specific set of devices - for instance, you might want your corporate VPN to only be reachable from devices owned by your company. You can't really trust a device that self attests to its identity, for instance by reporting its MAC address or serial number, for a couple of reasons:
  • These aren't fixed - MAC addresses are trivially reprogrammable, and serial numbers are typically stored in reprogrammable flash at their most protected
  • A malicious device could simply lie about them
If we want a high degree of confidence that the device we're talking to really is the device it claims to be, we need something that's much harder to spoof. For devices with a TPM this is the TPM itself. Every TPM has an Endorsement Key (EK) that's associated with a certificate that chains back to the TPM manufacturer. By verifying that certificate path and having the TPM prove that it's in posession of the private half of the EK, we know that we're communicating with a genuine TPM[1].

Android has a broadly equivalent thing called ID Attestation. Android devices can generate a signed attestation that they have certain characteristics and identifiers, and this can be chained back to the manufacturer. Obviously providing signed proof of the device identifier is kind of problematic from a privacy perspective, so the short version[2] is that only apps installed using a corporate account rather than a normal user account are able to do this.

But that's still not ideal - the device identifiers involved included the IMEI and serial number of the device, and those could potentially be used to correlate devices across privacy boundaries since they're static[3] identifiers that are the same both inside a corporate work profile and in the normal user profile, and also remains static if you move between different employers and use the same phone[4]. So, since Android 12, ID Attestation includes an "Enterprise Specific ID" or ESID. The ESID is based on a hash of device-specific data plus the enterprise that the corporate work profile is associated with. If a device is enrolled with the same enterprise then this ID will remain static, if it's enrolled with a different enterprise it'll change, and it just doesn't exist outside the work profile at all. The other device identifiers are no longer exposed.

But device ID verification isn't enough to solve the underlying problem here. When we receive a device ID attestation we know that someone at the far end has posession of a device with that ID, but we don't know that that device is where the packets are originating. If our VPN simply has an API that asks for an attestation from a trusted device before routing packets, we could pass that on to said trusted device and then simply forward the attestation to the VPN server[5]. We need some way to prove that the the device trying to authenticate is actually that device.

The answer to this is key provenance attestation. If we can prove that an encryption key was generated on a trusted device, and that the private half of that key is stored in hardware and can't be exported, then using that key to establish a connection proves that we're actually communicating with a trusted device. TPMs are able to do this using the attestation keys generated in the Credential Activation process, giving us proof that a specific keypair was generated on a TPM that we've previously established is trusted.

Android again has an equivalent called Key Attestation. This doesn't quite work the same way as the TPM process - rather than being tied back to the same unique cryptographic identity, Android key attestation chains back through a separate cryptographic certificate chain but contains a statement about the device identity - including the IMEI and serial number. By comparing those to the values in the device ID attestation we know that the key is associated with a trusted device and we can now establish trust in that key.

"But Matthew", those of you who've been paying close attention may be saying, "Didn't Android 12 remove the IMEI and serial number from the device ID attestation?" And, well, congratulations, you were apparently paying more attention than Google. The key attestation no longer contains enough information to tie back to the device ID attestation, making it impossible to prove that a hardware-backed key is associated with a specific device ID attestation and its enterprise enrollment.

I don't think this was any sort of deliberate breakage, and it's probably more an example of shipping the org chart - my understanding is that device ID attestation and key attestation are implemented by different parts of the Android organisation and the impact of the ESID change (something that appears to be a legitimate improvement in privacy!) on key attestation was probably just not realised. But it's still a pain.

[1] Those of you paying attention may realise that what we're doing here is proving the identity of the TPM, not the identity of device it's associated with. Typically the TPM identity won't vary over the lifetime of the device, so having a one-time binding of those two identities (such as when a device is initially being provisioned) is sufficient. There's actually a spec for distributing Platform Certificates that allows device manufacturers to bind these together during manufacturing, but I last worked on those a few years back and don't know what the current state of the art there is

[2] Android has a bewildering array of different profile mechanisms, some of which are apparently deprecated, and I can never remember how any of this works, so you're not getting the long version

[3] Nominally, anyway. Cough.

[4] I wholeheartedly encourage people not to put work accounts on their personal phones, but I am a filthy hypocrite here

[5] Obviously if we have the ability to ask for attestation from a trusted device, we have access to a trusted device. Why not simply use the trusted device? The answer there may be that we've compromised one and want to do as little as possible on it in order to reduce the probability of triggering any sort of endpoint detection agent, or it may be because we want to run on a device with different security properties than those enforced on the trusted device.

comment count unavailable comments

12 December, 2024 12:16PM

hackergotchi for Dirk Eddelbuettel

Dirk Eddelbuettel

RcppCCTZ 0.2.13 on CRAN: Maintenance

A new release 0.2.13 of RcppCCTZ is now on CRAN.

RcppCCTZ uses Rcpp to bring CCTZ to R. CCTZ is a C++ library for translating between absolute and civil times using the rules of a time zone. In fact, it is two libraries. One for dealing with civil time: human-readable dates and times, and one for converting between between absolute and civil times via time zones. And while CCTZ is made by Google(rs), it is not an official Google product. The RcppCCTZ page has a few usage examples and details. This package was the first CRAN package to use CCTZ; by now several others packages (four the last time we counted) include its sources too. Not ideal, but beyond our control.

This version include most routine package maintenance as well as one small contributed code improvement. The changes since the last CRAN release are summarised below.

Changes in version 0.2.13 (2024-12-11)

  • No longer set compilation standard as recent R version set a sufficiently high minimum

  • Qualify a call to cctz::format (Michael Quinn in #44)

  • Routine updates to continuous integration and badges

  • Switch to Authors@R in DESCRIPTION

Courtesy of my CRANberries, there is a diffstat report relative to to the previous version. More details are at the RcppCCTZ page; code, issue tickets etc at the GitHub repository. If you like this or other open-source work I do, you can sponsor me at GitHub.

This post by Dirk Eddelbuettel originated on his Thinking inside the box blog. Please report excessive re-aggregation in third-party for-profit settings.

12 December, 2024 04:04AM

December 11, 2024

Divine Attah-Ohiemi

From Sisterly Wisdom to Debian Dreams: My Outreachy Journey

Discovering Open Source: How I Got Introduced
Hey there! I’m Divine Attah-Ohiemi, a sophomore studying Computer Science. My journey into the world of open source was anything but grand. It all started with a simple question to my sister: “How do people get jobs without experience?” Her answer? Open source! I dove into this vibrant community, and it felt like discovering a hidden treasure chest filled with knowledge and opportunities.

Choosing Debian: Why This Community?
Why Debian, you ask? Well, I applied to Outreachy twice, and both times, I chose Debian. It’s not just my first operating system; it feels like home. The Debian community is incredibly welcoming, like a big family gathering where everyone supports each other. Whether I was updating my distro or poring over documentation, the care and consideration in this community were palpable. It reminded me of the warmth of homeschooling with relatives. Plus, knowing that Debian's name comes from its creator Ian and his wife Debra adds a personal touch that makes me feel even more honored to contribute to making the website better!

Why I Applied to Outreachy: What Inspired Me
Outreachy is my golden ticket to the open source world! As a 19-year-old, I see this internship as a unique opportunity to gain invaluable experience while contributing to something meaningful. It’s the perfect platform for me to learn, grow, and connect with like-minded individuals who share my passion for technology and community.

I’m excited for this journey and can’t wait to see where it takes me! 🌟

11 December, 2024 02:25PM by Divine Attah-Ohiemi

December 10, 2024

Stephan Lachnit

Creating a bootable USB stick that can still be used as normal storage

As someone who daily drives rolling release operating systems, having a bootable USB stick with a live install of Debian is essential. It has saved reverting broken packages and making my device bootable again at least a couple of times. However, creating a bootable USB stick usually uses the entire storage of the USB stick, which seems unnecessary given that USB sticks easily have 64GiB or more these days while live ISOs still don’t use more than 8GiB.

In this how-to, I will explain how to create a bootable USB stick, that also holds a FAT32 partition, which allows the USB stick used as usual.

Creating the partition partble

The first step is to create the partition table on the new drive. There are several tools to do this, I recommend the ArchWiki page on this topic for details. For best compatibility, MBR should be used instead of the more modern GPT. For simplicity I just went with the GParted since it has an easy GUI, but feel free to use any other tool. The layout should look like this:

Type  │ Partition │ Suggested size
──────┼───────────┼───────────────
EFI   │ /dev/sda1 │           8GiB
FAT32 │ /dev/sda2 │      remainder

Notes:

  • The disk names are just an example and have to be adjusted for your system.
  • Don’t set disk labels, they don’t appear on the new install anyway and some UEFIs might not like it on your boot partition.
  • If you used GParted, create the EFI partition as FAT32 and set the esp flag.

Mounting the ISO and USB stick

The next step is to download your ISO and mount it. I personally use a Debian Live ISO for this. To mount the ISO, use the loop mounting option:

mkdir -p /tmp/mnt/iso
sudo mount -o loop /path/to/image.iso /tmp/mnt/iso

Similarily, mount your USB stick:

mkdir -p /tmp/mnt/usb
sudo mount /dev/sda1 /tmp/mnt/usb

Copy the ISO contents

To copy the contents from the ISO to the USB stick, use this command:

sudo cp -r -L /tmp/mnt/iso/. /tmp/mnt/usb

The -L option is required to deference the symbolic links present in the ISO, since FAT32 does not support symbolic links. Note that you might get warning about cyclic symbolic links. Nothing can be done to fix those, except hoping that they won’t cause a problem. For me this never was a problem though.

Finishing touches

Umnount the ISO and USB stick:

sudo umount /tmp/mnt/usb
sudo umount /tmp/mnt/iso

Now the USB stick should be bootable and have a usable data partition that can be used on essentially every operating system. I recommend testing that the stick is bootable to make sure it actually works in case of an emergency.

10 December, 2024 02:53PM

hackergotchi for Emmanuel Kasper

Emmanuel Kasper

Too many open files in Minikube Pod

Right now playing with minikube, to run a three nodes highly available Kubernetes control plane. I am using the docker driver of minikube, so each Kubernetes node component is running inside a docker container, instead of using full blown VMs.

In my experience this works better than Kind, as using Kind you cannot correctly restart a cluster deployed in highly available mode.

This is the topology of the cluster:

$ minikube node list 
minikube        192.168.49.2
minikube-m02    192.168.49.3
minikube-m03    192.168.49.4

Each kubernetes node is actually a docker container:

$ docker ps
CONTAINER ID   IMAGE                                 COMMAND                  CREATED          STATUS         PORTS                                                                                                                                  NAMES
977046487e5e   gcr.io/k8s-minikube/kicbase:v0.0.45   "/usr/local/bin/entr…"   31 minutes ago   Up 6 minutes   127.0.0.1:32812->22/tcp, 127.0.0.1:32811->2376/tcp, 127.0.0.1:32810->5000/tcp, 127.0.0.1:32809->8443/tcp, 127.0.0.1:32808->32443/tcp   minikube-m03
8be3549f0c4c   gcr.io/k8s-minikube/kicbase:v0.0.45   "/usr/local/bin/entr…"   31 minutes ago   Up 6 minutes   127.0.0.1:32807->22/tcp, 127.0.0.1:32806->2376/tcp, 127.0.0.1:32805->5000/tcp, 127.0.0.1:32804->8443/tcp, 127.0.0.1:32803->32443/tcp   minikube-m02
4b39f1c47c23   gcr.io/k8s-minikube/kicbase:v0.0.45   "/usr/local/bin/entr…"   31 minutes ago   Up 6 minutes   127.0.0.1:32802->22/tcp, 127.0.0.1:32801->2376/tcp, 127.0.0.1:32800->5000/tcp, 127.0.0.1:32799->8443/tcp, 127.0.0.1:32798->32443/tcp   minikube

The whole list of pods running in the cluster is as this:

$ minikube kubectl -- get pods --all-namespaces
kube-system   coredns-6f6b679f8f-85n9l               0/1     Running   1 (44s ago)   3m32s
kube-system   coredns-6f6b679f8f-pnhxv               0/1     Running   1 (44s ago)   3m32s
kube-system   etcd-minikube                          1/1     Running   1 (44s ago)   3m37s
kube-system   etcd-minikube-m02                      1/1     Running   1 (42s ago)   3m21s
...
kube-system   kube-proxy-84gm6                       0/1     Error     1 (31s ago)   3m4s

There is this container in Error status, let us check the logs

$ minikube kubectl -- logs -n kube-system kube-proxy-84gm6
E1210 11:50:42.117036       1 run.go:72] "command failed" err="failed complete: too many open files"

This can be fixed by increasing the number of inotify watchers:

# cat > /etc/sysctl.d/minikube.conf <<EOF
fs.inotify.max_user_watches = 524288
fs.inotify.max_user_instances = 512
EOF
# sysctl --system
...
* Applying /etc/sysctl.d/minikube.conf ...
* Applying /etc/sysctl.conf ...
...
fs.inotify.max_user_watches = 524288
fs.inotify.max_user_instances = 512

Restart the cluster

$ minikube stop
$ minikube start 
$ minikube kubectl -- get pods --all-namespaces | grep Error
$

10 December, 2024 12:10PM by Manu

December 09, 2024

hackergotchi for Gunnar Wolf

Gunnar Wolf

Some tips for those who still administer Drupal7-based sites

A bit of history: Drupal at my workplace (and in Debian)

My main day-to-day responsibility in my workplace is, and has been for 20 years, to take care of the network infrastructure for UNAM’s Economics Research Institute. One of the most visible parts of this responsibility is to ensure we have a working Web presence, and that it caters for the needs of our academic community.

I joined the Institute in January 2005. Back then, our designer pushed static versions of our webpage, completely built in her computer. This was standard practice at the time, and lasted through some redesigns, but I soon started advocating for the adoption of a Content Management System. After evaluating some alternatives, I recommended adopting Drupal. It took us quite a bit to do the change: even though I clearly recall starting work toward adopting it as early as 2006, according to the Internet Archive, we switched to a Drupal-backed site around June 2010. We started using it somewhere in the version 6’s lifecycle.

As for my Debian work, by late 2012 I started getting involved in the maintenance of the drupal7 package, and by April 2013 I became its primary maintainer. I kept the drupal7 package up to date in Debian until ≈2018; the supported build methods for Drupal 8 are not compatible with Debian (mainly, bundling third-party libraries and updating them without coordination with the rest of the ecosystem), so towards the end of 2016, I announced I would not package Drupal 8 for Debian.

By March 2016, we migrated our main page to Drupal 7. By then, we already had several other sites for our academics’ projects, but my narrative follows our main Web site. I did manage to migrate several Drupal 6 (D6) sites to Drupal 7 (D7); it was quite involved process, never transparent to the user, and we did have the backlash of long downtimes (or partial downtimes, with sites half-available only) with many of our users. For our main site, we took the opportunity to do a complete redesign and deployed a fully new site.

You might note that March 2016 is after the release of D8 (November 2015). I don’t recall many of the specifics for this decision, but if I’m not mistaken, building the new site was a several months long process — not only for the technical work of setting it up, but for the legwork of getting all of the needed information from the different areas that need to be represented in the Institute. Not only that: Drupal sites often include tens of contributed themes and modules; the technological shift the project underwent between its 7 and 8 releases was too deep, and modules took a long time (if at all — many themes and modules were outright dumped) to become available for the new release.

Naturally, the Drupal Foundation wanted to evolve and deprecate the old codebase. But the pain to migrate from D7 to D8 is too big, and many sites have remained under version 7 — Eight years after D8’s release, almost 40% of Drupal installs are for version 7, and a similar proportion runs a currently-supported release (10 or 11). And while the Drupal Foundation made a great job at providing very-long-term support for D7, I understand the burden is becoming too much, so close to a year ago (and after pushing several times the D7, they finally announced support will finish this upcoming January 5.

Drupal 7 must go!

I found the following usage graphs quite interesting: the usage statistics for all Drupal versions follows a very positive slope, peaking around 2014 during the best years of D7, and somewhat stagnating afterwards, staying since 2015 at the 25000–28000 sites mark (I’m very tempted to copy the graphs, but builtwith’s terms of use are very clear in not allowing it). There is a sharp drop in the last year — I attribute it to the people that are leaving D7 for other technologies after its end-of-life announcement. This becomes clearer looking only at D7’s usage statistics: D7 peaks at ≈15000 installs in 2016 stays there for close to 5 years, and has a sharp drop to under 7500 sites in the span of one year.

D8 has a more “regular” rise, peak and fall peaking at ~8500 between 2020 and 2021, and down to close to 2500 for some months already; D9 has a very brief peak of almost 9000 sites in 2023 and is now close to half of it. Currently, the Drupal king appears to be D10, still on a positive slope and with over 9000 sites. Drupal 11 is still just a blip in builtwith’s radar, with… 3 registered sites as of September 2024 :-Þ

After writing this last paragraph, I came across the statistics found in the Drupal webpage; the methodology for acquiring its data is completely different: while builtwith’s methodology is their trade secret, you can read more about how Drupal’s data is gathered (and agree or disagree with it 😉, but at least you have a page detailing 12 years so far of reported data, producing the following graph (which can be shared under the CC BY-SA license 😃):

Drupal usage statistics by version 2013–2024

This graph is disgregated into minor versions, and I don’t want to come up with yet another graph for it 😉 but it supports (most of) the narrative I presented above… although I do miss the recent drop builtwith reported in D7’s numbers!

And what about Backdrop?

During the D8 release cycle, a group of Drupal developers were not happy with the depth of the architectural changes that were being adopted, particularly the transition to the Symfony PHP component framework, and forked the D7 codebase to create the Backdrop CMS, a modern version of Drupal, without dropping the known and tested architecture it had. The Backdrop developers keep working closely together with the Drupal community, and although its usage numbers are way smaller than Drupal’s, seems to be sustainable and lively. Of course, as I presented their numbers in the previous section, you can see Backdrop’s numbers in builtwith… are way, way lower.

I have found it to be a very warm and welcoming community, eager to receive new members. And, thanks to its contributed D2B Migrate module, I found it is quite easy to migrate a live site from Drupal 7 to Backdrop.

Migration by playbook!

So… Well, I’m an academic. And (if it’s not obvious to you after reading so far 😉), one of the things I must do in my job is to write. So I decided to write an article to invite my colleagues to consider Backdrop for their D7 sites in Cuadernos Técnicos Universitarios de la DGTIC, a young journal in our university for showcasing technical academical work. And now that my article got accepted and published, I’m happy to share it with you — of course, if you can read Spanish 😉 But anyway…

Given I have several sites to migrate, and that I’m trying to get my colleagues to follow suite, I decided to automatize the migration by writing an Ansible playbook to do the heavy lifting. Of course, the playbook’s users will probably need to tweak it a bit to their personal needs. I’m also far from an Ansible expert, so I’m sure there is ample room fo improvement in my style.

But it works. Quite well, I must add.

But with this size of database…

I did stumble across a big pebble, though. I am working on the migration of one of my users’ sites, and found that its database is… huge. I checked the mysqldump output, and it got me close to 3GB of data. And given the D2B_migrate is meant to work via a Web interface (my playbook works around it by using a client I wrote with Perl’s WWW::Mechanize), I repeatedly stumbled with PHP’s maximum POST size, maximum upload size, maximum memory size…

I asked for help in Backdrop’s Zulip chat site, and my attention was taken off fixing PHP to something more obvious: Why is the database so large? So I took a quick look at the database (or rather: my first look was at the database server’s filesystem usage). MariaDB stores each table as a separate file on disk, so I looked for the nine largest tables:

# ls -lhS|head
total 3.8G
-rw-rw---- 1 mysql mysql 2.4G Dec 10 12:09 accesslog.ibd
-rw-rw---- 1 mysql mysql 224M Dec  2 16:43 search_index.ibd
-rw-rw---- 1 mysql mysql 220M Dec 10 12:09 watchdog.ibd
-rw-rw---- 1 mysql mysql 148M Dec  6 14:45 cache_field.ibd
-rw-rw---- 1 mysql mysql  92M Dec  9 05:08 aggregator_item.ibd
-rw-rw---- 1 mysql mysql  80M Dec 10 12:15 cache_path.ibd
-rw-rw---- 1 mysql mysql  72M Dec  2 16:39 search_dataset.ibd
-rw-rw---- 1 mysql mysql  68M Dec  2 13:16 field_revision_field_idea_principal_articulo.ibd
-rw-rw---- 1 mysql mysql  60M Dec  9 13:19 cache_menu.ibd

A single table, the access log, is over 2.4GB long. The three following tables are, cache tables. I can perfectly live without their data in our new site! But I don’t want to touch the slightest bit of this site until I’m satisfied with the migration process, so I found a way to exclude those tables in a non-destructive way: given D2B_migrate works with a mysqldump output, and given that mysqldump locks each table before starting to modify it and unlocks it after its job is done, I can just do the following:

$ perl -e '$output = 1; while (<>) { $output=0 if /^LOCK TABLES `(accesslog|search_index|watchdog|cache_field|cache_path)`/; $output=1 if /^UNLOCK TABLES/; print if $output}' < /tmp/d7_backup.sql  > /tmp/d7_backup.eviscerated.sql; ls -hl /tmp/d7_backup.sql /tmp/d7_backup.eviscerated.sql
-rw-rw-r-- 1 gwolf gwolf 216M Dec 10 12:22 /tmp/d7_backup.eviscerated.sql
-rw------- 1 gwolf gwolf 2.1G Dec  6 18:14 /tmp/d7_backup.sql

Five seconds later, I’m done! The database is now a tenth of its size, and D2B_migrate is happy to take it. And I’m a big step closer to finishing my reliance on (this bit of) legacy code for my highly-visible sites 😃

09 December, 2024 07:55PM

Thorsten Alteholz

My Debian Activities in November 2024

Debian LTS

This was my hundred-twenty-fifth month that I did some work for the Debian LTS initiative, started by Raphael Hertzog at Freexian. During my allocated time I uploaded or worked on:

  • [DLA 3968-1] netatalk security update to fix four CVEs related to heap buffer overflow and writing arbitrary files. The patches have been prepared by the maintainer.
  • [DLA 3976-1] tgt update to fix one CVE related to not using a propper seed for rand()
  • [DLA 3977-1] xfpt update to fix one CVE related to a stack-based buffer overflow
  • [DLA 3978-1] editorconfig-core update to fix two CVEs related to buffer overflows.

I also continued to work on a fix for glewlwyd, which is more difficult than expected. Besides I started to work on ffmpeg and haproxy.

Last but not least I did a week of FD this month and attended the monthly LTS/ELTS meeting.

Debian ELTS

This month was the seventy-sixth ELTS month. During my allocated time I uploaded or worked on:

  • [ELA-1259-1]editorconfig-core security update for two CVEs in Buster to fix buffer overflows.

I also started to work on a fix for kmail-account-wizzard. Unfortunately preparing a testing environment takes some time and I did not finish testing this month. Besides I started to work on ffmpeg and haproxy.

Last but not least I did a week of FD this month and attended the monthly LTS/ELTS meeting.

Debian Printing

Unfortunately I didn’t found any time to work on this topic.

Debian Matomo

Unfortunately I didn’t found any time to work on this topic.

Debian Astro

This month I uploaded new packages or new upstream or bugfix versions of:

I also sponsored an upload of calceph.

Debian IoT

This month I uploaded new upstream or bugfix versions of:

Debian Mobcom

This month I uploaded new packages or new upstream or bugfix versions of:

misc

This month I uploaded new upstream or bugfix versions of:

I also did some NMU of opensta, kdrill, glosstex, irsim, pagetools, afnix, cpm, to fix some RC bugs.

FTP master

This month I accepted 266 and rejected 16 packages. The overall number of packages that got accepted was 269.

09 December, 2024 06:51PM by alteholz

Paul Wise

FLOSS Activities November 2024

Focus

This month I didn't have any particular focus. I just worked on issues in my info bubble.

Changes

Issues

Review

Communication

  • Respond to queries from Debian users and contributors on IRC

Sponsors

The SWH work was sponsored. All other work was done on a volunteer basis.

09 December, 2024 02:07AM

hackergotchi for Freexian Collaborators

Freexian Collaborators

Debian Contributions: OpenMPI transitions, cPython 3.12.7+ update uploads, Python 3.13 Transition, and more! (by Anupa Ann Joseph, Stefano Rivera)

Debian Contributions: 2024-11

Contributing to Debian is part of Freexian’s mission. This article covers the latest achievements of Freexian and their collaborators. All of this is made possible by organizations subscribing to our Long Term Support contracts and consulting services.

Transition management, by Emilio Pozuelo Monfort

Emilio has been helping finish the mpi-defaults switch to mpich on 32-bit architectures, and the openmpi transitions. This involves filing bugs for the reverse dependencies, doing NMUs, and requesting removals for outdated (Not Built from Source) binaries on 32-bit architectures where openmpi is no longer available. Those transitions got entangled with a few others, such as the petsc stack, and were blocking many packages from migrating to testing. These transitions were completed in early December.

cPython 3.12.7+ update uploads, by Stefano Rivera

Python 3.12 had failed to build on mips64el, due to an obscure dh_strip failure. The mips64el porters never figured it out, but the missing build on mips64el was blocking migration to Debian testing. After waiting a month, enough changes had accumulated in the upstream 3.12 maintenance git branch that we could apply them in the hope of changing the output enough to avoid breaking dh_strip. This worked.

Of course there were other things to deal with too. A test started failing due to a Debian-specific patch we carry for python3.x-minimal, and it needed to be reworked. And Stefano forgot to strip the trailing + from PY_VERSION, which confuses some python libraries. This always requires another patch when applying git updates from the maintenance branch. Stefano added a build-time check to catch this mistake in the future. Python 3.12.7 migrated.

Python 3.13 Transition, by Stefano Rivera and Colin Watson

During November the Python 3.13-add transition started. This is the first stage of supporting a new version of Python in Debian archive (after preparatory work), adding it as a new supported but non-default version. All packages with compiled Python extensions need to be re-built to add support for the new version.

We have covered the lead-up to this transition in the past. Due to preparation, many of the failures we hit were expected and we had patches waiting in the bug tracker. These could be NMUed to get the transition moving. Others had been known about but hadn’t been worked on, yet.

Some other packages ran into new issues, as we got further into the transition than we’d been able to in preparation. The whole Debian Python team has been helping with this work.

The rebuild stage of the 3.13-add transition is now over, but many packages need work before britney will let python3-defaults migrate to testing.

Limiting build concurrency based on available RAM, by Helmut Grohne

In recent years, the concurrency of CPUs has been increasing as has the demand for RAM by linkers. What has not been increasing as quickly is the RAM supply in typical machines. As a result, we more frequently run into situations where the package builds exhaust memory when building at full concurrency. Helmut initiated a discussion about generalizing an approach to this in Debian packages. Researching existing code that limits concurrency as well as providing possible extensions to debhelper and dpkg to provide concurrency limits based on available system RAM. Thus far there is consensus on the need for a more general solution, but ideas are still being collected for the precise solution.

MiniDebConf Toulouse at Capitole du Libre

The whole Freexian Collaborator team attended MiniDebConf Toulouse, part of the Capitole du Libre event. Several members of the team gave talks:

Stefano and Anupa worked as part of the video team, streaming and recording the event’s talks.

Miscellaneous contributions

  • Stefano looked into packaging the latest upstream python-falcon version in Debian, in support of the Python 3.13 transition. This appeared to break python-hug, which is sadly looking neglected upstream, and the best course of action is probably its removal from Debian.

  • Stefano uploaded videos from various 2024 Debian events to PeerTube and YouTube.

  • Stefano and Santiago visited the site for DebConf 2025 in Brest, after the MiniDebConf in Toulouse, to meet with the local team and scout out the venue.

    The on-going DebConf 25 organization work of last month also included handling the logo and artwork call for proposals.

  • Stefano helped the press team to edit a post for bits.debian.org on OpenStreetMap’s migration to Debian.

  • Carles implemented multiple language support on po-debconf-manager and tested it using Portuguese-Brazilian during MiniDebConf Toulouse. The system was also tested and improved by reviewing more than 20 translations to Catalan, creating merge requests for those packages, and providing user support to new users. Additionally, Carles implemented better status transitions, configuration keys management and other small improvements.

  • Helmut sent 32 patches for cross build failures. The wireplumber one was an interactive collaboration with Dylan Aïssi.

  • Helmut continued to monitor the /usr-move, sent a patch for lib64readline8 and continued several older patch conversations. lintian now reports some aliasing issues in unstable.

  • Helmut initiated a discussion on the semantics of *-for-host packages. More feedback is welcome.

  • Helmut improved the crossqa.debian.net infrastructure to fail running lintian less often in larger packages.

  • Helmut continued maintaining rebootstrap mostly dropping applied patches and continuing discussions of submitted patches.

  • Helmut prepared a non-maintainer upload of gzip for several long-standing bugs.

  • Colin came up with a plan for resolving the multipart vs. python-multipart name conflict, and began work on converting reverse-dependencies.

  • Colin upgraded 42 Python packages to new upstream versions. Some were complex: python-catalogue had some upstream version confusion, pydantic and rpds-py involved several Rust package upgrades as prerequisites, and python-urllib3 involved first packaging python-quart-trio and then vendoring an unpackaged test-dependency.

  • Colin contributed Incus support to needrestart upstream.

  • Lucas set up a machine to do a rebuild of all ruby reverse dependencies to check what will be broken by adding ruby 3.3 as an alternative interpreter. The tool used for this is mass-rebuild and the initial rebuilds have already started. The ruby interpreter maintainers are planning to experiment with debusine next time.

  • Lucas is organizing a Debian Ruby sprint towards the end of January in Paris. The plan of the team is to finish any missing bits of Ruby 3.3 transition at the time, try to push Rails 7 transition and fix RC bugs affecting the ruby ecosystem in Debian.

  • Anupa attended a Debian Publicity team meeting in-person during MiniDebCamp Toulouse.

  • Anupa moderated and posted in the Debian Administrator group in LinkedIn.

09 December, 2024 12:00AM by Anupa Ann Joseph, Stefano Rivera

December 08, 2024

hackergotchi for Paulo Henrique de Lima Santana

Paulo Henrique de Lima Santana

Bits from MiniDebConf Toulouse 2024

Intro

I always find it amazing the opportunities I have thanks to my contributions to the Debian Project. I am happy to receive this recognition through the help I receive with travel to attend events in other countries.

This year, two MiniDebConfs were scheduled for the second half of the year in Europe: the traditional edition in Cambridge in UK and a new edition in Toulouse in France. After weighing the difficulties and advantages that I would have to attend one of them, I decided to choose Toulouse, mainly because it was cheaper and because it was in November, giving me more time to plan the trip. I contacted the current DPL Andreas Tille explaining my desire to attend the event and he kindly approved my request for Debian to pay for the tickets. Thanks again to Andreas!

MiniDebConf Toulouse 2024 was held in November 16th and 17th (Saturday and Sunday) and took place in one of the rooms of a traditional Free Software event in the city named Capitole du Libre. Before MiniDebConf, the team organized a MiniDebCamp in November 14th and 15th at a coworking space.

The whole experience promised to be incredible, and it was! From visiting a city in France for the first time, to attending a local Free Software event, and sharing four days with people from the Debian community from various countries.

Travel and the city

My plan was to leave Belo Horizonte on Monday, pass through São Paulo, and arrive in Toulouse on Tuesday night. I was going to spend the whole of Wednesday walking around the city and then participate in the MiniDebCamp on Thursday.

But the flight that was supposed to leave São Paulo in the early hours of Monday to Tuesday was cancelled due to a problem with the airplane and I had spent all Tuesday waiting. I was rebooked on another flight that left in the evening and arrived in Toulouse on Wednesday afternoon. Even though I was very tired from the trip, I still took advantage of the end of the day to walk around the city. But it was a shame to have lost an entire day of sightseeing.

On Thursday I left early in the morning to walk around a little more before going to the MiniDebCamp venue. I walked around a lot and saw several tourist attractions. The city is really very beautiful, as they say, especially the houses and buildings made of pink bricks. I was impressed by the narrow and winding streets; at one point it seemed like I was walking through a maze. I arrived to a corner and there would be 5 streets crossing in different directions.

The riverbank that runs through the city is very beautiful and people spend their evenings there just hanging out. There was a lot of history around there.

I stayed in an airbnb 25 minutes walking from the coworking space and only 10 minutes from the event venue. It was a very spacious apartment that was much cheaper than a hotel.

MiniDebConf Toulouse 2024

MiniDebConf Toulouse 2024

MiniDebCamp

I arrived at the coworking space where the MiniDebCamp was being held and met up with several friends. I also met some new people, talked about the translation work we do in Brazil, and other topics.

We already knew that the organization would pay for lunch for everyone during the two days of MiniDebCamp, and at a certain point they told us that we could go to the room (which was downstairs from the coworking space) to have lunch. They set up a table with quiches, breads, charcuterie and LOTS of cheese :-) There were several types of cheese and they were all very good. I just found it a little strange because I’m not used to having cheese for lunch, but the experience was amazing anyway :-)

MiniDebConf Toulouse 2024

MiniDebConf Toulouse 2024

In the evening, we went as a group to dinner at a restaurant in front of the Capitolium, the city’s main tourist attraction.

On the second day, in the morning, I walked around the city a bit more, then went to the coworking space and had another incredible cheese table for lunch.

Video Team

One of my ideas for going to Toulouse was to be able to help the video team in setting up the equipment for broadcasting and recording the talks. I wanted to follow this work from the beginning and learn some details, something I can’t do before the DebConfs because I always arrive after the people have already set up the infrastructure. And later reproduce this work in the MiniDebConfs in Brazil, such as the one in Maceió that is already scheduled for May 1-4, 2025.

As I had agreed with the people from the video team that I would help set up the equipment, on Friday night we went to the University and stayed in the room working. I asked several questions about what they were doing, about the equipment, and I was able to clear up several doubts. Over the next two days I was handling one of the cameras during the talks. And on Sunday night I helped put everything away.

Thanks to olasd, tumbleweed and ivodd for their guidance and patience.

MiniDebConf Toulouse 2024

The event in general

There was also a meeting with some members of the publicity team who were there with the DPL. We went to a cafeteria and talked mainly about areas that could be improved in the team.

The talks at MiniDebConf were very good and the recordings are also available here.

I ended up not watching any of the talks from the general schedule at Capitole du Libre because they were in French. It’s always great to see free software events abroad to learn how they are done there and to bring some of those experiences to our events in Brazil.

I hope that MiniDebConf in Toulouse will continue to take place every year, or that the French community will hold the next edition in another city and I will be able to join again :-) If everything goes well, in July next year I will return to France to join DebConf25 in Brest.

MiniDebConf Toulouse 2024

More photos

08 December, 2024 10:00AM

December 07, 2024

Dominique Dumont

New cme command to update Debian Standards-Version field

Hi

While updating my Debian package, I often have to update a field from debian/control file.

This field is named Standards-Version and it declares which version of Debian policy the package complies to. When updating this field, one must follow the upgrading checklist.

That being said, I maintain a lot of similar package and I often have to update this Standards-Version field.

This field can be updated manually with cme fix dpkg (see Managing Debian packages with cme). But this command may make other changes and does not commit the result.

So I’ve created a new update-standards-version cme script that:

  • udpate Standards-Version field
  • commit the changed

For instance:

$ cme run update-standards-version 
Reading package lists... Done
Building dependency tree... Done
Reading state information... Done
Connecting to api.ftp-master.debian.org to check 31 package versions. Please wait...
Got info from api.ftp-master.debian.org for 31 packages.
Warning in 'source Standards-Version': Current standards version is '4.7.0'. Please read https://www.debian.org/doc/debian-policy/upgrading-checklist.html for the changes that may be needed on your package
to upgrade it from standard version '4.6.2' to '4.7.0'.

Offending value: '4.6.2'

Changes applied to dpkg-control configuration:
- source Standards-Version: '4.6.2' -> '4.7.0'
[master 552862c1] control: declare compliance with Debian policy 4.7.0
 1 file changed, 1 insertion(+), 1 deletion(-)

Here’s the generated commit. Note that the generated log mentions the new policy version:

$ git show
commit 552862c1f24479b1c0c8c35a6289557f65e8ff3b (HEAD -> master)
Author: Dominique Dumont <dod[at]debian.org>
Date:   Sat Dec 7 19:06:14 2024 +0100

    control: declare compliance with Debian policy 4.7.0

diff --git a/debian/control b/debian/control
index cdb41dc0..e888012e 100644
--- a/debian/control
+++ b/debian/control
@@ -48,7 +48,7 @@ Build-Depends-Indep: dh-sequence-bash-completion,
                      libtext-levenshtein-damerau-perl,
                      libyaml-tiny-perl,
                      po-debconf
-Standards-Version: 4.6.2
+Standards-Version: 4.7.0
 Vcs-Browser: https://salsa.debian.org/perl-team/modules/packages/libconfig-model-perl
 Vcs-Git: https://salsa.debian.org/perl-team/modules/packages/libconfig-model-perl.git
 Homepage: https://github.com/dod38fr/config-model/wiki

Notes:

  • this script can run only if there’s not pending change. Please commit or stash these changes before running this script.
  • this script requires:
    • cme >= 1.041
    • libconfig-model-perl >= 2.155
    • libconfig-model-dpkg-perl >= 3.006

I hope this will be useful to all my fellow Debian developers to reduce the boring parts of packaging activities.

All the best

07 December, 2024 05:52PM by dod

hackergotchi for Louis-Philippe Véronneau

Louis-Philippe Véronneau

lintian.debian.org: Episode IV – A New Hope

After weeks – dare I say months – of work, it is finally done. lintian.debian.org is back online!

Screenshot of the new lintian.debian.org website

Many, many thanks to everyone who worked hard to make this possible:

  • Thanks to Nicolas Peugnet, the author of lintian-ssg, who handed us this custom static site generator on a silver platter. I'm happy I didn't have to code this myself :)
  • Thanks to Otto Kekäläinen, maintainer of the lintian-ssg package in Debian, who worked in tandem with Nicolas to iron out problems.
  • Thanks to Philipp Kern, who did the work on the DSA side to put the website back online.

All in all, I did very little (mostly coordinating these fine folks) and they should get the credit for this very useful service being back.

07 December, 2024 01:45AM by Louis-Philippe Véronneau

December 06, 2024

hackergotchi for Bálint Réczey

Bálint Réczey

Firebuild 0.8.3 is out with 100+ fixes and experimental macOS support!

The new Firebuild release contains plenty of small fixes and a few notable improvements.

Experimental macOS support

The most frequently asked question from people getting to know Firebuild was if it worked on their Mac and the answer sadly used to be that well, it did, but only in a Linux VM. This was far from what they were looking for. 🙁

Linux and macOS have common UNIX roots, but porting Firebuild to macOS included bigger challenges, like ensuring that dyld(1), macOS’s dynamic loader initializes the preloaded interceptor library early enough to catch all interesting calls, and avoid using anything that uses malloc() or thread local variables which are not yet set up then.

Preloading libraries on Linux is really easy, running LD_PRELOAD=my_lib.so ls just works if the library exports the symbols to be interposed, while macOS employs multiple lines of defense to prevent applications from using unknown libraries. Firebuild’s guide for making DYLD_INSERT_LIBRARIES honored on Macs can be helpful with other projects as well that rely on injecting libraries.

Since GitHub’s Arm64 macOS runners don’t allow intercepting binaries with arm64e ABI yet, Firebuild’s Apple Silicon tests are run at Bitrise, who are proud to be first to provide the latest Xcode stacks and were also quick to make the needed changes to their infrastructure to support Firebuild (thanks! ❤).

Firebuild on macOS can already accelerate simple projects and rebuild itself with Xcode. Since Xcode introduces a lot of nondeterminism to the build, Firebuild can’t shine in acceleration with Xcode yet, but can provide nice reports to show which part of the build is the most time consuming and how each sub-command is called.

If you would like to try Firebuild on macOS please compile it from the GitHub repository for now. Precompiled binaries will be distributed on the Mac App Store and via CI providers. Contact us to get notified when those channels become available.

Dealing with the ‘Epochalypse’

Glibc’s API provides many functions with time parameters and some of those functions are intercepted by Firebuild. Time parameters used to be passed as 32-bit values on 32-bit systems, preventing them to accurately represent timestamps after year 2038, which is known as the Y2038 problem or the Epochalypse.

To deal with the problem glibc 2.34 started providing new function symbol variants with 64-bit time parameters, e.g clock_gettime64() in addition to clock_gettime(). The new 64-bit variants are used when compiling consumers of the API with _TIME_BITS=64 defined.

Processes intercepted by Firebuild may have been compiled with or without _TIME_BITS=64, thus libfirebuild now provides both variants on affected systems running glibc >= 34 to work safely with binaries using 64-bit and 32-bit time representation.

Many Linux distributions already stopped supporting 32-bit architectures, but Debian and Ubuntu still supports armhf, for example, where the Y2038 problem still applies. Both Debian and Ubuntu performed a transition rebuilding every library (and their reverse dependencies) with -D_FILE_OFFSET_BITS=64 set where the libraries exported symbols that changed when switching to 64-bit time representation (thanks to Steve Langasek for driving this!) . Thanks to the transition most programs are ready for 2038, but interposer libraries are trickier to fix and if you maintain one it might be a good idea to check if it works well both 32-bit and 64-bit libraries. Faketime, for example is not fixed yet, see #1064555.

Select passed through environment variables with regular expressions

Firebuild filters out most of the environment variables set when starting a build to make the build more reproducible and achieve higher cache hit rate. Extra environment variables to pass through can be specified on the command line one by one, but with many similarly named variables this may become hard to maintain. With regular expressions this just became easier:

firebuild -o 'env_vars.pass_through += "MY_VARS_.*"' my_build_command

If you are not interested in acceleration just would like to explore what the build does by generating a report you can simply pass all variables:

firebuild -r -o 'env_vars.pass_through += ".*"' my_build_command

Other highlights from the 0.8.3 release

  • Fixed and nicer report in Chrome and other WebKit based browsers
  • Support GLibc 2.39 by intercepting pidfd_spawn() and pidfd_spawnp()
  • Even faster Rust build acceleration

For all the changes please check out the release page on GitHub! 🚀

(This post is also published on The Firebuild blog.)

06 December, 2024 09:53PM by Réczey Bálint

Reproducible Builds (diffoscope)

diffoscope 284 released

The diffoscope maintainers are pleased to announce the release of diffoscope version 284. This version includes the following changes:

[ Chris Lamb ]
* Simplify tests_quines.py::test_{differences,differences_deb} to use
  assert_diff and not mangle the expected test output.
* Update some tests to support file(1) version 5.46.
  (Closes: reproducible-builds/diffoscope#395)

You find out more by visiting the project homepage.

06 December, 2024 12:00AM

December 05, 2024

Reproducible Builds

Reproducible Builds in November 2024

Welcome to the November 2024 report from the Reproducible Builds project!

Our monthly reports outline what we’ve been up to over the past month and highlight items of news from elsewhere in the world of software supply-chain security where relevant. As ever, if you are interested in contributing to the Reproducible Builds project, please visit our Contribute page on our website.

Table of contents:

  1. Reproducible Builds mourns the passing of Lunar
  2. Introducing reproduce.debian.net
  3. New landing page design
  4. SBOMs for Python packages
  5. Debian updates
  6. Reproducible builds by default in Maven 4
  7. PyPI now supports digital attestations
  8. “Dependency Challenges in OSS Package Registries”
  9. Zig programming language demonstrated reproducible
  10. Website updates
  11. Upstream patches
  12. Misc development news
  13. Reproducibility testing framework

Reproducible Builds mourns the passing of Lunar

The Reproducible Builds community sadly announced it has lost its founding member, Lunar. Jérémy Bobbio aka ‘Lunar’ passed away on Friday November 8th in palliative care in Rennes, France.

Lunar was instrumental in starting the Reproducible Builds project in 2013 as a loose initiative within the Debian project. He was the author of our earliest status reports and many of our key tools in use today are based on his design. Lunar’s creativity, insight and kindness were often noted.

You can view our full tribute elsewhere on our website. He will be greatly missed.


Introducing reproduce.debian.net

In happier news, this month saw the introduction of reproduce.debian.net. Announced at the recent Debian MiniDebConf in Toulouse, reproduce.debian.net is an instance of rebuilderd operated by the Reproducible Builds project.

rebuilderd is our server designed monitor the official package repositories of Linux distributions and attempts to reproduce the observed results there.

In November, reproduce.debian.net began rebuilding Debian unstable on the amd64 architecture, but throughout the MiniDebConf, it had attempted to rebuild 66% of the official archive. From this, it could be determined that it is currently possible to bit-for-bit reproduce and corroborate approximately 78% of the actual binaries distributed by Debian — that is, using the .buildinfo files hosted by Debian itself.

reproduce.debian.net also contains instructions how to setup one’s own rebuilderd instance, and we very much invite everyone with a machine to spare to setup their own version and to share the results. Whilst rebuilderd is still in development, it has been used to reproduce Arch Linux since 2019. We are especially looking for installations targeting Debian architectures other than i386 and amd64.


New landing page design

As part of a very productive partnership with the Sovereign Tech Fund and Neighbourhoodie, we are pleased to unveil our new homepage/landing page.

We are very happy with our collaboration with both STF and Neighbourhoodie (including many changes not directly related to the website), and look forward to working with them in the future.

SBOMs for Python packages

The Python Software Foundation has announced a new “cross-functional project for SBOMs and Python packages”. Seth Michael Larson writes that the project is “specifically looking to solve these issues”:

  • Enable Python users that require SBOM documents (likely due to regulations like CRA or SSDF) to self-serve using existing SBOM generation tools.
  • Solve the “phantom dependency” problem, where non-Python software is bundled in Python packages but not recorded in any metadata. This makes the job of software composition analysis (SCA) tools difficult or impossible.
  • Make the adoption work by relevant projects such as build backends, auditwheel-esque tools, as minimal as possible. Empower users who are interested in having better SBOM data for the Python projects they are using to be able to contribute engineering time towards that goal.

A GitHub repository for the initiative is available, and there are a number of queries, comments and remarks on Seth’s Discourse forum post.


Debian updates

There was significant development within Debian this month. Firstly, at the recent MiniDebConf in Toulouse, France, Holger Levsen gave a Debian-specific talk on rebuilding packages distributed from ftp.debian.org — that is to say, how to reproduce the results from the official Debian build servers:

Holger described the talk as follows:

For more than ten years, the Reproducible Builds project has worked towards reproducible builds of many projects, and for ten years now we have build Debian packages twice—with maximal variations applied—to see if they can be build reproducible still.

Since about a month, we’ve also been rebuilding trying to exactly match the builds being distributed via ftp.debian.org. This talk will describe the setup and the lessons learned so far, and why the results currently are what they are (spoiler: they are less than 30% reproducible), and what we can do to fix that.

The Debian Project Leader, Andreas Tille, was present at the talk and remarked later in his Bits from the DPL update that:

It might be unfair to single out a specific talk from Toulouse, but I’d like to highlight the one on reproducible builds. Beyond its technical focus, the talk also addressed the recent loss of Lunar, whom we mourn deeply. It served as a tribute to Lunar’s contributions and legacy. Personally, I’ve encountered packages maintained by Lunar and bugs he had filed. I believe that taking over his packages and addressing the bugs he reported is a meaningful way to honor his memory and acknowledge the value of his work.

Holger’s slides and video in .webm format are available.


Next, rebuilderd is the server to monitor package repositories of Linux distributions and attempt to reproduce the observed results. This month, version 0.21.0 released, most notably with improved support for binNMUs by Jochen Sprickerhof and updating the rebuilderd-debian.sh integration to the latest debrebuild version by Holger Levsen. There has also been significant work to get the rebuilderd package into the Debian archive, in particular, both rust-rebuilderd-common version 0.20.0-1 and rust-rust-lzma version 0.6.0-1 were packaged by kpcyrd and uploaded by Holger Levsen.

Related to this, Holger Levsen submitted three additional issues against rebuilderd as well:

  • rebuildctl should be more verbose when encountering issues. []
  • Please add an option to used randomised queues. []
  • Scheduling and re-scheduling multiple packages at once. []

… and lastly, Jochen Sprickerhof submitted one an issue requested that rebuilderd downloads the source package in addition to the .buildinfo file [] and kpcyrd also submitted and fixed an issue surrounding dependencies and clarifying the license []


Separate to this, back in 2018, Chris Lamb filed a bug report against the sphinx-gallery package as it generates unreproducible content in various ways. This month, however, Dmitry Shachnev finally closed the bug, listing the multiple sub-issues that were part of the problem and how they were resolved.


Elsewhere, Roland Clobus posted to our mailing list this month, asking for input on a bug in Debian’s ca-certificates-java package. The issue is that the Java key management tools embed timestamps in its output, and this output ends up in the /etc/ssl/certs/java/cacerts file on the generated ISO images. A discussion resulted from Roland’s post suggesting some short- and medium-term solutions to the problem.


Holger Levsen uploaded some packages with reproducibility-related changes:


Lastly, 12 reviews of Debian packages were added, 5 were updated and 21 were removed this month adding to our knowledge about identified issues in Debian.


Reproducible builds by default in Maven 4

On our mailing list this month, Hervé Boutemy reported the latest release of Maven (4.0.0-beta-5) has reproducible builds enabled by default. In his mailing list post, Hervé mentions that “this story started during our Reproducible Builds summit in Hamburg”, where he created the upstream issue that builds on a “multi-year” effort to have Maven builds configured for reproducibility.


PyPI now supports digital attestations

Elsewhere in the Python ecosystem and as reported on LWN and elsewhere, the Python Package Index (PyPI) has announced that it has finalised support for PEP 740 (“Index support for digital attestations”).

Trail of Bits, who performed much of the development work, has an in-depth blog post about the work and its adoption, as well as what is left undone:

One thing is notably missing from all of this work: downstream verification. […]

This isn’t an acceptable end state (cryptographic attestations have defensive properties only insofar as they’re actually verified), so we’re looking into ways to bring verification to individual installing clients. In particular, we’re currently working on a plugin architecture for pip that will enable users to load verification logic directly into their pip install flows.

There was an in-depth discussion on LWN’s announcement page, as well as on Hacker News.


Dependency Challenges in OSS Package Registries

At BENEVOL, the Belgium-Netherlands Software Evolution workshop in Namur, Belgium, Tom Mens and Alexandre Decan presented their paper, “An Overview and Catalogue of Dependency Challenges in Open Source Software Package Registries”.

The abstract of their paper is as follows:

While open-source software has enabled significant levels of reuse to speed up software development, it has also given rise to the dreadful dependency hell that all software practitioners face on a regular basis. This article provides a catalogue of dependency-related challenges that come with relying on OSS packages or libraries. The catalogue is based on the scientific literature on empirical research that has been conducted to understand, quantify and overcome these challenges. []

A PDF of the paper is available online.


Zig programming language demonstrated reproducible

Motiejus Jakšty posted an interesting and practical blog post on his successful attempt to reproduce the Zig programming language without using the pre-compiled binaries checked into the repository, and despite the circular dependency inherent in its bootstrapping process.

As a summary, Motiejus concludes that:

I can now confidently say (and you can also check, you don’t need to trust me) that there is nothing hiding in zig1.wasm [the checked-in binary] that hasn’t been checked-in as a source file.

The full post is full of practical details, and includes a few open questions.


Website updates

Notwithstanding the significant change to the landing page (screenshot above), there were an enormous number of changes made to our website this month. This included:

  • Alex Feyerke and Mariano Giménez:

  • Bernhard M. Wiedemann:

    • Update the “System images” page to document the e2fsprogs approach. []
  • Chris Lamb:

  • FC (Fay) Stegerman:

    • Replace more inline markdown with HTML on the “Success stories” page. []
    • Add some links, fix some other links and correct some spelling errors on the “Tools” page. []
  • Holger Levsen:

    • Add a historical presentation (“Reproducible builds everywhere eg. in Debian, OpenWrt and LEDE”) from October 2016. []
    • Add jochensp and Oejet to the list of known contributors. [][]
  • Julia Krüger:

  • Ninette Adhikari & hulkoba:

    • Add/rework the list of success stories into a new page that clearly shows milestones in Reproducible Builds. [][][][][][]
  • Philip Rinn:

    • Import 47 historical weekly reports. []
  • hulkoba:

    • Add alt text to almost all images (!). [][]
    • Fix a number of links on the “Talks”. [][]
    • Avoid so-called ‘ghost’ buttons by not using <button> elements as links, as the affordance of a <button> implies an action with (potentially) a side effect. [][]
    • Center the sponsor logos on the homepage. []
    • Move publications and generate them instead from a data.yml file with an improved layout. [][]

    • Make a large number of small but impactful stylisting changes. [][][][]

    • Expand the “Tools” to include a number of missing tools, fix some styling issues and fix a number of stale/broken links. [][][][][][]


Upstream patches

The Reproducible Builds project detects, dissects and attempts to fix as many currently-unreproducible packages as possible. We endeavour to send all of our patches upstream where appropriate. This month, we wrote a large number of such patches, including:


Misc development news


Reproducibility testing framework

The Reproducible Builds project operates a comprehensive testing framework running primarily at tests.reproducible-builds.org in order to check packages and other artifacts for reproducibility. In November, a number of changes were made by Holger Levsen, including:

  • reproduce.debian.net-related changes:

    • Create and introduce a new reproduce.debian.net service and subdomain []
    • Make a large number of documentation changes relevant to rebuilderd. [][][][][]
    • Explain a temporary workaround for a specific issue in rebuilderd. []
    • Setup another rebuilderd instance on the o4 node and update installation documentation to match. [][]
    • Make a number of helpful/cosmetic changes to the interface, such as clarifying terms and adding links. [][][][][]
    • Deploy configuration to the /opt and /var directories. [][]
    • Add an infancy (or ‘alpha’) disclaimer. [][]
    • Add more notes to the temporary rebuilderd documentation. []
    • Commit an nginx configuration file for reproduce.debian.net’s “Stats” page. []
    • Commit a rebuilder-worker.conf configuration for the o5 node. []
  • Debian-related changes:

    • Grant jspricke and jochensp access to the o5 node. [][]
    • Build the qemu package with the nocheck build flag. []
  • Misc changes:

    • Adapt the update_jdn.sh script for new Debian trixie systems. []
    • Stop installing the PostgreSQL database engine on the o4 and o5 nodes. []
    • Prevent accidental reboots of the o4 node because of a long-running job owned by josch. [][]

In addition, Mattia Rizzolo addressed a number of issues with reproduce.debian.net [][][][]. And lastly, both Holger Levsen [][][][] and Vagrant Cascadian [][][][] performed node maintenance.


If you are interested in contributing to the Reproducible Builds project, please visit our Contribute page on our website. However, you can get in touch with us via:

05 December, 2024 12:47PM

December 04, 2024

Scarlett Gately Moore

Hacked and tis the season for surgeries

I am still here. Sadly while I battle this insane infection from my broken arm I got back in July, the hackers got my blog. I am slowly building it back up. Further bad news is I have more surgeries, first one tomorrow. Furthering my current struggles I cannot start my job search due to hospitalization and recovery. Please consider a donation. https://gofund.me/6e99345d

On the open source work front, I am still working on stuff, mostly snaps ( Apps 24.08.3 released )

Thank you everyone that voted me into the Ubuntu Community Council!

I am trying to stay positive, but it seems I can’t catch a break. I will have my computer in the hospital and will work on what I can. Have a blessed day and see you soon.

Scarlett

04 December, 2024 05:30PM by sgmoore

Sven Hoexter

Looking at x509 Certificate Chains

Sometimes you've to look at the content of x509 certificate chains. Usually one finds them pem encoded and concatenated in a text file.

Since the openssl x509 subcommand only decodes the first certificate it will find in a file, I did something like this:

csplit -z -f 'cert' fullchain.pem '/-----BEGIN CERTIFICATE-----/' '{*}'
for x in cert*; do openssl x509 -in $x -noout -text; done

Apparently that's the "wrong" way and the more appropriate way is using the openssl crl2pkcs7 subcommand albeit we do not try to parse a revocation list here.

  openssl crl2pkcs7 -nocrl -certfile fullchain.pem | \
  openssl pkcs7 -print_certs -noout

Learned that one in a webinar presented by Victor Dukhovni. If you're new to the topic worth watching.

04 December, 2024 05:23PM

Enrico Zini

How to right click

I climbed on top of a mountain with a beautiful view, and when I started readying my new laptop for a work call (as one does on top of mountains), I realised that I couldn't right click and it kind of spoiled the mood.

Clicking on the bottom right corner of my touchpad left-clicked. Clicking with two fingers left-clicked. Alt-clicking, Super-clicking, Control-clicking, left clicked.

Here's there are two ways to simulate mouse buttons with touchpads in Wayland:

  • clicking on different areas at the bottom of the touchpad
  • double or triple-tapping, as long as the fingers are not too far apart

Skippable digression:

I'm not sure why Gnome insists in following Macs for defaults, which is what people with non-Mac hardware are less likely to be used to.

In my experience, Macs are as arbitrarily awkward to use as anything else, but they managed to build a community where if you don't understand how it works you get told you're stupid. All other systems (including Gnome) have communities where instead you get told (as is generally the case) that the system design is stupid, which at least gives you some amount of validation in your suffering.

Oh well.

How to configure right click

Surprisingly, this is not available in Gnome Shell settings. It can be found in gnome-tweaks: under "Keyboard & Mouse", "Mouse Click Emulation", one can choose between "Fingers" or "Area".

I tried both and went for "Area": I use right-drag a lot to resize windows, and I couldn't find a way, at least with this touchpad, to make it work consistently in "Fingers" mode.

04 December, 2024 04:51PM

hackergotchi for Bits from Debian

Bits from Debian

"Ceratopsian" will be the default theme for Debian 13

The theme "Ceratopsian" by Elise Couper has been selected as the default theme for Debian 13 "trixie". The theme is inspired by Trixie's (the fictional character from Toy Story) frill and is also influenced by a previously used theme called "futurePrototype" by Alex Makas.

Ceratopsian wallpaper. Click to see the whole theme proposal

Ceratopsian Website banner. Click to see the whole theme proposal

After the Debian Desktop Team made the call for proposing themes, a total of six choices were submitted. The desktop artwork poll was open to the public, and we received 2817 responses ranking the different choices, of which Ceratopsian has been ranked as the winner among them.

We'd like to thank all the designers that have participated and have submitted their excellent work in the form of wallpapers and artwork for Debian 13.

Congratulations, Elise, and thank you very much for your contribution to Debian!

04 December, 2024 12:30PM by Jonathan Carter

December 02, 2024

Bits from the DPL

This is bits from DPL for November.

MiniDebConf Toulouse

I had the pleasure of attending the MiniDebConf in Toulouse, which featured a range of engaging talks, complementing those from the recent MiniDebConf in Cambridge. Both events were preceded by a DebCamp, which provided a valuable opportunity for focused work and collaboration.

DebCamp

During these events, I participated in numerous technical discussions on topics such as maintaining long-neglected packages, team-based maintenance, FTP master policies, Debusine, and strategies for separating maintainer script dependencies from runtime dependencies, among others. I was also fortunate that members of the Publicity Team attended the MiniDebCamp, giving us the opportunity to meet in person and collaborate face-to-face.

Independent of the ongoing lengthy discussion on the Debian Devel mailing list, I encountered the perspective that unifying Git workflows might be more critical than ensuring all packages are managed in Git. While I'm uncertain whether these two questions--adopting Git as a universal development tool and agreeing on a common workflow for its use--can be fully separated, I believe it's worth raising this topic for further consideration.

Attracting newcomers

In my own talk, I regret not leaving enough time for questions--my apologies for this. However, I want to revisit the sole question raised, which essentially asked: Is the documentation for newcomers sufficient to attract new contributors? My immediate response was that this question is best directed to new contributors themselves, as they are in the best position to identify gaps and suggest improvements that could make the documentation more helpful.

That said, I'm personally convinced that our challenges extend beyond just documentation. I don't get the impression that newcomers are lining up to join Debian only to be deterred by inadequate documentation. The issue might be more about fostering interest and engagement in the first place.

My personal impression is that we sometimes fail to convey that Debian is not just a product to download for free but also a technical challenge that warmly invites participation. Everyone who respects our Code of Conduct will find that Debian is a highly diverse community, where joining the project offers not only opportunities for technical contributions but also meaningful social interactions that can make the effort and time truly rewarding.

In several of my previous talks (you can find them on my talks page –just search for "team," and don't be deterred if you see "Debian Med" in the title; it's simply an example), I emphasized that the interaction between a mentor and a mentee often plays a far more significant role than the documentation the mentee has to read. The key to success has always been finding a way to spark the mentee's interest in a specific topic that resonates with their own passions.

Bug of the Day

In my presentation, I provided a brief overview of the Bug of the Day initiative, which was launched with the aim of demonstrating how to fix bugs as an entry point for learning about packaging. While the current level of interest from newcomers seems limited, the initiative has brought several additional benefits.

I must admit that I'm learning quite a bit about Debian myself. I often compare it to exploring a house's cellar with a flashlight –you uncover everything from hidden marvels to things you might prefer to discard. I've also come across traces of incredibly diligent people who have invested their spare time polishing these hidden treasures (what we call NMUs). The janitor, a service in Salsa that automatically updates packages, fits perfectly into this cellar metaphor, symbolizing the ongoing care and maintenance that keep everything in order. I hadn't realized the immense amount of silent work being done behind the scenes--thank you all so much for your invaluable QA efforts.

Reproducible builds

It might be unfair to single out a specific talk from Toulouse, but I'd like to highlight the one on reproducible builds. Beyond its technical focus, the talk also addressed the recent loss of Lunar, whom we mourn deeply. It served as a tribute to Lunar's contributions and legacy. Personally, I've encountered packages maintained by Lunar and bugs he had filed. I believe that taking over his packages and addressing the bugs he reported is a meaningful way to honor his memory and acknowledge the value of his work.

Advent calendar bug squashing

I’d like to promote an idea originally introduced by Thorsten Alteholz, who in 2011 proposed a Bug Squashing Advent Calendar for the Debian Med team. (For those unfamiliar with the concept of an Advent Calendar, you can find an explanation on Wikipedia.) While the original version included a fun graphical element —which we’ve had to set aside due to time constraints (volunteers, anyone?)— we’ve kept the tradition alive by tackling one bug per day from December 1st to 24th each year. This initiative helps clean up issues that have accumulated over the year.

Regardless of whether you celebrate the concept of Advent, I warmly recommend this approach as a form of continuous bug-squashing party for every team. Not only does it contribute to the release readiness of your team’s packages, but it’s also an enjoyable and bonding activity for team members.

Best wishes for a cheerful and productive December

Andreas.

02 December, 2024 11:00PM by Andreas Tille

hackergotchi for Jonathan Dowland

Jonathan Dowland

jungle/acid/etc

I thought it had been a full year since I last shared a playlist, but it's been two! I had a plan to produce more, but it seems I haven't. Instead here's a few tracks I've discovered recently which share a common theme.

In August I stumbled across a Sound on Sound video interviewing Pete Cannon, who creates authentically old-school Jungle music using tools and techniques from the time, including AKAI samplers and the Commodore Amiga computer.

Here's three tracks that I found since then. Some 8-bit Amiga-jungle,

by

some slower-paced acid house from someone ostensibly based on Whitley Bay,

by

and a darker piece I heard on the radio.

by

02 December, 2024 10:00PM

hackergotchi for Junichi Uekawa

Junichi Uekawa

Graph for my furusato tax.

Graph for my furusato tax. Exceeding 150-man will inevitably exceed 50-man limit for Ichiji-shotoku. So added some rough calculation there. graph.

02 December, 2024 05:39AM by Junichi Uekawa

Charles

Hello World

Or how it took more than a year for me to set up this website -

As the computer science tradition demands, we must start with a Hello World.

Though I have to say this hello world took quite a long time to reach the internet. I’ve been thinking about setting up this website for way over a year, but there are always too many things to decide - what Static Site Generator will I use? Where should I get a domain from? Which registrar would be better now? What if I want to set up a mail server, is it good enough? Oh, and what about the theme, which one to choose? Can I get one simple enough to not fetch javascript or css from external sources?

This was taking so long that even my friends were saying “Please, just share your screen and let’s do it now!”. Well, rejoice friends, now it’s done!

02 December, 2024 02:18AM

December 01, 2024

hackergotchi for Guido Günther

Guido Günther

Free Software Activities November 2024

Another short status update of what happened on my side last month. The larger blocks are the Phosh 0.43 release, the initial file chooser portal, phosh-osk-stub now handling digit, number, phone and PIN input purpose via special layouts as well as Phoc mostly catching up with wlroots 0.18 and the current development version targeting 0.19.

phosh

  • When taking a screenshot via keybinding or power button long press save screenshots to clipboard and disk (MR)
  • Robustify Screenshot CI job (MR)
  • Update CI pipeline (MR)
  • Fix notifications banners that aren't tall enough not being shown (MR). Another 4y old bug hopefully out of the way.
  • Add rfkill mock and docs (MR). Useful for HKS testing.
  • Release 0.43~rc1 and 0.43
  • Drop libsoup workaround (MR)
  • Ensure notification only takes its actual height (MR)

phoc

  • Move wlroots 0.18 update forward (MR). Needs a bit more work before we can make it default.
  • Catch up with wlroots development branch (MR) allowing us to test current wlroots again.
  • Some of the above already applies to main so schedule it for 0.44 (MR)

phosh-mobile-settings

  • Don't mark release notes as translatable to save some i18n effort (MR)
  • Release 0.43~rc1 and 0.43.0

libphosh-rs

phosh-osk-stub

  • Add layouts for PIN, number and phone input purpose (MR)
  • Release 0.43~rc1
  • Ensure translation get picked up, various cleanups and release 0.43.0 (MR)
  • Make desktop file match app-id (MR)

phosh-tour

  • Fix typo and reduce number of strings to translate (MR)
  • Add translator comments (MR). This, the above and additional fixes in p-m-s were prompted by i18n feedback from Alexandre Franke, thanks a lot!
  • Release 0.43.0

pfs

  • Initial version of the adaptive file chooser dialog using gtk-rs. See demo.
  • Allow to activate via double click (for non-touch use) (MR)

xdg-desktop-portal-phosh

  • Use pfs to provide a file chooser portal (MR)

meta-phosh

  • Slightly improve point release handling (MR)
  • Improve string freeze announcements and add phosh-tour (MR)

Debian

  • Upload Phosh 0.43.0~rc1 and 0.43.0 (MR, MR, MR, MR, MR, MR, MR, MR, MR, MR, MR)
  • meta-phosh: Add Recommend: for xdg-desktop-portal-phosh (MR)
  • phosh-osk-data got accepted, create repo, brush up packaging and upload to unstable (MR
  • phosh-osk-stub: Recommend data packager (MR
  • Phosh: drop reverts (MR)
  • varnam-schemes: Fix autopkgtest (MR)
  • varnam-schemes: Improve packaging (MR)
  • Prepare govarnam 1.9.1 (MR)

Calls

  • ussd: Set input purpose and switch to AdwDialog (MR, Screenshot)

libcall-ui

  • Drop libhandy leftover (MR)

git-buildpackage

  • Improve docs and cleanup markdown (MR)
  • Mention gbp push in intro (MR)
  • Use application instead of productname entities to improve reading flow (MR)

wlroots

  • Drop mention of wlr_renderer_begin_with_buffer (MR)

python-dbusmock

  • Add mock for gsd-rfkill (MR)

xdg-spec

  • Sync notification categories with the portal spec (MR)
  • Add categories for SMS (MR)
  • Add a pubdate so it's clear the specs aren't stale (MR) (got fixed in a different and better way, thanks Matthias!)

ashpd

  • Allow to set filters in file chooser portal demo (MR)

govarnam

  • Robustify file generation (MR)

varnam-schemes

  • Unbreak tests on non intel/amd architectures (e.g. arm64) (MR)

Reviews

This is not code by me but reviews I did on other peoples code. The list is incomplete, but I hope to improve on this in the upcoming months. Thanks for the contributions!

  • flathub: livi runtime and gst update (MR)
  • phosh: Split linters into their own test suite (MR)
  • phosh; QuickSettings follow-up (MR)
  • phosh: Accent color fixes (MR)
  • phosh: Notification animation (MR)
  • phosh: end-session dialog timeout fix (MR)
  • phosh: search daemon (MR)
  • phosh-ev: Migrate to newer gtk-rs and async_channel (MR)
  • phosh-mobile-settings: Update gmobile (MR)
  • phosh-mobile-settings: Make panel-switcher scrollable (MR)
  • phosh-mobile-settings: i18n comments (MR)
  • gbp doc updates (MR)
  • gbp handle suite names with number prefix (MR)
  • Debian libvirt dependency changes (MR
  • Chatty: misc improvements (MR
  • iio-sensor-proxy: buffer driver without trigger (MR)
  • gbp doc improvements (MR)
  • gbp: More doc improvements (MR)
  • gbp: Clean on failure (MR)
  • gbp: DEP naming consistency (MR)

Help Development

If you want to support my work see donations. This includes a list of hardware we want to improve support for. Thanks a lot to all current and past donors.

Comments?

Join the Fediverse thread

01 December, 2024 06:55PM

hackergotchi for Colin Watson

Colin Watson

Free software activity in November 2024

Most of my Debian contributions this month were sponsored by Freexian.

You can also support my work directly via Liberapay.

Conferences

I attended MiniDebConf Toulouse 2024, and the MiniDebCamp before it. Most of my time was spent with the Freexian folks working on debusine; Stefano gave a talk about its current status with a live demo (frantically fixed up over the previous couple of days, as is traditional) and with me and others helping to answer questions at the end. I also caught up with some people I haven’t seen in ages, ate a variety of delicious cheeses, and generally had a good time. Many thanks to the organizers and sponsors!

After the conference, Freexian collaborators spent a day and a half doing some planning for next year, and then went for an afternoon visiting the Cité de l’espace.

Rust team

I upgraded these packages to new upstream versions, as part of upgrading pydantic and rpds-py:

  • rust-archery
  • rust-jiter (noticing an upstream test bug in the process)
  • rust-pyo3 (fixing CVE-2024-9979)
  • rust-pyo3-build-config
  • rust-pyo3-ffi
  • rust-pyo3-macros
  • rust-pyo3-macros-backend
  • rust-regex
  • rust-regex-automata
  • rust-regex
  • rust-serde
  • rust-serde-derive
  • rust-serde-json
  • rust-speedate
  • rust-triomphe

Python team

Last month, I mentioned that we still need to work out what to do about the multipart vs. python-multipart name conflict in Debian (#1085728). We eventually managed to come up with an agreed plan; Sandro has uploaded a renamed binary package to experimental, and I’ve begun work on converting reverse-dependencies (asgi-csrf, fastapi, python-curies, and starlette done so far). There’s a bit more still to do, but I expect we can finish it soon.

I fixed problems related to adding Python 3.13 support in:

I fixed some packaging problems that resulted in failures any time we add a new Python version to Debian:

I fixed other build/autopkgtest failures in:

I packaged python-quart-trio, needed for a new upstream version of python-urllib3, and contributed a small packaging tweak upstream.

I backported a twisted fix that caused problems in other packages, including breaking debusine‘s tests.

I disentangled some upstream version confusion in python-catalogue, and upgraded to the current upstream version.

I upgraded these packages to new upstream versions:

Other small fixes

I contributed Incus support to needrestart upstream.

In response to Helmut’s Cross building talk at MiniDebConf Toulouse, I fixed libfilter-perl to support cross-building (5b4c2e10, f9788c27).

I applied a patch to move aliased files from / to /usr in iprutils (#1087733).

I adjusted debconf to use the new /usr/lib/apt/apt-extracttemplates path (#1087523).

I upgraded putty to 0.82.

01 December, 2024 03:00PM by Colin Watson

hackergotchi for Junichi Uekawa

Junichi Uekawa

Lots of travel and back to Tokyo.

Lots of travel and back to Tokyo. Then I got sick. Trying to work on my bass piece, but it's really hard and I am having hard time getting to a reasonable shape. Discussions on Debconf 2026 bid. Hoping it will materialize soon.

01 December, 2024 06:36AM by Junichi Uekawa

hackergotchi for Sandro Knauß

Sandro Knauß

QML Dependency tracking in Debian

Tracking library dependencies work in Debian to resolve from symbols usage to a library and add this to the list of dependencies. That is working for years now. The KDE community nowadays create more and more QML based applications. Unfortunately QML is a interpreted language, this means missing QML dependencies will only be an issue at runtime.

To fix this I created dh_qmldeps, that searches for QML dependencies at build time and will fail if it can't resolve the QML dependency.

Me didn't create an own QML interpreter, just using qmlimportscanner behind the scenes and process the output further to resolve the QML modules to Debian packages.

The workflow is like follows:

The package compiles normally and split to the binary packages. Than dh_qmldeps scans through the package content to find QML content ( .qml files, or qmldirfor QML modules). All founded files will be scanned by qmlimportscanner, the output is a list of depended QML modules. As QML modules have a standardized file path, we can ask the Debian system, which packages ship this file path. We end up with a list of Debian packages in the variable ${qml6:Depends}. This variable can be attached to the list of dependencies of the scanned package. A maintainer can also lower some dependencies to Recommends or Suggest, if needed.

You can find the source code on salsa and usage documentation you can find on https://qt-kde-team.pages.debian.net/dh_qmldeps.html.

The last weeks I now enabled dh_qmldeps for newly every package, that creates a QML6 module package. So the first bugs are solved and it should be usable for more packages.

By scanning with qmlimportscanner trough all code, I found several non-existing QML modules:

  • import QtQuick3DPrivate qt6-multimedia - no Private QML module QTBUG-131753.
  • import QtQuickPrivate qt6-graphs - no Private QML module QTBUG-131754.
  • import QtQuickTimeline qt6-quicktimeline - the correct QML name is QtQuick.Timeline QTBUG-131755.
  • import QtQuickControls2 qt6-webengine - looks like a porting bug as the QML6 modules name is QtQuick.Controls QTBUG-131756.
  • import QtGraphicalEffects kquickimageeditor - the correct name is for QML6 is qt5compat.graphicaleffects, properly as it is an example nobody checks it kquickimageeditor!7.

YEAH - the first milestone is reached. We are able to simply handle QML modules.

But QML applications there is still room for improvement. In apps the QML files are inside the executable. Additionally applications create internal QML modules, that are shipped directly in the same executable. I still search for a good way to analyse an executable to get a list of internal QML modules and a list of included QML files. Any ideas are welcomed :)

As workaround dh_qmldeps scans currently all QML files inside the application source code.

01 December, 2024 12:00AM by Sandro Knauß

November 30, 2024

Dima Kogan

Strava track filtering validation

After years of seeing people's strava tracks, I became convinced that they insufficiently filter the data, resulting in over-estimating the effort. Today I did a bit of lazy analysis, and half-confirmed this: in the one case I looked at, strava reported reasonable elevation gain numbers, but greatly overestimated the distance traveled.

I looked at a single gps track of a long bike ride. This was uploaded to strava manually, as a .gpx file. I can imagine that different things happen if you use the strava app or some device that integrates with the service (the filtering might happen before the data hits the server, and the server could decide to not apply any more filtering).

I processed the data with a simple hysteretic filter, ignoring small changes in position and elevation, trying out different thresholds in the process. I completely ignore the timestamps, and only look at the differences between successive points. This handles the usual GPS noise; it does not handle GPS jumps, which I completely ignore in this analysis. Ignoring these would produce inflated elevation/gain numbers, but I'm working with a looong track, so hopefully this is a small effect.

Clearly this is not scientific, but it's something.

The code

Parsing .gpx is slow (this is a big file), so I cache that into a .vnl:

import sys
import gpxpy

filename_in  = 'INPUT.gpx'
filename_out = 'OUTPUT.gpx'

with open(filename_in, 'r') as f:
    gpx = gpxpy.parse(f)

f_out = open(filename_out, 'w')

tracks = gpx.tracks
if len(tracks) != 1:
    print("I want just one track", file=sys.stderr)
    sys.exit(1)
track = tracks[0]

segments = track.segments
if len(segments) != 1:
    print("I want just one segment", file=sys.stderr)
    sys.exit(1)
segment = segments[0]

time0 = segment.points[0].time
print("# time lat lon ele_m")
for p in segment.points:
    print(f"{(p.time - time0).seconds} {p.latitude} {p.longitude} {p.elevation}",
          file = f_out)

And I process this data with the different filters (this is a silly Python loop, and is slow):

#!/usr/bin/python3

import sys
import numpy as np
import numpysane as nps
import gnuplotlib as gp
import vnlog
import pyproj

geod = None
def dist_ft(lat0,lon0, lat1,lon1):

    global geod
    if geod is None:
        geod = pyproj.Geod(ellps='WGS84')
    return \
        geod.inv(lon0,lat0, lon1,lat1)[2] * 100./2.54/12.




f = 'OUTPUT.gpx'

track,list_keys,dict_key_index = \
    vnlog.slurp(f)

t      = track[:,dict_key_index['time' ]]
lat    = track[:,dict_key_index['lat'  ]]
lon    = track[:,dict_key_index['lon'  ]]
ele_ft = track[:,dict_key_index['ele_m']] * 100./2.54/12.



@nps.broadcast_define( ( (), ()),
                       (2,))
def filter_track(ele_hysteresis_ft,
                 dxy_hysteresis_ft):

    dist        = 0.0
    ele_gain_ft = 0.0

    lon_accepted = None
    lat_accepted = None
    ele_accepted = None

    for i in range(len(lat)):

        if ele_accepted is not None:
            dxy_here  = dist_ft(lat_accepted,lon_accepted, lat[i],lon[i])
            dele_here = np.abs( ele_ft[i] - ele_accepted )

            if dxy_here < dxy_hysteresis_ft and dele_here < ele_hysteresis_ft:
                continue

            if ele_ft[i] > ele_accepted:
                ele_gain_ft += dele_here;

            dist += np.sqrt(dele_here * dele_here +
                            dxy_here  * dxy_here)

        lon_accepted = lon[i]
        lat_accepted = lat[i]
        ele_accepted = ele_ft[i]

    # lose the last point. It simply doesn't matter

    dist_mi = dist / 5280.
    return np.array((ele_gain_ft, dist_mi))




Nele_hysteresis_ft    = 20
ele_hysteresis0_ft    = 5
ele_hysteresis1_ft    = 100
ele_hysteresis_ft_all = np.linspace(ele_hysteresis0_ft,
                                    ele_hysteresis1_ft,
                                    Nele_hysteresis_ft)

Ndxy_hysteresis_ft = 20
dxy_hysteresis0_ft = 5
dxy_hysteresis1_ft = 1000
dxy_hysteresis_ft  = np.linspace(dxy_hysteresis0_ft,
                                 dxy_hysteresis1_ft,
                                 Ndxy_hysteresis_ft)


# shape (Nele,Ndxy,2)
gain,distance = \
    nps.mv( filter_track( nps.dummy(ele_hysteresis_ft_all,-1),
                          dxy_hysteresis_ft),
            -1,0 )


# Stolen from mrcal
def options_heatmap_with_contours( plotoptions, # we update this on output

                                   *,
                                   contour_min           = 0,
                                   contour_max,
                                   contour_increment     = None,
                                   do_contours           = True,
                                   contour_labels_styles = 'boxed',
                                   contour_labels_font   = None):
    r'''Update plotoptions, return curveoptions for a contoured heat map'''

    gp.add_plot_option(plotoptions,
                       'set',
                       ('view equal xy',
                        'view map'))

    if do_contours:
        if contour_increment is None:
            # Compute a "nice" contour increment. I pick a round number that gives
            # me a reasonable number of contours

            Nwant = 10
            increment = (contour_max - contour_min)/Nwant

            # I find the nearest 1eX or 2eX or 5eX
            base10_floor = np.power(10., np.floor(np.log10(increment)))

            # Look through the options, and pick the best one
            m   = np.array((1., 2., 5., 10.))
            err = np.abs(m * base10_floor - increment)
            contour_increment = -m[ np.argmin(err) ] * base10_floor

        gp.add_plot_option(plotoptions,
                           'set',
                           ('key box opaque',
                            'style textbox opaque',
                            'contour base',
                            f'cntrparam levels incremental {contour_max},{contour_increment},{contour_min}'))

        if contour_labels_font is not None:
            gp.add_plot_option(plotoptions,
                               'set',
                               f'cntrlabel format "%d" font "{contour_labels_font}"' )
        else:
            gp.add_plot_option(plotoptions,
                               'set',
                               f'cntrlabel format "%.0f"' )

        plotoptions['cbrange'] = [contour_min, contour_max]

        # I plot 3 times:
        # - to make the heat map
        # - to make the contours
        # - to make the contour labels
        _with = np.array(('image',
                          'lines nosurface',
                          f'labels {contour_labels_styles} nosurface'))
    else:
        gp.add_plot_option(plotoptions, 'unset', 'key')
        _with = 'image'

    using = \
        f'({dxy_hysteresis0_ft}+$1*{float(dxy_hysteresis1_ft-dxy_hysteresis0_ft)/(Ndxy_hysteresis_ft-1)}):' + \
        f'({ele_hysteresis0_ft}+$2*{float(ele_hysteresis1_ft-ele_hysteresis0_ft)/(Nele_hysteresis_ft-1)}):3'
    plotoptions['_3d']     = True
    plotoptions['_xrange'] = [dxy_hysteresis0_ft,dxy_hysteresis1_ft]
    plotoptions['_yrange'] = [ele_hysteresis0_ft,ele_hysteresis1_ft]
    plotoptions['ascii']   = True # needed for using to work

    gp.add_plot_option(plotoptions, 'unset', 'grid')

    return \
        dict( tuplesize=3,
              legend = "", # needed to force contour labels
              using = using,
              _with=_with)




contour_granularity = 1000
plotoptions = dict()
curveoptions = \
    options_heatmap_with_contours( plotoptions, # we update this on output
                                   # round down to the nearest contour_granularity
                                   contour_min = (np.min(gain) // contour_granularity)*contour_granularity,
                                   # round up to the nearest contour_granularity
                                   contour_max = ((np.max(gain) + (contour_granularity-1)) // contour_granularity) * contour_granularity,
                                   do_contours = True)
gp.add_plot_option(plotoptions, 'unset', 'key')
gp.add_plot_option(plotoptions, 'set', 'size square')
gp.plot(gain,
        xlabel  = "Distance hysteresis (ft)",
        ylabel  = "Elevation hysteresis (ft)",
        cblabel = "Elevation gain (ft)",
        wait = True,
        **curveoptions,
        **plotoptions,
        title    = 'Computed gain vs filtering parameters')


contour_granularity = 10
plotoptions = dict()
curveoptions = \
    options_heatmap_with_contours( plotoptions, # we update this on output
                                   # round down to the nearest contour_granularity
                                   contour_min = (np.min(distance) // contour_granularity)*contour_granularity,
                                   # round up to the nearest contour_granularity
                                   contour_max = ((np.max(distance) + (contour_granularity-1)) // contour_granularity) * contour_granularity,
                                   do_contours = True)
gp.add_plot_option(plotoptions, 'unset', 'key')
gp.add_plot_option(plotoptions, 'set', 'size square')
gp.plot(distance,
        xlabel  = "Distance hysteresis (ft)",
        ylabel  = "Elevation hysteresis (ft)",
        cblabel = "Distance (miles)",
        wait = True,
        **curveoptions,
        **plotoptions,
        title    = 'Computed distance vs filtering parameters')

Results: gain

Strava says the gain was 46307ft. The analysis says:

strava-gain.png

strava-gain-zoom.png

These show the filtered gain for different values of the distance and gain hysteresis thresholds. The same data is shown at diffent zoom levels. There's no sweet spot, but we get 46307ft with a reasonable amount of filtering. Maybe 46307ft is a bit low even.

Results: distance

Strava says the distance covered was 322 miles. The analysis says:

strava-distance.png

strava-distance-zoom.png

Once again, there's no sweet spot, but we get 322 miles only if we apply no filtering at all. That's clearly too high, and is not reasonable. From the map (and from other people's strava routes) the true distance is closer to 305 miles. Why those people's strava numbers are more believable is anybody's guess.

30 November, 2024 10:48PM by Dima Kogan

Enrico Zini

New laptop setup

My new laptop Framework (Framework Laptop 13 DIY Edition (AMD Ryzen™ 7040 Series)) arrived, all the hardware works out of the box on Debian Stable, and I'm very happy indeed.

This post has the notes of all the provisioning steps, so that I can replicate them again if needed.

Installing Debian 12

Debian 12's installer just worked, with Secure Boot enabled no less, which was nice.

The only glitch is an argument with the guided partitioner, which was uncooperative: I have been hit before by a /boot partition too small, and I wanted 1G of EFI and 1G of boot, while the partitioner decided that 512Mb were good enough. Frustratingly, there was no way of changing that, nor I found how to get more than 1G of swap, as I wanted enough swap to fit RAM for hybernation.

I let it install the way it pleased, then I booted into grml for a round of gparted.

The tricky part of that was resizing the root btrfs filesystem, which is in an LV, which is in a VG, which is in a PV, which is in LUKS. Here's a cheatsheet.

Shrink partitions:

  • mount the root filesystem in /mnt
  • btrfs filesystem resize 6G /mnt
  • umount the root filesystem
  • lvresize -L 7G vgname/lvname
  • pvresize --setphysicalvolumesize /dev/mapper/pvname 8G
  • cryptsetup resize --device-size 9G name

note that I used an increasing size because I don't trust that each tool has a way of representing sizes that aligns to the byte. I'd be happy to find out that they do, but didn't want to find out the hard way that they didn't.

Resize with gparted:

Move and resize partitions at will. Shrinking first means it all takes a reasonable time, and you won't have to wait almost an hour for a terabyte-sized empty partition to be carefully moved around. Don't ask me why I know.

Regrow partitions:

  • cryptsetup resize name
  • pvresize /dev/mapper/pvname
  • lvresize -L 100% vgname/lvname
  • mount the root filesystem in /mnt
  • btrfs filesystem resize max /mnt
  • umount the root filesystem

Setup gnome

When I get a new laptop I have a tradition of trying to make it work with Gnome and Wayland, which normally ended up in frustration and a swift move to X11 and Xfce: I have a lot of long-time muscle memory involved in how I use a computer, and it needs to fit like prosthetics. I can learn to do a thing or two in a different way, but any papercut that makes me break flow and I cannot fix will soon become a dealbreaker.

This applies to Gnome as present in Debian Stable.

General Gnome settings tips

I can list all available settings with:

gsettings list-recursively

which is handy for grepping things like hotkeys.

I can manually set a value with:

gsettings set <schema> <key> <value>

and I can reset it to its default with:

gsettings reset <schema> <key>

Some applications like Gnome Terminal use "relocatable schemas", and in those cases you also need to specify a path, which can be discovered using dconf-editor:

gsettings set <schema>:<path> <key> <value>

Install appindicators

First thing first: app install gnome-shell-extension-appindicator, log out and in again: the Gnome Extension manager won't see the extension as available until you restart the whole session.

I have no idea why that is so, and I have no idea why a notification area is not present in Gnome by default, but at least now I can get one.

Fix font sizes across monitors

My laptop screen and monitor have significantly different DPIs, so:

gsettings set org.gnome.mutter experimental-features "['scale-monitor-framebuffer']"

And in Settings/Displays, set a reasonable scaling factor for each display.

Disable Alt/Super as hotkey for the Overlay

Seeing all my screen reorganize and reshuffle every time I accidentally press Alt leaves me disoriented and seasick:

gsettings set org.gnome.mutter overlay-key ''

Focus-follows-mouse and Raise-or-lower

My desktop is like my desktop: messy and cluttered. I have lots of overlapping window and I switch between them by moving the focus with the mouse, and when the visible part is not enough I have a handy hotkey mapped to raise-or-lower to bring forward what I need and send back what I don't need anymore.

Thankfully Gnome can be configured that way, with some work:

  • In gnome-shell settings, keyboard, shortcuts, windows, set "Raise window if covered, otherwise lower it" to "Super+Escape"
  • In gnome-tweak-tool, Windows, set "Focus on Hover"

This almost worked, but sometimes it didn't do what I wanted, like I expected to find a window to the front but another window disappeared instead. I eventually figured that by default Gnome delays focus changes by a perceivable amount, which is evidently too slow for the way I move around windows.

The amount cannot be shortened, but it can be removed with:

gsettings set org.gnome.shell.overrides focus-change-on-pointer-rest false

Mouse and keyboard shortcuts

Gnome has lots of preconfigured sounds, shortcuts, animations and other distractions that I do not need. They also either interfere with key combinations I want to use in terminals, or cause accidental window moves or resizes that make me break flow, or otherwise provide sensory overstimulation that really does not work for me.

It was a lot of work, and these are the steps I used to get rid of most of them.

Disable Super+N combinations that accidentally launch a questionable choice of programs:

for i in `seq 1 9`; do gsettings set org.gnome.shell.keybindings switch-to-application-$i '[]'; done

Gnome-Shell settings:

  • Multitasking:
    • disable hot corner
    • disable active edges
    • set a fixed number of workspaces
    • workspaces on all displays
    • switching includes apps from current workspace only
  • Sound:
    • disable system sounds
  • Keyboard
    • Compose Key set to Caps Lock
    • View and Customize Shortcuts:
      • Launchers
        • launch help browser: remove
      • Navigation
        • move to workspace on the left: Super+Left
        • move to workspace on the right: Super+Right
        • move window one monitor …: remove
        • move window one workspace to the left: Shift+Super+Left
        • move window one workspace to the right: Shift+Super+Right
        • move window to …: remove
        • switch system …: remove
        • switch to …: remove
        • switch windows …: disabled
      • Screenshots
        • Record a screenshot interactively: Super+Print
        • Take a screenshot interactively: Print
        • Disable everything else
      • System
        • Focus the active notification: remove
        • Open the applcation menu: remove
        • Restore the keyboard shortctus: remove
        • Show all applications: remove
        • Show the notification list: remove
        • Show the overvire: remove
        • Show the run command prompt: remove (the default Gnome launcher is not for me) Super+F2 (or remove to leave it to the terminal)
      • Windows
        • Close window: remove
        • Hide window: remove
        • Maximize window: remove
        • Move window: remove
        • Raise window if covered, otherwise lower it: Super+Escape
        • Resize window: remove
        • Restore window: remove
        • Toggle maximization state: remove
    • Custom shortcuts
      • xfrun4, launching xfrun4, bound to Super+F2
  • Accessibility:
    • disable "Enable animations"

gnome-tweak-tool settings:

  • Keyboard & Mouse
    • Overview shortcut: Right Super. This cannot be disabled, but since my keyboard doesn't have a Right Super button, that's good enough for me. Oddly, I cannot find this in gsettings.
  • Window titlebars
    • Double-Click: Toggle-Maximize
    • Middle-Click: Lower
    • Secondary-Click: Menu
  • Windows
    • Resize with secondary click

Gnome Terminal settings:

Thankfully 10 years ago I took notes on how to customize Gnome Terminal, and they're still mostly valid:

  • Shortcuts

    • New tab: Super+T
    • New window: Super+N
    • Close tab: disabled
    • Close window: disabled
    • Copy: Super+C
    • Paste: Super+V
    • Search: all disabled
    • Previous tab: Super+Page Up
    • Next tab: Super+Page Down
    • Move tab…: Disabled
    • Switch to tab N: Super+Fn (only available after disabling overview)
    • Switch to tab N with Alt+Fn cannot be configured in the UI: Alt+Fn is detected as simply Fn. It can however be set with gsettings:

      sh for i in `seq 1 12`; do gsettings set org.gnome.Terminal.Legacy.Keybindings:/org/gnome/terminal/legacy/keybindings/ switch-to-tab-$i "<Alt>F$i"; done

  • Profile

    • Text
      • Sound: disable terminal bell

Other hotkeys that got in my way and had to disable the hard way:

for n in `seq 1 12`; do gsettings set org.gnome.mutter.wayland.keybindings switch-to-session-$n '[]'; done
gsettings set org.gnome.desktop.wm.keybindings move-to-workspace-down '[]'
gsettings set org.gnome.desktop.wm.keybindings move-to-workspace-up '[]'
gsettings set org.gnome.desktop.wm.keybindings panel-main-menu '[]'
gsettings set org.gnome.desktop.interface menubar-accel '[]'

Note that even after removing F10 from being bound to menubar-accel, and after having to gsetting binding to F10 as is:

$ gsettings list-recursively|grep F10
org.gnome.Terminal.Legacy.Keybindings switch-to-tab-10 '<Alt>F10'

I still cannot quit Midnight Commander using F10 in a terminal, as that moves the focus in the window title bar. This looks like a Gnome bug, and a very frustrating one for me.

Appearance

Gnome-Shell settings:

  • Appearance:
    • dark mode

gnome-tweak-tool settings:

  • Fonts
    • Antialiasing: Subpixel
  • Top Bar
    • Clock/Weekday: enable (why is this not a default?)

Gnome Terminal settings:

  • General
    • Theme variant: Dark (somehow it wasn't picked by up from the system settings)
  • Profile
    • Colors
      • Background: #000

Other decluttering and tweaks

Gnome Shell Settings:

  • Search
    • disable application search
  • Removable media
    • set everything to "ask what to do"
  • Default applications
    • Web: Chromium
    • Mail: mutt
    • Calendar: khal is not sadly an option
    • Video: mpv
    • Photos: Geequie

Set a delay between screen blank and lock: when the screen goes blank, it is important for me to be able to say "nope, don't blank yet!", and maybe switch on caffeine mode during a presentation without needing to type my password in front of cameras. No UI for this, but at least gsettings has it:

gsettings set org.gnome.desktop.screensaver lock-delay 30

Extensions

I enabled the Applications Menu extension, since it's impossible to find less famous applications in the Overview without knowing in advance how they're named in the desktop. This stole a precious hotkey, which I had to disable in gsettings:

gsettings set org.gnome.shell.extensions.apps-menu apps-menu-toggle-menu '[]'

I also enabled:

  • Removable Drive Menu: why is this not on by default?
  • Workspace Indicator
  • Ubuntu Appindicators (apt install gnome-shell-extension-appindicator and restart Gnome)

I didn't go and look for Gnome Shell extentions outside what is packaged in Debian, as I'm very wary about running JavaScript code randomly downloaded from the internet with full access over my data and desktop interaction.

I also took care of checking that the Gnome Shell Extensions web page complains about the lacking "GNOME Shell integration" browser extension, because the web browser shouldn't be allowed to download random JavaScript from the internet and run it with full local access.

Yuck.

Run program dialog

The default run program dialog is almost, but not quite, totally useless to me, as it does not provide completion, not even just for executable names in path, and so it ends up being faster to open a new terminal window and type in there.

It's possible, in Gnome Shell settings, to bind a custom command to a key. The resulting keybinding will now show up in gsettings, though it can be located in a more circuitous way by grepping first, and then looking up the resulting path in dconf-editor:

gsettings list-recursively|grep custom-key
org.gnome.settings-daemon.plugins.media-keys custom-keybindings ['/org/gnome/settings-daemon/plugins/media-keys/custom-keybindings/custom0/']

I tried out several run dialogs present in Debian, with sad results, possibly due to most of them not being tested on wayland:

  • fuzzel does not start
  • gmrun is gtk2, last updated in 2016, but works fine
  • kupfer segfaults as I type
  • rofi shows, but can't get keboard input
  • shellex shows a white bar at top of the screen and lots of errors on stderr
  • superkb wants to grab the screen for hotkeys
  • synapse searched news on the internet as I typed, which is a big no for me
  • trabucco crashes on startup
  • wofi works but looks like very much an acquired taste, though it has some completion that makes it more useful than Gnome's run dialog
  • xfrun4 (package xfce4-appfinder) struggles on wayland, being unable to center its window and with the pulldown appearing elsewhere in the screen, but it otherwise works

Both gmrun and xfrun4 seem like workable options, with xfrun4 being customizable with convenient shortcut prefixes, so xfrun4 it is.

TODO

  • Figure out what is still binding F10 to menu, and what I can do about it
  • Figure out how to reduce the size of window titlebars, which to my taste should be unobtrusive and not take 2.7% of vertical screen size each. There's a minwaita theme which isn't packaged in Debian. There's a User Theme extension, and then the whole theming can of worms to open. For another day.
  • Figure out if Gnome can be convinced to resize popup windows? Take the Gnome Terminal shortcut preferences for example: it takes ⅓ of the vertical screen and can only display ¼ of all available shortcuts, and I cannot find a valid reason why I shouldn't be allowed to enlarge it vertically.
  • Figure out if I can place shortcut launcher icons in the top panel, and how

I'll try to update these notes as I investigate.

Conclusion so far

I now have something that seems to work for me. A few papercuts to figure out still, but they seem manageable.

It all feels a lot harder than it should be: for something intended to be minimal, Gnome defaults feel horribly cluttered and noisy to me, continuosly getting in the way of getting things done until tamed into being out of the way unless called for. It felt like a device that boots into flashy demo mode, which needs to be switched off before actual use.

Thankfully it can be switched off, and now I have notes to do it again if needed.

gsettings oddly feels to me like a better UI than the interactive settings managers: it's more comprehensive, more discoverable, more scriptable, and more stable across releases. Most of the Q&A I found on the internet with guidance given on the UI was obsolete, while when given with gsettings command lines it kept being relevant. I also have the feeling that these notes would be easier to understand and follow if given as gsettings invocations instead of descriptions of UI navigation paths.

At some point I'll upgrade to Trixie and reevaluate things, and these notes will be a useful checklist for that.

Fingers crossed that this time I'll manage to stay on Wayland. If not, I know that Xfce is still there for me, and I can trust it to be both helpful and good at not getting in the way of my work.

30 November, 2024 08:13PM

hackergotchi for Aurelien Jarno

Aurelien Jarno

UEFI Unified Kernel Image for Debian Installer on riscv64

On the riscv64 port, the default boot method is UEFI, with U-Boot typically used as the firmware. This approach aligns more closely with other architectures, which avoid developping riscv64 specific code. For advanced users, booting using U-Boot and extlinux is possible, thanks to the kernel being built with CONFIG_EFI_STUB=y.

The same applies to the Debian Installer, which is provided as ISO images in various sizes and formats like on other architectures. These images can be put on a USB drive or an SD-card and booted directly from U-Boot in UEFI mode. Some users prefer to use the netboot "image", which in practice consists of a Linux kernel, an initrd, plus a set of Device Tree Blob (DTB) files.

However, booting this in UEFI mode is not straightforward, unless you use a TFTP server, which is also not trivial. Less known to users, there is also a corresponding mini.iso image, which contains all the above plus a bootloader. This offers a simpler alternative for installation, but depending on your (vendor) U-Boot version this still requires to go through a media.

Systemd version 257-rc2 comes with a great new feature, the ability to include multiple DTB files in a single UKI (Unified Kernel Image) file, with systemd-stub automatically loading the appropriate one for the current hardware. A UKI file combines a UEFI boot stub program, a Linux kernel image, an optional initrd, and further resources in a single UEFI PE file. This finally solves the DTB problem in the UEFI world for distributions, as a single image can work on multiple boards.

Building upon this, debian-installer on riscv64 now also creates a UEFI UKI mini.efi image, which contains systemd-stub, a Linux kernel, an initrd, plus a set of Device Tree Blob (DTB) files. Using this image also ensures that the system is booted in UEFI mode. Booting it with debian-installer is as simple as:

load mmc 0:1 mini.efo $kernel_addr_r # (can also be done using tftpboot, wget, etc.)
bootefi $kernel_addr_r

Additional parameters can be passed to the image using the U-Boot bootargs environment variable. For instance, to boot in rescue mode:

setenv bootargs "rescue/enable=true"

30 November, 2024 09:41AM by aurel32

Russell Coker