January 18, 2019

hackergotchi for Rhonda D'Vine

Rhonda D'Vine

Enigma

Just the other day a working colleague asked me what kind of music I listen to, especially when working. It's true, music helps me to focus better and work more concentrated. But it obviously depends on what kind of music it is. And there is one project I come to every now and then. The name is Enigma. It's not disturbing, good for background, with soothing and non-intrusive vocals. Here are the songs:

  • Return To Innocence: This is quite likely the song you know from them, which also got me hooked up originally.
  • Push The Limits: A powerful song. The album version is even a few minutes longer.
  • Voyageur: Love the rhythm and theme in this song.

Like always, enjoy.

/music | permanent link | Comments: 0 | Flattr this

18 January, 2019 03:57PM by Rhonda

hackergotchi for Keith Packard

Keith Packard

newt-duino

Newt-Duino: Newt on an Arduino

Here's our target system. The venerable Arduino Duemilanove. Designed in 2009, this board comes with the Atmel ATmega328 system on chip, and not a lot else. This 8-bit microcontroller sports 32kB of flash, 2kB of RAM and another 1kB of EEPROM. Squeezing even a tiny version of Python onto this device took some doing.

How Small is Newt Anyways?

From my other ?postings about Newt, the amount of memory and set of dependencies for running Newt has shrunk over time. Right now, a complete Newt system fits in about 30kB of text on both Cortex M and ATmel processors. That includes the parser and bytecode interpreter, plus garbage collector for memory management.

Bare Metal Arduino

The first choice to make is whether to take some existing OS and get that running, or to just wing it and run right on the metal. To save space, I decided (at least for now), to skip the OS and implement whatever hardware support I need myself.

Newt has limited dependencies; it doesn't use malloc, and the only OS interface the base language uses is getchar and putchar to the console. That means that a complete Newt system need not be much larger than the core Newt language and some simple I/O routines.

For the basic Arduino port, I included some simple serial I/O routines for the console to bring the language up. Once running, I've started adding some simple I/O functions to talk to the pins of the device.

Pushing data out of .data

The ATmega 328P, like most (all?) of the 8-bit Atmel processors cannot directly access flash memory as data. Instead, you are required to use special library functions. Whoever came up with this scheme clearly loved the original 8051 design because it worked the same way.

Modern 8051 clones (like the CC1111 we used to use for Altus Metrum stuff) fix this bug by creating a flat 16-bit address space for both flash and ram so that 16-bit pointers can see all of memory. For older systems, SDCC actually creates magic pointers and has run-time code that performs the relevant magic to fetch data from anywhere in the system. Sadly, no-one has bothered to do this with avr-gcc.

This means that any data accesses done in the normal way can only touch RAM. To make this work, avr-gcc places all data, read-write and read-only in RAM. Newt has some pretty big read-only data bits, including parse tables and error messages. With only 2kB of RAM but 32kB of flash, it's pretty important to avoid filling that with read-only data.

avr-gcc has a whole 'progmem' mechanism which allows you to direct data into living only in flash by decorating the declaration with PROGMEM:

const char PROGMEM error_message[] = "This is in flash, not RAM";

This is pretty easy to manage, the only problem is that attempts to access this data from your program will fail unless you use the pgm_read_word and pgm_read_byte functions:

const char *m = error_message;
char c;

while ((c = (char) pgm_read_byte(m++))) putchar(c);

avr-libc includes some functions, often indicated with a '_P' suffix, which take pointers to flash instead of pointers to RAM. So, it's possible to place all read-only data in flash and not in RAM, it's just a pain, especially when using portable code.

So, I hacked up the newt code to add macros for the parse tables, one to add the necessary decoration and three others to fetch elements of the parse table by address.

#define PARSE_TABLE_DECLARATION(t)  PROGMEM t
#define PARSE_TABLE_FETCH_KEY(a)    ((parse_key_t) pgm_read_word(a))
#define PARSE_TABLE_FETCH_TOKEN(a)  ((token_t) pgm_read_byte(a))
#define PARSE_TABLE_FETCH_PRODUCTION(a) ((uint8_t) pgm_read_byte(a))

With suitable hacks in Newt to use these macros, I could finally build newt for the Arduino.

Automatically Getting Strings out of RAM

A lot of the strings in Newt are printf format strings passed directly to a printf-ish functions. I created some wrapper macros to automatically move the format strings out of RAM and call functions expecting the strings to be in flash. Here's the wrapper I wrote for fprintf:

#define fprintf(file, fmt, args...) do {            \
    static const char PROGMEM __fmt__[] = (fmt);        \
    fprintf_P(file, __fmt__, ## args);          \
    } while(0)

This assumes that all calls to fprintf will take constant strings, which is true in Newt, but not true generally. I would love to automatically handle those cases using __builtin_const_p, but gcc isn't as helpful as it could be; you can't declare a string to be initialized from a variable value, even if that code will never be executed:

#define fprintf(file, fmt, args...) do {            \
    if (__builtin_const_p(fmt)) {               \
        static const char PROGMEM __fmt__[] = (fmt);    \
        fprintf_P(file, __fmt__, ## args);      \
    } else {                        \
        fprintf(file, fmt, ## args);            \
    }                           \
    } while(0)

This doesn't compile when 'fmt' isn't a constant string because the initialization of fmt, even though never executed, isn't legal. Suggestions on how to make this work would be most welcome. I only need this for sprintf, so for now, I've created a 'sprintf_const' macro which does the above trick that I use for all sprintf calls with a constant string format.

With this hack added, I saved hundreds of bytes of RAM, making enough space for an (wait for it) 900 byte heap within the interpreter. That's probably enough to do some simple robotics stuff, with luck sufficient for my robotics students.

It Works!

> def fact(x):
>  r = 1
>  for y in range(2,x):
>   r *= y
>  return r
>
> fact(10)
362880
> fact(20) * fact(10)
4.41426e+22
>

This example was actually run on the Arduino pictured above.

Future Work

All I've got running at this point is the basic language and a couple of test primitives to control the LED on D13. Here are some things I'd like to add.

Using EEPROM for Program Storage

To make the system actually useful as a stand-alone robot, we need some place to store the application. Fortunately, the ATmega328P has 1kB of EEPROM memory. I plan on using this for program storage and will parse the contents at start up time.

Simple I/O API

Having taught students using Lego Logo, I've learned that the fewer keystrokes needed to get the first light blinking the better the students will do. Seeking to replicate that experience, I'm thinking that the I/O API should avoid requiring any pin mode settings, and should have functions that directly manipulate any devices on the board. The Lego Logo API also avoids functions with more than one argument, preferring to just save state within the system. So, for now, I plan to replicate that API loosely:

def onfor(n):
  talkto(LED)
  on()
  time.sleep(n)
  off()

In the Arduino environment, a motor controller is managed with two separate bits, requiring two separate numbers and lots of initialization:

int motor_speed = 6;
int motor_dir = 5;

void setup() {
    pinMod(motor_dir, OUTPUT);
}

void motor(int speed, int dir) {
    digitalWrite(motor_dir, dir);
    analogWrite(motor_speed, speed);
}

By creating an API which knows how to talk to the motor controller as a unified device, we get:

def motor(speed, dir):
  talkto(MOTOR_1)
  setdir(dir)
  setpower(speed)

Plans

I'll see about getting the EEPROM bits working, then the I/O API running. At that point, you should be able to do anything with this that you can with C, aside from performance.

Then some kind of host-side interface, probably written in Java to be portable to whatever machine the user has, and I think the system should be ready to experiment with.

Links

The source code is available from my server at https://keithp.com/cgit/newt.git/, and also at github https://github.com/keith-packard/newt. It is licensed under the GPLv2 (or later version).

18 January, 2019 05:19AM

hackergotchi for Daniel Kahn Gillmor

Daniel Kahn Gillmor

New OpenPGP certificate for dkg, 2019

My old OpenPGP certificate will be 12 years old later this year. I'm transitioning to a new OpenPGP certificate.

You might know my old OpenPGP certificate as:

pub   rsa4096 2007-06-02 [SC] [expires: 2019-06-29]
      0EE5BE979282D80B9F7540F1CCD2ED94D21739E9
uid          Daniel Kahn Gillmor <dkg@fifthhorseman.net>
uid          Daniel Kahn Gillmor <dkg@debian.org>

My new OpenPGP certificate is:

pub   ed25519 2019-01-17 [C] [expires: 2021-01-16]
      723E343AC00331F03473E6837BE5A11FA37E8721
uid          Daniel Kahn Gillmor
uid          dkg@debian.org
uid          dkg@fifthhorseman.net

If you've certified my old certificate, I'd appreciate your certifying my new one. Please do confirm by contacting me via whatever channels you think are most appropriate (including in-person if you want to share food or drink with me!) before you re-certify, of course.

I've published the new certificate to the SKS keyserver network, as well as to my personal website -- you can fetch it like this:

wget -O- https://dkg.fifthhorseman.net/dkg-openpgp.key | gpg --import

A copy of this transition statement signed by both the old and new certificates is available on my website, and you can also find further explanation about technical details, choices, and rationale on my blog.

Technical details

I've made a few decisions differently about this certificate:

Ed25519 and Curve25519 for Public Key Material

I've moved from 4096-bit RSA public keys to the Bernstein elliptic curve 25519 for all my public key material (EdDSA for signing, certification, and authentication, and Curve25519 for encryption). While 4096-bit RSA is likely to be marginally stronger cryptographically than curve 25519, 25519 still appears to be significantly stronger than any cryptanalytic attack known to the public.

Additionally, elliptic curve keys and the signatures associated with them are tiny compared to 4096-bit RSA. I certified my new cert with my old one, and well over half of the new certificate is just certifications from the old key because they are so large.

This size advantage makes it easier for me ship the public key material (and signatures from it) in places that would be more awkward otherwise. See the discussion below.

Separated User IDs

The other thing you're likely to notice if you're considering certifying my key is that my User IDs are now split out. rather than two User IDs:

  • Daniel Kahn Gillmor <dkg@fifthhorseman.net>
  • Daniel Kahn Gillmor <dkg@debian.org>

I now have the name separate from the e-mail addresses, making three User IDs:

  • Daniel Kahn Gillmor
  • dkg@fifthhorseman.net
  • dkg@debian.org

There are a couple reasons i've done this.

One reason is to simplify the certification process. Traditional OpenPGP User ID certification is an all-or-nothing process: the certifier is asserting that both the name and e-mail address belong to the identified party. But this can be tough to reason about. Maybe you know my name, but not my e-mail address. Or maybe you know my over e-mail, but aren't really sure what my "real" name is (i'll leave questions about what counts as a real name to a more philosophical blog post). You ought to be able to certify them independently. Now you can, since it's possible to certify one User ID independently of another.

Another reason is because i plan to use this certificate for e-mail, among other purposes. In e-mail systems, the human name is a confusing distraction, as the real underlying correspondent is the e-mail address. E-mail programs should definitely allow their users to couple a memorable name with an e-mail address, but it should be more like a petname. The bundling of a human "real" name with the e-mail address by the User ID itself just provides more points of confusion for the mail client.

If the user communicates with a certain person by e-mail address, the key should be bound to the e-mail protocol address on its own. Then the user themselves can decide what other monikers they want to use for the person; the User ID shouldn't force them to look at a "real" name just because it's been bundled together.

Split out ACLU identity

Note that my old certificate included some additional identities, including job-specific e-mail addresses. I've split out my job-specific cryptographic credentials to a different OpenPGP key entirely. If you want to mail me at dkg@aclu.org, you can use key 888E6BEAC41959269EAA177F138F5AB68615C560 (which is also published on my work bio page).

This is in part because the folks who communicate with me at my ACLU address are more likely to have old or poorly-maintained e-mail systems than other people i communicate with, and they might not be able to handle curve 25519. So the ACLU key is using 3072-bit RSA, which is universally supported by any plausible OpenPGP implementation.

This way i can experiment with being more forward-looking in my free software and engineering community work, and shake out any bugs that i might find there, before cutting over the e-mails that come in from more legal- and policy-focused colleagues.

Isolated Subkey Capabilities

In my new certificate, the primary key is designated certification-only. There are three subkeys, one each for authentication, encryption, and signing. The primary key also has a longer expiration time (2 years as of this writing), while the subkeys have 1 year expiration dates.

Isolating this functionality helps a little bit with security (i can take the certification key entirely offline while still being able to sign non-identity data), and it also offers a pathway toward having a more robust subkey rotation schedule. As i build out my tooling for subkey rotation, i'll probably make a few more blog posts about that.

Autocrypt-friendly

Finally, several of these changes are related to the Autocrypt project, a really great collaboration of a group of mail user agent developers, designers, UX experts, trainers, and users, who are providing guidance to make encrypted e-mail something that normal humans can use without having to think too much about it.

Autocrypt treats the OpenPGP certificate User IDs as merely decorative, but its recommended form of the User ID for an OpenPGP certificate is just the bare e-mail address. With the User IDs split out as described above, i can produce a minimized OpenPGP certificate that's Autocrypt-friendly, including only the User ID that matches my sending e-mail address precisely, and leaving out the rest.

I'm proud to be associated with the Autocrypt project, and have been helping to shepherd some of the Autocrypt functionality into different clients (my work on my own MUA of choice, notmuch is currently stalled, but i hope to pick it back up again soon). Having an OpenPGP certificate that works well with Autocrypt, and that i can stuff into messages even from clients that aren't fully-Autocrypt compliant yet is useful to me for getting things tested.

Documenting workflow vs. tooling

Some people may want to know "how did you make your OpenPGP cert like this?" For those folks, i'm sorry but this is not a step-by-step technical howto. I've read far too many "One True Way To Set Up Your OpenPGP Certificate" blog posts that haven't aged well, and i'm not confident enough to tell people to run the weird arbitrary commands that i ran to get things working this way.

Furthermore, i don't want people to have to run those commands.

If i think there are sensible ways to set up OpenPGP certificates, i want those patterns built into standard tooling for normal people to use, without a lot of command-line hackery.

So if i'm going to publish a "how to", it would be in the form of software that i think can be sensibly maintained and provides a sane user interface for normal humans. I haven't written that tooling yet, but i need to change keys first, so for now you just get this blog post in English. But feel free to tell me what you think i could do better!

18 January, 2019 04:43AM by Daniel Kahn Gillmor (dkg)

hackergotchi for Dirk Eddelbuettel

Dirk Eddelbuettel

RcppArmadillo 0.9.200.7.0

armadillo image

A new RcppArmadillo bugfix release arrived at CRAN today. The version 0.9.200.7.0 is another minor bugfix release, and based on the new Armadillo bugfix release 9.200.7 from earlier this week. I also just uploaded the Debian version, and Uwe’s systems have already create the CRAN Windows binary.

Armadillo is a powerful and expressive C++ template library for linear algebra aiming towards a good balance between speed and ease of use with a syntax deliberately close to a Matlab. RcppArmadillo integrates this library with the R environment and language–and is widely used by (currently) 559 other packages on CRAN.

This release just brings minor upstream bug fixes, see below for details (and we also include the updated entry for the November bugfix release).

Changes in RcppArmadillo version 0.9.200.7.0 (2019-01-17)

  • Upgraded to Armadillo release 9.200.7 (Carpe Noctem)

  • Fixes in 9.200.7 compared to 9.200.5:

    • handling complex compound expressions by trace()

    • handling .rows() and .cols() by the Cube class

Changes in RcppArmadillo version 0.9.200.5.0 (2018-11-09)

  • Upgraded to Armadillo release 9.200.5 (Carpe Noctem)

  • Changes in this release

    • linking issue when using fixed size matrices and vectors

    • faster handling of common cases by princomp()

Courtesy of CRANberries, there is a diffstat report relative to previous release. More detailed information is on the RcppArmadillo page. Questions, comments etc should go to the rcpp-devel mailing list off the R-Forge page.

This post by Dirk Eddelbuettel originated on his Thinking inside the box blog. Please report excessive re-aggregation in third-party for-profit settings.

18 January, 2019 02:00AM

January 17, 2019

hackergotchi for Anastasia Tsikoza

Anastasia Tsikoza

Week 5: Resolving the blocker

Photo by Pascal Debrunner on Unsplash

Hi, I am Anastasia, an Outreachy intern working on the project “Improve integration of Debian derivatives with Debian infrastructure and community”. Here are my other posts that haven’t appeared in the feed on time:

This post is about my work on the subscription feature for Debian derivatives - first of the two main issues to be resolved within my internship. And this week’s topic from the organizers is “Think About Your Audience”, especially newcomers to the community and future Outreachy applicants. So I’ll try to write about the feature keeping the most important details but taking into account that the readers might be unfamiliar with some terms and concepts.

What problem I was trying to solve

As I wrote here, around the middle of December the Debian derivatives census was re-enabled. It means that the census scripts started running on a scheduled basis. Every midnight they scanned through the derivatives’ wiki pages, retrieved all the info needed, processed it and formed files and directories for future use.

At each stage different kinds of errors could occur. Usually that means the maintainers should be contacted so that they update wiki pages of their derivatives or fix something in their apt repositories. Checking error logs, identifying new issues, notifying maintainers about them and keeping in mind who has already been notified and who has not - all this always was a significant amount of work which should have been eventually automated (at least partially).

What exactly I needed to do

By design, the subscription system had nothing in common with the daily spam mailing. Notifications should have only been sent when new issues appeared, which doesn’t happen too often.

It was decided to create several small logs for each derivative and then merge them into one. The resulting log should have also had comments, like which issues are new and which ones have been fixed since the last log was sent.

So my task could be divided into three parts:

  • create a way for folks to subscribe
  • write a script joining small logs into separate log for each particular derivative
  • write a script sending the log to subscribers (if it has to be sent)

The directory with the small logs was supposed to be kept until the next run to be compared with the new small logs. If some new issues appeared, the combined log would be sent to subscribers; otherwise it would simply be stored in the derivative’s directory.

Overcoming difficulties

I ran into first difficulties almost immediately after my changes were deployed. I forgot about the parallel execution of make: in the project’s crontab file we use the make --jobs=3, which means three parallel Make jobs are creating files and directories concurrently. So all can turn to chaos if you weren’t extra careful with the makefiles.

Every time I needed to fix something, Paul had to run a cron job and create most of the files from zero. And for nearly two days we’ve pulled together, coordinating actions through IRC. At the first day’s end the main problem seemed to be solved and I let myself rejoice… and then things went wrong. I was at my wits’ end and felt extremely tired, so I went off to bed. The next day I found the solution seconds after waking up!

Cool things

When finally all went without a hitch, it was a huge relief to me. And I am happy that some folks have already signed up for the errors, and hope they consider the feature useful.

Also I was truly pleased to know that Paul had first learned about makefiles’ order-only prerequisites from my code! That was quite a surprise :)

More details

If you are curious, in the Projects section a more detailed report will appear soon.

17 January, 2019 11:42PM

hackergotchi for Jonathan Dowland

Jonathan Dowland

multi-coloured Fedoras

My blog post about my Red Hat shell prompt proved very popular, although some people tried my instructions and couldn't get it to work, so here's some further information.

The unicode code-point I am using is "🎩", U+1F3A9 TOP HAT. My Terminal font of choice is Noto Mono Regular which does not contain a glyph for that code-point. Therefore the font system my Terminal uses (pango) is pulling it from another font. In my case, it's using OverPass Mono, an open source font created for Red Hat branding. The TOP HAT glyph in OverPass Mono is a Fedora. This is a nice little easter egg included by the font authors. (There's at least one other easter egg in the font, by the way!)

It's pure chance that my font system chose OverPass Mono to pull that glyph.

The simplest way to get this to work is (probably) to download and install "OverPass Mono" and set your Terminal font to it for all characters. Alternatively you may need to look at configuring Pango (or whatever font system you are using).

If your Terminal and font system support multicolour emojis, then the above alone may not work for you. Some of my colleagues got a nasty surprise when upgrading their systems: their red hats turned blue. It's possible to add a trailing Unicode modifying code-point that instructs the font system to not render the prior glyph using multicolour emojis. This reportedly fixes it:

hat="🎩"
combiner="$(perl -CS -e 'print "\N{VARIATION SELECTOR-15}"')"
# ^ alternatively, "$(printf '\xef\xb8\x8e')"
export PS1="\[\e[31m\]${hat}${combiner}\[\e[0m\]"

(Thanks ilmari for the perl trick)

17 January, 2019 04:57PM

hackergotchi for Julien Danjou

Julien Danjou

Serious Python released!

Serious Python released!

Today I'm glad to announce that my new book, Serious Python, has been released.

However, you wonder… what is Serious Python?

Well, Serious Python is the the new name of The Hacker's Guide to Python — the first book I published. Serious Python is the 4th update of that book — but with a brand a new name and a new editor!

Serious Python released!

For more than a year, I've been working with the editor No Starch Press to enhance this book and bring it to the next level! I'm very proud of what we achieved, and working with a whole team on this book has been a fantastic experience.

The content has been updated to be ready for 2019: pytest is now a de-facto standard for testing, so I had to write about it. On the other hand, Python 2 support was less a focus, and I removed many mentions of Python 2 altogether. Some chapters have been reorganized, regrouped and others got enhanced with new content!

The good news: you can get this new edition of the book with a 15% discount for the next 24 hours using the coupon code SERIOUSPYTHONLAUNCH on the book page.

The book is also released as part as No Starch collection. They also are in charge of distributing the paperback copy of the book. If you want a version of the book that you can touch and hold in your arms, look for it in No Starch shop, on Amazon or in your favorite book shop!

Serious Python released!No Starch version of Serious Python cover

17 January, 2019 04:56PM by Julien Danjou

Kunal Mehta

Eliminating PHP polyfills

The Symfony project has recently created a set of pure-PHP polyfills for both PHP extensions and newer language features. It allows developers to add requirements upon those functions or language additions without increasing the system requirements upon end users. For the most part, I think this is a good thing, and valuable to have. We've done similar things inside MediaWiki as well for CDB support, Memcached, and internationalization, just to name a few.

But the downside is that on platforms where it is possible to install the missing PHP extensions or upgrade PHP itself, we're shipping empty code. MediaWiki requires both the ctypes and mbstring PHP extensions, and our servers have those, so there's no use in deploying polyfills for those, because they'll never be used. In September, Reedy and I replaced the polyfills with "unpolyfills" that simply provide the correct package, so the polyfill is skipped by composer. That removed about 3,700 lines of code from what we're committing, reviewing, and deploying - a big win.

Last month I came across the same problem in Debian: #911832. The php-symfony-polyfill package was failing tests on the new PHP 7.3, and up for removal from the next stable release (Buster). On its own, the package isn't too important, but was a dependency of other important packages. In Debian, the polyfills are even more useless, since instead of depending upon e.g. php-symfony-polyfill-mbstring, the package could simply depend upon the native PHP extension, php-mbstring. In fact, there was already a system designed to implement those kinds of overrides. After looking at the dependencies, I uploaded a fixed version of php-webmozart-assert, filed bugs for two other packages. and provided patches for symfony. I also made a patch to the default overrides in pkg-php-tools, so that any future package that depends upon a symfony polyfill should now automatically depend upon the native PHP extension if necessary.

Ideally composer would support alternative requirements like ext-mbstring | php-symfony-polyfill-mbstring, but that's been declined by their developers. There's another issue that is somewhat related, but doesn't do much to reduce the installation of polyfills when unnecessary.

17 January, 2019 07:50AM by legoktm

January 16, 2019

Reproducible builds folks

Reproducible Builds: Weekly report #194

Here’s what happened in the Reproducible Builds effort between Sunday January 6 and Saturday January 12 2019:

Packages reviewed and fixed, and bugs filed

Website development

There were a number of updates to the reproducible-builds.org project website this week, including:

Test framework development

There were a number of updates to our Jenkins-based testing framework that powers tests.reproducible-builds.org this week, including:

  • Holger Levsen:
    • Arch Linux-specific changes:
      • Use Debian’s sed, untar and others with sudo as they are not available in the bootstrap.tar.gz file ([], [], [], [], etc.).
      • Fix incorrect sudoers(5) regex. []
      • Only move old schroot away if it exists. []
      • Add and drop debug code, cleanup cruft, exit on cleanup(), etc. ([], [])
      • cleanup() is only called on errors, thus exit 1. []
    • Debian-specific changes:
      • Revert “Support arbitrary package filters when generating deb822 output” ([]) and re-open the corresponding merge request
      • Show the total number of packages in a package set. []
    • Misc/generic changes:
      • Node maintenance. ([, [], [], etc.)
  • Mattia Rizzolo:
    • Fix the NODE_NAME value in case it’s not a full-qualified domain name. []
    • Node maintenance. ([], etc.)
  • Vagrant Cascadian:

This week’s edition was written by Bernhard M. Wiedemann, Chris Lamb, Holger Levsen & reviewed by a bunch of Reproducible Builds folks on IRC & the mailing lists.

16 January, 2019 04:39PM

hackergotchi for Iain R. Learmonth

Iain R. Learmonth

A Solution for Authoritative DNS

I’ve been thinking about improving my DNS setup. So many things will use e-mail verification as a backup authentication measure that it is starting to show as a real weak point. An Ars Technica article earlier this year talked about how “[f]ederal authorities and private researchers are alerting companies to a wave of domain hijacking attacks that use relatively novel techniques to compromise targets at an almost unprecedented scale.”

The two attacks that are mentioned in that article, changing the nameserver and changing records, are something that DNSSEC could protect against. Records wouldn’t have to be changed on my chosen nameservers, a BGP-hijacking could just give another server the queries for records on my domain instead and then reply with whatever it chooses.

After thinking for a while, my requirements come down to:

  • Offline DNSSEC signing
  • Support for storing signing keys on a HSM (YubiKey)
  • Version control
  • No requirement to run any Internet-facing infrastructure myself

After some searching I discovered GooDNS, a “good” DNS hosting provider. They have an interesting setup that looks to fit all of my requirements. If you’re coming from a more traditional arrangement with either a self-hosted name server or a web panel then this might seem weird, but if you’ve done a little “infrastructure as code” then maybe it is not so weird.

The inital setup must be completed via the web interface. You’ll need to have an hardware security module (HSM) for providing a time based one time password (TOTP), an SSH key and optionally a GPG key as part of the registration. You will need the TOTP to make any changes via the web interface, the SSH key will be used to interact with the git service, and the GPG key will be used for any email correspondance including recovery in the case that you lose your TOTP HSM or password.

You must validate your domain before it will be served from the GooDNS servers. There are two options for this, one for new domains and one “zero-downtime” option that is more complex but may be desirable if your domain is already live. For new domains you can simply update your nameservers at the registrar to validate your domain, for existing domains you can add a TXT record to the current DNS setup that will be validated by GooDNS to allow for the domain to be configured fully before switching the nameservers. Once the domain is validated, you will not need to use the web interface again unless updating contact, security or billing details.

All the DNS configuration is managed in a single git repository. There are three branches in the repository: “master”, “staging” and “production”. These are just the default branches, you can create other branches if you like. The only two that GooDNS will use are the “staging” and “production” branches.

GooDNS provides a script that you can install at /usr/local/bin/git-dns (or elsewhere in your path) which provides some simple helper commands for working with the git repository. The script is extremely readable and so it’s easy enough to understand and write your own scripts if you find yourself needing something a little different.

When you clone your git repository you’ll find one text file on the master branch for each of your configured zones:

irl@computer$ git clone git@goodns.net:irl.git
Cloning into 'irl1'...
remote: Enumerating objects: 3, done.
remote: Total 3 (delta 0), reused 0 (delta 0), pack-reused 3
Receiving objects: 100% (3/3), 22.55 KiB | 11.28 MiB/s, done.
Resolving deltas: 100% (1/1), done.
irl@computer$ ls
irl1.net   learmonth.me
irl@computer$ cat irl1.net
@ IN SOA ns1.irl1.net. hostmaster.irl1.net. (
            _SERIAL_
            28800
            7200
            864000
            86400
            )

@           IN      NS        ns1.goodns.net.
@           IN      NS        ns2.goodns.net.
@           IN      NS        ns3.goodns.net.

In the backend GooDNS is using OpenBSD 6.4 servers with nsd(8). This means that the zone files use the same syntax. If you don’t know what this means then that is fine as the documentation has loads of examples in it that should help you to configure all the record types you might need. If a record type is not yet supported by nsd(8), you can always specify the record manually and it will work just fine.

One thing you might note here is that the string _SERIAL_ appears instead of a serial number. The git-dns script will replace this with a serial number when you are ready to publish the zone file.

I’ll assume that you already have you GPG key and SSH key set up, now let’s set up the DNSSEC signing key. For this, we will use one of the four slots of the YubiKey. You could use either 9a or 9e, but here I’ll use 9e as 9a is already the SSH key for me.

To set up the token, we will need the yubico-piv-tool. Be extremely careful when following these steps especially if you are using a production device. Try to understand the commands before pasting them into the terminal.

First, make sure the slot is empty. You should get an output similar to the following one:

irl@computer$ yubico-piv-tool -s 9e -a status 
CHUID:  ...
CCC:    No data available
PIN tries left: 10

Now we will use git-dns to create our key signing key (KSK):

irl@computer$ git dns kskinit --yubikey-neo
Successfully generated a new private key.
Successfully generated a new self signed certificate.
Found YubiKey NEO.
Slots available:
 (1) 9a - Not empty
 (2) 9e - Empty
Which slot to use for DNSSEC signing key? 2
Successfully imported a new certificate.
CHUID:  ...
CCC:    No data available
Slot 9e:    
    Algorithm:  ECCP256
    Subject DN: CN=irl1.net
    Issuer DN:  CN=irl1.net
    Fingerprint:    97dda8a441a401102328ab6ed4483f08bc3b4e4c91abee8a6e144a6bb07a674c
    Not Before: Feb 01 13:10:10 2019 GMT
    Not After:  Feb 01 13:10:10 2021 GMT
PIN tries left: 10

We can see the public key for this new KSK:

irl@computer$ git dns pubkeys
irl1.net. DNSKEY 256 3 13 UgGYfiNse1qT4GIojG0VGcHByLWqByiafQ8Yt7/Eit2hCPYYcyiE+TX8HP8al/SzCnaA8nOpAkqFgPCI26ydqw==

Next we will create a zone signing key (ZSK). These are stored in the keys/ folder of your git repository but are not version controlled. You can optionally encrypt these with GnuPG (and so requiring the YubiKey to sign zones) but I’ve not done that here. Operations using slot 9e do not require the PIN so leaving the YubiKey connected to the computer is pretty much the same as leaving the KSK on the disk. Maybe a future YubiKey will not have this restriction or will add more slots.

irl@computer$ git dns zskinit
Created ./keys/
Successfully generated a new private key.
irl@computer$ git dns pubkeys
irl1.net. DNSKEY 256 3 13 UgGYfiNse1qT4GIojG0VGcHByLWqByiafQ8Yt7/Eit2hCPYYcyiE+TX8HP8al/SzCnaA8nOpAkqFgPCI26ydqw=
irl1.net. DNSKEY 257 3 13 kS7DoH7fxDsuH8o1vkvNkRcMRfTbhLqAZdaT2SRdxjRwZSCThxxpZ3S750anoPHV048FFpDrS8Jof08D2Gqj9w==

Now we can go to our domain registrar and add DS records to the registry for our domain using the public keys. First though, we should actually sign the zone. To create a signed zone:

irl@computer$ git dns signall
Signing irl1.net...
Signing learmonth.me...
[production 51da0f0] Signed all zone files at 2019-02-01 13:28:02
 2 files changed, 6 insertions(+), 0 deletions(-)

You’ll notice that all the zones were signed although we only created one set of keys. Set ups where you have one shared KSK and individual ZSK per zone are possible but they provide questionable additional security. Reducing the number of keys required for DNSSEC helps to keep them all under control.

To make these changes live, all that is needed is to push the production branch. To keep things tidy, and to keep a backup of your sources, you can push the master branch too. git-dns provides a helper function for this:

irl@computer$ git dns push
Pushing master...done
Pushing production...done
Pushing staging...done

If I now edit a zone file on the master branch and want to try out the zone before making it live, all I need to do is:

irl@computer$ git dns signall --staging
Signing irl1.net...
Signing learmonth.me...
[staging 72ea1fc] Signed all zone files at 2019-02-01 13:30:12
 2 files changed, 8 insertions(+), 0 deletions(-)
irl@computer$ git dns push
Pushing master...done
Pushing production...done
Pushing staging...done

If I now use the staging resolver or lookup records at irl1.net.staging.goodns.net then I’ll see the zone live. The staging resolver is a really cool idea for development and testing. They give you a couple of unique IPv6 addresses just for you that will serve your staging zone files and act as a resolver for everything else. You just have to plug these into your staging environment and everything is ready to go. In the future they are planning to allow you to have more than one staging environment too.

All that is left to do is ensure that your zone signatures stay fresh. This is easy to achieve with a cron job:

0 3 * * * /usr/local/bin/git-dns cron --repository=/srv/dns/irl1.net --quiet

I monitor the records independently and disable the mail output from this command but you might want to drop the --quiet if you’d like to get mails from cron on errors/warnings.

On the GooDNS blog they talk about adding an Onion service for the git server in the future so that they do not have logs that could show the location of your DNSSEC signing keys, which allows you to have even greater protection. They already support performing the git push via Tor but the addition of the Onion service would make it faster and more reliable.


Unfortunately, GooDNS is entirely fictional and you can’t actually manage your DNS in this way, but wouldn’t it be nice? This post has drawn inspiration from the following:

16 January, 2019 04:30PM

hackergotchi for Daniel Silverstone

Daniel Silverstone

Plans for 2019

At the end of last year I made eight statements about what I wanted to do throughout 2019. I tried to split them semi-evenly between being a better adult human and being a better software community contributor. I have had a few weeks now to settle my thoughts around what they mean and I'd like to take some time to go through the eight and discuss them a little more.

I've been told that doing this reduces the chance of me sticking to the points because simply announcing the points and receiving any kind of positive feedback may stunt my desire to actually achieve the goals. I'm not sure about that though, and I really want my wider friends community to help keep me honest about them all. I've set a reminder for April 7th to review the situation and hopefully be able to report back positively on my progress.


My list of goals was stated in a pair of tweets:

  1. Continue to lose weight and get fit. I'd like to reach 80kg during the year if I can
  2. Begin a couch to 5k and give it my very best
  3. Focus my software work on finishing projects I have already started
  4. Where I join in other projects be a net benefit
  5. Give back to the @rustlang community because I've gained so much from them already
  6. Be better at tidying up
  7. Save up lots of money for renovations
  8. Go on a proper holiday

Weight and fitness

Some of you may be aware already, others may not, that I have been making an effort to shed some of my excess weight over the past six or seven months. I "started" in May of 2018 weighing approximately 141kg and I am, as of this morning, weighing approximately 101kg. Essentially that's a semi-steady rate of 5kg per month, though it has, obviously, been slowing down of late.

In theory, given my height of roughly 178cm I should aim for a weight of around 70kg. I am trying to improve my fitness and to build some muscle and as such I'm aiming long-term for roughly 75kg. My goal for this year is to continue my improvement and to reach and maintain 80kg or better. I think this will make a significant difference to my health and my general wellbeing. I'm already sleeping better on average, and I feel like I have more energy over all. I bought a Garmin Vivoactive 3 and have been using that to track my general health and activity. My resting heart rate has gone down a few BPM over the past six months, and I can see my general improvement in sleep etc over that time too. I bought a Garmin Index Scale to track my weight and body composition, and that is also showing me good values as well as encouraging me to weigh myself every day and to learn how to interpret the results.

I've been managing my weight loss partly by means of a 16:8 intermittent fasting protocol, combined with a steady calorie deficit of around 1000kcal/day. While this sounds pretty drastic, I was horrendously overweight and this was critical to getting my weight to shift quickly. I expect I'll reduce that deficit over the course of the year, hence I'm only aiming for a 20kg drop over a year rather than trying to maintain what could in theory be a drop of 30kg or more.

In addition to the IF/deficit, I have been more active. I bought an e-bike and slowly got going on that over the summer, along with learning to enjoy walks around my local parks and scrubland. Since the weather got bad enough that I didn't want to be out of doors I joined a gym where I have been going regularly since September. Since the end of October I have been doing a very basic strength training routine and my shoulders do seem to be improving for it. I can still barely do a pushup but it's less embarassingly awful than it was.

Given my efforts toward my fitness, my intention this year is to extend that to include a Couch to 5k type effort. Amusingly, Garmin offer a self adjusting "coach" called Garmin Coach which I will likely use to guide me through the process. While I'm not committing to any, maybe I'll get involved in some parkruns this year too. I'm not committing to reach an ability to run 5k because, quite simply, my bad leg may not let me, but I am committing to give it my best. My promise to myself was to start some level of jogging once I hit 100kg, so that's looking likely by the end of this month. Maybe February is when I'll start the c25k stuff in earnest.

Adulting

I have put three items down in this category to get better at this year. One is a big thing for our house. I am, quite simply put, awful at tidying up. I leave all sorts of things lying around and I am messy and lazy. I need to fix this. My short-term goal in this respect is to pick one room of the house where the mess is mostly mine, and learn to keep it tidy before my checkpoint in April. I think I'm likely to choose the Study because it's where others of my activities for this year will centre and it's definitely almost entirely my mess in there. I'm not yet certain how I'll learn to do this, but it has been a long time coming and I really do need to. It's not fair to my husband for me to be this awful all the time.

The second of these points is to explicitly save money for renovations. Last year we had a new bathroom installed and I've been seriously happy about that. We will need to pay that off this year (we have the money, we're just waiting as long as we can to earn the best interest on it first) and then I'll want to be saving up for another spot of renovations. I'd like to have the kitchen and dining room done - new floor, new units and sink in the kitchen, fix up the messy wall in the dining room, have them decorated, etc. I imagine this will take quite a bit of 2019 to save for, but hopefully this time next year I'll be saying that we managed that and it's time for the next part of the house.

Finally I want to take a proper holiday this year. It has been a couple of years since Rob and I went to Seoul for a month, and while that was excellent, it was partly "work from home" and so I'd like to take a holiday which isn't also a conference, or working from home, or anything other than relaxation and seeing of interesting things. This will also require saving for, so I imagine we won't get to do it until mid to late 2019, but I feel like this is part of a general effort I've been making to take care of myself more. The fitness stuff above being physical, but a proper holiday being part of taking better care of my mental health.

Software, Hardware, and all the squishy humans in between

2018 was not a great year for me in terms of getting projects done. I have failed to do almost anything with Gitano and I did not doing well with Debian or other projects I am part of. As such, I'm committing to do better by my projects in 2019.

First, and foremost, I'm pledging to focus my efforts on finishing projects which I've already started. I am very good at thinking "Oh, that sounds fun" and starting something new, leaving old projects by the wayside and not getting them to any state of completion. While software is never entirely "done", I do feel like I should get in-progress projects to a point that others can use them and maybe contribute too.

As such, I'll be making an effort to sort out issues which others have raised in Gitano (though I doubt I'll do much more feature development for it) so that it can be used by NetSurf and so that it doesn't drop out of Debian. Since the next release of Debian is due soon, I will have to pull my finger out and get this done pretty soon.

I have been working, on and off, with Rob on a new point-of-sale for our local pub Ye Olde Vic and I am committing to get it done to a point that we can experiment with using it in the pub by the summer. Also I was working on a way to measure fluid flow through a pipe so that we can correlate the pulled beer with the sales and determine wastage etc. I expect I'll get back to the "beer'o'meter" once the point-of-sale work is in place and usable. I am not going to commit to getting it done this year, but I'd like to make a dent in the remaining work for it.

I have an on-again off-again relationship with some code I wrote quite a while ago when learning Rust. I am speaking of my Yarn implementation called (imaginatively) rsyarn. I'd like to have that project reworked into something which can be used with Cargo and associated tooling nicely so that running cargo test in a Rust project can result in running yarns as well.

There may be other projects which jump into this category over the year, but those listed above are the ones I'm committing to make a difference to my previous lackadaisical approach.

On a more community-minded note, one of my goals is to ensure that I'm always a net benefit to any project I join or work on in 2019. I am very aware that in a lot of cases, I provide short drive-by contributions to projects which can end up costing that project more than I gave them in benefit. I want to stop that behaviour and instead invest more effort into fewer projects so that I always end up a net benefit to the project in question. This may mean spending longer to ensure that an issue I file has enough in it that I may not need to interact with it again until verification of a correct fix is required. It may mean spending time fixing someone elses' issues so that there is the engineering bandwidth for someone else to fix mine. I can't say for sure how this will manifest, beyond being up-front and requesting of any community I decide to take part in, that they tell me if I end up costing more than I'm bringing in benefit.

Rust and the Rust community

I've mentioned Rust above, and this is perhaps the most overlappy of my promises for 2019. I want to give back to the Rust community because over the past few years as I've learned Rust and learned more and more about the community, I've seen how much of a positive effect they've had on my life. Not just because they made learning a new programming langauge so enjoyable, but because of the community's focus on programmers as human beings. The fantastic documentation ethics, and the wonderfully inclusive atmosphere in the community meant that I managed to get going with Rust so much more effectively than with almost any other language I've ever tried to learn since Lua.

I have, since Christmas, been slowly involving myself in the Rust community more and more. I joined one of the various Discord servers and have been learning about how crates.io is managed and I have been contributing to rustup.rs which is the initial software interface most Rust users encounter and forms such an integral part of the experience of the ecosystem that I feel it's somewhere I can make a useful impact.

While I can't say a significant amount more right now, I hope I'll be able to blog more in the future on what I'm up to in the Rust community and how I hope that will benefit others already in, and interested in joining, the fun that is programming in Rust.


In summary, I hope at least some of you will help to keep me honest about my intentions for 2019, and if, in return, I can help you too, please feel free to let me know.

16 January, 2019 11:29AM by Daniel Silverstone

hackergotchi for Daniel Lange

Daniel Lange

Openssh taking minutes to become available, booting takes half an hour ... because your server waits for a few bytes of randomness

So, your machine now needs minutes to boot before you can ssh in where it used to be seconds before the Debian Buster update?

Problem

Linux 3.17 (2014-10-05) learnt a new syscall getrandom() that, well, gets bytes from the entropy pool. Glibc learnt about this with 2.25 (2017-02-05) and two tries and four years after the kernel, OpenSSL used that functionality from release 1.1.1 (2018-09-11). OpenSSH implemented this natively for the 7.8 release (2018-08-24) as well.

Now the getrandom() syscall will block1 if the kernel can't provide enough entropy. And that's frequenty the case during boot. Esp. with VMs that have no input devices or IO jitter to source the pseudo random number generator from.

First seen in the wild January 2017

I vividly remember not seeing my Alpine Linux VMs back on the net after the Alpine 3.5 upgrade. That was basically the same issue.

Systemd. Yeah.

Systemd makes this behaviour worse, see issue #4271, #4513 and #10621.
Basically as of now the entropy file saved as /var/lib/systemd/random-seed will not - drumroll - add entropy to the random pool when played back during boot. Actually it will. It will just not be accounted for. So Linux doesn't know. And continues blocking getrandom(). This is obviously different from SysVinit times2 when /var/lib/urandom/random-seed (that you still have lying around on updated systems) made sure the system carried enough entropy over reboot to continue working right after enough of the system was booted.

#4167 is a re-opened discussion about systemd eating randomness early at boot (hashmaps in PID 0...). Some Debian folks participate in the recent discussion and it is worth reading if you want to learn about the mess that booting a Linux system has become.

While we're talking systemd ... #10676 also means systems will use RDRAND in the future despite Ted Ts'o's warning on RDRAND [Archive.org mirror and mirrored locally as 130905_Ted_Tso_on_RDRAND.pdf, 205kB as Google+ will be discontinued in April 2019].

Debian

Debian is seeing the same issue working up towards the Buster release, e.g. Bug #912087.

The typical issue is:

[    4.428797] EXT4-fs (vda1): mounted filesystem with ordered data mode. Opts: data=ordered
[ 130.970863] random: crng init done

with delays up to tens of minutes on systems with very little external random sources.

This is what it should look like:

[    1.616819] random: fast init done
[    2.299314] random: crng init done

Check dmesg | grep -E "(rng|random)" to see how your systems are doing.

If this is not fully solved before the Buster release, I hope some of the below can end up in the release notes3.

Solutions

You need to get entropy into the random pool earlier at boot. There are many ways to achieve this and - currently - all require action by the system administrator.

Kernel boot parameter

From kernel 4.19 (Debian Buster currently runs 4.18 [Update: but will be getting 4.19 before release according to Ben via Mika]) you can set RANDOM_TRUST_CPU at compile time or random.trust_cpu=on on the kernel command line. This will make Intel / AMD system trust RDRAND and fill the entropy pool with it. See the warning from Ted Ts'o linked above.

Using a TPM

The Trusted Platform Module has an embedded random number generator that can be used. Of course you need to have one on your board for this to be useful. It's a hardware device.

Load the tpm-rng module (ideally from initrd) or compile it into the kernel (config HW_RANDOM_TPM). Now, the kernel does not "trust" the TPM RNG by default, so you need to add

rng_core.default_quality=1000

to the kernel command line. 1000 means "trust", 0 means "don't use". So you can chose any value in between that works for you depending on how much you consider your TPM to be unbugged.

VirtIO

For Virtual Machines (VMs) you can forward entropy from the host (that should be running longer than the VMs and have enough entropy) via virtio_rng.

So on the host, you do:

kvm ... -object rng-random,filename=/dev/urandom,id=rng0 -device virtio-rng-pci,rng=rng0,bus=pci.0,addr=0x7

and within the VM newer kernels should automatically load virtio_rng and use that.

You can confirm with dmesg as per above.

Or check:

# cat /sys/devices/virtual/misc/hw_random/rng_available
virtio_rng.0
# cat /sys/devices/virtual/misc/hw_random/rng_current
virtio_rng.0

Patching systemd

The Fedora bugtracker has a bash / python script that replaces the systemd rnd seeding with a (better) working one. The script can also serve as a good starting point if you need to script your own solution, e.g. for reading from an entropy provider available within your (secure) network.

Chaoskey

The wonderful Keith Packard and Bdale Garbee have developed a USB dongle, ChaosKey, that supplies entropy to the kernel. Hard- and software are open source.

Jitterentropy_RNG

Kernel 4.2 introduced jitterentropy_rng which will use the jitter in CPU timings to generate randomness.

modprobe jitterentropy_rng

This apparently needs a userspace daemon though (read: design mistake) so

apt install jitterentropy-rngd (available from Buster/testing).

The current version 1.0.8-3 installs nicely on Stretch. dpkg -i is your friend.

But - drumroll - that daemon doesn't seem to use the kernel module at all.

That's where I stopped looking at that solution. At least for now. There are extensive docs if you want to dig into this yourself.

Haveged

apt install haveged

Haveged is a user-space daemon that gathers entropy though the timing jitter any CPU has. It will only run "late" in boot but may still get your openssh back online within seconds and not minutes.

It is also - to the best of my knowledge - not verified at all regarding the quality of randomness it generates. The haveged design and history page provides and interesting read and I wouldn't recommend haveged if you have alternatives. If you have none, haveged is a wonderful solution though as it works reliably. And unverified entropy is better than no entropy. Just forget this is 2018 2019 :-).

Updates

14.01.2019

Stefan Fritsch, the Apache2 maintainer in Debian, OpenBSD developer and a former Debian security team member stumbled over the systemd issue preventing Apache libssl to initialize at boot in a Debian bug #916690 - apache2: getrandom call blocks on first startup, systemd kills with timeout.

The bug has been retitled "document getrandom changes causing entropy starvation" hinting at not fixing the underlying issue but documenting it in the Debian Buster release notes.

Unhappy with this "minimal compromise" Stefan wrote a comprehensive summary of the current situation to the Debian-devel mailing list. The discussion spans over December 2018 and January 2019 and mostly iterated what had been written above already. The discussion has - so far - not reached any consensus. There is still the "systemd stance" (not our problem, fix the daemons and the "ssh/apache stance" (fix systemd, credit entropy).

The "document in release notes" minimal compromise was brought up again and Stefan warned of the problems this would create for Buster users:

> I'd prefer having this documented in the release notes:
> https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=916690
> with possible solutions like installing haveged, configuring virtio-rng,
> etc. depending on the situation.

That would be an extremely user-unfriendly "solution" and would lead to 
countless hours of debugging and useless bug reports.

This is exactly why I wrote this blog entry and keep it updated. We need to either fix this or tell everybody we can reach before upgrading to Buster. Otherwise this will lead to huge amounts of systems dead on the network after what looked like a successful upgrade.

Some interesting tidbits were mentioned within the thread:

Raphael Hertzog fixed the issue for Kali Linux by installing haveged by default. Michael Prokop did the same for the grml distribution within its December 2018 release.

Ben Hutchings pointed to an interesting thread on the debian-release mailing list he kicked off in May 2018. Multiple people summarized the options and the fact that there is no "general solution that is both correct and easy" at the time.

Sam Hartman identified Debian Buster VMs running under VMware as an issue, because that supervisor does not provide virtio-rng. So Debian VMs wouldn't boot into ssh availability within a reasonable time. This is an issue for real world use cases albeit running a proprietary product as the supervisor.

16.01.2019

Daniel Kahn Gillmor wrote in to explain a risk for VMs starting right after the boot of the host OS:

If that pool is used by the guest to generate long-term secrets because it appears to be well-initialized, that could be a serious problem.
(e.g. "Mining your P's and Q's" by Heninger et al -- https://factorable.net/weakkeys12.extended.pdf)
I've just opened https://bugs.launchpad.net/qemu/+bug/1811758 to report a way to improve that situation in qemu by default.

So ... make sure that your host OS has access to a hardware random number generator or at least carries over its random seed properly across reboots. You could also delay VM starts until the crng on the host Linux is fully initialized (random: crng init done).
Otherwise your VMs may get insufficiently generated pseudo-random numbers and won't even know.


  1. it will return with EAGAIN in the GRND_NONBLOCK use case. The blocking behaviour when lacking entropy is a security measure as per Bug #1559 of Google's Project Zero

  2. Update 18.12.2018: "SysVinit times" ::= "The times when most Linux distros used SysVinit over other init systems." So Wheezy and previous for Debian. Some people objected to the statement, so I added this footnote as a clarification. See the discussion in the comments below. 

  3. there is no Buster branch in the release notes repository yet (2018-12-17) 

16 January, 2019 08:20AM by Daniel Lange

Russ Allbery

Review: Aerial Magic Season 1

Review: Aerial Magic Season 1, by walkingnorth

Series: Aerial Magic #1
Publisher: LINE WEBTOON
Copyright: 2018
Format: Online graphic novel
Pages: 156
Aerial Magic is a graphic novel published on the LINE WEBTOON platform by the same author as the wonderful Always Human, originally in weekly episodes. It is readable for free, starting with the prologue. I was going to wait until all seasons were complete and then review the entire work, like I did with Always Human, but apparently there are going to be five seasons and I don't want to wait that long. This is a review of the first season, which is now complete in 25 episodes plus a prologue.

As with Always Human, the pages metadata in the sidebar is a bit of a lie: a very rough guess on how many pages this would be if it were published as a traditional graphic novel (six times the number of episodes, since each episode seems a bit longer than in Always Human). A lot of the artwork is large panels, so it may be an underestimate. Consider it only a rough guide to how long it might take to read.

Wisteria Kemp is an apprentice witch. This is an unusual thing to be — not the witch part, which is very common in a society that appears to use magic in much the way that we use technology, but the apprentice part. Most people training for a career in magic go to university, but school doesn't agree with Wisteria. There are several reasons for that, but one is that she's textblind and relies on a familiar (a crow-like bird named Puppy) to read for her. Her dream is to be accredited to do aerial magic, but her high-school work was... not good, and she's very afraid she'll be sent home after her ten-day trial period.

Magister Cecily Moon owns a magical item repair shop in the large city of Vectum and agreed to take Wisteria on as an apprentice, something that most magisters no longer do. She's an outgoing woman with a rather suspicious seven-year-old, two other employees, and a warm heart. She doesn't seem to have the same pessimism Wisteria has about her future; she instead is more concerned with whether Wisteria will want to stay after her trial period. This doesn't reassure Wisteria, nor do her initial test exercises, all of which go poorly.

I found the beginning of this story a bit more painful than Always Human. Wisteria has such a deep crisis of self-confidence, and I found Cecily's lack of awareness of it quite frustrating. This is not unrealistic — Cecily is clearly as new to having an apprentice as Wisteria is to being one, and is struggling to calibrate her style — but it's somewhat hard reading since at least some of Wisteria's unhappiness is avoidable. I wish Cecily had shown a bit more awareness of how much harder she made things for Wisteria by not explaining more of what she was seeing. But it does set up a highly effective pivot in tone, and the last few episodes were truly lovely. Now I'm nearly as excited for more Aerial Magic as I would be for more Always Human.

walkingnorth's art style is much the same as that in Always Human, but with more large background panels showing the city of Vectum and the sky above it. Her faces are still exceptional: expressive, unique, and so very good at showing character emotion. She occasionally uses an exaggerated chibi style for some emotions, but I feel like she's leaning more on subtlety of expression in this series and doing a wonderful job with it. Wisteria's happy expressions are a delight to look at. The backgrounds are not generally that detailed, but I think they're better than Always Human. They feature a lot of beautiful sky, clouds, and sunrise and sunset moments, which are perfect for walkingnorth's pastel palette.

The magical system underlying this story doesn't appear in much detail, at least yet, but what is shown has an interesting animist feel and seems focused on the emotions and memories of objects. Spells appear to be standardized symbolism that is known to be effective, which makes magic something like cooking: most people use recipes that are known to work, but a recipe is not strictly required. I like the feel of it and the way that magic is woven into everyday life (personal broom transport is common), and am looking forward to learning more in future seasons.

As with Always Human, this is a world full of fundamentally good people. The conflict comes primarily from typical interpersonal conflicts and inner struggles rather than any true villain. Also as with Always Human, the world features a wide variety of unremarked family arrangements, although since it's not a romance the relationships aren't quite as central. It makes for relaxing and welcoming reading.

Also as in Always Human, each episode features its own soundtrack, composed by the author. I am again not reviewing those because I'm a poor music reviewer and because I tend to read online comics in places and at times where I don't want the audio, but if you like that sort of thing, the tracks I listened to were enjoyable, fit the emotions of the scene, and were unobtrusive to listen to while reading.

This is an online comic on a for-profit publishing platform, so you'll have to deal with some amount of JavaScript and modern web gunk. I at least (using up-to-date Chrome on Linux with UMatrix) had fewer technical problems with delayed and partly-loaded panels than I had with Always Human.

I didn't like this first season quite as well as Always Human, but that's a high bar, and it took some time for Always Human to build up to its emotional impact as well. What there is so far is a charming, gentle, and empathetic story, full of likable characters (even the ones who don't seem that likable at first) and a fascinating world background. This is an excellent start, and I will certainly be reading (and reviewing) later seasons as they're published.

walkingnorth has a Patreon, which, in addition to letting you support the artist directly, has various supporting material such as larger artwork and downloadable versions of the music.

Rating: 7 out of 10

16 January, 2019 04:02AM

January 15, 2019

hackergotchi for Keith Packard

Keith Packard

newt-lola

Newt: Replacing Bison and Flex

Bison and Flex (or any of the yacc/lex family members) are a quick way to generate reliable parsers and lexers for language development. It's easy to write a token recognizer in Flex and a grammar in Bison, and you can readily hook code up to the resulting parsing operation. However, neither Bison nor Flex are really designed for embedded systems where memory is limited and malloc is to be avoided.

When starting Newt, I didn't hesitate to use them though; it's was nice to use well tested and debugged tools so that I could focus on other parts of the implementation.

With the rest of Newt working well, I decided to go take another look at the cost of lexing and parsing to see if I could reduce their impact on the system.

A Custom Lexer

Most mature languages end up with a custom lexer. I'm not really sure why this has to be the case, but it's pretty rare to run across anyone still using Flex for lexical analysis. The lexical structure of Python is simple enough that this isn't a huge burden; the hardest part of lexing Python code is in dealing with indentation, and Flex wasn't really helping with that part much anyways.

So I decided to just follow the same path and write a custom lexer. The result generates only about 1400 bytes of Thumb code, a significant savings from the Flex version which was about 6700 bytes.

To help make the resulting language LL, I added lexical recognition of the 'is not' and 'not in' operators; instead of attempting to sort these out in the parser, the lexer does a bit of look-ahead and returns a single token for both of these.

Parsing on the Cheap

Many of common languages are "almost" LL; this may come from using recursive-descent parsers. In 'pure' form, recursive descent parsers can only recognize LL languages, but it's easy to hack them up to add a bit of look ahead here and there to make them handle non-LL cases.

Which is one reason we end up using parser generators designed to handle LALR languages instead; that class of grammars does cover most modern languages fairly well, only requiring a small number of kludges to paper over the remaining gaps. I think the other reason I like using Bison is that the way an LALR parser works makes it really easy to handle synthetic attributes during parsing. Synthetic attributes are those built from collections of tokens that match an implicit parse tree of the input.

The '$[0-9]+' notation within Bison actions represent the values of lower-level parse tree nodes, while '$$' is the attribute value passed to higher levels of the tree.

However, LALR parser generators are pretty complicated, and the resulting parse tables are not tiny. I wondered how much space I could save by using a simpler parser structure, and (equally important), one designed for embedded use. Because Python is supposed to be an actual LL language, I decided to pull out an ancient tool I had and give it a try.

Lola: Reviving Ancient Lisp Code

Back in the 80s, I wrote a little lisp called Kalypso. One of the sub-projects resulted an LL parser generator called Lola. LL parsers are a lot easier to understand than LALR parsers, so that's what I wrote.

A program written in a long-disused dialect of lisp offers two choices:

1) Get the lisp interpreter running again

2) Re-write the program in an available language.

I started trying to get Kalypso working again, and decided that it was just too hard. Kalypso was not very portably written and depended on a lot of historical architecture, including the structure of a.out files and the mapping of memory.

So, as I was writing a Python-like language anyways, I decided to transliterate Lola into Python. It's now likely the least "Pythonic" program around as it reflects a lot of common Lisp-like programming ideas. I've removed the worst of the recursive execution, but it is still full of list operations. The Python version uses tuples for lists, and provides a 'head' and 'rest' operation to manipulate them. I probably should have just called these 'car' and 'cdr'...

One benefit of using Lisp was that I could write the grammar as s-expressions and avoid needing a parser for the Lola input language. Of course, Lola is a parser generator, so it actually bootstraps itself by having the grammar for the Lola language written as Python data structures, generating a parser for that and then parsing the user's input. Here's what the Lola grammar looks like, in Lola grammar syntax:

start       : non-term start
        |
        ;
non-term    : SYMBOL @NONTERM@ COLON rules @RULES@ SEMI
        ;
rules       : rule rules-p
        ;
rules-p     : VBAR rule rules-p
        |
        ;
rule        : symbols @RULE@
        ;
symbols     : SYMBOL @SYMBOL@ symbols
        |
        ;

Lola now has a fairly clean input syntax, including the ability to code actions in C (or whatever language). It has two output modules; a Python module that generates just the Python parse table structure, and a C module that generates a complete parser, ready to incorporate into your application much as Bison does.

Lola is available in my git repository, https://keithp.com/cgit/lola.git/

Actions in Bison vs Lola

Remember how I said that Bison makes processing synthetic attributes really easy? Well, the same is not true of the simple LL parser generated by Lola.

Actions in Lola are chucks of C code executed when they appear at to top of the parse stack. However, there isn't any context for them in the parsing process itself; the parsing process discards any knowledge of production boundaries. As a result, the actions have to manually track state on a separate attribute stack. There are pushes to this stack in one action that are expected to be matched by pops in another.

The resulting actions are not very pretty, and writing them somewhat error prone. I'd love to come up with a cleaner mechanism, and I've got some ideas, but those will have to wait for another time.

Bison vs Lola in Newt

Bison generated 4kB of parse tables and a 1470 byte parser. Lola generates 2kB of parse tables and a a 1530 byte parser. So, switching has saved about 2kB of memory. Most of the parser code in both cases is probably the actions, which my guess as to why they're similar. I think the 2kB savings is worth it, but it's a close thing for sure.

15 January, 2019 07:13PM

hackergotchi for Rapha&#235;l Hertzog

Raphaël Hertzog

Freexian’s report about Debian Long Term Support, December 2018

A Debian LTS logoLike each month, here comes a report about the work of paid contributors to Debian LTS.

Individual reports

In December, about 224 work hours have been dispatched among 13 paid contributors. Their reports are available:

  • Abhijith PA did 8 hours (out of 8 hours allocated).
  • Antoine Beaupré did 24 hours (out of 24 hours allocated).
  • Ben Hutchings did 15 hours (out of 20 hours allocated, thus keeping 5 extra hours for January).
  • Brian May did 10 hours (out of 10 hours allocated).
  • Chris Lamb did 18 hours (out of 18 hours allocated).
  • Emilio Pozuelo Monfort did 44 hours (out of 30 hours allocated + 39.25 extra hours, thus keeping 25.25 extra hours for January).
  • Hugo Lefeuvre did 20 hours (out of 20 hours allocated).
  • Lucas Kanashiro did 3 hours (out of 4 hours allocated, thus keeping one extra hour for January).
  • Markus Koschany did 30 hours (out of 30 hours allocated).
  • Mike Gabriel did 21 hours (out of 10 hours allocated and 1 extra hour from November and 10 hours additionally allocated during the month).
  • Ola Lundqvist did 8 hours (out of 8 hours allocated + 7 extra hours, thus keeping 7 extra hours for January).
  • Roberto C. Sanchez did 12.75 hours (out of 12 hours allocated + 0.75 extra hours from November).
  • Thorsten Alteholz did 30 hours (out of 30 hours allocated).

Evolution of the situation

In December we managed to dispatch all the hours available to contributors again, and we had one new contributor in training. Still, we continue to be looking for new contributors. Please contact Holger if you are interested to become a paid LTS contributor.

The security tracker currently lists 37 packages with a known CVE and the dla-needed.txt file has 31 packages needing an update.

Thanks to our sponsors

New sponsors are in bold.

No comment | Liked this article? Click here. | My blog is Flattr-enabled.

15 January, 2019 10:26AM by Raphaël Hertzog

January 14, 2019

Petter Reinholdtsen

CasparCG Server for TV broadcast playout in Debian

The layered video playout server created by Sveriges Television, CasparCG Server, entered Debian today. This completes many months of work to get the source ready to go into Debian. The first upload to the Debian NEW queue happened a month ago, but the work upstream to prepare it for Debian started more than two and a half month ago. So far the casparcg-server package is only available for amd64, but I hope this can be improved. The package is in contrib because it depend on the non-free fdk-aac library. The Debian package lack support for streaming web pages because Debian is missing CEF, Chromium Embedded Framework. CEF is wanted by several packages in Debian. But because the Chromium source is not available as a build dependency, it is not yet possible to upload CEF to Debian. I hope this will change in the future.

The reason I got involved is that the Norwegian open channel Frikanalen is starting to use CasparCG for our HD playout, and I would like to have all the free software tools we use to run the TV channel available as packages from the Debian project. The last remaining piece in the puzzle is Open Broadcast Encoder, but it depend on quite a lot of patched libraries which would have to be included in Debian first.

As usual, if you use Bitcoin and want to show your support of my activities, please send Bitcoin donations to my address 15oWEoG9dUPovwmUL9KWAnYRtNJEkP1u1b.

14 January, 2019 11:10PM

hackergotchi for Jonathan Dowland

Jonathan Dowland

Amiga/Gotek boot test

This is the fourth part in a series of blog posts. The previous post was part 3: preliminaries.

A500 mainboard

A500 mainboard

In 2015 Game 2.0, a Retro Gaming exhibition visited the Centre for Life in Newcastle. On display were a range of vintage home computers and consoles, rigged up so you could play them. There was a huge range of machines, including some Arcade units and various (relatively) modern VR systems that were drawing big crowds but something else caught my attention: a modest little Amiga A500, running the classic puzzle game, "Lemmings".

A couple of weeks ago I managed to disassemble my Amiga and remove the broken floppy disk drive. The machine was pretty clean under the hood, considering its age. I fed new, longer power and data ribbon cables out of the floppy slot in the case (in order to attach the Gotek Floppy Emulator externally) and re-assembled it.

Success! Lemmings!

Success! Lemmings!

I then iterated a bit with setting up disk images and configuring the firmware on the Gotek. It was supplied with FlashFloppy, a versatile and open source firmware that can operate in a number of different modes and read disk images in a variety of formats. I had some Amiga images in "IPF" format, others in "ADF" and also some packages with "RP9" suffixes. After a bit of reading around, I realised the "IPF" ones weren't going to work, the "RP9" ones were basically ZIP archives of other disk images and metadata, and the "ADF" format was supported.

Amiga & peripherals on my desk

Amiga & peripherals on my desk

For my first boot test of the Gotek adaptor, the disk image really had to be Lemmings. Success! Now that I knew the hardware worked, I spent some time re-arranging my desk at home, to try and squeeze the Amiga, its peripherals and the small LCD monitor alongside the equipment I use for my daily work. It was a struggle but they just about fit.

The next step was to be planning out and testing a workflow for writing to virtual floppies via the Gotek. Unfortunately, before I could get as far as performing the first write test, my hardware developed a problem…

14 January, 2019 07:42PM

Amiga floppy recovery project, part 3: preliminaries

This is the third part in a series of blog posts, following Amiga floppy recovery project, part 2. The next part is Amiga/Gotek boot test.

The first step for my Amiga project was to recover the hardware from my loft and check it all worked.

When we originally bought the A500 (in, I think, 1991) we bought a RAM expansion at the same time. The base model had a whole 512KiB of RAM, but it was common for people to buy a RAM expander that doubled the amount of memory to a whopping 1 MiB. The official RAM expander was the Amiga 501, which fit into a slot on the underside of the Amiga, behind a trapdoor.

The 501 also featured a real-time clock (RTC), which was powered by a backup NiCad battery soldered onto the circuit board. These batteries are notorious for leaking over a long enough time-frame, and our Amiga had been in a loft for at least 20 years. I had heard about this problem when I first dug the machine back out in 2015, and had a vague memory that I checked the board at the time and could find no sign of leakage, but reading around the subject more recently made me nervous, so I double-checked.

AMRAM-NC-issue 2 RAM expansion

AMRAM-NC-issue 2 RAM expansion

Lo and behold, we don't have an official Commodore RAM expander: we were sold a third-party one, an "AMRAM-NC-issue 2". It contains the 512KiB RAM and a DIP switch, but no RTC or corresponding battery, so no leakage. The DIP switch was used to enable and disable the RAM expansion. Curiously it is currently flicked to "disable". I wonder if we ever actually had it switched on?

The follow-on Amiga models A500+ and A600 featured the RTC and battery directly on the machine's mainboard. I wonder if that has resulted in more of those units being irrevocably damaged from leaking batteries, compared to the A500. My neighbours had an A600, but they got rid of it at some point in the intervening decades. If I were looking to buy an Amiga model today, I'd be quite tempted by the A600, due to its low profile, lacking the numpad, and integrated composite video output and RF modulator.

Kickstart 1.3 (firmware) prompt

Kickstart 1.3 (firmware) prompt

I wasn't sure whether I was going to have to rescue my old Philips CRT monitor from the loft. It would have been a struggle to find somewhere to house the Amiga and the monitor combined, as my desk at home is already a bit cramped. Our A500 was supplied with a Commodore A520 RF adapter which we never used in the machine's heyday. Over the Christmas break I tested it and it works, meaning I can use the A500 with my trusty 15" TFT TV (which has proven very useful for testing old equipment, far outlasting many monitors I've had since I bought it).

A520 RF modulator and external FDD

A520 RF modulator and external FDD

Finally I recovered my old Amiga external floppy disk drive. From what I recall this had very light usage in the past, so hopefully it still works, although I haven't yet verified it. I had partially disassembled this back in 2015, intending to re-purpose the drive into the Amiga. Now I have the Gotek, my plans have changed, so I carefully re-assembled it. Compared to enclosures I've used for PC equipment, it's built like a tank!

The next step is to remove the faulty internal floppy disk drive from the A500 and wire up the Gotek. I was thwarted from attempting this over the Christmas break. The Amiga enclosure's top part is screwed on with T9 Torx screws, and I lacked the right screwdriver part to remove it. I've now obtained the right screwdriver bit and can proceed.

14 January, 2019 07:42PM

hackergotchi for Bits from Debian

Bits from Debian

"futurePrototype" will be the default theme for Debian 10

The theme "futurePrototype" by Alex Makas has been selected as default theme for Debian 10 'buster'.

futurePrototype Login screen. Click to see the whole theme proposal

After the Debian Desktop Team made the call for proposing themes, a total of eleven choices have been submitted, and any Debian contributor has received the opportunity to vote on them in a survey. We received 3,646 responses ranking the different choices, and futurePrototype has been the winner among them.

We'd like to thank all the designers that have participated providing nice wallpapers and artwork for Debian 10, and encourage everybody interested in this area of Debian, to join the Design Team.

Congratulations, Alex, and thank you very much for your contribution to Debian!

14 January, 2019 12:15PM by Laura Arjona Reina, Niels Thykier and Jonathan Carter

Russ Allbery

Review: The Wonder Engine

Review: The Wonder Engine, by T. Kingfisher

Series: The Clocktaur War #2
Publisher: Red Wombat Tea Company
Copyright: 2018
ASIN: B079KX1XFD
Format: Kindle
Pages: 318

The Wonder Engine is the second half of The Clocktaur War duology, following Clockwork Boys. Although there is a substantial transition between the books, I think it's best to think of this as one novel published in two parts. T. Kingfisher is a pen name for Ursula Vernon when she's writing books for adults.

The prologue has an honest-to-God recap of the previous book, and I cannot express how happy that makes me. This time, I read both books within a month of each other and didn't need it, but I've needed that sort of recap so many times in the past and am mystified by the usual resistance to including one.

Slate and company have arrived in Anuket City and obtained temporary housing in an inn. No one is trying to kill them at the moment; indeed, the city seems oblivious to the fact that it's in the middle of a war. On the plus side, this means that they can do some unharried investigation into the source of the Clocktaurs, the war machines that are coming ever closer to smashing their city. On the minus side, it's quite disconcerting, and ominous, that the Clocktaurs involve so little apparent expenditure of effort.

The next steps are fairly obvious: pull on the thread of research of the missing member of Learned Edmund's order, follow the Clocktaurs and scout the part of the city they're coming from, and make contact with the underworld and try to buy some information. The last part poses some serious problems for Slate, though. She knows the underworld of Anuket City well because she used to be part of it, before making a rather spectacular exit. If anyone figures out who she is, death by slow torture is the best she can hope for. But the underworld may be their best hope for the information they need.

If this sounds a lot like a D&D campaign, I'm giving the right impression. The thief, ranger, paladin, and priest added a gnole to their company in the previous book, but otherwise closely match a typical D&D party in a game that's de-emphasizing combat. It's a very good D&D campaign, though, with some excellent banter, the intermittent amusement of Vernon's dry sense of humor, and some fascinating tidbits of gnole politics and gnole views on humanity, which were my favorite part of the book.

Somewhat unfortunately for me, it's also a romance. Slate and Caliban, the paladin, had a few exchanges in passing in the first book, but much of The Wonder Engine involves them dancing around each other, getting exasperated with each other, and trying to decide if they're both mutually interested and if a relationship could possibly work. I don't object to the relationship, which is quite fun in places and only rarely drifts into infuriating "why won't you people talk to each other" territory. I do object to Caliban, who Slate sees as charmingly pig-headed, a bit simple, and physically gorgeous, and who I saw as a morose, self-righteous jerk.

As mentioned in my review of the previous book, this series is in part Vernon's paladin rant, and much more of that comes into play here as the story centers more around Caliban and digs into his relationship with his god and with gods in general. Based on Vernon's comments elsewhere, one of the points is to show a paladin in a different (and more religiously realistic) light than the cliche of being one crisis of faith away from some sort of fall. Caliban makes it clear that when you've had a god in your head, a crisis of faith is not the sort of thing that actually happens, since not much faith is required to believe in something you've directly experienced. (Also, as is rather directly hinted, religions tend not to recruit as paladins the people who are prone to thinking about such things deeply enough to tie themselves up in metaphysical knots.) Guilt, on the other hand... religions are very good at guilt.

Caliban is therefore interesting on that level. What sort of person is recruited as a paladin? How does that person react when they fall victim to what they fight in other people? What's the relationship between a paladin and a god, and what is the mental framework they use to make sense of that relationship? The answers here are good ones that fit a long-lasting structure of organized religious warfare in a fantasy world of directly-perceivable gods, rather than fragile, crusading, faith-driven paladins who seem obsessed with the real world's uncertainty and lack of evidence.

None of those combine into characteristics that made me like Caliban, though. While I can admire him as a bit of world-building, Slate wants to have a relationship with him. My primary reaction to that was to want to take Slate aside and explain how she deserves quite a bit better than this rather dim piece of siege equipment, no matter how good he might look without his clothes on. I really liked Slate in the first book; I liked her even better in the second (particularly given how the rescue scene in this book plays out). Personally, I think she should have dropped Caliban down a convenient well and explored the possibilities of a platonic partnership with Grimehug, the gnole, who was easily my second-favorite character in this book.

I will give Caliban credit for sincerely trying, at least in between the times when he decided to act like an insufferable martyr. And the rest of the story, while rather straightforward, enjoyably delivered on the setup in the first book and did so with a lot of good banter. Learned Edmund was a lot more fun as a character by the end of this book than he was when introduced in the first book, and that journey was fun to see. And the ultimate source of the Clocktaurs, and particularly how they fit into the politics of Anuket City, was more interesting than I had been expecting.

This book is a bit darker than Clockwork Boys, including some rather gruesome scenes, a bit of on-screen gore, and quite a lot of anticipation of torture (although thankfully no actual torture scenes). It was more tense and a bit more uncomfortable to read; the ending is not a light romp, so you'll want to be in the right mood for that.

Overall, I do recommend this duology, despite the romance. I suspect some (maybe a lot) of my reservations are peculiar to me, and the romance will work better for other people. If you like Vernon's banter (and if you don't, we have very different taste) and want to see it applied at long novel length in a D&D-style fantasy world with some truly excellent protagonists, give this series a try.

The Clocktaur War is complete with this book, but the later Swordheart is set in the same universe.

Rating: 8 out of 10

14 January, 2019 05:32AM

January 13, 2019

Russell Coker

Are Men the Victims?

A very famous blog post is Straight White Male: The Lowest Difficulty Setting There Is by John Scalzi [1]. In that post he clearly describes that life isn’t great for straight white men, but that there are many more opportunities for them.

Causes of Death

When this post is mentioned there are often objections, one common objection is that men have a lower life expectancy. The CIA World factbook (which I consider a very reliable source about such matters) says that the US life expectancy is 77.8 for males and 82.3 for females [2]. The country with the highest life expectancy is Monaco with 85.5 for males and 93.4 years for females [3]. The CDC in the US has a page with links to many summaries about causes of death [4]. The causes where men have higher rates in 2015 are heart disease (by 2.1%), cancer (by 1.7%), unintentional injuries (by 2.8%), and diabetes (by 0.4%). The difference in the death toll for heart disease, cancer, unintentional injuries, and diabetes accounts for 7% of total male deaths. The male top 10 lists of causes of death also includes suicide (2.5%) and chronic liver disease (1.9%) which aren’t even in the top 10 list for females (which means that they would each comprise less than 1.6% of the female death toll).

So the difference in life expectancy would be partly due to heart problems (which are related to stress and choices about healthy eating etc), unintentional injuries (risk seeking behaviour and work safety), cancer (the CDC reports that smoking is more popular among men than women [5] by 17.5% vs 13.5%), diabetes (linked to unhealthy food), chronic liver disease (alcohol), and suicide. Largely the difference seems to be due to psychological and sociological issues.

The American Psychological Association has for the first time published guidelines for treating men and boys [6]. It’s noteworthy that the APA states that in the past “psychology focused on men (particularly white men), to the exclusion of all others” and goes on to describe how men dominate the powerful and well paid jobs. But then states that “men commit 90 percent of homicides in the United States and represent 77 percent of homicide victims”. They then go on to say “thirteen years in the making, they draw on more than 40 years of research showing that traditional masculinity is psychologically harmful and that socializing boys to suppress their emotions causes damage that echoes both inwardly and outwardly”. The article then goes on to mention use of alcohol, tobacco, and unhealthy eating as correlated with “traditional” ideas about masculinity. One significant statement is “mental health professionals must also understand how power, privilege and sexism work both by conferring benefits to men and by trapping them in narrow roles”.

The news about the new APA guidelines focuses on the conservative reaction, the NYT has an article about this [7].

I think that there is clear evidence that more flexible ideas about gender etc are good for men’s health and directly connect to some of the major factors that affect male life expectancy. Such ideas are opposed by conservatives.

Risky Jobs

Another point that is raised is the higher rate of work accidents for men than women. In Australia it was illegal for women to work in underground mines (one of the more dangerous work environments) until the late 80’s (here’s an article about this and other issues related to women in the mining industry [8]).

I believe that people should be allowed to work at any job they are qualified for. I also believe that we need more occupational health and safety legislation to reduce the injuries and deaths at work. I don’t think that the fact that a group of (mostly male) politicians created laws to exclude women from jobs that are dangerous and well-paid while also not creating laws to mitigate the danger is my fault. I’ll vote against such politicians at every opportunity.

Military Service

Another point that is often raised is that men die in wars.

In WW1 women were only allowed to serve in the battlefield as nurses. Many women died doing that. Deaths in war has never been an exclusively male thing. Women in many countries are campaigning to be allowed to serve equally in the military (including in combat roles).

As far as I am aware the last war where developed countries had conscription was the Vietnam war. Since then military technology has developed to increasingly complex and powerful weapons systems with an increasing number of civilians and non-combat military personnel supporting each soldier who is directly involved in combat. So it doesn’t seem likely that conscription will be required for any developed country in the near future.

But not being directly involved in combat doesn’t make people safe. NPR has an interesting article about the psychological problems (potentially leading up to suicide) that drone operators and intelligence staff experience [9]. As an aside the article reference two women doing that work.

Who Is Ignoring These Things?

I’ve been accused of ignoring these problems, it’s a general pattern on the right to accuse people of ignoring these straight white male problems whenever there’s a discussion of problems that are related to not being a straight white man. I don’t think that I’m ignoring anything by failing to mention death rates due to unsafe workplaces in a discussion about the treatment of trans people. I try to stay on topic.

The New York Times article I cited shows that conservatives are the ones trying to ignore these problems. When the American Psychological Association gives guidelines on how to help men who suffer psychological problems (which presumably would reduce the suicide rate and bring male life expectancy closer to female life expectancy) they are attacked by Fox etc.

My electronic communication (blog posts, mailing list messages, etc) is mostly connected to the free software community, which is mostly male. The majority of people who read what I write are male. But it seems that the majority of positive feedback when I write about such issues is from women. I don’t think there is a problem of women or left wing commentators failing men. I think there is a problem of men and conservatives failing men.

What Can We Do?

I’m sure that there are many straight white men who see these things as problems but just don’t say anything about it. If you don’t want to go to the effort of writing a blog post then please consider signing your name to someone else’s. If you are known for your work (EG by being a well known programmer in the Linux community) then you could just comment “I agree” on a post like this and that makes a difference while also being really easy to do.

Another thing that would be good is if we could change the hard drinking culture that seems connected to computer conferences etc. Kara has an insightful article on Model View Culture about drinking and the IT industry [10]. I decided that drinking at Linux conferences had got out of hand when about 1/3 of the guys at my table at a conference dinner vomited.

Linux Conf Au (the most prestigious Linux conference) often has a Depression BoF which is really good. I hope they have one this year. As an aside I have problems with depression, anyone who needs someone to talk to about such things and would rather speak to me than attend a BoF is welcome to contact me by email (please take a failure to reply immediately as a sign that I’m behind on checking my email not anything else) or social media.

If you have any other ideas on how to improve things please make a comment here, or even better write a blog post and link to it in a comment.

13 January, 2019 01:08PM by etbe

Russ Allbery

DocKnot 2.00

This is a new major release of the utility I use to generate package documentation. It's the start of a restructure that will eventually let me merge more of my package maintenance tools into this package (and possibly eventually my web site building tools).

The functions previously provided by the docknot command-line tool have been moved to docknot generate, and the arguments have been changed around a bit. There's also a new docknot generate-all, and more default values so that one doesn't have to pass in as many arguments. The Perl module has been similarly restructured, with documentation generation moved into a new App::DocKnot::Generate module.

On the documentation template front, this release also adds a separate TESTING section for Perl modules and changes some of the templating for standard documentation of how to run the test suite.

You can get the latest release from the DocKnot distribution page or from CPAN.

13 January, 2019 05:12AM

January 11, 2019

hackergotchi for Ben Hutchings

Ben Hutchings

Debian LTS work, December 2018

I was assigned 20 hours of work by Freexian's Debian LTS initiative and worked 15 hours. I carried the remaining hours over to January.

I prepared and released another stable update for Linux 3.16 (3.16.62) and rebased jessie's linux package on this version, but did not upload a new release yet.

I also discussed the outstanding speculation-related vulnerabilities affecting Xen in jessie.

11 January, 2019 11:50PM

hackergotchi for Joachim Breitner

Joachim Breitner

Teaching to read Haskell

TL;DR: New Haskell tutorial at http://haskell-for-readers.nomeata.de/.

Half a year ago, I left the normal academic career path and joined the DFINITY Foundation, a non-profit start-up that builds a blockchain-based “Internet Computer” which will, if everything goes well, provide a general purpose, publicly owned, trustworthy service hosting platform.

DFINITY heavily bets on Haskell as a programming language to quickly develop robust and correct programs (and it was my Haskell experience that opened this door for me). DFINITY also builds heavily on innovative cryptography and cryptographic protocols to make the Internet Computer work, and has assembled an impressive group of crypto researchers.

Crypto is hard, and so is implementing crypto. How do we know that the Haskell code correctly implements what the cryptography researchers designed? Clearly, our researchers will want to review the code and make sure that everything is as intended.

But surprisingly, not everybody is Haskell-literate. This is where I come in, given that I have taught Haskell classes before, and introduce Haskell to those who do not know it well enough yet.

At first I thought I’d just re-use the material I created for the CIS 194 Haskell course at the University of Pennsylvania. But I noticed that I am facing quite a different audience. Instead of young students with fairly little computer scientist background who can spent quite a bit of time to learn to write Haskell, I am now addressing senior and very smart computer scientists with many other important things to do, who want to learn to read Haskell.

Certainly, a number of basics are needed in either case; basic functional programming for example. But soon, the needs diverge:

  • In order to write Haskell, I have to learn how to structure a program, how to read error message and deal with Haskell’s layout rule, but I don’t need to know all idioms and syntax features yet.
  • If I want to read Haskell, I need to navigate possibly big files, recognize existing structure, and deal with a plenitude of syntax, but I don’t need to worry about setting up a compiler or picking the right library.

So I set out to create a new Haskell course, “Haskell for Readers”, that is specifically tailored to this audience. It leaves out a few things that are not necessary for reading Haskell, is relatively concise and densely packed, but it starts with the basics and does not assume prior exposure to functional programming.

As it behooves for a non-profit-foundation, DFINITY is happy for me to develop the lecture material in the open, and release it to the public under a permissive creative commons license, so I invite you to read the in-progress document, and maybe learn something. Of course, my hope is to also get constructive feedback in return, and hence profit from this public release. Sources on GitHub.

11 January, 2019 09:17PM by Joachim Breitner (mail@joachim-breitner.de)

Mike Gabriel

Upcoming FreeRDP v1.1 updates for Debian jessie (LTS) and Debian stretch (please test!)

Recently, Bernhard Miklautz, Martin Fleisz and myself have been working on old FreeRDP code. Our goal was, to get FreeRDP in Debian jessie LTS and Debian stretch working again against recent Microsoft RDP servers.

It has been done now.

Context

In Debian LTS, we were discussing a complex update of the freerdp (v1.1) package. That was before X-mas.

The status of FreeRDP v1.1 (jessie/stretch) then was and still is:

  • Since March 2018 freerdp in stretch (and jessie) (Git snapshot of never released v1.1) has been unusable against latest Microsoft Windows servers. All MS Windows OS versions switched to RDP proto version 6 plus CredSSP version 3 and the freerdp versions in Debian jessie/stretch do not support that, yet.
  • For people using Debian stretch, the only viable work-around is using freerdp2 from stretch-backports.
  • People using Debian jessie LTS don't have any options (except from upgrading to stretch and using freerdp2 from stretch-bpo).
  • Currently, we know of four unfixed no-DSA CVE issues in freerdp (v1.1) (that are fixed in buster's freerdp2).

With my Debian LTS contributor hat on, I have started working on the open freerdp CVE issues (whose backported fixes luckily appeared in a Ubuntu security update, so not much work on this side) and ...

... I have started backporting the required patches (at least these: [0a,0b,0c]) to get RDP proto version 6 working in Debian jessie's and Debian stretch's freerdp v1.1 version. It turned out later that the third referenced patch [0c] is not required.

With the LTS team it was agreed that this complete endeavour for LTS only makes sense if the stable release team is open to accepting such a complex change to Debian stretch, too.

While working on these patches, I regularly got feedback from FreeRDP upstream developer Bernhard Miklautz. That was before X-mas. Over the X-mas holidays (when I took time off with the family), Bernhard Miklautz and also Martin Fleisz from FreeRDP upstream took over and a couple of days ago I was presented with a working solution. Well done, my friends. Very cool and very awesome!

As already said, recently, more and more people installed FreeRDP v2 from stretch-backports (if on stretch), but we know of many people / sysadmins that are not allowed to use packages from Debian backports' repository. Using FreeRDPv2 from stretch-backports is still a good (actually the best) option for people without strict software policies. But to those, who are not permitted to use software from Debian backports, now we can provide you with a solution.

Please test FreeRDP v1.1 upload candidates

We would love to get some feedback from brave test users. Actually, if the new update works for you, there is no need for giving feedback. However, let us know when things fail for you.

Packages have been upload to my personal staging repository:
https://packages.sunweavers.net/debian/pool/main/f/freerdp/

APT URL (stretch):

deb http://packages.sunweavers.net/debian stretch main

APT URL (jessie):

deb http://packages.sunweavers.net/debian jessie main

Obtain the archive key:

$ wget -qO - http://packages.sunweavers.net/archive.key | sudo apt-key add -

Install the FreeRDP-X11 package:

% sudo apt update
$ sudo apt install freerdp-x11

As the staging repo contains various other packages, please disable that repo immediately after having installed the new FreeRDP package versions. Thanks!

Next steps

The changeset (.debdiff) has already been sent for pre-approval to the Debian stable (aka stretch) release team [2].

I will at least postpone the upload by some more days (let's say 5 days) to give people a chance for giving feedback. When these days are over and once (and if) I have got the release team's ACK to proceed, I will upload the updated package.

Once FreeRDP has been updated in Debian stretch, I will do an immediate upload of nearly the same package (with some formal changes) to Debian jessie LTS (installable via security.debian.org).

For Debian stretch, the updated FreeRDP version will be available to all Debian stretch users with the next Debian stable point release at the latest (if nothing of the above gets delayed). The release team may give this update some priority and make it available via stable-updates prior to the next point release.

For Debian jessie, the updated FreeRDP version will be available once the update has been acknowledged by the Debian stable release team.

References

11 January, 2019 03:06PM by sunweaver

hackergotchi for Dirk Eddelbuettel

Dirk Eddelbuettel

pinp 0.0.7: More small YAML options

A good six months after the previous release, another small feature release of our pinp package for snazzier one or two column Markdown-based pdf vignettes got onto CRAN minutes ago as another [CRAN-pretest-publish] release indicating a fully automated process (as can be done for packages free of NOTES, WARNING, ERRORS, and without ‘changes to worse’ in their reverse dependency checks).

One new option was suggested (and implemented) by Ilya Kashnitsky: the bold and small subtitle carrying a default of ‘this version built on …’ with the date is now customisable; motivation was for example stating a post-publication DOI which is indeed useful. In working with DOI I also finally realized that I was blocking displays of DOIs in the references: the PNAS style use \doi{} for a footer display (which we use e.g. for vignette info) shadowing the use via the JSS.cls borrowed from the Journal of Statistical Software setup. So we renamed the YAML header option to doi_footer for clarity, still support the old entries for backwards compatibility (yes, we care), renamed the macro for this use — and with an assist from LaTeX wizard Achim Zeileis added a new \doi{} now displaying DOIs in the references as they should! We also improved some internals as e.g. the Travis CI checks but I should blog about that another time, and documented yet more YAML header options in the vignette.

A screenshot of the package vignette can be seen below. Additional screenshots of are at the pinp page.

The NEWS entry for this release follows.

Changes in pinp version 0.0.7 (2019-01-11)

  • Added some more documentation for different YAML header fields.

  • A new option has been added for a 'date_subtitle' (Ilya Kashnitsky in #64 fixing #63).

  • 'doi' YAML option renamed to 'doi_footer' to permit DOIs in refs, 'doi' header still usable (Dirk in #66 fixing #65).

  • The 'doi' macro was redefined to create a hyperlink.

Courtesy of CRANberries, there is a comparison to the previous release. More information is on the tint page. For questions or comments use the issue tracker off the GitHub repo.

This post by Dirk Eddelbuettel originated on his Thinking inside the box blog. Please report excessive re-aggregation in third-party for-profit settings.

11 January, 2019 11:24AM

Hideki Yamane

Debian Bug Squash Party Tokyo 2019-01

Hi, we'll hold an event "Debian Bug Squash Party Tokyo 2019-01" (19th, Jan).
Happy bug squashing, see you there! :)

11 January, 2019 01:55AM by Hideki Yamane (noreply@blogger.com)

January 10, 2019

hackergotchi for Bits from Debian

Bits from Debian

DebConf19 is looking for sponsors!

DebConf19 will be held in Curitiba, Brazil from July 21th to 28th, 2019. It will be preceded by DebCamp, July 14th to 19th, and Open Day on the 20th.

DebConf, Debian's annual developers conference, is an amazing event where Debian contributors from all around the world gather to present, discuss and work in teams around the Debian operating system. It is a great opportunity to get to know people responsible for the success of the project and to witness a respectful and functional distributed community in action.

The DebConf team aims to organize the Debian Conference as a self-sustaining event, despite its size and complexity. The financial contributions and support by individuals, companies and organizations are pivotal to our success.

There are many different possibilities to support DebConf and we are in the process of contacting potential sponsors from all around the globe. If you know any organization that could be interested or who would like to give back resources to FOSS, please consider handing them the sponsorship brochure or contact the fundraising team with any leads. If you are a company and want to sponsor, please contact us at sponsors@debconf.org.

Let’s work together, as every year, on making the best DebConf ever. We are waiting for you at Curitiba!

DebConf19 logo

10 January, 2019 05:30PM by Laura Arjona Reina, Andre Bianchi and Paulo Santana

hackergotchi for Jonathan Dowland

Jonathan Dowland

Amiga floppy recovery project, part 2

This is the second part in a series of blog posts. The first part is Amiga floppy recovery project. The next part is Amiga floppy recovery project, part 3: preliminaries.

Happy New Year!

Nearly four years ago I wrote about my ambition to read the data from my old collection of Amiga floppy disks. I'd wanted to do this for a long time prior to writing that. I knew we still had the Amiga but my family were fairly sure we didn't have the floppies. It was the floppies turning up in 2015 that made the project viable.

I haven't done much for this project in the last four years. I began to catalogue the floppies, and the Amiga itself, the floppies and a 2½ foot pile of old Amiga magazines moved from my parent's loft to mine (as they moved house). During the move, we did briefly power on the Amiga to make sure it still worked. It booted fine, but we couldn't read any disks. I tried about 5 before I felt it was more likely the drive was dead than all the floppies we had tried. If that were the case, then the broken drive might damage any disks we tried in it.

The next step is to replace the internal disk drive in the Amiga, so that I could boot software. After that there are a number of different approaches I could take to move data onto a modern machine, including attempting to connect the Amiga to a network for the first time in its life. I have an external floppy disk drive that I considered swapping into the Amiga, but didn't attempt it.

This Christmas my brother bought me a Gotek floppy disk emulator, modified for use with Amiga computers. This is a lightweight device that can be connected to the Amiga in place of the real floppy disk drive, re-using the existing power and data ribbon cables. It features a USB port, two buttons and a small LCD numeric display. The intention is that you insert a USB drive containing Amiga disk images, and the Gotek presents them to the Amiga as if they were real disks.

Thanks to this device, I can now finally embark upon my disk-reading project!

10 January, 2019 04:41PM

hackergotchi for Joachim Breitner

Joachim Breitner

Tiny nonces paper is online

Nadia Heninger and me just have put the preprint version of our paper “Biased Nonce Sense: Lattice Attacks against Weak ECDSA Signatures in Cryptocurrencies”, to be presented at Financial Crypto 2019, online. In this work, we see how many private keys used on the Bitcoin, Ethereum and Ripple blockchain, as well as in HTTPS and SSH, were used in an unsafe way and hence can be compromised. The resulting numbers are not large – 300 Bitcoin keys, with a balance of around $54 – but it shows (again and again) that it can be tricky to get crypto right, and that if you don’t get it right, you can lose your money.

Brief summary

When you create a cryptographic signatures using ECDSA (the elliptic curve digital signature algorithm), you need to come up with the nonce, a 256 bit random number. It is really important to use a different nonce every time, otherwise it is easy for someone else to take your signatures (which might be stored for everyone to read on the Bitcoin blockchain) and calculate your private key using relatively simple math, and with your private key they can spend all your Bitcoins. In fact, there is evidence that people out there continuously monitor the blockchains for signatures with such repeated nonces and immediately extract the money from compromised keys.

Less well known, but still nothing new to the crypto (as in cryptopgraphy) community is the that an attacker can calculate the key from signature that use different, but similar nonces: For example if they are close by each other (only the low bits differ), or if they differ by exactly a large power of two (only the high bits differ). This uses a fancy and powerful technique based on lattices. Our main contribution here is to bridge crypto (as in cryptopgraphy) and crypto (as in cryptocurrency) and see if such vulnerabilities actually exist out there.

And indeed, there are some. Not many (which is good), but they do exist, and clearly due to more than one source. Unfortunately, it is really hard to find out who made these signatures, and with which code, so we can only guess about the causes of these bugs. A large number of affected signatures are related to multisig transactions, so we believe that maybe hardware tokens could be the cause here.

Observing programming bugs

Even though we could not identify the concrete implementations that caused these problems, we could still observe some interesting details about them. The most curious is certainly this one:

One set of signatures, which incidentally were created by an attacker who emptied out accounts of compromised keys (e.g. those that are created with a weak password, or otherwise leaked onto the internet), was using nonces that shared the low 128 bits, and hence revealed the (already compromised) private key of the account he emptied out. Curiously, these low 128 bits are precisely the upper 128 bits of the private key.

So it very much looks like the attacker hacked up a program that monitors the blockchain and empties out accounts, and probably did so using a memory unsafe language like C, and got the size of the buffers for nonce and key wrong, so maybe they did properly filled the nonce with good random data, but when they wrote the secret key, the memory locations overlapped and they overrode parts of their randomness with the very non-random secret key. Oh well.

Do I need to worry?

Probably not. The official blockchain clients get their crypto right (at least this part), and use properly random nonces, so as a user you don’t have to worry. In fact, since 2016, the Bitcoin client uses deterministic signatures (RFC6979) which completely removes the need for randomness in the process.

If you are using non-standard libraries, or if you write your own crypto routines (which you should only ever do if you have a really really good reason for it) you should make sure that these use RFC6979. This is even more important on embedded devices or hardware tokens where a good source of randomness might be hard to come by.

Discrete logarithm in secp256k1 with lookup table

In the course of this work I wanted to find out if small nonces (<264) were used even when the key created only one of these – the lattice-based attacks need at least two signatures to work. So I created code that calculates the discrete log in the secp256k1 curve up to an exponent of (<264). This is made feasible using a lookup table for smaller exponents (<239 in our case – just large enough to still fit into 2.2TB of RAM).

This exercise turned out to be not very useful; we did not find any additional keys, but I had fun writing up the program, implemented in C and working very close to the raw data, juggling big binary files mmap’ed into memory, and implementing custom lookup indices and such. In the hope that this might be useful to someone, I share the code at https://github.com/nomeata/secp265k1-lookup-table.

10 January, 2019 08:04AM by Joachim Breitner (mail@joachim-breitner.de)

Nonce sense paper online

Nadia Heninger and I just have put the preprint version of our paper “Biased Nonce Sense: Lattice Attacks against Weak ECDSA Signatures in Cryptocurrencies”, to be presented at Financial Crypto 2019, online. In this work, we see how many private keys used on the Bitcoin, Ethereum and Ripple blockchain, as well as in HTTPS and SSH, were used in an unsafe way and hence can be compromised. The resulting numbers are not large – 300 Bitcoin keys, with a balance of around $54 – but it shows (again and again) that it can be tricky to get crypto right, and that if you don’t get it right, you can lose your money.

Brief summary

When you create a cryptographic signatures using ECDSA (the elliptic curve digital signature algorithm), you need to come up with the nonce, a 256 bit random number. It is really important to use a different nonce every time, otherwise it is easy for someone else to take your signatures (which might be stored for everyone to read on the Bitcoin blockchain) and calculate your private key using relatively simple math, and with your private key they can spend all your Bitcoins. In fact, there is evidence that people out there continuously monitor the blockchains for signatures with such repeated nonces and immediately extract the money from compromised keys.

Less well known, but still nothing new to the crypto (as in cryptography) community is the that an attacker can calculate the key from signature that use different, but similar nonces: For example if they are close by each other (only the low bits differ), or if they differ by exactly a large power of two (only the high bits differ). This uses a fancy and powerful technique based on lattices. Our main contribution here is to bridge crypto (as in cryptography) and crypto (as in cryptocurrency) and see if such vulnerabilities actually exist out there.

And indeed, there are some. Not many (which is good), but they do exist, and clearly due to more than one source. Unfortunately, it is really hard to find out who made these signatures, and with which code, so we can only guess about the causes of these bugs. A large number of affected signatures are related to multisig transactions, so we believe that maybe hardware tokens could be the cause here.

Observing programming bugs

Even though we could not identify the concrete implementations that caused these problems, we could still observe some interesting details about them. The most curious is certainly this one:

One set of signatures, which incidentally were created by an attacker who emptied out accounts of compromised keys (e.g. those that are created with a weak password, or otherwise leaked onto the internet), was using nonces that shared the low 128 bits, and hence revealed the (already compromised) private key of the account he emptied out. Curiously, these low 128 bits are precisely the upper 128 bits of the private key.

So it very much looks like the attacker hacked up a program that monitors the blockchain and empties out accounts, and probably did so using a memory unsafe language like C, and got the size of the buffers for nonce and key wrong, so maybe they did properly filled the nonce with good random data, but when they wrote the secret key, the memory locations overlapped and they overrode parts of their randomness with the very non-random secret key. Oh well.

Do I need to worry?

Probably not. The official blockchain clients get their crypto right (at least this part), and use properly random nonces, so as a user you don’t have to worry. In fact, since 2016, the Bitcoin client uses deterministic signatures (RFC6979) which completely removes the need for randomness in the process.

If you are using non-standard libraries, or if you write your own crypto routines (which you should only ever do if you have a really really good reason for it) you should make sure that these use RFC6979. This is even more important on embedded devices or hardware tokens where a good source of randomness might be hard to come by.

Discrete logarithm in secp256k1 with lookup table

In the course of this work I wanted to find out if small nonces (<264) were used even when the key created only one of these – the lattice-based attacks need at least two signatures to work. So I created code that calculates the discrete log in the secp256k1 curve up to an exponent of (<264). This is made feasible using a lookup table for smaller exponents (<239 in our case – just large enough to still fit into 2.2TB of RAM).

This exercise turned out to be not very useful; we did not find any additional keys, but I had fun writing up the program, implemented in C and working very close to the raw data, juggling big binary files mmap’ed into memory, and implementing custom lookup indices and such. In the hope that this might be useful to someone, I share the code at https://github.com/nomeata/secp265k1-lookup-table.

10 January, 2019 08:04AM by Joachim Breitner (mail@joachim-breitner.de)

Russ Allbery

Review: Bright Earth

Review: Bright Earth, by Philip Ball

Publisher: University of Chicago
Copyright: 2001
Printing: 2003
ISBN: 0-226-03628-6
Format: Trade paperback
Pages: 337

The subtitle Art and the Invention of Color does a good job advertising the topic of Bright Earth: a history of the creation of color pigments for art (specifically European painting; more on that in a moment). It starts with a brief linguistic and scientific introduction to color, sketches what's known about use and creation of color pigments in antiquity, and then settles down for serious historical study starting in the Middle Ages. Ball catalogs pigment choices, discusses manufacturing methods, and briefly surveys the opinions of various schools of art on color from before the Renaissance through to the modern art of today. He also takes two fascinating (albeit too brief) side trips to discuss aging of pigments and the problem of reproducing color art.

This is one of those non-fiction books whose primary joy for me was to introduce me to problems and constraints that were obvious in retrospect but that I'd never thought about. If someone had asked me whether painters were limited in their subject matter and methods by the colors available to them, I probably would have said "huh" and agreed, but I never thought to ask the question. Like a lot of people of my age in the US, I grew up watching Bob Ross's The Joy of Painting and its familiar list of oil paints: phthalo green, alizarin crimson, and so forth. But of course that rich palette is a product of modern chemistry. Early Renaissance painters had to make do with fewer options, many of them requiring painstaking preparation that painters or their assistants did themselves before the popularity of art and the rise of professional color makers. They knew, and were shaped by, their materials in a way that one cannot be when one buys tubes of paint from an art store.

Similarly, I was familiar with additive color mixing from physics and from computer graphics projects, and had assumed that a few reasonable primaries would provide access to the entire palette. I had never considered the now-obvious problem of subtractive mixing with impure primaries: since the pigments are removing colors from white light, mixing together multiple pigments quickly gets you a muddy brown, not a brilliant secondary color. The resulting deep distrust of mixing pigments that dates back to antiquity further limits the options available to painters.

Ball's primary topic is the complicated interplay between painting and science. Many of the new colors of the Renaissance were byproducts or accidents of alchemy, and were deeply entangled in the obsession with the transmutation of metals into gold. Most of the rest were repurposed dyes from the much more lucrative textile business. Enlightenment chemistry gave rise to a whole new palette, but the chemistry of colors is complex and fickle. Going into this book, I had a superficial impression that particular elements or compounds had particular colors, and finding pigments would be a matter of finding substances that happened to have that color. Ball debunks that idea quickly: small variations in chemical structure, and thus small variations in preparation, can produce wildly different colors. Better chemistry led to more and better colors, but mostly by accident or trial and error until surprisingly recently. The process to make a color almost always came first; understanding of why it worked might be delayed centuries.

In school, I was an indifferent art student at best, so a lot of my enjoyment of Bright Earth came from its whirlwind tour of art history through the specific lens of color. I hadn't understood why medieval European paintings seem so artificial and flat before reading this book, or why, to my modern eye, Renaissance work suddenly became more beautiful and interesting. I had also never thought about the crisis that photography caused for painting, or how much that explains of the modern move away from representational art. And I had seriously underestimated the degree to which colors are historically symbolic rather than representational. This material may be old news for those who paid attention in art history courses (or, *cough*, took them in the first place), but I enjoyed the introduction. (I often find topics more approachable when presented through an idiosyncratic lens like this.)

Ball is clear, straightforward, and keeps the overall picture coherent throughout, which probably means that he's simplifying dramatically given that the scope of this book is nothing less than the entire history of European and American painting. But I'm a nearly complete newcomer to this topic, and he kept me afloat despite the flood of references to paintings that I've never seen or thought about, always providing enough detail for me to follow his points about color. You definitely do not have to already know art history to get a lot out of this book.

I do have one caveat and one grumble. The caveat is that, despite the subtitle, this book is not about art in general. It's specifically about painting, and more specifically focused on the subset of painting that qualifies as "fine art." Ball writes just enough about textiles to hint that the vast world of dyes may be even more interesting, and were certainly more important to more people, but textiles are largely omitted from this story. More notably, one would not be able to tell from this book that eastern Asia or Africa or pre-colonial America exist, let alone have their own artistic conventions and history. Ball's topic is relentlessly limited to Europe, and then the United States, except for a few quick trips to India or Afghanistan for raw materials. There's nothing inherently wrong with this — Ball already has more history than he can fully cover in only Europe and the United States — but it would have been nice to read a more explicit acknowledgment and at least a few passing mentions of how other cultures approached this problem.

The grumble is just a minor mismatch of interests between Ball and myself, namely that the one brief chapter on art reproduction was nowhere near enough for me, and I would have loved to read three or four chapters (or a whole book) on that topic. I suspect my lack of appreciation of paintings has a lot to do with the challenges of reproducing works of art in books or on a computer screen, and would have loved more technical detail on what succeeds and what fails and how one can tell whether a reproduction is "correct" or not. I would have traded off a few alchemical recipes for more on that modern problem. Maybe I'll have to find another book.

As mentioned above, I'm not a good person to recommend books about art to anyone who knows something about art. But with that disclaimer, and the warning that the whirlwind tour of art history mixed with the maddening ambiguity of color words can be a bit overwhelming in spots, I enjoyed reading this more than I expected and will gladly recommend it.

Bright Earth does not appear to be available as an ebook, and I think that may be a wise choice. The 66 included color plates help a great deal, and I wouldn't want to read this book without them. Unless any future ebook comes with very good digital reproductions, you may want to read this book in dead tree form.

Rating: 7 out of 10

10 January, 2019 04:26AM

hackergotchi for Gunnar Wolf

Gunnar Wolf

Finally, a sensible increase in participation for Tor in Mexico!

/Known fact: Latin America's share of participation in different aspects of the free software movement is very low.

There are many hypotheses for this, but all in all, it's mainly economics related: Only a tiny minority of us in this geographic region can spare the time, energy and money needed to donate part of our work and life to a project, no matter how much we agree with it. Of course, this cannot explain it wholly; there are many issues that further contribute with this low participation. Free software development is mostly carried out in English (much more so even than programming in general, although basically any programing language "reeks" of English).

In mid-2017, the Tor project acknowledged this and created the Global South Initiative. At first, I heard about it when the global-south@lists.torproject.org mailing list was started, and started interacting there right away. Roughly a month later, we started to plan for what is now our research/documentation project. We even managed to somehow attract the Tor community at large for the Tor Meeting last September/October in Mexico City (which was a *great* opportunity!)

One of the issues we have been pushing for, with marginal success rate until very recently, is to get more people involved running Tor relays or, if possible, exit nodes. Of course, when I asked officially for permission to set up an exit node at the university (I want to do things the right way), I was right away slammed and denied.

But... Patience, time, hardware donation by Derechos Digitales, and some determination have led us to the fact that... 18 months ago, we only had one or two active Tor relays. Now, the reality is finally changing!

Thanks to many individuals willing to donate their time and resources, we currently have eleven relays (eight of them which I can recognize by name and thank their respective owners — The linked page will probably give different results, as it varies over time).

As for the diversity this brings to the network, it's well summed up by the aggregated search:

Four autonomous systems; the only ISP that's usable for home users we have been able to identify is Axtel, with which we have five relays currently running; three at UNAM, the biggest university in the country; one in CINVESTAV, an important research facility; finally, one in Mega Cable, which surprises me, as Mega Cable does not provide a reachable IP for any of the subscribers we have probed! (Maybe it's run by corporate users or something like that?)

And, very notably: I have to recognize and thank our friends at Red en Defensa de los Derechos Digitales (R3D), as they have set up our –so far– only exit node (via the Axtel ISP). Wow!

Ten relays, mind you, is still a tiny contribution. Due to the bandwidth we are currently able to offer (and many many many other factors I cannot go into details, as I don't even know them all), Mexico as a country is currently providing approximately 0.05% (that is, one out of each 2000) Tor connections as a guard (entry) node, a slightly higher amount as a middle node, and a slightly lower amount as an exit node. But it is steadily increasing, and that's great!

AttachmentSize
relays.png85.37 KB
aggr_relays.png52.3 KB

10 January, 2019 12:23AM by gwolf

January 09, 2019

hackergotchi for Markus Koschany

Markus Koschany

My Free Software Activities in December 2018

Welcome to gambaru.de. Here is my monthly report that covers what I have been doing for Debian. If you’re interested in Java, Games and LTS topics, this might be interesting for you.

Debian Games

  • I used this month to polish some of my team-maintained packages and to slightly improve the debian packaging in openyahtzee, monopd, opencity, pangzero, powermanga, ri-li, tecnoballz, whichwayisup, atanks, ufoai and dreamchess.
  • I fixed RC bug #915453 in supertuxkart.
  • I released a new version of debian-games,  a collection of metapackages to ease the installation of games in Debian. I plan to do another update in January. This one will then almost be the final state for Buster but there is usually another last minor update during deep freeze to include even the latest changes.
  • I also packaged a new upstream version of enemylines3, which was merely a bug fix release though. Nevertheless I could drop two Debian patches. Yeah.

Debian Java

Misc

  • I updated osmo, tofrodos and iftop and applied a patch by Andreas Henriksson for wbar to  fix a reproducibility issue on merged-usr systems.
  • The browser extension privacybadger was updated to version 2018.12.17.
  • I prepared a security update of libarchive for Stretch released as DSA-4360-1.
  • I reported a FTBFS that got recently fixed in moria. (#916030)

NMU

Debian LTS

This was my thirty-fourth month as a paid contributor and I have been paid to work 30 hours on Debian LTS, a project started by Raphaël Hertzog. In that time I did the following:

  • From 17.12.2018 until 06.01.2019 I was in charge of our LTS frontdesk. I investigated and triaged CVE in graphiscmagick, sqlite3, libvncserver, pspp, yara, terminology, sssd, libarchive, freecol, rabbitmq-server, hoteldruid, libraw, nagios3, gnupg2, igraph, python3.4, radare2, imagemagick, tar, poppler, tcpreplay,  libcaca, binutils, liblas, mxml, jasper, aria2, systemd, libpff, libsixel, libspring-security-2.0-java, nasm, yaml-cpp and yaml-cpp0.3.
  • DLA-1630-1. I triaged and investigated 39 CVE in libav. Later I issued a security update for libav fixing 14 of them.
  • DLA-1612-1. Issued a security update for libarchive fixing 2 CVE.
  • DLA-1615-1. Issued a security update for nagios3 fixing 5 CVE.
  • DLA-1616-1. Issued a security update for libextractor fixing 2 CVE.
  • DLA-1628-1. Issued a security update for jasper fixing 8 CVE (announced 9). It turned out that CVE-2018-19139 has not been fixed yet.

ELTS

Extended Long Term Support (ELTS) is a project led by Freexian to further extend the lifetime of Debian releases. It is not an official Debian project but all Debian users benefit from it without cost. The current ELTS release is Debian 7 „Wheezy“. This was my seventh month and I have been paid to work 15 hours on ELTS.

  • I was in charge of our ELTS frontdesk from 17.12.2018 until 06.01.2019 and I triaged CVE in libarchive, gnutls26, rabbitmq-server, binutils, wget, tar, krb5, jasper and systemd.
  • ELA-72-1. Issued a security update for jasper fixing 5 CVE. I analyzed the remaining open issues, prepared patches myself and forwarded them upstream.
  • ELA-73-1. Issued a security update for libcaca fixing 4 CVE.
  • ELA-74-1. Issued a security update for sqlite3 fixing 3 CVE.

Thanks for reading and see you next time.

09 January, 2019 07:43PM by Apo

hackergotchi for Gergely Nagy

Gergely Nagy

One hat less

Almost a week ago, on a snowy night of January 3, I hung up my Debian Developer hat, and sent my retirement letter in. This has been a long time coming, and I contemplated doing this many times over the past year or two, but never got the courage and the willpower to retire. It wasn't easy. Debian has been part of my life for the past two decades, and I've been a developer for about eighteen years. I considered - and to some extent, still do - Debian my second family. There have been times when Debian was my escape hatch, something I could focus on to help me pass the days by. I've made friends, I've met people, I learned humility, I learned empathy. If it weren't for Debian, I wouldn't be the person I am today. Retiring felt like admitting defeat for a long time. But it isn't.

It's not defeat when you admit that your circumstances changed. It's not defeat when you let others take better care of the stuff you've been neglecting for way too long. It's not defeat when you're taking a break. It's not defeat when you're reducing the amount of stress that keeps you awake at night. It's not defeat when you do the right thing. It's not defeat when you gracefully retire: rather, it is a start of something new.

I originally joined Debian for two reasons: I was a Debian user for a while by then, and felt that I should be giving back, as I received so much from Debian. The second reason wasn't this noble, but after eighteen years, I guess I could admit it: the other reason I applied was vanity. Having a @debian.org email-address was something.

At the time, I was... a different person. I was full of myself at times, I was angry, perhaps a tad too elitist at times too. I'm not proud of old me, but it's part of me. I grew, and became a better person, there's no shame in being able to grow - quite the contrary. And Debian helped immensely. I've had role models in the project, who I look up to even to this day, who helped shape me one way or the other.

There are two people I need to mention specifically: Martin Michlmayr and Rhonda D'Vine.

Martin was my Application Manager when I applied, I considered him a mentor, a friend. The example he set were instrumental in shaping me too. Rhonda helped me get to my first ever conference: I got on a train to Vienna, and she took me to LinuxTag in her car from there, and then back again. That first LinuxTag, the path that led there, the conference itself, was formative, and both Martin and Rhonda had a big part in it being so. Thank you again - so many years later, I still smile when I think back. Those years we were in touch, meant a lot to me.

I often feel nostalgic about the times way back then, when I was active on IRC. I miss the old bunch (oh, so many people I haven't talked to in years!). I miss being active in Debian. I miss talking to fellow developers. But life took me elsewhere, and there's only so many hours in a day. Over the years Debian and me, we grew apart. There was always something more important, so my Debian contributions dropped (though, I had some ups too, while I was ftp-master assistant for example). By 2018, I barely did anything, and one of my resolutions for the new year was that I'll be taking better care of my health, mental and physical alike. Part of this is reducing my workload, reducing the stress levels, and things I feel guilty about.

It's a strange feeling. A mix of sadness and relief. On the flip side, as Lars put it, I'm now part of the retired DD club, and that's one illustrious club to be part of. First I joined the Debian Parents Club, now the Retired Debian Developers Club. This is a good start of a new chapter.

At some point though, I will be back. Debian is part of my life, and always will be. One way or the other, there will be a way back. It may take years, or a decade, but I'll put on that swirly hat again.

09 January, 2019 11:45AM by Gergely Nagy

hackergotchi for Daniel Lange

Daniel Lange

Apple Time Machine backups on Debian 9 (Stretch)

Netatalk 3.1.12 has been released which fixes an 18 year old RCE bug. The Medium write up on CVE-2018-1160 by Jacob Baines is quite an entertaining read.

The full release notes for 3.1.12 are unfortunately not even half as interesting.

Warning: Read the original blog post before installing for the first time. Be sure to read the original blog post if you are new to Netatalk3 on Debian Jessie or Stretch!
You'll get nowhere if you install the .debs below and don't know about the upgrade path from 2.2.x which is still in the Debian archive. So RTFA.

For Debian Buster (Debian 10) we'll have Samba 4.9 which has learnt (from Samba 4.8.0 onwards) how to emulate a SMB time machine share. I'll make a write up how to install this once Buster stabilizes. This luckily means there will be no need to continue supporting Netatalk in normal production environments. So I guess bug #690227 won't see a proper fix anymore. Waiting out problems helps at times, too :/.

Update instructions and downloads:

Continue reading " Apple Time Machine backups on Debian 9 (Stretch)"

09 January, 2019 10:29AM by Daniel Lange

Andrej Shadura

resvg: worth having in Debian?

Yesterday I have discovered resvg, an MPL 2.0-licensed SVG rendering and optimisation library and a tool, written in Rust. It is said to be faster than some SVG renderers while currently slower than librsvg:

Speed comparison for the Elementary Icon Theme with resvg/cairo being faster than QtSvg but slower than librsvg and resvg/qt being slower than QtSvg Speed comparison for the Oxygen Icon Theme with resvg/cairo being faster than QtSvg but slower than librsvg and resvg/qt being slower than QtSvg

It aims to support the static subset of SVG better than other libraries:

SVG test suite results: resvg 249, Inkscape 253, librsvg 222

The author writes:

One of the major differences from other rendering libraries is that resvg does a lot of preprocessing before rendering. It converts shapes to paths, resolves attributes, removes groups and invisible elements, fixes a lot of issues in malformed SVG files. Then it creates a simple render tree with all elements and attributes resolved. And only then it starts to render. So it's very easy to implement a new rendering backend.

  • librsvg, currently, is heavily tied to the cairo library, unlike resvg
  • librsvg is heavily tied to GNOME which makes it painful to distribute outside the Linux ecosystem
  • librsvg doesn't really preprocess input files, rendering them as is
  • librsvg has a minimal support of the edge-cases, which leads to rendering errors

I’m thinking of packaging this for Debian, but I would be interested to know what others think of this.

09 January, 2019 10:05AM by Andrej Shadura

Help the Conservancy raise the remaining $14 000

The Software Freedom Conservancy is having the last 7 days to collect the remaining less than $14 000 of the fundraiser generously matched by Private Internet Access. All donations up to $90 000 will be matched until 15 January.

Conservancy is an organisation sponsoring nearly 50 free software projects helping them, most importantly with accounting, paying developers and defending their trademarks and ensuring license compliance.

Conservancy is currently home to almost fifty member projects

Read more about what the Conservancy does on their website. The matching funds expire soon, so if you can, please donate directly before 15 January.

09 January, 2019 09:37AM by Andrej Shadura

Elana Hashman

Hot chocolate

My roommate once told me that “[I] make the best hot chocolate [he's] ever tasted.” This was a little surprising to me, because my hot chocolate recipe is not sophisticated, but I guess it does taste pretty good. 😊

Today I will share my secrets with the world.

Homemade hot chocolate

Ingredients:

  • 1 tsp granulated sugar
  • 1 tbsp unsweetened cocoa powder
  • splash of vanilla extract (or paste, if you're really fancy)
  • hot water
  • 10-12oz milk (I like whole milk)
  • pinch salt

Recipe:

  1. Mix the sugar, cocoa, and vanilla in a microwave-safe liquid measuring cup. Add a small amount of hot water and stir until the mixture is fully combined.
  2. Add milk while stirring until you have about one serving of hot chocolate, about 10-12oz. Taste.
  3. Adjust any ingredients to taste (more sugar, more cocoa, etc.) and add salt.
  4. Microwave for about 1 minute on high. Stir and taste for a temperature check.
  5. Microwave for about 30 more seconds or until sufficiently hot.
  6. Pour into a mug or thermos for serving.
  7. Enjoy!

Variations

Spicy hot chocolate: substitute ¼ tsp cinnamon for the vanilla. Add a dash of cayenne.

Peppermint hot chocolate: substitute a splash of mint extract for the vanilla.

Non-dairy hot chocolate: use your favourite non-dairy milk substitute in place of milk. If it's vanilla-flavoured, leave out the vanilla. If it's sweetened, you may need to reduce the amount of sugar added.

09 January, 2019 03:00AM by Elana Hashman

January 08, 2019

hackergotchi for Jonathan McDowell

Jonathan McDowell

MP130 Maple Leaf and Debian

Like any conference one of the nice things about DebConf is the random interesting people you meet. At the DebConf18 conference dinner I had the pleasure of talking to Jia-Bin Huang, from Centrum Embedded Systems. He told me about how he was using Debian as the basis for his MapleBoard (mostly in Chinese, unfortunately) project, which is based on the Allwinner H3 and has thousands of shipped units. The next day I went to take a look at the website (Google Translate proved helpful), which has details of the board including full schematics of the board. I reached out to Jia-Bin to see if he was still at the conference and if he had any boards with him, but he’d already returned to Taipei. He generously offered to send me one and a few weeks later (mostly due to UK customs delays) I had an “Economy Edition” board in my hands.

MapleBoard MP130

Getting up and running was simple; the board came with a pl2303 based USB serial cable, but I found it a little unreliable so ended up using my own cable. The supplied documentation was in Chinese, but the login details were clearly visible - username mpl1, password 1234. After logging in I found it was running a lightly customized Debian Stretch variant, with the following packages not from the main Debian repository:

base-files              9.9+mp4     Maple base system miscellaneous files
debian-archive-keyring  2017.7+mp1  GnuPG archive keys of the Debian archive
distro-info-data        0.36+mp1    information about the distributions' releases (data files)
linux-image-4.15.2a…    4.15.2a…    Linux kernel, version 4.15.2a-02769-g6d0ce60c8d56
maplewebde              0.1~rc4-2   Web interface to communicate with mapleboard

maplewebde seems to be a web interface for interacting with the board, but it’s in Chinese so I wasn’t able to make much sense of it. I was more interested in the kernel - how much customisation had been done, and how much was already mainlined. Happily the Linux sunxi folk have done a lot of great work in getting things upstream, so although the supplied kernel was using its own drivers (largely branded Mapleboard rather than Allwinner) all the necessary pieces were in mainline. I did a little bit of cleanup of the supplied device tree configuration file, and am pleased to say that as of 5.0-rc1 a multi_v7_defconfig will build a kernel and a sun8i-h3-mapleboard-mp130.dtb file which Just Work™ on the device.

What about the board itself? I haven’t thrown a lot at it, but it feels reasonably responsive compared to some other ARM boards I’ve played with. Quad 1GHz cores and 1GB RAM is a nice enough base spec, though it is ARMv7 so 32-bit only. Unfortunately the “Economy Edition” doesn’t have HDMI on board or I’d have seen about trying to use it as a media player - the video decoding engine apparently has Free drivers based on a vdpau backend, which is pretty cool. There’s no SATA, so it can’t be pressed into service as a build machine easily. I suspect in the long run I’ll press it into service as part of my home automation setup, somewhere I need more oompf than an ESP8266 but still want low power consumption - there are a number of GPIOs conveniently brought out to a 40 pin header.

In general terms of the target market my understanding is the board is largely pitched as a development board, with Centrum being prepared to do customisations for sufficiently sized runs. The open nature of the board, and availability of good support upstream (even if it’s come from the community rather than Allwinner) certainly makes it an attractive option if you’re looking for a 32-bit ARM platform, and although I’m not aware of any option other than ordering from Taiwan at present I’ve found Jia-Bin to be helpful with any queries I’ve had. Also I quite like supporting companies that are using Debian. :) Worth a look if you’re in the market for such a thing.

08 January, 2019 07:07PM

Ingo Juergensmann

Too much disk IO on sda in RAID10 setup - part 2

Some days ago I blogged about my issue with one of the disks in my server having a high utilization and latency. There have been several ideas and guesses what the reason might be, but I think I found the root cause today:

SMART Self-test log structure revision number 1
Num  Test_Description    Status                  Remaining  LifeTime(hours)  LBA_of_first_error
# 1  Extended offline    Completed: read failure       90%     50113         60067424
# 2  Short offline       Completed without error       00%      3346   

After the RAID-Sync-Sunday last weekend I removed sda from my RAIDs today and started a "smartctl -t long /dev/sda". This test was quickly aborted because it already ran into an read error after just a few minutes. Currently I'm still running a "badblocks -w" test and this is the result so far:

# badblocks -s -c 65536 -w /dev/sda
Testing with pattern 0xaa: done
Reading and comparing: 42731372done, 4:57:27 elapsed. (0/0/0 errors)
42731373done, 4:57:30 elapsed. (1/0/0 errors)
42731374done, 4:57:33 elapsed. (2/0/0 errors)
42731375done, 4:57:36 elapsed. (3/0/0 errors)
 46.82% done, 6:44:41 elapsed. (4/0/0 errors)

Long story short: I already ordered a replacement disk!

But what's also interesting is this:

I removed the disk today at approx. 12:00 and you can see the immediate effect on the other disks/LVs (the high blue graph from sda shows the badblocks test), although the RAID10 is now in degraded mode. Interesting what effect (currently) 4 defect blocks can have to a RAID10 performance without smartctl taking notice of this. Smartctl only reported an issue after issueing the selftest. It's also strange that the latency and high utilization slowly increased over time, like 6 months or so.

Kategorie: 
 

08 January, 2019 07:07PM by ij

François Marier

Erasing Persistent Storage Securely on Linux

Here are some notes on how to securely delete computer data in a way that makes it impractical for anybody to recover that data. This is an important thing to do before giving away (or throwing away) old disks.

Ideally though, it's better not to have to rely on secure erasure and start use full-disk encryption right from the start, for example, using LUKS. That way if the secure deletion fails for whatever reason, or can't be performed (e.g. the drive is dead), then it's not a big deal.

Rotating hard drives

With ATA or SCSI hard drives, DBAN seems to be the ideal solution.

  1. Burn it on CD,
  2. boot with it,
  3. and following the instructions.

Note that you should disconnect any drives you don't want to erase before booting with that CD.

This is probably the most trustworth method of wiping since it uses free and open source software to write to each sector of the drive several times. The methods that follow rely on proprietary software built into the firmware of the devices and so you have to trust that it is implemented properly and not backdoored.

ATA / SATA solid-state drives

Due to the nature of solid-state storage (i.e. the lifetime number of writes is limited), it's not a good idea to use DBAN for those. Instead, we must rely on the vendor's implementation of ATA Secure Erase.

First, set a password on the drive:

hdparm --user-master u --security-set-pass p /dev/sdX

and then issue a Secure Erase command:

hdparm --user-master u --security-erase-enhanced p /dev/sdX

NVMe solid-state drives

For SSDs using an NVMe connector, simply request a User Data Erase

nvme format -s1 /dev/nvme0n1

08 January, 2019 04:55PM

hackergotchi for Bits from Debian

Bits from Debian

New Debian Developers and Maintainers (November and December 2018)

The following contributors got their Debian Developer accounts in the last two months:

  • Abhijith PA (abhijith)
  • Philippe Thierry (philou)
  • Kai-Chung Yan (seamlik)
  • Simon Qhuigley (tsimonq2)
  • Daniele Tricoli (eriol)
  • Molly de Blanc (mollydb)

The following contributors were added as Debian Maintainers in the last two months:

  • Nicolas Mora
  • Wolfgang Silbermayr
  • Marcos Fouces
  • kpcyrd
  • Scott Martin Leggett

Congratulations!

08 January, 2019 01:00PM by Jean-Pierre Giraud

Reproducible builds folks

Reproducible Builds: Weekly report #193

Happy new year from the Reproducible Builds project! Here’s what happened in the Reproducible Builds effort between Sunday December 30th and Saturday January 5th:

Packages reviewed and fixed, and bugs filed

Test framework development

There were a number of updates to our Jenkins-based testing framework that powers tests.reproducible-builds.org this week, including:

  • Holger Levsen:
  • Mattia Rizzolo:
    • Set real_year=2019 in the reproducible health check script to 2019. []
    • Node maintenance. ([1], [2], etc.)

This week’s edition was written by Bernhard M. Wiedemann, Chris Lamb and Eli Schwartz & reviewed by a bunch of Reproducible Builds folks on IRC & the mailing lists.

08 January, 2019 12:00PM

hackergotchi for Benjamin Mako Hill

Benjamin Mako Hill

A Research Symbiont!

Plush fish with attached parasitic lamprey.The award itself is a plush fish with a parasitic lamprey attached to it.

As of yesterday, I’m officially a research symbiont! A committee of health scientists saw fit to give me a Research Symbiont Award which is awarded annually to “a scientist working in any field who has shared data beyond the expectations of their field.” The award was announced at Pacific Symposium on Biocomputing and came with a trip to Hawaii (which I couldn’t take!) and the awesome plush fish with a parasitic lamprey shown in the picture.

You can read lots more about the award on the Research Symbiont Awards website and you can hear a little more about the reasons I got one in a blog post on Community Data Science Collective blog and in a short video I recorded for the award ceremony.

Sharing data in ways that are useful to others is a ton of work. It takes more time than you might imagine to prepare, polish, validate, test, and document data for others to use. I think I spent more time working with Andrés Monroy-Hernández on the Scratch Research Dataset than I have any single empirical paper! Although I spent the time doing it because I think it’s an important way to contribute to science, recognition in the form of an award—and a cute stuffed parasitic fish—is super appreciated as well!

If you know of research symbionts, you should consider nominating them next year!

08 January, 2019 03:28AM by Benjamin Mako Hill

January 07, 2019

hackergotchi for Dirk Eddelbuettel

Dirk Eddelbuettel

BH 1.69.0-1 on CRAN

Boost

The BH package provides a sizeable portion of the Boost C++ libraries as a set of template headers for use by R. It is quite popular, and frequently used together with Rcpp. The BH CRAN page shows e.g. that it is used by rstan, dplyr as well as a few other packages. The current count of reverse dependencies is at 164.

Boost releases every four months. The last release we packaged was 1.66 from February—and this BH release gets us to Boost 1.69 released just three or so weeks ago. And as blogged last month, we made a pre-release (following several reverse-depends checks) to allow three packages affected by changes in Boost to adapt. My RcppStreams package was one, and we made an update release 0.1.2 just yesterday. This BH release was also preceded by another reverse-depends check on the weekend.

Sine the 1.66 release of BH, CRAN tightened policies some more. Pragmas suppressing compiler warnings are now verboten so I had to disable a few. Expect compilations of packages using Boost, and BH, to be potentially very noisy. Consider adding flags to your local ~/.R/Makeconf and we should add them to the src/Makevars as much as we can (eg my ticket #3961 to dplyr). Collecting a few of these on a BH wiki page may not be a bad idea.

A list of our local changes follows. The diff to upstream Boost is in the repo as well.

Changes in version 1.69.0-1 (2019-01-07)

  • Upgraded to Boost 1.69.0 (plus the few local tweaks)

  • Applied the standard minimal patch with required changes, as well as the newer changeset for diagnostics pragma suppression.

  • Following a pre-release in December, maintainers of three packages affected by the 1.66 to 1.69 were contacted, and changes were prepared.

Via CRANberries, there is a diffstat report relative to the previous release.

Comments and suggestions about BH are welcome via the mailing list or the issue tracker at the GitHub repo.

This post by Dirk Eddelbuettel originated on his Thinking inside the box blog. Please report excessive re-aggregation in third-party for-profit settings.

07 January, 2019 11:26PM

hackergotchi for Shirish Agarwal

Shirish Agarwal

Book Lovers meetup and Review of the Hollywood Station (Novel)


I went to a booklover’s meetup almost after a year the last time yesterday. I went as the last time I had gone it was just 5-7 people and in small intimate groups you can talk about books or stories you like in depth rather than just share a glimpse of the books that turn you on. Although it was a bit crowdy it was fun. The only downer is that I wasn’t able to get a list of all the books people were reading or/and recommending and the group is on whatsapp 😦

Anyways, few years ago, I had chanced upon a book called ‘The New Centurions’ by Joseph Wambaugh. It was a very different kind of writing as it was written from a Police Officer’s POV while mostly we see crime and investigation of the crime from a third-person perspective. More than that, it sort of shares the emotional turmoil that a Police Officer goes through. It also shed light on how truth is also greyer to him rather than black and white as it seems to us. While I had read that book few years ago, during a book sale some months ago, unnoticed went in The Hollywood Station . While this book is also a classic in itself I was surprised to read that the New York Police Department was operating under a consent decree for a period of five years which was later made 7 years.

There were 2-3 incidents in the book though, which shook me quite a lot. While there aren’t any overtly sexual stuff in the book, there are quite a few suicide cases which the Police Officers experience. After reading any one of the experiences, I had to stay away for almost 2 weeks or a bit more just to get the images out of my head. The scenes or situations described are not detailed or graphic in nature, it’s just the way they happen that leads you to emphatic and wonder what you would have done in that situation.

I would share one of the less scarier ones. Once while on patrol, they get a message about an elderly Causcasian gentleman who apparently lives by himself on the outskirts and with his loaded gun has been firing in the air and harassing neighbors. A whole bevy of officers come in case it becomes a shoot-out. They find an old gentleman whom they are not sure of whether he is on drugs or whatever (the officers are not sure). They ask him to surrender and he seems to comply. The place is semi-dark . While surrending, one of the cops sees what seems to be a gun and calls it out. He is riddled with holes When one of the officers turns over the palm of the dead person’s hand, he finds it had a water gun which even didn’t even have water. While searching, they later find a sort of suicide note in which he laments about his life, thanks and absolves the officers of the law and asks his ashes to be spread over the bay. One of the young officers remarks that he didn’t wear the badge to be an executioner but to protect and serve.

While reading the book and even afters, I am and was pretty sure if I were to be party to any such events, it would absolutely gaurantee me becoming an inmate in a mental asylum. The only resemblances in character formation at least in mainstream Bollywood cinema would be Paresh Rawal’s breakthrough performance as the cop on his last day when Mumbai bombings happened in Mumbai Meri Jaan and more recently Jitendra Joshi’s act as Constable Katekar in Sacred Games Season 1.

I found both the books to be pretty rich in both police trainings, philosophy, limitations which a Police Department has. I am sure there is a bit of bias as Joseph Wambaugh himself was a Police Officer and served the Police Force for a period of 14 years. He also shares some case-laws which would have made for some very dry-reading if read only in context of just a case-law but becomes more important as certain context is given to you.

One of the positives of reading these books were I actively searched about consent decree, its usage being seen through various eyes, from an eye of civil-right activist, from the political establishment and of course its application in the Indian judicial system, if any in reforming the police force. As shared in a blog post last month on the surveillance order it tells of the many steps the Indian judicial still needs to take to raise awareness on such a topic.


07 January, 2019 02:09PM by shirishag75

hackergotchi for Steve McIntyre

Steve McIntyre

Rebuilding the entire Debian archive twice on arm64 hardware for fun and profit

I've posted this analysis to Debian mailing lists already, but I'm thinking it's also useful as a blog post too. I've also fixed a few typos and added a few more details that people have suggested.

This has taken a while in coming, for which I apologise. There's a lot of work involved in rebuilding the whole Debian archive, and many days spent analysing the results. You learn quite a lot, too! :-)

I promised way back before DebConf 18 last August that I'd publish the results of the rebuilds that I'd just started. Here they are, after a few false starts. I've been rebuilding the archive specifically to check if we would have any problems building our 32-bit Arm ports (armel and armhf) using 64-bit arm64 hardware. I might have found other issues too, but that was my goal.

The logs for all my builds are online at

https://www.einval.com/debian/arm/rebuild-logs/

for reference. See in particular

for automated analysis of the build logs that I've used as the basis for the stats below.

Executive summary

As far as I can see we're basically fine to use arm64 hosts for building armel and armhf, so long as those hosts include hardware support for the 32-bit A32 instruction set. As I've mentioned before, that's not a given on all arm64 machines, but there are sufficient machine types available that I think we should be fine. There are a couple of things we need to do in terms of setup - see Machine configuration below.

Methodology

I (naively) just attempted to rebuild all the source packages in unstable main, at first using pbuilder to control the build process and then later using sbuild instead. I didn't think to check on the stated architectures listed for the source packages, which was a mistake - I would do it differently if redoing this test. That will have contributed quite a large number of failures in the stats below, but I believe I have accounted for them in my analysis.

I built lots of packages, using a range of machines in a small build farm at home:
  • Macchiatobin
  • Seattle
  • Synquacer
  • Multiple Mustangs

using my local mirror for improved performance when fetching build-deps etc. I started off with a fixed list of packages that were in unstable when I started each rebuild, for the sake of simplicity. That's one reason why I have two different numbers of source packages attempted for each arch below. If packages failed due to no longer being available, I simply re-queued using the latest version in unstable at that point.

I then developed a script to scan the logs of failed builds to pick up on patterns that matched with obvious causes. Once that was done, I worked through all the failures to (a) verify those patterns, and (b) identify any other failures. I've classified many of the failures to make sense of the results. I've also scanned the Debian BTS for existing bugs matching my failed builds (and linked to them), or filed new bugs where I could not find matches.

I did not investigate fully every build failure. For example, where a package has never been built before on armel or armhf and failed here I simply noted that fact. Many of those are probably real bugs, but beyond the scope of my testing.

For reference, all my scripts and config are in git at

https://git.einval.com/cgi-bin/gitweb.cgi?p=buildd-scripts.git

armel results

Total source packages attempted 28457
Successfully built 25827
Failed 2630

Almost half of the failed builds were simply due to the lack of a single desired build dependency (nodejs:armel, 1289). There were a smattering of other notable causes:

  • 100 log(s) showing build failures (java/javadoc)
    Java build failures seem particularly opaque (to me!), and in many cases I couldn't ascertain if it was a real build problem or just maven being flaky. :-(
  • 15 log(s) showing Go 32-bit integer overflow
    Quite a number of go packages are blindly assuming sizes for 64-bit hosts. That's probably fair, but seems unfortunate.
  • 8 log(s) showing Sbuild build timeout
    I was using quite a generous timeout (12h) with sbuild, but still a very small number of packages failed. I'd earlier abandoned pbuilder for sbuild as I could not get it to behave sensibly with timeouts.
The stats that matter are the arch-specific failures for armel:
  • 13 log(s) showing Alignment problem
  • 5 log(s) showing Segmentation fault
  • 1 log showing Illegal instruction
and the new bugs I filed:
  • 3 bugs for arch misdetection
  • 8 bugs for alignment problems
  • 4 bugs for arch-specific test failures
  • 3 bugs for arch-specific misc failures

Considering the number of package builds here, I think these numbers are basically "lost in the noise". I have found so few issues that we should just go ahead. The vast majority of the failures I found were either already known in the BTS (260), unrelated to what I was looking for, or both.

See below for more details about build host configuration for armel builds.

armhf results

Total source packages attempted 28056
Successfully built 26772
Failed 1284

FTAOD: I attempted fewer package builds for armhf as we simply had a smaller number of packages when I started that rebuild. A few weeks later, it seems we had a few hundred more source packages for the armel rebuild.

The armhf rebuild showed broadly the same percentage of failures, if you take into account the nodejs difference - it exists in the armhf archive, so many hundreds more packages could build using it.

In a similar vein for notable failures:

  • 89 log(s) showing build failures (java/javadoc)
    Similar problems, I guess...
  • 15 log(s) showing Go 32-bit integer overflow
    That's the same as for armel, I'm assuming (without checking!) that they're the same packages.
  • 4 log(s) showing Sbuild build timeout
    Only 4 timeouts compared to the 8 for armel. Maybe a sign that armhf will be slightly quicker in build time, so less likely to hit a timeout? Total guesswork on small-number stats! :-)

Arch-specific failures found for armhf:

  • 11 log(s) showing Alignment problem
  • 4 log(s) showing Segmentation fault
  • 1 log(s) showing Illegal instruction

and the new bugs I filed:

  • 1 bugs for arch misdetection
  • 8 bugs for alignment problems
  • 10 bugs for arch-specific test failures
  • 3 bugs for arch-specific misc failures

Again, these small numbers tell me that we're fine. I liked to 139 existing bugs in the BTS here.

Machine configuration

To be able to support 32-bit builds on arm64 hardware, there are a few specific hardware support issues to consider.

Alignment

Our 32-bit Arm kernels are configured to fix up userspace alignment faults, which hides lazy programming at the cost of a (sometimes massive) slowdown in performance when this fixup is triggered. The arm64 kernel cannot be configured to do this - if a userspace program triggers an alignment exception, it will simply be handed a SIGBUS by the kernel. This was one of the main things I was looking for in my rebuild, common to both armel and armhf. In the end, I only found a very small number of problems.

Given that, I think we should immediately turn off the alignment fixups on our existing 32-bit Arm buildd machines. Let's flush out any more problems early, and I don't expect to see many.

To give credit here: Ubuntu have been using arm64 machines for building 32-bit Arm packages for a while now, and have already been filing bugs with patches which will have helped reduce this problem. Thanks!

Deprecated / retired instructions

In theory(!), alignment is all we should need to worry about for armhf builds, but our armel software baseline needs two additional pieces of configuration to make things work, enabling emulation for

  • SWP (low-level locking primitive, deprecated since ARMv6 AFAIK)
  • CP15 barriers (low-level barrier primitives, deprecated since ARMv7)

Again, there is quite a performance cost to enabling emulation support for these instructions but it is at least possible!

In my initial testing for rebuilding armhf only, I did not enable either of these emulations. I was then finding lots of "Illegal Instruction" crashes due to CP15 barrier usage in armhf Haskell and Mono programs. This suggests that maybe(?) the baseline architecture in these toolchains is incorrectly set to target ARMv6 rather than ARMv7. That should be fixed and all those packages rebuilt at some point.

UPDATES

  • Peter Green pointed out that ghc in Debian armhf is definitely configured for ARMv7, so maybe there is a deeper problem.
  • Edmund Grimley Evans suggests that the Haskell problem is coming from how it drives LLVM, linking to #864847 that he filed in 2017.

Bug highlights

There are a few things I found that I'd like to highlight:

  • In the glibc build, we found an arm64 kernel bug (#904385) which has since been fixed upstream thanks to Will Deacon at Arm. I've backported the fix for the 4.9-stable kernel branch, so the fix will be in our Stretch kernels soon.
  • There's something really weird happening with Vim (#917859). It FTBFS for me with an odd test failure for both armel-on-arm64 and armhf-on-arm64 using sbuild, but in a porter box chroot or directly on my hardware using debuild it works just fine. Confusing!
  • I've filed quite a number of bugs over the last few weeks. Many are generic new FTBFS reports for old packages that haven't been rebuilt in a while, and some of them look un-maintained. However, quite a few of my bugs are arch-specific ones in better-maintained packages and several have already had responses from maintainers or have already been fixed. Yay!
  • Yesterday, I filed a slew of identical-looking reports for packages using MPI and all failing tests. It seems that we have a real problem hitting openmpi-based packages across the archive at the moment (#918157 in libpmix2). I'm going to verify that on my systems shortly.

Other things to think about

Building in VMs

So far in Debian, we've tended to run our build machines using chroots on raw hardware. We have a few builders (x86, arm64) configured as VMs on larger hosts, but as far as I can see that's the exception so far. I know that OpenSUSE and Fedora are both building using VMs, and for our Arm ports now we have more powerful arm64 hosts available it's probably the way we should go here.

In testing using "linux32" chroots on native hardware, I was explicitly looking to find problems in native architecture support. In the case of alignment problems, they could be readily "fixed up / hidden" (delete as appropriate!) by building using 32-bit guest kernels with fixups enabled. If I'd found lots of those, that would be a safer way to proceed than instantly filing lots of release-critical FTBFS bugs. However, given the small number of problems found I'm not convinced it's worth worrying about.

Utilisation of hardware

Another related issue is in how we choose to slice up build machines. Many packages will build very well in parallel, and that's great if you have something like the Synquacer with many small/slow cores. However, not all our packages work so well and I found that many are still resolutely chugging through long build/test processes in single threads. I experimented a little with my config during the rebuilds and what seemed to work best for throughput was kicking off one build per 4 cores on the machines I was using. That seems to match up with what the Fedora folks are doing (thanks to hrw for the link!).

Migrating build hardware

As I mentioned earlier, to build armel and armhf sanely on arm64 hardware, we need to be using arm64 machines that include native support for the 32-bit A32 instruction set. While we have lots of those around at the moment, some newer/bigger arm64 server platforms that I've seen announced do not include it. (See an older mail from me for more details. We'll need to be careful about this going forwards and keep using (at least) some machines with A32. Maybe we'll migrate arm64-only builds onto newer/bigger A64-only machines and keep the older machines for armel/armhf if that becomes a problem?

At least for the foreseeable future, I'm not worried about losing A32 support. Arm keeps on designing and licensing ARMv8 cores that include it...

Thanks

I've spent a lot of time looking at existing FTBFS bugs over the last weeks, to compare results against what I've been seeing in my build logs. Much kudos to people who have been finding and filing those bugs ahead of me, in particular Adrian Bunk and Matthias Klose who have filed many such bugs. Also thanks to Helmut Grohne for his script to pull down a summary of FTBFS bugs from UDD - that saved many hours of effort!

Finally...

Please let me know if you think you've found a problem in what I've done, or how I've analysed the results here. I still have my machines set up for easy rebuilds, so reproducing things and testing fixes is quite easy - just ask!

07 January, 2019 12:57PM

hackergotchi for Jonathan Dowland

Jonathan Dowland

ZDBSP

A Debian package of ZDBSP is now available in the unstable distribution.

I mentioned ZDBSP when I wrote about glBSP. It's my hope to remove glBSP from the archive, leaving ZDBSP as the sole Doom nodebuilder in Debian. I might try to do this in time for the next stable release. That means probably actioning the removal before the 12th March.

If you use glBSP for any purpose, please give ZDBSP a look and make sure it works for you.

07 January, 2019 12:24PM

hackergotchi for Julien Danjou

Julien Danjou

Why You Should Care That Your SQL DDL is Transactional

Why You Should Care That Your SQL DDL is Transactional

I don't write a lot about database management. How come? I'm a software engineer, and like many of my peers, I leverage databases to store data. I should talk more about this! What made me write this today is that I've discovered that many of my peers wouldn't be able to understand the title of this post.

However, I can tell you that once you've finished reading this, you'll thank me!

DDL?

To understand what this post is about, let's start with DDL. DDL is the abbreviation of Data Definition Language. In summary, a DDL is a language that allows defining your data structure. A famous one is the SQL DDL — and that's the one I talk about here.

I'm sure you already used it if you created a relational database with CREATE TABLE foo (id INTEGER). That is a DDL statement.

In SQL, there's a lot of DDL operations you can do, such as creating a table, renaming a table, creating or removing a column, converting a column to a new type, etc.

Those DDL statements are commonly used in two cases:

  1. When creating your database' tables for the first time. You issue a bunch of CREATE TABLE statements, and your database is ready to be used.
  2. When updating your database by adding, removing or modifying tables or columns. This is typically done when upgrading your application to a new version.

The fact that our DDL is transactional or not in option 1. has often little impact in practice. It's can still be useful, where for example you could get an error because the disk is full — having the ability to roll back in this case can be a life saviour.

In our case here, we'll talk about why you need a transactional DDL when upgrading your database.

Transactional You Said?

What transactional means here? It means that we can issue those DDL statements inside a transaction.

Wait, what's a transaction? To make it simple, in a database, a transaction is a group of operations that are treated as a single coherent operation, independently of other transactions. The final operation has to be atomic, consistent, isolated and durable — therefore that ACID property you keep reading about while always wondering what it meant. The operations composing the transaction are either entirely executed, or not at all.

In our case, having the DDL being transactional means one simple thing: the ability to execute several operations (e.g., several ALTER TABLE) in a single operation, that can be either committed or rolled back.

Let's use an example. Here's a table ingredients with a name column created with:

CREATE TABLE ingredients (
  name text NOT NULL
);

In this table, there is a list of ingredients in the form of water 20 mL, flour 300 g, etc.

Now, we're upgrading our application, and we want to handle the quantity of ingredients in their columns to make it easier to query the data. Let's say we're going to handle quantity and quantity units for our ingredients. We need to add two new columns to our table schema, quantity and unit:

ALTER TABLE ingredients ADD COLUMN quantity integer NOT NULL;
ALTER TABLE ingredients ADD COLUMN unit text NOT NULL;

We also need to convert the name by splitting it into <name> <quantity> <unit> and insert this into the new columns. We can do this like that:

UPDATE ingredients SET name=split_part(name, ' ', 1), quantity=split_part(name, ' ', 2)::int, unit=split_part(name, ' ', 3);
In this example, I'm using the split_part operator from PostgreSQL to split the string.

With the UPDATE statement, the name column containing flour 300 grams now contains flour, and the columns quantity and unit respectively stores 300 and grams.

When we run our upgrade procedure consisting of those two ALTER TABLE and one UPDATE, we got our final table like this:

# SELECT * FROM ingredients;
 name  │ quantity │ unit
───────┼──────────┼──────
 flour │      300 │ g
(1 row)

Exactly what we want.

Ok, So What?

In the previous example, everything worked fine. Our 300 grams of flour string is split, converted and stored into the three different columns. However, let's think about what happens if the conversion fails because our ingredient name is foobar:

# ALTER TABLE ingredients ADD COLUMN quantity integer;
ALTER TABLE
# ALTER TABLE ingredients ADD COLUMN unit text;
ALTER TABLE
# UPDATE ingredients SET name=split_part(name, ' ', 1), quantity=split_part(name, ' ', 2)::int, unit=split_part(name, ' ', 3);
ERROR:  invalid input syntax for integer: ""

Right, so in this case our update failed because it's impossible to convert an empty string to an integer.

We're going to fix this piece of data in our database (manually or automatically, whatever) to make it work, changing foobar to something like foobar 1 kg.

Then, when we rerun the upgrade script, this is what happens:

# ALTER TABLE ingredients ADD COLUMN quantity integer NOT NULL;
ERROR:  column "quantity" of relation "ingredients" already exists

The upgrade script failed earlier — not in the UPDATE statement. It has a good reason to fail: the column quantity already exists.

Why is that? Well, when we run the upgrade procedure the first time, we did not run it inside a transaction. Every DDL statement was committed right after its execution. Therefore, the current state of our database is half-migrated: we have the new schema installed, but not the data migrated.

This sucks. This should not happen. Ever.

Why?

Some database systems (e.g., MySQL) do not support DDL running in a transaction, so you have no choice than running the three operations (ALTER, ALTER and then UPDATE) as three distinct operations: if any of those fails, there's no way to recover and get back to the initial state.

If you're using a database that supports running DDL statements inside a transaction (e.g., PostgreSQL), we can run your upgrade script like this:

postgres=# BEGIN;
postgres=# ALTER TABLE ingredients ADD COLUMN quantity integer;
ALTER TABLE
postgres=# ALTER TABLE ingredients ADD COLUMN unit text;
ALTER TABLE
postgres=# UPDATE ingredients SET name=split_part(name, ' ', 1), quantity=split_part(name, ' ', 2)::int, unit=split_part(name, ' ', 3);
ERROR:  invalid input syntax for integer: ""
postgres=# ROLLBACK;
ROLLBACK

Since the transaction failed, we ended up doing a ROLLBACK. When checking the state of the database, we can see the state did not change:

# \d ingredients;
           Table "public.ingredients"
 Column │ Type │ Collation │ Nullable │ Default
────────┼──────┼───────────┼──────────┼─────────
 name   │ text │           │ not null │

Therefore, it's possible to fix our database content and rerun the migration procedure without being in a half-migrated state.

A Database That Lies

When you're giving data to a database, you're trusting it. It'd be awful if it were lying to you, right? Check this out:

mysql> CREATE TABLE ingredients (name text NOT NULL);
Query OK, 0 rows affected (0.03 sec)

mysql> BEGIN;
Query OK, 0 rows affected (0.00 sec)

mysql> ALTER TABLE ingredients ADD COLUMN quantity integer;
Query OK, 0 rows affected (0.05 sec)
Records: 0  Duplicates: 0  Warnings: 0

mysql> ALTER TABLE ingredients ADD COLUMN unit text;
Query OK, 0 rows affected (0.01 sec)
Records: 0  Duplicates: 0  Warnings: 0

mysql> ROLLBACK;
Query OK, 0 rows affected (0.00 sec)

mysql> DESC ingredients;
+----------+---------+------+-----+---------+-------+
| Field    | Type    | Null | Key | Default | Extra |
+----------+---------+------+-----+---------+-------+
| name     | text    | NO   |     | NULL    |       |
| quantity | int(11) | YES  |     | NULL    |       |
| unit     | text    | YES  |     | NULL    |       |
+----------+---------+------+-----+---------+-------+
3 rows in set (0.00 sec)

In the output above, you can see that we issued two DDL statements inside a transaction and that we then rolled back that transaction. MySQL did not output any error at any time, making us think that it did not alter our table. However, when checking the schema of the database, we can see that nothing has been rolled back. Not only MySQL does not support transactional DDL, but it also fails you entirely and lie about what it's doing.

How Important is That?

Transactional DDL is a feature that is often ignored by software engineers, whereas it's a key feature for managing your database life cycle.

I'm writing this post today because I've been hit by this multiple times over the last years. OpenStack made the choice years ago to go with MySQL, and in consequences, every database upgrade script that fails in the middle of the procedure leave the database is an inconsistence state. In that case, it means that either:

  1. The operator have to determine where the upgrade script stopped, roll back the upgrade by itself, fix the failure, and rerun the upgrade procedure.
  2. The developer must anticipate every case of potential upgrade failure, write a roll back procedure for each of this case by and test every one of those cases.
  3. Use a database system that handles transactional DDL.

No need to tell you that option 3. is the best, option 2. is barely possible to implement and option 1. is what reality looks like. In Gnocchi, we picked option 3. by recommending operators to use PostgreSQL.

Next time you use a database, think carefully about what your upgrade procedure will look like!

07 January, 2019 11:00AM by Julien Danjou

hackergotchi for Daniel Lange

Daniel Lange

Xfce 4.12 not suspending on laptop-lid close

Xfce 4.12 as default in Ubuntu/Xubuntu 18.04 LTS did not suspend a laptop after closing the lid. In fact running xfce4-power-manager --quit ; xfce4-power-manager --no-daemon --debug showed that xfce4 wasn't seeing a laptop lid close event at all.

To the contrary acpi_listen nicely finds button/lid LID close and button/lid LID open events when folding the screen and opening it up again.

As so often the wonderful docs / community of Arch Linux to the rescue. This forum thread from 2015 received the correct answer in 2017:

Xfce4 basically recognizes systemd and thus disables its built-in power-management options for handling these "button events" (but doesn't tell you so in the config UI for power-manager). Systemd is configured to handle these events by default (/etc/systemd/logind.conf has HandleLidSwitch=suspend but for unknown reasons decides not to honor that).

So best is to teach Xfce4 to handle the events again as in pre-systemd times:

xfconf-query -c xfce4-power-manager -p /xfce4-power-manager/logind-handle-lid-switch -s false

Now the UI options will work again as intended and the laptop suspends on lid close and resumes on lid open.

Update:

07.01.19: Changed XFCE -> Xfce as per Corsac's suggestion in the comments below. Thank you!

Background info:

The name "XFCE" was originally an acronym for "XForms Common Environment", but since that time it has been rewritten twice and no longer uses the XForms toolkit. The name survived, but it is no longer capitalized as "XFCE", but rather as "Xfce". The developers' current stance is that the initialism no longer stands for anything specific. After noting this, the FAQ on the Xfce Wiki comments "(suggestion: X Freakin' Cool Environment)".

(quoted from Wikipedia's Xfce article also found in the Xfce docs FAQ).

07 January, 2019 09:37AM by Daniel Lange

Dima Kogan

Where are all these crashes coming from?

Check this out!

wrecks.png

(click for an interactive-but-very-slow map thing).

So let's say you're walking around in the mountains, and BAM! You run into a crashed airplane (pretend this happens all the time). You wonder "what is this?" "When did it go down?" "How?" Usually it doesn't happen this way for me (rather like this), but in any case this is all useful.

It turns out that most (all?) such incidents are investigated by the NTSB. And they then write a report. And then somebody puts all the reports into a database and makes them available on the internet. How helpful! For my purposes, I mostly care about the location each impact site. Some reports actually have a lat/lon pair, while others have a range/bearing from some "observing" airport. Many reports don't have either, but there isn't a lot I can do about that.

In any case, I just did some typing, and produced the map linked above:

https://caltopo.com/m/N13E

This shows all the incidents after 1982 that have a location listed in an NTSB report. The locations are represented by a red circle and/or a range/bearing area, depending on what the report contains. Each object contains a link to the report itself. The data isn't very accurate it looks like (sometimes not even close to right), but the text of the reports often has more specific information. I have all the data in the US, but the above link is just the general area of the San Gabriel mountains; the web interface is way too slow to show more. Generating maps for other areas is trivial: the code, docs, and full final dataset appear here.

And I just learned that in 2016 a Boeing 777 cleared Mt. Wilson by 1000 ft, not counting the towers. Some people live dangerously.

07 January, 2019 07:56AM by Dima Kogan

hackergotchi for Dirk Eddelbuettel

Dirk Eddelbuettel

RcppStreams 0.1.2

Streamulus

A maintenance release of RcppStreams arrived on CRAN this afternoon. RcppStreams brings the excellent Streamulus C++ template library for event stream processing to R.

Streamulus, written by Irit Katriel, uses very clever template meta-programming (via Boost Fusion) to implement an embedded domain-specific event language created specifically for event stream processing.

This release provides a minor update, motivated by the upcoming BH release 1.69 for which we made a pre-release – and RcppStreams is one of three packages identified by that pre-release as needed minor accomodations for the new Boost release. We also updated packaging to current CRAN standards, but made no other changes.

The NEWS file entries follows below:

Changes in version 0.1.2 (2019-01-05)

  • Added symbol registration

  • Converted a few files from CRLF to CR [CRAN checks]

  • Added a few final newlines [CRAN checks]

  • Disabled one optional output to enable BH 1.69 builds

Courtesy of CRANberries, there is also a copy of the DESCRIPTION file for this initial release. More detailed information is on the RcppStreams page page and of course on the Streamulus page.

This post by Dirk Eddelbuettel originated on his Thinking inside the box blog. Please report excessive re-aggregation in third-party for-profit settings.

07 January, 2019 02:12AM

Hideki Yamane

Call for tester: to brave unstable user, please check Secure Boot feature

As described in Debian Wiki, for check test the "secure boot" feature on a real hardware, we need more testers.

So, brave "sid" users, please test it with your hardware and add its result to Wiki page.


As always should you or any of your hardware be trouble or killed, I will disavow all of knowledge or your action. Good luck!

07 January, 2019 02:00AM by Hideki Yamane (noreply@blogger.com)

January 06, 2019

Russ Allbery

James Nicoll's 100 SF/F Books

On December 27th, Tor.com published an article by James Nicoll entitled 100 SF/F Books You Should Consider Reading in the New Year. Since then, various people I know have been posting the list annotated with the books they've read and liked. That seemed fun, so I thought I'd join in.

I've read 28 out of the 99 books (and shorter fiction) listed. (One entry was for a musical work; I appreciate James's inclusiveness, but I appreciate my orderly classification system more, and it's not a book.) I'm a bit surprised that's not higher, since James is one of my primary sources of book reviews. Lots of ones I've not read are in various to-read piles, though.

Among those that I have read, average rating is 7.43, which shows why I pay attention to James's reviews. Interestingly, though, that's slightly below the average rating for the BookRiot SF/F by Female Authors list, at 7.48. I thought perhaps the BookRiot list had more "obvious" good books and James was trying to be more obscure, but looking it over, that doesn't seem to be the case. The BookRiot list just has marginally more books I liked, at least of the ones I've read so far (sometimes via choosing a different book by the same author).

Anyway, here's my version of the list for anyone who's interested. I recommend reading the original article for James's short descriptions, covers, and purchase links.

06 January, 2019 09:49PM

Thorsten Alteholz

My Debian Activities in December 2018

FTP master

This month I accepted 276 packages, which is bit more than two months before. On the other side I rejected 34 uploads, which is the same as last month. The overall number of packages that got accepted this month was 442.

Debian LTS

This was my fifty fourth month that I did some work for the Debian LTS initiative, started by Raphael Hertzog at Freexian.

This month my all in all workload has been 30h. This was the first month were I did not upload any package, but only prepared a wireshark package for testing. It is available at people.d.o and contains patches for 31 CVEs. As lots of dissectors are affected, I would be very glad if others could have a look at it.

I also started to work on CVEs of exiv2 and krb5.

Last but not least I did some days of frontdesk duties.

Debian ELTS

This month was the seventh ELTS month.

During my allocated time I prepared a wireshark package for testing. It is also available at people.d.o and contains patches for 26 CVEs. As with the LTS version I would be very glad if others could have a look at it.

I also started to work on CVEs of krb5.

As like in LTS, I also did some days of frontdesk duties.

Other stuff

I improved packaging of …

I uploaded new upstream versions of …

Thanks to all the people that tested the meep and mpb packages and filed bugs. I am now trying to clean up the mess :-). At least my packages no longer depend on guile-2.0!

I didn’t have to sponsor package for Nicolas Mora anymore. He finally became a DM and I gave him upload rights for his packages. So now he can take care of his stuff on his own :-).
The next sponsee is Tommi Höynälänmaa. Initially he wanted to have theme-d and theme-d-gnome in Debian. Due to dependencies being orphaned and RC buggy he now has to take care of …

He is really doing a nice job.

The Debian Med Advent Calendar was again really successful this year. There was no new record, but with 81, a fair number of bugs could have been closed.

year number of bugs closed
2011 63
2012 28
2013 73
2014 5
2015 150
2016 95
2017 105
2018 81

Well done everybody who participated, especially Andreas Tille who closed by far the most bugs!

06 January, 2019 06:09PM by alteholz

Russell Coker

Photograph Your Work

One thing I should have learned before (but didn’t) and hope I’ve learned now is to photograph sysadmin work.

If you work as a sysadmin you probably have a good phone, if you are going to run ssh from a phone or use a phone to read docs while in a server room with connectivity problems you need a phone with a good screen. You will also want a phone that has current security support. Such a phone will have a reasonable amount of storage space, I doubt that you can get a phone with less than 32G of storage that has a decent screen and Android security support. Admittedly Apple has longer security support for iPhones than Google does for Nexus/Pixel phones so it might be possible to get an older iPhone with a decent screen and hardly any space (but that’s not the point here).

If you have 32G of storage on your phone then there’s no real possibility of using up your storage space by photographing a day’s work. You could probably take an unreasonable number of photos of a week’s work as well as a few videos and not use up much of that.

The first time I needed photos recently was about 9 months ago when I was replacing some network gear (new DSL modem and switch for a client). The network sockets in the rack weren’t labelled and I found it unreasonably difficult to discover where everything was (the tangle of cables made tracking them impossible). What I should have done is to photograph the cables before I started and then I would have known where to connect everything. A 12MP camera allows zooming in on photos to get details, so a couple of quick shots of that rack would have saved me a lot of time – and in the case where everything goes as planned taking a couple of photos isn’t going to delay things.

Last night there was a power failure in a server room that hosts a couple of my machines. When power came back on the air-conditioner didn’t start up and the end result was a server with one of it’s disks totally dead (maybe due to heat, maybe power failures, maybe it just wore out). For unknown reasons BTRFS wouldn’t allow me to replace the disk in the RAID-1 array so I needed to copy the data to a new disk and create a new mirror (taking a lot of my time and also giving downtime). While I was working on this the filesystem would only mount read-only so no records of the kernel errors were stored. If I had taken photos of the screen I would have records of this which might allow me to reproduce the problem and file a bug report. Now I have no records, I can’t reproduce it, and I have a risk that next time a disk dies in a BTRFS RAID-1 I’ll have the same problem. Also presumably random people all over the world will suffer needless pain because of this while lacking the skills to file a good bug report because I didn’t make good enough records to reproduce it.

Hopefully next time I’m in a situation like this I’ll think to take some photos instead of just rebooting and wiping the evidence.

As an aside I’ve been finding my phone camera useful for zooming in on serial numbers that I can’t read otherwise. I’ve got new glasses on order that will hopefully address this, but in the mean time it’s the only way I can read the fine print. Another good use of a phone camera is recording error messages that scroll past too quickly to read and aren’t logged. Some phones support slow motion video capture (up to 120fps or more) and even for phones that don’t you can use slow play (my favourite Android video player MX Player works well at 5% normal speed) to capture most messages that are too quick to read.

06 January, 2019 11:58AM by etbe

January 05, 2019

Ingo Juergensmann

Too much disk IO on sda in RAID10 setup

I have a RAID10 setup with 4x 2 TB WD Red disks in my server. Although the setup works fairly well and has enough throughput there is one strange issue with that setup: /dev/sda has more utilzation/load than the other 3 disks. See the blue line in the following graph which represents utilization by week for sda:

As you can see from the graphs and from the numbers below sda has a 2-3 times higher utilization than sdb, sdc or sdd, especially when looking at disk latency graph by Munin:

Although the graphs are a little confusing you can easily spot the big difference from the below values. And it's not only Munin showing this strange behaviour of sda, but also atop: 

Here you see that sda is 94% busy although the writes to the disks are a little bit lower than on the other disks. The screenshot of atop was before I moved MySQL/MariaDB to my NVMe disk 4 weeks ago. But you can also spot that sda is slowing down the RAID10.

So the big question is: why is utilization and latency of sda that high? it's the same disk model as the other disks. All disks are connected to a Supermicro X9SRi-F mainboard. The first two SATA ports are 6 Gbit/s, the other 4 ports are 3 Gbit/s ports:

sda sdb sdc sdd
=== START OF INFORMATION SECTION ===
Model Family:     Western Digital Red
Device Model:     WDC WD20EFRX-68AX9N0
Serial Number:    WD-WMC301414725
LU WWN Device Id: 5 0014ee 65887fe2c
Firmware Version: 80.00A80
User Capacity:    2,000,398,934,016 bytes [2.00 TB]
Sector Sizes:     512 bytes logical, 4096 bytes physical
Device is:        In smartctl database [for details use: -P show]
ATA Version is:   ACS-2 (minor revision not indicated)
SATA Version is:  SATA 3.0, 6.0 Gb/s (current: 6.0 Gb/s)
Local Time is:    Sat Jan  5 17:24:58 2019 CET
SMART support is: Available - device has SMART capability.
SMART support is: Enabled
=== START OF INFORMATION SECTION ===
Model Family:     Western Digital Red
Device Model:     WDC WD20EFRX-68AX9N0
Serial Number:    WD-WMC301414372
LU WWN Device Id: 5 0014ee 65887f374
Firmware Version: 80.00A80
User Capacity:    2,000,398,934,016 bytes [2.00 TB]
Sector Sizes:     512 bytes logical, 4096 bytes physical
Device is:        In smartctl database [for details use: -P show]
ATA Version is:   ACS-2 (minor revision not indicated)
SATA Version is:  SATA 3.0, 6.0 Gb/s (current: 6.0 Gb/s)
Local Time is:    Sat Jan  5 17:27:01 2019 CET
SMART support is: Available - device has SMART capability.
SMART support is: Enabled
=== START OF INFORMATION SECTION ===
Model Family:     Western Digital Red
Device Model:     WDC WD20EFRX-68AX9N0
Serial Number:    WD-WMC301411045
LU WWN Device Id: 5 0014ee 603329112
Firmware Version: 80.00A80
User Capacity:    2,000,398,934,016 bytes [2.00 TB]
Sector Sizes:     512 bytes logical, 4096 bytes physical
Device is:        In smartctl database [for details use: -P show]
ATA Version is:   ACS-2 (minor revision not indicated)
SATA Version is:  SATA 3.0, 6.0 Gb/s (current: 3.0 Gb/s)
Local Time is:    Sat Jan  5 17:30:15 2019 CET
SMART support is: Available - device has SMART capability.
SMART support is: Enabled
=== START OF INFORMATION SECTION ===
Model Family:     Western Digital Red
Device Model:     WDC WD20EFRX-68AX9N0
Serial Number:    WD-WMC301391344
LU WWN Device Id: 5 0014ee 60332a8aa
Firmware Version: 80.00A80
User Capacity:    2,000,398,934,016 bytes [2.00 TB]
Sector Sizes:     512 bytes logical, 4096 bytes physical
Device is:        In smartctl database [for details use: -P show]
ATA Version is:   ACS-2 (minor revision not indicated)
SATA Version is:  SATA 3.0, 6.0 Gb/s (current: 3.0 Gb/s)
Local Time is:    Sat Jan  5 17:30:26 2019 CET
SMART support is: Available - device has SMART capability.
SMART support is: Enabled

There is even the same firmware version of the disks. Usually I would have expected a slower disk IO from the 3 Gbit/s disks (sdc & sdd), but not from sda. All disks are configured in BIOS to use AHCI.

I cannot explain why sda has higher latency and more utilization than the other disks. Any ideas are welcome and appreciated. You can also reach me in the Fediverse (Friendica: https://nerdica.net/profile/ij & Mastodon: ">https://nerdculture.de/) or via XMPP at ij@nerdica.net.

Kategorie: 
 

05 January, 2019 04:37PM by ij

Julian Andres Klode

Setting up an email server, part 1: The Forwarder

This week, I’ve been working on rolling out mail services on my server. I started working on a mail server setup at the end of November, while the server was not yet in use, but only for about two days, and then let it rest.

As my old shared hosting account expired on January 1, I had to move mail forwarding duties over to the new server. Yes forwarding - I do plan to move hosting the actual email too, but at the moment it’s “just” forwarding to gmail.

The Software

As you might know from the web server story, my server runs on Ubuntu 18.04. I set up a mail server on this system using

  • Postfix for SMTP duties (warning, they oddly do not have an https page)
  • rspamd for spam filtering, and signing DKIM / ARC
  • bind9 for DNS resolving
  • postsrsd for SRS

You might wonder why bind9 is in there. It turns out that DNS blacklists used by spam filters block the caching DNS servers you usually use, so you have to use your own recursive DNS server. Ubuntu offers you the choice between bind9 and dnsmasq in main, and it seems like bind9 is more appropriate here than dnsmasq.

Setting up postfix

Most of the postfix configuration is fairly standard. So, let’s skip TLS configuration and outbound SMTP setups (this is email, and while they support TLS, it’s all optional, so let’s not bother that much here).

The most important part is restrictions in main.cf.

First of all, relay restrictions prevent us from relaying emails to weird domains:

# Relay Restrictions
smtpd_relay_restrictions = reject_non_fqdn_recipient reject_unknown_recipient_domain permit_mynetworks permit_sasl_authenticated defer_unauth_destination

We also only accept mails from hosts that know their own full qualified name:

# Helo restrictions (hosts not having a proper fqdn)
smtpd_helo_required = yes
smtpd_helo_restrictions = permit_mynetworks reject_invalid_helo_hostname reject_non_fqdn_helo_hostname reject_unknown_helo_hostname

We also don’t like clients (other servers) that send data too early, or have an unknown hostname:

smtpd_data_restrictions = reject_unauth_pipelining
smtpd_client_restrictions = permit_mynetworks reject_unknown_client_hostname

I also set up a custom apparmor profile that’s pretty lose, I plan to migrate to the one in the apparmor git eventually but it needs more testing and some cleanup.

Sender rewriting scheme

For SRS using postsrsd, we define the SRS_DOMAIN in /etc/default/postsrsd and then configure postfix to talk to it:

# Handle SRS for forwarding
recipient_canonical_maps = tcp:localhost:10002
recipient_canonical_classes= envelope_recipient,header_recipient

sender_canonical_maps = tcp:localhost:10001
sender_canonical_classes = envelope_sender

This has a minor issue that it also rewrites the Return-Path when it delivers emails locally, but as I am only forwarding, I’m worrying about that later.

rspamd basics

rspamd is a great spam filtering system. It uses a small core written in C and a bunch of Lua plugins, such as:

  • IP score, which keeps track of how good a specific server was in the past
  • Replies, which can check whether an email is a reply to another one
  • DNS blacklisting
  • DKIM and ARC validation and signing
  • DMARC validation
  • SPF validation

It also has a nice web UI:

rspamd web ui status

rspamd web ui status

rspamd web ui investigating a spam message

rspamd web ui investigating a spam message

Setting up rspamd is quite easy. You basically just drop a bunch of configuration overrides into /etc/rspamd/local.d and you’re done. Heck, it mostly works out of the box. There’s a fancy rspamadm configwizard too.

What you do want for rspamd is a redis server. redis is needed in many places, such as rate limiting, greylisting, dmarc, reply tracking, ip scoring, neural networks.

I made a few changes to the defaults:

  • I enabled subject rewriting instead of adding headers, so spam mail subjects get [SPAM] prepended, in local.d/actions.conf:

    reject = 15;
    rewrite_subject = 6;
    add_header = 6;
    greylist = 4;
    subject = "[SPAM] %s";
    
  • I set autolearn = true; in local.d/classifier-bayes.conf to make it learn that an email that has a score of at least 15 (those that are rejected) is spam, and emails with negative scores are ham.

  • I set extended_spam_headers = true; in local.d/milter_headers.conf to get a report from rspamd in the header seeing the score and how the score came to be.

ARC setup

ARC is the ‘Authenticated Received Chain’ and is currently a DMARC working group work item. It allows forwarders / mailing lists to authenticate their forwarding of the emails and the checks they have performed.

rspamd is capable of validating and signing emails with ARC, but I’m not sure how much influence ARC has on gmail at the moment, for example.

There are three parts to setting up ARC:

  1. Generate a DKIM key pair (use rspamadm dkim_keygen)
  2. Setup rspamd to sign incoming emails using the private key
  3. Add a DKIM TXT record for the public key. rspamadm helpfully tells you how it looks like.

For step two, what we need to do is configure local.d/arc.conf. You can basically use the example configuration from the rspamd page, the key point for signing incoming email is to specifiy sign_incoming = true; and use_domain_sign_inbound = "recipient"; (FWIW, none of these options are documented, they are fairly new, and nobody updated the documentation for them).

My configuration looks like this at the moment:

# If false, messages with empty envelope from are not signed
allow_envfrom_empty = true;
# If true, envelope/header domain mismatch is ignored
allow_hdrfrom_mismatch = true;
# If true, multiple from headers are allowed (but only first is used)
allow_hdrfrom_multiple = false;
# If true, username does not need to contain matching domain
allow_username_mismatch = false;
# If false, messages from authenticated users are not selected for signing
auth_only = true;
# Default path to key, can include '$domain' and '$selector' variables
path = "${DBDIR}/arc/$domain.$selector.key";
# Default selector to use
selector = "arc";
# If false, messages from local networks are not selected for signing
sign_local = true;
#
sign_inbound = true;
# Symbol to add when message is signed
symbol_signed = "ARC_SIGNED";
# Whether to fallback to global config
try_fallback = true;
# Domain to use for ARC signing: can be "header" or "envelope"
use_domain = "header";
use_domain_sign_inbound = "recipient";
# Whether to normalise domains to eSLD
use_esld = true;
# Whether to get keys from Redis
use_redis = false;
# Hash for ARC keys in Redis
key_prefix = "ARC_KEYS";

This would also sign any outgoing email, but I’m not sure that’s necessary - my understanding is that we only care about ARC when forwarding/receiving incoming emails, not when sending them (at least that’s what gmail does).

Other Issues

There are few other things to keep in mind when running your own mail server. I probably don’t know them all yet, but here we go:

  • You must have a fully qualified hostname resolving to a public IP address
  • Your public IP address must resolve back to the fully qualified host name
  • Again, you should run a recursive DNS resolver so your DNS blacklists work (thanks waldi for pointing that out)
  • Setup an SPF record. Mine looks like this:

    jak-linux.org. 3600 IN TXT "v=spf1 +mx ~all"

    this states that all my mail servers may send email, but others probably should not (a softfail). Not having an SPF record can punish you; for example, rspamd gives missing SPF and DKIM a score of 1.

  • All of that software is sandboxed using AppArmor. Makes you question its security a bit less!

Source code, outlook

As always, you can find the Ansible roles on GitHub. Feel free to point out bugs! 😉

In the next installment of this series, we will be looking at setting up Dovecot, and configuring DKIM. We probably also want to figure out how to run notmuch on the server, keep messages in matching maildirs, and have my laptop synchronize the maildir and notmuch state with the server. Ugh, sounds like a lot of work.

05 January, 2019 03:34PM

Niels Thykier

Improvements to apt-file since stretch

The list of changes for apt-file in buster is rather short, but I would still like to mention a few of them in this post.

New list-indices command:

In stretch, apt-file migrated to use apt’s new acquire system and that meant a lot of changes to apt-file’s interface.  Among other, you could no longer search for source packages via the “-a source” but had to use “-Idsc” instead.  While the manpage has some runes about these names, but not everyone finds them helpful.

To do a bit better, we have included a new list-indices command in apt-file that enables you see exactly which indices that apt-file recognises and whether we can find any data in them.  This looks like the following:

$ apt-file list-indices
+-----------------+-----------------------------+-----------------+
| Index Name (-I) | DefaultEnabled (Apt config) | Index Status    |
+-----------------+-----------------------------+-----------------+
| deb             | <unset>                     | Ok              |
| udeb            | false                       | Empty (code: 4) |
| dsc             | false                       | Empty (code: 4) |
+-----------------+-----------------------------+-----------------+

At the moment, the command outputs three columns:

  • “Index Name (-I)” – This is the value you pass to -I/–index-names to search in that index.
  • “DefaultEnabled (Apt config)” – This is the value apt-file found while asking apt-config about the index configuration.  As you can disable fetching of Contents files on a “per sources.list” basis, so the above is only the default for URIs you fetch from.
  • “Index Status” – This tells you whether apt-file can actually search for anything in that index.  It comes in 3 basic status: Ok (i.e. some searchable data found), Empty (no Contents files found) and Error (an unexpected error happened while checking the status but it did not stop apt-file from generating the table).

Note: It is intended to be human-readable and is not (intended to be) machine readable.  If you need this data in a script or for automation purposes, please use apt-config plus apt-get indextargets directly and extra the data you want via those tools.  The output format of this command may change without notice (if another format is found better suited, etc.).

Status feedback (terminal-only):

The new version of apt-file also provides some feedback to your terminal while you wait for your search.  It comes in the form of a single line showing you what apt-file is doing. Below is the output from apt-file (being interrupted) at various stages.

$ apt-file show lintian
Searching for contents of the packages ...^C
$ apt-file show lintian
Searching, found 370 results so far ...^C

(note: The number of “results” here is prior to de-duplicating the results.  You may see a much larger number here than in the final result)

The output is disabled when you redirect stdout, so most scripts are unaffected by this change:

$ apt-file show lintian | head -n1
lintian: /etc/lintianrc

Faster by default:

Historically, Contents files had a human-readable header that described what the files contented, how to read the file and column headers.  Unfortunately, they made apt-file’s life harder as we had to reliably ignore these headers without losing actual data from them.  However, both dak and reprepro has stopped adding the headers, so apt-file can now skip the header filtering by default.

If you still need to search for files in Contents files with the headers, you can re-enable the check (with related performance hit) by setting apt-file::Parser::Check-For-Description-Header to true in your configuration.  E.g.

$ time apt-file search nothing-else-matches
                                          
real    0m1.853s
[...]
$ time apt-file search -o apt-file::Parser::Check-For-Description-Header=true nothing-else-matches
                                          
real    0m7.875s

Actual performance benefit largely depends on cache size and (apparently) also what you search for.  🙂

05 January, 2019 09:03AM by Niels Thykier