October 21, 2018

hackergotchi for Dirk Eddelbuettel

Dirk Eddelbuettel

RApiDatetime 0.0.4: Updates and Extensions

The first update in a little while brings us release 0.0.4 of RApiDatetime which got onto CRAN this morning via the lovely automated sequence of submission, pretest-recheck and pretest-publish.

RApiDatetime provides seven entry points for C-level functions of the R API for Date and Datetime calculations. The functions asPOSIXlt and asPOSIXct convert between long and compact datetime representation, formatPOSIXlt and Rstrptime convert to and from character strings, and POSIXlt2D and D2POSIXlt convert between Date and POSIXlt datetime. This releases brings asDatePOSIXct as a seventh courtesy of Josh Ulrich. All these functions are all fairly useful, but not one of them was previously exported by R for C-level use by other packages. Which is silly as this is generally extremely carefully written and tested code.

I also updated the exported base R code to what is in R 3.5.1 right now, added a few #nocov declarations (not all which are reflected at the codecov page yet, and added a dependency badge at the GitHub repo.

Changes in RApiDatetime version 0.0.4 (2018-10-21)

  • New function asDatePOSIXct (Josh Ulrich in #2)

  • Synchronized with upstream code in base R (Dirk in #3)

Courtesy of CRANberries, there is a comparison to the previous release. More information is on the rapidatetime page.

For questions or comments please use the issue tracker off the GitHub repo.

This post by Dirk Eddelbuettel originated on his Thinking inside the box blog. Please report excessive re-aggregation in third-party for-profit settings.

21 October, 2018 07:12PM

hackergotchi for Vincent Bernat

Vincent Bernat

BGP LLGR: robust and reactive BGP sessions

On a BGP-routed network with multiple redundant paths, we seek to achieve two goals concerning reliability:

  1. A failure on a path should quickly bring down the related BGP sessions. A common expectation is to recover in less than a second by diverting the traffic to the remaining paths.

  2. As long as a path is operational, the related BGP sessions should stay up, even under duress.

Detecting failures fast: BFD⚓︎

To quickly detect a failure, BGP can be associated with BFD, a protocol to detect faults in bidirectional paths,1 defined in RFC 5880 and RFC 5882. BFD can use very low timers, like 100 ms.

However, when BFD runs in a process on top of a generic kernel,2 notably when running BGP on the host, it is not unexpected to loose a few BFD packets on adverse conditions: the daemon handling the BFD sessions may not get enough CPU to answer in a timely manner. In this scenario, it is not unlikely for all the BGP sessions to go down at the same time, creating an outage, as depicted in the last case in the diagram below.

BGP and failed sessions
Examples of failures on a network using BGP as the underlying routing protocol. A link failure is detected by BFD and the failed path is removed from the ECMP route. However, when high CPU usage on the bottom router prevents BFD packets to be processed timely, all paths are removed.

So far, we have two contradicting roads:

  • lower the BFD timers to quickly detect a failure along the path, or
  • raise the BFD timers to ensure BGP sessions remain operational.

Fix false positives: BGP LLGR⚓︎

Long-lived BGP Graceful Restart is a new BGP capability to retain stale routes for a longer period after a session failure but treating them as least-preferred. It also defines a well-known community to share this information with other routers. It is defined in the Internet-Draft draft-uttaro-idr-bgp-persistence-04 and several implementations already exist:

  • Juniper JunOS (since 15.1, see the documentation),
  • Cisco IOS XR (unfortunately only for VPN and FlowSpec families),
  • BIRD (since 1.6.5 and 2.0.3, both yet to be released, sponsored by Exoscale), and
  • GoBGP (since 1.33).

The following illustration shows what happens during two failure scenarios. Like without LLGR, in ❷, a link failure is detected by BFD and the failed path is removed from the route as two other paths remain with a higher preference. A couple of minutes later, the faulty path has its stale timer expired and will not be used anymore. Shortly after, in ❸, the bottom router experiences high CPU usage, preventing BFD packets to be processed timely. The BGP sessions are closed and the remaining paths become stale but as there is no better path left, they are still used until the LLGR timer expires. In the meantime, we expect the BGP sessions to resume.

BGP with LLGR
Examples of failures on a network using BGP as the underlying routing protocol with LLGR enabled.

From the point of view of the top router, the first failed path was considered as stale because the BGP session with R1 was down. However, during the second failure, the two remaining paths were considered as stale because they were tagged with the well-known community LLGR_STALE (65535:6) by R2 and R3.

Another interesting point of BGP LLGR is the ability to restart the BGP daemon without any impact—as long as all paths keep a steady state shortly before and during restart. This is quite interesting when running BGP on the host.3

BIRD⚓︎

Let’s see how to configure BIRD 1.6. As BGP LLGR is built on top of the regular BGP graceful restart (BGP GR) capability, we need to enable both. The timer for BGP LLGR starts after the timer for BGP GR. During a regular graceful restart, routes are kept with the same preference. Therefore it is important to set this timer to 0.

template bgp BGP_LLGR {
  bfd graceful;
  graceful restart yes;
  graceful restart time 0;
  long lived graceful restart yes;
  long lived stale time 120;
}

When a problem appears on the path, the BGP session goes down and the LLGR timer starts:

$ birdc show protocol R1_1 all
name     proto    table    state  since       info
R1_1     BGP      master   start  11:20:17    Connect
  Preference:     100
  Input filter:   ACCEPT
  Output filter:  ACCEPT
  Routes:         1 imported, 0 exported, 0 preferred
  Route change stats:     received   rejected   filtered    ignored   accepted
    Import updates:              2          0          0          0          4
    Import withdraws:            0          0        ---          0          0
    Export updates:             12         10          0        ---          2
    Export withdraws:            1        ---        ---        ---          0
  BGP state:          Connect
    Neighbor address: 2001:db8:104::1
    Neighbor AS:      65000
    Neighbor graceful restart active
    LL stale timer:   112/-

The related paths are marked as stale (as reported by the s in 100s) and tagged with the well-known community LLGR_STALE:

$ birdc show route 2001:db8:10::1/128 all
2001:db8:10::1/128 via 2001:db8:204::1 on eth0.204 [R1_2 10:35:01] * (100) [i]
        Type: BGP unicast univ
        BGP.origin: IGP
        BGP.as_path:
        BGP.next_hop: 2001:db8:204::1 fe80::5254:3300:cc00:5
        BGP.local_pref: 100
                   via 2001:db8:104::1 on eth0.104 [R1_1 11:22:51] (100s) [i]
        Type: BGP unicast univ
        BGP.origin: IGP
        BGP.as_path:
        BGP.next_hop: 2001:db8:104::1 fe80::5254:3300:6800:5
        BGP.local_pref: 100
        BGP.community: (65535,6)

We are left with only one path for the route in the kernel:

$ ip route show 2001:db8:10::1
2001:db8:10::1 via 2001:db8:204::1 dev eth0.204 proto bird metric 1024 pref medium

To upgrade BIRD without impact, it needs to run with the -R flag and the graceful restart yes directive should be present in the kernel protocols. Then, before upgrade, stop it using SIGKILL instead of SIGTERM to avoid a clean close of the BGP sessions.

Juniper JunOS⚓︎

With JunOS, we only have to enable BGP LLGR for each family—assuming BFD is already configured:

# Enable BGP LLGR
edit protocols bgp group peers family inet6 unicast
set graceful-restart long-lived restarter stale-time 2m

Once a path is failing, the associated BGP session goes down and the BGP LLGR timer starts:

> show bgp neighbor 2001:db8:104::4
Peer: 2001:db8:104::4+179 AS 65000 Local: 2001:db8:104::1+57667 AS 65000
  Group: peers                 Routing-Instance: master
  Forwarding routing-instance: master
  Type: Internal    State: Connect        Flags: <>
  Last State: Active        Last Event: ConnectRetry
  Last Error: None
  Export: [ LOOPBACK NOTHING ]
  Options: <Preference HoldTime Ttl AddressFamily Multipath Refresh>
  Options: <BfdEnabled LLGR>
  Address families configured: inet6-unicast
  Holdtime: 6 Preference: 170
  NLRI inet6-unicast:
  Number of flaps: 2
  Last flap event: Restart
  Time until long-lived stale routes deleted: inet6-unicast 00:01:05
  Table inet6.0 Bit: 20000
    RIB State: BGP restart is complete
    Send state: not advertising
    Active prefixes:              0
    Received prefixes:            1
    Accepted prefixes:            1
    Suppressed due to damping:    0
    LLGR-stale prefixes:          1

The associated path is marked as stale and is therefore inactive as there are better paths available:

> show route 2001:db8:10::4 extensive
[…]
BGP    Preference: 170/-101
       Source: 2001:db8:104::4
       Next hop type: Router, Next hop index: 778
       Next hop: 2001:db8:104::4 via em1.104, selected
       Protocol next hop: 2001:db8:104::4
       Indirect next hop: 0xb1d27c0 1048578 INH Session ID: 0x15c
       State: <Int Ext>
       Inactive reason: LLGR stale
       Local AS: 65000 Peer AS: 65000
       Age: 4  Metric2: 0
       Communities: llgr-stale
       Accepted LongLivedStale
       Localpref: 100
       Router ID: 1.0.0.4
[…]

Have a look at the GitHub repository for the complete configurations as well as the expected outputs during normal operations. There is also a variant with the configurations of BIRD and JunOS when acting as a BGP route reflector. Now that FRR got BFD support, I hope it will get LLGR support as well.

  1. With point-to-point links, BGP can immediately detect a failure without BFD. However, with a pair of fibers, the failure may be undirectional, leaving it undetected by the other end until the expiration of the hold timer. ↩︎

  2. On a Juniper MX, BFD is usually handled directly by the real-time microkernel running on the packet forwarding engine. The BFD control packet contains a bit indicating if BFD is implemented by the forwarding plane or by the control plane. Therefore, you can check with tcpdump how a router implements BFD. Here is an example where 10.71.7.1, a Linux host running BIRD, implements BFD in the control plane, while 10.71.0.3, a Juniper MX, does not:

    $ sudo tcpdump -pni vlan181 port 3784
IP 10.71.7.1 > 10.71.0.3: BFDv1, Control, State Up, Flags: [none]
IP 10.71.0.3 > 10.71.7.1: BFDv1, Control, State Up, Flags: [Control Plane Independent]

    ↩︎

  3. Such a feature is the selling point of BGP graceful restart. However, without LLGR, non-functional paths are kept with the same preference and are not removed from ECMP routes. ↩︎

21 October, 2018 06:00PM by Vincent Bernat

Petter Reinholdtsen

Web browser integration of VLC with Bittorrent support

Bittorrent is as far as I know, currently the most efficient way to distribute content on the Internet. It is used all by all sorts of content providers, from national TV stations like NRK, Linux distributors like Debian and Ubuntu, and of course the Internet archive.

Almost a month ago a new package adding Bittorrent support to VLC became available in Debian testing and unstable. To test it, simply install it like this:

apt install vlc-plugin-bittorrent

Since the plugin was made available for the first time in Debian, several improvements have been made to it. In version 2.2-4, now available in both testing and unstable, a desktop file is provided to teach browsers to start VLC when the user click on torrent files or magnet links. The last part is thanks to me finally understanding what the strange x-scheme-handler style MIME types in desktop files are used for. By adding x-scheme-handler/magnet to the MimeType entry in the desktop file, at least the browsers Firefox and Chromium will suggest to start VLC when selecting a magnet URI on a web page. The end result is that now, with the plugin installed in Buster and Sid, one can visit any Internet Archive page with movies using a web browser and click on the torrent link to start streaming the movie.

Note, there is still some misfeatures in the plugin. One is the fact that it will hang and block VLC from exiting until the torrent streaming starts. Another is the fact that it will pick and play a random file in a multi file torrent. This is not always the video file you want. Combined with the first it can be a bit hard to get the video streaming going. But when it work, it seem to do a good job.

For the Debian packaging, I would love to find a good way to test if the plugin work with VLC using autopkgtest. I tried, but do not know enough of the inner workings of VLC to get it working. For now the autopkgtest script is only checking if the .so file was successfully loaded by VLC. If you have any suggestions, please submit a patch to the Debian bug tracking system.

As usual, if you use Bitcoin and want to show your support of my activities, please send Bitcoin donations to my address 15oWEoG9dUPovwmUL9KWAnYRtNJEkP1u1b.

21 October, 2018 07:50AM

October 20, 2018

hackergotchi for Michal &#268;iha&#345;

Michal Čihař

Weblate 3.2.2

Weblate 3.2.2 has been released today. It's a second bugfix release for 3.2 fixing several minor issues which appeared in the release.

Full list of changes:

  • Remove no longer needed Babel dependency.
  • Updated language definitions.
  • Improve documentation for addons, LDAP and Celery.
  • Fixed enabling new dos-eol and auto-java-messageformat flags.
  • Fixed running setup.py test from PyPI package.
  • Improved plurals handling.
  • Fixed translation upload API failure in some corner cases.
  • Fixed updating Git configuration in case it was changed manually.

If you are upgrading from older version, please follow our upgrading instructions.

You can find more information about Weblate on https://weblate.org, the code is hosted on Github. If you are curious how it looks, you can try it out on demo server. Weblate is also being used on https://hosted.weblate.org/ as official translating service for phpMyAdmin, OsmAnd, Turris, FreedomBox, Weblate itself and many other projects.

Should you be looking for hosting of translations for your project, I'm happy to host them for you or help with setting it up on your infrastructure.

Further development of Weblate would not be possible without people providing donations, thanks to everybody who have helped so far! The roadmap for next release is just being prepared, you can influence this by expressing support for individual issues either by comments or by providing bounty for them.

Filed under: Debian English SUSE Weblate

20 October, 2018 11:00AM

hackergotchi for Steve Kemp

Steve Kemp

So I wrote a basic BASIC

So back in June I challenged myself to write a BASIC interpreter in a weekend. The next time I mentioned it was to admit defeat. I didn't really explain in any detail, because I thought I'd wait a few days and try again and I was distracted at the time I wrote my post.

As it happened that was over four months ago, so clearly it didn't work out. The reason for this was because I was getting too bogged down in the wrong kind of details. I'd got my heart set on doing this the "modern" way:

  • Write a lexer to spit the input into tokens
    • LINE-NUMBER:10, PRINT, "Hello, World"
  • Then I'd take those tokens and form an abstract syntax tree.
  • Finally I'd walk the tree evaluating as I went.

The problem is that almost immediately I ran into problems, my naive approach didn't have a good solution for identifying line-numbers. So I was too paralysed to proceed much further.

I sidestepped the initial problem and figured maybe I should just have a series of tokens, somehow, which would be keyed off line-number. Obviously when you're interpreting "traditional" BASIC you need to care about lines, and treat them as important because you need to handle fun-things like this:

10 PRINT "STEVE ROCKS"
20 GOTO 10

Anyway I'd parse each line, assuming only a single statement upon a line (ha!) you can divide it into:

  • Number - i.e. line-number.
  • Statement.
  • Newline to terminate.

Then you could have:

code{blah} ..
code[10] = "PRINT STEVE ROCKS"
code[20] = "GOTO 10"

Obviously you spot the problem there, if you think it through. Anyway. I've been thinking about it off and on since then, and the end result is that for the past two evenings I've been mostly writing a BASIC interpreter, in golang, in 20-30 minute chunks.

The way it works is as you'd expect (don't make me laugh ,bitterly):

  • Parse the input into tokens.
  • Store those as an array.
  • Interpet each token.
    • No AST
    • No complicated structures.
    • Your program is literally an array of tokens.

I cheated, horribly, in parsing line-numbers which turned out to be exactly the right thing to do. The output of my naive lexer was:

INT:10, PRINT, STRING:"Hello World", NEWLINE, INT:20, GOTO, INT:10

Guess what? If you (secretly) prefix a newline to the program you're given you can identify line-numbers just by keeping track of your previous token in the lexer. A line-number is any number that follows a newline. You don't even have to care if they sequential. (Hrm. Bug-report?)

Once you have an array of tokens it becomes almost insanely easy to process the stream and run your interpreter:

 program[] = { LINE_NUMBER:10, PRINT, "Hello", NEWLINE, LINE_NUMBER:20 ..}

 let offset := 0
 for( offset < len(program) ) {
    token = program[offset]

    if ( token == GOTO ) { handle_goto() ; }
    if ( token == PRINT ) { handle_print() ; }
    .. handlers for every other statement
    offset++
 }

Make offset a global. And suddenly GOTO 10 becomes:

  • Scan the array, again, looking for "LINE_NUMBER:10".
  • Set offset to that index.

Magically it all just works. Add a stack, and GOSUB/RETURN are handled with ease too by pushing/popping the offset to it.

In fact even the FOR-loop is handled in only a few lines of code - most of the magic happening in the handler for the "NEXT" statement (because that's the part that needs to decide if it needs to jump-back to the body of the loop, or continue running.

OK this is a basic-BASIC as it is missing primtives (CHR(), LEN,etc) and it only cares about integers. But the code is wonderfully simple to understand, and the test-case coverage is pretty high.

I'll leave with an example:

10 REM This is a program
00 REM
 01 REM This program should produce 126 * 126 * 10
 02 REM  = 158760
 03 REM
 05 GOSUB 100
 10 FOR i = 0 TO 126
 20  FOR j = 0 TO 126 STEP 1
 30   FOR k = 0 TO 10
 40    LET a = i * j * k
 50   NEXT k
 60  NEXT j
 70 NEXT i
 75 PRINT a, "\n"
 80 END
100 PRINT "Hello, I'm multiplying your integers"
110 RETURN

Loops indented for clarity. Tokens in upper-case only for retro-nostalgia.

Find it here, if you care:

I had fun. Worth it.

I even "wrote" a "game":

20 October, 2018 05:15AM

October 19, 2018

hackergotchi for Robert McQueen

Robert McQueen

GNOME Foundation Hackfest 2018

This week, the GNOME Foundation Board of Directors met at the Collabora office in Cambridge, UK, for the second annual Foundation Hackfest. We were also joined by the Executive Director, Neil McGovern, and Director of Operations, Rosanna Yuen. This event was started by last year’s board and is a great opportunity for the newly-elected board to set out goals for the coming year and get some uninterrupted hacking done on policies, documents, etc. While it’s fresh in our mind, we wanted to tell you about some of the things we have been working on this week and what the community can hope to see in the coming months.

Wednesday: Goals

On Wednesday we set out to define the overall goals of the Foundation, so we could focus our activities for the coming years, ensuring that we were working on the right priorities. Neil helped to facilitate the discussion using the Charting Impact process. With that input, we went back to the purpose of the Foundation and mapped that to ten and five year goals, making sure that our current strategies and activities would be consistent with reaching those end points. This is turning out to be a very detailed and time-consuming process. We have made a great start, and hope to have something we can share for comments and input soon. The high level 10-year goals we identified boiled down to:

  • Sustainable project and foundation
  • Wider awareness and mindshare – being a thought leader
  • Increased user base

As we looked at the charter and bylaws, we identified a long-standing issue which we need to solve — there is currently no formal process to cover the “scope” of the Foundation in terms of which software we support with our resources. There is the release team, but that is only a subset of the software we support. We have some examples such as GIMP which “have always been here”, but at present there is no clear process to apply or be included in the Foundation. We need a clear list of projects that use resources such as CI, or have the right to use the GNOME trademark for the project. We have a couple of similar proposals from Allan Day and Carlos Soriano for how we could define and approve projects, and we are planning to work with them over the next couple of weeks to make one proposal for the board to review.

Thursday: Budget forecast

We started the second day with a review of the proposed forecast from Neil and Rosanna, because the Foundation’s financial year starts in October. We have policies in place to allow staff and committees to spend money against their budget without further approval being needed, which means that with no approved budget, it’s very hard for the Foundation to spend any money. The proposed budget was based off the previous year’s actual figures, with changes to reflect the increased staff headcount, increased spend on CI, increased staff travel costs, etc, and ensure after the year’s spending, we follow the reserves policy to keep enough cash to pay the foundation staff for a further year. We’re planning to go back and adjust a few things (internships, marketing, travel, etc) to make sure that we have the right resources for the goals we identified.

We had some “hacking time” in smaller groups to re-visit and clarify various policies, such as the conference and hackfest proposal/approval process, travel sponsorship process and look at ways to support internationalization (particularly to indigenous languages).

Friday: Foundation Planning

The Board started Friday with a board-only (no staff) meeting to make sure we were aligned on the goals that we were setting for the Executive Director during the coming year, informed by the Foundation goals we worked on earlier in the week. To avoid the “seven bosses” problem, there is one board member (myself) responsible for managing the ED’s priorities and performance. It’s important that I take advantage of the opportunity of the face to face meeting to check in with the Board about their feedback for the ED and things I should work together with Neil on over the coming months.

We also discussed a related topic, which is the length of the term that directors serve on the Foundation Board. With 7 staff members, the Foundation needs consistent goals and management from one year to the next, and the time demands on board members should be reduced from previous periods where the Foundation hasn’t had an Executive Director. We want to make sure that our “ten year goals” don’t change every year and undermine the strategies that we put in place and spend the Foundation resources on. We’re planning to change the Board election process so that each director has a two year term, so half of the board will be re-elected each year. This also prevents the situation where the majority of the Board is changed at the same election, losing continuity and institutional knowledge, and taking months for people to get back up to speed.

We finished the day with a formal board meeting to approve the budget, more hack time on various policies (and this blog!). Thanks to Collabora for use of their office space, food, and snacks – and thanks to my fellow Board members and the Foundation’s wonderful and growing staff team

19 October, 2018 03:38PM by ramcq

hackergotchi for Michal &#268;iha&#345;

Michal Čihař

translation-finder 0.1

Setting up translation components in Weblate can be tricky in some cases, especially if you lack knowledge of the translation format you are using. Also this is something we wanted to automate from the very beginning, but there were always more pressing things to implement. But now the time is coming as I've just made first beta release of translation-finder, tool to help with this.

The translation-finder will look at filesystem (eg. checked out repository) and tries to find translatable files. So far the heuristics is pretty simple, but still it detects just fine most of the projects currently hosted on our hosted localization platform. Still if you find issue with that, you're welcome to provide feedback in our issue tracker.

The integration into Weblate will come in next weeks and will be able to enjoy this new feature in the 3.3 release.

Filed under: Debian English SUSE Weblate

19 October, 2018 02:30PM

hackergotchi for Daniel Pocock

Daniel Pocock

Debian GSoC 2018 report

One of my major contributions to Debian in 2018 has been participation as a mentor and admin for Debian in Google Summer of Code (GSoC).

Here are a few observations about what happened this year, from my personal perspective in those roles.

Making a full report of everything that happens in GSoC is close to impossible. Here I consider issues that span multiple projects and the mentoring team. For details on individual projects completed by the students, please see their final reports posted in August on the mailing list.

Thanking our outgoing administrators

Nicolas Dandrimont and Sylvestre Ledru retired from the admin role after GSoC 2016 and Tom Marble has retired from the Outreachy administration role, we should be enormously grateful for the effort they have put in as these are very demanding roles.

When the last remaining member of the admin team, Molly, asked for people to step in for 2018, knowing the huge effort involved, I offered to help out on a very temporary basis. We drafted a new delegation but didn't seek to have it ratified until the team evolves. We started 2018 with Molly, Jaminy, Alex and myself. The role needs at least one new volunteer with strong mentoring experience for 2019.

Project ideas

Google encourages organizations to put project ideas up for discussion and also encourages students to spontaneously propose their own ideas. This latter concept is a significant difference between GSoC and Outreachy that has caused unintended confusion for some mentors in the past. I have frequently put teasers on my blog, without full specifications, to see how students would try to respond. Some mentors are much more precise, telling students exactly what needs to be delivered and how to go about it. Both approaches are valid early in the program.

Student inquiries

Students start sending inquiries to some mentors well before GSoC starts. When Google publishes the list of organizations to participate (that was on 12 February this year), the number of inquiries increases dramatically, in the form of personal emails to the mentors, inquiries on the debian-outreach mailing list, the IRC channel and many project-specific mailing lists and IRC channels.

Over 300 students contacted me personally or through the mailing list during the application phase (between 12 February and 27 March). This is a huge number and makes it impossible to engage in a dialogue with every student. In the last years where I have mentored, 2016 and 2018, I've personally but a bigger effort into engaging other mentors during this phase and introducing them to some of the students who already made a good first impression.

As an example, Jacob Adams first inquired about my PKI/PGP Clean Room idea back in January. I was really excited about his proposals but I knew I simply didn't have the time to mentor him personally, so I added his blog to Planet Debian and suggested he put out a call for help. One mentor, Daniele Nicolodi replied to that and I also introduced him to Thomas Levine. They both generously volunteered and together with Jacob, ensured a successful project. While I originally started the clean room, they deserve all the credit for the enhancements in 2018 and this emphasizes the importance of those introductions made during the early stages of GSoC.

In fact, there were half a dozen similar cases this year where I have interacted with a really promising student and referred them to the mentor(s) who appeared optimal for their profile.

After my recent travels in the Balkans, a number of people from Albania and Kosovo expressed an interest in GSoC and Outreachy. The students from Kosovo found that their country was not listed in the application form but the Google team very promptly added it, allowing them to apply for GSoC for the first time. Kosovo still can't participate in the Olympics or the World Cup, but they can compete in GSoC now.

At this stage, I was still uncertain if I would mentor any project myself in 2018 or only help with the admin role, which I had only agreed to do on a very temporary basis until the team evolves. Nonetheless, the day before student applications formally opened (12 March) and after looking at the interest areas of students who had already made contact, I decided to go ahead mentoring a single project, the wizard for new students and contributors.

Student selections

The application deadline closed on 27 March. At this time, Debian had 102 applications, an increase over the 75 applications from 2016. Five applicants were female, including three from Kosovo.

One challenge we've started to see is that since Google reduced the stipend for GSoC, Outreachy appears to pay more in many countries. Some women put more effort into an Outreachy application or don't apply for GSoC at all, even though there are far more places available in GSoC each year. GSoC typically takes over 1,000 interns in each round while Outreachy can only accept approximately 50.

Applicants are not evenly distributed across all projects. Some mentors/projects only receive one applicant and then mentors simply have to decide if they will accept the applicant or cancel the project. Other mentors receive ten or more complete applications and have to spend time studying them, comparing them and deciding on the best way to rank them and make a decision.

Given the large number of project ideas in Debian, we found that the Google portal didn't allow us to use enough category names to distinguish them all. We contacted the Google team about this and they very quickly increased the number of categories we could use, this made it much easier to tag the large number of applications so that each mentor could filter the list and only see their own applicants.

The project I mentored personally, a wizard for helping new students get started, attracted interest from 3 other co-mentors and 10 student applications. To help us compare the applications and share data we gathered from the students, we set up a shared spreadsheet using Debian's Sandstorm instance and Ethercalc. Thanks to Asheesh and Laura for setting up and maintaining this great service.

Slot requests

Switching from the mentor hat to the admin hat, we had to coordinate the requests from each mentor to calculate the total number of slots we wanted Google to fund for Debian's mentors.

Once again, Debian's Sandstorm instance, running Ethercalc, came to the rescue.

All mentors were granted access, reducing the effort for the admins and allowing a distributed, collective process of decision making. This ensured mentors could see that their slot requests were being counted correctly but it means far more than that too. Mentors put in a lot of effort to bring their projects to this stage and it is important for them to understand any contention for funding and make a group decision about which projects to prioritize if Google doesn't agree to fund all the slots.

Management tools and processes

Various topics were discussed by the team at the beginning of GSoC.

One discussion was about the definition of "team". Should the new delegation follow the existing pattern, reserving the word "team" for the admins, or should we move to the convention followed by the DebConf team, where the word "team" encompasses a broader group of the volunteers? A draft delegation text was prepared but we haven't asked for it to be ratified, this is a pending task for the 2019 team (more on that later).

There was discussion about the choice of project management tools, keeping with Debian's philosophy of only using entirely free tools. We compared various options, including Redmine with the Agile (Kanban) plugin, Kanboard (as used by DebConf team), and more Sandstorm-hosted possibilities, such as Wekan and Scrumblr. Some people also suggested ideas for project management within their Git repository, for example, using Org-mode. There was discussion about whether it would be desirable for admins to run an instance of one of these tools to manage our own workflow and whether it would be useful to have all students use the same tool to ease admin supervision and reporting. Personally, I don't think all students need to use the same tool as long as they use tools that provide public read-only URLs, or even better, a machine-readable API allowing admins to aggregate data about progress.

Admins set up a Git repository for admin and mentor files on Debian's new GitLab instance, Salsa. We tried to put in place a process to synchronize the mentor list on the wiki, the list of users granted team access in Salsa and the list of mentors maintained in the GSoC portal. This could be taken further by asking mentors and students to put a Moin Category tag on the bottom of their personal pages on the wiki, allowing indexes to be built automatically.

Students accepted

On 23 April, the list of selected students was confirmed. Shortly afterward, a Debian blog appeared welcoming the students.

OSCAL 2018, Albania and Kosovo visit

I traveled to Tirana, Albania for OSCAL'18 where I was joined by two of the Kosovan students selected by Debian. They helped run the Debian booth, comprising a demonstration of software defined radio from Debian Hams.

Enkelena Haxhiu and I gave a talk together about communications technology. This was Enkelena's first talk. In the audience was Arjen Kamphuis, he was one of the last people to ask a question at the end. His recent disappearance is a disturbing mystery.

DebConf18

A GSoC session took place at DebConf18, the video is available here and includes talks from GSoC and Outreachy participants past and present.

Final results

Many of the students have already been added to Planet Debian where they have blogged about what they did and what they learned in GSoC. More will appear in the near future.

If you like their project, if you have ideas for an event where they could present it or if you simply live in the same region, please feel free to contact the students directly and help them continue their free software adventure with us.

Meeting more students

Google's application form for organizations like Debian asks us what we do to stay in contact with students after GSoC. Crossing multiple passes in the Swiss and Italian alps to find Sergio Alberti at Capo di Lago is probably one of the more exotic answers to that question.

Looking back at past internships

I first mentored students in GSoC 2013. Since then, I've been involved in mentoring a total of 12 students in GSoC and 3 interns in Outreachy as well as introducing many others to mentors and organizations. Several of them stay in touch and it's always interesting to hear about their successes as they progress in their careers and in their enjoyment of free software.

The Outreachy organizers have chosen a picture of two of my former interns, Urvika Gola (Outreachy 2016) and Pranav Jain (GSoC 2016) for the mentors page of their web site. This is quite fitting as both of them have remained engaged and become involved in the mentoring process.

Lessons from GSoC 2018, preparing for 2019

One of the big challenges we faced this year is that as the new admin team was only coming together for the first time, we didn't have any policies in place before mentors and students started putting significant effort in to their proposals.

Potential mentors start to put in significant effort from February, when the list of participating organizations is usually announced by Google. Therefore, it seems like a good idea to make any policies clear to potential mentors before the end of January.

We faced a similar challenge with selecting mentors to attend the GSoC mentor summit. While some ideas were discussed about the design of a selection process or algorithm, the admins fell back on the previous policy based on a random selection as mentors may have anticipated that policy was still in force when they signed up.

As I mentioned already, there are several areas where GSoC and Outreachy are diverging, this already led to some unfortunate misunderstandings in both directions, for example, when people familiar with Outreachy rules have been unaware of GSoC differences and vice-versa and I'll confess to being one of several people who has been confused at least once. Mentors often focus on the projects and candidates and don't always notice the annual rule changes. Unfortunately, this requires involvement and patience from both the organizers and admins to guide the mentors through any differences at each step.

The umbrella organization question

One of the most contentious topics in Debian's GSoC 2018 program was the discussion of whether Debian can and should act as an umbrella organization for smaller projects who are unlikely to participate in GSoC in their own right.

As an example, in 2016, four students were mentored by Savoir Faire Linux (SFL), makers of the Ring project, under the Debian umbrella. In 2017, Ring joined the GNU Project and they mentored students under the GNU Project umbrella organization. DebConf17 coincidentally took place in Montreal, Canada, not far from the SFL headquarters and SFL participated as a platinum sponsor.

Google's Mentor Guide explicitly encourages organizations to consider this role, but does not oblige them to do so either:

Google’s program administrators actually look quite fondly on the umbrella organizations that participate each year.

For an organization like Debian, with our philosophy, independence from the cloud and distinct set of tools, such as the Salsa service mentioned earlier, being an umbrella organization gives us an opportunity to share the philosophy and working methods for mutual benefit while also giving encouragement to related projects that we use.

Some people expressed concern that this may cut into resources for Debian-centric projects, but it appears that Google has not limited the number of additional places in the program for this purpose. This is one of the significant differences with Outreachy, where the number of places is limited by funding constraints.

Therefore, if funding is not a constraint, I feel that the most important factor to evaluate when considering this issue is the size and capacity of the admin team. Google allows up to five people to be enrolled as admins and if enough experienced people volunteer, it can be easier for everybody whereas with only two admins, the minimum, it may not be feasible to act as an umbrella organization.

Within the team, we observed various differences of opinion: for example some people were keen on the umbrella role while others preferred to restrict participation to Debian-centric projects. We have the same situation with Outreachy: some mentors and admins only want to do GSoC, while others only do Outreachy and there are others, like myself, who have supported both programs equally. In situations like this, nobody is right or wrong.

Once that fundamental constraint, the size of the admin team, is considered, I personally feel that any related projects engaged on this basis can be evaluated for a wide range of synergies with the Debian community, including the people, their philosophy, the tools used and the extent to which their project will benefit Debian's developers and users. In other words, this doesn't mean any random project can ask to participate under the Debian umbrella but those who make the right moves may have a chance of doing so.

Financial

Google pays each organization an allowance of USD 500 for each slot awarded to the organization, plus some additional funds related to travel. This generally corresponds to the number of quality candidates identified by the organization during the selection process, regardless of whether the candidate accepts an internship or not. Where more than one organization requests funding (a slot) for the same student, both organizations receive a bounty, we had at least one case like this in 2018.

For 2018, Debian has received USD 17,200 from Google.

GSoC 2019 and beyond

Personally, as I indicated in January that I would only be able to do this on a temporary basis, I'm not going to participate as an admin in 2019 so it is a good time for other members of the community to think about the role. Each organization who wants to participate needs to propose a full list of admins to Google in January 2019, therefore, now is the time for potential admins to step forward, decide how they would like to work together as a team and work out the way to recruit mentors and projects.

Thanks to all the other admins, mentors, the GSoC team at Google, the Outreachy organizers and members of the wider free software community who supported this initiative in 2018. I'd particularly like to thank all the students though, it is really exciting to work with people who are so open minded, patient and remain committed even when faced with unanticipated challenges and adversity.

19 October, 2018 08:26AM by Daniel.Pocock

October 18, 2018

Petter Reinholdtsen

Release 0.2 of free software archive system Nikita announced

This morning, the new release of the Nikita Noark 5 core project was announced on the project mailing list. The free software solution is an implementation of the Norwegian archive standard Noark 5 used by government offices in Norway. These were the changes in version 0.2 since version 0.1.1 (from NEWS.md):

  • Fix typos in REL names
  • Tidy up error message reporting
  • Fix issue where we used Integer.valueOf(), not Integer.getInteger()
  • Change some String handling to StringBuffer
  • Fix error reporting
  • Code tidy-up
  • Fix issue using static non-synchronized SimpleDateFormat to avoid race conditions
  • Fix problem where deserialisers were treating integers as strings
  • Update methods to make them null-safe
  • Fix many issues reported by coverity
  • Improve equals(), compareTo() and hash() in domain model
  • Improvements to the domain model for metadata classes
  • Fix CORS issues when downloading document
  • Implementation of case-handling with registryEntry and document upload
  • Better support in Javascript for OPTIONS
  • Adding concept description of mail integration
  • Improve setting of default values for GET on ny-journalpost
  • Better handling of required values during deserialisation
  • Changed tilknyttetDato (M620) from date to dateTime
  • Corrected some opprettetDato (M600) (de)serialisation errors.
  • Improve parse error reporting.
  • Started on OData search and filtering.
  • Added Contributor Covenant Code of Conduct to project.
  • Moved repository and project from Github to Gitlab.
  • Restructured repository, moved code into src/ and web/.
  • Updated code to use Spring Boot version 2.
  • Added support for OAuth2 authentication.
  • Fixed several bugs discovered by Coverity.
  • Corrected handling of date/datetime fields.
  • Improved error reporting when rejecting during deserializatoin.
  • Adjusted default values provided for ny-arkivdel, ny-mappe, ny-saksmappe, ny-journalpost and ny-dokumentbeskrivelse.
  • Several fixes for korrespondansepart*.
  • Updated web GUI:
    • Now handle both file upload and download.
    • Uses new OAuth2 authentication for login.
    • Forms now fetches default values from API using GET.
    • Added RFC 822 (email), TIFF and JPEG to list of possible file formats.

The changes and improvements are extensive. Running diffstat on the changes between git tab 0.1.1 and 0.2 show 1098 files changed, 108666 insertions(+), 54066 deletions(-).

If free and open standardized archiving API sound interesting to you, please contact us on IRC (#nikita on irc.freenode.net) or email (nikita-noark mailing list).

As usual, if you use Bitcoin and want to show your support of my activities, please send Bitcoin donations to my address 15oWEoG9dUPovwmUL9KWAnYRtNJEkP1u1b.

18 October, 2018 12:40PM

October 17, 2018

hackergotchi for Michal &#268;iha&#345;

Michal Čihař

wlc 0.9

wlc 0.9, a command line utility for Weblate, has been just released. There are several new commands like translation file upload or repository cleanup. The codebase has been also migrated to use requests instead of urllib.

Full list of changes:

  • Switched to requests
  • Added support for cleanup command.
  • Added support for upload command.

wlc is built on API introduced in Weblate 2.6 and still being in development, you need at least Weblate 2.10 (or use on our hosting offering). You can find usage examples in the wlc documentation.

Filed under: Debian English SUSE Weblate

17 October, 2018 03:00PM

October 16, 2018

hackergotchi for Matthew Garrett

Matthew Garrett

Initial thoughts on MongoDB's new Server Side Public License

MongoDB just announced that they were relicensing under their new Server Side Public License. This is basically the Affero GPL except with section 13 largely replaced with new text, as follows:

If you make the functionality of the Program or a modified version available to third parties as a service, you must make the Service Source Code available via network download to everyone at no charge, under the terms of this License. Making the functionality of the Program or modified version available to third parties as a service includes, without limitation, enabling third parties to interact with the functionality of the Program or modified version remotely through a computer network, offering a service the value of which entirely or primarily derives from the value of the Program or modified version, or offering a service that accomplishes for users the primary purpose of the Software or modified version.

“Service Source Code” means the Corresponding Source for the Program or the modified version, and the Corresponding Source for all programs that you use to make the Program or modified version available as a service, including, without limitation, management software, user interfaces, application program interfaces, automation software, monitoring software, backup software, storage software and hosting software, all such that a user could run an instance of the service using the Service Source Code you make available.

MongoDB admit that this license is not currently open source in the sense of being approved by the Open Source Initiative, but say:We believe that the SSPL meets the standards for an open source license and are working to have it approved by the OSI.

At the broadest level, AGPL requires you to distribute the source code to the AGPLed work[1] while the SSPL requires you to distribute the source code to everything involved in providing the service. Having a license place requirements around things that aren't derived works of the covered code is unusual but not entirely unheard of - the GPL requires you to provide build scripts even if they're not strictly derived works, and you could probably make an argument that the anti-Tivoisation provisions of GPL3 fall into this category.

A stranger point is that you're required to provide all of this under the terms of the SSPL. If you have any code in your stack that can't be released under those terms then it's literally impossible for you to comply with this license. I'm not a lawyer, so I'll leave it up to them to figure out whether this means you're now only allowed to deploy MongoDB on BSD because the license would require you to relicense Linux away from the GPL. This feels sloppy rather than deliberate, but if it is deliberate then it's a massively greater reach than any existing copyleft license.

You can definitely make arguments that this is just a maximalist copyleft license, the AGPL taken to extreme, and therefore it fits the open source criteria. But there's a point where something is so far from the previously accepted scenarios that it's actually something different, and should be examined as a new category rather than already approved categories. I suspect that this license has been written to conform to a strict reading of the Open Source Definition, and that any attempt by OSI to declare it as not being open source will receive pushback. But definitions don't exist to be weaponised against the communities that they seek to protect, and a license that has overly onerous terms should be rejected even if that means changing the definition.

In general I am strongly in favour of licenses ensuring that users have the freedom to take advantage of modifications that people have made to free software, and I'm a fan of the AGPL. But my initial feeling is that this license is a deliberate attempt to make it practically impossible to take advantage of the freedoms that the license nominally grants, and this impression is strengthened by it being something that's been announced with immediate effect rather than something that's been developed with community input. I think there's a bunch of worthwhile discussion to have about whether the AGPL is strong and clear enough to achieve its goals, but I don't think that this SSPL is the answer to that - and I lean towards thinking that it's not a good faith attempt to produce a usable open source license.

(It should go without saying that this is my personal opinion as a member of the free software community, and not that of my employer)

[1] There's some complexities around GPL3 code that's incorporated into the AGPLed work, but if it's not part of the AGPLed work then it's not covered

comment count unavailable comments

16 October, 2018 10:43PM

Reproducible builds folks

Reproducible Builds: Weekly report #181

Here’s what happened in the Reproducible Builds effort between Sunday October 7 and Saturday October 13 2018:

Another brief reminder that another Reproducible Builds summit will be taking place between 11th—13th December 2018 in Mozilla’s offices in Paris. If you are interested in attending please send an email to holger@layer-acht.org. More details can also be found on the corresponding event page of our website.

diffoscope development

diffoscope is our in-depth “diff-on-steroids” utility which helps us diagnose reproducibility issues in packages) was updated this week, including contributions from:

Packages reviewed and fixed, and bugs filed

Test framework development

There were a large number of updates to our Jenkins-based testing framework that powers tests.reproducible-builds.org by Holger Levsen this month, including:

In addition, Mattia Rizzolo performed some node administration (1, 2).

Misc.

This week’s edition was written by Bernhard M. Wiedemann, Chris Lamb, Holger Levsen & reviewed by a bunch of Reproducible Builds folks on IRC & the mailing lists.

16 October, 2018 08:23PM

hackergotchi for Julien Danjou

Julien Danjou

More GitHub workflow automation

More GitHub workflow automation

The more you use computers, the more you see the potentials for automating everything. Who doesn't love that? By building Mergify those last months, we've decided it was time bring more automation to the development workflow.

Mergify's first version was a minimal viable product around automating the merge of pull requests. As I wrote a few months ago, we wanted to automate the merge of pull requests when it was ready to be merged. For most projects, this is easy and consists of a simple rule: "it must be approved by a developer and pass the CI".

Evolving on Feedback

For the first few months, we received a lot of feedback from our users. They were enthusiastic about the product but were frustrated by a couple of things.

First, Mergify would mess up with branch protections. We thought that people wanted the GitHub UI to match their rules. As I'll explain later, it turns out to be only partially true, and we found a workaround.

Then, Mergify's abilities were capped by some of the limitations of the GitHub workflow and API. For example, GitHub would only allow rules per branch, whereas our users wanted to have rules applied based on a lot of different criteria.

Building the Next Engine

We rolled up our sleeves and started to build that new engine. The first thing was to get rid of the GitHub branch protection feature altogether and leveraging the Checks API to render something useful to the users in the UI. You can now have a complete overview of the rules that will be applied to your pull requests in the UI, making it easy to understand what's happening.

More GitHub workflow automation

Then, we wrote a new matching engine that would allow matching any pull requests based on any of its attributes. You can now automate your workflow with a finer-grained configuration.

What Does It Look Like?

Here's a simple rule you could write:

pull_request_rules:
  - name: automatic merge on approval and CI pass
    conditions:
     - "#approved-reviews-by>=1"
     - status-success=continuous-integration/travis-ci/pr
     - label!=work-in-progress
    actions:
      merge:
        method: merge

With that, any pull request that has been approved by a collaborator, passes the Travis CI job and does not have the label work-in-progress will be automatically merged by Mergify.

You could use even more actions to backport this pull request to another branch, close the pull request or add/remove labels. We're starting to see users building amazing workflow with that engine!

We're thrilled by this new version we launched this week and glad we're getting amazing feedback (again) from our users.

When you give it a try, drop me a note and let me know what you think about it!

16 October, 2018 12:39PM by Julien Danjou

October 15, 2018

hackergotchi for Michal &#268;iha&#345;

Michal Čihař

uTidylib 0.4

Two years ago, I've taken over uTidylib maintainership. Two years has passed without any bigger contribution, but today there is a new version with support for recent html-tidy and Python 3.

The release still can't be uploaded to PyPI (see https://github.com/pypa/warehouse/issues/4860), but it's available for download from my website or tagged in the GitHub repository.

Full list of changes is quite small:

  • Compatibility with html-tidy 5.6.0.
  • Added support for Python 3.

Anyway as I can not update PyPI entry, the downloads are currently available only on my website: https://cihar.com/software/utidylib/

Filed under: Debian English SUSE uTidylib

15 October, 2018 02:30PM

hackergotchi for Robert McQueen

Robert McQueen

Flatpaks, sandboxes and security

Last week the Flatpak community woke to the “news” that we are making the world a less secure place and we need to rethink what we’re doing. Personally, I’m not sure this is a fair assessment of the situation. The “tl;dr” summary is: Flatpak confers many benefits besides the sandboxing, and even looking just at the sandboxing, improving app security is a huge problem space and so is a work in progress across multiple upstream projects. Much of what has been achieved so far already delivers incremental improvements in security, and we’re making solid progress on the wider app distribution and portability problem space.

Sandboxing, like security in general, isn’t a binary thing – you can’t just say because you have a sandbox, you have 100% security. Like having two locks on your front door, two front doors, or locks on your windows too, sensible security is about defense in depth. Each barrier that you implement precludes some invalid or possibly malicious behaviour. You hope that in total, all of these barriers would prevent anything bad, but you can never really guarantee this – it’s about multiplying together probabilities to get a smaller number. A computer which is switched off, in a locked faraday cage, with no connectivity, is perfectly secure – but it’s also perfectly useless because you cannot actually use it. Sandboxing is very much the same – whilst you could easily take systemd-nspawn, Docker or any other container technology of choice and 100% lock down a desktop app, you wouldn’t be able to interact with it at all.

Network services have incubated and driven most of the container usage on Linux up until now but they are fundamentally different to desktop applications. For services you can write a simple list of permissions like, “listen on this network port” and “save files over here” whereas desktop applications have a much larger number of touchpoints to the outside world which the user expects and requires for normal functionality. Just thinking off the top of my head you need to consider access to the filesystem, display server, input devices, notifications, IPC, accessibility, fonts, themes, configuration, audio playback and capture, video playback, screen sharing, GPU hardware, printing, app launching, removable media, and joysticks. Without making holes in the sandbox to allow access to these in to your app, it either wouldn’t work at all, or it wouldn’t work in the way that people have come to expect.

What Flatpak brings to this is understanding of the specific desktop app problem space – most of what I listed above is to a greater or lesser extent understood by Flatpak, or support is planned. The Flatpak sandbox is very configurable, allowing the application author to specify which of these resources they need access to. The Flatpak CLI asks the user about these during installation, and we provide the flatpak override command to allow the user to add or remove these sandbox escapes. Flatpak has introduced portals into the Linux desktop ecosystem, which we’re really pleased to be sharing with snap since earlier this year, to provide runtime access to resources outside the sandbox based on policy and user consent. For instance, document access, app launching, input methods and recursive sandboxing (“sandbox me harder”) have portals.

The starting security position on the desktop was quite terrible – anything in your session had basically complete access to everything belonging to your user, and many places to hide.

  • Access to the X socket allows arbitrary input and output to any other app on your desktop, but without it, no app on an X desktop would work. Wayland fixes this, so Flatpak has a fallback setting to allow Wayland to be used if present, and the X socket to be shared if not.
  • Unrestricted access to the PulseAudio socket allows you to reconfigure audio routing, capture microphone input, etc. To ensure user consent we need a portal to control this, where by default you can play audio back but device access needs consent and work is under way to create this portal.
  • Access to the webcam device node means an app can capture video whenever it wants – solving this required a whole new project.
  • Sandboxing access to configuration in dconf is a priority for the project right now, after the 1.0 release.

Even with these caveats, Flatpak brings a bunch of default sandboxing – IPC filtering, a new filesystem, process and UID namespace, seccomp filtering, an immutable /usr and /app – and each of these is already a barrier to certain attacks.

Looking at the specific concerns raised:

  • Hopefully from the above it’s clear that sandboxing desktop apps isn’t just a switch we can flick overnight, but what we already have is far better than having nothing at all. It’s not the intention of Flatpak to somehow mislead people that sandboxed means somehow impervious to all known security issues and can access nothing whatsoever, but we do want to encourage the use of the new technology so that we can work together on driving adoption and making improvements together. The idea is that over time, as the portals are filled out to cover the majority of the interfaces described, and supported in the major widget sets / frameworks, the criteria for earning a nice “sandboxed” badge or submitting your app to Flathub will become stricter. Many of the apps that access --filesystem=home are because they use old widget sets like Gtk2+ and frameworks like Electron that don’t support portals (yet!). Contributions to improve portal integration into other frameworks and desktops are very welcome and as mentioned above will also improve integration and security in other systems that use portals, such as snap.
  • As Alex has already blogged, the freedesktop.org 1.6 runtime was something we threw together because we needed something distro agnostic to actually be able to bootstrap the entire concept of Flatpak and runtimes. A confusing mishmash of Yocto with flatpak-builder, it’s thankfully nearing some form of retirement after a recent round of security fixes. The replacement freedesktop-sdk project has just released its first stable 18.08 release, and rather than “one or two people in their spare time because something like this needs to exist”, is backed by a team from Codethink and with support from the Flatpak, GNOME and KDE communities.
  • I’m not sure how fixing and disclosing a security problem in a relatively immature pre-1.0 program (in June 2017, Flathub had less than 50 apps) is considered an ongoing problem from a security perspective. The wording in the release notes?

Zooming out a little bit, I think it’s worth also highlighting some of the other reasons why Flatpak exists at all – these are far bigger problems with the Linux desktop ecosystem than app security alone, and Flatpak brings a huge array of benefits to the table:

  • Allowing apps to become agnostic of their underlying distribution. The reason that runtimes exist at all is so that apps can specify the ABI and dependencies that they need, and you can run it on whatever distro you want. Flatpak has had this from day one, and it’s been hugely reliable because the sandboxed /usr means the app can rely on getting whatever they need. This is the foundation on which everything else is built.
  • Separating the release/update cadence of distributions from the apps. The flip side of this, which I think is huge for more conservative platforms like Debian or enterprise distributions which don’t want to break their ABIs, hardware support or other guarantees, is that you can still get new apps into users hands. Wider than this, I think it allows us huge new freedoms to move in a direction of reinventing the distro – once you start to pull the gnarly complexity of apps and their dependencies into sandboxes, your constraints are hugely reduced and you can slim down or radically rethink the host system underneath. At Endless OS, Flatpak literally changed the structure of our engineering team, and for the first time allowed us to develop and deliver our OS, SDK and apps in independent teams each with their own cadence.
  • Disintermediating app developers from their users. Flathub now offers over 400 apps, and (at a rough count by Nick Richards over the summer) over half of them are directly maintained by or maintained in conjunction with the upstream developers. This is fantastic – we get the releases when they come out, the developers can choose the dependencies and configuration they need – and they get to deliver this same experience to everyone.
  • Decentralised. Anyone can set up a Flatpak repo! We started our own at Flathub because there needs to be a center of gravity and a complete story to build out a user and developer base, but the idea is that anyone can use the same tools that we do, and publish whatever/wherever they want. GNOME uses GitLab CI to publish nightly Flatpak builds, KDE is setting up the same in their infrastructure, and Fedora is working on completely different infrastructure to build and deliver their packaged applications as Flatpaks.
  • Easy to build. I’ve worked on Debian packages, RPMs, Yocto, etc and I can honestly say that flatpak-builder has done a very good job of making it really easy to put your app manifest together. Because the builds are sandboxed and each runtimes brings with it a consistent SDK environment, they are very reliably reproducible. It’s worth just calling this out because when you’re trying to attract developers to your platform or contributors to your app, hurdles like complex or fragile tools and build processes to learn and debug all add resistance and drag, and discourage contributions. GNOME Builder can take any flatpak’d app and build it for you automatically, ready to hack within minutes.
  • Different ways to distribute apps. Using OSTree under the hood, Flatpak supports single-file app .bundles, pulling from OSTree repos and OCI registries, and at Endless we’ve been working on peer-to-peer distribution like USB sticks and LAN sharing.

Nobody is trying to claim that Flatpak solves all of the problems at once, or that what we have is anywhere near perfect or completely secure, but I think what we have is pretty damn cool (I just wish we’d had it 10 years ago!). Even just in the security space, the overall effort we need is huge, but this is a journey that we are happy to be embarking together with the whole Linux desktop community. Thanks for reading, trying it out, and lending us a hand.

15 October, 2018 01:40PM by ramcq

hackergotchi for Lars Wirzenius

Lars Wirzenius

Rewrote summain from Python to Rust

I've been learning Rust lately. As part of that, I rewrote my summain program from Python to Rust (see summainrs). It's not quite a 1:1 rewrite: the Python version outputs RFC822-style records, the Rust one uses YAML. The Rust version is my first attempt at using multithreading, something I never added to the Python version.

Results:

  • Input is a directory tree with 8.9 gigabytes of data in 9650 files and directories.
  • Each file gets stat'd, and regular files get SHA256 computed.
  • Run on a Thinkpad X220 laptop with a rotating hard disk. Two CPU cores, 4 hyperthreads. Mostly idle, but desktop-py things running in the background. (Not a very systematic benchmark.)
  • Python version: 123 seconds wall clock time, 54 seconds user, 6 second system time.
  • Rust version: 61 seconds wall clock (50% of the speed), 56 seconds user (104%), and 4 seconds system time (67&).

A nice speed improvement, I think. Especially, since the difference between the single and multithreaded version of the Rust program is four characters (par_iter instead of iter in the process_chunk function).

15 October, 2018 08:13AM

hackergotchi for Louis-Philippe Véronneau

Louis-Philippe Véronneau

A Good Harvest of the Devil's Lettuce

Hop cones layed out for drying

You might have heard that Canada's legalising marijuana in 2 days. Even though I think it's a pretty good idea, this post is not about pot, but about another type of Devil's Lettuce: hops.

As we all know, homebrewing beer is a gateway into growing hops, a highly suspicious activity that attracts only marginals and deviants. Happy to say I've been successfully growing hops for two years now and this year's harvest has been bountiful.

Two years ago, I planted two hops plants, one chinook and one triple pearl. A year prior to this I had tried to grow a cascade plant in a container on my balcony, but it didn't work out well. This time I got around to plant the rhizomes in the ground under my balcony and had the bines grow on ropes.

Although I've been having trouble with the triple pearl (the soil where I live is thick and heavy clay - not the best for hops), the chinook has been growing pretty well.

Closeup of my chinook hops on the bines

Harvest time is always fun and before taking the bines down, I didn't know how much cones I would get this year. I'd say compared to last year, I tripled my yield. With some luck (and better soil), I should be able to get my triple pearl to produce cones next year.

Here a nice poem about the usefulness of hops written by Thomas Tusser in 1557:

      The hop for his profit I thus do exalt,
      It strengtheneth drink and it flavoureth malt;
      And being well-brewed long kept it will last,
      And drawing abide, if ye draw not too fast.

So remember kids, don't drink and upload and if you decide to grow some of the Devil's Lettuce, make sure you use it to flavoureth malt and not your joint. The ones waging war on drugs might not like it.

15 October, 2018 03:30AM by Louis-Philippe Véronneau

October 14, 2018

hackergotchi for Dirk Eddelbuettel

Dirk Eddelbuettel

RcppCCTZ 0.2.5

A new bugfix release 0.2.5 of RcppCCTZ got onto CRAN this morning – just a good week after the previous release.

RcppCCTZ uses Rcpp to bring CCTZ to R. CCTZ is a C++ library for translating between absolute and civil times using the rules of a time zone. In fact, it is two libraries. One for dealing with civil time: human-readable dates and times, and one for converting between between absolute and civil times via time zones. And while CCTZ is made by Google(rs), it is not an official Google product. The RcppCCTZ page has a few usage examples and details. This package was the first CRAN package to use CCTZ; by now at least three others do—but decided in their infinite wisdom to copy the sources yet again into their packages. Sigh.

This version corrects two bugs. We were not properly accounting for those poor systems that do not natively have nanosecond resolution. And I missed a feature in the Rcpp DatetimeVector class by not setting the timezone on newly created variables; this too has been fixed.

Changes in version 0.2.5 (2018-10-14)

  • Parsing to Datetime was corrected on systems that do not have nanosecond support in C++11 chrono (#28).

  • DatetimeVector objects are now created with their timezone attribute when available.

  • The toTz functions is now vectorized (#29).

  • More unit tests were added, and some conditioning on Solaris (mostly due to missing timezone info) was removed.

We also have a diff to the previous version thanks to CRANberries. More details are at the RcppCCTZ page; code, issue tickets etc at the GitHub repository.

This post by Dirk Eddelbuettel originated on his Thinking inside the box blog. Please report excessive re-aggregation in third-party for-profit settings.

14 October, 2018 11:04PM

Jeremy Bicha

Google Cloud Print in Ubuntu

There is an interesting hidden feature available in Ubuntu 18.04 LTS and newer. To enable this feature, first install cpdb-backend-gcp.

sudo apt install cpdb-backend-gcp

Make sure you are signed in to Google with GNOME Online Accounts. Open the Settings app1 to the Online Accounts page. If your Google account is near the top above the Add an account section, then you’re all set.

Currently, only LibreOffice is supported. Hopefully, for 19.04, other GTK+ apps will be able to use the feature.

This feature was developed by Nilanjana Lodh and Abhijeet Dubey when they were Google Summer of Code 2017 participants. Their mentors were Till Kamppeter, Aveek Basu, and Felipe Borges.

Till has been trying to get this feature installed by default in Ubuntu since 18.04 LTS, but it looks like it won’t make it in until 19.04.

I haven’t seen this feature packaged in any other Linux distros yet. That might be because people don’t know about this feature so that’s why I’m posting about it today! If you are a distro packager, the 3 packages you need are cpdb-libs , cpdb-backend-gcp, and cpdb-backend-cups. The final package enables easy printing to any IPP printer. (I didn’t mention it earlier because I believe Ubuntu 18.04 LTS already supports that feature through a different package.)

Save to Google Drive

In my original blog post, I confused the cpdb feature with a feature that already exists in GTK3 built with GNOME Online Accounts support. This should already work on most distros.

When you print a document, there will be an extra Save to Google Drive option. Saving to Google Drive saves a PDF of your document to your Google Drive account.

This post was edited on October 16 to mention that cpdb only supports LibreOffice now and that Save to Google Drive is a GTK3 feature instead.

October 17: Please see Felipe’s comments. It turns out that even Google Cloud Print works fine in distros with recent GTK3. The point of the cpdb feature is to make this work in apps that don’t use GTK3. So I guess the big benefit now is that you can use Google Cloud Print or Save to Google Drive from LibreOffice.

14 October, 2018 02:31PM by Jeremy Bicha

October 13, 2018

Julian Andres Klode

The demise of G+ and return to blogging (w/ mastodon integration)

I’m back to blogging, after shutting down my wordpress.com hosted blog in spring. This time, fully privacy aware, self hosted, and integrated with mastodon.

Let’s talk details: In spring, I shutdown my wordpress.com hosted blog, due to concerns about GDPR implications with comment hosting and ads and stuff. I’d like to apologize for using that, back when I did this (in 2007), it was the easiest way to get into blogging. Please forgive me for subjecting you to that!

Recently, Google announced the end of Google+. As some of you might know, I posted a lot of medium-long posts there, rather than doing blog posts; especially after I disabled the wordpress site.

With the end of Google+, I want to try something new: I’ll host longer pieces on this blog, and post shorter messages on @juliank@mastodon.social. If you follow the Mastodon account, you will see toots for each new blog post as well, linking to the blog post.

Mastodon integration and privacy

Now comes the interesting part: If you reply to the toot, your reply will be shown on the blog itself. This works with a tiny bit of JavaScript that talks to a simple server-side script that finds toots from me mentioning the blog post, and then replies to that.

This protects your privacy, because mastodon.social does not see which blog post you are looking at, because it is contacted by the server, not by you. Rendering avatars requires loading images from mastodon.social’s file server, however - to improve your privacy, all avatars are loaded with referrerpolicy='no-referrer', so assuming your browser is half-way sane, it should not be telling mastodon.social which post you visited either. In fact, the entire domain also sets Referrer-Policy: no-referrer as an http header, so any link you follow will not have a referrer set.

The integration was originally written by @bjoern@mastodon.social – I have done some moderate improvements to adapt it to my theme, make it more reusable, and replace and extend the caching done in a JSON file with a Redis database.

Source code

This blog is free software; generated by the Hugo snap. All source code for it is available:

(Yes I am aware that hosting the repositories on GitHub is a bit ironic given the whole focus on privacy and self-hosting).

The theme makes use of Hugo pipes to minify and fingerprint JavaScript, and vendorizes all dependencies instead of embedding CDN links, to, again, protect your privacy.

Future work

I think I want to make the theme dark, to be more friendly to the eyes. I also might want to make the mastodon integration a bit more friendly to use. And I want to get rid of jQuery, it’s only used for a handful of calls in the Mastodon integration JavaScript.

If you have any other idea for improvements, feel free to join the conversation in the mastodon toot, send me an email, or open an issue at the github projects.

Closing thoughts

I think the end of Google+ will be an interesting time, requring a lot of people in the open source world to replace one of their main communication channels with a different approach.

Mastodon and Diaspora are both in the race, and I fear the community will split or everyone will have two accounts in the end. I personally think that Mastodon + syndicated blogs provide a good balance: You can quickly write short posts (up to 500 characters), and you can host long articles on your own and link to them.

I hope that one day diaspora* and mastodon federate together. If we end up with one federated network that would be the best outcome.

13 October, 2018 09:03PM

Ingo Juergensmann

Xen & Databases

I'm running PostgreSQL and MySQL on my server that both serve different databases to Wordpress, Drupal, Piwigo, Friendica, Mastodon, whatever...

In the past the databases where colocated in my mailserver VM whereas the webserver was running on a different VM. Somewhen I moved the databases from domU to dom0, maybe because I thought that the databases would be faster running on direct disk I/O in the dom0 environment, but can't remember the exact rasons anymore.

However, in the meantime the size of the databases grew and the number of the VMs did, too. MySQL and PostgreSQL are both configured/optimized to run with 16 GB of memory in dom0, but in the last months I experienced high disk I/O especially for MySQL and slow I/O performance in all the domU VMs because of that.

Currently iotop shows something like this:

Total DISK READ :     131.92 K/s | Total DISK WRITE :    1546.42 K/s
Actual DISK READ:     131.92 K/s | Actual DISK WRITE:       2.40 M/s
  TID  PRIO  USER     DISK READ  DISK WRITE  SWAPIN     IO>    COMMAND
 6424 be/4 mysql       0.00 B/s    0.00 B/s  0.00 % 60.90 % mysqld
18536 be/4 mysql      43.97 K/s   80.62 K/s  0.00 % 35.59 % mysqld
 6499 be/4 mysql       0.00 B/s   29.32 K/s  0.00 % 13.18 % mysqld
20117 be/4 mysql       0.00 B/s    3.66 K/s  0.00 % 12.30 % mysqld
 6482 be/4 mysql       0.00 B/s    0.00 B/s  0.00 % 10.04 % mysqld
 6495 be/4 mysql       0.00 B/s    3.66 K/s  0.00 % 10.02 % mysqld
20144 be/4 postgres    0.00 B/s   73.29 K/s  0.00 %  4.87 % postgres: hubzilla hubzi~
 2920 be/4 postgres    0.00 B/s 1209.28 K/s  0.00 %  3.52 % postgres: wal writer process
11759 be/4 mysql       0.00 B/s   25.65 K/s  0.00 %  0.83 % mysqld
18736 be/4 mysql       0.00 B/s   14.66 K/s  0.00 %  0.17 % mysqld
21768 be/4 root        0.00 B/s    0.00 B/s  0.00 %  0.02 % [kworker/1:0]
 2922 be/4 postgres    0.00 B/s   69.63 K/s  0.00 %  0.00 % postgres: stats collector process

MySQL data site is below configured max memory size for MySQL, so everything should more or less fit into memory. Yet, there is still a large amount of disk I/O by MySQL, much more than by PostgreSQL. Of course there is much I/O done by writes to the database.

However, I'm thinking of changing my setup again back to domU based database setup again, maybe one dedicated VM for both DBMS' or even two dedicated VMs for each of them? I'm not quite sure how Xen reacts to the current work load?

Back in the days when I did 3D computer graphic I did a lot of testing with different settings in regards of priorities and such. Basically one would think that giving the renderer more CPU time would speed of the rendering, but this turned out to be wrong: the higher the render tasks priority was, the slower the rendering got, because disk I/O (and other tasks that were necessary for the render task to work) got slowed down. When running the render task at lowest priority all the other necessary tasks could run on higher speed and return the CPU more quickly, which resulted in shorter render times.

So, maybe I experience something similar with the databases on dom0 here as well: dom0 is busy doing database work and this slows down all the other tasks (== domU VMs). When I would move databases back to domU this would enable dom0 again to better do its basic job of taking care of the domUs?

Of course, this is also a quite philosophical question, but what is the recommended setup? Is it better to separate the databases in two different VMs or just one? Or is running the databases on dom0 the best option?

I'm interested in your feedback, so please comment! :-)

UPDATE: you can also contact me @ij@nerdculture.de on Mastodon or on Friendica at https://nerdica.net/profile/ij

Kategorie: 
Debian
Tags: 
Debian
MySQL
PostgreSQL
Xen
 

13 October, 2018 07:21PM by ij

Jeremy Bicha

Shutter removed from Debian & Ubuntu

This week, the popular screenshot app Shutter was removed from Debian Unstable & Ubuntu 18.10. (It had already been removed from Debian “Buster” 6 months ago and some of its “optional” dependencies had already been removed from Ubuntu 18.04 LTS).

Shutter will need to be ported to gtk3 before it can return to Debian. (Ideally, it would support Wayland desktops too but that’s not a blocker for inclusion in Debian.)

See the Debian bug for more discussion.

I am told that flameshot is a nice well-maintained screenshot app.

I believe Snap or Flatpak are great ways to make apps that use obsolete libraries available on modern distros that can no longer keep those libraries around. There isn’t a Snap or Flatpak version of Shutter yet, so hopefully someone interested in that will help create one.

13 October, 2018 06:29PM by Jeremy Bicha

hackergotchi for Dirk Eddelbuettel

Dirk Eddelbuettel

RcppNLoptExample 0.0.1: Use NLopt from C/C++

A new package of ours, RcppNLoptExample, arrived on CRAN yesterday after a somewhat longer-than-usual wait for new packages as CRAN seems really busy these days. As always, a big and very grateful Thank You! for all they do to keep this community humming.

So what does it do ?

NLopt is a very comprehensive library for nonlinear optimization. The nloptr package by Jelmer Ypma has long been providing an excellent R interface.

Starting with its 1.2.0 release, the nloptr package now exports several C symbols in a way that makes it accessible to other R packages without linking easing the installation on all operating systems.

The new package RcppNLoptExample illustrates this facility with an example drawn from the NLopt tutorial. See the (currently single) file src/nlopt.cpp.

How / Why ?

R uses C interfaces. These C interfaces can be exported between packages. So when the usual library(nloptr) (or an import via NAMESPACE) happens, we now also get a number of C functions registered.

And those are enough to run optimization from C++ as we simply rely on the C interface provided. Look careful at the example code: the objective function and the constraint functions are C functions, and the body of our example invokes C functions from NLopt. This just works. For either C code, or C++ (where we rely on Rcpp to marshal data back and forth with ease).

On the other hand, if we tried to use the NLopt C++ interface which brings with it someinterface code we would require linking to that code (which R cannot easily export across packages using its C interface). So C it is.

Status

The package is pretty basic but fully workable. Some more examples should get added, and a helper class or two for state would be nice. The (very short) NEWS entry follows:

Changes in version 0.0.1 (2018-10-01)

  • Initial basic package version with one example from NLopt tutorial

Code, issue tickets etc are at the GitHub repository.

This post by Dirk Eddelbuettel originated on his Thinking inside the box blog. Please report excessive re-aggregation in third-party for-profit settings.

13 October, 2018 02:56PM

Elana Hashman

PyGotham 2018 Talk Resources

At PyGotham in 2018, I gave a talk called "The Black Magic of Python Wheels". I based this talk on my two years of work on auditwheel and the manylinux platform, hoping to share some dark details of how the proverbial sausage is made.

It was a fun talk, and I enjoyed the opportunity to wear my Python Packaging Authority hat:

The Black Magic of Python Wheels

Follow-up readings

All the PEPs referenced in the talk

In increasing numeric order.

  • PEP 376 "Database of Installed Python Distributions"
  • PEP 426 "Metadata for Python Software Packages 2.0"
  • PEP 427 "The Wheel Binary Package Format 1.0"
  • PEP 513 "A Platform Tag for Portable Linux Built Distributions" (aka manylinux1)
  • PEP 571 "The manylinux2010 Platform Tag"

Image licensing info

13 October, 2018 04:00AM by Elana Hashman

hackergotchi for Dirk Eddelbuettel

Dirk Eddelbuettel

GitHub Streak: Round Five

Four years ago I referenced the Seinfeld Streak used in an earlier post of regular updates to to the Rcpp Gallery:

This is sometimes called Jerry Seinfeld’s secret to productivity: Just keep at it. Don’t break the streak.

and then showed the first chart of GitHub streaking

github activity october 2013 to october 2014github activity october 2013 to october 2014

And three year ago a first follow-up appeared in this post:

github activity october 2014 to october 2015github activity october 2014 to october 2015

And two years ago we had a followup

github activity october 2015 to october 2016github activity october 2015 to october 2016

And last year we another one

github activity october 2016 to october 2017github activity october 2016 to october 2017

As today is October 12, here is the newest one from 2017 to 2018:

github activity october 2017 to october 2018github activity october 2017 to october 2018

Again, special thanks go to Alessandro Pezzè for the Chrome add-on GithubOriginalStreak.

This post by Dirk Eddelbuettel originated on his Thinking inside the box blog. Please report excessive re-aggregation in third-party for-profit settings.

13 October, 2018 01:11AM

October 12, 2018

hackergotchi for Bastian Venthur

Bastian Venthur

Introducing Litestats

Profiling in Python has always been easy, however, analyzing the profiler's output not so much. After the profile has been created you can use Python's pstats module but it feels quite clumsy and not really empowering. For Python 2 there has been RunSnakeRun, a very convenient tool for analyzing the profiler output, unfortunately that tool hasn't been updated since 2014. I recently ported it to Python3 and wxPython4 but I'm probably not going to maintain that code properly as I'm not very comfortable with wxPython.

I still wanted something nicer than pstats for profiling so I wrote litestats. Litestats is a simple command line tool that takes the output of the Python profiler and transforms the data into a sqlite3 database. You can now easily analyze the profiler output using sqlite on the command line, the sqlitebrowser for a graphical interface or use the data base as the foundation of your very own tooling around the analysis.

How does it work?

Litestats reads the dump of the profiler and creates a normalized data base with tree tables:

  • functions: contains each function (callers and callees) with filename, line number and function name
  • stats contains the statistics (primitive/total calls, total/cumulative time) for all functions
  • calls a caller-callee mapping

While this provides an exact representation of the dump, those tables would be cumbersome to use. So litestats additionally creates three views emulating pstats print_stats(), print_callers() and print_callees() functionality:

  • pstats
  • callers
  • callees

Installation

Litestats has no requirements other than Python itself and is available on PyPI:

$ pip install litestats

Usage

$ # run the profiler and dump the output
$ python3 -m cProfile -o example.prof example.py
$ # convert dump to sqlite3 db
$ litestats example.prof
$ # example.prof.sqlite created

You can now use the sqlite3 data base to investigate the profiler dump:

sqlite> select *
   ...> from pstats
   ...> order by cumtime desc
   ...> limit 20;

ncalls      tottime     ttpercall             cumtime     ctpercall   filename:lineno(function)
----------  ----------  --------------------  ----------  ----------  ------------------------------------
18/1        0.000161    8.94444444444444e-06  0.067797    0.067797    ~:0(<built-in method builtins.exec>)
1           1.0e-06     1.0e-06               0.067755    0.067755    <string>:1(<module>)
1           4.0e-06     4.0e-06               0.067754    0.067754    /usr/lib/python3.7/runpy.py:195(run_
1           6.0e-06     6.0e-06               0.066135    0.066135    /usr/lib/python3.7/runpy.py:62(_run_
1           1.1e-05     1.1e-05               0.066113    0.066113    /home/venthur/Documents/projects/lit
1           6.6e-05     6.6e-05               0.055152    0.055152    /home/venthur/Documents/projects/lit
1           4.1e-05     4.1e-05               0.0549      0.0549      /home/venthur/Documents/projects/lit
1           0.050196    0.050196              0.050196    0.050196    ~:0(<method 'executescript' of 'sqli
20/3        8.9e-05     4.45e-06              0.011064    0.003688    <frozen importlib._bootstrap>:978(_f
20/3        4.8e-05     2.4e-06               0.011005    0.00366833  <frozen importlib._bootstrap>:948(_f
20/3        7.5e-05     3.75e-06              0.01083     0.00361     <frozen importlib._bootstrap>:663(_l
15/3        3.5e-05     2.33333333333333e-06  0.01073     0.00357666  <frozen importlib._bootstrap_externa
29/5        2.5e-05     8.62068965517241e-07  0.010215    0.002043    <frozen importlib._bootstrap>:211(_c
3           6.0e-06     2.0e-06               0.010087    0.00336233  ~:0(<built-in method builtins.__impo
28/6        9.0e-06     3.21428571428571e-07  0.008977    0.00149616  <frozen importlib._bootstrap>:1009(_
1           9.0e-06     9.0e-06               0.00841     0.00841     /home/venthur/Documents/projects/lit
16          0.000138    8.625e-06             0.004802    0.00030012  <frozen importlib._bootstrap_externa
1           4.5e-05     4.5e-05               0.004143    0.004143    /usr/lib/python3.7/logging/__init__.
1           0.004038    0.004038              0.004038    0.004038    ~:0(<method 'commit' of 'sqlite3.Con
13          3.3e-05     2.53846153846154e-06  0.002368    0.00018215  <frozen importlib._bootstrap_externa

12 October, 2018 10:00PM by Bastian Venthur

hackergotchi for Dirk Eddelbuettel

Dirk Eddelbuettel

binb 0.0.3: Now with Monash

The third release of the binb package just arrived on CRAN, and it comes with a new (and very crispy) theme: Monash. With that we are also thrilled to welcome Rob Hyndman as a co-author.

Here is a quick demo combining all (by now four) themes:

Also, Ista made the IQSS theme more robust to font selection. Other changes:

Changes in binb version 0.0.3 (2018-10-12)

  • The IQSS theme now has a fallback font if Libertinus is unavailable (Ista in #7)

  • Added support for 'Monash' theme (Rob Hyndman in #10 and #11 closing #9)

  • Simplified some options for the 'Monash' theme (Dirk in #13)

  • The IQSS theme can now set an alternate titlegraphic (Ista in #14)

CRANberries provides the usual summary of changes to the previous version.

For questions or comments use the issue tracker off the GitHub repo.

This post by Dirk Eddelbuettel originated on his Thinking inside the box blog. Please report excessive re-aggregation in third-party for-profit settings.

12 October, 2018 12:36PM

October 11, 2018

Deepanshu Gajbhiye

Google Summer of code at Debian Final Report

Google Summer of code 2018

Table of contents

  1. Introduction
  2. Final Summary
  3. Deliverable
  4. Weekly reports & Blog posts
  5. Other contributions
  6. Thank you

Introduction

Virtual LTSP server project automates installation and configuration of LTSP server with vagrant. It is the easiest way to create LTSP setup. We have developed the project to do the same for Linux mint 19 and Debian 9. We also created several scripts for testing, create ltsp client, manage accounts, etc. Also created packer scripts to create vagrant boxes that we will use in the project.

Final Summary

Google Summer of Code was a great opportunity to work with really smart and amazing people. I learned a lot in the process. It took my understanding of Vagrant, bash-scripting, packer, ltsp, Debian packaging to a whole another level. I started with a basic provisioner script to install ltsp to vagrant box. Then we make several improvements in it.

Later made several features and improvements in it. After that a major issue was the client was unable to boot. In order to solve this issue I searched through all the content on the internet about ltsp. Even asked the active developers of ltsp on how to fix this issue. They have been working on ltsp since 2006 but they haven’t encountered this problem yet. After struggling for lots of I solved it! Looking at the complexity and the time it took mentors told me to write a separate blog post about it. We have also created Virtual ltsp server for Debian 9. Also one for linux mint 19.

I had to create a new linux mint vagrant box with xfce. Which was really fun. I also automated its creation with packer scripts. We also did port from Edubuntu packages to Debian. It is locally built and installed via a small script. In the end, we added features like automatic login, guest login, and several scripts to optimize the workflow for the user. This was a really short summary of the work done. More details can be found on the weekly reports.

Deliverables

Virtual LTSP Server

docker-scripts/dev--LTSP

Pull request made

docker-scripts/dev--LTSP

Commits made

  • bionic branch —
  • buster branch —

Issues worked on

docker-scripts/dev--LTSP

Packer scripts to create vagrant box

docker-scripts/dev--LTSP

Linux mint tara vagrant box

Vagrant Cloud by HashiCorp

Ported edubuntu packages from ubuntu to debian

Documentation

docker-scripts/dev--LTSP

Weekly reports & Blog posts

  • Week1:

GSoC weekly report of Deepanshu Gajbhiye week 1

  • Week2:
  • Week3:
  • Week4:
  • Week5:
  • Week6:
  • Week7:
  • Week8:
  • Week9:
  • Week10:
  • Week11:
  • Week12:

Other contributions

https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=905006

Thank you

I am very thankful to Google and Debian for accepting me to Google Summer of Code. Working for GSoC was an amazing experience. I will definately participate next year as well.

Thank to my mentors Dashamir Hoxha and Akash Shende for their solid support, quick response, patience and trust on me.

Thank you Daniel Pocock for encouragement and Big thanks to Debian, ltsp, Vagrant community for being very helpful.

11 October, 2018 04:26PM by Deepanshu

Antoine Beaupré

Archived a part of my CD collection

After about three days of work, I've finished archiving a part of my old CD collection. There were about 200 CDs in a cardboard box that were gathering dust. After reading Jonathan Dowland's post about CD archival, I got (rightly) worried it would be damaged beyond rescue so I sat down and did some research on the rescue mechanisms. My notes are in rescue and I hope to turn this into a more agreeable LWN article eventually.

I post this here so I can put a note in the box with a permanent URL for future reference as well.

Remaining work

All the archives created were dumped in the ~/archive or ~/mp3 directories on curie. Data needs to be deduplicated, replicated, and archived somewhere more logical.

Inventory

I have a bunch of piles:

  • a spindle of disks that consists mostly of TV episodes, movies, distro and Windows images/ghosts. not imported.
  • a pile of tapes and Zip drives. not imported.
  • about fourty backup disks. not imported.
  • about five "books" disks of various sorts. ISOs generated. partly integrated in my collection, others failed to import or were in formats that were considered non-recoverable
  • a bunch of orange seeds piles
    • Burn Your TV masters and copies
    • apparently live and unique samples - mostly imported in mp3
    • really old stuff with tons of dupes - partly sorted through, in jams4, reste still in the pile
  • a pile of unidentified disks

All disks were eventually identified as trash, blanks, perfect, finished, defective, or not processed. A special needs attention stack was the "to do" pile, and would get sorted through the other piles. each pile was labeled with a sticky note and taped together summarily.

A post-it pointing to the blog post was included in the box, along with a printed version of the blog post summarizing a snapshot of this inventory.

Here is a summary of what's in the box.

Type Count Note
trash 13 non-recoverable. not detected by the Linux kernel at all and no further attempt has been made to recover them.
blanks 3 never written to, still usable
perfect 28 successfully archived, without errors
finished 4 almost perfect: but mixed-mode or multi-session
defective 21 found to have errors but not considered important enough to re-process
total 69
not processed ~100 visual estimate

11 October, 2018 03:52PM

hackergotchi for Mario Lang

Mario Lang

"Learning from machines" by Ashi Krishnan

I need to share this talk by Ashi Krishnan because I think it is very well done, and I find the content extremely fascinating and interesting.

If you are into consciousness and/or dreaming, do yourself a favour and allocate 40 minutes of your life for this one.

Thanks Ashi, you made my day!

11 October, 2018 09:00AM by Mario Lang

hackergotchi for Lars Wirzenius

Lars Wirzenius

On flatpak, snap, distros, and software distribution

I don't think any of Flatpak, Snappy, traditional Linux distros, non-traditional Linux distros, containers, online services, or other forms of software distribution are a good solution for all users. They all fail in some way, and each of them requires continued, ongoing effort to be acceptable even within their limitations.

This week, there's been some discussion about Flatpak, a software distribution approach that's (mostly) independent of traditional Linux distributions. There's also, Snappy, which is Canonical's similar thing.

The discussion started with the launch of a new website attacking Flatpak as a technology. I'm not going to link to it, since it's an anonymous attack and rant, and not constructive. I'd rather have a constructive discussion. I'm also not going to link to rebuttals, and will just present my own view, which I hope is different enough to be interesting.

The website raises the issue that Flatpak's sandboxing is not as good as it should be. This seems to be true. Some of Flatpak's defenders respond that it's an evolving technology, which seems fair. It's not necessary to be perfect; it's important to be better than what came before, and to constantly improve.

The website also raises the point that a number of flatpaks themselves contain unfixes security problems. I find this to be more worrying than an imperfect sandbox. A security problem inside a perfect sandbox can still be catastrophic: it can leak sensitive data, join a distributed denial of service attack, use excessive CPU and power, and otherwise cause mayhem. The sandbox may help in containing the problem somewhat, but to be useful for valid use, the sandbox needs to allow things that can be used maliciously.

As a user, I want software that's...

  • easy to install and update
  • secure to install (what I install is what the developers delivered)
  • always up to date with security fixes, including for any dependencies (embedded in the software or otherwise)
  • reasonably up to date with other bug fixes
  • sufficiently up to date with features I want (but I don't care a about newer features that I don't have a use for)
  • protective of my freedoms and privacy and other human rights, which includes (but is not restricted to) being able to self-host services and work offline

As a software developer, I additionally want my own software to be...

  • effortless to build
  • automatically tested in a way that gives me confidence it works for my users
  • easy to deliver to my users
  • easy to debug
  • not be broken by changes to build and runtime dependencies, or at least make such changes be extremely obvious, meaning they result in a build error or at least an error during automated tests

These are requirements that are hard to satisfy. They require a lot of manual effort, and discipline, and I fear the current state of software development isn't quite there yet. As an example, the Linux kernel development takes great care to never break userland, but that requires a lot of care when making changes, a lot of review, and a lot of testing, and a willingness to go to extremes to achieve that. As a result, upgrading to a newer kernel version tends to be a low-risk operation. The glibc C library, used by most Linux distributions, has a similar track record.

But Linux and glibc are system software. Flatpak is about desktop software. Consider instead LibreOffice, the office suite. There's no reason why it couldn't be delivered to users as a Flatpak (and indeed it is). It's a huge piece of software, and it needs a very large number of libraries and other dependencies to work. These need to be provided inside the LibreOffice Flatpak, or by one or more of the Flatpak "runtimes", which are bundles of common dependencies. Making sure all of the dependencies are up to date can be partly automated, but not fully: someone, somewhere, needs to make the decision that a newer version is worth upgrading to right now, even if it requires changes in LibreOffice for the newer version to work.

For example, imagine LO uses a library to generate PDFs. A new version of the library reduces CPU consumption by 10%, but requires changes, because the library's API (programming interface) has changed radically. The API changes are necessary to allow the speedup. Should LibreOffice upgrade to the new version of not? If 10% isn't enough of a speedup to warrant the effort to make the LO changes, is 90%? An automated system could upgrade the library, but that would then break the LO build, resulting in something that doesn't work anymore.

Security updates are easier, since they usually don't involve API changes. An automated system could upgrade dependencies for security updates, and then trigger automated build, test, and publish of a new Flatpak. However, this is made difficult by there is often no way to automatically, reliably find out that there is a security fix released. Again, manual work is required to find the security problem, to fix it, to communicate that there is a fix, and to upgrade the dependency. Some projects have partial solutions for that, but there seems to be nothing universal.

I'm sure most of this can be solved, some day, in some manner. It's definitely an interesting problem area. I don't have a solution, but I do think it's much too simplistic to say "Flatpaks will solve everything", or "the distro approach is best", or "just use the cloud".

11 October, 2018 08:24AM

hackergotchi for Norbert Preining

Norbert Preining

Debian/TeX Live updates 20181009

MOre than a month has passed, and we went through a CVE and some other complications, but finally I managed to build and upload a new set of packages of TeX Live for Debian.

During this update some color profiles (icc) that had unclear licenses have been removed, which for now creates problems with the pdfx package. So if you use the pdfx package, please explicitly specify a color profile. The next upload will again allow using pdfx without specifying a profile in which case a default profile is used. I have uploaded already a set of free profiles to CTAN and they arrived in TeX Live, but pdfx package isn’t updated till now.

Other than this I don’t recall anything spectacular new or changed, but it is still a long list of packages being updated 😉

Please enjoy.

New packages

bxwareki, chs-physics-report, ctanbib, dehyph, firamath, firamath-otf, jigsaw, kalendarium, kvmap, libertinus, libertinus-fonts, libertinus-type1, metapost-colorbrewer, pst-feyn, pst-lsystem, pst-marble, ptex-manual, quantikz, rank-2-roots, tex-locale, utexasthesis, widows-and-orphans.

Updated packages

achemso, apa6, arabluatex, archaeologie, arydshln, axodraw2, babel, babel-belarusian, babel-french, bangorcsthesis, beamer, beamerswitch, bezierplot, biblatex-anonymous, biblatex-chem, biblatex-ext, biblatex-manuscripts-philology, biblatex-publist, bidi, breqn, bxjscls, bxorigcapt, bxwareki, caption, catechis, clrstrip, cochineal, context-handlecsv, cooking-units, covington, csplain, cstex, doi, ducksay, duckuments, dvipdfmx, eplain, epstopdf, europecv, exercisebank, fei, fira, fontawesome5, gentombow, hyperref, hyphen-german, hyphen-latin, hyphen-thai, hyph-utf8, ifluatex, jadetex, jlreq, jslectureplanner, l3build, l3experimental, l3kernel, l3packages, latex-bin, latexindent, latex-make, lettrine, libertinus-otf, libertinust1math, libertinus-type1, listings, lshort-chinese, lualibs, luamplib, luaotfload, luatexja, lwarp, make4ht, memdesign, mltex, mptopdf, nicematrix, nimbus15, oberdiek, ocgx2, onedown, overpic, parskip, pdftex, pdfx, perception, platex, platex-tools, plautopatch, poetry, pst-eucl, pst-plot, pstricks, ptex, reledmac, returntogrid, sourceserifpro, svg, tableof, tetex, tex4ht, textualicomma, thesis-gwu, thumbpdf, tkz-base, tkz-doc, tkz-graph, tkz-kiviat, tkz-linknodes, tkz-tab, tlcockpit, tugboat, tugboat-plain, ucsmonograph, ulthese, univie-ling, updmap-map, uri, witharrows, xcharter, xepersian, xetex, xits, xmltex, yafoot, zhlipsum.

11 October, 2018 06:28AM by Norbert Preining

October 10, 2018

hackergotchi for Neil Williams

Neil Williams

Code Quality & Formatting for Python

I've recently added two packages (and their dependencies) to Debian and thought I'd cover a bit more about why.

Black

black, the uncompromising Python code formatter, has arrived in Debian unstable and testing.

black is being adopted by the LAVA Software Community Project in a gradual way and the new CI will be checking that files which have been formatted by black stay formatted by black in merge requests.

There are endless ways to format Python code and pycodestyle and pylint are often too noisy to use without long lists of ignored errors and warnings. Black takes the stress out of maintaining a large Python codebase as long as a few simple steps are taken:

  • Changes due to black are not functional changes. A merge request applying black to a source code file must not include functional changes. Just the change done by black. This makes code review manageable.
  • Changes made by black are recorded and once made, CI is used to ensure that there are no regressions.
  • Black is only run on files which are not currently being changed in existing merge requests. This is a simple sanity provision, rebasing functional changes after running black is not fun.

Consistent formatting goes a long way to helping humans spot problematic code.

See https://black.readthedocs.io/en/stable/ or apt-get install python-black-doc for a version which doesn't "call home".

Radon

So much for code formatting, that's nice and all but what can matter more is an overview of the complexity of the codebase.

We're experimenting with running radon as part of our CI to get a CodeClimate report which GitLab should be able to understand.

https://docs.codeclimate.com/docs/workflow

https://git.metabarcoding.org/help/ci/examples/code_climate.md

(Take a bow http://vincentsanders.blogspot.com/2018/09/all-i-wanted-to-do-is-check-error-code.html - Vince gave me the idea by mentioning his use of Cyclomatic Complexity.)

What we're hoping to achieve here is a failed CI test if the complexity of critical elements increases and a positive indication if the code complexity of areas which are currently known to be complex can be reduced without losing functionality.

Initially, just having the data is a bonus. The first try at CodeClimate support took the best part of an hour to scan our code repository. radon took 3 seconds.

See https://radon.readthedocs.io/en/latest/ or apt-get install python-radon-doc for a version which doesn't "call home".

(It would be really nice for upstreams to understand that putting badges in their sphinx documentation templates makes things harder to distribute fairly. Fine, have a nice web UI for your own page but remove the badges from the pages in the released tarballs, e.g. with a sphinx build time option.)

One more mention - bandit

I had nothing to do with introducing this to Debian but I am very grateful that it exists in Debian. bandit is proving to be very useful in our CI, providing SAST reports in GitLab. As with many tools of it's kind, it is noisy at first. However, with a few judicious changes and the use of the # nosec comment to rule out scanning of things like unit tests which deliberately tried to be insecure, we have substantially reduced the number of reports produced with bandit.

Having the tools available is so important to actually fixing problems before the software gets released.

10 October, 2018 01:00PM by Neil Williams

hackergotchi for Michal &#268;iha&#345;

Michal Čihař

Weblate 3.2.1

Weblate 3.2.1 has been released today. It's a bugfix release for 3.2 fixing several minor issues which appeared in the release.

Full list of changes:

  • Document dependency on backports.csv on Python 2.7.
  • Fix running tests under root.
  • Improved error handling in gitexport module.
  • Fixed progress reporting for newly added languages.
  • Correctly report Celery worker errors to Sentry.
  • Fixed creating new translations with Qt Linguist.
  • Fixed occasional fulltext index update failures.
  • Improved validation when creating new components.
  • Added support for cleanup of old suggestions.

If you are upgrading from older version, please follow our upgrading instructions.

You can find more information about Weblate on https://weblate.org, the code is hosted on Github. If you are curious how it looks, you can try it out on demo server. Weblate is also being used on https://hosted.weblate.org/ as official translating service for phpMyAdmin, OsmAnd, Turris, FreedomBox, Weblate itself and many other projects.

Should you be looking for hosting of translations for your project, I'm happy to host them for you or help with setting it up on your infrastructure.

Further development of Weblate would not be possible without people providing donations, thanks to everybody who have helped so far! The roadmap for next release is just being prepared, you can influence this by expressing support for individual issues either by comments or by providing bounty for them.

Filed under: Debian English SUSE Weblate

10 October, 2018 11:00AM

hackergotchi for Lars Wirzenius

Lars Wirzenius

New job: WMF release engineering

I've started my new job. I now work in the release engineering team at Wikimedia, the organisation that runs sites such as Wikipedia. We help put new versions of the software that runs the sites into production. My role is to help make that process more automated and frequent.

10 October, 2018 06:07AM

October 09, 2018

hackergotchi for Benjamin Mako Hill

Benjamin Mako Hill

hackergotchi for Holger Levsen

Holger Levsen

20181008-lts-201809

My LTS work in September

In September I only managed to spend 2.5h working on jessie LTS on:

  • finishing work on patches for samba, but then failed to release the DLA for it until now. Expect an upload soon. Sorry for the delay, various RL issues took their toll.

09 October, 2018 03:49PM

hackergotchi for Markus Koschany

Markus Koschany

My Free Software Activities in September 2018

Welcome to gambaru.de. Here is my monthly report that covers what I have been doing for Debian. If you’re interested in Java, Games and LTS topics, this might be interesting for you.

Debian Games

  • Yavor Doganov continued his heroics in September and completed the port to GTK 3 of teg, a risk-like game. (#907834) Then he went on to fix gnome-breakout.
  • I packaged a new upstream release of freesweep, a minesweeper game, which fixed some minor bugs but unfortunately not #907750.
  • I spent most of the time this month on packaging a newer upstream version of unknown-horizons, a strategy game similar to the old Anno games. After also upgrading the fife engine, fifechan and NMUing python-enet, the game is up-to-date again.
  • More new upstream versions this month: atomix, springlobby, pygame-sdl2, and renpy.
  • I updated widelands to fix an incomplete appdata file (#857644) and to make the desktop icon visible again.
  • I enabled gconf support in morris (#908611) again because gconf will be supported in Buster.
  • Drascula, a classic adventure game, refused to start because of changes to the ScummVM engine. It is working now. (#908864)
  • In other news I backported freeorion to Stretch and sponsored a new version of the runescape wrapper for Carlos Donizete Froes.

Debian Java

  • Only late in September I found the time to work on JavaFX but by then Emmanuel Bourg had already done most of the work and upgraded OpenJFX to version 11. We now have a couple of broken packages (again) because JavaFX is no longer tied to the JRE but is designed more like a library. Since most projects still cling to JavaFX 8 we have to fix several build systems by accommodating those new circumstances.  Surely there will be more to report next month.
  • A Ubuntu user reported that importing furniture libraries was no longer possible in sweethome3d (LP: #1773532) when it is run with OpenJDK 10. Although upstream is more interested in supporting Java 6, another user found a fix which I could apply too.
  • New upstream versions this month: jboss-modules, libtwelvemonkeys-java, robocode, apktool, activemq (RC #907688), cup and jflex. The cup/jflex update required a careful order of uploads because both packages depend on each other. After I confirmed that all reverse-dependencies worked as expected, both parsers are up-to-date again.
  • I submitted two point updates for dom4j and tomcat-native to fix several security issues in Stretch.

Misc

  • Firefox 60 landed in Stretch which broke all xul-* based browser plugins. I thought it made sense to backport at least two popular addons, ublock-origin and https-everywhere, to Stretch.
  • I also prepared another security update for discount (DSA-4293-1) and uploaded  libx11 to Stretch to fix three open CVE.

Debian LTS

This was my thirty-first month as a paid contributor and I have been paid to work 29,25 hours on Debian LTS, a project started by Raphaël Hertzog. In that time I did the following:

  • From 24.09.2018 until 30.09.2018 I was in charge of our LTS frontdesk. I investigated and triaged CVE in dom4j, otrs2, strongswan, python2.7, udisks2, asterisk, php-horde, php-horde-core, php-horde-kronolith, binutils, jasperreports, monitoring-plugins, percona-xtrabackup, poppler, jekyll and golang-go.net-dev.
  • DLA-1499-1. Issued a security update for discount fixing 4 CVE.
  • DLA-1504-1. Issued a security update for ghostscript fixing 14 CVE.
  • DLA-1506-1. Announced a security update for intel-microcode.
  • DLA-1507-1. Issued a security update for libapache2-mod-perl2 fixing 1 CVE.
  • DLA-1510-1. Issued a security update for glusterfs fixing 11 CVE.
  • DLA-1511-1. Issued an update for reportbug.
  • DLA-1513-1. Issued a security update for openafs fixing 3 CVE.
  • DLA-1517-1. Issued a security update for dom4j fixing 1 CVE.
  • DLA-1523-1. Issued a security update for asterisk fixing 1 CVE.
  • DLA-1527-1 and DLA-1527-2. Issued a security update for ghostscript fixing 2 CVE and corrected an incomplete fix for CVE-2018-16543 later.
  • I reviewed and uploaded strongswan and otrs2 for Abhijith PA.

ELTS

Extended Long Term Support (ELTS) is a project led by Freexian to further extend the lifetime of Debian releases. It is not an official Debian project but all Debian users benefit from it without cost. The current ELTS release is Debian 7 „Wheezy“. This was my fourth month and I have been paid to work 15  hours on ELTS.

  • I was in charge of our ELTS frontdesk from 10.09.2018 until 16.09.2018 and I triaged CVE in samba, activemq, chromium-browser, curl, dom4j, ghostscript, firefox-esr, elfutils, gitolite, glib2.0, glusterfs, imagemagick, lcms2, lcms, jhead, libpodofo, libtasn1-3, mgetty, opensc, openafs, okular, php5, smarty3, radare, sympa, wireshark, zsh, zziplib and intel-microcode.
  • ELA-35-1. Issued a security update for samba fixing 1 CVE.
  • ELA-36-1. Issued a security update for curl fixing 1 CVE.
  • ELA-37-2. Issued a regression update for openssh.
  • ELA-39-1. Issued a security update for intel-microcode addressing 6 CVE.
  • ELA-42-1. Issued a security update for libapache2-mod-perl2 fixing 1 CVE.
  • ELA-45-1. Issued a security update for dom4j fixing 1 CVE.
  • I started to work on a security update for the Linux kernel which will be released shortly.

Thanks for reading and see you next time.

09 October, 2018 11:06AM by Apo

hackergotchi for Norbert Preining

Norbert Preining

TeX Live Database as Graph Database

For a presentation at the Neo4j User Meeting in Tokyo I have converted the TeX Live Database into a Graph Database and represented dependencies between all kind of packages as well as files and their respective packages as nodes and relations.

Update 20181010: I have worked out the first step mentioned in further work and got rid of uuids completely and use package names/revisions and file names as identifier. I also added a new node type for TLPDB and renamed the relation between packages and files from contains to includes. The former now refers to the relation between TLPDB and packages. The text and code has been updated to reflect this.

Before going into the details how I did represent the TeX Live Database tlpdb as graph, let us recall a few concepts of how packages are managed and arranged in TeX Live. Each package in TeX Live has a cateogry. The currently available categories are Package, ConTeXt, Collection, Scheme, TLCore. They can be categorized into four groups:

  • Basic macro packages These are the meat of TeX Live, the actual stuff our upstream authors are writing. Typically LaTeX or font packages, they are either of category Package or ConTeXt.
  • Packages for binaries These are packages that ship “binary” files – which means files that are installed into the bin directory and are executable. Some of these are actually scripts and not binaries, though.
  • Collections A Collection contains basic and binary packages, and might depend on other collections. We guarantee that the set of collections is a partition of the available files, which allows distributors like Debian etc to make sure that no file is included two times in different packages.
  • Schemata These are the top-level groups that are presented to the user during installation. They depend on collections and other packages, and try to provide a meaningful selection.

The TeX Live Database itself is modeled after the Debian package database, and contains stanzas for each package. A typical example for a package would be (slightly abbreviated):

category Package
revision 15878
catalogue one2many
shortdesc Generalising mathematical index sets
longdesc In the discrete branches of mathematics and the computer
...
longdesc one-line change.
docfiles size=98
 texmf-dist/doc/latex/12many/12many.pdf details="Package documentation"
 texmf-dist/doc/latex/12many/README details="Readme"
srcfiles size=6
 texmf-dist/source/latex/12many/12many.dtx
 texmf-dist/source/latex/12many/12many.ins
runfiles size=1
 texmf-dist/tex/latex/12many/12many.sty
catalogue-ctan /macros/latex/contrib/12many
catalogue-date 2016-06-24 19:18:15 +0200
catalogue-license lppl
catalogue-topics maths
catalogue-version 0.3
...

A typical example for a collection would be:

name collection-langjapanese
category Collection
revision 48752
shortdesc Japanese
longdesc Support for Japanese; additional packages in
longdesc collection-langcjk.
depend collection-langcjk
depend ascmac
depend babel-japanese
...

and a typical example of a schema would be:

name scheme-medium
category Scheme
revision 44177
shortdesc medium scheme (small + more packages and languages)
longdesc This is the medium TeX Live collection: it contains plain TeX,
longdesc LaTeX, many recommended packages, and support for most European
longdesc languages.
depend collection-basic
depend collection-binextra
depend collection-context
depend collection-fontsrecommended
...

In total, we are currently at the following values: 9 Schemata, 41 Collections, 6718 Packages (Package, TLCore, ConTeXt), and about 181839 files.

Representation in Neo4j

Representation as graph was relatively straight-forward: We decided for separate nodes for each package and each file, and relations of dependency (depend in the above examples), inclusion (files being included in a package), and containment (a package is contained in a certain tlpdb revision).

We used a simple Perl script tl-dump-neo4j which uses the TeX Live provided Perl modules to read and parse the TeX Live Database to generate CSV files for each node type and each relation type. These CSV files were then imported into a Neo4j database with neo4j-import. For each node type one csv file was generated with three fields, an UUID consisting of the name and the revision separated by a colon, the name of the package and the revision. Example of the file node-Package.csv containing the Packages:

uuid:ID,name,revision
cslatex:47536,cslatex,47536
mcf2graph:48046,mcf2graph,48046
substances:40989,substances,40989
layouts:42428,layouts,42428
guide-to-latex:45712,guide-to-latex,45712

For the files contained in the database I use the file name as identifier, thus the respective csv only contains one field, the file name (enclosed in quotes to make sure that spaces are not mistreated).

There is a node type TLPDB with only identifier revision that carries the current version of the tlpdb used.

The three relations (depends, contains, and includes) then used the assigned UUIDs to define the relation: For packages it is the “name:revision”, for files the filename. The start of edge-depends.csv file is:

:START_ID,:END_ID
cslatex:47536,csplain:48563
cslatex:47536,latex:47860
cslatex:47536,cm:45811
cslatex:47536,tex-ini-files:40533
cslatex:47536,latex-fonts:28888
cslatex:47536,hyphen-base:48303

Only for the includes relation we added an additional tag giving the type of file (run/bin/doc/src according to the group the file is in the tlpdb). The start of edge-includes.csv is given below:

:START_ID,type,:END_ID
libertinus-type1:48618,run,"texmf-dist/fonts/tfm/public/libertinus-type1/LibertinusSans-tosf-sc-ly1.tfm"
sourcesanspro:42852,run,"texmf-dist/fonts/tfm/adobe/sourcesanspro/SourceSansPro-ExtraLight-tlf-sc-ly1.tfm"
cbfonts:31624,run,"texmf-dist/fonts/tfm/public/cbfonts/grxc0700.tfm"
clearsans:34405,doc,"texmf-dist/doc/fonts/clearsans/clear-samples.pdf"
fonts-tlwg:47499,run,"texmf-dist/fonts/tfm/public/fonts-tlwg/rkinnari_o.tfm"

The last relation is contains which sets up connections between tlpdb revisions and the contained packages. The start of edge-contains.csv is given below:

:START_ID,:END_ID
48864,ltxmisc:21927
48864,tex4ebook:47672
48864,tamefloats:27345
48864,matc3mem:35773
48864,todo:17746

With this in place a simple call to neo4j-import produced a ready-to-go Neo4j Database:

$ ls
edge-contains.csv    node-ConTeXt.csv  node-TLCore.csv
edge-depends.csv     node-Files.csv    node-TLPDB.csv
edge-includes.csv    node-Package.csv
node-Collection.csv  node-Scheme.csv
$ neo4j-import --into ../graphdb \
   --nodes:TLPDB node-TLPDB.csv \
   --nodes:Collection node-Collection.csv \
   --nodes:ConTeXt node-ConTeXt.csv \
   --nodes:Files node-Files.csv \
   --nodes:Package node-Package.csv \
   --nodes:Scheme node-Scheme.csv \
   --nodes:TLCore node-TLCore.csv \
   --relationships:contains edge-contains.csv \
   --relationships:includes edge-includes.csv \
   --relationships:depends edge-depends.csv
...
IMPORT DONE in 2s 93ms. 
Imported:
  168129 nodes
  172280 relationships
  175107 properties
Peak memory usage: 1.03 GB

Sample queries

Return all schemata:

match (s:Scheme) return s;

Return all dependencies from a schema to something else then a collection:

match p = (s:Scheme) -[:depends]-> (q)
  where NOT 'Collection' IN LABELS(q)
  return p;

Here we use LABELS to find all the labels of a node.

Check whether the same package is contained in two different collections:

match (c1:Collection) -[:depends]-> (p)
  <-[:depends]- (c2:Collection) return c1, c2, p;

Fortunately, only collections are targets of multiple depends, which is fine 😉

Search for cycles in the dependencies:

match p = (n)-[:depends*]-> (n) return p;

Here we use the * operator to search for arbitrary long paths. Interestingly we got one result, namely that ConTeXt depends on itself, something that is not good anyway.

Search for files that are included in multiple packages:

match (p1) -[:includes]-> (f)
  <- [:includes]- (p2) return p1, p2, f;

Fortunately here we didn't get any result. Anyway, this is checked every day with a simple grep/awk program 😉

Show all the documentation files for one package:

match (p) -[:includes {type:'doc'}]-> (f)
  where p.name = "tlcockpit"
  return p,f;

Graph Algorithm with Neo4j

The Neo4j Team also provides a set of graph alogrithm readily available by installing and activating a plugin. This plugin can be downloaded from this Neo4j Github Page. In my case this resulted in the download of graph-algorithms-algo-3.4.7.0.jar, which I did put into the plugins folder of my Neo4j installation. On Debian this is /var/lib/neo4j/plugins/. To get it to actually run one needs to allow running it by adding the following line to the Neo4j config file (on Debian /etc/neo4j/neo4j.conf):

dbms.security.procedures.unrestricted=algo.*

After a restart of Neo4j one is ready to use all the algorithms provided in this jar.

First let us check the Google Page Rank (whatever it might mean for the current case):

CALL algo.pageRank.stream(null, 'depends', {iterations:20, dampingFactor:0.85})
  YIELD nodeId, score
  MATCH (node) WHERE id(node) = nodeId
  RETURN node.name AS page,score
  ORDER BY score DESC

which gives the following output (in table mode):

╒══════════════════════════════╤═══════════════════╕
│"page"                        │"score"            │
╞══════════════════════════════╪═══════════════════╡
│"context"                     │4.868265000000001  │
├──────────────────────────────┼───────────────────┤
│"hyphen-base"                 │4.667172000000001  │
├──────────────────────────────┼───────────────────┤
│"hyph-utf8"                   │4.0754105          │
├──────────────────────────────┼───────────────────┤
│"kpathsea"                    │1.8529665          │
├──────────────────────────────┼───────────────────┤
│"plain"                       │0.982524           │
├──────────────────────────────┼───────────────────┤

In the similar vein is the Betweenness Centrality:

CALL algo.betweenness.stream(null, 'depends', {direction:'out'})
  YIELD nodeId, centrality
  MATCH (pkg) WHERE id(pkg) = nodeId
  RETURN pkg.name AS pkg,centrality
  ORDER BY centrality DESC;

which gives the following output:

╒══════════════════════════╤═══════════════════╕
│"pkg"                     │"centrality"       │
╞══════════════════════════╪═══════════════════╡
│"collection-basic"        │1675.4717032967033 │
├──────────────────────────┼───────────────────┤
│"collection-latexextra"   │1212.0             │
├──────────────────────────┼───────────────────┤
│"context"                 │947.3333333333334  │
├──────────────────────────┼───────────────────┤
│"collection-latex"        │744.8166666666666  │
├──────────────────────────┼───────────────────┤
│"collection-pictures"     │586.0              │
├──────────────────────────┼───────────────────┤

Finally let us look at the triangle computation:

CALL algo.triangleCount.stream(null, 'depends', {concurrency:4})
  YIELD nodeId, triangles, coefficient
  MATCH (p) WHERE id(p) = nodeId
  RETURN p.name AS name, triangles, coefficient
  ORDER BY triangles DESC

which yields the following output:

╒═════════════════════════╤═══════════╤═══════════════════════╕
│"name"                   │"triangles"│"coefficient"          │
╞═════════════════════════╪═══════════╪═══════════════════════╡
│"collection-basic"       │109        │0.042644757433489826   │
├─────────────────────────┼───────────┼───────────────────────┤
│"scheme-full"            │46         │0.05897435897435897    │
├─────────────────────────┼───────────┼───────────────────────┤
│"collection-latex"       │43         │0.04154589371980676    │
├─────────────────────────┼───────────┼───────────────────────┤
│"scheme-tetex"           │42         │0.022950819672131147   │
├─────────────────────────┼───────────┼───────────────────────┤
│"collection-context"     │39         │0.04318936877076412    │
├─────────────────────────┼───────────┼───────────────────────┤

Future Work

[DONE 20181010 - see above] During the presentation we got the suggestion to use hash values of the node content instead of arbitrarily computed uuids to allow for better upgrades/additions in case the values of nodes did remain the same.

Furthermore, it would be interesting to parse the full information of packages (including revision numbers, catalogue information etc), save them into the nodes, and regularly update the database to see the development of packages. To make this actually work out we need the first step of using hashes, though.

Closing

Considering that all of the above plus the actual presentation slides were written in less than one day, one can see that developing a graph database based on Neo4j and playing around with it is a rather trivial procedure. The difficult part is normally to find the "right" set of node types and relation types, as well as their attributes. In the case of the TeX Live database this was quite trivial, which allowed for an easy and direct representation in Neo4j.

We made the graph (read-only) available for experimentation at http://texlive.info:7474/browser/ (with user/pass neo4j).

We hope that these simple examples of graphs help others to kick-start more interesting and deeper projects using the power of graphs.

09 October, 2018 04:03AM by Norbert Preining

hackergotchi for Lucas Nussbaum

Lucas Nussbaum

Sending mail from mutt with queueing and multiple accounts/profiles?

I’m looking into upgrading my email setup. I use mutt, and need two features:

  • Local queueing of emails, so that I can write emails when offline, queue them, and send them later when I’m online.
  • Routing of emails through several remote SMTP servers, ideally depending on the From header, so that emails pass SPF/DKIM checks.

I currently use nullmailer, which does queueing just fine, but cannot apparently handle several remote SMTP servers.

There’s also msmtp, which can handle several “accounts” (remote SMTPs). But apparently not when using queueing using msmtpq.

What are you using yourself?

09 October, 2018 03:37AM by lucas

October 08, 2018

hackergotchi for Neil McGovern

Neil McGovern

GNOME ED Update – September

We’ve now moved my reporting to the board to a monthly basis, so this blog should get updated monthly too! So here’s what I’ve been up to in September.

Recruitment

Recruitment continues for our four positions that we announced earlier this year, but I’m pleased to say we’re in the final stages for these. For those interested, the process went a little bit like this:

  • Applicants sent in a CV and cover letter
  • If they were suitable for the position on a quick read of the CV and letter, they got a short questionnaire asking for more details, such as “What do you know about the GNOME Foundation?”
  • Those with interesting answers get sent to a first interview, which mostly technical
  • Then, those who are still in the process are invited to a second interview, which is competency-based
  • At the end of all this, we hope to make an offer to the best candidate!

End of year

For those who don’t know, the Foundation’s financial year runs from the start of October to the end of September. This means we have quite a bit of work to do to:

  1. Finalise the accounts for last year and submit our tax returns
  2. Make a new budget for the forthcoming year

Work has already begun on this, and I hope to finalise the new budget with the board at the Foundation Hackfest being held next week.

Libre Application Summit

LAS was held in Denver, Colorado, and I attended. There were 20 talks and three BoF
sessions held, as well as a number of social events. From looking around, there were probably around 60-70 people, including representatives from KDE and Elementary. It was particularly pleasing to see a number of students from the local university attend and present a lightning talk.

I also had meetings with System76 and Private Internet Access, as well as a couple of local companies.

Speaking of System76, we also had a nice tour of their new factory. I knew they were taking manufacturing in-house, but I didn’t realise the extent of this process. It’s not just assembly, but taking raw sheet metal, bending it into the right shape and painting them!

My meetings with PIA were also interesting – I got to see the new VPN client that they have, which I’m assured will be free software when released. There was a couple of issues I could see about how to integrate that with GNOME, and we had a good session running through these.

Other conferences coming up

In October, I’m hoping to attend Sustain Summit 2018, in London, followed by Freenode.Live in Bristol, UK. I’ll be speaking at the latter, which is in November. Then, after a couple of days at home, GNOME is going to SeaGL! Meet me and Rosanna in Seattle at the GNOME booth!

Friends of GNOME

Another thing that happened was fixing the Friends of GNOME signup page. For some reason, unknown to us, when you submitted the form to PayPal, it redirected to the home page rather than the payment page. This didn’t happen if you selected “EUR” as the payment method, or if you selected “EUR” and then “USD” before submitting. After lots of head scratching (an analysis of the POST data showed that it was /identical/ in each case) I changed the POST to a GET, and it suddenly started working again. Confusion all around, but it should now be working again.

08 October, 2018 06:27PM by Neil McGovern

Reproducible builds folks

Reproducible Builds: Weekly report #180

Here’s what happened in the Reproducible Builds effort between Sunday September 30 and Saturday October 6 2018:

Packages reviewed and fixed, and bugs filed

Test framework development

There were a huge number of updates to our Jenkins-based testing framework that powers tests.reproducible-builds.org by Holger Levsen this month, including:

In addition, Alexander Couzens added a comment regarding OpenWrt/LEDE which was subsequently amended by Holger.

Misc.

This week’s edition was written by Bernhard M. Wiedemann, Chris Lamb, heinrich5991, Holger Levsen, Marek Marczykowski-Górecki, Vagrant Cascadian & reviewed by a bunch of Reproducible Builds folks on IRC & the mailing lists.

08 October, 2018 05:32PM

hackergotchi for Holger Levsen

Holger Levsen

20180908-lts-201809

My LTS work in September

In September I only managed to spend 2.5h working on jessie LTS on:

  • finishing work on patches for samba, but then failed to release the DLA for it until now. Expect an upload soon. Sorry for the delay, various RL issues took their toll.

08 October, 2018 03:06PM

Petter Reinholdtsen

Fetching trusted timestamps using the rfc3161ng python module

I have earlier covered the basics of trusted timestamping using the 'openssl ts' client. See blog post for 2014, 2016 and 2017 for those stories. But some times I want to integrate the timestamping in other code, and recently I needed to integrate it into Python. After searching a bit, I found the rfc3161 library which seemed like a good fit, but I soon discovered it only worked for python version 2, and I needed something that work with python version 3. Luckily I next came across the rfc3161ng library, a fork of the original rfc3161 library. Not only is it working with python 3, it have fixed a few of the bugs in the original library, and it has an active maintainer. I decided to wrap it up and make it available in Debian, and a few days ago it entered Debian unstable and testing.

Using the library is fairly straight forward. The only slightly problematic step is to fetch the required certificates to verify the timestamp. For some services it is straight forward, while for others I have not yet figured out how to do it. Here is a small standalone code example based on of the integration tests in the library code:

#!/usr/bin/python3

"""

Python 3 script demonstrating how to use the rfc3161ng module to
get trusted timestamps.

The license of this code is the same as the license of the rfc3161ng
library, ie MIT/BSD.

"""

import os
import pyasn1.codec.der
import rfc3161ng
import subprocess
import tempfile
import urllib.request

def store(f, data):
    f.write(data)
    f.flush()
    f.seek(0)

def fetch(url, f=None):
    response = urllib.request.urlopen(url)
    data = response.read()
    if f:
        store(f, data)
    return data

def main():
    with tempfile.NamedTemporaryFile() as cert_f,\
    	 tempfile.NamedTemporaryFile() as ca_f,\
    	 tempfile.NamedTemporaryFile() as msg_f,\
    	 tempfile.NamedTemporaryFile() as tsr_f:

        # First fetch certificates used by service
        certificate_data = fetch('https://freetsa.org/files/tsa.crt', cert_f)
        ca_data_data = fetch('https://freetsa.org/files/cacert.pem', ca_f)

        # Then timestamp the message
        timestamper = \
            rfc3161ng.RemoteTimestamper('http://freetsa.org/tsr',
                                        certificate=certificate_data)
        data = b"Python forever!\n"
        tsr = timestamper(data=data, return_tsr=True)

        # Finally, convert message and response to something 'openssl ts' can verify
        store(msg_f, data)
        store(tsr_f, pyasn1.codec.der.encoder.encode(tsr))
        args = ["openssl", "ts", "-verify",
                "-data", msg_f.name,
	        "-in", tsr_f.name,
		"-CAfile", ca_f.name,
                "-untrusted", cert_f.name]
        subprocess.check_call(args)

if '__main__' == __name__:
   main()

The code fetches the required certificates, store them as temporary files, timestamp a simple message, store the message and timestamp to disk and ask 'openssl ts' to verify the timestamp. A timestamp is around 1.5 kiB in size, and should be fairly easy to store for future use.

As usual, if you use Bitcoin and want to show your support of my activities, please send Bitcoin donations to my address 15oWEoG9dUPovwmUL9KWAnYRtNJEkP1u1b.

08 October, 2018 10:30AM

October 07, 2018

Dima Kogan

Generating manpages from python and argparse

I find the python ecosystem deeply frustrating. On some level, they fixed some issues in previous languages. But at the same time, they chose to completely flout long-standing conventions, and have rewritten the world in ways that are different for no good reason. And when you rewrite the world, you always end up rewriting only the parts you care about, so your new implementation lacks pieces that other people find very important.

Today's annoyance: manpages. I have some tools written in python that I'm going to distribute, and since this isn't intended to be user-hostile, I want to ship manpages. The documentation already exists in the form of docstrings and argparse option decriptions, so I don't want to write the manpages, but they should be generated for me. I looked around and I can't find anything anywhere. There're some hoaky hacks people have come up with to do some of this, but I don't see any sort of "standard" way to do this at all. Even when I reduce the requirements to almost nothing, I still can't find anything. Please tell me if I missed something.

Anyway, I came up with yet another hoaky hack. It works for me. Sample project:

https://github.com/dkogan/python-argparse-generate-manpages-example

This has a python tool called frobnicate, and a script to generate its manpage as a .pod. The Makefile has rules to make this .pod and to convert it into a manpage. It works by running frobnicate --help, and parsing that output.

The --help message is set up to be maximally useful by including the main docstring into the message. This is good not just for the manpages, but to make an informative --help. And this has a nice consequence in that the manpage generator only needs to look at the --help output.

It is assumed that the main docstring (and thus the --help message) is formatted like a manpage would be, beginning with a synopsis. This isn't usually done in python, but it should be; they just like being contrarian for no good reason.

With those assumptions I can parse the --help output, and produce a reasonable manpage. Converted to html, it looks like this. Not the most exciting thing in the world, but that's the point.

This could all be cleaned up, and made less brittle. That would be great. In the meantime, it solves my use case. I'm releasing this into the public domain. Please use it and hack it up. If you fix it and give the changes back, I wouldn't complain. And if there're better ways to have done this, somebody please tell me.

Update

A few people reached out to me with suggestions of tools they have found and/or used for this purpose. A survey:

argparse-manpage

This lives here and here. I actually found this thing in my search, but it didn't work at all, and I didn't want to go into the rabbithole of debugging it. Well, I debugged it now and know what it needed. Issues

  • First off, there're 3 binaries that could be executed:
    • argparse-manpage
    • wrap
    • bin/argparse-manpage

    It appears that only the first one is meant to be functional. The rest throw errors that require the user to debug this thing

  • To make it more exciting, bin/argparse-manpage doesn't work out of the box on Debian-based boxes: it begins with 
    #!/bin/python

    Which isn't something that exists there. Changing this to 

    #!/usr/bin/python

    makes it actually runnable. Once you know that this is the thing that needs running, of course.

  • All right, now that it can run, we need to figure out more of this. The "right" invocation for the example project is this: 
    $ argparse-manpage/bin/argparse-manpage --pyfile python-argparse-generate-manpages-example/frobnicate  --function parse_args > frobnicator.1

    Unsurprisingly it doesn't work

  • argparse-manpage wants a function that returns an ArgumentParser object. This went against what the example program did: return the already-parsed options.
  • Once this is done, it still doesn't work. Apparently it attempts to actually run the program, and the program barfs that it wasn't given enough arguments. So for manpage-creation purposes I disable the actual option parsing, and make the program do nothing
  • And it still doesn't work. Apparently the function that parses the arguments doesn't see the argparse import when argparse-manpage works with it (even though it works fine when you just run the program). Moving that import into the function makes it work finally. A patch to the test program combining all of the workarounds together: 
diff --git a/frobnicate b/frobnicate
index 89da764..475ac63 100755
--- a/frobnicate
+++ b/frobnicate
@@ -16,6 +16,7 @@ import argparse


 def parse_args():
+    import argparse
     parser = \
         argparse.ArgumentParser(description = __doc__,
                                 formatter_class=argparse.RawDescriptionHelpFormatter)
@@ -29,8 +30,8 @@ def parse_args():
                         nargs='+',
                         help='''Inputs to process''')

-    return parser.parse_args()
+    return parser



-args = parse_args()
+#args = parse_args().parse_args()

Now that we actually get output, let's look at it. Converted to html, looks like this. Some chunks are missing because I didn't pass them on the commandline (author, email, etc). But it also generated information about the arguments only. I.e. the description in the main docstring is missing even though argparse was given it, and it's reported with --help. And the synopsis section contains the short options instead of an example, like it should.

help2man

This is in Debian in the help2man package. It works out of the box at least. Invocation: 

help2man python-argparse-generate-manpages-example/frobnicate --no-discard-stderr > frobnicator.1

In html looks like this. Better. At least the description made it in. The usage message is mistakenly in the NAME section, and the SYNOPSIS section is missing with the synopsis ending up in the DESCRIPTION section. The tool was not given enough information to do this correctly. It could potentially do something POD-like where the user is responsible for actually writing out all the sections, and help2man would just pick out and reformat the option-parsing stuff. This would be very heuristic, and impossible to do "right".

Others

There's apparently some way to do this with sphinx. I generally avoid the python-specific reimaginings of the world, so I haven't touched those. Besides, the "python" part is a detail in this project. The sphinx thing is here. And apparently you can invoke it like this: 

python3 -m sphinx -b man ...

Thoughts

Not sure what a better solution to all this would be. Maybe some sort of docstring2man tool that works like pod2man combined with docopt instead of argparse.

07 October, 2018 02:42PM by Dima Kogan

October 06, 2018

hackergotchi for Norbert Preining

Norbert Preining

TLCockpit v1.0

Today I released v1.0 of TLCockpit, the GUI front-end for the TeX Live Manager tlmgr.

If you are looking for a general introduction to TLCockpit, please see the blog introducing it. Here I only want to introduce the changes made since the last announcement here:

  • copyable information: The package information window didn’t allow to copy text from it, which has been fixed as far as possible.
  • placeholders: Add placeholders to tables when they are empty.
  • Cygwin support: Thanks to lots of debugging and support from Ken Brown we now support running on Cygwin
  • Java version checks: Java previous to version 8 are not supported and the program bails out. For Java later than version 8 we give a warning since in most cases it will not run due to ScalaFX incompatibilities.
    • CTAN and will soon be available via tlmgr update. As usual, please use the issue page of the github project to report problems.

    Enjoy.

06 October, 2018 03:02AM by Norbert Preining

Vishal Gupta

DebDialer : Handling phone numbers on Linux Desktops | GSoC 2018

This summer I had the chance to contribute to Debian as a part of GSoC. I built a desktop application, debdialer for handling tel: URLs and (phone numbers in general) on the Linux Desktop. It is written in Python 3.5.2 and uses PyQt4 to display a popup window. Alternatively, there is also a no-gui option that uses dmenu for input and terminal for output. There is also a modified apk of KDE-Connect to link debdialer with the user’s Android Phone. The pop-up window has numeric and delete buttons, so the user can either use the GUI or keyboard to modify numbers.

Alt Text DebDialer

Features

(Screenshots and how-to)
  1. Adds contact using .vcf file (Add vcard to Contacts)
  2. Adds number in dialer as contact (Add to Contacts)
  3. Sending dialer number to Android phone (DIAL ON ANDROID PHONE)
  4. Parsing numbers from file (Open File)
  5. Automatic formatting of numbers and setting of details

Installation

Installing with pip installs the python package but does not set up the desktop file. Hence, the following script needs to be run.

# Optional dependencies. Atleast one of them is required.
sudo apt install dmenu
sudo apt install python3-pyqt4

curl -L https://salsa.debian.org/comfortablydumb-guest/Hello-from-the-Debian-side/raw/master/install.sh -s | bash

Links :

06 October, 2018 02:59AM by Vishal Gupta

October 05, 2018

John Goerzen

The Python Unicode Mess

Unicode has solved a lot of problems. Anyone that remembers the mess of ISO-8859-* vs. CP437 (and of course it’s even worse for non-Western languages) can attest to that. And of course, these days they’re doing the useful work of…. codifying emojis.

Emojis aside, things aren’t all so easy. Today’s cause of pain: Python 3. So much pain.

Python decided to fully integrate Unicode into the language. Nice idea, right?

But here come the problems. And they are numerous.

gpodder, for instance, frequently exits with tracebacks due to Python errors converting podcast titles with smartquotes into ASCII. Then you have the case where the pexpect docs say to use logfile = sys.stdout to show the interaction with the virtual terminal. Only that causes an error these days.

But processing of filenames takes the cake. I was recently dealing with data from 20 years ago, before UTF-8 was a filename standard. These filenames are still valid on Unix. tar unpacks them, and they work fine. But you start getting encoding errors from Python trying to do things like store filenames in strings. For a Python program to properly support all valid Unix filenames, it must use “bytes” instead of strings, which has all sorts of annoying implications. What’s the chances that all Python programs do this correctly? Yeah. Not high, I bet.

I recently was processing data generated by mtree, which uses octal escapes for special characters in filenames. I thought this should be easy in Python, eh?

That second link had a mention of an undocumented function, codecs.escape_decode, which does it right. I finally had to do this:

    if line.startswith(b'#'):
        continue
    fields = line.split()
    filename = codecs.escape_decode(fields[0])[0]
    filetype = getfield(b"type", fields[1:])
    if filetype == b"file":

And, whatever you do, don’t accidentally write if filetype == "file" — that will silently always evaluate to False, because "file" tests different than b"file". Not that I, uhm, wrote that and didn’t notice it at first…

So if you want to actually handle Unix filenames properly in Python, you:

  • Must have a processing path that fully avoids Python strings.
  • Must use sys.{stdin,stdout}.buffer instead of just sys.stdin/stdout
  • Must supply filenames as bytes to various functions. See PEP 0471 for this comment: “Like the other functions in the os module, scandir() accepts either a bytes or str object for the path parameter, and returns the DirEntry.name and DirEntry.path attributes with the same type as path. However, it is strongly recommended to use the str type, as this ensures cross-platform support for Unicode filenames. (On Windows, bytes filenames have been deprecated since Python 3.3).” So if you want to be cross-platform, it’s even worse, because you can’t use str on Unix nor bytes on Windows.

Update: Would you like to receive filenames on the command line? I’ll hand you this fine mess. And the environment? it’s not even clear.

05 October, 2018 09:10PM by John Goerzen

hackergotchi for Daniel Pocock

Daniel Pocock

Stigmatizing volunteers who miss an event

In various free software communities, I've come across incidents where people have been criticized inappropriately when they couldn't attend an event or didn't meet other people's expectations. This has happened to me a few times and I've seen it happen to other people too.

As it turns out, this is an incredibly bad thing to do. I'm not writing about this to criticize any one person or group in return. Rather, it is written in the hope that people who are still holding grudges like this might finally put them aside and also to reassure other volunteers that you don't have to accept this type of criticism.

Here are some of the comments I received personally:

"Last year, you signed up for the conference but didn't attend, cancelling on the last minute, when you had already been ..."

"says the person who didn't attend any of the two ... he was invited to, because, well, he had no time"

"you didn't stayed at the booth enough at ..., never showed up at the booth at the ... and never joined ..."

Having seen this in multiple places, I don't want this blog to focus on any one organization, person or event.

In all these cases, the emails were sent to large groups on CC, one of them appeared on a public list. Nobody else stepped in to point out how wrong this is.

Out of these three incidents, one of them subsequently apologized and I sincerely thank him for that.

The emails these were taken from were quite negative and accusatory. In two of these cases, the accusation was being made after almost a year had passed. It leaves me wondering how many people in the free software community are holding grudges like this and for how long.

Personally, going to an event usually means giving talks and workshops. Where possible, I try to involve other people in my presentations too and may disappear for an hour or skip a social gathering while we review slides. Every volunteer, whether they are speakers, organizers or whatever else usually knows the most important place where they should be at any moment and it isn't helpful to criticize them months later without even asking, for example, about what they were doing rather than what they weren't doing.

Think about some of the cases where a volunteer might cancel their trip or leave an event early:

  • At the last minute they decided to go to the pub instead.
  • They never intended to go in the first place and wanted to waste your time.
  • They are not completely comfortable telling you their reason because they haven't got to know you well enough or they don't want to put it in an email.
  • There is some incredibly difficult personal issue that may well be impossible for them to tell you about because it is uncomfortable or has privacy implications. Imagine if a sibling commits suicide, somebody or their spouse has a miscarriage, a child with a mental health issue or a developer who is simply burnt out. A lot of people wouldn't tell you about tragedies in this category and they are entitled to their privacy.

When you think about it, the first two cases are actually really unlikely. You don't do that yourself, so why do you assume or imply that any other member of the community would behave that way?

So it comes down to the fact that when something like this happens, it is probably one of the latter two cases.

Even if it really was one of the first two cases, criticizing them won't make them more likely to show up next time, it has no positive consequences.

In the third case, if the person doesn't trust you well enough to tell you the reason they changed their plans, they are going to trust you even less after this criticism.

In the fourth case, your criticism is going to be extraordinarily hurtful for them. Blaming them, criticizing them, stigmatizing them and even punishing them and impeding their future participation will appear incredibly cruel from the perspective of anybody who has suffered from some serious tragedy: yet these things have happened right under our noses in respected free software projects.

What is more, the way the subconscious mind works and makes associations, they are going to be reminded about that tragedy or crisis when they see you (or one of your emails) in future. They may become quite brisk in dealing with you or go out of their way to avoid you.

Many organizations have adopted codes of conduct recently. In Debian, it calls on us to assume good faith. The implication is that if somebody doesn't behave the way you hope or expect, or if somebody tells you there is a personal problem without giving you any details, the safest thing to do and the only thing to do is to assume it is something in the most serious category and treat them with the respect that you would show them if they had fully explained such difficult circumstances to you.

05 October, 2018 08:35AM by Daniel.Pocock

Sergio Alberti

Unusual meetings

Last week, I had the pleasure of meeting Daniel Pocock in one of my favourite places (which is Capo Di Lago). As his blog shows, he was almost close to the area where I live. So he had the courtesy to join me for dinner, despite the long trip he had faced.

It was interesting to finally know someone inside the Debian organization. We discussed about various conferences on free software, how Debian works, my work during the GSoC and the heating system he’s working on in his house.

Finally, he gave me an overview of Software Defined Radio, Ham Radio, RTL-SDR dongles and Deb packages needed to manage it (rtl-sdr and gqrx-sdr). I didn’t know anything about it. This made me curious.

Thanks Daniel.

meeting with daniel

05 October, 2018 08:33AM

Rodrigo Siqueira

XDC 2018 Report

XDC 2018 Report

X.Org Developer’s Conference (XDC) is the summit meeting for people that work with graphics in all the world to meet each other for three days. There you will find people working with compositors, direct rendering management (DRM), graphics applications, and so forth; all these people at the same place create a unique learning opportunity. Finally, you can feel the community spirit in every table, talk, and corner.

The XDC has many exciting talks, social events, and space for discussion with developers. All of this enabled thanks to the organization team, which did a great job by organizing the conference; they selected a great university that had a perfect structure for hosting the event. They also included social events that introduced some background about the history of the La Coruna; In my case, I enjoyed to learn a bit of the local history. About the food, the conference provided coffee breaks and lunch during all the days, all of them great!

About the community

In my case, I put as much effort as possible to learn from people. In the first day, I had a great conversation with Daniel Stone, Roman Gilg, Drew DeVault, Guido Günther and other about compositors. In particular, Daniel Stone explained a lot of details about Weston and Wayland; he also taught me how to test Weston on top of VKMS, and how to see logs. Additionally, Daniel gave me an excellent idea: add writeback to VKMS to provide visual output and other features. In the same day, Daniel explained me many things about the community organization and his work to maintain the Gitlab instance for the Freedesktop; I really enjoyed every second of our conversation.

Additionally, I met a group of Sway developers during lunch. After a small chat, for some reason they took their laptops and started to play with Sway; I got really impressed with their work and enthusiasm. Then, I decided that I wanted to learn how to contribute with Sway for two reasons: I want to learn more about graphics in the user space (compositors), and I want to use a new desktop environment. Afterwards, I started asking Drew to teach me how to compile and use Sway. He was really kind, he showed me many things about compositor then pointed me directions to better get into this world.

On the second day, I was obsessed about writeback, and I tried to talk with Brian Starkey; he is the original author of the patch that added writeback to DRM. We spoke for one hour, Bryan explained me so many details about writeback and gave me some ideas on how I could implement it on VKMS. In the end, he also sent me an email with diagrams that he made on-the-fly and some extra explanation about the subject. I am happy that I had the opportunity to learn so many things from him. In the same day, I also got a chance to talk to Arkadiusz Hiler about some of the patches that I sent to IGT, and I also made lots of questions about IGT. He explained with details, how I could read the intel CI report and other related stuff. I hope that after his explanations I can improve my patches and also send much more for IGT.

On the third day, Haneen and I worked together to learn as much as we could by asking many things to Daniel Vetter. We used the opportunity to clarify many of our questions, and also discuss some patches we sent. At the end of our conversation, I applied to become co-mentor in the Outreachy; I hope that I can help bringing new people to become part of this fantastic community.

This is just a brief summary of XDC, I took every single opportunity that I had to talk to people and learned something new.

I finally met Haneen, Daniel Vetter, Sean Paul, Martin Peres, and Arkadiusz Hiler

One exciting thing about working remotely it is the fact that you talk with many people without really know them in person. In particular, I worked with Haneen for such a long time, but I have never seen her; however, in the XDC I finally met her! It was really nice to talk to her, both of us were together most of the time trying to understand as much as we could; as a result, we always helped each other in the event to better understand the concepts that someone would explained us.

I also met Daniel Vetter, and Sean Paul, both of them were our mentors during summer. I really enjoyed to talk with them and put a face on the nickname. Additionally, I met Martin Peres, thanks to him I created this website to keep reporting my work and also thanks to him I could enjoy XDC.

Finally, I met Hiler from Intel. He provided many feedback on the patches I sent IGT; he also helped a lot in the IRC channel. I really liked to meet him in person.

Feedbacks

During the event, Haneen and I received so many feedback on how we could improve the VKMS that we decided to do a workshop in the last day. The workshop was great, a lot of people joined, and we took note of new ideas. From the conversation, we emerged the following list of tasks:

Now that I learned a lot and collected so many feedback, I will work on the following steps:

  1. Implement writeback support
  2. Implement configfs system
  3. Mentor a newcomer in the outreachy

05 October, 2018 03:00AM

October 04, 2018

hackergotchi for Norbert Preining

Norbert Preining

Wendy Berliner, Deborah Eyre: Great Minds And How To Grow Them

I never read any of these typical Howto books, and when it comes to raising up my child I am also quite confident that I know more or less what I want to do and what I consider good. But then, sometimes, you ponder things like how you could do better. And in one of these weak moments I started reading the book Great Minds and How To Grow Them. What a huge amount of wasted time it was …

This book is a failure from the first to the last page: It asks parents to talk to the kids and make them ask questions. That is all that is written there. There are long lists of situations, categorized into all kind of complex expressions to fake the image of being scientifically. That is probably what made me fed up the most, the aura of being scientific, quoting lots of (I guess rather dubious) research with foot notes but not proper reference system. My favorite scientific quote was:

According to Daniel Levitin (2006) it takes 10,000 hours of practice to make an expert. So children who are successful in school and beyond are those who have put in the practice.
– Great Minds and How To Grow Them, Chapter 2

which is repeated at least one more time. Checking the reference leads to Levitin, D.J. (2006) This Is Your Brainb on Music: The Science of a Human Obsession. Dutton: Penguin. Well, I don’t think this is a proper scientific reference, it is used only in a way to achieve the appearance of science. In addition, let us ponder on this number of hours a bit. Assuming I want to become expert in some topic, how long will it take. If I want to manage it in 10 years (!!!) I need to study 10000/3650 = roughly 3 hours a day, for 10 years. Now that is definitely incorrect. If I tell this my daughter she will simply NOT become an expert in anything …

Other things in the book is the pseudo-scientific categorization, here a few examples:

Meta-thinking: This is a set of four characteristics that relate to thinking about thinking. They consist of: meta-cognition; self-regulation; strategy planning; intellectual confidence.
…
Linking: This is a set of six characteristics about linking what you learn together. The six characteristics are: generalisation; connection finding; big picture thinking; abstraction; imagination; seeing alternative perspectives.
– Great Minds and How To Grow Them, Chapter 3

Aehmm, yes, lots of impressive lists. I really think that anyone who can read this book and follow the argumentation there (how thin the argumentation lines might be) is already in the state of mind that he knows what is best for kids, without having to read through these pages.

I am closing with a quote from a great review by David James, the deputy head (academic) at Bryanston School:

So, how do we grow great minds in our children? The answer, it seems, is simple: you talk to them and encourage them to ask questions. The authors are careful to avoid writing that, in essence, you grow great minds by being very middle class in possession of a significant amount of cultural capital (they reassure us at one point that a good education can be realised “even if you have to use a foodbank”), but it clearly helps.
– David James: Book review: Great Minds and How to Grow Them

Final verdict: not worth taking place in your book shelf, not even the electronic one.

04 October, 2018 03:02PM by Norbert Preining

Petter Reinholdtsen

Automatic Google Drive sync using grive in Debian

A few days, I rescued a Windows victim over to Debian. To try to rescue the remains, I helped set up automatic sync with Google Drive. I did not find any sensible Debian package handling this automatically, so I rebuild the grive2 source from the Ubuntu UPD8 PPA to do the task and added a autostart desktop entry and a small shell script to run in the background while the user is logged in to do the sync. Here is a sketch of the setup for future reference.

I first created ~/googledrive, entered the directory and ran 'grive -a' to authenticate the machine/user. Next, I created a autostart hook in ~/.config/autostart/grive.desktop to start the sync when the user log in:

[Desktop Entry]
Name=Google drive autosync
Type=Application
Exec=/home/user/bin/grive-sync

Finally, I wrote the ~/bin/grive-sync script to sync ~/googledrive/ with the files in Google Drive.

#!/bin/sh
set -e
cd ~/
cleanup() {
    if [ "$syncpid" ] ; then
        kill $syncpid
    fi
}
trap cleanup EXIT INT QUIT
/usr/lib/grive/grive-sync.sh listen googledrive 2>&1 | sed "s%^%$0:%" &
syncpdi=$!
while true; do
    if ! xhost >/dev/null 2>&1 ; then
        echo "no DISPLAY, exiting as the user probably logged out"
        exit 1
    fi
    if [ ! -e /run/user/1000/grive-sync.sh_googledrive ] ; then
        /usr/lib/grive/grive-sync.sh sync googledrive
    fi
    sleep 300
done 2>&1 | sed "s%^%$0:%"

Feel free to use the setup if you want. It can be assumed to be GNU GPL v2 licensed (or any later version, at your leisure), but I doubt this code is possible to claim copyright on.

As usual, if you use Bitcoin and want to show your support of my activities, please send Bitcoin donations to my address 15oWEoG9dUPovwmUL9KWAnYRtNJEkP1u1b.

04 October, 2018 01:20PM

Sanjay Prajapat

October 03, 2018

hackergotchi for Daniel Pocock

Daniel Pocock

Renewables, toilets, wifi and freedom

The first phase of the Dublin project was recently completed: wifi, a toilet and half a kitchen. An overhaul of the heating and hot water infrastructure and energy efficiency improvements are coming next.

The Irish state provides a range of Government grants for energy efficiency and renewable energy sources (not free as in freedom, but free as in cash).

The Potterton boiler is the type of dinosaur this funding is supposed to nudge into extinction. Nonetheless, it has recently passed another safety inspection:

Not so smart

This relic has already had a date with the skip, making way for smart controls:

Renewable energy

Given the size of the property and the funding possibilities, I'm keen to fully explore options for things like heat pumps and domestic thermal stores.

Has anybody selected and managed this type of infrastructure using entirely free solutions, for example, Domoticz or Home Assistant? Please let me know, I'm keen to try these things, contribute improvements and blog about the results.

Next project: intrusion detection

With neighbours like these, who needs cat burglars? Builders assure me he has been visiting on a daily basis and checking their work.

Time for a smart dog to stand up to this trespasser?

03 October, 2018 09:42PM by Daniel.Pocock

hackergotchi for Norbert Preining

Norbert Preining

Firefox got maniac

I don’t know what, I don’t know why, but Firefox behaves completely maniac on one of my computers. Opening simple tabs beats up 4 Web Content threads to nearly 100% CPU time, switching tabs the same.

By now I have:

  • removed all of .mozilla
  • started in –safe-mode
  • disabled all extensions
  • tried upstream provided Firefox instead of Debian’s
  • made tripple flips while balancing three raw eggs on my nose

plus some more trials like removing locally installed fonts. But nothing has helped. Firefox is maniac. Opening a new tab with my Debian Q&A page sometimes, but not always, just shoots up the CPU usage as shown above. Yes, that are *4* Web Content all around/above 60%. This is all on an up-to-date Debian/sid.

Behavior is erratic. Often after suspend to ram and wake up it is getting worse. No idea what else I could do…

Anyone having an idea, please let me know!

Update 2018-10-04: For everyone interested, I am now suspecting a problem with the fontconfig library. The reasons why I come to this proposal is:

  • the turning wheel on white background I experience often, I also see when new font packages are installed and fc-cache etc is updated
  • another program (electron based) started to crash in libfontconfig with: 
    (gdb) bt
#0  0x00007f416edf1dc0 in  () at /usr/lib/x86_64-linux-gnu/libfontconfig.so.1
#1  0x00007f416ede5ede in FcConfigSubstituteWithPat () at /usr/lib/x86_64-linux-gnu/libfontconfig.so.1
#2  0x00007f416edf69fd in FcFontRenderPrepare () at /usr/lib/x86_64-linux-gnu/libfontconfig.so.1
#3  0x00007f416edf6e84 in FcFontMatch () at /usr/lib/x86_64-linux-gnu/libfontconfig.so.1

I hope I can actual show that this is the problem by finding a problematic font I have installed. I have lots of fonts in /usr/local, in particular CJK fonts that sometimes are a bit challenging.

03 October, 2018 04:32PM by Norbert Preining

hackergotchi for Jonathan Dowland

Jonathan Dowland

Red Hat shell prompt

I do a lot of my work in a virtual machine running Red Hat Enterprise Linux. In order to distinguish the terminal windows that are running shells within the VM from the ones running on my laptop's natively, I configure the shell prompt to look different.

For my other systems, I use my punctual prompt system, but for this VM I thought it would be more fun to have something a bit more on-brand.

A Red Hat prompt

Here's a bashrc snippet to achieve this:

hat="🎩"
export PS1="\[\e[31m\]$hat\[\e[0m\]"

I ended up switching to zsh for my shell in this VM because of a bash bug exposed by trying this out. You shouldn't hit it for the snippet above, but I originally specified the red colour using 256 color escape codes, and this caused width-error bugs when long lines wrapped. zsh doesn't have the same bug, but it can be avoided in bash anyway by using the much older 8 colour escape codes, as above.

03 October, 2018 12:55PM

October 02, 2018

hackergotchi for Ritesh Raj Sarraf

Ritesh Raj Sarraf

sg3-utils and NVMe disks

sg3-utils, version 1.44, was recently uploaded to Debian. This new upstream release has happened almost 2.5 years after the last release. One important feature to emphasize about is some support for NVMe disks, which are now getting more common on latest range of laptops.

rrs@priyasi:~$ sudo sg_inq /dev/nvme0
[sudo] password for rrs: 
Identify controller for /dev/nvme0:
  Model number: PC401 NVMe SK hynix 512GB               
  Serial number: ES82N092210402J4U   
  Firmware revision: 80003E00
  Version: 1.2
  Optional admin command support:
    Device self-test
    Firmware download and commit
    Format NVM
    Security send and receive
  Optional NVM command support:
    Save and Select fields non-zero
    Dataset management
    Write uncorrectable
    Compare
  PCI vendor ID VID/SSVID: 0x1c5c/0x1c5c
  IEEE OUI Identifier: 0xace42e
  Controller ID: 0x1
  Number of namespaces: 1
  Maximum data transfer size: 32 pages
  Namespace 1 (of 1):
    Namespace size/capacity: 1000215216/1000215216 blocks
    Namespace utilization: 765481168 blocks
    EUI-64: 0xace42e8170034999
    Number of LBA formats: 2
    Index LBA size: 0
    LBA format 0 support: <-- active
      Logical block size: 512 bytes
      Approximate namespace size: 512 GB
      Metadata size: 0 bytes
      Relative performance: Best [0x0]
    LBA format 1 support:
      Logical block size: 4096 bytes
      Approximate namespace size: 0 GB
      Metadata size: 0 bytes
      Relative performance: Best [0x0]

More details are available on the upstream page. Here's the snippet for NVMe disks support.

NVME Support

On one hand NVM Express (usually written as "NVMe") doesn't have much in common with SCSI. They have different command sets and don't share many transports. The SOP/PQI project which uses the SCSI command set over PCIe (PCI Express) has not caught on, probably closely related to NVMe's success on PCIe.

On the other hand both SCSI and NVMe offer alternative approaches to accessing solid state disks (SSDs), approaches that have much in common. SCSI has been around a little longer (1981 compared to 2011) . In the early days of NVMe there was a SCSI to NVMe Translation Layer (SNTL) but that was dropped in 2017 from the Linux kernel as Intel were not prepared to support it any longer. Strange, since the equivalent mechanism for (mainly) SATA disks called SCSI to ATA Translation (SAT) has been quite successful both for SATA and the SAS (SCSI serial protocol).

There are some signs of the two protocols co-existing with the NVME-MI (Management Interface) group accepting the SES-3 standard for (storage) enclosure management. To this end two NVME-MI commands have been added: SES Send and SES Receive. The library underpinning sg3_utils has been extended to detect NVMe devices in Linux and FreeBSD (not yet working for Windows) has been extended and to pass-through SCSI SEND DIAGNOSTIC and RECEIVE DIAGNOSTIC RESULTS commands to those new NVME-MI commands. The sg3_utils library implements a small SNTL that emulates the mandatory SPC commands (e.g. INQUIRY and REPORT LUNS) and a few others so that the sg_ses utility can interact with a NVMe enclosure with almost no code changes to sg_ses itself. sg_inq has been modified to detect NVMe devices and partially decode the Identify Controller and Identify namespace responses. The lsscsi utility (see the lsscsi package) now lists NVMe namespaces (after SCSI logical units) and NVMe controllers (after SCSI hosts (HBAs)).

The sg_senddiag utility applied to a NVMe device will attempt to tunnel the associated data in NVME-MI SES Send and SES Recieve commands. That should fail unless the NVMe controller has an enclosure attached. The sg_raw command can issue NVME Admin command. The sg_inq command has two modes when applied to a NVME (namespace or controller); without options it will send Identify controller and namespace (assuming there is one) and partially decode the response; with the "--page=sinq" option ("sinq" means standard Inquiry, so for SCSI devices that is the same as the default (no option) usage) will issue a SCSI INQUIRY which the above mentioned SNTL will emulate an INQUIRY response.

Since there is no standard SNTL, this package has taken the liberty of defining its own "vendor specific" VPD page: 0xde . When fetched (with sg_inq, sg_vpd or sdparm) on a NVMe device it will yield the Identify controller response which is 4096 bytes long.
 

Categories: 

Keywords: 

Like: 

02 October, 2018 06:30PM by Ritesh Raj Sarraf

hackergotchi for Jonathan McDowell

Jonathan McDowell

Controlling my heating with Home Assistant

My original post about home automation discussed the fact that one of my motivations was improving control over my central heating system. In the last few weeks I’ve finally brought enough pieces together to make that a reality. My boiler is controlled by a Siemens RCR10/433 thermostat. Ross Harper has a good write-up about decoding the Siemens RCR10/433 and I was able to extend my Energenie Atmel 433MHz transmitter to treat the boiler as another switch. Slightly different timing than the Energenie switches, but exactly the same principle. My TEMPer USB clone provides a reading of the living room temperature. Finally mqtt-arp lets me work out whether anyone is home or not.

Everything is tied together with Home Assistant. The configuration has ended up more involved than I expected, but it’s already better than the old 24 hour timer. There’s definitely still room for improvement in terms of behaviour as the weather starts to get colder and I collect further data. Presently it looks like this:

Home Assistant heating controls

Top left is the control card; “Heating” is a climate control with a target temperature and a current temperature (from the living room sensor), an on/off state for the boiler (linked to the 433MHz transmitter), a switch to indicate if the timer is on or not and finally a slider to control what the target temperature should be when the heating is active.

Top right is a history card showing the various temperature sensors around the house as well as the target temperature state of the heating over the past 24 hours.

The bottom two cards show the timer times for week days and weekends. I did consider making a full 7 day timer, but this ends up good enough and I couldn’t find a better way to represent a set of start + end times that would have allowed a clean interface display of a full week. The times control when the “Heating timer” control in the top left is switched on + off.

These 4 cards provide the ability to see the current state of the heating, and tweak it, ideally meaning there’s no need to hand edit config files during normal operation. Rough theory of operation is:

  • If the timer is on and someone is at home, raise the target temperature to the value set in the temperature slider.
  • If the timer turns off or everyone leaves the house, lower the target temperature to 5°C.

The core is a generic thermostat:

climate:
  - platform: generic_thermostat
    name: Heating
    heater: switch.gas_boiler
    target_sensor: sensor.living_room_temperature
    min_temp: 5
    max_temp: 25
    ac_mode: false
    hot_tolerance: 0.5
    cold_tolerance: 0.1
    min_cyle_duration:
      minutes: 5
    keep_alive:
      minutes: 30
    initial_operation_mode: 'auto'

This is always active, and climate.set_temperature used to control what the target temperature is.

The temperature control slider is a simple input_number:

input_number:
  heating_temperature:
    name: Temperature
    min: 5
    max: 25
    step: 0.5
    icon: mdi:thermometer

The timer is where it gets complex. There are 8 input_datetime entries to deal with the different start/stop times. It seems like there should be an easier way, but this is what I have:

Heating start/stop time inputs 
input_datetime:
  weekday_morning_start:
    name: Week day morning start
    has_time: true
    has_date: false
  weekday_morning_stop:
    name: Week day morning stop
    has_time: true
    has_date: false
  weekend_morning_start:
    name: Weekend morning start
    has_time: true
    has_date: false
  weekend_morning_stop:
    name: Weekend morning stop
    has_time: true
    has_date: false
  weekday_evening_start:
    name: Week day evening start
    has_time: true
    has_date: false
  weekday_evening_stop:
    name: Week day evening stop
    has_time: true
    has_date: false
  weekend_evening_start:
    name: Weekend evening start
    has_time: true
    has_date: false
  weekend_evening_stop:
    name: Weekend evening stop
    has_time: true
    has_date: false

For the automations I also needed to add a time & date sensor:

sensor:
  - platform: time_date
    display_options:
      - 'time'

And finally the output input_boolean to represent if the timer is active or not:

input_boolean:
  heating_timer:
    name: Heating timer
    icon: mdi:toggle-switch

These get tied together with a bunch of automations:

Automations for heating timer 
automation:
  - id: heating_morning_on_wd
    alias: Turn heating on (weekday mornings)
    trigger:
      platform: template
      value_template: "{{ states('sensor.time') == (states.input_datetime.weekday_morning_start.attributes.timestamp | int | timestamp_custom('%H:%M', False)) }}"
    condition:
      condition: time
      weekday:
        - mon
        - tue
        - wed
        - thu
        - fri
    action:
      service: input_boolean.turn_on
      data_template:
        entity_id: input_boolean.heating_timer

  - id: heating_morning_off_wd
    alias: Turn heating off (weekday mornings)
    trigger:
      platform: template
      value_template: "{{ states('sensor.time') == (states.input_datetime.weekday_morning_stop.attributes.timestamp | int | timestamp_custom('%H:%M', False)) }}"
    condition:
      condition: time
      weekday:
        - mon
        - tue
        - wed
        - thu
        - fri
    action:
      service: input_boolean.turn_off
      data_template:
        entity_id: input_boolean.heating_timer

  - id: heating_evening_on_wd
    alias: Turn heating on (weekday evenings)
    trigger:
      platform: template
      value_template: "{{ states('sensor.time') == (states.input_datetime.weekday_evening_start.attributes.timestamp | int | timestamp_custom('%H:%M', False)) }}"
    condition:
      condition: time
      weekday:
        - mon
        - tue
        - wed
        - thu
        - fri
    action:
      service: input_boolean.turn_on
      data_template:
        entity_id: input_boolean.heating_timer

  - id: heating_evening_off_wd
    alias: Turn heating off (weekday evenings)
    trigger:
      platform: template
      value_template: "{{ states('sensor.time') == (states.input_datetime.weekday_evening_stop.attributes.timestamp | int | timestamp_custom('%H:%M', False)) }}"
    condition:
      condition: time
      weekday:
        - mon
        - tue
        - wed
        - thu
        - fri
    action:
      service: input_boolean.turn_off
      data_template:
        entity_id: input_boolean.heating_timer

  - id: heating_morning_on_we
    alias: Turn heating on (weekend mornings)
    trigger:
      platform: template
      value_template: "{{ states('sensor.time') == (states.input_datetime.weekend_morning_start.attributes.timestamp | int | timestamp_custom('%H:%M', False)) }}"
    condition:
      condition: time
      weekday:
        - sat
        - sun
    action:
      service: input_boolean.turn_on
      data_template:
        entity_id: input_boolean.heating_timer

  - id: heating_morning_off_we
    alias: Turn heating off (weekend mornings)
    trigger:
      platform: template
      value_template: "{{ states('sensor.time') == (states.input_datetime.weekend_morning_stop.attributes.timestamp | int | timestamp_custom('%H:%M', False)) }}"
    condition:
      condition: time
      weekday:
        - sat
        - sun
    action:
      service: input_boolean.turn_off
      data_template:
        entity_id: input_boolean.heating_timer

  - id: heating_evening_on_we
    alias: Turn heating on (weekend evenings)
    trigger:
      platform: template
      value_template: "{{ states('sensor.time') == (states.input_datetime.weekend_evening_start.attributes.timestamp | int | timestamp_custom('%H:%M', False)) }}"
    condition:
      - condition: time
        weekday:
          - sat
          - sun
    action:
      service: input_boolean.turn_on
      data_template:
        entity_id: input_boolean.heating_timer

  - id: heating_evening_off_we
    alias: Turn heating off (weekend evenings)
    trigger:
      platform: template
      value_template: "{{ states('sensor.time') == (states.input_datetime.weekend_evening_stop.attributes.timestamp | int | timestamp_custom('%H:%M', False)) }}"
    condition:
      condition: time
      weekday:
        - sat
        - sun
    action:
      service: input_boolean.turn_off
      data_template:
        entity_id: input_boolean.heating_timer

The timer boolean switch and the group.all_devices presence information are then tied together to raise/lower the target temperature as appropriate. I’ve used 4 automations for this - one triggered for timer on, one for timer off, one for someone arriving at home, one for everyone leaving. Again, there might be a better way, but this does what I need:

Automations to raise/lower target temperature 
automation:
  - id: heating_timer_on
    alias: Turn heating on based on timer
    trigger:
      platform: state
      entity_id: input_boolean.heating_timer
      to: 'on'
    condition:
      condition: state
      entity_id: group.all_devices
      state: 'home'
    action:
      service: climate.set_temperature
      data_template:
        entity_id: climate.heating
        temperature: "{{ states('input_number.heating_temperature') }}"

  - id: heating_timer_off
    alias: Turn heating off based on timer
    trigger:
      platform: state
      entity_id: input_boolean.heating_timer
      to: 'off'
    action:
      service: climate.set_temperature
      data:
        entity_id: climate.heating
        temperature: 5

  - id: heating_on_when_get_home
    alias: Turn heating on on arrival if timer on
    trigger:
      platform: state
      entity_id: group.all_devices
      from: "not_home"
      to: "home"
    condition:
      condition: state
      entity_id: input_boolean.heating_timer
      state: 'on'
    action:
      service: climate.set_temperature
      data_template:
        entity_id: climate.heating
        temperature: "{{ states('input_number.heating_temperature') }}"

  - id: heating_off_when_leave_home
    alias: Turn heating off when we leave home
    trigger:
      platform: state
      entity_id: group.all_devices
      from: "home"
      to: "not_home"
    action:
      service: climate.set_temperature
      data:
        entity_id: climate.heating
        temperature: 5

Finally there’s the UI configuration, which I’ve done using Lovelace. The use of ‘:’ as the name for the climate.heating element is a kludge - I haven’t figured out yet how to name each individual data entry it adds to the history graph. I’m not particularly fond of the input method for controlling times - something closer to the Android analog clock with digits at the top would be nicer, but I’m not a UI guy and this works well enough.

Lovelace configuration for heating controls 
views:
  - title: Heating
    cards:
      - type: entities
        title: Controls
        show_header_toggle: false
        entities:
          - climate.heating
          - switch.gas_boiler
          - input_boolean.heating_timer
          - input_number.heating_temperature

      - type: history-graph
        title: Temperatures
        entities:
          - entity: sensor.kitchen_temperature
            name: Kitchen
          - entity: sensor.living_room_temperature
            name: Living Room
          - entity: sensor.master_bedroom_temperature
            name: Master Bedroom
          - entity: sensor.outside
            name: Outside
          - entity: sensor.study_temperature
            name: Study
          - entity: climate.heating
            name: ":"

      - type: entities
        title: Week day Timer
        entities:
          - input_datetime.weekday_morning_start
          - input_datetime.weekday_morning_stop
          - input_datetime.weekday_evening_start
          - input_datetime.weekday_evening_stop

      - type: entities
        title: Weekend Timer
        entities:
          - input_datetime.weekend_morning_start
          - input_datetime.weekend_morning_stop
          - input_datetime.weekend_evening_start
          - input_datetime.weekend_evening_stop

02 October, 2018 05:15PM

hackergotchi for Junichi Uekawa

Junichi Uekawa

Started Q4.

Started Q4. The strong typhoon passed by. 2018 is close to end already and I look back and wonder what I accomplished.

02 October, 2018 11:25AM by Junichi Uekawa