June 21, 2018

hackergotchi for Lars Wirzenius

Lars Wirzenius

Ick ALPHA-6 released: CI/CD engine

It gives me no small amount of satisfaction to announce the ALPHA-6 version of ick, my fledgling continuous integration and deployment engine. Ick has been now deployed and used by other people than myself.

Ick can, right now:

  • Build system trees for containers.
  • Use system trees to run builds in containers.
  • Build Debian packages.
  • Publish Debian packages via its own APT repository.
  • Deploy to a production server.

There's still many missing features. Ick is by no means ready to replace your existing CI/CD system, but if you'd like to have a look at ick, and help us make it the CI/CD system of your dreams, now is a good time to give it a whirl.

(Big missing features: web UI, building for multiple CPU architectures, dependencies between projects, good documentation, a development community. I intend to make all of these happen in due time. Help would be welcome.)

21 June, 2018 04:34PM

June 20, 2018

John Goerzen

Making a difference

Every day, ask yourself this question: What one thing can I do today that will make this democracy stronger and honor and support its institutions? It doesn’t have to be a big thing. And it probably won’t shake the Earth. The aggregation of them will shake the Earth.

– Benjamin Wittes

I have written some over the past year or two about the dangers facing the country. I have become increasingly alarmed about the state of it. And that Benjamin Wittes quote, along with the terrible tragedy, spurred me to action. Among other things, I did two things I never have done before:

I registered to protest on June 30.

I volunteered to do phone banking with SwingLeft.

And I changed my voter registration from independent to Republican.

No, I have not gone insane. The reason for the latter is that here in Kansas, the Democrats rarely field candidates for most offices. The real action happens in the Republican primary. So if I can vote in that primary, I can have a voice in keeping the crazy out of office. It’s not much, but it’s something.

Today we witnessed, hopefully, the first victory in our battle against the abusive practices happening to children at the southern border. Donald Trump caved, and in so doing, implicitly admitted the lies he and his administration have been telling about the situation. This only happened because enough people thought like Wittes: “I am small, but I can do SOMETHING.” When I called the three Washington offices of my senators and representatives — far-right Republicans all — it was apparent that I was by no means the first to give them an earful about this, and that they were changing their tone because of what they heard. Mind you, they hadn’t taken any ACTION yet, but the calls mattered. The reporting mattered. The attention mattered.

I am going to keep doing what little bit I can. I hope everyone else will too. Let us shake the Earth.

20 June, 2018 06:24PM by John Goerzen

hackergotchi for Julien Danjou

Julien Danjou

Stop merging your pull requests manually

Stop merging your pull requests manually

If there's something that I hate, it's doing things manually when I know I could automate them. Am I alone in this situation? I doubt so.

Nevertheless, every day, they are thousands of developers using GitHub that are doing the same thing over and over again: they click on this button:

Stop merging your pull requests manually

This does not make any sense.

Don't get me wrong. It makes sense to merge pull requests. It just does not make sense that someone has to push this damn button every time.

It does not make any sense because every development team in the world has a known list of pre-requisite before they merge a pull request. Those requirements are almost always the same, and it's something along those lines:

  • Is the test suite passing?
  • Is the documentation up to date?
  • Does this follow our code style guideline?
  • Have N developers reviewed this?

As this list gets longer, the merging process becomes more error-prone. "Oops, John just clicked on the merge button while there were not enough developer that reviewed the patch." Rings a bell?

In my team, we're like every team out there. We know what our criteria to merge some code into our repository are. That's why we set up a continuous integration system that runs our test suite each time somebody creates a pull request. We also require the code to be reviewed by 2 members of the team before it's approbated.

When those conditions are all set, I want the code to be merged.

Without clicking a single button.

That's exactly how Mergify started.

Stop merging your pull requests manually

Mergify is a service that pushes that merge button for you. You define rules in the .mergify.yml file of your repository, and when the rules are satisfied, Mergify merges the pull request.

No need to press any button.

Take a random pull request, like this one:

Stop merging your pull requests manually

This comes from a small project that does not have a lot of continuous integration services set up, just Travis. In this pull request, everything's green: one of the owners reviewed the code, and the tests are passing. Therefore, the code should be already merged: but it's there, hanging, chilling, waiting for someone to push that merge button. Someday.

With Mergify enabled, you'd just have to put this .mergify.yml a the root of the repository:

rules:
  default:
    protection:
      required_status_checks:
        contexts:
          - continuous-integration/travis-ci
      required_pull_request_reviews:
        required_approving_review_count: 1

With such a configuration, Mergify enables the desired restrictions, i.e., Travis passes, and at least one project member reviewed the code. As soon as those conditions are positive, the pull request is automatically merged.

We built Mergify as a free service for open-source projects. The engine powering the service is also open-source.

Now go check it out and stop letting those pull requests hang out one second more. Merge them!

If you have any question, feel free to ask us or write a comment below! And stay tuned — as Mergify offers a few other features that I can't wait to talk about!

20 June, 2018 03:53PM by Julien Danjou

Craig Small

Odd dependency on Google Chrome

For weeks I have had problems with Google Chrome. It would work very few times and then for reasons I didn’t understand, would stop working. On the command line you would get several screens of text, but never would the Chrome window appear.

So I tried the Beta, and it worked… once.

Deleted all the cache and configuration and it worked… once.

Every time the process would be in an infinite loop listening to a Unix socket (fd 7) but no window for the second and subsequent starts of Chrome.

By sheer luck in the screenfulls of spam I noticed this:

Gkr-Message: 21:07:10.883: secret service operation failed: The name org.freedesktop.secrets was not provided by any .service files

Hmm, so I noticed every time I started a fresh new Chrome, I logged into my Google account. So, once again clearing things I started Chrome, didn’t login and closed and reopened.  I had Chrome running the second time! Alas, not with all the stuff synchronised.

An issue for Mailspring put me onto the right path. installing gnome-keyring (or the dependencies p11-kit and gnome-keyring-pkcs11) fixed Chrome.

So if Chrome starts but you get no window, especially if you use cinnamon, try that trick.

 

 

20 June, 2018 11:21AM by Craig

hackergotchi for Jonathan Carter

Jonathan Carter

Plans for DebCamp18

Dates

I’m going to DebCamp18! I should arrive at NCTU around noon on Saturday, 2018-07-21.

My Agenda

  • DebConf Video: Research if/how MediaDrop can be used with existing Debian video archive backends (basically, just a bunch of files on http).
  • DebConf Video: Take a better look at PeerTube and prepare a summary/report for the video team so that we better know if/how we can use it for publishing videos.
  • Debian Live: I have a bunch of loose ideas that I’d like to formalize before then. At the very least I’d like to file a bunch of paper cut bugs for the live images that I just haven’t been getting to. Live team may also need some revitalization, and better co-ordination with packagers of the various desktop environments in terms of testing and release sign-offs. There’s a lot to figure out and this is great to do in person (might lead to a DebConf BoF as well).
  • Debian Live: Current live weekly images have Calamares installed, although it’s just a test and there’s no indication yet on whether it will be available on the beta or final release images, we’ll have to do a good assessment on all the consequences and weigh up what will work out the best. I want to put together an initial report with live team members who are around.
  • AIMS Desktop: Get core AIMS meta-packages in to Debian… no blockers on this but just haven’t had enough quite time to do it (And thanks to AIMS for covering my travel to Hsinchu!)
  • Get some help on ITPs that have been a little bit more tricky than expected:
    • gamemode – Adjust power saving and cpu governor settings when launching games
    • notepadqq – A linux clone of notepad++, a popular text editor on Windows
    • Possibly finish up zram-tools which I just don’t get the time for. It aims to be a set of utilities to manage compressed RAM disks that can be used for temporary space, compressed in-memory swap, etc.
  • Debian Package of the Day series: If there’s time and interest, make some in-person videos with maintainers about their packages.
  • Get to know more Debian people, relax and socialize!

20 June, 2018 08:32AM by jonathan

Athos Ribeiro

Triggering Debian Builds on OBS

This is my fifth post of my Google Summer of Code 2018 series. Links for the previous posts can be found below:

My GSoC contributions can be seen at the following links

Debian builds on OBS

OBS supports building Debian packages. To do so, one must properly configure a project so OBS knows it is building a .deb package and to have the packages needed to handle and build debian packages installed.

openSUSE’s OBS instance has repositories for Debian 8, Debian 9, and Debian testing.

We will use base Debian projects in our OBS instance as Download on Demand projects and use subprojects to achieve our final goal (build packages agains Clang). By using the same configurations as the ones in the openSUSE public projects, we could perform builds in Debian 8 and Debian 9 in our local OBS deploys. However, builds for Debian Testing and Unstable were failing.

With further investigation, we realized the OBS version packaged in Debian cannot decompress control.tar.xz files in .deb packages, which is the default compression format for the control tarball since dpkg-1.19 (it used to be control.tar.gz before that). This issue was reported on the OBS repositories and was fixed on a Pull Request that is not included in the current Debian OBS version yet. For now, we apply this patch in our OBS instance on our salt states.

After applying the patch, the builds on Debian 8 and 9 are still finishing with success, but builds against Debian Testing and Unstable are getting stuck in a blocked state: dependencies are being downloaded, the OBS scheduler stalls for a while, the downloaded packages get cleaned up, and then the dependencies are downloaded again. OBS backend enters in a loop doing the described procedure and never assigns a build to a worker. No logs with hints leading to a possible issue are issued, giving us no clue of the current problem.

Although I am inclined to believe we have a problem with our dependencies list, I am still debugging this issue during this week and will bring more news on my next post.

Refactoring project configuration files

Reshabh opened a Pull Request in our salt repository with the OBS configuration files for Ubuntu, also based on the openSUSE’s OBS public configurations. Based on Sylvestre comments, I have been refactoring the Debian configuration files based on the OBS docuemtation. One of the proposed improvements is to use debootstrap to mount the builder chroot. This will allow us to reduce the number of dependencies listed in the projects configuration files. The issue which generated debootstrap support in OBS is available at https://github.com/openSUSE/obs-build/issues/111 and may lead to more interesting resources on the matter.

Next steps (A TODO list to keep on the radar)

  • Fix OBS builds on Debian Testing and Unstable
  • Write patches for the OBS worker issue described in post 3
  • Change the default builder to perform builds with clang
  • Trigger new builds by using the dak/mailing lists messages
  • Verify the rake-tasks.sh script idempotency and propose patch to opencollab repository
  • Separate salt recipes for workers and server (locally)
  • Properly set hostnames (locally)

20 June, 2018 02:26AM

June 19, 2018

Shashank Kumar

Google Summer of Code 2018 with Debian - Week 5

During week 5, there were 3 merge requests undergoing review process simultaneously. I learned a lot about how code should be written in order to assist the reader since the code is read more times than the time it is written.

Services and Utility

After the user has entered their information on the signin or signup screen, the job of querying the database was given to a module named updatedb. The job of updatedb was to clean user input, hash password, query the database and respond with appropriate result after the database query is executed. In a discussion with Sanyam, he said updatedb doesn't conform to its name with what functions it incorporated. And explained the virtue of Service and Utility modules/functions and that this is the best place to restructure code with the same.

Utility functions can be described roughly as the functions which perform some operations on the data without caring much about the relationship of the data with respect to the application. So, generating uuid, cleaning email address, cleaning full name and hashing password becomes out utility functions and can be seen in utils.py for signup and similarly for signin.

Service functions can be described roughly as the functions which while performing operations on the data take their relationship with the application into account. Hence, these functions are not generic and application specific. sign_up_user is one such service function which received user information, calls utility functions to modify that information, query the database with respect to the signup operation i.e. adding the new user's detail to the database or raise SignUpError if details are already present. This can be seen in services module for signup and signin as well.

Persisting database connection

This is how the connection to the database used to work before the review. The settings module used to create the connection to the database, create table schema if not present and close the connection. Few constants are saved in the module to be used by signup and signin in order to connect to the database. But, the problem is, now database connection has to be established everytime there's a query to be executed by the services of signup or signin. Since the sqlite3 database is saved in a file alongside the application, I though it'll not be a problem to make connection whenever needed. But it overhead on the OS now which can slow down the application when scaled. To resolve this, now settings return the connection object which can be used again in any other module.

Integrating SignUp with Dashboard

While the SignUp feature was being reviewed the Dashbaord was merged and I had to refactor SignUp merge request accordingly. The natural flow of this should be the SignUp being the default screen up on the UI and after successful signup operation the Dashboard should be displayed. To achieve such a flow, I used screen manager which handles different screens and transition between them with predefined animation. This is defined in main module and the entire flow can be seen in action below.

Designing Tutorials and Tools menu

Once user is on the Dashboard, they have an option of picking up from different modules and going through the tutorials and tools available in the respective modules. The idea is to display difficulty tip as well so it becomes easier for the user to begin. Hence, below is what I've designed in order to incorporate the same.

New Contributor Wizard - Tutorials and Tools Menu

Implementing Tutorials and Tools menu

Now comes the fun part, thinking about the architecture of the modules just designed in order for them to take shape of some code in the application. The idea here is to define them in a json file to be picked from the respective module afterwards. This way it'll be easier to add new tutorials and tools and hence we have this resultant json. The developement of this feature can be followed on this merge request

Now remains the quest to design and implement the structure of tutorials which can be generalized in a way that it can be populated using a json file. This will provide flexibility to the developer of tutorials and a UI module can also be implemented to modify this json to add new tutorials without even knowing how to code. Sounds amazing right? We'll see how it works out soon. If you have any suggestions this make sure to comment down below, on the merge request or reach out to me.

The Conclusion

Since the SignUp has also been merged I'll have to refactor SignIn now to integrate all of it in one happy application and complete the natural flow of things. Also, the design and development of tools/tutorials is underway and by the next blog is out you might be able to test the application with atleast one tool or tutorial from one of the modules on the dashboard.

19 June, 2018 06:30PM by Shashank Kumar

hackergotchi for Benjamin Mako Hill

Benjamin Mako Hill

How markets coopted free software’s most powerful weapon (LibrePlanet 2018 Keynote)

Several months ago, I gave the closing keynote address at LibrePlanet 2018. The talk was about the thing that scares me most about the future of free culture, free software, and peer production.

A video of the talk is online on Youtube and available as WebM video file (both links should skip the first 3m 19s of thanks and introductions).

Here’s a summary of the talk:

App stores and the so-called “sharing economy” are two examples of business models that rely on techniques for the mass aggregation of distributed participation over the Internet and that simply didn’t exist a decade ago. In my talk, I argue that the firms pioneering these new models have learned and adapted processes from commons-based peer production projects like free software, Wikipedia, and CouchSurfing.

The result is an important shift: A decade ago,  the kind of mass collaboration that made Wikipedia, GNU/Linux, or Couchsurfing possible was the exclusive domain of people producing freely and openly in commons. Not only is this no longer true, new proprietary, firm-controlled, and money-based models are increasingly replacing, displacing, outcompeting, and potentially reducing what’s available in the commons. For example, the number of people joining Couchsurfing to host others seems to have been in decline since Airbnb began its own meteoric growth.

In the talk, I talk about how this happened and what I think it means for folks of that are committed to working in commons. I talk a little bit about the free culture and free software should do now that mass collaboration, these communities’ most powerful weapon, is being used against them.

I’m very much interested in feedback provided any way you want to reach me including in person, over email, in comments on my blog, on Mastodon, on Twitter, etc.


Work on the research that is reflected and described in this talk was supported by the National Science Foundation (awards IIS-1617129 and IIS-1617468). Some of the initial ideas behind this talk were developed while working on this paper (official link) which was led by Maximilian Klein and contributed to by Jinhao Zhao, Jiajun Ni, Isaac Johnson, and Haiyi Zhu.

19 June, 2018 06:03PM by Benjamin Mako Hill

hackergotchi for Sean Whitton

Sean Whitton

I'm going to DebCamp18, Hsinchu, Taiwan

Here’s what I’m planning to work on – please get in touch if you want to get involved with any of these items.

DebCamp work

Throughout DebCamp and DebConf

  • Debian Policy: sticky bugs; process; participation; translations

  • Helping people use dgit and git-debrebase

    • Writing up or following up on feature requests and bugs

    • Design work with Ian and others

19 June, 2018 03:43PM

hackergotchi for Raphaël Hertzog

Raphaël Hertzog

Freexian’s report about Debian Long Term Support, May 2018

A Debian LTS logoLike each month, here comes a report about the work of paid contributors to Debian LTS.

Individual reports

In May, about 202 work hours have been dispatched among 12 paid contributors. Their reports are available:

Evolution of the situation

The number of sponsored hours increased to 190 hours per month thanks to a few new sponsors who joined to benefit from Wheezy’s Extended LTS support.

We are currently in a transition phase. Wheezy is no longer supported by the LTS team and the LTS team will soon take over security support of Debian 8 Jessie from Debian’s regular security team.

Thanks to our sponsors

New sponsors are in bold.

No comment | Liked this article? Click here. | My blog is Flattr-enabled.

19 June, 2018 08:27AM by Raphaël Hertzog

hackergotchi for Erich Schubert

Erich Schubert

Predatory publishers: SciencePG

I got spammed again by SciencePG (“Science Publishing Group”).

One of many (usually Chinese or Indian) fake publishers, that will publish anything as long as you pay their fees. But, unfortunately, once you published a few papers, you inevitably land on their spam list: they scrape the websites of good journals for email adresses, and you do want your contact email address on your papers.

However, this one is particularly hilarious: They have a spelling error right at the top of their home page!

SciencePG spelling

Fail.

Speaking of fake publishers. Here is another fun example:

Kim Kardashian, Satoshi Nakamoto, Tomas Pluskal
Wanion: Refinement of RPCs.
Drug Des Int Prop Int J 1(3)- 2018. DDIPIJ.MS.ID.000112.

Yes, that is a paper in the “Drug Designing & Intellectual Properties” International (Fake) Journal. And the content is a typical SciGen generated paper that throws around random computer buzzword and makes absolutely no sense. Not even the abstract. The references are also just made up. And so are the first two authors, VIP Kim Kardashian and missing Bitcoin inventor Satoshi Nakamoto…

In the PDF version, the first headline is “Introductiom”, with “m”…

So Lupine Publishers is another predatory publisher, that does not peer review, nor check if the article is on topic for the journal.

Via Retraction Watch

Conclusion: just because it was published somewhere does not mean this is real, or correct, or peer reviewed…

19 June, 2018 08:12AM by Erich Schubert

Reproducible builds folks

Reproducible Builds: Weekly report #164

Here’s what happened in the Reproducible Builds effort between Sunday June 10 and Saturday June 16 2018:

diffoscope development

diffoscope is our in-depth “diff-on-steroids” utility which helps us diagnose reproducibility issues in packages. This week, version 96 was uploaded to Debian unstable by Chris Lamb. It includes contributions already covered by posts in previous weeks as well as new ones from:

tests.reproducible-builds.org development

There were a number of changes to our Jenkins-based testing framework that powers tests.reproducible-builds.org, including:

Packages reviewed and fixed, and bugs filed

Misc.

This week’s edition was written by Bernhard M. Wiedemann, Chris Lamb & reviewed by a bunch of Reproducible Builds folks on IRC & the mailing lists.

19 June, 2018 07:40AM

Arthur Del Esposte

GSoC Status Update - First Month

In the past month I have been working on my GSoC project in Debian’s Distro Tracker. This project aims at designing and implementing new features in Distro Tracker to better support Debian teams to track the health of their packages and to prioritize their work efforts. In this post, I will describe the current status of my contributions, highlight the main challenges, and point the next steps.

Work Management and Communication

I communicate with Lucas Kanashiro (my mentor) constantly via IRC and personally at least once a week as we live in the same city. We have a weekly meeting with Raphael Hertzog at #debian-qa IRC channel to report advances, collect feedback, solve technical doubts, and planning the next steps.

I created a new repository in Salsa to save the log of our IRC meetings and to track my tasks through the repository’s issue tracker.Besides that, once a month I’ll post a new status update in my blog, such as this one, with more details regarding my contributions.

Advances

When GSoC officially started, Distro Tracker already had some team-related features. Briefly, a team is an entity composed by one or more users that are interested in the same set of packages. Teams are created manually by users and anyone may join public teams. The team page aggregates some basic information about the team and the list of packages of interest.

Distro Tracker offers a page to enable users to browser public teams which shows a paginated, sorted list of names. It used to be hard to find a team based on this list since Distro Tracker has more 110 teams distributed over 6 pages. In this sense, I created a new search field with auto-complete on the top of teams page to enable users to find a team’s page faster, as show in the following figure:

Search Field for Teams Page

Also, I have been working on improving the current teams infrastructure to enable Debian’s teams to better track the health of their packages. Initially, we decided to use the current data available in Distro Tracker to create the first version of a new team’s page based on PET.

Presenting team’s packages data in a table on the team’s page would be a relatively trivial task. However, Distro Tracker architecture aims to provide a generic core which can be extended through specific distro applications, such as Kali Linux. The core source code provides generic infrastructure to import data related to deb packages and also to present them in HTML pages. Therefore, we had to consider this Distro Tracker requirement to properly provide a extensible infrastructure to show packages data through tables in so that it would be easy to add new table fields and to change the default behavior of existing columns provided by the core source code.

So, based on the previously existing panels feature and on Hertzog’s suggestions, I designed and developed a framework to create customizable package tables for teams. This framework is composed of two main classes:

  • BaseTableField - A base class representing fields to be displayed on package tables. Among other things, it must define the column name and a template to render the cell content for a package.
  • BasePackageTable - A base class representing package tables which are displayed on a team page. It may have several BaseTableFields to display package’s information. Different tables may show a different list of packages based on its scope.

We have been discussing my implementation in an open Merge Request, although we are very close to the version that should be incorporated. The following figures show the comparison between the earlier PET’s table and our current implementation.

PET Packages Table Distro Tracker Packages Table
PET Packages Table Current Teams Page

Currently, the team’s page only have one table, which displays all packages related to that team. We are already presenting a very similar set of data to PET’s table. More specifically, the following columns are shown:

  • Package - displays the package name on the cell. It is implemented by the core’s GeneralInformationTableField class
  • VCS - by default, it displays the type of package’s repository (i.e. GIT, SVN) or Unknown. It is implemented by the core’s VcsTableField class. However, Debian app extend this behavior by adding the changelog version on the latest repository tag and displaying issues identified by Debian’s VCS Watch.
  • Archive - displays the package version on distro archive. It is implemented by the core’s ArchiveTableField class.
  • Bugs - displays the total number of bugs of a package. It is implemented by the core’s BugsTableField class. Ideally, each third-party apps should extend this field table to both add links for their bug tracker system.
  • Upstream - displays the upstream latest version available. This is a specific table field implemented by Debian app since this data is imported through Debian-specific tasks. In this sense, it is not available for other distros.

As the table’s cells are small to present detailed information, we have added Popper.js, a javascript library to display popovers. In this sense, some columns show a popover with more details regarding its content which is displayed on mouse hover. The following figure shows the popover to the Package column:

Package's Popover

In additional to designing the table framework, the main challenge were to avoid the N+1 problem which introduces performance issues since for a set of N packages displayed in a table, each field element must perform 1 or more lookup for additional data for a given package. To solve this problem, each subclass of BaseTableField must define a set of Django’s Prefetch objects to enable BasePackageTable objects to load all required data in batch in advance through prefetch_related, as listed bellow.

class BasePackageTable(metaclass=PluginRegistry):
    @property
    def packages_with_prefetch_related(self):
        """
        Returns the list of packages with prefetched relationships defined by
        table fields
        """
        package_query_set = self.packages
        for field in self.table_fields:
            for l in field.prefetch_related_lookups:
                package_query_set = package_query_set.prefetch_related(l)

        additional_data, implemented = vendor.call(
            'additional_prefetch_related_lookups'
        )
        if implemented and additional_data:
            for l in additional_data:
                package_query_set = package_query_set.prefetch_related(l)
        return package_query_set

    @property
    def packages(self):
        """
        Returns the list of packages shown in the table. One may define this
        based on the scope
        """
        return PackageName.objects.all().order_by('name')


class ArchiveTableField(BaseTableField):
    prefetch_related_lookups = [
        Prefetch(
            'data',
            queryset=PackageData.objects.filter(key='general'),
            to_attr='general_archive_data'
        ),
        Prefetch(
            'data',
            queryset=PackageData.objects.filter(key='versions'),
            to_attr='versions'
        )
    ]

    @cached_property
    def context(self):
        try:
            info = self.package.general_archive_data[0]
        except IndexError:
            # There is no general info for the package
            return

        general = info.value

        try:
            info = self.package.versions[0].value
            general['default_pool_url'] = info['default_pool_url']
        except IndexError:
            # There is no versions info for the package
            general['default_pool_url'] = '#'

        return general

Finally, it is worth noticing that we also improved the team’s management page by moving all team management features to a single page and improving its visual structure:

Teams Management

Next Steps

Now, we are moving towards adding other tables with different scopes, such as the tables presented by PET:

PET tables

To this end, we will introduce the Tag model class to categorize the packages based on their characteristics. Thus, we will create an additional task responsible for tagging packages based on their available data. The relationship between packages and tags should be ManyToMany. In the end, we want to perform a simple query to define the scope of a new table, such as the following example to query all packages with Release Critical (RC) bugs:

class RCPackageTable(BasePackageTable):
    def packages(self):
      tag = Tag.objects.filter(name='rc-bugs')
      return tag.packages.all()

We probably will need to work on Debian’s VCSWatch to enable it to receive update through Salsa’s webhook, especially for real-time monitoring of repositories.


Let’s get moving on! \m/

19 June, 2018 03:00AM

hackergotchi for Gunnar Wolf

Gunnar Wolf

Demoting multi-factor authentication

I started teaching at Facultad de Ingeniería, UNAM in January 2013. Back then, I was somewhat surprised (for good!) that the university required me to create a digital certificate for registering student grades at the end of the semester. The setup had some not-so-minor flaws (i.e. the private key was not generated at my computer but centrally, so there could be copies of it outside my control — Not only could, but I noted for a fact a copy was kept at the relevant office at my faculty, arguably to be able to timely help poor teachers if they lost their credentials or patience), but was decent...
Authentication was done via a Java applet, as there needs to be a verifiably(?)-secure way to ensure the certificate was properly checked at the client without transfering it over the network. Good thing!
But... Java applets grow out of favor. I don't think I have ever been able to register my grading from a Linux desktop (of course, I don't have a typical Linux desktop, so luck might smile to other people). But last semester and this semester I suffered even to get the grades registered from Windows — Seems that every browser has deprecated the extensions for the Java runtime, and applets are no longer a thing. I mean, I could get the Oracle site to congratulate me for having Java 8 installed, but it just would not run the university's applet!
So, after losing the better part of an already-busy evening... I got a mail. It says (partial translation mine):

Subject: Problems to electronically sign at UNAM

We are from the Advance Electronic Signature at UNAM. We are sending you this mail as we have detected you have problems to sign the grades, probably due to the usage of Java.

Currently, we have a new Electronic Signature system that does not use Java, we can migrate you to this system.
(...)

The certificate will thus be stored in the cloud, we will deposit it at signing time, you just have to enter the password you will have assigned.
(...)

Of course, I answered asking which kind of "cloud" was it, as we all know that the cloud does not exist, it's just other people's computers... And they decided to skip this question.

You can go see what is required for this implementation at https://www.fea.unam.mx/Prueba de la firma (Test your signature): It asks me for my CURP (publicly known number that identifies every Mexican resident). Then, it asks me for a password. And that's it. Yay :-Þ

Anyway I accepted, as losing so much time to grade is just too much. And... Yes, many people will be happy. Partly, I'm releieved by this (I have managed to hate Java for over 20 years). I am just saddened by the fact we have lost an almost-decent-enough electronic signature implementation and fallen back to just a user-password scheme. There are many ways to do crypto verification on the client side nowadays; I know JavaScript is sandboxed and cannot escape to touch my filesystem, but... It is amazing we are losing this simple and proven use case.

And it's amazing they are pulling it off as if it were a good thing.

19 June, 2018 01:11AM by gwolf

June 18, 2018

hackergotchi for Benjamin Mako Hill

Benjamin Mako Hill

Honey Buckets

When I was growing up in Washington state, a company called Honey Bucket held a dominant position in the local portable toilet market. Their toilets are still a common sight in the American West.

Honey Bucket brand portable toilet. Photo by donielle. (CC BY-SA)

They were so widespread when I was a child that I didn’t know that “Honey Bucket” was the name of a company at all until I moved to Massachusetts for college. I thought “honey bucket” was just the generic term for toilets that could be moved from place-to-place!

So for the first five years that I lived in Massachusetts, I continued to call all portable toilets “honey buckets.”

Until somebody asked me why I called them that—five years after moving!—all my friends in Massachusetts thought that “honey bucket” was just a personal, idiosyncratic, and somewhat gross, euphemism.

18 June, 2018 10:40PM by Benjamin Mako Hill

Russell Coker

Cooperative Learning

This post is about my latest idea for learning about computers. I posted it to my local LUG mailing list and received no responses. But I still think it’s a great idea and that I just need to find the right way to launch it.

I think it would be good to try cooperative learning about Computer Science online. The idea is that everyone would join an IRC channel at a suitable time with virtual machine software configured and try out new FOSS software at the same time and exchange ideas about it via IRC. It would be fairly informal and people could come and go as they wish, the session would probably go for about 4 hours but if people want to go on longer then no-one would stop them.

I’ve got some under-utilised KVM servers that I could use to provide test VMs for network software, my original idea was to use those for members of my local LUG. But that doesn’t scale well. If a larger group people are to be involved they would have to run their own virtual machines, use physical hardware, or use trial accounts from VM companies.

The general idea would be for two broad categories of sessions, ones where an expert provides a training session (assigning tasks to students and providing suggestions when they get stuck) and ones where the coordinator has no particular expertise and everyone just learns together (like “let’s all download a random BSD Unix and see how it compares to Linux”).

As this would be IRC based there would be no impediment for people from other regions being involved apart from the fact that it might start at 1AM their time (IE 6PM in the east coast of Australia is 1AM on the west coast of the US). For most people the best times for such education would be evenings on week nights which greatly limits the geographic spread.

While the aims of this would mostly be things that relate to Linux, I would be happy to coordinate a session on ReactOS as well. I’m thinking of running training sessions on etbemon, DNS, Postfix, BTRFS, ZFS, and SE Linux.

I’m thinking of coordinating learning sessions about DragonflyBSD (particularly HAMMER2), ReactOS, Haiku, and Ceph. If people are interested in DragonflyBSD then we should do that one first as in a week or so I’ll probably have learned what I want to learn and moved on (but not become enough of an expert to run a training session).

One of the benefits of this idea is to help in motivation. If you are on your own playing with something new like a different Unix OS in a VM you will be tempted to take a break and watch YouTube or something when you get stuck. If there are a dozen other people also working on it then you will have help in solving problems and an incentive to keep at it while help is available.

So the issues to be discussed are:

  1. What communication method to use? IRC? What server?
  2. What time/date for the first session?
  3. What topic for the first session? DragonflyBSD?
  4. How do we announce recurring meetings? A mailing list?
  5. What else should we setup to facilitate training? A wiki for notes?

Finally while I list things I’m interested in learning and teaching this isn’t just about me. If this becomes successful then I expect that there will be some topics that don’t interest me and some sessions at times when I am have other things to do (like work). I’m sure people can have fun without me. If anyone has already established something like this then I’d be happy to join that instead of starting my own, my aim is not to run another hobbyist/professional group but to learn things and teach things.

There is a Wikipedia page about Cooperative Learning. While that’s interesting I don’t think it has much relevance on what I’m trying to do. The Wikipedia article has some good information on the benefits of cooperative education and situations where it doesn’t work well. My idea is to have a self-selecting people who choose it because of their own personal goals in terms of fun and learning. So it doesn’t have to work for everyone, just for enough people to have a good group.

18 June, 2018 12:28PM by etbe

John Goerzen

Memories, Father’s Day, and an 89-year-old plane

“Oh! I have slipped the surly bonds of Earth
And danced the skies on laughter-silvered wings;
Sunward I’ve climbed, and joined the tumbling mirth
of sun-split clouds, — and done a hundred things”

– John Gillespie Magee, Jr.

I clicked on the radio transmitter in my plane.

O’Neill Traffic, Bonanza xx departing to the south. And Trimotor, thanks for flight #1. We really enjoyed it.

And we had. Off to my left, a 1929 Ford Trimotor airliner was heading off into the distance, looking as if it were just hanging in the air, glinting in the morning sun, 1000 feet above the ground. Earlier that morning, my boys and I had been passengers in that very plane. But now we had taken off right after them, as they were taking another load of passengers up for a flight and we were flying back home. To my right was my 8-year-old, and my 11-year-old was in back, both watching out the windows. The radio clicked on, and the three of us heard the other pilot’s response:

Oh thank you. We’re glad you came!

A few seconds later, they were gone out of sight.

The experience of flying in an 89-year-old airliner is quite something. As with the time we rode on the Durango & Silverton railroad, it felt like stepping back into a time machine — into the early heyday of aviation.

Jacob and Oliver had been excited about this day a long time. We had tried to get a ride when it was on tour in Oklahoma, much closer, but one of them got sick on the drive that day and it didn’t work out. So Saturday morning, we took the 1.5-hour-flight up to northern Nebraska. We’d heard they’d have a pancake breakfast fundraiser, and the boys were even more excited. They asked to set the alarm early, so we’d have no risk of missing out on airport pancakes.

Jacob took this photo of the sunrise at the airport while I was doing my preflight checks:

IMG_1574

Here’s one of the beautiful views we got as we flew north to meet the Trimotor.

IMG_20180616_070810_v1

It was quite something to share a ramp with that historic machine. Here’s a photo of our plane not far from the Trimotor.

IMG_20180616_082051

After we got there, we checked in for the flight, had a great pancake and sausage breakfast, and then into the Trimotor. The engines fired up with a most satisfying low rumble, and soon we were aloft — cruising along at 1000 feet, in that (by modern standards) noisy, slow, and beautiful machine. We explored the Nebraska countryside from the air before returning 20 minutes later. I asked the boys what they thought.

“AWESOME!” was the reply. And I agreed.

IMG_20180616_090828

Jacob and Oliver have long enjoyed pretending to be flight attendants when we fly somewhere. They want me to make airline-sounding announcements, so I’ll say something like, “This is your captain speaking. In a few moments, we’ll begin our descent into O’Neill. Flight attendants, prepare the cabin for arrival.” Then Jacob will say, “Please return your tray tables that you don’t have to their full upright and locked position, make sure your seat belt is tightly fastened, and your luggage is stowed. This is your last chance to visit the lavatory that we don’t have. We’ll be on the ground shortly.”

Awhile back, I loaded up some zip-lock bags with peanuts and found some particularly small bottles of pop. Since then, it’s become tradition on our longer flights for them to hand out bags of peanuts and small quantities of pop as we cruise along — “just like the airlines.” A little while back, I finally put a small fridge in the hangar so they get to choose a cold beverage right before we leave. (We don’t typically have such things around, so it’s a special treat.)

Last week, as I was thinking about Father’s Day, I told them how I remembered visiting my dad at work, and how he’d let me get a bottle of Squirt from the pop machine there (now somewhat rare). So when we were at the airport on Saturday, it brought me a smile to hear, “DAD! This pop machine has Squirt! Can we get a can? It’s only 75 cents!” “Sure – after our Trimotor flight.” “Great! Oh, thank you dad!”

I realized then I was passing a small but special memory on to another generation. I’ve written before of my childhood memories of my dad, and wondering what my children will remember of me. Martha isn’t old enough yet to remember her cackles of delight as we play peek-a-boo or the books we read at bedtime. Maybe Jacob and Oliver will remember our flights, or playing with mud, or researching dusty maps in a library, playing with radios, or any of the other things we do. Maybe all three of them will remember the cans of Squirt I’m about to stock that hangar fridge with.

But if they remember that I love them and enjoy doing things with them, they will have remembered the most important thing. And that is another special thing I got from my parents, and can pass on to another generation.

18 June, 2018 07:59AM by John Goerzen

hackergotchi for Steve Kemp

Steve Kemp

Monkeying around with intepreters - Result

So I challenged myself to writing a BASIC intepreter over the weekend, unfortunately I did not succeed.

What I did was take an existing monkey-repl and extend it with a series of changes to make sure that I understood all the various parts of the intepreter design.

Initially I was just making basic changes:

  • Added support for single-line comments.
    • For example "// This is a comment".
  • Added support for multi-line comments.
    • For example "/* This is a multi-line comment */".
  • Expand \n and \t in strings.
  • Allow the index operation to be applied to strings.
    • For example "Steve Kemp"[0] would result in S.
  • Added a type function.
    • For example "type(3.13)" would return "float".
    • For example "type(3)" would return "integer".
    • For example "type("Moi")" would return "string".

Once I did that I overhauled the built-in functions, allowing callers to register golang functions to make them available to their monkey-scripts. Using this I wrote a simple "standard library" with some simple math, string, and file I/O functions.

The end result was that I could read files, line-by-line, or even just return an array of the lines in a file:

 // "wc -l /etc/passwd" - sorta
 let lines = file.lines( "/etc/passwd" );
 if ( lines ) {
    puts( "Read ", len(lines), " lines\n" )
 }

Adding file I/O was pretty neat, although I only did reading. Handling looping over a file-contents is a little verbose:

 // wc -c /etc/passwd, sorta.
 let handle = file.open("/etc/passwd");
 if ( handle < 0 ) {
   puts( "Failed to open file" )
 }

 let c = 0;       // count of characters
 let run = true;  // still reading?

 for( run == true ) {

    let r = read(handle);
    let l = len(r);
    if ( l > 0 ) {
        let c = c + l;
    }
    else {
        let run = false;
    }
 };

 puts( "Read " , c, " characters from file.\n" );
 file.close(handle);

This morning I added some code to interpolate hash-values into a string:

 // Hash we'll interpolate from
 let data = { "Name":"Steve", "Contact":"+358449...", "Age": 41 };

 // Expand the string using that hash
 let out = string.interpolate( "My name is ${Name}, I am ${Age}", data );

 // Show it worked
 puts(out + "\n");

Finally I added some type-conversions, allowing strings/floats to be converted to integers, and allowing other value to be changed to strings. With the addition of a math.random function we then got:

 // math.random() returns a float between 0 and 1.
 let rand = math.random();

 // modify to make it from 1-10 & show it
 let val = int( rand * 10 ) + 1 ;
 puts( "math.random() -> ", val , "\n");

The only other signification change was the addition of a new form of function definition. Rather than defining functions like this:

 let hello = fn() { puts( "Hello, world\n" ) };

I updated things so that you could also define a function like this:

 function hello() { puts( "Hello, world\n" ) };

(The old form still works, but this is "clearer" in my eyes.)

Maybe next weekend I'll try some more BASIC work, though for the moment I think my monkeying around is done. The world doesn't need another scripting language, and as I mentioned there are a bunch of implementations of this around.

The new structure I made makes adding a real set of standard-libraries simple, and you could embed the project, but I'm struggling to think of why you would want to. (Though I guess you could pretend you're embedding something more stable than anko and not everybody loves javascript as a golang extension language.)

18 June, 2018 03:15AM

Arthur Del Esposte

GSoC Status Update - First Month

In the past month I have been working on my GSoC project in Debian’s Distro Tracker. This project aims at designing and implementing new features in Distro Tracker to better support Debian teams to track the health of their packages and to prioritize their work efforts. In this post, I will describe the current status of my contributions, highlight the main challenges, and point the next steps.

Work Management and Communication

I communicate with Lucas Kanashiro (my mentor) constantly via IRC and personally at least once a week as we live in the same city. We have a weekly meeting with Raphael Hertzog at #debian-qa IRC channel to report advances, collect feedback, solve technical doubts, and planning the next steps.

I created a new repository in Salsa to save the log of our IRC meetings and to track my tasks through the repository’s issue tracker.Besides that, once a month I’ll post a new status update in my blog, such as this one, with more details regarding my contributions.

Advances

When GSoC officially started, Distro Tracker already had some team-related features. Briefly, a team is an entity composed by one or more users that are interested in the same set of packages. Teams are created manually by users and anyone may join public teams. The team page aggregates some basic information about the team and the list of packages of interest.

Distro Tracker offers a page to enable users to browser public teams which shows a paginated, sorted list of names. It used to be hard to find a team based on this list since Distro Tracker has more 110 teams distributed over 6 pages. In this sense, I created a new search field with auto-complete on the top of teams page to enable users to find a team’s page faster, as show in the following figure:

Search Field for Teams Page

Also, I have been working on improving the current teams infrastructure to enable Debian’s teams to better track the health of their packages. Initially, we decided to use the current data available in Distro Tracker to create the first version of a new team’s page based on PET.

Presenting team’s packages data in a table on the team’s page would be a relatively trivial task. However, Distro Tracker architecture aims to provide a generic core which can be extended through specific distro applications, such as Kali Linux. The core source code provides generic infrastructure to import data related to deb packages and also to present them in HTML pages. Therefore, we had to consider this Distro Tracker requirement to properly provide a extensible infrastructure to show packages data through tables in so that it would be easy to add new table fields and to change the default behavior of existing columns provided by the core source code.

So, based on the previously existing panels feature and on Hertzog’s suggestions, I designed and developed a framework to create customizable package tables for teams. This framework is composed of two main classes:

  • BaseTableField - A base class representing fields to be displayed on package tables. Among other things, it must define the column name and a template to render the cell content for a package.
  • BasePackageTable - A base class representing package tables which are displayed on a team page. It may have several BaseTableFields to display package’s information. Different tables may show a different list of packages based on its scope.

We have been discussing my implementation in an open Merge Request, although we are very close to the version that should be incorporated. The following figures show the comparison between the earlier PET’s table and our current implementation.

PET Packages Table Distro Tracker Packages Table
PET Packages Table Current Teams Page

Currently, the team’s page only have one table, which displays all packages related to that team. We are already presenting a very similar set of data to PET’s table. More specifically, the following columns are shown:

  • Package - displays the package name on the cell. It is implemented by the core’s GeneralInformationTableField class
  • VCS - by default, it displays the type of package’s repository (i.e. GIT, SVN) or Unknown. It is implemented by the core’s VcsTableField class. However, Debian app extend this behavior by adding the changelog version on the latest repository tag and displaying issues identified by Debian’s VCS Watch.
  • Archive - displays the package version on distro archive. It is implemented by the core’s ArchiveTableField class.
  • Bugs - displays the total number of bugs of a package. It is implemented by the core’s BugsTableField class. Ideally, each third-party apps should extend this field table to both add links for their bug tracker system.
  • Upstream - displays the upstream latest version available. This is a specific table field implemented by Debian app since this data is imported through Debian-specific tasks. In this sense, it is not available for other distros.

As the table’s cells are small to present detailed information, we have added Popper.js, a javascript library to display popovers. In this sense, some columns show a popover with more details regarding its content which is displayed on mouse hover. The following figure shows the popover to the Package column:

Package's Popover

In additional to designing the table framework, the main challenge were to avoid the N+1 problem which introduces performance issues since for a set of N packages displayed in a table, each field element must perform 1 or more lookup for additional data for a given package. To solve this problem, each subclass of BaseTableField must define a set of Django’s Prefetch objects to enable BasePackageTable objects to load all required data in batch in advance through prefetch_related, as listed bellow.

class BasePackageTable(metaclass=PluginRegistry):
    @property
    def packages_with_prefetch_related(self):
        """
        Returns the list of packages with prefetched relationships defined by
        table fields
        """
        package_query_set = self.packages
        for field in self.table_fields:
            for l in field.prefetch_related_lookups:
                package_query_set = package_query_set.prefetch_related(l)

        additional_data, implemented = vendor.call(
            'additional_prefetch_related_lookups'
        )
        if implemented and additional_data:
            for l in additional_data:
                package_query_set = package_query_set.prefetch_related(l)
        return package_query_set

    @property
    def packages(self):
        """
        Returns the list of packages shown in the table. One may define this
        based on the scope
        """
        return PackageName.objects.all().order_by('name')


class ArchiveTableField(BaseTableField):
    prefetch_related_lookups = [
        Prefetch(
            'data',
            queryset=PackageData.objects.filter(key='general'),
            to_attr='general_archive_data'
        ),
        Prefetch(
            'data',
            queryset=PackageData.objects.filter(key='versions'),
            to_attr='versions'
        )
    ]

    @cached_property
    def context(self):
        try:
            info = self.package.general_archive_data[0]
        except IndexError:
            # There is no general info for the package
            return

        general = info.value

        try:
            info = self.package.versions[0].value
            general['default_pool_url'] = info['default_pool_url']
        except IndexError:
            # There is no versions info for the package
            general['default_pool_url'] = '#'

        return general

Finally, it is worth noticing that we also improved the team’s management page by moving all team management features to a single page and improving its visual structure:

Teams Management

Next Steps

Now, we are moving towards adding other tables with different scopes, such as the tables presented by PET:

PET tables

To this end, we will introduce the Tag model class to categorize the packages based on their characteristics. Thus, we will create an additional task responsible for tagging packages based on their available data. The relationship between packages and tags should be ManyToMany. In the end, we want to perform a simple query to define the scope of a new table, such as the following example to query all packages with Release Critical (RC) bugs:

class RCPackageTable(BasePackageTable):
    def packages(self):
      tag = Tag.objects.filter(name='rc-bugs')
      return tag.packages.all()

We probably will need to work on Debian’s VCSWatch to enable it to receive update through Salsa’s webhook, especially for real-time monitoring of repositories.


Let’s get moving on! \m/

18 June, 2018 03:00AM

June 17, 2018

hackergotchi for Clint Adams

Clint Adams

Before the combination with all the asterisks

We assembled at the rally point on the wrong side of the tracks. When consensus was achieved, we began our march to the Candy Kingdom. Before we had made it even a single kilometer, a man began yelling at us.

„It’s not here,” he exclaimed. “It’s that way.”

This seemed incredible. It became apparent that, despite his fedora, he was probably the King of Ooo.

Nevertheless, we followed him in the direction he indicated. He did not offer us space in his vehicle, but we managed to catch up eventually.

„It’s to the right of the cafe. Look for сиська,” he announced.

It occurred to me that the only sign I had seen that said сиська was right by where he had intercepted us. It also occurred to me that the cafe had three sides, and “right” was rather ambiguous.

There was much confusion until the Banana Man showed up.

Posted on 2018-06-17
Tags: mintings

17 June, 2018 09:49PM

hackergotchi for Bits from Debian

Bits from Debian

Debian Artwork: Call for Proposals for Debian 10 (Buster)

This is the official call for artwork proposals for the Buster cycle.

For the most up to date details, please refer to the wiki.

We would also like to take this opportunity to thank Juliette Taka Belin for doing the Softwaves theme for stretch.

The deadlines for submissions is: 2018-09-05

The artwork is usually picked based on which themes look the most:

  • ''Debian'': admittedly not the most defined concept, since everyone has their own take on what Debian means to them.
  • ''plausible to integrate without patching core software'': as much as we love some of the insanely hot looking themes, some would require heavy GTK+ theming and patching GDM/GNOME.
  • ''clean / well designed'': without becoming something that gets annoying to look at a year down the road. Examples of good themes include Joy, Lines and Softwaves.

If you'd like more information, please use the Debian Desktop mailing list.

17 June, 2018 11:30AM by Jonathan Carter

June 16, 2018

Arturo Borrero González

Netfilter Workshop 2018 Berlin summary

Netfilter logo

This weekend we had Netfilter Workshop 2018 in Berlin, Germany.

Lots of interesting talks happened, mostly surrounding nftables and how to move forward from the iptables legacy world to the new, modern nft framework.

In a nutshell, the Netfilter project, the FLOSS community driven project, has agreed to consider iptables as a legacy tool. This confidence comes from the maturity of the nftables framework, which is fairly fully-compliant with the old iptables API, including extensions (matches and targets).

Starting now, next iptables upstream releases will include the old iptables binary as /sbin/iptables-legacy, and the same for the other friends.

To summarize:

  • /sbin/iptables-legacy
  • /sbin/iptables-legacy-save
  • /sbin/iptables-legacy-restore
  • /sbin/ip6tables-legacy
  • /sbin/ip6tables-legacy-save
  • /sbin/ip6tables-legacy-restore
  • /sbin/arptables-legacy
  • /sbin/ebtables-legacy

The new binary will be using the nf_tables kernel backend instead, what was formely known as ‘iptables-compat’. Should you find some rough edges with the new binary, you could always use the old -legacy tools. This is for people who want to keep using the old iptables semantics, but the recommendation is to migrate to nftables as soon as possible.

Moving to nftables will add the benefits of improved performance, new features, new semantics, and in general, a modern framework. All major distributions will implement these changes soon, including RedHat, Fedora, CentOS, Suse, Debian and derivatives. We also had some talks regarding firewalld, the firewalling service in use by some rpm-based distros. They gained support for nftables starting with v0.6.0. This is great news, since firewalld is the main firewalling top-level mechanism in these distributions. Good news is that the libnftables high level API is in great shape. It recently gained a new high level JSON API thanks to Phil Sutter. The firewalld tool will use this new JSON API soon.

I gave a talk about the status of Netfilter software packages at Debian, and shared my plans to implement these iptables -> nftables changes in the near future.

We also had an interesting talk by a CloudFlare engineer about how they use the TPROXY Netfilter infraestructure to serve thousand customers. Some discussion happened about caveats and improvements and how nftables could be a better fit if it gains TPROXY-like features. In the field of networking at scale, some vmware engineers also joined the conversation for nft connlimit and nf_conncount, a new approach in nftables for rate-limiting/policing based on conntrack data. This was followed up by a presentation by Pablo Neira about the new flow offload infrastructure for nftables, which can act as a complete kernel bypass in case of packet forwarding.

The venue

Jozsef Kadlecsik shared a deep and detailed investigation on ipset vs nftables and how we could match both frameworks. He gave an overview of what’s missing, what’s already there and what could be a benefit from users migrating from ipset to nftables.

We had some space for load-balancing as well. Laura García shared the last news regarding the nftlb project, the nftables-based load balancer. She shared some interesting numbers about how reptoline affects Netfilter performance. She mentioned that the impact of reptoline is about 17% in nftables and 40% for iptables for her use cases.

Florian Westphal gave a talk regarding br_netfilter and how we could improve the linux kernel networking stack from the Netfilter point of view for bridge use cases. Right now all sorts of nasty things are done to store required information and context for packets traveling bridges (which may need to be evaluated by Netfilter). We have a lot of marging for improvement and Florian’s plan is to invest time in these.

We had a very interesting legal talk by Dr. Till Jaeger regarding GPL enforcement in Germany, related to the Patrick McHardly situation. Some good work is being done in this field to defend the community against activities which hurts the interest of all the Linux users and developers.

Harsha Sharma, 18 years old from India, gave a talk explaining her work on nftables to the rest of Netfilter contributors. This is possible thanks to internship programs like Outreachy and Google Summer of Code. Varsha and Harsha, both are so brave for traveling so far from home to join a mostly european-white-men-only meeting. We where joined by 3 women this workshop and I would like to believe this is a symbol of our inclusiveness, of being a healthy community.

The group

The workshop was sponsorized by vmware, zevenet, redhat, intra2net, oisf, stamus networks, and suricata.

16 June, 2018 05:28PM

hackergotchi for Steve Kemp

Steve Kemp

Monkeying around with intepreters

Recently I've had an overwhelming desire to write a BASIC intepreter. I can't think why, but the idea popped into my mind, and wouldn't go away.

So I challenged myself to spend the weekend looking at it.

Writing an intepreter is pretty well-understood problem:

  • Parse the input into tokens, such as "LET", "GOTO", "INT:3"
    • This is called lexical analysis / lexing.
  • Taking those tokens and building an abstract syntax tree.
    • The AST
  • Walking the tree, evaluating as you go.
    • Hey ho.

Of course BASIC is annoying because a program is prefixed by line-numbers, for example:

 10 PRINT "HELLO, WORLD"
 20 GOTO 10

The naive way of approaching this is to repeat the whole process for each line. So a program would consist of an array of input-strings each line being treated independently.

Anyway reminding myself of all this fun took a few hours, and during the course of that time I came across Writing an intepreter in Go which seems to be well-regarded. The book walks you through creating an interpreter for a language called "Monkey".

I found a bunch of implementations, which were nice and clean. So to give myself something to do I started by adding a new built-in function rnd(). Then I tested this:

let r = 0;
let c = 0;

for( r != 50 ) {
   let r = rnd();
   let c = c + 1;
}

puts "It took ";
puts c;
puts " attempts to find a random-number equalling 50!";

Unfortunately this crashed. It crashed inside the body of the loop, and it seemed that the projects I looked at each handled the let statement in a slightly-odd way - the statement wouldn't return a value, and would instead fall-through a case statement, hitting the next implementation.

For example in monkey-intepreter we see that happen in this section. (Notice how there's no return after the env.Set call?)

So I reported this as a meta-bug to the book author. It might be the master source is wrong, or might be that the unrelated individuals all made the same error - meaning the text is unclear.

Anyway the end result is I have a language, in go, that I think I understand and have been able to modify. Now I'll have to find some time to go back to BASIC-work.

I found a bunch of basic-intepreters, including ubasic, but unfortunately almost all of them were missing many many features - such as implementing operations like RND(), ABS(), COS().

Perhaps room for another interpreter after all!

16 June, 2018 11:01AM

June 15, 2018

Sven Hoexter

imagine you no longer own your infrastructure

Sounds crazy and nobody would ever do that, but just for a moment imagine you no longer own your infrastructure.

Imagine you just run your container on something like GKE with Kubernetes.

Imagine you build your software with something like Jenkins running in a container, using the GKE provided docker interface to build stuff in another container.

And for a $reason imagine you're not using the Google provided container registry, but your own one hosted somewhere else on the internet.

Of course you access your registry via HTTPS, so your connection is secured at the transport level.

Now imagine your certificate is at the end of its validity period. Like ending the next day.

Imagine you just do what you do every time that happens, and you just order a new certificate from one of the left over CAs like DigiCert.

You receive your certificate within 15 minutes.

You deploy it to your registry.

You validate that your certificate chain validates against different certificate stores.

The one shipped in ca-certificates on various Debian releases you run.

The one in your browser.

Maybe you even test it with Google Chrome.

Everything is cool and validates. I mean, of course it does. DigiCert is a known CA player and the root CA certificate was created five years ago. A lot of time for a CA to be included and shipped in many places.

But still there is one issue. The docker commands you run in your build jobs fail to pull images from your registry because the certificate can not be validated.

You take a look at the underlying OS and indeed it's not shipping the 5 year old root CA certificate that issued your intermediate CA that just issued your new server certificate.

If it were your own infrastructure you would now just ship the missing certificate.

Maybe by including it in your internal ca-certificates build.

Or by just deploying it with ansible to /usr/share/ca-certificates/myfoo/ and adding that to the configuration in /etc/ca-certificates.conf so update-ca-certificates can create the relevant hash links for you in /etc/ssl/certs/.

But this time it's not your infrastructure and you can not modify the operating system context your docker container are running in.

Sounds insane, right? Luckily we're just making up a crazy story and something like that would never happen in the real world, because we all insist on owning our infrastructure.

15 June, 2018 07:04PM

hackergotchi for Sune Vuorela

Sune Vuorela

Partially initialized objects

I found this construct some time ago. It took some reading to understand why it worked. I’m still not sure if it is actually legal, or just works only because m_derivedData is not accessed in Base::Base.

struct Base {
    std::string& m_derivedData;
    Base(std::string& data) : m_derivedData(data) {
    }
};

struct Derived : public Base {
    std::string m_data;
    struct Derived() : Base(m_data), m_data("foo") {
    }
};

15 June, 2018 06:15PM by Sune Vuorela

Andrej Shadura

Working in open source: part 1

Three years ago on this day I joined Collabora to work on free software full-time. It still feels a bit like yesterday, despite so much time passing since then. In this post, I’m going to reconstruct the events of that year.

Back in 2015, I worked for Alcatel-Lucent, who had a branch in Bratislava. I can’t say I didn’t like my job — quite contrary, I found it quite exciting: I worked with mobile technologies such as 3G and LTE, I had really knowledgeable and smart colleagues, and it was the first ‘real’ job (not counting the small business my father and I ran) where using Linux for development was not only not frowned upon, but was a mandatory part of the standard workflow, and running it on your workstation was common too, even though not official.

However, after working for Alcatel-Lucent for a year, I found I don’t like some of the things about this job. We developed proprietary software for the routers and gateways the company produced, and despite the fact we used quite a lot of open source libraries and free software tools, we very rarely contributed anything back, and if this happened at all, it usually happened unofficially and not on the company’s time. Each time I tried to suggest we need to upstream our local changes so that we don’t have to maintain three different patchsets for different upstream versions ourselves, I was told I know nothing about how the business works, and that doing that would give up the control on the code, and we can’t do that. At the same time, we had no issue incorporating permissively-licensed free software code. The more I worked at Alcatel-Lucent, the more I felt I am just getting useless knowledge of a proprietary product I will never be able to reuse once and if I leave the company. At some point, in a discussion at work someone said that doing software development (including my free software work) even on my free time may constitute a conflict of interests, and the company may be unhappy about it. Add to that that despite relatively flexible hours, working from home was almost never allowed, as was working from other offices of the company.

These were the major reasons I quit my job at Alcatel-Lucent, and my last day was 10 April 2018. Luckily, we reached an agreement that I will still get my normal pay while on the notice period despite not actually going to the office or doing any work, which allowed me to enjoy two months of working on my hobby projects while not having to worry about money.

To be honest, I don’t want to seem like I quit my job just because it was all proprietary software, and I did plan to live from donations or something, it wasn’t quite like that. While still working for Alcatel-Lucent, I was offered a job which was developing real-time software running inside the Linux kernel. While I have declined this job offer, mostly because it was a small company with less than a dozen employees, and I would need to take over the responsibility for a huge piece of code — which was, in fact, also proprietary, this job offer taught me this thing: there were jobs out there where my knowledge of Linux was of an actual use, even in the city I lived in. The other thing I learnt was this: there were remote Linux jobs too, but I needed to become self-employed to be able to take them, since my immigration status at the moment didn’t allow me to be employed abroad.

Picture of the business license. Text in Slovak: ‘Osvedčenie o živnostenskom opravnení. Andrei Shadura’.

The business license I received within a few days of quitting my job

Feeling free as a bird, having the business registered, I’ve spent two months hacking, relaxing, travelling to places in Slovakia and Ukraine, and thinking about how am I going to earn money when my two months vacation ends.

A street in Trenčín; the castle can be seen above the building’s roof.

In Trenčín

The obvious idea was to consult, but that wouldn’t guarantee me constant income. I could consult on Debian or Linux in general, or on version control systems — in 2015 I was an active member of the Kallithea project and I believed I could help companies migrate from CVS and Subversion to Mercurial and Git hosted internally on Kallithea. (I’ve actually also got a job offer from Unity Technologies to hack on Kallithea and related tools, but I had to decline it since it would require moving to Copenhagen, which I wasn’t ready for, despite liking the place when I visited them in May 2015.)

Another obvious idea was working for Red Hat, but knowing how slow their HR department was, I didn’t put too much hope into it. Besides, when I contacted them, they said they need to get an approval for me to work for them remotely and as a self-employed, lowering my chances on getting a job there without having to relocate to Brno or elsewhere.

At some point, reading Debian Planet, I found a blog post by Simon McVittie on polkit, in which he mentioned Collabora. Soon, I applied, had my interviews and a job offer.

To be continued later today…

15 June, 2018 02:08PM by Andrej Shadura

hackergotchi for Steinar H. Gunderson

Steinar H. Gunderson

Qt flag types

typeid(Qt::AlignRight) = Qt::AlignmentFlag (implicitly convertible to QVariant
typeid(Qt::AlignRight | Qt::AlignVCenter) = QFlags<Qt::AlignmentFlag> (not implicitly convertible to QVariant)
typeid(Qt::AlignRight + Qt::AlignVCenter) = int (implicitly convertible to QVariant)

Qt, what is wrong with you?

15 June, 2018 08:00AM

hackergotchi for Daniel Pocock

Daniel Pocock

The questions you really want FSFE to answer

As the last man standing as a fellowship representative in FSFE, I propose to give a report at the community meeting at RMLL.

I'm keen to get feedback from the wider community as well, including former fellows, volunteers and anybody else who has come into contact with FSFE.

It is important for me to understand the topics you want me to cover as so many things have happened in free software and in FSFE in recent times.

last man standing

Some of the things people already asked me about:

  • the status of the fellowship and the membership status of fellows
  • use of non-free software and cloud services in FSFE, deviating from the philosophy that people associate with the FSF / FSFE family
  • measuring both the impact and cost of campaigns, to see if we get value for money (a high level view of expenditure is here)

What are the issues you would like me to address? Please feel free to email me privately or publicly. If I don't have answers immediately I would seek to get them for you as I prepare my report. Without your support and feedback, I don't have a mandate to pursue these issues on your behalf so if you have any concerns, please reply.

Your fellowship representative

15 June, 2018 07:28AM by Daniel.Pocock

bisco

Third GSoC Report

The last two weeks went by pretty fast, probably also because the last courses this semester started and i have a lot of additional work to do.

I closed the last report with writing about the implementation of the test suite. I’ve added a lot more tests since then and there are now around 80 tests that are run with every commit. Using unit tests that do some basic testing really makes life a lot easier- next time i start a software project i’ll definitly start early on with writing tests. I’ve also read a bit about the difference of integration and unit tests. A unit test should only test one specific functionality, so i refactored some of the old tests and made them more granular.

I then also looked into coding style checkers and decided to go with flake8. There were a huge pile of coding style violations in my code, most of them lines that were more than 79 characters. I’ve integrated flake8 in the test suite and removed all the violations. One more thing about python: i’ve read python3 with pleasure which gives a great overview about some of the new features of python3 and i’ve made some notes about stuff i want to integrate (i.e. pathlib)

Regarding the functionality of nacho i’ve added the possibility to delete an account. SSH keys are now validated on upload and it is possilbe to configure the key types that are allowed. I initially just checked if the key string consists of valid base64 encoded data, but that was not really a good solution so i decided to use sshpubkeys to check the validity of the keys. Nacho now also checks the profile image before storing it in the LDAP database- it is possible to configure the image size and list allowed image types, which is verified using python-magic. I also made a big change concerning the configuration: all the relevant configuration options are now moved to a seperate configuration file in json format, which is parsed when nacho is started. This makes it also a lot easier to have default values and to let users override them in their local config. I also updated the documentation and the debian package.

Now that the issues with nacho are slowly becoming smaller, i’ll start to look into existing SSO solutions that then can be used with the LDAP backend. There are four solutions i’ve on my list at the moment, that are keycloak, ipsilon, lemonldap-ng and glewlwyd.

15 June, 2018 05:28AM

hackergotchi for Gunnar Wolf

Gunnar Wolf

«Understanding the Digital World» — By Brian Kernighan

I came across Kernighan's 2017 book, Understanding the Digital World — What You Need to Know about Computers, the Internet, Privacy, and Security. I picked it up thanks to a random recommendation I read somewhere I don't recall. And it's really a great read.
Of course, basically every reader that usually comes across this blog will be familiar with Kernighan. Be it because his most classic books from the 1970s, The Unix Programming Environment or The C Programming Language, or from the much more recent The Practice of Programming or The Go Programming Language, Kernighan is a world-renowned authority for technical content, for highly technical professionals at the time of their writing — And they tend to define the playing field later on.
But this book I read is... For the general public. And it is superb at that.
Kernighan states in his Preface that he teaches a very introductory course at Princeton (a title he admits to be too vague, Computers in our World) to people in the social sciences and humanities field. And this book shows how he explains all sorts of scary stuff to newcomers.
As it's easier than doing a full commentary on it, I'll just copy the table of contents (only to the section level, it gets just too long if I also list subsections). The list of contents is very thorough (and the book is only 238 pages long!), but take a look at basically every chapter... And picture explaining those topics to computing laymen. An admirable feat!

  • Part I: Hardware
    • 1. What's in a computer?
      • Logical construction
      • Physical construction
      • Moore's Law
      • Summary
    • 2. Bits, Bytes, and Representation of Information
      • Analog versus Digital
      • Analog-Digital Conversion
      • Bits, Bytes and Binary
      • Summary
    • 3. Inside the CPU
      • The Toy Computer
      • Real CPUs
      • Caching
      • Other Kinds of Computers
      • Summary

    Wrapup on Hardware

  • Part II: Software
    • 4. Algorithms
      • Linear Algorithms
      • Binary Search
      • Sorting
      • Hard Problems and Complexity
      • Summary
    • 5. Programming and Programming Languages
      • Assembly Language
      • High Level Languages
      • Software Development
      • Intellectual Property
      • Standards
      • Open Source
      • Summary
    • 6. Software Systems
      • Operating Systems
      • How an Operating System works
      • Other Operating Systems
      • File Systems
      • Applications
      • Layers of Software
      • Summary
    • 7. Learning to Program
      • Programming Language Concepts
      • A First JavaScript Example
      • A Second JavaScript Example
      • Loops
      • Conditionals
      • Libraries and Interfaces
      • How JavaScript Works
      • Summary

    Wrapup on Software

  • Part III: Communications
    • 8. Networks
      • Telephones and Modems
      • Cable and DSL
      • Local Area Networks and Ethernet
      • Wireless
      • Cell Phones
      • Bandwidth
      • Compression
      • Error Detection and Correction
      • Summary
    • The Internet
      • An Internet Overview
      • Domain Names and Addresses
      • Routing
      • TCP/IP protocols
      • Higher-Level Protocols
      • Copyright on the Internet
      • The Internet of Things
      • Summary
    • 10. The World Wide Web
      • How the Web works
      • HTML
      • Cookies
      • Active Content in Web Pages
      • Active Content Elsewhere
      • Viruses, Worms and Trojan Horses
      • Web Security
      • Defending Yourself
      • Summary
    • 11. Data and Information
      • Search
      • Tracking
      • Social Networks
      • Data Mining and Aggregation
      • Cloud Computing
      • Summary
    • 12. Privacy and Security
      • Cryptography
      • Anonymity
      • Summary
    • 13. Wrapping up

I must say, I also very much enjoyed learning of my overall ideological alignment with Brian Kernighan. I am very opinionated, but I believe he didn't make me do a even mild scoffing — and he goes to many issues I have strong feelings about (free software, anonymity, the way the world works...)
So, maybe I enjoyed this book so much because I enjoy teaching, and it conveys great ways to teach the topics I'm most passionate about. But, anyway, I have felt for several days the urge to share this book with the group of people that come across my blog ☺

15 June, 2018 12:07AM by gwolf

June 14, 2018

hackergotchi for Kees Cook

Kees Cook

security things in Linux v4.17

Previously: v4.16.

Linux kernel v4.17 was released last week, and here are some of the security things I think are interesting:

Jailhouse hypervisor

Jan Kiszka landed Jailhouse hypervisor support, which uses static partitioning (i.e. no resource over-committing), where the root “cell” spawns new jails by shrinking its own CPU/memory/etc resources and hands them over to the new jail. There’s a nice write-up of the hypervisor on LWN from 2014.

Sparc ADI

Khalid Aziz landed the userspace support for Sparc Application Data Integrity (ADI or SSM: Silicon Secured Memory), which is the hardware memory coloring (tagging) feature in Sparc M7. I’d love to see this extended into the kernel itself, as it would kill linear overflows between allocations, since the base pointer being used is tagged to belong to only a certain allocation (sized to a multiple of cache lines). Any attempt to increment beyond, into memory with a different tag, raises an exception. Enrico Perla has some great write-ups on using ADI in allocators and a comparison of ADI to Intel’s MPX.

new kernel stacks cleared on fork

It was possible that old memory contents would live in a new process’s kernel stack. While normally not visible, “uninitialized” memory read flaws or read overflows could expose these contents (especially stuff “deeper” in the stack that may never get overwritten for the life of the process). To avoid this, I made sure that new stacks were always zeroed. Oddly, this “priming” of the cache appeared to actually improve performance, though it was mostly in the noise.

MAP_FIXED_NOREPLACE

As part of further defense in depth against attacks like Stack Clash, Michal Hocko created MAP_FIXED_NOREPLACE. The regular MAP_FIXED has a subtle behavior not normally noticed (but used by some, so it couldn’t just be fixed): it will replace any overlapping portion of a pre-existing mapping. This means the kernel would silently overlap the stack into mmap or text regions, since MAP_FIXED was being used to build a new process’s memory layout. Instead, MAP_FIXED_NOREPLACE has all the features of MAP_FIXED without the replacement behavior: it will fail if a pre-existing mapping overlaps with the newly requested one. The ELF loader has been switched to use MAP_FIXED_NOREPLACE, and it’s available to userspace too, for similar use-cases.

pin stack limit during exec

I used a big hammer and pinned the RLIMIT_STACK values during exec. There were multiple methods to change the limit (through at least setrlimit() and prlimit()), and there were multiple places the limit got used to make decisions, so it seemed best to just pin the values for the life of the exec so no games could get played with them. Too much assumed the value wasn’t changing, so better to make that assumption actually true. Hopefully this is the last of the fixes for these bad interactions between stack limits and memory layouts during exec (which have all been defensive measures against flaws like Stack Clash).

Variable Length Array removals start

Following some discussion over Alexander Popov’s ongoing port of the stackleak GCC plugin, Linus declared that Variable Length Arrays (VLAs) should be eliminated from the kernel entirely. This is great because it kills several stack exhaustion attacks, including weird stuff like stepping over guard pages with giant stack allocations. However, with several hundred uses in the kernel, this wasn’t going to be an easy job. Thankfully, a whole bunch of people stepped up to help out: Gustavo A. R. Silva, Himanshu Jha, Joern Engel, Kyle Spiers, Laura Abbott, Lorenzo Bianconi, Nikolay Borisov, Salvatore Mesoraca, Stephen Kitt, Takashi Iwai, Tobin C. Harding, and Tycho Andersen. With Linus Torvalds and Martin Uecker, I also helped rewrite the max() macro to eliminate false positives seen by the -Wvla compiler option. Overall, about 1/3rd of the VLA instances were solved for v4.17, with many more coming for v4.18. I’m hoping we’ll have entirely eliminated VLAs by the time v4.19 ships.

That’s in for now! Please let me know if you think I missed anything. Stay tuned for v4.18; the merge window is open. :)

© 2018, Kees Cook. This work is licensed under a Creative Commons Attribution-ShareAlike 3.0 License.
Creative Commons License

14 June, 2018 11:23PM by kees

Athos Ribeiro

Some notes on the OBS Documentation

This is my fourth post of my Google Summer of Code 2018 series. Links for the previous posts can be found below:

Open Build Service Manuals

OBS provides several manuals on their web site, including an admin and a user guide. Since I needed to travel to an academic conference last week (too many hours in airplanes), I took some time to read the full OBS documentation to have a better understanding of the tool we have been deploying. While reading the documentation, I took some notes on relevant points for our GSoC project (and sent a patch to fix a few typos in OBS documentaion, which I discuss below.

Hardware requirements

There is no need to distribute all different services in OBS server since our instance will not process heavy build loads. We do want to separate the server services from the OBS workers (package builders) so expensive builds will not compromise our server performance.

According to OBS documentation, we need

  • 1 core for each scheduler architecture
  • 4GB ram for each scheduler architecture
  • 50GB disk per architecture for each build distribution supported

We are working with a single build distribution (Debian unstable). Therefore, we need 50GB disk for our OBS instance for each supported architecture (unless we want to mirror the whole distribution instead of using the Download on Demand OBS feature).

We would like to work with 3 different architectures: i686, x86_64 and arm. Hence, we need 150GB, 12GB ram and 3 cores according to the OBS admin guide.

Summary:

  • 12GB RAM
  • 150GB disk
  • 3 cores

OBS Instance Configuration

We want to change some instance configurations like

  • Change OBS instance description
  • Set administrator email
  • Disable new users sign up: since all builds in this OBS instance will be fired automatically and no new projects will be configured for now, we will not allow people to create accounts in our OBS instance.

It is important to note that the proper way of changing a project’s configuration is through the API calls. Therefore, we will need to make such calls in our salt scripts.

To list OBS configurations:

osc -A https://irill8.siege.inria.fr api /configuration

To redefine OBS configurations:

osc -A https://irill8.siege.inria.fr api /configuration -T new_config_file.xml

Workers configuration

OBS workers need to be allowed to connect to the server in /etc/obs/BSConfig.pm. The server accepts connections from any node in the network by default, but we can (and should) force OBS to accept connections only from our own nodes.

Source Services

OBS provide a way to run scripts to change sources before builds. This may be useful for building against Clang.

To create a source service, we must create a script in the /usr/lib/obs/service/ directory and create a new _service file either in the package or in the project repository level.

_service is a XML file pointing to our script under /usr/lib/obs/service/ and providing possible parameters to the script:

<services>
 <service name="foobar.sh" mode="MODE">
 <param name="PARAMETER1">PARAMETER1_VALUE</param>
 </service>
</services>

Self signed certificates

For testing purposes, there is no need to generate proper SSL certificates, we can generate and self sign our own:

mkdir /srv/obs/certs
openssl genrsa -out /srv/obs/certs/server.key 1024
openssl req -new -key /srv/obs/certs/server.key -out /srv/obs/certs/server.csr
openssl x509 -req -days 365 -in /srv/obs/certs/server.csr -signkey /srv/obs/certs/server.key -out /srv/obs/certs/server.crt
cat /srv/obs/certs/server.key /srv/obs/certs/server.crt > /srv/obs/certs/server.pem

Finally, we must trust our certificate:

cp /srv/obs/certs/server.pem /etc/ssl/certs/
c_rehash /etc/ssl/certs/

Message bus

OBS supports rabbitMQ usage to publish events such as build results, package updates, etc. In the future, we could also set a rabbitMQ instance so other services can listen to a queue with our Clang build results.

Next steps (A TODO list to keep on the radar)

  • Write patches for the OBS worker issue described in post 3
  • Configure Debian projects on OBS with salt, not manually
  • Change the default builder to perform builds with clang
  • Trigger new builds by using the dak/mailing lists messages
  • Verify the rake-tasks.sh script idempotency and propose patch to opencollab repository
  • Separate salt recipes for workers and server (locally)
  • Properly set hostnames (locally)

14 June, 2018 04:06AM

hackergotchi for Louis-Philippe Véronneau

Louis-Philippe Véronneau

IMAP Spam Begone (ISBG) version 2.1.0 is out!

When I first started at the non-profit where I work, one of the problems people had was rampant spam on their email boxes. The email addresses we use are pretty old (+10 years) and over time they have been added to all the possible spam lists there are.

That would not be a real problem if our email hosting company did not have very bad spam filters. They are a worker's coop and charge us next to nothing for hosting our emails, but sadly they lack the resources to run a real bayesian-based spam filtering solution like SpamAssassin. "Luckily" for us, it seems that a lot of ISPs and email hosting enterprises also tend to have pretty bad spam filtering on the email boxes they provide and there were a few programs out there to fix this.

One of the solutions I found to alleviate this problem was to use IMAP Spam Begone (ISBG), a script that makes it easy to scan an IMAP inbox for spam using your own SpamAssassin server and get your spam moved around via IMAP. Since then, I've been maintaining the upstream project.

At the time, ISBG was somewhat abandoned and was mostly a script made of old python2 code. No classes, no functions, just a long script that ran from top to bottom.

Well, I'm happy to say that ISBG now has a new major release! Version 2.1.0 is out and replaces the last main release, 1.0.0. From a script, ISBG has now evolved into a full-fledged python module using classes and functions. Although the code still works with python2, everything is now python3 compliant as well. We even started using CI tests recently!

That, and you know, tons of bugs were fixed. I'd like to thank all the folks who submitted patches, as very few of the actual code was written by me.

If you want to give ISBG a try, you can find the documentation here. Here's also a nice terminal capture I made of ISBG working in verbose mode:

14 June, 2018 04:00AM by Louis-Philippe Véronneau

June 13, 2018

Elana Hashman

Looking back on "Teaching Python: The Hard Parts"

One of my goals when writing talks is to produce content with a long shelf life. Because I'm one of those weird people that prefers to write new talks for new events, I feel like it'd be a waste of effort if my talks didn't at least age well. So how do things measure up if I look back on one of my oldest?

"Teaching Python: The Hard Parts" remains one of my most popular talks, despite presenting it just one time at PyCon US 2016. For most of the past two years, it held steady as a top 10 talk from PyCon 2016 by popularity on YouTube (although it was recently overtaken by a few hundred views 😳), even when counting it against the keynotes (!), and most of the YouTube comments are shockingly nice (!!).

Well, actually

Not everyone was a fan. Obviously I should have known better than to tell instructors they didn't have to use Python 3:

Matt Williams: Obviously Python 3 should be taught over Python 2. In a few years time 2 will be completely unsupported http://pythonclock.org/

Did I give bad advice? Was mentiontioning the usability advantage of better library support and documentation SEO with Python 2 worth the irreparable damage I might have done to the community?

Matt's not the only one with a chip on his shoulder: the Python 2 → 3 transition has been contentious, and much ink has been spilled on the topic. A popular author infamously wrote a long screed claiming "PYTHON 3 IS SUCH A FAILURE IT WILL KILL PYTHON". Spoiler alert: Python is still alive, and the author updated his book for Python 3.

I've now spent a few years writing 2/3 compatible code, and am on the cusp of dropping Python 2 entirely. I've felt bad for not weighing in on the topic publicly, because people might have looked to this talk for guidance and wouldn't know my advice has changed over the past two years.

A little history

I wrote this talk based on my experiences teaching Python in the winter and fall of 2014, and presented it in early 2016. Back then, it wasn't clear if Python 3 adoption was going to pick up: Hynek wrote an article about Python 3 adoption a few months before PyCon that contained the ominous subheading "GLOOM". Python 3 only reached a majority mindshare of Python developers in May 2017!

Why? That's a topic long enough to fill a series of blog posts, but briefly: the number of breaking changes introduced in the first few releases in Python 3, coupled with the lack of compelling features to incentivize migration led to slow adoption. Personally, until the Python 3.3 release, I don't think Python 3 had that balance right to really take off. Version 3.3 was released in fall of 2012. Python 3.4 was only released in early 2014, just before I mentored at my first set of workshops!

This is a long-winded way to say, "when I gave this talk, it wasn't clear that telling workshop organizers to teach Python 3 would be good advice, because the ecosystem wasn't there yet."

The brave new world

But this is no longer the case! Python 3 adoption is overtaking Python 2 use, even in the enterprise space. The Python 2 clock keeps on ticking. Latest releases of Python 3 have compelling features to lure users, including strong, native concurrency support, formatted strings, better cross-system path support, and type hints.

This is to say, if I had to pick just one change to make to this talk if I gave it today, I would tell folks

USE PYTHON 3! ✨

Other updates

  • The documentation for packaging Python is a lot better now. There have been many good talks presented on the subject.
  • Distributing Python is still hard. There isn't a widely adopted practice for cross-platform management of compiled dependencies yet, although wheels are picking up steam. I'm currently working on the manylinux2010 update to address this problem on Linux systems.

Endorsements

Not to let one YouTube commenter rain on my parade, I am thrilled to say that some people in the community have written some awfully nice things about my talk. Thanks to all for doing so—pulling this together really brightened my day!

Blog Posts

Roxanne Johnson writes, "Elana Hashman’s PyCon talk on Teaching Python: The Hard Parts had me nodding so hard I thought I might actually be headbanging." 😄

Georgia Reh writes, "I am just in love with this talk. Any one who has seen me speak about teaching git knows I try really hard to not overload students with information, and Elana has a very clear idea of what a beginner needs to know when learning python versus what they can learn later." 💖

Tweets

When I presented this talk, I was too shy to attach my twitter handle to my slides, so all these folks tweeted at me by name. Wow!

Other

My talk was included in the "Awesome Python in Education" list. How cool 😎

Declaring a small victory

Writing this post has convinced me that "Teaching Python: The Hard Parts" meets some arbitrary criteria for "sufficiently forward-thinking." Much of the content still strikes me as fresh: as an occasional mentor for various technical workshops, I still keep running into trouble with platform diversity, the command line, and packaging; the "general advice" section is evergreen for Python and non-Python workshops alike. So with all that said, here's hoping that looking back on this talk will keep it alive. Give it a watch if you haven't seen it before!

If you like what you see and you're interested in checking out my speaking portfolio or would like to invite me to speak at your conference or event, do check out my talks page.

13 June, 2018 11:00PM by Elana Hashman

hackergotchi for Shirish Agarwal

Shirish Agarwal

students, suicides, pressures and solutions.

Couple of days back, I heard of a student whose body was found hung from the ceiling in a college nearby. It felt a bit shocked as I had visited that college just sometime back. It is also possible that I may have run into him and even had a conversation with him. No names were shared and even if there were shared it’s doubtful I would remember him as during events you meet so many people, it’s difficult to parse and remember names 😦 . I do feel sort of stretched at events but that’s life I guess.

As no suicide note was found, the police are investigating from all angles as to the nature of the death. While it’s too early to come to conclusions whether the student decided to take his own life or someone else decided to end his life for some reason or the other, I saw that nobody whom I talked to felt perturbed even a tiny bit probably because it has become a new normal. The major reasons apart from those shared in a blog post are that the costs of the education is too high for today’s students.

There are also perceived career biases that people have, believing that Computer Science is better than being a lawyer, even though IT layoffs have become a new normal. In the above specific case, it was reported that apparently the student who killed himself wanted to be a lawyer while the family wanted him to do CS (Computer Science) .

Also the whole reskilling and STEM culture may be harder as at least Government syllabuses are 10-15 years too late. The same goes for the teachers who would have to change a lot and sadly, it is too common for teachers to be paid a pittance, even college professors.

I know of quite a few colleges in the city in different domains where suicides have taken place, the authorities have tried putting wellness rooms where students who feel depressed could share their feelings but probably due to feelings of shame or weaknesses, the ones who are most at risk do not allow the true feelings to surface. The eastern philosophy of ‘saving face’ is killing our young ones. There is one non-profit I know, Connecting NGO 18002094353 (toll-free) and 9922001122 (mobile) that students or whoever is in depression can call. The listeners don’t give any advice as they are not mental health experts but just give a patient hearing. Sometimes sharing or describing whatever you are facing may give enough either hope or a mini-solution that you can walk towards.

I hope people would use the resources listed above.

Update – 15/06/2018 – A friend/acquaintance recently passed a link which helped her and her near and dear ones to better support her throughout her facing depression. It pretty much seems like a yo-yo but that’s how people might feel in a given situation.

I was shared an email where I had asked the concerned non-profit to see if it needed any more addition to the blog post and this is what I heard from them –

Hello Shirish,

Warm Greetings from Connecting NGO

I read the blog link sent by you and the article looks good. I dont think anything needs to be added to that. Someday if you can come to the office, we can sit and talk about articles regarding emotional distress and suicides and how they need to be written. You have done a good job and thanks for sharing the link.
We will surely try to get in touch with X college sometime this month and talk with the teaching staff and authorities there along with the students. Thanks for the lead. Hoping to see you soon.

Regards,
Vikramsinh Pawar
Senior Programme Coordinator,

I was simply being cautious and short of words as words carelessly used could be a trigger as well.

On one of the groups I am a member of, I came to know of another institute where there have been quite a few suicides. A few of us have decided to visit the institute with a trained mental health professional and see if we can be of any assistance in anyway, in some ways sharing our tales of loss in the hopes that others are able to grieve their loss or at least come to terms with.

We have also asked the non-profit so maybe they would also do an intervention on their own.

13 June, 2018 07:07PM by shirishag75

hackergotchi for Sean Whitton

Sean Whitton

Debian Policy call for participation -- June 2018

I’d like to push a substantive release of Policy but I’m waiting for DDs to review and second patches in the following bugs. I’d be grateful for your involvement!

If a bug already has two seconds, or three seconds if the proposer of the patch is not a DD, please consider reviewing one of the others, instead, unless you have a particular interest in the topic of the bug.

If you’re not a DD, you are welcome to review, but it might be a more meaningful contribution to spend your time writing patches bugs that lack them, instead.

#786470 [copyright-format] Add an optional “License-Grant” field

#846970 Proposal for a Build-Indep-Architecture: control file field

#864615 please update version of posix standard for scripts (section 10.4)

#880920 Document Rules-Requires-Root field

#891216 Requre d-devel consultation for epoch bump

#897217 Vcs-Hg should support -b too

13 June, 2018 02:15PM

Enrico Zini

Progress bar for file descriptors

I ran gzip on an 80Gb file, it's processing, but who knows how much it has done yet, and when it will end? I wish gzip had a progressbar. Or MySQL. Or…

Ok. Now every program that reads a file sequentially can have a progressbar:

https://gitlab.com/spanezz/fdprogress

fdprogress

Print progress indicators for programs that read files sequentially.

fdprogress monitors file descriptor offsets and prints progressbars comparing them to file sizes.

Pattern can be any glob expression.

usage: fdprogress [-h] [--verbose] [--debug] [--pid PID] [pattern]

show progress from file descriptor offsets

positional arguments:
  pattern            file name to monitor

optional arguments:
  -h, --help         show this help message and exit
  --verbose, -v      verbose output
  --debug            debug output
  --pid PID, -p PID  PID of process to monitor

pv

pv has a --watchfd option that does most of what fdprogress is trying to do: use that instead.

fivi

fivi also exists, with specific features to show progressbars for filter commands.

13 June, 2018 11:43AM

hackergotchi for Norbert Preining

Norbert Preining

Microsoft fixed the Open R Debian package

I just got notice that Microsoft has updated the Debian packaging of Open R to properly use dpkg-divert. I checked the Debian packaging scripts and they now properly divert R and Rscript, and revert back to the Debian provided (r-base) version after removal of the packages.

The version 3.5.0 has been rereleased, if you have downloaded it from MRAN you will need to redownload the file and be careful to use the new one, the file name of the downloaded file is the same.

Thanks Microsoft for the quick fix, it is good news that those playing with Open R will not be left with a hosed system.

PS: I guess this post will by far not get the incredible attention the first one got 😉

13 June, 2018 10:09AM by Norbert Preining

June 12, 2018

hackergotchi for Jonathan McDowell

Jonathan McDowell

Hooking up Home Assistant to Alexa + Google Assistant

I have an Echo Dot. Actually I have two; one in my study and one in the dining room. Mostly we yell at Alexa to play us music; occasionally I ask her to set a timer, tell me what time it is or tell me the news. Having setup Home Assistant it seemed reasonable to try and enable control of the light in the dining room via Alexa.

Perversely I started with Google Assistant, even though I only have access to it via my phone. Why? Because the setup process was a lot easier. There are a bunch of hoops to jump through that are documented on the Google Assistant component page, but essentially you create a new home automation component in the Actions on Google interface, connect it with the Google OAuth stuff for account linking, and open up your Home Assistant instance to the big bad internet so Google can connect.

This final step is where I differed from the provided setup. My instance is accessible internally at home, but I haven’t wanted to expose it externally yet (and I suspect I never well, but instead have the ability to VPN back in to access or similar). The default instructions need you to open up API access publicly, and configure up Google with your API password, which allows access to everything. I’d rather not.

So, firstly I configured up my external host with an Apache instance and a Let’s Encrypt cert (luckily I have a static IP, so this was actually the base host that the Home Assistant container runs on). Rather than using this to proxy the entire Home Assistant setup I created a unique /external/google/randomstring proxy just for the Google Assistant API endpoint. It looks a bit like this:

<VirtualHost *:443>
  ServerName my.external.host

  ProxyPreserveHost On
  ProxyRequests off

  RewriteEngine on

  # External access for Google Assistant
  ProxyPassReverse /external/google/randomstring http://hass-host:8123/api/google_assistant
  RewriteRule ^/external/google/randomstring$ http://hass-host:8123/api/google_assistant?api_password=myapipassword [P]
  RewriteRule ^/external/google/randomstring/auth$ http://hass-host:8123/api/google_assistant/auth?%{QUERY_STRING}&&api_password=myapipassword [P]

  SSLEngine on
  SSLCertificateFile /etc/ssl/my.external.host.crt
  SSLCertificateKeyFile /etc/ssl/private/my.external.host.key
  SSLCertificateChainFile /etc/ssl/lets-encrypt-x3-cross-signed.crt
</VirtualHost>

This locks down the external access to just being the Google Assistant end point, and means that Google have a specific shared secret rather than the full API password. I needed to configure up Home Assistant as well, so configuration.yaml gained:

google_assistant:
  project_id: homeautomation-8fdab
  client_id: oFqHKdawWAOkeiy13rtr5BBstIzN1B7DLhCPok1a6Jtp7rOI2KQwRLZUxSg00rIEib2NG8rWZpH1cW6N
  access_token: l2FrtQyyiJGo8uxPio0hE5KE9ZElAw7JGcWRiWUZYwBhLUpH3VH8cJBk4Ct3OzLwN1Fnw39SR9YArfKq
  agent_user_id: noodles@earth.li
  api_key: nyAxuFoLcqNIFNXexwe7nfjTu2jmeBbAP8mWvNea
  exposed_domains:
    - light

Setting up Alexa access is more complicated. Amazon Smart Home skills must call an AWS Lambda - the code that services the request is essential a small service run within Lambda. Home Assistant supports all the appropriate requests, so the Lambda code is a very simple proxy these days. I used Haaska which has a complete setup guide. You must do all 3 steps - the OAuth provider, the AWS Lambda and the Alexa Skill. Again, I wanted to avoid exposing the full API or the API password, so I forked Haaska to remove the use of a password and instead use a custom URL. I then added the following additional lines to the Apache config above:

# External access for Amazon Alexa
ProxyPassReverse /external/amazon/stringrandom http://hass-host:8123/api/alexa/smart_home
RewriteRule /external/amazon/stringrandom http://hass-host:8123/api/alexa/smart_home?api_password=myapipassword [P]

In the config.json I left the password field blank and set url to https://my.external.host/external/amazon/stringrandom. configuration.yaml required less configuration than the Google equivalent:

alexa:
  smart_home:
    filter:
      include_entities:
        - light.dining_room_lights
        - light.living_room_lights
        - light.kitchen
        - light.snug

(I’ve added a few more lights, but more on the exact hardware details of those at another point.)

To enable in Alexa I went to the app on my phone, selected the “Smart Home” menu option, enabled my Home Assistant skill and was able to search for the available devices. I can then yell “Alexa, turn on the snug” and magically the light turns on.

Aside from being more useful (due to the use of the Dot rather than pulling out a phone) the Alexa interface is a bit smoother - the command detection is more reliable (possibly due to the more limited range of options it has to work out?) and adding new devices is a simple rescan. Adding new devices with Google Assistant seems to require unlinking and relinking the whole setup.

The only problem with this setup so far is that it’s only really useful for the room with the Alexa in it. Shouting from the living room in the hope the Dot will hear is a bit hit and miss, and I haven’t yet figured out a good alternative method for controlling the lights there that doesn’t mean using a phone or a tablet device.

12 June, 2018 08:21PM

John Goerzen

Syncing with a memory: a unique use of tar –listed-incremental

I have a Nextcloud instance that various things automatically upload photos to. These automatic folders sync to a directory on my desktop. I wanted to pull things out of that directory without deleting them, and only once. (My wife might move them out of the directory on her computer, and I might arrange them into targets on my end.)

In other words, I wanted to copy a file from a source to a destination, but remember what had been copied before so it only ever copies once.

rsync doesn’t quite do this. But it turns out that tar’s listed-incremental feature can do exactly that. Ordinarily, it would delete files that were deleted on the source. But if we make the tar file with the incremental option, but extract it without, it doesn’t try to delete anything at extract time.

Here’s my synconce script:

#!/bin/bash

set -e

if [ -z "$3" ]; then
    echo "Syntax: $0 snapshotfile sourcedir destdir"
    exit 5
fi

SNAPFILE="$(realpath "$1")"
SRCDIR="$2"
DESTDIR="$(realpath "$3")"

cd "$SRCDIR"
if [ -e "$SNAPFILE" ]; then
    cp "$SNAPFILE" "${SNAPFILE}.new"
fi
tar "--listed-incremental=${SNAPFILE}.new" -cpf - . | \
    tar -xf - -C "$DESTDIR"
mv "${SNAPFILE}.new" "${SNAPFILE}"

Just have the snapshotfile be outside both the sourcedir and destdir and you’re good to go!

12 June, 2018 11:27AM by John Goerzen

hackergotchi for Dirk Eddelbuettel

Dirk Eddelbuettel

R 3.5.0 on Debian and Ubuntu: An Update

Overview

R 3.5.0 was released a few weeks ago. As it changes some (important) internals, packages installed with a previous version of R have to be rebuilt. This was known and expected, and we took several measured steps to get R binaries to everybody without breakage.

The question of but how do I upgrade without breaking my system was asked a few times, e.g., on the r-sig-debian list as well as in this StackOverflow question.

Debian

Core Distribution As usual, we packaged R 3.5.0 as soon as it was released – but only for the experimental distribution, awaiting a green light from the release masters to start the transition. A one-off repository [drr35](https://github.com/eddelbuettel/drr35) was created to provide R 3.5.0 binaries more immediately; this was used, e.g., by the r-base Rocker Project container / the official R Docker container which we also update after each release.

The actual transition was started last Friday, June 1, and concluded this Friday, June 8. Well over 600 packages have been rebuilt under R 3.5.0, and are now ready in the unstable distribution from which they should migrate to testing soon. The Rocker container r-base was also updated.

So if you use Debian unstable or testing, these are ready now (or will be soon once migrated to testing). This should include most Rocker containers built from Debian images.

Contributed CRAN Binaries Johannes also provided backports with a -cran35 suffix in his CRAN-mirrored Debian backport repositories, see the README.

Ubuntu

Core (Upcoming) Distribution Ubuntu, for the upcoming 18.10, has undertaken a similar transition. Few users access this release yet, so the next section may be more important.

Contributed CRAN and PPA Binaries Two new Launchpad PPA repositories were created as well. Given the rather large scope of thousands of packages, multiplied by several Ubuntu releases, this too took a moment but is now fully usable and should get mirrored to CRAN ‘soon’. It covers the most recent and still supported LTS releases as well as the current release 18.04.

One PPA contains base R and the recommended packages, RRutter3.5. This is source of the packages that will soon be available on CRAN. The second PPA (c2d4u3.5) contains over 3,500 packages mainly derived from CRAN Task Views. Details on updates can be found at Michael’s R Ubuntu Blog.

This can used for, e.g., Travis if you managed your own sources as Dirk’s r-travis does. We expect to use this relatively soon, possibly as an opt-in via a variable upon which run.sh selects the appropriate repository set. It will also be used for Rocker releases built based off Ubuntu.

In both cases, you may need to adjust the sources list for apt accordingly.

Others

There may also be ongoing efforts within Arch and other Debian-derived distributions, but we are not really aware of what is happening there. If you use those, and coordination is needed, please feel free to reach out via the the r-sig-debian list.

Closing

In case of questions or concerns, please consider posting to the r-sig-debian list.

Dirk, Michael and Johannes, June 2018

12 June, 2018 01:27AM

June 11, 2018

Reproducible builds folks

Reproducible Builds: Weekly report #163

Here’s what happened in the Reproducible Builds effort between Sunday June 3 and Saturday June 9 2018:

Development work

Upcoming events

tests.reproducible-builds.org development

There were a number of changes to our Jenkins-based testing framework that powers tests.reproducible-builds.org, including:

In addition, Mattia Rizzolo has been working in a large refactor of the Python part of the setup.

Documentation updates

Misc.

This week’s edition was written by Bernhard M. Wiedemann, Chris Lamb, Holger Levsen, Mattia Rizzolo, Santiago Torres, Vagrant Cascadian & reviewed by a bunch of Reproducible Builds folks on IRC & the mailing lists.

11 June, 2018 09:45PM

hackergotchi for Sune Vuorela

Sune Vuorela

Kirigaming – Kolorfill

Last time, I was doing a recipe manager. This time I’ve been doing a game with javascript and QtQuick, and for the first time dipping my feet into the Kirigami framework.

I’ve named the game Kolorfill, because it is about filling colors. It looks like this:

Kolorfill

The end goal is to make the board into one color in as few steps as possible. The way to do it is “Paint bucket”-tool from top left corner with various colors.

But enough talk. Let’s see some code:
https://cgit.kde.org/scratch/sune/kolorfill.git/

And of course, there is some QML tests for the curious.
A major todo item is saving the high score and getting that to work. Patches welcome. Or pointer to what QML components that can help me with that.

11 June, 2018 07:07PM by Sune Vuorela

Shashank Kumar

Google Summer of Code 2018 with Debian - Week 4

After working on designs and getting my hands dirty with KIVY for the first 3 weeks, I became comfortable with my development environment and was able to deliver features within a couple of days with UI, tests, and documentation. In this blog, I explain how I converted all my Designs into Code and what I've learned along the way.

The Sign Up

New Contributor Wizard - SignUp

In order to implement above design in KIVY, the best way is to write a user kv-lang. It involves writing a kv file which contains widget tree of the layout and a lot more. One can learn more about kv-lang from the documentation. To begin with, let us look at the simplest kv file.

BoxLayout:
    Label:
        text: 'Hello'
    Label:
        text: 'World'
KV Language

In KIVY, in order to build UI widgets are used. Also, widget base class is what is derived to create all other UI elements like layouts, button, label and so on in KIVY. Indentation is used in kv just like in Python to define children. In our kv file above, we're using BoxLayout which allows us to arrange all its children in either horizontal(by default) or vertical orientation. So, both the Labels will be oriented horizontally one after another.

Just like children widgets, one can also set values to properties like Hello to text of the first Label in above code. More information about what properties can be defined for BoxLayout and Label can be seen from their API documentaion. All which remains is importing this .kv (say sample.kv) file from your module which runs KIVY app. You might notice that for now Language and Timezone are kept static. The reason is, Language support architecture is yet to be finalized and both the options would require a Drop Down list, design and implementation for which will be handled separately.

In order for me to build the UI following the design, I had to experiment with widgets. When all was done, signup.kv file contained the resultant UI.

Validations

Now, the good part is we have a UI, the user can input data. And the bad part is user can input any data! So, it's very important to validate whether the user is submitting data in the correct format or not. Specifically for Sign Up module, I had to validate Email, Passwords and Full Name submitted by the user. Validation module can be found here which contains classes and methods for what I intended to do.

It's important that user gets feedback after validation if something is wrong with the input. This is done by exchanging the Label's text with error message and color with bleeding red by calling prompt_error_message for unsuccessful validation.

Updating The Database

After successful validation, Sign Up module steps forward to update the database in sqlite3 module. But before that, Email and Full Name is cleaned for any unnecessary whitespaces, tabs and newline characters. Universally unique identifier or uuid is generated for the user_id. Plain text Password in changed to sha256 hash string for security. Finally, sqlite3 is integrated to updatedb.py to update the database. SQlite database is stored in a single file and named new_contributor_wizard.db. For user information, the table named USERS is created if not present during initialization of UpdateDB instance. Finally, information is stored or error is returned if the Email already exists. This is how the USERS schema looks like.

id VARCHAR(36) PRIMARY KEY,
email UNIQUE,
pass VARCHAR(64),
fullname TEXT,
language TEXT,
timezone TEXT

After the Database is updated, i.e. successful account creation of user, the natural flow is to take the user to the Dashboard screen. In order to make this feature atomic, integration with Dashboard would be done once all 3 (SignUp, SignIn, and Dashboard) features are merged. So, in order to showcase successful sign-up, I've used text confirmation. Below is the screencast of how the feature looks and what changes it makes in the database.

The Sign In

New Contributor Wizard - SignIn

If you look into the difference in UI of SignIn module in comparison with the SignUp, you might notice a few changes.

  • The New Contributor Wizard is now right-aligned
  • Instead of 2 columns taking user information, here we have just one with Email and Password

Hence, the UI experiences only a little change and the result can be seen in singin.py.

Validations

Just like in the Sign Up modules, we are not trusting user's input to be sane. Hence, we validate whether the user is giving us a good format Email and Password. The resultant validations of Sign In modules can be seen in validations.py.

Updating The Database

After successful validation, next step would be cleaning Email and hashing the Password entered by the user. Here we have two possibilities of unsuccessful signin,

  • Either the Email entered by the user doesn't exist in the database
  • Or the Password entered by the user is not correct

Else, the user is signed in successfully. For the unsuccessful signin, I have created a exceptions.py module to prompt the error correctly. updatedb.py contains the database operations for Sign In module.

The Exceptions

Exceptions.py of Sign In contains Exception classes and they are defined as

  • UserError - this class is used to throw an exception when Email doesn't exist
  • PasswordError - this class is used to throw an exception when Password doesn't match the one saved in the database with the corresponding email.

All these modules are integrated with signin.py and the resultant feature can be seen in action in the screencast below. Also, here's the merge request for the same.

The Dashboard

New Contributor Wizard - Dashboard

The Dashboard is completely different than the above two modules. If New Contributor Wizard is the culmination of different user stories and interactive screen then Dashboard is the protagonist of all the other features. A successful SignIn or SignUp will direct the user to the Dashboard. All the tutorials and tools will be available to the user henceforth.

The UI

There are 2 segments of the Dashboard screen, one is for all the menu options on the left and another is for the tutorials and tools for the selected menu option on the right. So, it was needed to change the screen on the right all the time while selecting the menu options. KIVY provides a widget named Screen Manager to manage such an issue gracefully. But in order to have control over the transition of just a part of the screen rather than the entire screen, one has to dig deep into the API and work it out. Here's when I remembered a sentence from the Zen of Python, "Simple is better than complex" and I chose the simple way of changing the screen i.e. by adding/removing widget functions.

In the dashboard.py, I'm overidding on_touch_down function to check which menu option the user clicks on and calling enable_menu accordingly.

The menu options on the left are not the Button widget. I had an option of using the Button directly but it would need customization to make them look pretty. Instead, I used BoxLayout and Label to incorporate a button like feature. In enable_menu I only check on top of which option user is clicking using the touch API. Now, all I have to do is highlight the selected option and unfocus all the other options. The final UI can be seen here in dashboard.kv.

Courseware

Along with highlighting the selected option, Dashboard also changes to the courseware i.e. tools and tutorials for the selected option on the right. To provide a modular structure to application, all these options are build as separate modules and then integrated into the Dashboard. Here are all the modules for the courseware build for the Dashboard,

  • blog - Users will be given tools to create and deploy their blogs and also learn the best practices.
  • cli - Understanding Command Line Interface will be the goal with all the tutorials provided in this module.
  • communication - Communication module will have tutorials for IRC and mailing lists and showcase best communication practices. The tools in this module will help user subscribe to the mailing lists of different open source communities.
  • encryption - Encrypting communication and data will be tough using this module.
  • how_to_use - This would be an introductory module for the user for them to understand how to user this application.
  • vcs - Version Control Systems like git is important while working on a project whether personal or with a team and everything in between.
  • way_ahead - This module will help users reach out to different open source communities and organizations. It will also showcase open source project to the user with respect to their preference and information about programs like Google Summer of Code and Outreachy.
Settings

Below the menu are the options for settings. These settings also have separate modules just like courseware. Specifically, they are described as

  • application_settings - Would help out user to manage setting which are specific to KIVY application like resolutions.
  • theme_settings - User can manage theme related setting like color schema using this option
  • profile_settings - Would help the user manage information about themselves

The merge request which incorporates the Dashboard feature in the project can be seen in action in the screencast below.

The Conclusion

The week 4 was a bit satisfying for me as I felt like adding value to the project with these merge requests. As soon as the merge requests are reviewed and merged in the repository, I'll work on integrating all these features together to create a seamless experience as it should be for the user. There are few necessary modifications to be made in the features like supporting multiple languages and adding the gradient to the background as it can be seen in the design. I'll create issues on redmine for the same and will work on them as soon as integration is done. My next task would be designing how tutorials and tasks would look in the right segment of the Dashboard.

11 June, 2018 06:30PM by Shashank Kumar

hackergotchi for Norbert Preining

Norbert Preining

Microsoft’s failed attempt on Debian packaging

Just recently Microsoft Open R 3.5 was announced, as an open source implementation of R with some improvements. Binaries are available for Windows, Mac, and Linux. I dared to download and play around with the files, only to get shocked how incompetent Microsoft is in packaging.

From the microsoft-r-open-mro-3.5.0 postinstall script:

#!/bin/bash

#TODO: Avoid hard code VERSION number in all scripts
VERSION=`echo $DPKG_MAINTSCRIPT_PACKAGE | sed 's/[[:alpha:]|(|[:space:]]//g' | sed 's/\-*//' | awk  -F. '{print $1 "." $2 "." $3}'`
INSTALL_PREFIX="/opt/microsoft/ropen/${VERSION}"

echo $VERSION

ln -s "${INSTALL_PREFIX}/lib64/R/bin/R" /usr/bin/R
ln -s "${INSTALL_PREFIX}/lib64/R/bin/Rscript" /usr/bin/Rscript

rm /bin/sh
ln -s /bin/bash /bin/sh

First of all, the ln -s will not work in case the standard R package is installed, but much worse, forcibly relinking /bin/sh to bash is something I didn’t expect to see.

Then, looking at the prerm script, it is getting even more funny:

#!/bin/bash

VERSION=`echo $DPKG_MAINTSCRIPT_PACKAGE | sed 's/[[:alpha:]|(|[:space:]]//g' | sed 's/\-*//' | awk  -F. '{print $1 "." $2 "." $3}'`
INSTALL_PREFIX="/opt/microsoft/ropen/${VERSION}/"

rm /usr/bin/R
rm /usr/bin/Rscript
rm -rf "${INSTALL_PREFIX}/lib64/R/backup"

Stop, wait, you are removing /usr/bin/R without even checking that it points to the R you have installed???

I guess Microsoft should read a bit up, in particular about dpkg-divert and proper packaging. What came in here was such an exhibition of incompetence that I can only assume they are doing it on purpose.

PostScriptum: A short look into the man page of dpkg-divert will give a nice example how it should be done.

PPS: I first reported these problems in the R Open Forums and later got an answer that they look into it.

11 June, 2018 09:13AM by Norbert Preining

John Goerzen

Running Digikam inside Docker

After my recent complaint about AppImage, I thought I’d describe how I solved my problem. I needed a small patch to Digikam, which was already in Debian’s 5.9.0 package, and the thought of rebuilding the AppImage was… unpleasant.

I thought – why not just run it inside Buster in Docker? There are various sources on the Internet for X11 apps in Docker. It took a little twiddling to make it work, but I did.

My Dockerfile was pretty simple:

FROM debian:buster
MAINTAINER John Goerzen 

RUN apt-get update && \
    apt-get -yu dist-upgrade && \
    apt-get --install-recommends -y install firefox-esr digikam digikam-doc \
         ffmpegthumbs imagemagick minidlna hugin enblend enfuse minidlna pulseaudio \
         strace xterm less breeze && \
    apt-get clean && rm -rf /var/lib/apt/lists/* /tmp/* /var/tmp/*
RUN adduser --disabled-password --uid 1000 --gecos "John Goerzen" jgoerzen && \
    rm -r /home/jgoerzen/.[a-z]*
RUN rm /etc/machine-id
CMD /usr/bin/docker

RUN mkdir -p /nfs/personalmedia /run/user/1000 && chown -R jgoerzen:jgoerzen /nfs /run/user/1000

I basically create the container and my account in it.

Then this script starts up Digikam:

#!/bin/bash

set -e

# This will be unnecessary with docker 18.04 theoretically....  --privileged see
# https://stackoverflow.com/questions/48995826/which-capabilities-are-needed-for-statx-to-stop-giving-eperm
# and https://bugs.launchpad.net/ubuntu/+source/docker.io/+bug/1755250

docker run -ti \
       -v /tmp/.X11-unix:/tmp/.X11-unix -v "/run/user/1000/pulse:/run/user/1000/pulse" -v /etc/machine-id:/etc/machine-id \
       -v /etc/localtime:/etc/localtime \
       -v /dev/shm:/dev/shm -v /var/lib/dbus:/var/lib/dbus -v /var/run/dbus:/var/run/dbus -v /run/user/1000/bus:/run/user/1000/bus  \
       -v "$HOME:$HOME" -v "/nfs/personalmedia/Pictures:/nfs/personalmedia/Pictures" \
     -e DISPLAY="$DISPLAY" \
     -e XDG_RUNTIME_DIR="$XDG_RUNTIME_DIR" \
     -e DBUS_SESSION_BUS_ADDRESS="$DBUS_SESSION_BUS_ADDRESS" \
     -e LANG="$LANG" \
     --user "$USER" \
     --hostname=digikam \
     --name=digikam \
     --privileged \
     --rm \
     jgoerzen/digikam "$@"  /usr/bin/digikam

The goal here was not total security isolation; if it had been, then all the dbus mounting and $HOME mounting was a poor idea. But as an alternative to AppImage — well, it worked perfectly. I could even get security updates if I wanted.

11 June, 2018 07:35AM by John Goerzen

June 10, 2018

hackergotchi for Michal &#268;iha&#345;

Michal Čihař

Weblate 3.0.1

Weblate 3.0.1 has been released today. It contains several bug fixes, most importantly possible migration issue on users when migrating from 2.20. There was no data corruption, just some of the foreign keys were possibly not properly migrated. Upgrading from 3.0 to 3.0.1 will fix this as well as going directly from 2.20 to 3.0.1.

Full list of changes:

  • Fixed possible migration issue from 2.20.
  • Localization updates.
  • Removed obsolete hook examples.
  • Improved caching documentation.
  • Fixed displaying of admin documentation.
  • Improved handling of long language names.

If you are upgrading from older version, please follow our upgrading instructions, the upgrade is more complex this time.

You can find more information about Weblate on https://weblate.org, the code is hosted on Github. If you are curious how it looks, you can try it out on demo server. Weblate is also being used on https://hosted.weblate.org/ as official translating service for phpMyAdmin, OsmAnd, Turris, FreedomBox, Weblate itself and many other projects.

Should you be looking for hosting of translations for your project, I'm happy to host them for you or help with setting it up on your infrastructure.

Further development of Weblate would not be possible without people providing donations, thanks to everybody who have helped so far! The roadmap for next release is just being prepared, you can influence this by expressing support for individual issues either by comments or by providing bounty for them.

Filed under: Debian English SUSE Weblate

10 June, 2018 08:15PM

hackergotchi for Dirk Eddelbuettel

Dirk Eddelbuettel

RcppZiggurat 0.1.5

ziggurats

A maintenance release 0.1.5 of RcppZiggurat is now on the CRAN network for R.

The RcppZiggurat package updates the code for the Ziggurat generator which provides very fast draws from a Normal distribution. The package provides a simple C++ wrapper class for the generator improving on the very basic macros, and permits comparison among several existing Ziggurat implementations. This can be seen in the figure where Ziggurat from this package dominates accessing the implementations from the GSL, QuantLib and Gretl—all of which are still way faster than the default Normal generator in R (which is of course of higher code complexity).

Per a request from CRAN, we changed the vignette to accomodate pandoc 2.* just as we did with the most recent pinp release two days ago. No other changes were made. Other changes that have been pending are a minor rewrite of DOIs in DESCRIPTION, a corrected state setter thanks to a PR by Ralf Stubner, and a tweak for function registration to have user_norm_rand() visible.

The NEWS file entry below lists all changes.

Changes in version 0.1.5 (2018-06-10)

  • Description rewritten using doi for references.

  • Re-setting the Ziggurat generator seed now correctly re-sets state (Ralf Stubner in #7 fixing #3)

  • Dynamic registration reverts to manual mode so that user_norm_rand() is visible as well (#7).

  • The vignette was updated to accomodate pandoc 2* [CRAN request].

Courtesy of CRANberries, there is also a diffstat report for the most recent release. More information is on the RcppZiggurat page.

This post by Dirk Eddelbuettel originated on his Thinking inside the box blog. Please report excessive re-aggregation in third-party for-profit settings.

10 June, 2018 06:27PM

RcppGSL 0.3.6

A maintenance update 0.3.6 of RcppGSL is now on CRAN. The RcppGSL package provides an interface from R to the GNU GSL using the Rcpp package.

Per a request from CRAN, we changed the vignette to accomodate pandoc 2.* just as we did with the most recent pinp release two days ago. No other changes were made. The (this time really boring) NEWS file entry follows:

Changes in version 0.3.6 (2018-06-10)

  • The vignette was updated to accomodate pandoc 2* [CRAN request].

Courtesy of CRANberries, a summary of changes to the most recent release is available.

More information is on the RcppGSL page. Questions, comments etc should go to the issue tickets at the GitHub repo.

This post by Dirk Eddelbuettel originated on his Thinking inside the box blog. Please report excessive re-aggregation in third-party for-profit settings.

10 June, 2018 06:20PM

RcppClassic 0.9.10

A maintenance release RcppClassic 0.9.9 is now at CRAN. This package provides a maintained version of the otherwise deprecated first Rcpp API; no new projects should use it.

Per a request from CRAN, we changed the vignette to accomodate pandoc 2.* just as we did with the most recent pinp release two days ago. No other changes were made.

CRANberries also reports the changes relative to the previous release.

Questions, comments etc should go to the rcpp-devel mailing list off the R-Forge page.

This post by Dirk Eddelbuettel originated on his Thinking inside the box blog. Please report excessive re-aggregation in third-party for-profit settings.

10 June, 2018 04:36PM

hackergotchi for Ben Hutchings

Ben Hutchings

Debian LTS work, May 2018

I was assigned 15 hours of work by Freexian's Debian LTS initiative and worked all those hours.

I uploaded the pending changes to linux at the beginning of the month, one of which had been embargoed. I prepared and released another update to the Linux 3.2 longterm stable branch (3.2.102). I then made a final upload of linux based on that.

10 June, 2018 03:05PM

John Goerzen

Please stop making the library situation worse with attempts to fix it

I recently had a simple-sounding desire. I would like to run the latest stable version of Digikam. My desktop, however, runs Debian stable, which has 5.3.0, not 5.9.0.

This is not such a simple proposition.


$ ldd /usr/bin/digikam | wc -l
396

And many of those were required at versions that weren’t in stable.

I had long thought that AppImage was a rather bad idea, but I decided to give it a shot. I realized it was worse than I had thought.

The problems with AppImage

About a year ago, I wrote about the problems Docker security. I go into much more detail there, but the summary for AppImage is quite similar. How can I trust all the components in the (for instance) Digikam AppImage image are being kept secure? Are they using the latest libssl and libpng, to avoid security issues? How will I get notified of a security update? (There seems to be no mechanism for this right now.) An AppImage user that wants to be secure has to manually answer every one of those questions for every application. Ugh.

Nevertheless, the call of better facial detection beckoned, and I downloaded the Digikam AppImage and gave it a whirl. The darn thing actually fired up. But when it would play videos, there was no sound. Hmmmm.

I found errors like this:

Cannot access file ././/share/alsa/alsa.conf

Nasty. I spent quite some time trying to make ALSA work, before a bunch of experimentation showed that if I ran alsoft-conf on the host, and selected only the PulseAudio backend, then it would work. I reported this bug to Digikam.

Then I thought it was working — until I tried to upload some photos. It turns out that SSL support in Qt in the AppImage was broken, since it was trying to dlopen an incompatible version of libssl or libcrypto on the host. More details are in the bug I reported about this also.

These are just two examples. In the rather extensive Googling I did about these problems, I came across issue after issue people had with running Digikam in an AppImage. These issues are not limited to the ALSA and SSL issues I describe here. And they are not occurring due to some lack of skill on the part of Digikam developers.

Rather, they’re occurring because AppImage packaging for a complex package like this is hard. It’s hard because it’s based on a fiction — the fiction that it’s possible to make an AppImage container for a complex desktop application act exactly the same, when the host environment is not exactly the same. Does the host run PulseAudio or ALSA? Where are its libraries stored? How do you talk to dbus?

And it’s not for lack of trying. The scripts to build the Digikam appimage support runs to over 1000 lines of code in the AppImage directory, plus another 1300 lines of code (at least) in CMake files that handle much of the work, and another 3000 lines or so of patches to 3rd-party packages. That’s over 5000 lines of code! By contrast, the Debian packaging for the same version of Digikam, including Debian patches but excluding the changelog and copyright files, amounts to 517 lines. Of course, it is reusing OS packages for the dependencies that were already built, but this amounts to a lot simpler build.

Frankly I don’t believe that AppImage really lives up to its hype. Requiring reinventing a build system and making some dangerous concessions on security for something that doesn’t really work in the end — not good in my book.

The library problem

But of course, AppImage exists for a reason. That reason is that it’s a real pain to deal with so many levels of dependencies in software. Even if we were to compile from source like the old days, and even if it was even compatible with the versions of the dependencies in my OS, that’s still a lot of work. And if I have to build dependencies from source, then I’ve given up automated updates that way too.

There’s a lot of good that ELF has brought us, but I can’t help but think that it wasn’t really designed for a world in which a program links 396 libraries (plus dlopens a few more). Further, this world isn’t the corporate Unix world of the 80s; Open Source developers aren’t big on maintaining backwards compatibility (heck, both the KDE and Qt libraries under digikam have both been entirely rewritten in incompatible ways more than once!) The farther you get from libc, the less people seem to care about backwards compatibility. And really, who can blame volunteers? You want to work on new stuff, not supporting binaries from 5 years ago, right?

I don’t really know what the solution is here. Build-from-source approaches like FreeBSD and Gentoo have plenty of drawbacks too. Is there some grand solution I’m missing? Some effort to improve this situation without throwing out all the security benefits that individually-packaged libraries give us in distros like Debian?

10 June, 2018 08:31AM by John Goerzen

June 09, 2018

hackergotchi for Lars Wirzenius

Lars Wirzenius

Hacker Noir developments

I've been slowly writing on would-be novel, Hacker Noir. See also my Patreon post. I've just pushed out a new public chapter, Assault, to the public website, and a patron-only chapter to Patreon: "Ambush", where the Team is ambushed, and then something bad happens.

The Assault chapter was hard to write. It's based on something that happened to me earlier this year. The Ambush chapter was much more fun.

09 June, 2018 06:47PM

New chapter of Hacker Noir on Patreon

For the 2016 NaNoWriMo I started writing a novel about software development, "Hacker Noir". I didn't finish it during that November, and I still haven't finished it. I had a year long hiatus, due to work and life being stressful, when I didn't write on the novel at all. However, inspired by both the Doctorow method and the Seinfeld method, I have recently started writing again.

I've just published a new chapter. However, unlike last year, I'm publishing it on my Patreon only, for the first month, and only for patrons. Then, next month, I'll be putting that chapter on the book's public site (noir.liw.fi), and another new chapter on Patreon.

I don't expect to make a lot of money, but I am hoping having active supporters will motivate me to keep writing.

I'm writing the first draft of the book. It's likely to be as horrific as every first-time author's first draft is. If you'd like to read it as raw as it gets, please do. Once the first draft is finished, I expect to read it myself, and be horrified, and throw it all away, and start over.

Also, I should go get some training on marketing.

09 June, 2018 06:45PM

hackergotchi for Dirk Eddelbuettel

Dirk Eddelbuettel

RcppDE 0.1.6

Another maintenance release, now at version 0.1.6, of our RcppDE package is now on CRAN. It follows the most recent (unblogged, my bad) 0.1.5 release in January 2016 and the 0.1.4 release in September 2015.

RcppDE is a "port" of DEoptim, a popular package for derivative-free optimisation using differential evolution optimization, to C++. By using RcppArmadillo, the code becomes a lot shorter and more legible. Our other main contribution is to leverage some of the excellence we get for free from using Rcpp, in particular the ability to optimise user-supplied compiled objective functions which can make things a lot faster than repeatedly evaluating interpreted objective functions as DEoptim (and, in fairness, just like most other optimisers) does.

That is also what lead to this upload: Kyle Baron noticed an issue when nesting a user-supplied compiled function inside a user-supplied compiled objective function -- and when using the newest Rcpp. This has to do with some cleanups we made for how RNG state is, or is not, set and preserved. Kevin Ushey was (once again) a real trooper here and added a simple class to Rcpp (in what is now the development version 0.12.17.2 available on the Rcpp drat repo) and used that here to (selectively) restore behaviour similarly to what we had in Rcpp (but which created another issue for another project). So all that is good now in all use cases. We also have some other changes contributed by Yi Kang some time ago for both JADE style randomization and some internal tweaks. Some packaging details were updated, and that sums up release 0.1.6.

Courtesy of CRANberries, there is also a diffstat report for the most recent release.

This post by Dirk Eddelbuettel originated on his Thinking inside the box blog. Please report excessive re-aggregation in third-party for-profit settings.

09 June, 2018 05:11PM

June 08, 2018

Manuel A. Fernandez Montecelo

Talk about the Debian GNU/Linux riscv64 port at RISC-V workshop

About a month ago I attended the RISC-V workshop (conference, congress) co-organised by the Barcelona Supercomputing Center (BSC) and Universitat Politècnica de Catalunya (UPC).

There I presented a talk with the (unimaginative) name of “Debian GNU/Linux Port for RISC-V 64-bit”, talking about the same topic as many other posts of this blog.

There are 2-3 such RISC-V Workshop events per year, one somewhere in Silicon Valley (initially at UC Berkeley, its birthplace) and the others spread around the world.

The demographics of this gathering are quite different to those of planet-debian; the people attending usually know a lot about hardware and often Linux, GNU toolchains and other FOSS, but sometimes very little about the inner workings of FOSS organisations such as Debian. My talk had these demographics as target, so a lot of its content will not teach anything new for most readers of planet-debian.

Still, I know that some readers are interested in parts of this, now that the slides and videos are published, so here it is:

Also very relevant is that they were using Debian (our very own riscv64 port, recently imported into debian-ports infra) in two of the most important hardware demos in the corridors. The rest were mostly embedded distros to showcase FPS games like Quake2, Doom or similar.


All the feedback that I received from many of the attendees about the availability of the port was very positive and they were very enthusiastic, basically saying that they and their teams were really delighted to be able to use Debian to test their different prototypes and designs, and to drive development.

Also, many used Debian daily in their work and research for other purposes, for example a couple of people were proudly showing to me Debian installed on their laptops.

For me, this feedback is a testament of how much of what we do everyday matters to the world out there.


For the historical curiosity, I also presented a similar talk in a previous workshop (2 years back) at CSAIL / MIT.

At that time the port was in a much more incipient state, mostly a proof of concept (for example the toolchain had not even started to be upstreamed). Links:

08 June, 2018 09:20PM by Manuel A. Fernandez Montecelo

hackergotchi for Junichi Uekawa

Junichi Uekawa

Recently I'm not writing any code.

Recently I'm not writing any code.

08 June, 2018 08:58PM by Junichi Uekawa

hackergotchi for Erich Schubert

Erich Schubert

Elsevier CiteScore™ missing the top conference in data mining

Elsevier Scopus is crap.

It’s really time to abandon Elsevier. German universities canceled their subscriptions. Sweden apparently began now to do so, too. Because Elsevier (and to a lesser extend, other publishers) overcharge universities badly.

Meanwhile, Elsevier still struggles to pretend it offers additional value. For example with the ‘‘horribly incomplete’’ Scopus database. For computer science, Scopus etc. are outright useless.

Elsevier just advertised (spammed) their “CiteScore™ metrics”. “Establishing a new standard for measuring serial citation impact”. Not.

“Powered by Scopus, CiteScore metrics are a comprehensive, current, transparent and “ horribly incomplete for computer science.

An excerpt from Elsevier CiteScore™:

Proceedings of the ACM SIGKDD International Conference on Knowledge Discovery and Data Mining

Scopus coverage years:from 2002 to 2003, from 2005 to 2015(coverage discontinued in Scopus)

ACM SIGKDD is the top conference for data mining (there are others like NIPS with more focus in machine learning - I’m referring to the KDD subdomain).

But for Elsevier, it does not seem to be important.

Forget Elsevier. Also forget Thomson Reuter’s ISI Web of Science. It’s just the same publisher-oriented crap.

Communications of the ACM: Research Evaluation For Computer Science

Niklaus Wirth, Turing Award winner, appears for minor papers from indexed publications, not his seminal 1970 Pascal report. Knuth’s milestone book series, with an astounding 15,000 citations in Google Scholar, does not figure. Neither do Knuth’s three articles most frequently cited according to Google.

Yes, if you ask Elsevier or Thomson Reuter’s, Donald Knuth’s “the art of computer programming” does not matter. Because it is not published by Elsevier.

They also ignore the fact that open-access gains importance quickly. Many very influencial papers such as “word2vec” have been published first in the open-access preprint server arXiv. Some never even were published anywhere else.

According to Google Scholar, the top venue for artificial intelligence is arXiv cs.LG, and stat.ML is ranked 5. And the top venue for computational linguistics is arXiv cs.CL. In databases and information systems the top venue WWW publishes via ACM, but using open-access links from their web page. The second, VLDB, operates their own server to publish PVLDB as open-access. And number three is arXiv cs.SI, number five is arXiv cs.DB.

Time to move to open-access, and away from overpriced publishers. If you want your paper to be read and cited, publish open-access and not with expensive walled gardens like Elsevier.

08 June, 2018 02:01PM by Erich Schubert

June 07, 2018

Thorsten Alteholz

My Debian Activities in May 2018

FTP master

This month I accepted 304 packages and rejected 20 uploads. The overall number of packages that got accepted this month was 420.

Debian LTS

This was my forty seventh month that I did some work for the Debian LTS initiative, started by Raphael Hertzog at Freexian.

This month my all in all workload has been 24.25h. During that time I did LTS uploads of:

    [DLA 1387-1] cups security update for one CVE
    [DLA 1388-1] wireshark security update for 9 CVEs

I continued to work on the bunch of wireshark CVEs and sorted all out that did not affect Jessie or Stretch. At the end I sent my dediff with patches for 20 Jessie CVEs and 38 CVES for Stretch to Moritz so that he could compare them with his own work. Unfortunately he didn’t use all of them.

The CVEs for krb5 were marked as no-dsa by the security team, so there was no upload for Wheezy. Building the package for cups was a bit annoying as the test suite didn’t want to run in the beginning.

I also tested the apache2 package from Roberto twice and let the package do a second round before the final upload.

Last but not least I did a week of frontdesk duties and prepared my new working environment for Jessie LTS and Wheezy ELTS.

Other stuff

During May I did uploads of …

  • libmatthew-java to fix a FTBFS with Java 9 due to a disappearing javah. In the end it resulted in a new upstream version.

I also prepared the next libosmocore transistion by uploading several osmocom packages to experimental. This has to continue in June.

Further I sponsored some glewlwyd packages for Nicolas Mora. He is right on his way to become a Debian Maintainer.

Last but not least I uploaded the new package libterm-readline-ttytter-per, which is needed to bring readline functionality to oysttyer, a command line twitter client.

07 June, 2018 10:51PM by alteholz

My Debian Activities in May 2018

FTP master

This month I accepted 304 packages and rejected 20 uploads. The overall number of packages that got accepted this month was 420.

Debian LTS

This was my forty seventh month that I did some work for the Debian LTS initiative, started by Raphael Hertzog at Freexian.

This month my all in all workload has been 24.25h. During that time I did LTS uploads of:

    [DLA 1387-1] cups security update for one CVE
    [DLA 1388-1] wireshark security update for 9 CVEs

I continued to work on the bunch of wireshark CVEs and sorted all out that did not affect Jessie or Stretch. At the end I sent my dediff with patches for 20 Jessie CVEs and 38 CVES for Stretch to Moritz so that he could compare them with his own work. Unfortunately he didn’t use all of them.

The CVEs for krb5 were marked as no-dsa by the security team, so there was no upload for Wheezy. Building the package for cups was a bit annoying as the test suite didn’t want to run in the beginning.

I also tested the apache2 package from Roberto twice and let the package do a second round before the final upload.

Last but not least I did a week of frontdesk duties and prepared my new working environment for Jessie LTS and Wheezy ELTS.

Other stuff

During May I did uploads of …

  • libmatthew-java to fix a FTBFS with Java 9 due to a disappearing javah. In the end it resulted in a new upstream version.

I also prepared the next libosmocore transistion by uploading several osmocom packages to experimental. This has to continue in June.

Further I sponsored some glewlwyd packages for Nicolas Mora. He is right on his way to become a Debian Maintainer.

Last but not least I uploaded the new package libterm-readline-ttytter-per, which is needed to bring readline functionality to oysttyer, a command line twitter client.

07 June, 2018 08:53PM by alteholz