March 18, 2019

18 March, 2019 03:12PM · XF86Config

sudo apt install debootstrap schroot

SYSROOT=/srv/chroot/buster-amd64-pelican
sudo mkdir -p ${SYSROOT:?}
sudo debootstrap --variant=minbase buster ${SYSROOT:?}

ls ${SYSROOT:?}

cat << EOF | sudo tee /etc/schroot/chroot.d/buster-amd64-pelican.conf
[buster-amd64-pelican]
users=$LOGNAME
root-users=$LOGNAME
source-users=$LOGNAME
source-root-users=$LOGNAME
type=directory
union-type=overlay
directory=/srv/chroot/buster-amd64-pelican
EOF

schroot -c source:buster-amd64-pelican -u root -- \
  bash -c "apt update && apt install --yes make pelican && apt clean"

$ schroot -l
chroot:buster-amd64-pelican
source:buster-amd64-pelican

$ schroot -c chroot:buster-amd64-pelican

$ schroot -c buster-amd64-pelican

(chroot)$ touch /var/tmp/this-is-an-empty-file
(chroot)$ ls /var/tmp
this-is-an-empty-file

(chroot)$ pwd
/home/arno/my-pelican-blog
(chroot)$ ls
content  Makefile  pelicanconf.py  ...

$ nano content/bla-bla-bla.md

$ schroot -c buster-amd64-pelican
(chroot)$ make html
(chroot)$ make serve

PY?=python3
PELICAN?=pelican

PY?=schroot -c buster-amd64-pelican -- python3
PELICAN?=schroot -c buster-amd64-pelican -- pelican

schroot -c source:$PROJECT -u root -- \
  bash -c "apt update && apt --yes dist-upgrade && apt clean"

sudo rm /etc/schroot/chroot.d/buster-amd64-pelican.conf
sudo rm -fr /srv/chroot/buster-amd64-pelican

#!/usr/bin/python3

import git
from pathlib import Path
import sys

def list_paths(root_tree, path=Path(".")):
    for blob in root_tree.blobs:
        yield path / blob.name
    for tree in root_tree.trees:
        yield from list_paths(tree, path / tree.name)

repo = git.Repo(".", search_parent_directories=True)
commit = repo.commit(sys.argv[1])
for path in list_paths(commit.tree):
    print(path)

---[ Linear mapping ]---
...
0xffffb07cfc402000-0xffffb07cfc403000    4K PTE   ro NX SHD AF NG    UXN MEM/NORMAL
0xffffb07cfc403000-0xffffb07cfc4d0000  820K PTE   RW NX SHD AF NG    UXN MEM/NORMAL
0xffffb07cfc4d0000-0xffffb07cfc4d1000    4K PTE   ro NX SHD AF NG    UXN MEM/NORMAL
0xffffb07cfc4d1000-0xffffb07cfc79d000 2864K PTE   RW NX SHD AF NG    UXN MEM/NORMAL

8010fad8:       e30c4488        movw    r4, #50312      ; 0xc488
8010fadc:       e34840d0        movt    r4, #32976      ; 0x80d0
...
8010fb1c:       e51b2030        ldr     r2, [fp, #-48]  ; 0xffffffd0
8010fb20:       e5943000        ldr     r3, [r4]
8010fb24:       e1520003        cmp     r2, r3
8010fb28:       1a000020        bne     8010fbb0
...
8010fbb0:       eb006738        bl      80129898 <__stack_chk_fail>

8010fc18:       e1a0300d        mov     r3, sp
8010fc1c:       e3c34d7f        bic     r4, r3, #8128   ; 0x1fc0
...
8010fc60:       e51b2030        ldr     r2, [fp, #-48]  ; 0xffffffd0
8010fc64:       e5943018        ldr     r3, [r4, #24]
8010fc68:       e1520003        cmp     r2, r3
8010fc6c:       1a000020        bne     8010fcf4
...
8010fcf4:       eb006757        bl      80129a58 <__stack_chk_fail>

drwxr-xr-x  3 jgoerzen jgoerzen      4096 Dec  3  2003 irclogs
-rw-r--r--  1 jgoerzen jgoerzen       373 Dec  3  2003 .vimrc
-rw-r--r--  1 jgoerzen jgoerzen       651 Nov 27  2003 .reportbugrc
drwx------  3 jgoerzen jgoerzen      4096 Sep  2  2003 .arch-params
-rw-r--r--  1 jgoerzen jgoerzen      1115 Aug 23  2003 .gopherrc
drwxr-xr-x  3 jgoerzen jgoerzen      4096 Jul 18  2003 .subversion
-rw-r--r--  1 jgoerzen jgoerzen     15317 Jun 21  2003 .pinerc
-rw-r--r--   1 root  root                      268 Dec 25  2005 libnet.cfg
-rw-r-----   1 root  root                     1305 Nov 11  2004 mrtg.cfg
-rw-r--r--   1 root  root                      552 Jul 31  2004 pam.conf
glockenspiel:/etc# shutdown -P now
Shared connection to glockenspiel.complete.org closed.
> I'd prefer having this documented in the release notes:
> https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=916690
> with possible solutions like installing haveged, configuring virtio-rng,
> etc. depending on the situation.

That would be an extremely user-unfriendly "solution" and would lead to 
countless hours of debugging and useless bug reports.

If that pool is used by the guest to generate long-term secrets because it appears to be well-initialized, that could be a serious problem.

(e.g. "Mining your P's and Q's" by Heninger et al -- https://factorable.net/weakkeys12.extended.pdf)

I've just opened https://bugs.launchpad.net/qemu/+bug/1811758 to report a way to improve that situation in qemu by default.

$ egrep '^[^#]' ~/.cargo/config
[source]
[source.debian-packages]
directory = "/usr/share/cargo/registry"
[source.crates-io]
replace-with = "debian-packages"
$

[dependencies]
GSL = "1"

GSL = { path = "../rust-GSL", optional = true }

from tornado.options import define, options
import tornado.netutil
import tornado.httpserver

define("web_port", type=int, default=None, help="listening port for web interface")

application = Application(self.db_url)

if options.web_port is None:
    sockets = tornado.netutil.bind_sockets(0, '127.0.0.1')
    self.web_port = sockets[0].getsockname()[:2][1]
    server = tornado.httpserver.HTTPServer(application)
    server.add_sockets(sockets)
else:
    server = tornado.httpserver.HTTPServer(application)
    server.listen(options.web_port)

.github
.gitignore
.gitattributes
.travis.yml
.appveyor.yml
.gitattributes

# don't export the github-pages source
/docs export-ignore
# export some other irrelevant directories
/foo export-ignore
# don't export the files necessary for CI
Gemfile export-ignore
.appveyor.yml export-ignore
.travis.yml export-ignore
# ignore Git related files
.gitattributes export-ignore
.gitignore export-ignore

import asyncio
from asyncio.log import logger
from contextlib import contextmanager
import os

class ExtendedResults:
    def __init__(self):
        self.rusage = None
        self.returncode = None

class SafeChildWatcherWithRusage(asyncio.SafeChildWatcher):
    """
    SafeChildWatcher that uses os.wait4 to also get rusage information.
    """
    rusage_results = {}

    @classmethod
    @contextmanager
    def monitor(cls, proc):
        """
        Return an ExtendedResults that gets filled when the process exits
        """
        assert proc.pid > 0
        pid = proc.pid
        extended_results = ExtendedResults()
        cls.rusage_results[pid] = extended_results
        try:
            yield extended_results
        finally:
            cls.rusage_results.pop(pid, None)

    def _do_waitpid(self, expected_pid):
        # The original is in asyncio/unix_events.py; on new python versions, it
        # makes sense to check changes to it and port them here
        assert expected_pid > 0

        try:
            pid, status, rusage = os.wait4(expected_pid, os.WNOHANG)
        except ChildProcessError:
            # The child process is already reaped
            # (may happen if waitpid() is called elsewhere).
            pid = expected_pid
            returncode = 255
            logger.warning(
                "Unknown child process pid %d, will report returncode 255",
                pid)
        else:
            if pid == 0:
                # The child process is still alive.
                return

            returncode = self._compute_returncode(status)
            if self._loop.get_debug():
                logger.debug('process %s exited with returncode %s',
                             expected_pid, returncode)

        extended_results = self.rusage_results.get(pid)
        if extended_results is not None:
            extended_results.rusage = rusage
            extended_results.returncode = returncode

        try:
            callback, args = self._callbacks.pop(pid)
        except KeyError:  # pragma: no cover
            # May happen if .remove_child_handler() is called
            # after os.waitpid() returns.
            if self._loop.get_debug():
                logger.warning("Child watcher got an unexpected pid: %r",
                               pid, exc_info=True)
        else:
            callback(pid, returncode, *args)

    @classmethod
    def install(cls):
        loop = asyncio.get_event_loop()
        child_watcher = cls()
        child_watcher.attach_loop(loop)
        asyncio.set_child_watcher(child_watcher)

from .hacks import SafeChildWatcherWithRusage
SafeChildWatcherWithRusage.install()

...

    @coroutine
    def run(self, *args, **kw):
        kw["stdin"] = asyncio.subprocess.PIPE
        kw["stdout"] = asyncio.subprocess.PIPE
        kw["stderr"] = asyncio.subprocess.PIPE
        self.started = time.time()

        self.proc = yield from asyncio.create_subprocess_exec(*args, **kw)

        from .hacks import SafeChildWatcherWithRusage
        with SafeChildWatcherWithRusage.monitor(self.proc) as results:
            yield from asyncio.tasks.gather(
                self.write_stdin(self.proc.stdin),
                self.read_stdout(self.proc.stdout),
                self.read_stderr(self.proc.stderr)
            )
        self.returncode = yield from self.proc.wait()
        self.rusage = results.rusage
        self.ended = time.time()

(defun anarcat/notmuch-html-convert ()
  """create an HTML part from a Markdown body

This will not work if there are *any* attachments of any form, those should be added after."""
  (interactive)
  (save-excursion
    ;; fetch subject, it will be the HTML version title
    (message "building HTML attachment...")
    (message-goto-subject)
    (beginning-of-line)
    (search-forward ":")
    (forward-char)
    (let ((beg (point))) (end-of-line)
         (setq subject (buffer-substring beg (point))))
    (message "determined title is %s..." subject)
    ;; wrap signature in a <pre>
    (message-goto-signature)
    (forward-line -1)
    ;; save and delete signature which requires special formatting
    (setq signature (buffer-substring (point) (point-max)))
    (delete-region (point) (point-max))
    ;; set region to top of body then end of buffer
    (end-of-buffer)
    (message-goto-body)
    (narrow-to-region (point) (mark))
    ;; run markdown on region
    (setq output-buffer-name "*notmuch-markdown-output*")
    (message "running markdown...")
    (markdown output-buffer-name)
    (widen)
    (save-excursion
      (set-buffer output-buffer-name)
      (end-of-buffer)
      ;; add signature formatted as <pre>
      (insert "\n<pre>")
      (insert signature)
      (insert "</pre>\n")
      (markdown-add-xhtml-header-and-footer subject))
    (message "done the dirty work, re-inserting everything...")
    ;; restore signature
    (message-goto-signature)
    (insert signature)
    (message-goto-body)
    (insert "<#multipart type=alternative>\n")
    (end-of-buffer)
    (insert "<#part type=text/html>\n")
    (insert-buffer output-buffer-name)
    (end-of-buffer)
    (insert "<#/multipart>\n")
    (let ((f (buffer-size (get-buffer output-buffer-name))))
      (message "appended HTML part (%s bytes)" f))))

 [1]: one of so many: https://www.georgedillon.com/web/html_email_is_evil_still.shtml
 [2]: https://en.wikipedia.org/wiki/Netscape_Communicator
 [3]: yes my age is showing
 [4]: to be fair, this article encouraged me quite a bit: https://blog.chaddickerson.com/2019/01/09/replacing-facebook/
 [5]: not the bass guitar one, unfortunately
 [6]: https://en.wikipedia.org/wiki/HTML_email#Adoption
 [7]: https://trey-jackson.blogspot.com/2008/01/emacs-tip-8-markdown.html

from tornado import web
import pathlib

class StaticFileHandler(web.StaticFileHandler):
    """
    StaticFileHandler that allows overriding paths in the static directory with
    system provided versions
    """
    SYSTEM_ASSET_PATH = pathlib.Path("/usr/share/javascript")

    @classmethod
    def get_absolute_path(self, root, path):
        path = pathlib.PurePath(path)
        if not path.parts:
            return super().get_absolute_path(root, path)

        system_dir = self.SYSTEM_ASSET_PATH.joinpath(path.parts[0])
        if system_dir.is_dir():
            # If that asset directory exists in the system, look for things in
            # there
            return self.SYSTEM_ASSET_PATH.joinpath(path)
        else:
            # Else go ahead with the default static dir
            return super().get_absolute_path(root, path)

    def validate_absolute_path(self, root, absolute_path):
        """
        Rewrite of tornado's validate_absolute_path not to raise an error for
        paths in /usr/share/javascript/
        """
        root = pathlib.Path(root)
        absolute_path = pathlib.Path(absolute_path)

        is_system_root = absolute_path.parts[:len(self.SYSTEM_ASSET_PATH.parts)] == self.SYSTEM_ASSET_PATH.parts
        is_static_root = absolute_path.parts[:len(root.parts)] == root.parts

        if not is_system_root and not is_static_root:
            raise web.HTTPError(403, "%s is not in root static directory or system assets path",
                                self.path)

        if absolute_path.is_dir() and self.default_filename is not None:
            # need to look at the request.path here for when path is empty
            # but there is some prefix to the path that was already
            # trimmed by the routing
            if not self.request.path.endswith("/"):
                self.redirect(self.request.path + "/", permanent=True)
                return
            absolute_path = absolute_path.joinpath(self.default_filename)
        if not absolute_path.exists():
            raise web.HTTPError(404)
        if not absolute_path.is_file():
            raise web.HTTPError(403, "%s is not a file", self.path)
        return str(absolute_path)

class DebianApplication(tornado.web.Application):
    def __init__(self, *args, **settings):
        from .static import StaticFileHandler
        settings.setdefault("static_handler_class", StaticFileHandler)
        super().__init__(*args, **settings)

    <link rel="stylesheet" href="{{static_url('bootstrap4/css/bootstrap.min.css')}}">
    <script src="{{static_url('jquery/jquery.min.js')}}"></script>
    <script src="{{static_url('popper.js/umd/popper.min.js')}}"></script>
    <script src="{{static_url('bootstrap4/js/bootstrap.min.js')}}"></script>

>>> import mailbox
>>> print(len(mailbox.mbox("/tmp/test.mbox")))
9
>>> print(len(mailbox.mbox("/tmp/test.mbox.gz")))
0
>>> print(len(mailbox.mbox("/tmp/test1.mbox.xz")))
0

import lzma
import gzip
import bz2
import mailbox

class StreamMbox(mailbox.mbox):
    """
    mailbox.mbox does not support opening a stream, which is sad.

    This is a subclass that works around it
    """
    def __init__(self, fd: BinaryIO, factory=None, create: bool = True):
        # Do not call parent __init__, just redo everything here to be able to
        # open a stream. This will need to be re-reviewed for every new version
        # of python's stdlib.

        # Mailbox constructor
        self._path = None
        self._factory = factory

        # _singlefileMailbox constructor
        self._file = fd
        self._toc = None
        self._next_key = 0
        self._pending = False       # No changes require rewriting the file.
        self._pending_sync = False  # No need to sync the file
        self._locked = False
        self._file_length = None    # Used to record mailbox size

        # mbox constructor
        self._message_factory = mailbox.mboxMessage

    def flush(self):
        raise NotImplementedError("StreamMbox is a readonly class")

class UsageExample:
    DECOMPRESS = {
        ".xz": lzma.open,
        ".gz": gzip.open,
        ".bz2": bz2.open,
    }

    @classmethod
    def scan(cls, path: Path) -> Generator[ScannedEmail, None, None]:
        decompress = cls.DECOMPRESS.get(path.suffix)
        if decompress is None:
            with open(path.as_posix(), "rb") as fd:
                yield from cls.scan_fd(path, fd)
        else:
            with decompress(path.as_posix(), "rb") as fd:
                yield from cls.scan_fd(path, fd)

    @classmethod
    def scan_fd(cls, path: Path, fd: BinaryIO) -> Generator[ScannedEmail, None, None]:
        mbox = StreamMbox(fd)
        for msg in mbox:
            ...

hard disk size	one pass	three passes
1 TB	2 h	6 h
2 TB	4 h	12 h
3 TB	6 h	18 h
4 TB	8 h	24 h (one day)
5 TB	10 h	30 h
6 TB	12 h	36 h
8 TB	16 h	48 h (two days)
10 TB	20 h	60 h
12 TB	24 h	72 h (three days)
14 TB	28 h	84 h
16 TB	32 h	96 h (four days)
18 TB	36 h	108 h
20 TB	40 h	120 h (five days)

March 18, 2019

March 17, 2019

Changes in RQuantLib version 0.4.8 (2019-03-17)

Changes in Rcpp version 1.0.1 (2019-03-17)

March 16, 2019

Changes in littler version 0.3.6 (2019-01-26)

Introduction

Create the chroot

A quick introduction to schroot

Chroot usage with Pelican

Maintenance

Last words

March 15, 2019

Introducing myself

Current contributions

Kernel team

Raspi3-firmware

U-Boot

Raspberry Pi Zero

Raspberry Pi 3

Future work

Multiple tools

Data accessibility

Integration and Uniformity

March 14, 2019

March 13, 2019

Meeting with Bhakts and ‘Good intentions’

Demonetisation

Important Legislations which need Public Scrutiny

The BJP Site hack

Arming yourself

Blowing a raspberry

When is a superset not a superset

Escape from the Pie Folk

Next steps

diffoscope development

Packages reviewed and fixed, and bugs filed

Test framework development

Outreachy

March 12, 2019

Problem

First seen in the wild January 2017

Systemd. Yeah.

Debian

Solutions

Kernel boot parameter

Using a TPM

VirtIO

Patching systemd

Chaoskey

Jitterentropy_RNG

Haveged

early-rng-init-tools

Updates

14.01.2019

16.01.2019

12.03.2019

March 11, 2019

Debian’s Perspective: Stability

Recognizing Personal Preference

The case for reportbug

Where Debian Should Improve

March 10, 2019

Debian Games

Debian Java

Misc

Debian LTS

ELTS

What does this mean?

Why?

Change process in Debian

Fragmented workflow and infrastructure

Old infrastructure: package uploads

Old infrastructure: bug tracker

Old infrastructure: mailing list archives

Debian is hard to machine-read

Complicated build stack

Developer experience pretty painful

I have more ideas