RoadMap Nov 2020

RoadMap to 2.0

State

Updated Nov 2020 (version 1.9.4).

'Test && polish period', addressing the POST section and optionally selected issues from BUILDING && REPOSITORY.

Milestones

  • New deb package layout: mini-buildd (the service), mini-buildd-utils (for users), python3-mini-buildd, mini-buildd-doc
  • Code migration to python 3.
  • Update to django 2.2 (minimal version).
  • Switch to twisted as http backend.
  • Support secure data channels (HTTPS && SFTP).
  • API upgrade to json.
  • Event message queue.
  • Periodic task scheduler (cron, internal use).
  • Automated repository snasphots (weekly, monthly, yearly).
  • WEB UI rewrite.

2.0 will

  • be 'upgrade-compatible' with 1.0 (especially, there is no SQL scheme change).
  • not be run-time compatible with 1.0.

Open Issues

Most of these detailed "issues" are in a 'note to self' fashion (possibly hard to understand for outsiders).

Done issues will be removed from this list (consult the Debian changelog for done things).

BUILDING && REPOSITORY

  • chroot check (dist-upgrade) may stall, blocking mbd from starting (Hint: Stalls in Chroot._mbd_schroot_run(). On 1.0, schroot-cleanup did remedy)
  • [WISH] Library of predefined sbuild --chroot-setup-command snippets (eatmydata, ccache)
  • QA
    • autopkgtest support?
    • reproducibility
    • piuparts support?
  • DEB_BUILD_OPTIONS vs DEB_BUILD_PROFILES
  • [BUG]: When service is stopped, sbuild continues (try w/ chromium, it builds in 13h).
  • [WISH] rm libeatmydata as default (?).
  • [BUG]: When source switches to another archive, you can update the chroot via chroot.check(). However, this needs to be done manually (confusing). Maybe this can be done prior to building?
  • [WISH] Better Daemon defaults for
    • Build queue size
    • Sbuild jobs: Is it safe to default to -j auto or -j N with N > 1 ???
      • debhelper-based packages only actually use this if enabled explicitly ('dh --parallel')
      • ?? Any cases where this could cause parallel builds where not wanted?
  • [CHECK]: Can we use more finegrainded PRIOS on archives (instead of NotAutomatic, ButAutomo...)??
  • [BISH]: auto-deactivation:
    1. When repo is (auto-)deactivated, API calls like show and search don't work any more either.
    2. When one source is auto-deactivated (likely for rolling unstable+testing), the whole repository is deactivated.
  • Sources/Archives: Support partial arch archives (?)
  • [WISH]: Timeout when checking sources (?)
  • [WISH] Make passive ftp port range configurable.
  • [FISH] Make use of [python3-]distro-info
  • [FEAT]: Implement "wake remote" support.
  • [FEAT]: config: Add "purge" action (bulk-delete instances) for chroots.
  • notify: Admin notify on deactivated daemon.
    • ->Daemon.notify for admin messages, repo.notify for packages.
    • ->Daemon.notify for email, rest as before

REVISIT/POST

  • Debian package dependencies:
    • p-d-registration
    • salsa pull: ftps patches for dput.
    • salsa pull: ftps patches for dput-ng.
    • salsa pull: gbp: allow setting --distribution for --snapshot.
  • Bugs:
    • UNREPRODUCIBLE: LOG: daemon.log (file log only) stops working after some time???
    • twisted:
      • UNREPRODUCIBLE: events: deadlock just calling events several times(?).
      • Warning: Sometimes on shutdown: stacktrace: Request.finish called on a request after its connection was lost; use Request.notifyFinish to keep track of this.
      • warning: SSL: 2020-06-29 15:24:11,220 W: /usr/lib/python3/dist-packages/twisted/internet/_sslverify.py:238: DeprecationWarning: NPN is deprecated. Protocols should switch to using ALPN.
      • Nov 2020: Late error w/ updatetestall.
    • UNREPRODUCIBLE: on portext: changes.gen_buildrequest: UnboundLocalError 'mini_buildd' on 'if f["md5sum"] == mini_buildd.misc.md5_of_file(p):'
    • UNREPRODUCIBLE: gpg-agent left-overs (?)
    • UNREPRODUCIBLE: sbuild_kill, package building keeps running
    • (unverified) Can't change user upload permissions?
    • testsuite: debrepro cleanup error "Device or resource busy": with newer 5er kernels? only sid chroot under buster?
      • root cause: fusermount -u fails in debrepro script: fusermount: failed to mark mounts private: Invalid argument. Later, removal of dir fails.
    • model data: need restart?
    • Intermediate unpredicatble state of daemon instance:
      After Daemon instance creation and before any initial _update_from_model() call, the singleton daemon instance is in an unexpected state for some code (value set to None in constructor). This may lead to confusing errors. Reproduce: './devel update' with the sleep set to 1 seconds.
  • Tasks:
    • debconf update (as late as possible, merge local branch).
      • debconf, auto SSL?
    • Keep status of packages/builds between shutdowns
  • django:
    • admin WEB config: Errors adding some objects (dirchroot, ...)
    • Harden against "dict has changed size during iteration" error: Select context, copy context.
  • Security audit:
    • HTTP error (security): OK to deliver str(exception) (html, header)?
  • Testing:
    • General GAU tests.
    • Upgrading from 1.0.x.
    • mini-buildd-ssh-*-command setup.
    • buster compat:
      • Can't find clipboard.js
  • Documentation:
    • DOC: create Debian NEWS file from doc/NEWS
    • [DOC] "../downgrade.buster-ab.pref" (if changing to '~9').
    • [DOC] Quickstart, APT bootstrap: Test and check for jessie, stretch, buster.
    • sbuild_kill: Leaves chroot snasphot?

Post 2.0

  • External user management (ldap, sso?)
  • gnupg: Switch to 'python3-gpg' (NOT 'python3-gpgme').
  • [WISH]: Auto ports and/or Port-Alarm on Debian Package Updates
  • SQL schema migration (with django >= 1.7)
  • Extended User Role Concept, with advanced right management workflows.
  • [FEAT]: Add suite "wildwest", including some way to add pre-build packages.
  • API: debdiff
  • setup:
    • sync config between instances
    • sync config w/ defaults
    • [WISH] admin/model objects: Show "used by count" (so we can see what objects may be obsolete).
    • [WISH] Archive: Rethink Archive::mbd_ping (see https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=740576)
    • [WISH] Integrate ssh upload (using mini-buildd user?) (currently in examples only)
  • [WISH] Automatic SSH upload support, either by scripting the setup or integrating SSH w/ twisted.
  • REPOSITORY
    • sid-test-nakedgoose: Add extra suite for arbitrary package (deb+src) installs.

Bummers