RoadMap Nov 2020

Table of Contents

RoadMap to 2.0
Post 2.0
Bummers

RoadMap to 2.0

State

Updated Nov 2020 (version 1.9.4).

'Test && polish period', addressing the POST section and optionally selected issues from BUILDING && REPOSITORY.

Milestones

New deb package layout: mini-buildd (the service), mini-buildd-utils (for users), python3-mini-buildd, mini-buildd-doc
Code migration to python 3.
Update to django 2.2 (minimal version).
Switch to twisted as http backend.
Support secure data channels (HTTPS && SFTP).
API upgrade to json.
Event message queue.
Periodic task scheduler (cron, internal use).
Automated repository snasphots (weekly, monthly, yearly).
WEB UI rewrite.

2.0 will

be 'upgrade-compatible' with 1.0 (especially, there is no SQL scheme change).
not be run-time compatible with 1.0.

Open Issues

Most of these detailed "issues" are in a 'note to self' fashion (possibly hard to understand for outsiders).

Done issues will be removed from this list (consult the Debian changelog for done things).

BUILDING && REPOSITORY

chroot check (dist-upgrade) may stall, blocking mbd from starting (Hint: Stalls in Chroot._mbd_schroot_run(). On 1.0, schroot-cleanup did remedy)
[WISH] Library of predefined sbuild --chroot-setup-command snippets (eatmydata, ccache)
QA
- autopkgtest support?
- reproducibility
- piuparts support?
DEB_BUILD_OPTIONS vs DEB_BUILD_PROFILES
[BUG]: When service is stopped, sbuild continues (try w/ chromium, it builds in 13h).
[WISH] rm libeatmydata as default (?).
[BUG]: When source switches to another archive, you can update the chroot via chroot.check(). However, this needs to be done manually (confusing). Maybe this can be done prior to building?
[WISH] Better Daemon defaults for
- Build queue size
- Sbuild jobs: Is it safe to default to -j auto or -j N with N > 1 ???
  
  debhelper-based packages only actually use this if enabled explicitly ('dh --parallel')
  
  ?? Any cases where this could cause parallel builds where not wanted?
[CHECK]: Can we use more finegrainded PRIOS on archives (instead of NotAutomatic, ButAutomo...)??
[BISH]: auto-deactivation:
1. When repo is (auto-)deactivated, API calls like show and search don't work any more either.
2. When one source is auto-deactivated (likely for rolling unstable+testing), the whole repository is deactivated.
Sources/Archives: Support partial arch archives (?)
[WISH]: Timeout when checking sources (?)
[WISH] Make passive ftp port range configurable.
[FISH] Make use of [python3-]distro-info
[FEAT]: Implement "wake remote" support.
[FEAT]: config: Add "purge" action (bulk-delete instances) for chroots.
notify: Admin notify on deactivated daemon.
- ->Daemon.notify for admin messages, repo.notify for packages.
- ->Daemon.notify for email, rest as before

REVISIT/POST

Debian package dependencies:
- p-d-registration
- salsa pull: ftps patches for dput.
- salsa pull: ftps patches for dput-ng.
- salsa pull: gbp: allow setting --distribution for --snapshot.
Bugs:
- UNREPRODUCIBLE: LOG: daemon.log (file log only) stops working after some time???
- twisted:
  
  UNREPRODUCIBLE: events: deadlock just calling events several times(?).
  
  Warning: Sometimes on shutdown: stacktrace: Request.finish called on a request after its connection was lost; use Request.notifyFinish to keep track of this.
  
  warning: SSL: 2020-06-29 15:24:11,220 W: /usr/lib/python3/dist-packages/twisted/internet/_sslverify.py:238: DeprecationWarning: NPN is deprecated. Protocols should switch to using ALPN.
  
  Nov 2020: Late error w/ updatetestall.
- UNREPRODUCIBLE: on portext: changes.gen_buildrequest: UnboundLocalError 'mini_buildd' on 'if f["md5sum"] == mini_buildd.misc.md5_of_file(p):'
- UNREPRODUCIBLE: gpg-agent left-overs (?)
- UNREPRODUCIBLE: sbuild_kill, package building keeps running
- (unverified) Can't change user upload permissions?
- testsuite: debrepro cleanup error "Device or resource busy": with newer 5er kernels? only sid chroot under buster?
  
  root cause: fusermount -u fails in debrepro script: fusermount: failed to mark mounts private: Invalid argument. Later, removal of dir fails.
- model data: need restart?
  
  Daemon change (for example: email regex), repository uploaders change, ...: Needs start/stop to become active.
  
  https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=838393
- Intermediate unpredicatble state of daemon instance:
  
  After Daemon instance creation and before any initial _update_from_model() call, the singleton daemon instance is in an unexpected state for some code (value set to None in constructor). This may lead to confusing errors. Reproduce: './devel update' with the sleep set to 1 seconds.
Tasks:
- debconf update (as late as possible, merge local branch).
  
  debconf, auto SSL?
- Keep status of packages/builds between shutdowns
django:
- admin WEB config: Errors adding some objects (dirchroot, ...)
- Harden against "dict has changed size during iteration" error: Select context, copy context.
Security audit:
- HTTP error (security): OK to deliver str(exception) (html, header)?
Testing:
- General GAU tests.
- Upgrading from 1.0.x.
- mini-buildd-ssh-*-command setup.
- buster compat:
  
  Can't find clipboard.js
Documentation:
- DOC: create Debian NEWS file from doc/NEWS
- [DOC] "../downgrade.buster-ab.pref" (if changing to '~9').
- [DOC] Quickstart, APT bootstrap: Test and check for jessie, stretch, buster.
- sbuild_kill: Leaves chroot snasphot?

Post 2.0

External user management (ldap, sso?)
gnupg: Switch to 'python3-gpg' (NOT 'python3-gpgme').
[WISH]: Auto ports and/or Port-Alarm on Debian Package Updates
SQL schema migration (with django >= 1.7)
Extended User Role Concept, with advanced right management workflows.
[FEAT]: Add suite "wildwest", including some way to add pre-build packages.
API: debdiff
setup:
- sync config between instances
- sync config w/ defaults
- [WISH] admin/model objects: Show "used by count" (so we can see what objects may be obsolete).
- [WISH] Archive: Rethink Archive::mbd_ping (see https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=740576)
- [WISH] Integrate ssh upload (using mini-buildd user?) (currently in examples only)
[WISH] Automatic SSH upload support, either by scripting the setup or integrating SSH w/ twisted.
REPOSITORY
- sid-test-nakedgoose: Add extra suite for arbitrary package (deb+src) installs.

Bummers

http://tools.ietf.org/html/rfc5147: This, in theory, could be some way to create a TOC for text/plain (buildlogs and such). However
- RFC only describes line/char based links, not text matching based (so scanning the text file would be needed still)
- As it seems, main browsers simply don't support it
  
  https://bugzilla.mozilla.org/show_bug.cgi?id=660583
  
  https://bugs.chromium.org/p/chromium/issues/detail?id=77024

Other articles

RoadMap Oct 2020

RoadMap to 2.0

State