State
Updated Nov 2020 (version 1.9.4).
'Test && polish period', addressing the POST section and optionally selected issues from BUILDING && REPOSITORY.
Milestones
- New deb package layout: mini-buildd (the service), mini-buildd-utils (for users), python3-mini-buildd, mini-buildd-doc
- Code migration to python 3.
- Update to django 2.2 (minimal version).
- Switch to twisted as http backend.
- Support secure data channels (HTTPS && SFTP).
- API upgrade to json.
- Event message queue.
- Periodic task scheduler (cron, internal use).
- Automated repository snasphots (weekly, monthly, yearly).
- WEB UI rewrite.
2.0 will
- be 'upgrade-compatible' with 1.0 (especially, there is no SQL scheme change).
- not be run-time compatible with 1.0.
Open Issues
Most of these detailed "issues" are in a 'note to self' fashion (possibly hard to understand for outsiders).
Done issues will be removed from this list (consult the Debian changelog for done things).
BUILDING && REPOSITORY
- chroot check (dist-upgrade) may stall, blocking mbd from starting (Hint: Stalls in Chroot._mbd_schroot_run(). On 1.0, schroot-cleanup did remedy)
- [WISH] Library of predefined sbuild --chroot-setup-command snippets (eatmydata, ccache)
- QA
- autopkgtest support?
- reproducibility
- piuparts support?
- DEB_BUILD_OPTIONS vs DEB_BUILD_PROFILES
- [BUG]: When service is stopped, sbuild continues (try w/ chromium, it builds in 13h).
- [WISH] rm libeatmydata as default (?).
- [BUG]: When source switches to another archive, you can update the chroot via chroot.check(). However, this needs to be done manually (confusing). Maybe this can be done prior to building?
- [WISH] Better Daemon defaults for
- Build queue size
- Sbuild jobs: Is it safe to default to -j auto or -j N with N > 1 ???
- debhelper-based packages only actually use this if enabled explicitly ('dh --parallel')
- ?? Any cases where this could cause parallel builds where not wanted?
- [CHECK]: Can we use more finegrainded PRIOS on archives (instead of NotAutomatic, ButAutomo...)??
- [BISH]: auto-deactivation:
- When repo is (auto-)deactivated, API calls like show and search don't work any more either.
- When one source is auto-deactivated (likely for rolling unstable+testing), the whole repository is deactivated.
- Sources/Archives: Support partial arch archives (?)
- [WISH]: Timeout when checking sources (?)
- [WISH] Make passive ftp port range configurable.
- [FISH] Make use of [python3-]distro-info
- [FEAT]: Implement "wake remote" support.
- [FEAT]: config: Add "purge" action (bulk-delete instances) for chroots.
- notify: Admin notify on deactivated daemon.
- ->Daemon.notify for admin messages, repo.notify for packages.
- ->Daemon.notify for email, rest as before
REVISIT/POST
- Debian package dependencies:
- p-d-registration
- salsa pull: ftps patches for dput.
- salsa pull: ftps patches for dput-ng.
- salsa pull: gbp: allow setting --distribution for --snapshot.
- Bugs:
- UNREPRODUCIBLE: LOG: daemon.log (file log only) stops working after some time???
- twisted:
- UNREPRODUCIBLE: events: deadlock just calling events several times(?).
- Warning: Sometimes on shutdown: stacktrace: Request.finish called on a request after its connection was lost; use Request.notifyFinish to keep track of this.
- warning: SSL: 2020-06-29 15:24:11,220 W: /usr/lib/python3/dist-packages/twisted/internet/_sslverify.py:238: DeprecationWarning: NPN is deprecated. Protocols should switch to using ALPN.
- Nov 2020: Late error w/ updatetestall.
- UNREPRODUCIBLE: on portext: changes.gen_buildrequest: UnboundLocalError 'mini_buildd' on 'if f["md5sum"] == mini_buildd.misc.md5_of_file(p):'
- UNREPRODUCIBLE: gpg-agent left-overs (?)
- UNREPRODUCIBLE: sbuild_kill, package building keeps running
- (unverified) Can't change user upload permissions?
- testsuite: debrepro cleanup error "Device or resource busy": with newer 5er kernels? only sid chroot under buster?
- root cause: fusermount -u fails in debrepro script: fusermount: failed to mark mounts private: Invalid argument. Later, removal of dir fails.
- model data: need restart?
- Daemon change (for example: email regex), repository uploaders change, ...: Needs start/stop to become active.
- https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=838393
- Intermediate unpredicatble state of daemon instance:
- After Daemon instance creation and before any initial _update_from_model() call, the singleton daemon instance is in an unexpected state for some code (value set to None in constructor). This may lead to confusing errors. Reproduce: './devel update' with the sleep set to 1 seconds.
- Tasks:
- debconf update (as late as possible, merge local branch).
- debconf, auto SSL?
- Keep status of packages/builds between shutdowns
- django:
- admin WEB config: Errors adding some objects (dirchroot, ...)
- Harden against "dict has changed size during iteration" error: Select context, copy context.
- Security audit:
- HTTP error (security): OK to deliver str(exception) (html, header)?
- Testing:
- General GAU tests.
- Upgrading from 1.0.x.
- mini-buildd-ssh-*-command setup.
- buster compat:
- Can't find clipboard.js
- Documentation:
- DOC: create Debian NEWS file from doc/NEWS
- [DOC] "../downgrade.buster-ab.pref" (if changing to '~9').
- [DOC] Quickstart, APT bootstrap: Test and check for jessie, stretch, buster.
- sbuild_kill: Leaves chroot snasphot?
Post 2.0
- External user management (ldap, sso?)
- gnupg: Switch to 'python3-gpg' (NOT 'python3-gpgme').
- [WISH]: Auto ports and/or Port-Alarm on Debian Package Updates
- SQL schema migration (with django >= 1.7)
- Extended User Role Concept, with advanced right management workflows.
- [FEAT]: Add suite "wildwest", including some way to add pre-build packages.
- API: debdiff
- setup:
- sync config between instances
- sync config w/ defaults
- [WISH] admin/model objects: Show "used by count" (so we can see what objects may be obsolete).
- [WISH] Archive: Rethink Archive::mbd_ping (see https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=740576)
- [WISH] Integrate ssh upload (using mini-buildd user?) (currently in examples only)
- [WISH] Automatic SSH upload support, either by scripting the setup or integrating SSH w/ twisted.
- REPOSITORY
- sid-test-nakedgoose: Add extra suite for arbitrary package (deb+src) installs.
Bummers
- http://tools.ietf.org/html/rfc5147: This, in theory, could be some way to create a TOC for text/plain (buildlogs and such). However
- RFC only describes line/char based links, not text matching based (so scanning the text file would be needed still)
- As it seems, main browsers simply don't support it