mini-buildd Development TODO
(0) MOTM: Sep 2016
Some notes/news about release planning:
Within stretch development, lots of changes in 1.0.x were needed to accomodate for new versions of packages in sid.
As this needed/needs quite some elaborate changes to the code anyway, I also broadened the view on the 1.0.x stable branch to allow more changes and even some (compatible) new features in.
I.e., I am currently trying to work down the open Debian Bugs plus some pet-peeves (you may find here in this roadmap) in 1.0.x, going for a point to actually merge this to master to finally start working on 1.2.x.
I am currently thinking that 1.2.x may actually not feature very much more than the python3 switch and https.
(1) 1.0.x Stable
Subject for patching up stable 1.0.x.
See also: https://bugs.debian.org/cgi-bin/pkgreport.cgi?repeatmerged=no&src=mini-buildd.
Known/possible issues atm:
- OK: Update project setup: src/ (PYTHONPATH).
- OK: [BUG ]: 'reset passwd' error 500. => [MYST] wheezy, only: p-d-r: password reset fails: Template engine can't find 'registration/password_reset_email.txt'.
- OK: [BUG ]: shm mount leftovers??
- OK: [BUG] When distribution is wrong (like for example squeeze-wheezy-experimental) mini-buildd keeps var/log/wheezy/_failed/... log cruft (forever it seems)
- OK: [BUG] chroot removal fails when no appr. archive exists
- OK: [FIXED] py-exception when uploading "jessie-repoid-unstable-experimental "
- PENDING (fix accepted by sbuild) [CHECK] sbuild >= 0.66: Does no longer work for non-multiarch distributions(?). Maybe related: https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=793561 - build log: Couldn't find any package whose name or description matched "sbuild-build-depends-weslok-archive-keyring-dummy:amd64"
- [BUG] (?) 'mini-buildd-tool -q' does not suppress server logs like "[hellfield] I: API call 'getsourceslist' by user 'AnonymousUser'"
- [BUG] WEB: Internal ports for *-snapshot offer *-experimental (and vice versa). (will always be rejected/same version).
- Check if there also could be better error checking in port/portext API call itself
- [BUG] Daemon change (for example: email regex): Needs start/stop to become active.
- [BUG] (unverified) Changing user upload permissions in admin config fails (?).
- [BUG] chroots: Sources used for bootstrapping stick. 'check' may fail when sources eventually become invalid.
- => Use/update with current source when doing chroot 'check'.
- [BUG] when sbuild fails in setup stage, sbuild idles around (?)
- [CHECK]: schroot 1.7
- [FEATURE]: HTML: Show python/django/cherrypy/pyftplib versions. (templetatags?)
- [CHECK]: Can we use more finegrainded PRIOS on archives (instead of NotAutomatic, ButAutomo...)??
- [DOC ]: Client API: Document API, article, more examples and tests.
- [FEAT]: chroot check: when 'apt-get update fails', retry after 'rm -rf /var/lib/apt/lists/' (?)
- [FEAT]: automated ports: Username2Changelog
- [FEAT]: python-mini-buildd should: Suggest or Recommend python-django,python-django-registration (python client API).
- [CHECK]: ftp/buildd process hangs, breaking ftp::
- # hellfield: /home/absurd [0] # root? ps aux | grep mini mini-bu+ 5481 0.0 0.0 10524 132 ? S Mai01 0:00 /build/util-linux-rIVWM5/util-linux-2.28/tailf /build/util-linux-rIVWM5/util-linux-2.28/tests/output/tailf/simple.input mini-bu+ 25047 0.0 0.0 2088 104 ? S Mai01 0:00 /build/util-linux-Wae87U/util-linux-2.28/tailf /build/util-linux-Wae87U/util-linux-2.28/tests/output/tailf/simple.input mini-bu+ 32185 0.0 0.0 36020 3760 ? Ss Mai01 0:01 /lib/systemd/systemd --user mini-bu+ 32188 0.0 0.0 207256 2868 ? S Mai01 0:00 (sd-pam) root 32526 0.0 0.0 14204 2424 pts/0 S+ 11:02 0:00 grep --color mini
- [FEAT]: config: Add "purge" action (bulk-delete instances) for chroots.
- [FEAT]: test packages: Why do we not build to experimental?
- [FEAT]: Integrate/add superportext.
- [FEATURE] API: getsourceslist:
- 'mini-buildd-tool vm154 getsourceslist --suite='(stable|stable-rollback0)' wheezy' should work; or new option 'with_rollbacks'.
- 'mini-buildd-tool vm154 getsourceslist --mirror=XXX
- [CHECK] What about port[ext] when there is no version restriction (debdev)?
- [CHECK] chroot check/updating: Should we use dist-upgrade (avoid some failures for sid/testing)?
- [FEATURE] 'Complete' per-upload config (More ways to influence build via changes magic lines):
- New flags brainstorm:
- OK: INTERNAL_APT_PRIORITY
- DISTRIBUTION_APT_PRIORITY... ?
- DEB_BUILDOPTIONS
(2) 1.1.x Development
(2.0) 1.2.0 Release Goals
1.2 will have secure data channels as only big new feature, and else focus on code stabilisation and cleanup:
- Support secure data channels (HTTPS, SFTP).
- Code migration to python3.
- Code cleanup and speed optimizations (python code, algos, calls, reprepro, ...).
- Improve documentation.
Be sure it does not:
- Change SQL scheme, i.e., no incompatible changes to django models.
- Change the inter-instance API (except for HTTPS/FTPS support if needed).
- Change the user API's (except for HTTPS/FTPS support if needed).
(2.1) 1.1.x Development Kickstart
This will most likely take place after the jessie Release; it will mark the point we might not be able any more to merge fixes back from stable.
- Debian packaging
- deb: +"mini-buildd-tool" package
- deb: -"mini-buildd-common" package
- Cleanup /usr/share/doc/examples/:
- python-mini-buildd vs mini-buildd
- examples versus actual package data
- lintian clean
- Code cleanups
- Harden package dependencies to jessie/wheezy+bpo versions to remove as much compat code as possible:
- Remove doc/apidoc.py
- django < 1.7
- ...
- Bulk-migrate code to python3
- Fixing up pylint:
- Fix/cleanup in-code exceptions
- [WISH] pylint: Move all disabled tests from pylintrc to source code
- Revisit logging setup
- All logging should go to mini-buildd's configure logging
- django logs to mini-buildd's (see reverted e89a653ec92be56d708f750f1ba2b2d152401e6b "Make django logging use mini-buildd's.", premature solution)
- pyftpdlib and cherrypy ?
- Get django deprecation warnings logged too (currently via './devel testdjango_X.Y' only).
(2.3) Patches
New add-on like features, w/o need to change much of the infrastructure. Not all may be actually picked up for 1.2:
- [WISH]: reprepro: Support "Content" files (is it as easy as: http://rowinggolfer.blogspot.de/2012/07/configuring-reprepro-for-apt-file.html ?).
- [FISH]: ~bpo8+1 instead ~bpo80+1: If jessie-backports uses this, update default in mbd too
- [FISH]: feature/better_buildstatus Show build status and lintian status directly, with colors: i386(BL) amd64(BL)
- [WISH] API(status): Add archive_id; add suite names (=> client_api)
- [WISH] API(package_status): Status for package+version: building, installed, failed (for tools polling for build results).
- [WISH] Integrate ssh upload (using mini-buildd user?) (currently in examples only)
- [WISH] Include "super-portext" example.
- [WISH] Config option "mirrors"
- API call 'mirrors'.
- Bundle with API call getsourceslist (others?).
- keyring package: sources.list library
- [WISH] Status: Add remove/migrate/?? to last packages
- [CHECK] http://tools.ietf.org/html/rfc5147
- Link to parts of the build log via fragment to text/plain
- [WISH] Make passive ftp port range configurable.
- [WISH] keyring: Auto-migrate keyring packages.
- [WISH] Drop sbuild keys (sbuild key workaround in misc.py) (See https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=834330: Only needed for <wheezy).
- [WISH] Library of predefined sbuild --chroot-setup-command snippets (eatmydata, ccache)
- [WISH] rm libeatmydata as default (?).
- [WISH] Web API should work with POST; we should use POST by default.
- [WISH] admin/model objects: Show "used by count" (so we can see what objects may be obsolete).
- [WISH] Archive: Rethink Archive::mbd_ping (see https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=740576)
- [WISH] BTRFS chroot backend
- [FISH] API: port/portext: --extra-changelog-entry
- [FISH] Make use of python3-distro-info
- [WISH] Better Daemon defaults for
- Build queue size
- Sbuild jobs: Is it safe to default to -j auto or -j N with N > 1 ???
- debhelper-based packages only actually use this if enabled explicitely ('dh --parallel')
- ?? Any cases where this could cause parallel builds where not wanted?
(3) Further Development Brainstorm
- [WISH]: Auto ports and/or Port-Alarm on Debian Package Updates
- SQL schema migration (with django 1.7)
- PYTHON API for mini-buildd API calls.
- piuparts check functionality (sbuild?)
- Sources/Archives: Support partial arch archives (?)
- Better 'user role concept', better keyring management workflow:
- Upload-Admin:
- Liste der Uploader, die auf eine Upload-Queue zugreifen koennen
- Uploader-Requests ("Anklopfen"):
- Uploader: "Moechte gerne auf Upload-Queues X, Y und Z uploaden"
- Queue-Admin von Queue X bekommt Notification: "Uploader A will auf X zugreifen"
- Queue-Admin kann Keys fuer Queue X listen: accepted + unaccepted keys
- Siehe auch: http://intothesaltmine.org/blog/html/pages/key_management.html
- notify: Admin notify on deactivated daemon.
- ->Daemon.notify for admin messages, repo.notify for packages.
- ->Daemon.notify for email, rest as before