Raw Development TODO Sep 2016

mini-buildd Development TODO

(0) MOTM: Sep 2016

Some notes/news about release planning:

Within stretch development, lots of changes in 1.0.x were needed to accomodate for new versions of packages in sid.

As this needed/needs quite some elaborate changes to the code anyway, I also broadened the view on the 1.0.x stable branch to allow more changes and even some (compatible) new features in.

I.e., I am currently trying to work down the open Debian Bugs plus some pet-peeves (you may find here in this roadmap) in 1.0.x, going for a point to actually merge this to master to finally start working on 1.2.x.

I am currently thinking that 1.2.x may actually not feature very much more than the python3 switch and https.

(1) 1.0.x Stable

Subject for patching up stable 1.0.x.

See also: https://bugs.debian.org/cgi-bin/pkgreport.cgi?repeatmerged=no&src=mini-buildd.

Known/possible issues atm:

  • OK: Update project setup: src/ (PYTHONPATH).
  • OK: [BUG ]: 'reset passwd' error 500. => [MYST] wheezy, only: p-d-r: password reset fails: Template engine can't find 'registration/password_reset_email.txt'.
  • OK: [BUG ]: shm mount leftovers??
  • OK: [BUG] When distribution is wrong (like for example squeeze-wheezy-experimental) mini-buildd keeps var/log/wheezy/_failed/... log cruft (forever it seems)
  • OK: [BUG] chroot removal fails when no appr. archive exists
  • OK: [FIXED] py-exception when uploading "jessie-repoid-unstable-experimental "
  • PENDING (fix accepted by sbuild) [CHECK] sbuild >= 0.66: Does no longer work for non-multiarch distributions(?). Maybe related: https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=793561 - build log: Couldn't find any package whose name or description matched "sbuild-build-depends-weslok-archive-keyring-dummy:amd64"
  • [BUG] (?) 'mini-buildd-tool -q' does not suppress server logs like "[hellfield] I: API call 'getsourceslist' by user 'AnonymousUser'"
  • [BUG] WEB: Internal ports for *-snapshot offer *-experimental (and vice versa). (will always be rejected/same version).
    • Check if there also could be better error checking in port/portext API call itself
  • [BUG] Daemon change (for example: email regex): Needs start/stop to become active.
  • [BUG] (unverified) Changing user upload permissions in admin config fails (?).
  • [BUG] chroots: Sources used for bootstrapping stick. 'check' may fail when sources eventually become invalid.
    • => Use/update with current source when doing chroot 'check'.
  • [BUG] when sbuild fails in setup stage, sbuild idles around (?)
  • [CHECK]: schroot 1.7
  • [FEATURE]: HTML: Show python/django/cherrypy/pyftplib versions. (templetatags?)
  • [CHECK]: Can we use more finegrainded PRIOS on archives (instead of NotAutomatic, ButAutomo...)??
  • [DOC ]: Client API: Document API, article, more examples and tests.
  • [FEAT]: chroot check: when 'apt-get update fails', retry after 'rm -rf /var/lib/apt/lists/' (?)
  • [FEAT]: automated ports: Username2Changelog
  • [FEAT]: python-mini-buildd should: Suggest or Recommend python-django,python-django-registration (python client API).
  • [CHECK]: ftp/buildd process hangs, breaking ftp::
    # hellfield: /home/absurd [0] # root? ps aux | grep mini mini-bu+ 5481 0.0 0.0 10524 132 ? S Mai01 0:00 /build/util-linux-rIVWM5/util-linux-2.28/tailf /build/util-linux-rIVWM5/util-linux-2.28/tests/output/tailf/simple.input mini-bu+ 25047 0.0 0.0 2088 104 ? S Mai01 0:00 /build/util-linux-Wae87U/util-linux-2.28/tailf /build/util-linux-Wae87U/util-linux-2.28/tests/output/tailf/simple.input mini-bu+ 32185 0.0 0.0 36020 3760 ? Ss Mai01 0:01 /lib/systemd/systemd --user mini-bu+ 32188 0.0 0.0 207256 2868 ? S Mai01 0:00 (sd-pam) root 32526 0.0 0.0 14204 2424 pts/0 S+ 11:02 0:00 grep --color mini
  • [FEAT]: config: Add "purge" action (bulk-delete instances) for chroots.
  • [FEAT]: test packages: Why do we not build to experimental?
  • [FEAT]: Integrate/add superportext.
  • [FEATURE] API: getsourceslist:
    • 'mini-buildd-tool vm154 getsourceslist --suite='(stable|stable-rollback0)' wheezy' should work; or new option 'with_rollbacks'.
    • 'mini-buildd-tool vm154 getsourceslist --mirror=XXX
  • [CHECK] What about port[ext] when there is no version restriction (debdev)?
  • [CHECK] chroot check/updating: Should we use dist-upgrade (avoid some failures for sid/testing)?
  • [FEATURE] 'Complete' per-upload config (More ways to influence build via changes magic lines):
    • New flags brainstorm:
      • OK: INTERNAL_APT_PRIORITY
      • DISTRIBUTION_APT_PRIORITY... ?
      • DEB_BUILDOPTIONS

(2) 1.1.x Development

(2.0) 1.2.0 Release Goals

1.2 will have secure data channels as only big new feature, and else focus on code stabilisation and cleanup:

  • Support secure data channels (HTTPS, SFTP).
  • Code migration to python3.
  • Code cleanup and speed optimizations (python code, algos, calls, reprepro, ...).
  • Improve documentation.

Be sure it does not:

  • Change SQL scheme, i.e., no incompatible changes to django models.
  • Change the inter-instance API (except for HTTPS/FTPS support if needed).
  • Change the user API's (except for HTTPS/FTPS support if needed).

(2.1) 1.1.x Development Kickstart

This will most likely take place after the jessie Release; it will mark the point we might not be able any more to merge fixes back from stable.

  • Debian packaging
    • deb: +"mini-buildd-tool" package
    • deb: -"mini-buildd-common" package
    • Cleanup /usr/share/doc/examples/:
      • python-mini-buildd vs mini-buildd
      • examples versus actual package data
    • lintian clean
  • Code cleanups
    • Harden package dependencies to jessie/wheezy+bpo versions to remove as much compat code as possible:
      • Remove doc/apidoc.py
      • django < 1.7
      • ...
    • Bulk-migrate code to python3
    • Fixing up pylint:
      • Fix/cleanup in-code exceptions
      • [WISH] pylint: Move all disabled tests from pylintrc to source code
  • Revisit logging setup
    • All logging should go to mini-buildd's configure logging
      • django logs to mini-buildd's (see reverted e89a653ec92be56d708f750f1ba2b2d152401e6b "Make django logging use mini-buildd's.", premature solution)
      • pyftpdlib and cherrypy ?
    • Get django deprecation warnings logged too (currently via './devel testdjango_X.Y' only).

(2.3) Patches

New add-on like features, w/o need to change much of the infrastructure. Not all may be actually picked up for 1.2:

  • [WISH]: reprepro: Support "Content" files (is it as easy as: http://rowinggolfer.blogspot.de/2012/07/configuring-reprepro-for-apt-file.html ?).
  • [FISH]: ~bpo8+1 instead ~bpo80+1: If jessie-backports uses this, update default in mbd too
  • [FISH]: feature/better_buildstatus Show build status and lintian status directly, with colors: i386(BL) amd64(BL)
  • [WISH] API(status): Add archive_id; add suite names (=> client_api)
  • [WISH] API(package_status): Status for package+version: building, installed, failed (for tools polling for build results).
  • [WISH] Integrate ssh upload (using mini-buildd user?) (currently in examples only)
  • [WISH] Include "super-portext" example.
  • [WISH] Config option "mirrors"
    • API call 'mirrors'.
    • Bundle with API call getsourceslist (others?).
    • keyring package: sources.list library
  • [WISH] Status: Add remove/migrate/?? to last packages
  • [CHECK] http://tools.ietf.org/html/rfc5147
    • Link to parts of the build log via fragment to text/plain
  • [WISH] Make passive ftp port range configurable.
  • [WISH] keyring: Auto-migrate keyring packages.
  • [WISH] Drop sbuild keys (sbuild key workaround in misc.py) (See https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=834330: Only needed for <wheezy).
  • [WISH] Library of predefined sbuild --chroot-setup-command snippets (eatmydata, ccache)
  • [WISH] rm libeatmydata as default (?).
  • [WISH] Web API should work with POST; we should use POST by default.
  • [WISH] admin/model objects: Show "used by count" (so we can see what objects may be obsolete).
  • [WISH] Archive: Rethink Archive::mbd_ping (see https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=740576)
  • [WISH] BTRFS chroot backend
  • [FISH] API: port/portext: --extra-changelog-entry
  • [FISH] Make use of python3-distro-info
  • [WISH] Better Daemon defaults for
    • Build queue size
    • Sbuild jobs: Is it safe to default to -j auto or -j N with N > 1 ???
      • debhelper-based packages only actually use this if enabled explicitely ('dh --parallel')
      • ?? Any cases where this could cause parallel builds where not wanted?

(3) Further Development Brainstorm

  • [WISH]: Auto ports and/or Port-Alarm on Debian Package Updates
  • SQL schema migration (with django 1.7)
  • PYTHON API for mini-buildd API calls.
  • piuparts check functionality (sbuild?)
  • Sources/Archives: Support partial arch archives (?)
  • Better 'user role concept', better keyring management workflow:
    • Upload-Admin:
      • Liste der Uploader, die auf eine Upload-Queue zugreifen koennen
      • Uploader-Requests ("Anklopfen"):
  • notify: Admin notify on deactivated daemon.
    • ->Daemon.notify for admin messages, repo.notify for packages.
    • ->Daemon.notify for email, rest as before