RoadMap (Raw Development TODO) Feb 2017

mini-buildd RoadMap

A.k.a. raw development todo.

(0) MOTM: Feb 2017

Some current notes/news about release planning:

I finally decided 1.0.29 to be sufficiently good to kick off 1.1.x development from there, and 1.0.x now again strictly for bugs only.

I will (hopefully frequently) upload to experimental to deliver current 1.1.x development carnage.

(1) 1.0.x Stable

Subject for patching up stable 1.0.x.

Nothing scheduled atm.

Potential bugs:

(2) 1.1.x Development

(2.0) 1.2.0 Release Goals

1.2 will have python3 migration and secure data channels as big new feature, and else focus on code stabilisation and cleanup:

  • Code migration to python3.
  • Support secure data channels (HTTPS, SFTP).
  • Code cleanup and speed optimizations (python code, algos, calls, reprepro, ...).
  • Improve documentation.

Be sure it does not:

  • Change SQL scheme, i.e., no incompatible changes to django models.
  • Change the inter-instance API (except for HTTPS/FTPS support if needed).
  • Change the user API's (except for HTTPS/FTPS support if needed).

(2.1) 1.1.x Development Kickstart Todos

Bulk updates and cleanups
  • Python 3 Initials
    • Bulk-update deps
    • Bulk-update code
  • Code cleanups
    • pylint: Bulk-Fix/cleanup in-code exceptions
    • pylint: Move all exceptions from pylintrc to source code
Debian packaging changes
  • deb: +"mini-buildd-tool" package
  • deb: -"mini-buildd-common" package
  • Cleanup /usr/share/doc/examples/:
    • python-mini-buildd vs mini-buildd
    • examples versus actual package data
  • lintian clean

(2.2) Issues

New add-on like features, w/o need to change much of the infrastructure. Not all may be actually picked up for 1.2:

  • [WISH]: Timeout when checking sources (?)
  • [WISH]: Refactor [1]
  • [WISH]: reprepro: Support "Content" files (is it as easy as: ?).
  • [FISH]: ~bpo8+1 instead ~bpo80+1: If jessie-backports uses this, update default in mbd too
  • [WISH] API(status): Add archive_id; add suite names (=> client_api)
  • [WISH] API(package_status): Status for package+version: building, installed, failed (for tools polling for build results).
  • [WISH] Integrate ssh upload (using mini-buildd user?) (currently in examples only)
  • [WISH] Config option "mirrors"
    • API call 'mirrors'.
    • Bundle with API call getsourceslist (others?).
    • keyring package: sources.list library
  • [WISH] Status: Add remove/migrate/?? to last packages
  • [CHECK]
    • Link to parts of the build log via fragment to text/plain
  • [WISH] Make passive ftp port range configurable.
  • [WISH] keyring: Auto-migrate keyring packages.
  • [WISH] Drop sbuild keys (sbuild key workaround in (See man sources.list =>[trusted=yes]: Only needed for <wheezy).
  • [WISH] Library of predefined sbuild --chroot-setup-command snippets (eatmydata, ccache)
  • [WISH] rm libeatmydata as default (?).
  • [WISH] Web API should work with POST; we should use POST by default.
  • [WISH] admin/model objects: Show "used by count" (so we can see what objects may be obsolete).
  • [WISH] Archive: Rethink Archive::mbd_ping (see
  • [FISH] Make use of python3-distro-info
  • [WISH] Better Daemon defaults for
    • Build queue size
    • Sbuild jobs: Is it safe to default to -j auto or -j N with N > 1 ???
      • debhelper-based packages only actually use this if enabled explicitely ('dh --parallel')
      • ?? Any cases where this could cause parallel builds where not wanted?
  • [FEAT]: Integrate/add superportext.
  • [HTML]: API: redirect to login and back on perm error.
  • [FEAT]: daemon check(): Add overall warnings and checks
    • Old schroot workaround link in /etc/schroot/..
    • cruft in logs/
  • [FEAT]: Implement "wake remote" support.
  • [FEAT]: config: Add "purge" action (bulk-delete instances) for chroots.
  • [FEATURE] API: getsourceslist:
    • 'mini-buildd-tool vm154 getsourceslist --suite='(stable|stable-rollback0)' wheezy' should work; or new option 'with_rollbacks'.
    • 'mini-buildd-tool vm154 getsourceslist --mirror=XXX
  • [CHECK]: Can we use more finegrainded PRIOS on archives (instead of NotAutomatic, ButAutomo...)??
  • [BUG] schroot: still more mount leftovers (=> hellfield)
  • [BUG] Daemon change (for example: email regex), repository uploaders change, ...: Needs start/stop to become active. -
  • [BUG] (unverified) Changing user upload permissions in admin config fails (?).
  • [BUG] when sbuild fails in setup stage, sbuild idles around (?)
  • [BUG] Intermediate unpredicatble state of daemon instance:
    After Daemon instance creation and before any initial _update_from_model() call, the singleton daemon instance is in an unexpected state for some code (value set to None in constructor). This may lead to confusing errors. Reproduce: './devel update' with the sleep set to 1 seconds.
  • [CHECK]: ftp/buildd process hangs, breaking ftp::
    # hellfield: /home/absurd [0] # root? ps aux | grep mini mini-bu+ 5481 0.0 0.0 10524 132 ? S Mai01 0:00 /build/util-linux-rIVWM5/util-linux-2.28/tailf /build/util-linux-rIVWM5/util-linux-2.28/tests/output/tailf/simple.input mini-bu+ 25047 0.0 0.0 2088 104 ? S Mai01 0:00 /build/util-linux-Wae87U/util-linux-2.28/tailf /build/util-linux-Wae87U/util-linux-2.28/tests/output/tailf/simple.input mini-bu+ 32185 0.0 0.0 36020 3760 ? Ss Mai01 0:01 /lib/systemd/systemd --user mini-bu+ 32188 0.0 0.0 207256 2868 ? S Mai01 0:00 (sd-pam) root 32526 0.0 0.0 14204 2424 pts/0 S+ 11:02 0:00 grep --color mini
  • [BUG] Failing start of ftpd not detected:
    socket -sl 8067 start => daemon start should fail; instead ftpd failure this is only logged.

(3) Further Development Brainstorm

  • [WISH]: Auto ports and/or Port-Alarm on Debian Package Updates
  • SQL schema migration (with django 1.7)
  • PYTHON API for mini-buildd API calls.
  • piuparts check functionality (sbuild?)
  • Sources/Archives: Support partial arch archives (?)
  • Better 'user role concept', better keyring management workflow:
    • Upload-Admin:
      • Liste der Uploader, die auf eine Upload-Queue zugreifen koennen
      • Uploader-Requests ("Anklopfen"):
  • notify: Admin notify on deactivated daemon.
    • ->Daemon.notify for admin messages, repo.notify for packages.
    • ->Daemon.notify for email, rest as before

(4) Annotations

[1]Dev note, for 1.2.x: This should be refactured like so.

Like so: