Roadmap ####### A more current version of this document (regarding the *next* roadmap) may be available on `http://mini-buildd.installiert.net `_. 2.6.x (next) ************ Upgrading from 2.4.x ==================== Brainstorm ========== Roughly, volatility of these items increases the lower they are placed. * reprepro: Handle multiple versions per distribution gracefully (this is already possible <5.4 w/ the 'components corner case') * reprepro 5.4 (*needs to move out of experimental 1st*): Allow multiple versions per distribution; eventually deprecates convoluted *rollback support* * reprepro 5.4 (*needs to move out of experimental 1st*): Drop ddeb exceptions ('Deb-Build-Options: noddebs', Ubuntu) * APT key rollover (doc or automated) * Support for deb822 sources * Merge test packages from API call 'test_packages' && testsuite * django: Integrate CAS/SSO-like features (python3-django-casclient?) * Watcher support: Autoport and/or alarm on Debian Package Updates * Make piuparts actually work (non-EXPERIMENTAL) * gnupg: Switch completely to 'python3-gpg' (*NOT* 'python3-gpgme') * Archive: Rethink Archive::mbd_ping (see https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=740576) * Release: Alternatives for NotAutomatic, ButAutomo...)?? Can we use more finegrained PRIOS on archives? * sid-test-wildwest: Add extra suite for arbitrary package (deb+src) installs? * Dependency check on package migration * Replace ftp by twisted/ssh? * Switch to jinja2 templates? * Conform to DEP-14 (git branch layout)? * Implicit deletion of a *Chroot* instance may leave assosicated data (chroot files) behind (i.e., when deleting resp. *Source*). See ``save|delete_model()`` in models.py -- this code is supposed to do that. 2.4.x ***** * Tighten software dependencies (python, twisted, reprepro,..) - in short, this now needs at least *Debian trixie* to run. * Web UI has become more user friendly and can now be used on mobile devices as well. * Add more user friendly "search" API call (allows for patterns and also finds binary package names). On the web UI, "search" replaces "ls" as the main search tool. * A lot of minor changes, fixes and improvements (see full Debian changelog). Upgrading from 2.2.x ==================== * :apicall:`show`: In (json) result, ``changes_paths`` is replaced by ``event_path`` and ``changes_path`` (f306e1b4). Only relevant should you have custom tool(s) parsing/interpreting the output. * **Distribution/lintian**: Now uses ``lintian_extra_options`` and *new* ``lintian_warnfail_options`` fields exclusively for lintian arguments (previously, values from setup depending on codename would be mixed in at run-time). This means full control per *Distribution* for the administrator now. Per-codename default values are now only preset for *new Distributions* via :apicall:`setup`. If you had customized ``lintian_extra_options``, you may want to recheck your lintian settings; otherwise, nothing needs to be done. As always, *inspect* may be used to show if your settings diverge from setup's default. * :apicall:`dput_conf`: By default now also outputs "recommendend global config". Should you really need the old (target only) bevahiour, use ``--without-global``. * :apicall:`reindex`: *NEW* Trigger reprepro reindex even if config has not changed. Maybe worthwhile to manually run once after upgrade -- check for details in :apicall:`reindex`. * :apicall:`search`: *NEW* Simpler package search (pattern matching and search in binary package names). On the web app, "search" replaces "ls" as the main search tool. * "Home" and "Events" on the web app are now the same and now actually only show the current internal events queue. I.e., you can no longer select older or all events here. Use :apicall:`search` to get a package events URI listing if you need to see specific/older events. Noteworthy external Bugs ======================== * libssl, python-cryptography, CRYPTOGRAPHY_OPENSSL_NO_LEGACY=1: * https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=965041 * https://github.com/pyca/cryptography/issues/11450 2.2.x ***** * First release to **support django SQL schema migration** (for app ``mini_buildd``), including initial model cleanup. * Crucial **Web UI improvements** (speed, usability, look&feel). Thanks Gerhard! * **Drops all <= bullseye compatibility support** -- i.e., installable on *bookworm* or newer only (needs ``python 3.11`` and ``django 3.2``). * Logging improvements && doubling down on **new style daemon for logging** (drops explicit log file ``daemon.log``). * Adds tool ``mini-buildd-cruft`` to help detect/remove deprecated files from older versions. * Many other micro improvements and fixes from a ~year-long development phase (see changelog). Upgrading from 2.0.x ==================== * Deprecated ``-v,--verbose,-q,--quiet`` option is **removed for good** (use ``--log-level``) * Deprecated ``mini-buildd -W,--httpd-bind`` option is **removed for good** (use use ``--http-endpoint``) You may also directly upgrade from 1.0.x -- in that case, all notes on :ref:`roadmap:Upgrading from 1.0.x` below also apply. 2.0.x ***** * New deb package layout: mini-buildd (the service), mini-buildd-utils (for users), python3-mini-buildd, mini-buildd-doc * Code migration to ``python 3`` * Update to ``django 2.2`` (minimal version) * Switch to ``twisted`` as http backend * Support secure data channels (HTTPS && SFTP) * API upgrade to json * Event message queue * Periodic task scheduler (cron, internal use) * Automated repository snasphots (weekly, monthly, yearly) * WEB UI rewrite * Simplified maintenance (via new API call ``setup``) * 'upgrade-compatible' (albeit **not** run-time-compatible) with 1.0.x (especially, there is no SQL schema change) * Installable on bookworm, bullseye and buster Upgrading from 1.0.x ==================== Package upgrade --------------- Your ``1.0.x`` installation will typically run on a ``buster``-based system. You may simply ``dist-upgrade`` / ``full-upgrade`` that system (1st to ``bullseye``, then to ``bookworm``) -- just make sure the ``mini-buildd`` package is **never purged** during that process. When done, you will need these extra APT sources: * ``bookworm``: ``Debian Backports (bookworm-backports)`` * ``bullseye``: ``Hellfield Archive (bullseye-ab-stable)`` Alternatively: buster-based upgrade ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ For convenience, it's still possible to run ``2.0.x`` under ``buster`` with some tweaks: Extra APT sources needed: * ``Debian Backports`` (buster-backports) * ``Hellfield Archive`` (buster-ab-stable) To upgrade, ``aptitude`` is imho the easiest way. Select the suggestion that de-installs py2 variants, and upgrades all the rest:: aptitude install mini-buildd/buster-ab-stable .. attention:: **buster-based install**: 500 Internal Error: ``[Errno 2] No such file or directory: 'S.gpg-agent.extra'`` We see this error seemingly random on buster installations (w/ ``2.2.27-1~bpo10+1``) only so far. * Removing a very old left-over ``~/.gnupg/gpg.conf`` seem to have helped so far on an actual host * In test suite / chroot, it usually pops up with profile '_all' * Never yet seen with default (no) profile or '_debian' Post-upgrade checklist ---------------------- Administrator ~~~~~~~~~~~~~ * ``mini-buildd`` command line arguments (``dpkg-reconfigure``) -- see :mbdcommand:`mini-buildd` * ``--httpd-bind`` is now deprecated in favor of ``--http-endpoint`` * ``--verbose``, ``--quiet`` are now deprecated in favor of ``--log-level``. Also, giving *none* of these options now yields the recommended production log level (before, ``--verbose`` was default) * ``--debug`` now has some options removed (see man page). In general, never use it for production, ``--log-level=DEBUG`` should be good enough * Decide if you want switch to ``https/ftps``: See :ref:`administrator:Running encrypted` * Inspect/extend your setup, see :mbdpage:`setup`. Especially, check for these "model diff"s (which must be fixed manually): * ``Daemon.ftpd_bind``: Deprecates 'non-endpoint' notation * ``Daemon.gnupg_keyserver``: Please change to traditional keyserver (like ``keyserver.ubuntu.com``) to retain functionality * ``Distribution.sbuildrc_snippet`` and ``Distribution.schroot_setup_script``. These most likely have legacy support code for ccache and eatmydata, which is now supported by builtin ``ccache`` and ``eatmydata`` -- see :ref:`administrator:Distribution` * Remake all chroots -- see :apicall:`remake_chroots`, :mbdpage:`builders` * Rebuild all keyring packages (so they no longer use deprecated ``apt-key``) -- see :apicall:`keyring_packages` * In case you hate the autogenerated complicated "Origin" text, you may now customize it -- see DB field ``Daemon.extra_options`` * In case you have a custom SSH upload setup (i.e., similar to what has been suggested in the 1.0.x manual): * You may need to sync your setup to what new helper :mbdcommand:`mini-buildd-ssh-setup` would set up (the script can setup new, but not upgrade -- some 'educated digging' is necessary I fear) * You can now add a custom dput snippet for convenience -- see :apicall:`dput_conf` * File ``~/.mini-buildd.pid`` is no longer used, and may be removed (unless you use a custom setup using the old initv start/stop script) Consumer ~~~~~~~~ * Keyring Package: The path to the convenience sources.list files has changed. If you happen to link these, you will need to update: * All support files are now under ``/usr/share/-archive-keyring`` (to be able to cleanly support installing multiple keyring packages) 1.0.x ***** .. |Hellfield| replace:: The Hellfield Archive .. _Hellfield: http://mini-buildd.installiert.net/pages/the-hellfield-archive.html Installation ============ Debian *stretch* ---------------- Add *Debian Backports* to your sources.list:: deb http://httpredir.debian.org/debian/ stretch-backports main contrib non-free Aptitude will serve you better to interactively resolve dependencies from backports. Go with the resolution that actually selects mini-buildd from *stretch-backports* (``bpo9``): .. code:: bash aptitude update aptitude install mini-buildd/stretch-backports # Server instance aptitude install python-mini-buildd/stretch-backports # Client support only Debian *jessie* --------------- Add *Debian Backports* to your sources.list:: deb http://httpredir.debian.org/debian/ jessie-backports main contrib non-free Aptitude will serve you better to interactively resolve dependencies from backports. Go with the resolution that actually selects mini-buildd from *jessie-backports* (``bpo8``): .. code:: bash aptitude update aptitude install mini-buildd/jessie-backports # Server instance aptitude install python-mini-buildd/jessie-backports # Client support only Debian *wheezy* --------------- Add *Debian Backports* and |Hellfield|_'s *stable suite* to your sources.list:: deb http://httpredir.debian.org/debian/ wheezy-backports main contrib non-free deb http://debian.installiert.net/hellfield/ab/ wheezy-ab-stable main contrib non-free Aptitude will serve you better to interactively resolve dependencies from *Debian Backports* and |Hellfield|_; also, *Recommended* dependencies are likely to produce conflicting package installs on this setup. Go with the resolution that actually selects mini-buildd from *wheezy-ab-stable* (``ab70``): .. code:: bash aptitude update aptitude --without-recommends install mini-buildd/wheezy-ab-stable # Server instance aptitude --without-recommends install python-mini-buildd/wheezy-ab-stable # Client support only Others ------ **Note that you are on your own here; this might or might not work for your target at hand, but you may try anyway ;).** First, check |Hellfield|_ if your target distribution is available at all. Then, for whatever distribution *CODENAME* you are using, add *stable + unstable + snapshot* from |Hellfield|_:: deb http://debian.installiert.net/hellfield/ab/ CODENAME-ab-stable main contrib non-free deb http://debian.installiert.net/hellfield/ab/ CODENAME-ab-unstable main contrib non-free deb http://debian.installiert.net/hellfield/ab/ CODENAME-ab-snasphot main contrib non-free It will also usually be a good idea to add any *backport fashion* source for your target if any such thing is available. Then try your luck with aptitude: .. code:: bash aptitude update aptitude --without-recommends install mini-buildd/CODENAME-ab-snapshot # Server instance aptitude --without-recommends install python-mini-buildd/CODENAME-ab-snapshot # Client support only Migrate packages from 0.8.x =========================== .. tip:: Can I skip migrating from 0.8.x? Yes -- a much simpler solution might be to just copy the old repository directory (``~/rep``) somewhere and serve it via some standard web server, and just continue to use it along with your new repo as long as needed. This roughly explains the steps needed to upgrade a mini-buildd 0.8.x installation to 1.0.x with **transferring the packages from the old 0.8.x repositories over**, so you can continue with the new 1.0.x repos only: 1. Upgrade mini-buildd from 0.8.x to 1.0. Chances are this might have already implicitly happened, with some update. You will then have 1.0 up and running, and ye olde 0.8.x repositories still available as read-only apt repositories. Just be sure you don't **purge** the old package, and then install 1.0, as this will remove the whole old repository. 2. Configure mini-buildd 1.0. This means you should, in the end, have a 1.0 repository with the **same identity** as the old 0.8.x repository, and with all distributions you want to migrate. 3. Import packages. Become mini-buildd user, and got to the new 1.0 reprepro repository you want to import to, and use the importer script to migrate packages: .. code:: bash :class: mini-buildd-code cd repositories/REPOID mini-buildd-import-08x ~/rep/squeeze-REPOID-experimental squeeze-REPOID-experimental mini-buildd-import-08x ~/rep/squeeze-REPOID squeeze-REPOID-unstable This example is for squeeze; repeat the imports for all base distributions you want to migrate. Thusly, ye olde ``*-experimental`` distribution will be migrated to the distribution with the same name in 1.0. Ye olde ``squeeze-REPOID`` goes to ``squeeze-REPOID-unstable``. For the latter, multiple package version will be automatically installed to the new *rollback distributions* (which are needed with reprepro to support multiple package versions). 4. (Optional) Fix up package status. All the migrated packages are now in 1.0 "unstable" distribution; you may think of bulk-migrating them all to "stable", if that were your semantics for the 0.8.x non-experimental distributions. Eventually, when everything is updated, you may of course (re)move the old 0.8.x directory ``~/rep/``. .. seealso:: :debbug:`790292` 0.8.x (deprecated) ****************** Vintage *0.8.x* upgrade support is available through the |Hellfield|_'s *ablg* (last generation) repository for *lenny*, *squeeze* and *wheezy* with one of these lines:: deb http://debian.installiert.net/hellfield/ablg/ lenny-ablg-stable main contrib non-free deb http://debian.installiert.net/hellfield/ablg/ squeeze-ablg-stable main contrib non-free deb http://debian.installiert.net/hellfield/ablg/ wheezy-ablg-stable main contrib non-free