mini_buildd.sbuild module

Attention

sbuild (retry-workaround in gnupg.py): Spurious build failures can't connect to the agent: IPC connect call failed (Debian Bug #849551)

Attention

sbuild: Fails to build “all” packages with “build archall” flag set to arch “x” in case DSP has >= 1 arch “all” and >=1 arch “y” binary package (Debian Bug #706086)

This is due to sbuild and in in more detail explained in the Debian bug report.

A bad one-package workaround would be to change the “build archall” flag to arch “y”.

Attention

sbuild (workaround in apt-update snippet): apt-get update “File has unexpected size … Mirror sync in progress?” failures

This may happen if by chance there are reprepro updates around the same time apt-get update runs in the build.

You may reproduce this quite reliably with the testsuite (it’s due to the immediate parent package remove, see ./devel):

  1. testsuite: Change installed-port test to (also) port for running codename (changelog + ./devel)

  2. ./devel updatetestall

  3. ./devel testsuite-packages installed-port

Internal sbuild snippet works around this problem by retrying apt-get update – but you still see the error lines in the buildlog if that happens.

Solution could be to lock HTTP delivery on any reprepro change, but for now this seems overkill to me.

Attention

sbuild+apt-cacher-ng (workaround in apt-update snippet): apt-get update stalls on parallel access

Phenomena: A build just always remains ongoing && the buildlog just stalls during apt-get update.

Repeat bug:

/usr/share/doc/mini-buildd/examples/apt-parallel-bug
#!/bin/bash -e
# Precondition: Have ``apt-cacher-ng`` installed, default config

# Phaenomena description: Running this (on a fairly fast and parallel host) yields to ``apt-get update`` to stall
#
# If that happens, ``ps aux | grep method`` will then show existing/standing ``/usr/lib/apt/methods/http``, ``/usr/lib/apt/methods/gpgv`` processes
# Killing any of these will make apt continue (with an error)

# Note: Acquire::http::Pipeline-Depth and Acquire::QueueMode don't seem to change anything
# Note: Using ``parallel --jobs=1`` remedies the bug

read -p "Will create chroots in $(pwd) and do other dirty sudo thing (Ctl-C to abort)"

# At least two working apt lines are needed to trigger bug
APT_SOURCES_LIST="\
deb [] http://localhost:3142/debian sid main contrib non-free non-free-firmware
deb [] http://localhost:3142/debian experimental main contrib non-free non-free-firmware
"

CHROOTS="sid0 sid1"

for C in ${CHROOTS}; do
	[ -e "${C}" ] || sudo debootstrap sid "${C}" http://localhost:3142/debian/

	# Configure sources.list for local apt-cacher-ng with at least two workng lines
	printf "%s" "${APT_SOURCES_LIST}" | sudo tee ./${C}/etc/apt/sources.list

	# Remove any already download source.list files, so proper HTTP download is triggered
	sudo rm -rf "./${C}/var/lib/apt/lists/"
done

# Run apt-get in parallel
seq 0 1 | parallel sudo chroot ./sid{}/ timeout --verbose 10s apt-get update

Remedy situation at hand:

  1. Kill corresponding process /usr/lib/apt/methods/gpgv or /usr/lib/apt/methods/http (apt-get update may be retried and build continues)

  2. Cancel resp. build(s) (then retry)

Related 3rd party findings:

mini_buildd.sbuild.CONFIG_DIR = '.config'

Build dir constants.

mini_buildd.sbuild.APT_GET = 'apt-get --quiet --yes --option=APT::Install-Recommends=false --option=Acquire::Languages=none --option=Dpkg::Use-Pty=false --option=Dpkg::Options::=--force-confdef --option=Dpkg::Options::=--force-confnew'

Quiet, non-interactive, least invasive and loggable apt-get call (Dpkg::Use-Pty=false is to avoid https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=539617).

mini_buildd.sbuild.BASH_VARS = 'MBD_SCRIPT="$(realpath "${0}")"  # Note: Older versions of realpath (<=wheezy) did not support options, just avoid them\nMBD_BASE="${MBD_SCRIPT%/*/*}"'

Safe canonized paths

class mini_buildd.sbuild.Blocks(type_)

Bases: Dir

usage()
validate(value)

Validate value from user space (string, space separated), return value as list

line(value)
class mini_buildd.sbuild.ConfigBlocks

Bases: Blocks

configure(file_, blocks)

Add configured blocks to provided file

class mini_buildd.sbuild.SetupBlocks

Bases: Blocks

configure(dir_, blocks)

Add configured blocks to provided dir

class mini_buildd.sbuild.SBuild(breq, extra_args)

Bases: object

BUILDLOG_STATUS_REGEX = re.compile('^(Status|Lintian|Piuparts|Autopkgtest|Build-Time|Package-Time|Space|Build-Space): [^ ]+.*$')
run()

Run sbuild && update buildresult from build log

Note

This will iterate all lines of the build log, and parse out the selection of sbuild’s summary status we need. In case the build log above does write the same output like ‘Status: xyz’, sbuild’s correct status at the bottom will override this later. Best thing, though, would be if sbuild would eventually provide a better way to get these values.

cancel(who)