angstrom-distribution/meta-angstrom/boot-validate

Boot validation for Angstrom images

A bitbake pass only proves the rootfs was assembled with the right files. Whether a preset really disables a unit, a timer really arms, a networkd drop-in really takes effect, or an ALSA device is really openable by the user that needs it, is only observable on a booted system. This skill boots the image and answers that.

Companion skills: angstrom-build (the standard kas-container build flow) and mackas-angstrom (the same on a macOS/mackas host — mackas exec, retrieve). Build there, validate here.

Two launch shapes, one validator

Everything downstream of "a console appeared" is identical, which is what scripts/boot-validate.py implements once for both. The backend choice (runqemu vs. the board fork) is a lookup in a BACKENDS table in the script, not an if/elif on machine name — see "Sweeping every qemu machine" below.

The qemu machine list itself is discovered at runtime from openembedded-core/meta/conf/machine/qemu*.conf (a sibling checkout — see mackas-angstrom/angstrom-build for the layout), not hardcoded, so it will not silently go stale the next time oe-core adds or drops a machine. --list-machines shows what was actually discovered (or says so if it fell back to a static list because no sibling checkout was found).

The validator: scripts/boot-validate.py

Waits for a login prompt, logs in, waits for the shell to actually be ready (not merely echoed — an early command gets swallowed during session setup), runs fenced sanity checks, then any extra probes, and prints a single BOOT-VALIDATION: PASS | PASS(warn) | FAIL line. Exit code 0 on PASS, non-zero on FAIL. The full console capture goes to a log file for evidence.

Two console backends, same logic:

• --launch 'CMD' — spawn CMD and drive its stdin/stdout as the console. This is the emulator path for both shapes. CMD must put the guest console on stdio (runqemu ... nographic, or -nographic for a hand-built qemu line).
• --port /dev/ttyUSB0 — a real serial port via pyserial, for validating the same image on real hardware. pyserial is imported only on this path, so the emulator path needs nothing installed.

--launch/--port plus --check/--host-cmd/--timeout/--poweroff/--password are implemented and verified end-to-end against a real headless beaglebone boot. --sweep/--machine (the oe-core qemu*-machine driver described below, which builds inside the container and needs its own machine-discovery logic) is not currently implemented — driving a single machine via --launch is the only working path right now.

Verdict rules:

• systemctl is-system-running → running = PASS.
• starting = PASS(warn), not a bare PASS. A single snapshot can't tell a real transient state (jobs still queued, about to settle) from a permanent one (a crash-looping unit whose start job never completes) — conflating the two let a genuinely broken app (a service stuck in a 30-restart loop, never coming up) report a bare PASS three attempts running before this was caught. Don't trust starting alone; the per-image --check probes (is-active, journalctl -u <service>) right below carry the real answer — always add them for any service the image is meant to run.
• degraded = PASS(warn) on purpose: the board booted; chase the failing units separately with systemctl --failed.
• No systemd at all (an init=/bin/sh or non-systemd boot) but a real uname -a = PASS(warn).
• No login prompt, no shell, or an empty uname = FAIL, with the last 40 console lines printed for triage.

Compare the reported `/etc/os-release` fields with the build you meant to test. A PASS on a stale image validates nothing. The script reports PRETTY_NAME, VERSION and BUILD_ID; Angstrom sets the first two (PRETTY_NAME="Angstrom <release>"), so BUILD_ID is normally absent and VERSION is the field to compare. --check probes are the way to add anything image-specific:

--check 'audio:cat /proc/asound/cards; ls -l /dev/snd'
--check 'presets:systemctl is-enabled busybox-syslog.service busybox-klogd.service'

Each runs after login and is reported verbatim under its name. Judge the printed text, not exit codes — see the gotchas.

Per-image configuration: hostfwd, checks, host-cmd

--sweep/--machine mode carries three per-image tables in the script, keyed on the image recipe name, so a boot-test config for a specific image lives in the repo instead of being typed at each invocation. All three are additive to whatever the caller passes on the command line, never a replacement for it.

`IMAGE_HOSTFWD` — {image: [(host_port, guest_port), ...]}. get_extra_qemu_args() composes the full QB_SLIRP_OPT string from it, keeping runqemu's own default 127.0.0.1:2222->22/127.0.0.1:2323->23 forwards (an override replaces the whole hostfwd list, not just adds to it — see the code comment). A row's extra qemu args come from IMAGE_HOSTFWD[image] when the image has an entry, falling back to the machine row's own extra_qemu_args (MACHINE_OVERRIDES) otherwise. Take the guest port from the recipe's own config/env file and record where you got it — a guessed port produces a forward that silently reaches nothing. Where an httpd or reverse proxy fronts the app, forward its port and probe through it: that is the path a real client takes.

The tables ship two worked examples rather than any real image — a REST-shaped myapp-image (8080 → 8080) and a SOAP-shaped mysoap-image (8090 → 80 for the fronting httpd, 9090 → 9090 for the app itself). Replace them when adding a real image; they exist to show the mechanism, not to be kept.

`IMAGE_CHECKS` — {image: ["NAME:CMD", ...]}, threaded through exactly like --check: run on the guest, after the sanity checks. Write real probes, not just systemctl --failed + is-active — is-active proves only that a process forked. The example entries show the shape worth copying:

• a REST cycle — GET an endpoint, extract a token from the response body, send it as a header on a second POST/GET, and assert on both status codes and the fields that must be present;
• a SOAP cycle — fetch ?wsdl and confirm it is one, then POST an XML envelope with a SOAPAction header, assert the expected element is in the response, and assert a Fault is not (a SOAP Fault is a perfectly valid HTTP 200, so status alone never catches it);
• per-app diagnostics either way: journalctl -u <unit> (last, so it also covers what the probes above provoked), a state/DB-file ls, an HTTP-status probe per web route.

Wait on the port, not on the unit. Type=simple reports active the instant ExecStart forks, long before an app framework finishes bootstrap and calls listen(). An is-active gate therefore races every API probe behind it. Poll the real port instead — for i in $(seq 1 40); do curl -s -o /dev/null --max-time 2 http://127.0.0.1:PORT/ && break; sleep 2; done.

A probe can fail the verdict: print a line starting with `BV-FAIL:`. The run then ends BOOT-VALIDATION: FAIL with every such line named in the verdict. Policy lives in the probe because only the probe knows what its own output means — test "$C" = 200 || echo BV-FAIL: GET / HTTP $C. Without it the verdict reads only systemd's own state, which is how an image whose every HTML route returned 500 reported a clean PASS: nothing in the check set read an HTTP status, and the one contradicting signal was a screenshot, i.e. a report-only --host-cmd. A TIMED OUT check fails the run too — no evidence gathered is not a pass.

Two probe-authoring traps, both found the hard way:

• The whole probe is one console line, sharing it with the run's own end marker. boot-validate.py sends echo BVSTART; <your probe>; echo BVEND, so a shell syntax error anywhere in the probe takes BVEND down with it and the harness reports TIMED OUT rather than anything about the syntax. An unquoted || echo (no such file) did exactly this and hid a probe's output for two full attempts. Quote anything containing (, ), &, ; or |.
• Fetching a stack trace: grep the message, don't `tail` the file. A web-framework trace is easily ~90 frames, so tail -n 60 returns middleware plumbing and never the exception that names the fault. Grep the log's LEVEL: message header lines instead, and cut -c1-300 — a single untruncated frame takes minutes to clock out over the serial console.

`--host-cmd NAME:CMD` (repeatable) — runs on the emulator host (wherever boot-validate.py itself runs — inside the kas-container for the runqemu backend), after the --check probes and before poweroff, guest still up. This is the way to drive a hostfwd'd port from the host side: curl against a forwarded API port, a headless-Chromium screenshot of a webUI, anything that needs to reach the guest over the network rather than through the console. Captured stdout/stderr and rc are mirrored into the log file and printed as HOST-CMD <name>: rc=<n> plus the output. Report-only: a failing --host-cmd never flips BOOT-VALIDATION from PASS to FAIL — it diagnoses the app sitting on top of a working boot, not the boot itself. IMAGE_HOST_CMDS threads per-image entries through --sweep/--machine the same table-driven way as IMAGE_CHECKS; the example entry is a headless-Chromium screenshot per web route (chromium --headless=new --no-sandbox --disable-gpu --virtual-time-budget=10000 --window-size=1280,1024 --screenshot=/repo/artifacts-boot-validate/<name>.png http://127.0.0.1:<hostfwd-port>/). It installs chromium on demand because it runs host-side, outside the read-only container mount; output lands in artifacts-boot-validate/ at the meta-angstrom repo root — add that to .git/info/exclude. Address the host-side `IMAGE_HOSTFWD` port here, not the guest port the `--check` probes use.

A screenshot is evidence only once you open it. Every failure mode here produces a PNG of the right size at the right time: a Chromium ERR_CONNECTION_RESET page when nothing is listening, a framework's own 500 | Server Error page when the app is up but the view layer is broken. Read the file, and cross-check that two shots of different routes are not byte-identical — identical hashes mean a global fault, not two working pages.