Make a passing CI/release result mean the checks actually ran. - .forgejo/workflows/ci.yml is fail-closed: it provisions the pinned toolchain or fails, then runs one 'make ci' (fmt-check, clippy, workspace tests, VM-harness safety tests, release build, static installer build). - Pin stable Rust 1.93.1 in installer-rs/rust-toolchain.toml; run cargo gates with --locked for reproducible dependencies. - phoebus-release gates now records passed/failed/blocked/skipped/not_run (only passed is a pass), executes real host gate commands, and imports the existing VM-harness results.json instead of a competing harness. Gates with no automation are always blocked. - Bind gate reports and harness evidence to a candidate name+SHA-256 (--candidate; PHOEBUS_VM_ARTIFACT[_SHA256]); stale or unbound evidence is failed, never passed. - publish refuses unless the report is candidate-bound and all-passed, the manifest verifies the artifact and a detached signature verifies; atomic publish preserves and restores the previous release on failure. - vmsafety.Results records the verified artifact digest; 3 new Python tests. - 20 new Rust regression tests; ADR 0012 and docs updated. Verified locally: make ci (364 Rust tests, 19 harness tests, release and static builds) exit 0; static binary is static-pie linked. CI runner, VM and model gates remain unexecuted here.
78 lines
4.2 KiB
Markdown
78 lines
4.2 KiB
Markdown
# ADR 0012: Release verification evidence and publication authorization
|
|
|
|
- **Status:** Accepted
|
|
- **Date:** 2026-09-22
|
|
|
|
## Context
|
|
|
|
Two failure modes made a passing result meaningless:
|
|
|
|
1. **CI skipped required checks.** `.forgejo/workflows/ci.yml` wrapped every
|
|
Rust step in `if command -v cargo; then ...; else echo "skipping"; fi`. A
|
|
runner without Cargo reported green while running nothing.
|
|
2. **Release gates did not execute and were not bound to an artifact.**
|
|
`phoebus-release gates --execute` ran no command (there were no host gates,
|
|
and every non-host gate was recorded `not_run`). `publish` accepted any
|
|
report whose results were all `passed`, without checking that the report
|
|
described the artifact being published, that the artifact matched its
|
|
manifest, or that a signature had been verified. A stale report for an old
|
|
build could therefore authorize a different one.
|
|
|
|
The project already has a disposable VM harness (`installer/tests/vm/*.py`)
|
|
that writes structured `results.json`. A second, competing harness is not
|
|
wanted.
|
|
|
|
## Decision
|
|
|
|
1. **CI is fail-closed.** The workflow provisions the pinned toolchain or fails
|
|
with a clear message; it never treats a missing prerequisite as a pass. The
|
|
whole required set is one command, `make ci`, so CI and local development run
|
|
the same checks (fmt, clippy, workspace tests, VM-harness safety tests,
|
|
release build, static installer build). The toolchain is pinned in
|
|
`installer-rs/rust-toolchain.toml` and dependencies use the committed
|
|
`Cargo.lock` (`--locked`).
|
|
2. **Gate outcomes are distinct.** A gate is `passed`, `failed`, `blocked`,
|
|
`skipped` or `not_run`; **only `passed` is a pass**. `blocked` means a
|
|
prerequisite (VM, graphical seat, model, CI runner, or an automated command)
|
|
is missing; `skipped` means an operator excluded it; `not_run` means
|
|
execution was not requested.
|
|
3. **Gates perform their advertised check.** Host gates run their real command
|
|
through the command layer and record the exit status. VM/seat/model gates
|
|
name the real harness scenario and are resolved from that harness's
|
|
`results.json` (reusing the existing harness); with no evidence they are
|
|
`blocked`. Gates whose automation does not exist yet have an empty command
|
|
and can only ever be `blocked`.
|
|
4. **Evidence is bound to the artifact.** `phoebus-release gates --candidate
|
|
<file>` records the candidate's name and SHA-256 in the report; the harness
|
|
records the digest it verified from `PHOEBUS_VM_ARTIFACT[_SHA256]`. Imported
|
|
evidence whose digest differs from the candidate is reported as `failed`
|
|
("stale evidence"), never passed.
|
|
5. **Publication is authorized, not assumed.** `phoebus-release publish`
|
|
refuses unless the report is bound to the exact candidate digest, every gate
|
|
passed, the candidate matches its manifest entry, and a detached signature
|
|
verifies. Check failures happen before the destination is touched.
|
|
`atomic_publish` keeps `latest.iso.prev` and restores it if the final rename
|
|
fails, so a failed or interrupted publication never removes the working
|
|
release.
|
|
|
|
## Consequences
|
|
|
|
- A green CI job means the listed checks actually ran. There is no "skipping"
|
|
path for a required check.
|
|
- `make release-gates` writes a report that reflects reality. Because
|
|
`no-model-recovery`, `updates-and-rollback` and `desktop-session` have no
|
|
automated harness yet, they are always `blocked`, so a full release cannot be
|
|
authorized until those checks are automated (or given reviewed manual
|
|
evidence). This is deliberate: it is preferable to be unable to publish than
|
|
to publish on a false pass.
|
|
- The release report and per-gate logs are diagnostics, not source; they are
|
|
gitignored, redacted with the environment redactor, and bounded in size.
|
|
- The remaining external prerequisite is a registered Forgejo Actions runner;
|
|
until one exists the workflow is unexecuted and is recorded as such.
|
|
|
|
## Status
|
|
|
|
Accepted and implemented. The gate engine, harness-evidence import, artifact
|
|
binding and publication refusals are covered by unit tests; `make ci` passes
|
|
locally. CI execution and VM/model gates remain unverified here (no runner, no
|
|
disposable VM or model in this environment).
|