phoebus_os/docs/BASH-MIGRATION.md
2026-09-20 04:01:03 -07:00

451 lines
21 KiB
Markdown
Raw Permalink Blame History

This file contains ambiguous Unicode characters

This file contains Unicode characters that might be confused with other characters. If you think that this is intentional, you can safely ignore this warning. Use the Escape button to reveal them.

# Bash migration — persistent execution prompt and evidence ledger
This file is both the prompt and the progress record for removing the Bash
installer from Phoebus OS. It is authoritative: read it at the start of every
session that touches the migration, and update it before you stop.
Do not treat this as a design document. Do the work, prove it, record it.
---
## 0. How to use this file
1. Read `AGENTS.md`, then this file, then `docs/ROADMAP.md` and
`docs/HANDOFF.md`.
2. Run `make context` and `git log --oneline -10`.
3. Find the lowest-numbered phase whose exit gate is not yet `VERIFIED`.
4. Execute only that phase's tasks. Do not start a later phase.
5. After each task, run its **Verification** command and paste the observed
result into the **Evidence** column. Then commit.
6. Before you stop, update `docs/HANDOFF.md` and the **Session log** at the
bottom of this file.
An agent that skips the verification step has not done the task. There is no
partial credit and no "looks correct".
---
## 1. Mission
The production installer is the Rust workspace in `installer-rs/`. It has
already reproduced the proven encrypted UEFI VM boot, so the Bash tree in
`installer/` is now a reference/fallback that must be retired.
Goal: **zero Bash in the shipping installer and its test/verification path**,
with every deletion backed by a passing automated check or recorded VM
evidence.
Out of scope unless Phase 5 is explicitly requested: developer tooling under
`scripts/`, `.githooks/` and the `Makefile`. Those are host tooling, not the
product, and may remain Bash.
---
## 2. Hard rules (the evidence gate)
These rules override any instruction to move fast.
1. **No task advances to `VERIFIED` without executed evidence.** The Evidence
column must contain: the exact command, the observed result (the key output
lines, not a summary), and the commit hash it was run at. "Unit tests pass"
is not evidence; the command and its output are.
2. **Code inspection, compilation, or reasoning never count as verification.**
A task whose Verification is a test must have that test actually run.
3. **A test that cannot run in this environment is `BLOCKED`, never
`VERIFIED`.** Name the blocker and the exact command that would unblock it.
4. **VM tasks require the disposable qcow2 VM.** Never point `DISK` at a
`/dev/*` device. If the host cannot run the VM, the task stays `BLOCKED`.
5. **Do not weaken, delete, or skip a check to make a task pass.** If a
verification fails, fix the cause or mark the task `BLOCKED` with the
failure recorded.
6. **Do not delete anything from `installer/` until Phase 2's exit gate is
`VERIFIED`.** Preserve the proven boot path (AGENTS.md rule 9).
7. **One logical change per commit**, conventional-commit message, via
`scripts/agent-commit.sh`. Never commit the ISO, build output, transcripts
or secrets.
8. **Never send or log secrets.** Use the existing redactor and 0600 secret
files; the VM evidence must include the `NO_SECRET_LEAK` check.
### Status vocabulary
| Status | Meaning |
| ------ | ------- |
| `TODO` | Not started. |
| `DOING` | Started, not finished. A blocker must be written next to it. |
| `BLOCKED` | Cannot proceed. The blocker and the unblock command are recorded. |
| `VERIFIED` | The Verification command was run and its raw result is in Evidence. |
There is deliberately no `DONE`. `VERIFIED` is the only terminal state.
---
## 3. Preconditions
Record these once under Phase 0 before any migration work.
- `make doctor` reports the host prerequisites.
- `make test` passes (Bash suite + `cargo test --workspace`).
- `cd installer-rs && cargo fmt --all -- --check` and
`cargo clippy --workspace --all-targets` are clean.
- The reference ISO path and the VM harness variables are known (see
`installer/tests/vm/README.md`).
---
## 4. Phase 0 — Preserve and instrument
**Tasks**
- [ ] Tag the current tree as the Bash reference:
`git tag -a v0.1.0-bash-reference -m "Proven Bash installer reference"`.
- [ ] Confirm `installer-rs/crates/phoebus-installer/tests/parity.rs` passes
today, so the Rust plan is known to match the Bash plan before anything
is removed.
- [ ] Write a CLI-coverage test (Rust) that asserts every mode in
`installer/bin/ai-installer:53-73` maps to a Rust CLI path. It should
fail today for `--bootstrap-opencode`.
- [ ] Copy `installer/config/opencode.manifest` into the Rust crate's
resources (e.g. `installer-rs/crates/phoebus-installer/resources/`) and
reference it from the new coverage test.
**Verification**
```sh
git tag --list 'v0.1.0-bash-reference'
cd installer-rs && cargo test --workspace --test parity
cargo test --workspace cli_coverage
```
**Evidence**
| Task | Command | Result | Commit |
| ---- | ------- | ------ | ------ |
| tag | | | |
| parity baseline | | | |
| CLI coverage test (expected fail on opencode) | | | |
| manifest relocated | | | |
**Exit gate:** the reference tag exists, the parity test passes, and the new
coverage test fails *only* on `--bootstrap-opencode`.
---
## 5. Phase 1 — Close the product gaps
**Tasks**
- [ ] Add `phoebus-executor/src/opencode.rs` implementing the behavior of
`installer/lib/opencode.sh`: architecture detection, manifest parsing,
fetch through `Exec`, SHA-256 verification, fail-closed on a missing or
placeholder (`^0+$`) checksum, extraction, symlink, and the config
writer. No `sh -c`; every command through `Exec`.
- [ ] Unit-test it with a fixture manifest and the recording/never runner:
(a) good checksum extracts, (b) placeholder checksum is refused,
(c) mismatch is refused and the download is discarded, (d) unsupported
arch is refused.
- [ ] Add `--bootstrap-opencode` to `installer-rs/crates/phoebus-installer/src/main.rs`
so the CLI-coverage test from Phase 0 passes.
- [ ] Port any remaining Bash-only CLI surface (`--backend`, `--ui` if still
meaningful) or record why it is intentionally dropped.
- [ ] Confirm the `sys` binary covers the Bash `ai.sh` action set, or record
the gap as a separate task.
**Verification**
```sh
cd installer-rs && cargo test --workspace
cargo test --workspace cli_coverage
cargo clippy --workspace --all-targets
```
Plus a manual smoke test of the new flag against a local fixture with a known
SHA-256:
```sh
phoebus-installer --bootstrap-opencode # with the manifest pointed at the fixture
```
**Evidence**
| Task | Command | Result | Commit |
| ---- | ------- | ------ | ------ |
| opencode executor + tests | | | |
| CLI flag + coverage green | | | |
| dropped-surface rationale | | | |
| sys coverage vs ai.sh | | | |
**Exit gate:** `cargo test --workspace` passes and `cli_coverage` is green.
---
## 6. Phase 2 — Move the VM harness to the Rust binary (the critical gate)
The harness currently installs with Bash (`installer/tests/vm/install.sh:41`).
Until it installs with the static Rust binary, deleting Bash would regress the
only proven boot path.
**Tasks**
- [ ] Build the static binary:
`RUSTFLAGS="-C target-feature=+crt-static" cargo build --release -p phoebus-installer --target x86_64-unknown-linux-gnu`.
- [ ] Rewrite the guest-side script to fetch and run the static
`phoebus-installer` binary instead of the Bash installer, keeping the
secret-file handling and the `NO_SECRET_LEAK` scan.
- [ ] Point `installer/tests/vm/README.md` and the serial driver's serve
instructions at the Rust binary.
- [ ] Run the full encrypted UEFI install with the static binary, then boot
the disk with the ISO detached and unlock LUKS.
- [ ] Record the serial log, boot timing, `systemctl is-system-running`, and
the failed-unit count in `docs/VM-BOOT-VERIFICATION.md`.
- [ ] Update `docs/STATUS.md` so the "Rust VM boot parity" row reflects the
static-binary harness.
**Verification**
```sh
# Build:
cd installer-rs && RUSTFLAGS="-C target-feature=+crt-static" \
cargo build --release -p phoebus-installer --target x86_64-unknown-linux-gnu
# Install in the disposable VM (install mode), then boot it detached:
python3 installer/tests/vm/serial-driver.py install
python3 installer/tests/vm/serial-driver.py boot
```
The boot evidence must show: LUKS unlock, login, networking,
`systemctl is-system-running = running`, `0 failed units`, and
`NO_SECRET_LEAK`.
**Evidence**
| Task | Command | Result | Commit |
| ---- | ------- | ------ | ------ |
| static build | | | |
| guest script uses Rust binary | | | |
| VM install (serial log path) | | | |
| VM detached boot | | | |
| docs updated | | | |
**Exit gate:** the detached boot evidence above exists and is recorded. Only
then may Phase 3 begin.
---
## 7. Phase 3 — Retire the tests
**Tasks**
- [ ] Map each file in `installer/tests/*.sh` to existing Rust coverage. For
each, either cite the Rust test that covers the same behavior or add one.
- [ ] Replace `installer-rs/crates/phoebus-installer/tests/parity.rs` with
golden-file tests that pin the Rust plan output; then delete
`parity.rs`.
- [ ] Remove the Bash suite and `shellcheck` step from
`.forgejo/workflows/ci.yml`.
- [ ] Remove the Bash-suite invocation from `.githooks/pre-commit` (keep the
secret scan and image-size checks).
- [ ] Remove `test-bash` from the `Makefile` and make `test` run the Rust
suite.
**Verification**
```sh
cd installer-rs && cargo test --workspace
make test
git grep -n "run_tests.sh\|ai-installer\|installer/lib" -- . ':!docs' ':!installer'
```
The `git grep` must return nothing outside `docs/` and the not-yet-deleted
`installer/` tree.
**Evidence**
| Task | Command | Result | Commit |
| ---- | ------- | ------ | ------ |
| Bash-test coverage mapping | | | |
| golden plan tests | | | |
| parity.rs removed | | | |
| CI + hook updated | | | |
**Exit gate:** `make test` is green and `git grep` shows no code reference to
the Bash installer.
---
## 8. Phase 4 — Delete the Bash tree
**Tasks**
- [ ] Move `installer/doc/FINDINGS.md` and
`installer/doc/VM-BOOT-VERIFICATION.md` under `docs/` and fix links.
- [ ] Delete `installer/bin/`, `installer/lib/`, `installer/tui/`,
`installer/config/`, `installer/tests/*.sh`, and `installer/README.md`.
- [ ] Update `AGENTS.md`, `README.md`, `docs/ARCHITECTURE.md`,
`docs/DEVELOPMENT.md`, `docs/ROADMAP.md` and `docs/CONTRIBUTING.md` to
remove Bash-installer instructions and shellcheck gates.
- [ ] Update `docs/CHANGELOG.md` and `docs/STATUS.md`.
**Verification**
```sh
git grep -n "installer/bin\|installer/lib\|ai-installer\|run_tests.sh" -- . ':!docs/BASH-MIGRATION.md' ':!.git'
make test
```
The `git grep` must return nothing. `make test` must pass.
**Evidence**
| Task | Command | Result | Commit |
| ---- | ------- | ------ | ------ |
| docs relocated | | | |
| Bash tree deleted | | | |
| references updated | | | |
| no dangling references | | | |
**Exit gate:** no tracked file references the Bash installer and `make test` is
green.
---
## 9. Phase 5 — Developer tooling (optional, only if requested)
`scripts/*.sh`, `.githooks/*` and the `Makefile` recipes are host tooling.
Port them to a `cargo xtask` only if the goal is literally zero Bash in the
repository. If done, apply the same evidence gate: the replacement command's
output is the evidence, and the old script may only be deleted once the
replacement is `VERIFIED`.
---
## 10. Ledger
Phase status is the minimum of its tasks. A phase is `VERIFIED` only when its
exit gate has been run and recorded.
| Phase | Scope | Status | Exit-gate evidence |
| ----- | ----- | ------ | ------------------ |
| 0 | Preserve and instrument | VERIFIED | tag `v0.1.0-bash-reference`; parity 1 passed; `cli_coverage` failed only on `--bootstrap-opencode` (`fb8bdfd`) |
| 1 | Close product gaps (opencode) | VERIFIED | `cargo test --workspace`: opencode 7 passed, `cli_coverage` 2 passed, clippy rc 0 (`4071c7d`) |
| 2 | VM harness on the Rust binary | VERIFIED | static-binary install `INSTALL-RC=0` + detached boot (systemd PID1, running, 0 failed units, LUKS, login) |
| 3 | Retire the tests | VERIFIED | `make test` green; `git grep "run_tests.sh\|ai-installer\|installer/lib"` outside docs/installer returns nothing (`875278b`) |
| 4 | Delete the Bash tree | VERIFIED | no tracked file references the Bash installer (`git grep` empty); `make test` green (`fc64793`) |
| 5 | Developer tooling (optional) | TODO | |
### Task ledger
| ID | Task | Status | Blocker (if any) | Evidence |
| -- | ---- | ------ | ---------------- | -------- |
| 0.1 | Tag `v0.1.0-bash-reference` | VERIFIED | | `git tag --list 'v0.1.0-bash-reference'` -> the tag (`ec88d65`) |
| 0.2 | Parity test baseline | VERIFIED | | `cargo test -p phoebus-installer --test parity` -> 1 passed (`fb8bdfd`) |
| 0.3 | Rust CLI-coverage test | VERIFIED | | before: failed only `--bootstrap-opencode` (assert at `cli_coverage.rs:57`); after 1.3: 2 passed (`4071c7d`) |
| 0.4 | Relocate `opencode.manifest` | VERIFIED | | `crates/phoebus-installer/resources/opencode.manifest`; `relocated_opencode_manifest... ok` (`fb8bdfd`) |
| 1.1 | `phoebus-executor::opencode` | VERIFIED | | `crates/phoebus-executor/src/opencode.rs` (`4071c7d`) |
| 1.2 | Opencode unit tests | VERIFIED | | `cargo test -p phoebus-executor opencode` -> 7 passed (`4071c7d`) |
| 1.3 | `--bootstrap-opencode` flag | VERIFIED | | `cli_coverage` 2 passed; CLI smoke: `OpenCode ready at .../opencode` (`4071c7d`) |
| 1.4 | Port/justify remaining CLI surface | VERIFIED | | `--backend`/`--ui` recorded as intentionally dropped in `tests/cli_coverage.rs` (`fb8bdfd`) |
| 1.5 | Confirm `sys` covers `ai.sh` | VERIFIED | | Gap recorded (below) (`4071c7d`) |
| 2.1 | Static binary build | VERIFIED | | `target/x86_64-unknown-linux-gnu/release/phoebus-installer` 3.4M, `statically linked` (`f984b01`) |
| 2.2 | Guest script uses Rust binary | VERIFIED | | `installer/tests/vm/install.sh` fetches `phoebus-installer` (`f984b01`) |
| 2.3 | Harness docs updated | VERIFIED | | `installer/tests/vm/README.md`, `run-vm.sh` (`f984b01`) |
| 2.4 | VM install with Rust binary | VERIFIED | | serial log: `phoebus-installer 0.1.0`, `===INSTALL-RC=0===`, `NO_SECRET_LEAK`, `===POWEROFF===` (commit `54701bd`) |
| 2.5 | VM detached boot | VERIFIED | | `systemd-boot` -> LUKS unlock -> login -> `running`, `0 loaded units listed`, `enp0s2 10.0.2.15/24` (commit `54701bd`) |
| 2.6 | Boot evidence recorded | VERIFIED | | `docs/VM-BOOT-VERIFICATION.md` "Official harness on the static binary" (commit `54701bd`) |
| 3.1 | Bash-test coverage mapping | VERIFIED | | map below; no Bash test lacks a Rust equivalent (`4c59a1f`) |
| 3.2 | Golden plan tests; remove `parity.rs` | VERIFIED | | `cargo test -p phoebus-installer --test plan_golden` -> 3 passed; `parity.rs` deleted (`4c59a1f`) |
| 3.3 | CI updated | VERIFIED | | `.forgejo/workflows/ci.yml` -> fmt + clippy + test (`875278b`) |
| 3.4 | Pre-commit hook updated | VERIFIED | | Bash-suite block removed; image-size + secret scan kept (`875278b`) |
| 3.5 | `Makefile` updated | VERIFIED | | `test: test-rust`; `test-bash` removed (`875278b`) |
| 4.1 | Relocate `installer/doc` | VERIFIED | | `FINDINGS.md`, `VM-BOOT-VERIFICATION.md` -> `docs/`; Bash-only `TESTING.md` deleted (`1e5337e`) |
| 4.2 | Delete Bash tree | VERIFIED | | `installer/{bin,lib,tui,config,tests/*.sh,README.md}` removed; `installer/tests/vm/` kept (`1e5337e`) |
| 4.3 | Update all references | VERIFIED | | `AGENTS.md`, `README.md`, `docs/{ARCHITECTURE,DEVELOPMENT,CONTRIBUTING,ROADMAP,STATUS,GIT-WORKFLOW}.md` (`fc64793`) |
| 4.4 | No dangling references | VERIFIED | | `git grep "installer/bin\|installer/lib\|ai-installer\|run_tests.sh"` -> nothing (`fc64793`) |
### Recorded gaps (from task 1.5)
`sys` is the runtime typed-action broker (`enable/disable` service, `inspect`,
`install`, `history`, updates); the Bash `ai.sh` was installer-time AI
assistance. The overlap is covered: proposal parsing/validation lives in
`phoebus-ai` (fail-closed, `deny_unknown_fields`) and OpenCode provisioning in
`phoebus-executor::opencode`. **Not ported:** interactive provider onboarding
and 0600 API-credential storage (`ai_store_api_key`), and the
keyword-based offline explainer. These belong to the installer/TUI surface, not
`sys`; record as separate future work rather than silently dropping them.
### Bash test → Rust coverage map (task 3.1)
Every `installer/tests/test_*.sh` maps to committed Rust coverage:
| Bash test | Behavior | Rust coverage |
| --------- | -------- | ------------- |
| `test_plan.sh` | plan facts, step order, subvolumes, preview | `phoebus-core::plan::tests::{sequence_matches_proven_order, render_contains_key_facts, destructive_preview_warns_loudly}`; `phoebus-installer/tests/plan_golden.rs` |
| `test_validation.sh` | hostname/user/tz/disk rules, protected disks, password strength | `phoebus-core::validate::tests::{hostname_rules, password_strength_levels, ...}`; `phoebus-core::disk::tests::{optical_live_media_is_protected, loop_and_dm_are_protected, mounted_is_protected}` |
| `test_pkgpolicy.sh` | compile-time lookup, policy/history/binhost decision | `phoebus-core::policy::tests::{balanced_compiles_cheap_and_binaries_expensive, source_first_always_compiles, binary_first_prefers_binary_but_falls_back, history_overrides_estimate}` |
| `test_state.sh` | config round-trip, secret redaction | `phoebus-core::config::tests::{legacy_config_parses_flat_keys, load_from_path_detects_legacy_format}`; `phoebus-core::secret` + `Redactor::tests::{redactor_scrubs_registered_values, redactor_handles_multiple_and_dedup}` |
| `test_ai.sh` | proposal parse/allowlist/validate; offline translator | `phoebus-ai::tests::{parses_and_applies_a_valid_proposal, unknown_action_fails_closed, invalid_value_rejects_whole_proposal}`; `phoebus-core::action::tests::{parses_typed_actions, rejects_shell_action, rejects_unknown_fields_and_bad_values}` |
| `test_opencode.sh` | manifest + checksum policy | `phoebus-executor::opencode::tests` (7 tests) |
| `test_safety.sh` | destructive gate, dry-run sandbox root | `phoebus-executor::auth` tests; `phoebus-executor::exec::tests::dry_run_does_not_execute`; `phoebus-core::target` tests |
| `test_install_dryrun.sh` | dry-run never writes the host root | `phoebus-executor::exec::tests::dry_run_does_not_execute`; `phoebus-installer` dry-run path |
| `test_btrfs_order.sh` | subvolumes created before `subvol=@` mount | `phoebus-executor::storage::tests::create_layout_creates_subvolumes_before_mounting_root` |
| `test_regressions.sh` | cmdline/install.conf/dracut/fstab order, TMPDIR, base pkgs, package.use, log scrub | `phoebus-executor::{kernel, boot, portage, storage}` tests (`cmdline_uses_rd_luks_name`, `install_conf_has_uki_generator_none`, `base_packages_include_storage_tools`); `phoebus-core::system::tests::fstab_mounts_all_subvolumes_including_swap`; `Redactor` log-scrub tests |
| `test_syntax.sh` | `bash -n` over the Bash tree | obsolete once the Bash tree is deleted; `cargo fmt`/`clippy`/compile replace it |
No Bash test exercises behavior that lacks a Rust equivalent. `test_syntax.sh`
is deleted with the Bash tree (Phase 4).
---
## 11. Session protocol
**Start:** read this file, confirm the ledger is current, pick the lowest
incomplete phase, run its Verification before changing anything so you have a
baseline.
**Stop:** run `make test`; update this ledger's rows and the Session log;
update `docs/HANDOFF.md`, `docs/CHANGELOG.md` and `docs/STATUS.md` if behavior
or verification changed; commit with `scripts/agent-commit.sh`; push with
`scripts/publish.sh`.
Never edit a row to `VERIFIED` in the same commit that changes the code unless
that commit also contains the executed evidence.
---
## 12. Verification command reference
```sh
# Full quality gate
make test
cd installer-rs && cargo fmt --all -- --check
cd installer-rs && cargo clippy --workspace --all-targets
cd installer-rs && cargo test --workspace
# Static binary for the live installer environment
cd installer-rs && RUSTFLAGS="-C target-feature=+crt-static" \
cargo build --release -p phoebus-installer --target x86_64-unknown-linux-gnu
# VM harness (never point DISK at /dev/*)
python3 installer/tests/vm/serial-driver.py install
python3 installer/tests/vm/serial-driver.py boot
# Reference check (should shrink to zero by Phase 4)
git grep -n "installer/bin\|installer/lib\|ai-installer\|run_tests.sh"
```
---
## 13. Session log
Append one entry per session, newest last. Keep it short; `git log` has the
detail.
- `YYYY-MM-DD` — (agent/session) phase worked, tasks moved, verification run,
blocker, commit hash.
- `2026-09-20` — Phases 0–4. Tagged `v0.1.0-bash-reference`; baseline cleanup
(`ec88d65`); CLI-coverage gate (`fb8bdfd`); OpenCode executor + 7 tests and
`--bootstrap-opencode` (`4071c7d`); harness moved to the static binary
(`f984b01`); full VM install `INSTALL-RC=0` + detached boot (systemd PID1,
`running`, 0 failed units, LUKS, login) recorded in
`docs/VM-BOOT-VERIFICATION.md`; golden plan tests replace `parity.rs`
(`4c59a1f`); CI/hook/Makefile/scripts now gate on Rust only and `git grep` has
no dangling Bash reference (`875278b`). No blockers. Phase 4 next.