phoebus_os/docs/VERIFICATION.md
Phoebus b4935578ac fix(installer): do not discard the validated AI state on unrelated edits
A fully-validated OpenCode install was refused at the final step because
selecting a disk or setting the account/LUKS secrets cleared the AI
validation. Bind the validation to a digest of only the AI connection
settings: non-AI config changes preserve it, while an AI provider/model/
endpoint/server or AI-credential change invalidates it. Add regression
tests and password-length placeholder hints.
2026-09-22 04:13:31 -07:00

138 lines
13 KiB
Markdown
Raw Permalink Blame History

This file contains ambiguous Unicode characters

This file contains Unicode characters that might be confused with other characters. If you think that this is intentional, you can safely ignore this warning. Use the Escape button to reveal them.

# Verification and findings register
Companion to `docs/STATUS.md`. The machine-readable per-run results live outside
git under `/var/tmp/phoebus-vm/<run-id>/results.json`; this file is the
human-readable register of what was found, fixed and still unverified.
## OpenCode adapter (2026-09-22, second pass)
Implemented a real OpenCode adapter (not just a probe), integrated through
canonical configuration (`ai.backend = "opencode"`), the GUI bridge, the
installer CLI and `sys ask`. Defects found while implementing and testing:
| ID | Defect | Fix | Test |
|----|--------|-----|------|
| O1 | No OpenCode proposal adapter; the GUI used the OpenAI-compatible transport | `phoebus-executor::opencode_client` implements the real session/message protocol as a `ModelTransport` | live + bridge end-to-end |
| O2 | `POST /session` was sent a `model` field per the OpenAPI schema | the shipped 1.18.32 rejects it (BadRequest); the model is selected on the message | real-HTTP integration |
| O3 | The probe had no credential input | `ProbeRequest.auth` + basic-auth header via stdin | auth integration test |
| O4 | The probe did not enforce the connection destination | `Endpoint::checked_target` resolves and requires loopback for plain http; the connection is pinned | `transport_policy_enforces_loopback_for_plain_http`, insecure-destination test |
| O5 | Handwritten HTTP read had no total deadline and a UTF-8-unsafe chunk decoder that fell back on malformed input | removed the handwritten parser; the established curl transport is used and bounds time, protocol, redirects and size | real-HTTP chunked/oversize/redirect/disconnect tests |
| O6 | The catalog was parsed without checking the HTTP status; a providers-shaped 500 could look like a catalog | status is checked before parsing; non-2xx is `incompatible` | `a_misleading_success_shaped_500_does_not_produce_a_catalog` |
| O7 | Catalog presence counted as readiness | `ready` is reserved for a validated response; the probe returns `model_catalog`/`reachable` | `an_empty_catalog_is_reachable_not_ready` |
| O8 | `probe_live_server_when_configured` passed when its env var was absent and only asserted reachability | replaced by env-gated `opencode_live` tests that assert a real inference; `PHOEBUS_OPENCODE_REQUIRE_LIVE=1` fails when the live run cannot happen | `make opencode-acceptance` |
| O9 | The credential test built an already-sanitised struct | tests feed credential-bearing server bodies and assert errors/catalogs never contain them | `one_shot_rejects_a_non_assistant_or_errored_reply`, `catalog_never_reads_provider_keys` |
| O10 | A provider name satisfied the mandatory-AI gate | both the bridge and the CLI require a validated interaction bound to the current configuration; config/secret changes invalidate it | `a_provider_name_alone_does_not_satisfy_the_ai_requirement`, `an_unreachable_provider_does_not_satisfy_the_ai_gate` |
| O11 | The proposal prompt allowed the model to key the change by action name | prompt now requires exactly `action` + `value` with explicit examples | live bridge `ai_check` |
| O12 | The GUI duplicate-session assertion accepted two pre-existing sessions | requires exactly one session before and after | harness |
| O13 | GUI navigation was inferred from a surviving process/screenshot | the frontend emits secret-free `phoebus-state` lines; the harness asserts the current step and AI state | harness |
| O14 | A model could return a malformed proposal with no second chance | one bounded corrective retry in `complete_proposal_at`/`complete_plan`, with the strict parser still final | `a_rejected_proposal_is_retried_once_with_a_correction`, `a_retry_that_still_fails_is_still_rejected` |
### Graphical OpenCode acceptance (VM verified)
`gui-driver.py opencode` on the built ISO (real model, no simulation):
- welcome → installer → AI step; selected **OpenCode agent** through the UI;
- typed the server URL, provider and model through the UI;
- **Test connection** obtained a real model response (green "Model responded");
- **Connect and validate** obtained and validated a proposal (green "Validated",
frontend state `aiConnected=true`);
- run `gui-opencode-20260922T102640Z-be168813`: **17/17 PASS**, exit 0; the full
GUI regression `gui-gui-20260922T102907Z-a9e5e824` was **25/25 PASS**.
- The model is reached through a guest-loopback relay to the host OpenCode
server (the SSH-tunnel shape), so the transport policy is satisfied.
- Authenticated server: the same harness with a password-protected server got
through authentication and a real Test connection; a flaky malformed proposal
was correctly rejected. Use `deepseek-flash` (reliable) over
`deepseek-v4-flash` (occasionally malformed).
Still **NOT RUN**: the full graphical installation (disk → account → review →
ERASE → install → detached boot) *through OpenCode*. The AI gate is now reached
and passed in the graphical VM; the remaining stages are the long package
installation. Two fresh-disk graphical installs and a separate physical machine
remain unverified.
### User trial findings (interactive VM, 2026-09-22)
Observed while a user drove the live ISO by hand (run
`/var/tmp/phoebus-vm/user-session*`):
| ID | Finding | Status |
|----|---------|--------|
| U1 | Timezone (and locale, keyboard) were **free-text**: the user was expected to know `America/New_York` etc. Only ~5 keymap presets were offered, then manual entry. | **Fixed**: searchable pickers populated from the live system (418 timezones from `zone.tab`, 513 locales, 82 keymaps), type-to-filter and click-to-choose; a custom value is still accepted. |
| U2 | The AI **provider/model** fields were free-text; the user had to know the exact model id. | **Fixed**: `ai_catalog` bridge op returns the OpenCode catalog (223 providers, real model ids) or an OpenAI-compatible `/models` list; the GUI shows selectable Provider/Model combos with a "Load list" action. |
| U3 | An invalid model produced only `OpenCode message failed: HTTP 500` — opaque. | **Fixed in source**: the error now names `provider/model` and says to check the model exists and the credentials are valid. (Needs the next ISO.) |
| U4 | At first boot the DMS app-launcher/overview overlay was open on top of the welcome window, hiding the primary action until dismissed. | **Open**: reproduced once; investigating whether it is a DMS first-run behaviour or a focus race. |
| U5 | The OpenCode model default `deepseek-chat` is not present on every server, so the first attempt failed. | **Mitigated**: the picker now lists only real models. Considering defaulting the model to unset and requiring a pick. |
| U6 | **Showstopper.** A fully-validated OpenCode install was refused at the final step with *"installing Phoebus requires a successful, validated AI interaction"*. The validated state was cleared by **unrelated** configuration and secret changes (selecting a disk, typing the account password, setting the LUKS passphrase). | **Fixed**: the AI validation is now bound to a digest of only the AI connection settings; non-AI config changes preserve it, and only an AI provider/model/endpoint/server change or an AI credential change invalidates it. Regression tests: `a_non_ai_config_change_keeps_the_validated_interaction`, `an_account_secret_does_not_invalidate_but_an_ai_key_does`. |
| U7 | The account/encryption step required several attempts; the password/passphrase minimum was only revealed as a red error after pressing Next, and the offending fields could be scrolled off-screen. | **Mitigated**: password/passphrase fields now carry an "At least 8 characters" / "Repeat …" placeholder hint before submission. |
U6 blocked a real install; U1–U3 and U6–U7 are implemented, U4 remains open.
The user's session reached the AI step, obtained and applied a validated
OpenCode proposal, selected the disk, completed the account step and the
review — and only then hit U6. Evidence: `/var/tmp/phoebus-vm/user-session3/`
(`shots/0008`–`0016`, `observations.log`).
OpenCode protocol/transport tests: 26 executor unit tests + 7 real-HTTP
integration tests + 2 live tests (skipped offline, real inference when
configured). Bridge gate tests: 5. Full suite: 336 Rust tests.
## Run matrix (this session, 2026-09-22)
```json
{
"source_revision": "7e9fa9c",
"iso_sha256": "428dbd7547db8690596bf5c3a0d7a6fa8aee4c9f9eca1e926c216712dd050b36",
"static_binary_sha256": "2f9cd4b36a12e469d2b6187166acaf8b11f7f382cc19f7ad08bb2a809c855591",
"runs": [
{"scenario": "gui", "result": "PASS 23/23", "exit_code": 0,
"artifacts": "/var/tmp/phoebus-vm/gui-gui-20260922T025537Z-69f754cf"},
{"scenario": "make test (rust + vmsafety)", "result": "PASS", "exit_code": 0},
{"scenario": "cargo fmt --check", "result": "PASS", "exit_code": 0},
{"scenario": "cargo clippy --workspace --all-targets", "result": "PASS (warnings only)", "exit_code": 0},
{"scenario": "opencode probe (live 1.18.32)", "result": "ready", "exit_code": 0},
{"scenario": "opencode probe (unauth LAN bind)", "result": "authentication_required", "exit_code": 1},
{"scenario": "noninteractive install + detached boot", "result": "NOT RUN on the final ISO", "exit_code": null},
{"scenario": "full graphical install past AI", "result": "BLOCKED", "exit_code": null},
{"scenario": "rollback VM acceptance", "result": "NOT RUN", "exit_code": null},
{"scenario": "AI sandbox canary VM test", "result": "NOT RUN", "exit_code": null},
{"scenario": "second physical computer OpenCode transport", "result": "BLOCKED", "exit_code": null}
]
}
```
## Confirmed defects fixed
| ID | Defect | Symptom | Fix | Regression test |
|----|--------|---------|-----|-----------------|
| D1 | Disk safety was not revalidated before destroying the target | A disk that disappeared, was mounted, became the live medium, or was replaced under the same name between selection and install would be partitioned on the strength of a stale review | `Installer::revalidate_target_disk` re-probes and compares identity in `storage_prepare` | `orchestrate::tests::{revalidation_refuses_a_disappeared_disk,revalidation_refuses_a_now_protected_disk,revalidation_refuses_reused_device_name_with_a_new_size,...}` |
| D2 | `probe_disks` parsed human-readable `lsblk` sizes as bytes | Every disk displayed `0 B`; the size-identity check was blind | pass `-b` to `lsblk` | `probe::tests::probe_disks_classifies_optical_as_protected` asserts the byte size |
| D3 | The bridge could replay or overlap a destructive install | A second `start_install` (retry, second frontend, reconnect) could re-run storage preparation | approval is single-use, consumed before work; `installing`/`installed` guards | `start_install_cannot_be_replayed_after_success` |
| D4 | The welcome's primary action was not keyboard-focused | No focus ring; Return did not open the installer | request activation + `forceActiveFocus()`; regression test also rejects the invalid `onActiveChanged` signal | `welcome_primary_action_is_keyboard_reachable` |
| D5 | Escape during installation silently closed the frontend | A destructive operation could be abandoned with no explanation | Escape is refused during progress and the UI says so | `progress_cannot_be_abandoned_silently` |
| D6 | Provider transport accepted unsafe URL schemes / unbounded responses / followed no protocol allowlist | A config-supplied endpoint could target `file:`, inject curl config, or return a huge body | `validate_endpoint`, `proto = "=http,https"`, `max-filesize`, redirects disabled | `endpoint_validation_rejects_unsafe_schemes_and_injection`, `curl_transport_refuses_an_unsafe_endpoint`, `curl_transport_bounds_the_response_and_disables_redirects` |
| D7 | `TargetRoot` followed symlinks out of the target | A planted symlink could redirect an installer write to the host | `reject_symlink_components` in `write`/`ensure_dir` | `target::tests::{refuses_to_write_through_an_escaping_symlink,refuses_a_symlinked_leaf}` |
| D8 | The VM harness documented disk safety but did not enforce it, and reported observations rather than results | A mistyped `DISK` could point at a real device; a timeout could look like success; shell interpolation in screenshot conversion | `vmsafety.py`; per-run OVMF/QMP; structured results; argument arrays | `test_vmsafety.py` (16 tests), run by `make test` |
## Suspected risks (reproduced or bounded)
- **Prompt injection in device labels / model output.** Typed proposals pass
`deny_unknown_fields` and a closed action set; device labels are display-only.
Bounded by tests (`phoebus-ai` parse/allowlist). No VM canary run.
- **Live sudoers is `NOPASSWD: ALL` (live-only).** It does not reach the
installed system, but the GUI bridge uses it. Narrowing is a recorded follow-up.
- **OpenCode `/config/providers` returns provider credentials in cleartext.**
Upstream behaviour; the probe keeps only counts and docs mandate loopback +
tunnel. Not a Phoebus defect, but a deployment hazard.
## Missing verification (blocked / not run)
- Full graphical installation past the AI step and two fresh-disk graphical
installs: **BLOCKED** — no provider key and no local model; normal install
requires a connected model.
- `install`/`boot` modes on the final ISO: **NOT RUN** (the earlier appliance
build passed `INSTALL-RC=0` + detached boot; the final ISO's binary changed,
so this is not automatically re-verified).
- Rollback acceptance, AI sandbox canary, installed Mango/dwl, physical seat,
NVIDIA: **NOT RUN / unverified**.
- Second physical computer OpenCode transport: **BLOCKED**; host-local forwarded
hop and unauthenticated-LAN refusal were tested instead.