A fully-validated OpenCode install was refused at the final step because selecting a disk or setting the account/LUKS secrets cleared the AI validation. Bind the validation to a digest of only the AI connection settings: non-AI config changes preserve it, while an AI provider/model/ endpoint/server or AI-credential change invalidates it. Add regression tests and password-length placeholder hints.
138 lines
13 KiB
Markdown
138 lines
13 KiB
Markdown
# Verification and findings register
|
||
|
||
Companion to `docs/STATUS.md`. The machine-readable per-run results live outside
|
||
git under `/var/tmp/phoebus-vm/<run-id>/results.json`; this file is the
|
||
human-readable register of what was found, fixed and still unverified.
|
||
|
||
## OpenCode adapter (2026-09-22, second pass)
|
||
|
||
Implemented a real OpenCode adapter (not just a probe), integrated through
|
||
canonical configuration (`ai.backend = "opencode"`), the GUI bridge, the
|
||
installer CLI and `sys ask`. Defects found while implementing and testing:
|
||
|
||
| ID | Defect | Fix | Test |
|
||
|----|--------|-----|------|
|
||
| O1 | No OpenCode proposal adapter; the GUI used the OpenAI-compatible transport | `phoebus-executor::opencode_client` implements the real session/message protocol as a `ModelTransport` | live + bridge end-to-end |
|
||
| O2 | `POST /session` was sent a `model` field per the OpenAPI schema | the shipped 1.18.32 rejects it (BadRequest); the model is selected on the message | real-HTTP integration |
|
||
| O3 | The probe had no credential input | `ProbeRequest.auth` + basic-auth header via stdin | auth integration test |
|
||
| O4 | The probe did not enforce the connection destination | `Endpoint::checked_target` resolves and requires loopback for plain http; the connection is pinned | `transport_policy_enforces_loopback_for_plain_http`, insecure-destination test |
|
||
| O5 | Handwritten HTTP read had no total deadline and a UTF-8-unsafe chunk decoder that fell back on malformed input | removed the handwritten parser; the established curl transport is used and bounds time, protocol, redirects and size | real-HTTP chunked/oversize/redirect/disconnect tests |
|
||
| O6 | The catalog was parsed without checking the HTTP status; a providers-shaped 500 could look like a catalog | status is checked before parsing; non-2xx is `incompatible` | `a_misleading_success_shaped_500_does_not_produce_a_catalog` |
|
||
| O7 | Catalog presence counted as readiness | `ready` is reserved for a validated response; the probe returns `model_catalog`/`reachable` | `an_empty_catalog_is_reachable_not_ready` |
|
||
| O8 | `probe_live_server_when_configured` passed when its env var was absent and only asserted reachability | replaced by env-gated `opencode_live` tests that assert a real inference; `PHOEBUS_OPENCODE_REQUIRE_LIVE=1` fails when the live run cannot happen | `make opencode-acceptance` |
|
||
| O9 | The credential test built an already-sanitised struct | tests feed credential-bearing server bodies and assert errors/catalogs never contain them | `one_shot_rejects_a_non_assistant_or_errored_reply`, `catalog_never_reads_provider_keys` |
|
||
| O10 | A provider name satisfied the mandatory-AI gate | both the bridge and the CLI require a validated interaction bound to the current configuration; config/secret changes invalidate it | `a_provider_name_alone_does_not_satisfy_the_ai_requirement`, `an_unreachable_provider_does_not_satisfy_the_ai_gate` |
|
||
| O11 | The proposal prompt allowed the model to key the change by action name | prompt now requires exactly `action` + `value` with explicit examples | live bridge `ai_check` |
|
||
| O12 | The GUI duplicate-session assertion accepted two pre-existing sessions | requires exactly one session before and after | harness |
|
||
| O13 | GUI navigation was inferred from a surviving process/screenshot | the frontend emits secret-free `phoebus-state` lines; the harness asserts the current step and AI state | harness |
|
||
| O14 | A model could return a malformed proposal with no second chance | one bounded corrective retry in `complete_proposal_at`/`complete_plan`, with the strict parser still final | `a_rejected_proposal_is_retried_once_with_a_correction`, `a_retry_that_still_fails_is_still_rejected` |
|
||
|
||
### Graphical OpenCode acceptance (VM verified)
|
||
|
||
`gui-driver.py opencode` on the built ISO (real model, no simulation):
|
||
|
||
- welcome → installer → AI step; selected **OpenCode agent** through the UI;
|
||
- typed the server URL, provider and model through the UI;
|
||
- **Test connection** obtained a real model response (green "Model responded");
|
||
- **Connect and validate** obtained and validated a proposal (green "Validated",
|
||
frontend state `aiConnected=true`);
|
||
- run `gui-opencode-20260922T102640Z-be168813`: **17/17 PASS**, exit 0; the full
|
||
GUI regression `gui-gui-20260922T102907Z-a9e5e824` was **25/25 PASS**.
|
||
- The model is reached through a guest-loopback relay to the host OpenCode
|
||
server (the SSH-tunnel shape), so the transport policy is satisfied.
|
||
- Authenticated server: the same harness with a password-protected server got
|
||
through authentication and a real Test connection; a flaky malformed proposal
|
||
was correctly rejected. Use `deepseek-flash` (reliable) over
|
||
`deepseek-v4-flash` (occasionally malformed).
|
||
|
||
Still **NOT RUN**: the full graphical installation (disk → account → review →
|
||
ERASE → install → detached boot) *through OpenCode*. The AI gate is now reached
|
||
and passed in the graphical VM; the remaining stages are the long package
|
||
installation. Two fresh-disk graphical installs and a separate physical machine
|
||
remain unverified.
|
||
|
||
### User trial findings (interactive VM, 2026-09-22)
|
||
|
||
Observed while a user drove the live ISO by hand (run
|
||
`/var/tmp/phoebus-vm/user-session*`):
|
||
|
||
| ID | Finding | Status |
|
||
|----|---------|--------|
|
||
| U1 | Timezone (and locale, keyboard) were **free-text**: the user was expected to know `America/New_York` etc. Only ~5 keymap presets were offered, then manual entry. | **Fixed**: searchable pickers populated from the live system (418 timezones from `zone.tab`, 513 locales, 82 keymaps), type-to-filter and click-to-choose; a custom value is still accepted. |
|
||
| U2 | The AI **provider/model** fields were free-text; the user had to know the exact model id. | **Fixed**: `ai_catalog` bridge op returns the OpenCode catalog (223 providers, real model ids) or an OpenAI-compatible `/models` list; the GUI shows selectable Provider/Model combos with a "Load list" action. |
|
||
| U3 | An invalid model produced only `OpenCode message failed: HTTP 500` — opaque. | **Fixed in source**: the error now names `provider/model` and says to check the model exists and the credentials are valid. (Needs the next ISO.) |
|
||
| U4 | At first boot the DMS app-launcher/overview overlay was open on top of the welcome window, hiding the primary action until dismissed. | **Open**: reproduced once; investigating whether it is a DMS first-run behaviour or a focus race. |
|
||
| U5 | The OpenCode model default `deepseek-chat` is not present on every server, so the first attempt failed. | **Mitigated**: the picker now lists only real models. Considering defaulting the model to unset and requiring a pick. |
|
||
| U6 | **Showstopper.** A fully-validated OpenCode install was refused at the final step with *"installing Phoebus requires a successful, validated AI interaction"*. The validated state was cleared by **unrelated** configuration and secret changes (selecting a disk, typing the account password, setting the LUKS passphrase). | **Fixed**: the AI validation is now bound to a digest of only the AI connection settings; non-AI config changes preserve it, and only an AI provider/model/endpoint/server change or an AI credential change invalidates it. Regression tests: `a_non_ai_config_change_keeps_the_validated_interaction`, `an_account_secret_does_not_invalidate_but_an_ai_key_does`. |
|
||
| U7 | The account/encryption step required several attempts; the password/passphrase minimum was only revealed as a red error after pressing Next, and the offending fields could be scrolled off-screen. | **Mitigated**: password/passphrase fields now carry an "At least 8 characters" / "Repeat …" placeholder hint before submission. |
|
||
|
||
U6 blocked a real install; U1–U3 and U6–U7 are implemented, U4 remains open.
|
||
The user's session reached the AI step, obtained and applied a validated
|
||
OpenCode proposal, selected the disk, completed the account step and the
|
||
review — and only then hit U6. Evidence: `/var/tmp/phoebus-vm/user-session3/`
|
||
(`shots/0008`–`0016`, `observations.log`).
|
||
|
||
OpenCode protocol/transport tests: 26 executor unit tests + 7 real-HTTP
|
||
integration tests + 2 live tests (skipped offline, real inference when
|
||
configured). Bridge gate tests: 5. Full suite: 336 Rust tests.
|
||
|
||
## Run matrix (this session, 2026-09-22)
|
||
|
||
```json
|
||
{
|
||
"source_revision": "7e9fa9c",
|
||
"iso_sha256": "428dbd7547db8690596bf5c3a0d7a6fa8aee4c9f9eca1e926c216712dd050b36",
|
||
"static_binary_sha256": "2f9cd4b36a12e469d2b6187166acaf8b11f7f382cc19f7ad08bb2a809c855591",
|
||
"runs": [
|
||
{"scenario": "gui", "result": "PASS 23/23", "exit_code": 0,
|
||
"artifacts": "/var/tmp/phoebus-vm/gui-gui-20260922T025537Z-69f754cf"},
|
||
{"scenario": "make test (rust + vmsafety)", "result": "PASS", "exit_code": 0},
|
||
{"scenario": "cargo fmt --check", "result": "PASS", "exit_code": 0},
|
||
{"scenario": "cargo clippy --workspace --all-targets", "result": "PASS (warnings only)", "exit_code": 0},
|
||
{"scenario": "opencode probe (live 1.18.32)", "result": "ready", "exit_code": 0},
|
||
{"scenario": "opencode probe (unauth LAN bind)", "result": "authentication_required", "exit_code": 1},
|
||
{"scenario": "noninteractive install + detached boot", "result": "NOT RUN on the final ISO", "exit_code": null},
|
||
{"scenario": "full graphical install past AI", "result": "BLOCKED", "exit_code": null},
|
||
{"scenario": "rollback VM acceptance", "result": "NOT RUN", "exit_code": null},
|
||
{"scenario": "AI sandbox canary VM test", "result": "NOT RUN", "exit_code": null},
|
||
{"scenario": "second physical computer OpenCode transport", "result": "BLOCKED", "exit_code": null}
|
||
]
|
||
}
|
||
```
|
||
|
||
## Confirmed defects fixed
|
||
|
||
| ID | Defect | Symptom | Fix | Regression test |
|
||
|----|--------|---------|-----|-----------------|
|
||
| D1 | Disk safety was not revalidated before destroying the target | A disk that disappeared, was mounted, became the live medium, or was replaced under the same name between selection and install would be partitioned on the strength of a stale review | `Installer::revalidate_target_disk` re-probes and compares identity in `storage_prepare` | `orchestrate::tests::{revalidation_refuses_a_disappeared_disk,revalidation_refuses_a_now_protected_disk,revalidation_refuses_reused_device_name_with_a_new_size,...}` |
|
||
| D2 | `probe_disks` parsed human-readable `lsblk` sizes as bytes | Every disk displayed `0 B`; the size-identity check was blind | pass `-b` to `lsblk` | `probe::tests::probe_disks_classifies_optical_as_protected` asserts the byte size |
|
||
| D3 | The bridge could replay or overlap a destructive install | A second `start_install` (retry, second frontend, reconnect) could re-run storage preparation | approval is single-use, consumed before work; `installing`/`installed` guards | `start_install_cannot_be_replayed_after_success` |
|
||
| D4 | The welcome's primary action was not keyboard-focused | No focus ring; Return did not open the installer | request activation + `forceActiveFocus()`; regression test also rejects the invalid `onActiveChanged` signal | `welcome_primary_action_is_keyboard_reachable` |
|
||
| D5 | Escape during installation silently closed the frontend | A destructive operation could be abandoned with no explanation | Escape is refused during progress and the UI says so | `progress_cannot_be_abandoned_silently` |
|
||
| D6 | Provider transport accepted unsafe URL schemes / unbounded responses / followed no protocol allowlist | A config-supplied endpoint could target `file:`, inject curl config, or return a huge body | `validate_endpoint`, `proto = "=http,https"`, `max-filesize`, redirects disabled | `endpoint_validation_rejects_unsafe_schemes_and_injection`, `curl_transport_refuses_an_unsafe_endpoint`, `curl_transport_bounds_the_response_and_disables_redirects` |
|
||
| D7 | `TargetRoot` followed symlinks out of the target | A planted symlink could redirect an installer write to the host | `reject_symlink_components` in `write`/`ensure_dir` | `target::tests::{refuses_to_write_through_an_escaping_symlink,refuses_a_symlinked_leaf}` |
|
||
| D8 | The VM harness documented disk safety but did not enforce it, and reported observations rather than results | A mistyped `DISK` could point at a real device; a timeout could look like success; shell interpolation in screenshot conversion | `vmsafety.py`; per-run OVMF/QMP; structured results; argument arrays | `test_vmsafety.py` (16 tests), run by `make test` |
|
||
|
||
## Suspected risks (reproduced or bounded)
|
||
|
||
- **Prompt injection in device labels / model output.** Typed proposals pass
|
||
`deny_unknown_fields` and a closed action set; device labels are display-only.
|
||
Bounded by tests (`phoebus-ai` parse/allowlist). No VM canary run.
|
||
- **Live sudoers is `NOPASSWD: ALL` (live-only).** It does not reach the
|
||
installed system, but the GUI bridge uses it. Narrowing is a recorded follow-up.
|
||
- **OpenCode `/config/providers` returns provider credentials in cleartext.**
|
||
Upstream behaviour; the probe keeps only counts and docs mandate loopback +
|
||
tunnel. Not a Phoebus defect, but a deployment hazard.
|
||
|
||
## Missing verification (blocked / not run)
|
||
|
||
- Full graphical installation past the AI step and two fresh-disk graphical
|
||
installs: **BLOCKED** — no provider key and no local model; normal install
|
||
requires a connected model.
|
||
- `install`/`boot` modes on the final ISO: **NOT RUN** (the earlier appliance
|
||
build passed `INSTALL-RC=0` + detached boot; the final ISO's binary changed,
|
||
so this is not automatically re-verified).
|
||
- Rollback acceptance, AI sandbox canary, installed Mango/dwl, physical seat,
|
||
NVIDIA: **NOT RUN / unverified**.
|
||
- Second physical computer OpenCode transport: **BLOCKED**; host-local forwarded
|
||
hop and unauthenticated-LAN refusal were tested instead.
|