phoebus_os/docs/VERIFICATION.md
Phoebus 359f083bc2
Some checks are pending
ci / test (push) Waiting to run
test(gui): drive the OpenCode AI step through the real UI
Add an 'opencode' harness mode with a guest-loopback relay (the tunnel
shape), keyboard selection of the OpenCode chip, UI entry of the server/
provider/model (and a 0600 server password), and assertions on the
frontend's secret-free state (tested, validated, applied). Record the VM
evidence and the remaining not-run full-install gate.
2026-09-22 03:32:07 -07:00

117 lines
10 KiB
Markdown

# Verification and findings register
Companion to `docs/STATUS.md`. The machine-readable per-run results live outside
git under `/var/tmp/phoebus-vm/<run-id>/results.json`; this file is the
human-readable register of what was found, fixed and still unverified.
## OpenCode adapter (2026-09-22, second pass)
Implemented a real OpenCode adapter (not just a probe), integrated through
canonical configuration (`ai.backend = "opencode"`), the GUI bridge, the
installer CLI and `sys ask`. Defects found while implementing and testing:
| ID | Defect | Fix | Test |
|----|--------|-----|------|
| O1 | No OpenCode proposal adapter; the GUI used the OpenAI-compatible transport | `phoebus-executor::opencode_client` implements the real session/message protocol as a `ModelTransport` | live + bridge end-to-end |
| O2 | `POST /session` was sent a `model` field per the OpenAPI schema | the shipped 1.18.32 rejects it (BadRequest); the model is selected on the message | real-HTTP integration |
| O3 | The probe had no credential input | `ProbeRequest.auth` + basic-auth header via stdin | auth integration test |
| O4 | The probe did not enforce the connection destination | `Endpoint::checked_target` resolves and requires loopback for plain http; the connection is pinned | `transport_policy_enforces_loopback_for_plain_http`, insecure-destination test |
| O5 | Handwritten HTTP read had no total deadline and a UTF-8-unsafe chunk decoder that fell back on malformed input | removed the handwritten parser; the established curl transport is used and bounds time, protocol, redirects and size | real-HTTP chunked/oversize/redirect/disconnect tests |
| O6 | The catalog was parsed without checking the HTTP status; a providers-shaped 500 could look like a catalog | status is checked before parsing; non-2xx is `incompatible` | `a_misleading_success_shaped_500_does_not_produce_a_catalog` |
| O7 | Catalog presence counted as readiness | `ready` is reserved for a validated response; the probe returns `model_catalog`/`reachable` | `an_empty_catalog_is_reachable_not_ready` |
| O8 | `probe_live_server_when_configured` passed when its env var was absent and only asserted reachability | replaced by env-gated `opencode_live` tests that assert a real inference; `PHOEBUS_OPENCODE_REQUIRE_LIVE=1` fails when the live run cannot happen | `make opencode-acceptance` |
| O9 | The credential test built an already-sanitised struct | tests feed credential-bearing server bodies and assert errors/catalogs never contain them | `one_shot_rejects_a_non_assistant_or_errored_reply`, `catalog_never_reads_provider_keys` |
| O10 | A provider name satisfied the mandatory-AI gate | both the bridge and the CLI require a validated interaction bound to the current configuration; config/secret changes invalidate it | `a_provider_name_alone_does_not_satisfy_the_ai_requirement`, `an_unreachable_provider_does_not_satisfy_the_ai_gate` |
| O11 | The proposal prompt allowed the model to key the change by action name | prompt now requires exactly `action` + `value` with explicit examples | live bridge `ai_check` |
| O12 | The GUI duplicate-session assertion accepted two pre-existing sessions | requires exactly one session before and after | harness |
| O13 | GUI navigation was inferred from a surviving process/screenshot | the frontend emits secret-free `phoebus-state` lines; the harness asserts the current step and AI state | harness |
| O14 | A model could return a malformed proposal with no second chance | one bounded corrective retry in `complete_proposal_at`/`complete_plan`, with the strict parser still final | `a_rejected_proposal_is_retried_once_with_a_correction`, `a_retry_that_still_fails_is_still_rejected` |
### Graphical OpenCode acceptance (VM verified)
`gui-driver.py opencode` on the built ISO (real model, no simulation):
- welcome → installer → AI step; selected **OpenCode agent** through the UI;
- typed the server URL, provider and model through the UI;
- **Test connection** obtained a real model response (green "Model responded");
- **Connect and validate** obtained and validated a proposal (green "Validated",
frontend state `aiConnected=true`);
- run `gui-opencode-20260922T102640Z-be168813`: **17/17 PASS**, exit 0; the full
GUI regression `gui-gui-20260922T102907Z-a9e5e824` was **25/25 PASS**.
- The model is reached through a guest-loopback relay to the host OpenCode
server (the SSH-tunnel shape), so the transport policy is satisfied.
- Authenticated server: the same harness with a password-protected server got
through authentication and a real Test connection; a flaky malformed proposal
was correctly rejected. Use `deepseek-flash` (reliable) over
`deepseek-v4-flash` (occasionally malformed).
Still **NOT RUN**: the full graphical installation (disk → account → review →
ERASE → install → detached boot) *through OpenCode*. The AI gate is now reached
and passed in the graphical VM; the remaining stages are the long package
installation. Two fresh-disk graphical installs and a separate physical machine
remain unverified.
OpenCode protocol/transport tests: 26 executor unit tests + 7 real-HTTP
integration tests + 2 live tests (skipped offline, real inference when
configured). Bridge gate tests: 5. Full suite: 336 Rust tests.
## Run matrix (this session, 2026-09-22)
```json
{
"source_revision": "7e9fa9c",
"iso_sha256": "428dbd7547db8690596bf5c3a0d7a6fa8aee4c9f9eca1e926c216712dd050b36",
"static_binary_sha256": "2f9cd4b36a12e469d2b6187166acaf8b11f7f382cc19f7ad08bb2a809c855591",
"runs": [
{"scenario": "gui", "result": "PASS 23/23", "exit_code": 0,
"artifacts": "/var/tmp/phoebus-vm/gui-gui-20260922T025537Z-69f754cf"},
{"scenario": "make test (rust + vmsafety)", "result": "PASS", "exit_code": 0},
{"scenario": "cargo fmt --check", "result": "PASS", "exit_code": 0},
{"scenario": "cargo clippy --workspace --all-targets", "result": "PASS (warnings only)", "exit_code": 0},
{"scenario": "opencode probe (live 1.18.32)", "result": "ready", "exit_code": 0},
{"scenario": "opencode probe (unauth LAN bind)", "result": "authentication_required", "exit_code": 1},
{"scenario": "noninteractive install + detached boot", "result": "NOT RUN on the final ISO", "exit_code": null},
{"scenario": "full graphical install past AI", "result": "BLOCKED", "exit_code": null},
{"scenario": "rollback VM acceptance", "result": "NOT RUN", "exit_code": null},
{"scenario": "AI sandbox canary VM test", "result": "NOT RUN", "exit_code": null},
{"scenario": "second physical computer OpenCode transport", "result": "BLOCKED", "exit_code": null}
]
}
```
## Confirmed defects fixed
| ID | Defect | Symptom | Fix | Regression test |
|----|--------|---------|-----|-----------------|
| D1 | Disk safety was not revalidated before destroying the target | A disk that disappeared, was mounted, became the live medium, or was replaced under the same name between selection and install would be partitioned on the strength of a stale review | `Installer::revalidate_target_disk` re-probes and compares identity in `storage_prepare` | `orchestrate::tests::{revalidation_refuses_a_disappeared_disk,revalidation_refuses_a_now_protected_disk,revalidation_refuses_reused_device_name_with_a_new_size,...}` |
| D2 | `probe_disks` parsed human-readable `lsblk` sizes as bytes | Every disk displayed `0 B`; the size-identity check was blind | pass `-b` to `lsblk` | `probe::tests::probe_disks_classifies_optical_as_protected` asserts the byte size |
| D3 | The bridge could replay or overlap a destructive install | A second `start_install` (retry, second frontend, reconnect) could re-run storage preparation | approval is single-use, consumed before work; `installing`/`installed` guards | `start_install_cannot_be_replayed_after_success` |
| D4 | The welcome's primary action was not keyboard-focused | No focus ring; Return did not open the installer | request activation + `forceActiveFocus()`; regression test also rejects the invalid `onActiveChanged` signal | `welcome_primary_action_is_keyboard_reachable` |
| D5 | Escape during installation silently closed the frontend | A destructive operation could be abandoned with no explanation | Escape is refused during progress and the UI says so | `progress_cannot_be_abandoned_silently` |
| D6 | Provider transport accepted unsafe URL schemes / unbounded responses / followed no protocol allowlist | A config-supplied endpoint could target `file:`, inject curl config, or return a huge body | `validate_endpoint`, `proto = "=http,https"`, `max-filesize`, redirects disabled | `endpoint_validation_rejects_unsafe_schemes_and_injection`, `curl_transport_refuses_an_unsafe_endpoint`, `curl_transport_bounds_the_response_and_disables_redirects` |
| D7 | `TargetRoot` followed symlinks out of the target | A planted symlink could redirect an installer write to the host | `reject_symlink_components` in `write`/`ensure_dir` | `target::tests::{refuses_to_write_through_an_escaping_symlink,refuses_a_symlinked_leaf}` |
| D8 | The VM harness documented disk safety but did not enforce it, and reported observations rather than results | A mistyped `DISK` could point at a real device; a timeout could look like success; shell interpolation in screenshot conversion | `vmsafety.py`; per-run OVMF/QMP; structured results; argument arrays | `test_vmsafety.py` (16 tests), run by `make test` |
## Suspected risks (reproduced or bounded)
- **Prompt injection in device labels / model output.** Typed proposals pass
`deny_unknown_fields` and a closed action set; device labels are display-only.
Bounded by tests (`phoebus-ai` parse/allowlist). No VM canary run.
- **Live sudoers is `NOPASSWD: ALL` (live-only).** It does not reach the
installed system, but the GUI bridge uses it. Narrowing is a recorded follow-up.
- **OpenCode `/config/providers` returns provider credentials in cleartext.**
Upstream behaviour; the probe keeps only counts and docs mandate loopback +
tunnel. Not a Phoebus defect, but a deployment hazard.
## Missing verification (blocked / not run)
- Full graphical installation past the AI step and two fresh-disk graphical
installs: **BLOCKED** — no provider key and no local model; normal install
requires a connected model.
- `install`/`boot` modes on the final ISO: **NOT RUN** (the earlier appliance
build passed `INSTALL-RC=0` + detached boot; the final ISO's binary changed,
so this is not automatically re-verified).
- Rollback acceptance, AI sandbox canary, installed Mango/dwl, physical seat,
NVIDIA: **NOT RUN / unverified**.
- Second physical computer OpenCode transport: **BLOCKED**; host-local forwarded
hop and unauthenticated-LAN refusal were tested instead.