From d96aae1cfa4c8008e5a1a599d75fdff8d6a68861 Mon Sep 17 00:00:00 2001 From: Phoebus Date: Tue, 29 Sep 2026 03:29:46 -0700 Subject: [PATCH] feat(ai): show a concrete action example; record VM live-agent and local-model evidence --- docs/CHANGELOG.md | 17 +++++++++++++++++ docs/HANDOFF.md | 12 ++++++++++++ docs/STATUS.md | 3 ++- docs/VM-BOOT-VERIFICATION.md | 18 +++++++++++++++--- .../crates/phoebus-ai/src/assistant.rs | 11 +++++++---- 5 files changed, 53 insertions(+), 8 deletions(-) diff --git a/docs/CHANGELOG.md b/docs/CHANGELOG.md index 2b44b51..071225c 100644 --- a/docs/CHANGELOG.md +++ b/docs/CHANGELOG.md @@ -5,6 +5,23 @@ this file is the curated, human-readable summary. ## Unreleased — Fast defaults with user-controlled source builds +- **VM live AI: real capable agent, and a local-model prompt fix (2026-09-29).** + Booted the golden overlay and, from inside the guest, reached the host's real + `opencode` agent through the loopback tunnel: `sys ask "list installed + packages" --ai-backend opencode --opencode-provider opencode-go + --opencode-model deepseek-v4.1-flash` returned a valid Assistant Plan + digest + (`CASE live-agent PASS`), and a hostile shell/private-file instruction was + refused/downgraded (`CASE live-agent-hostile-rejected PASS`) + (`serial-boot-20260929T102703Z-ffd1ec25`). A real local model (Qwen2.5-1.5B + GGUF served by `llama-server`) was also driven through the direct sandbox in + both scopes; it reached the model, but the 1.5B model could not satisfy the + exact typed schema (bare-string action, then a missing `rationale` reason) and + the correction prompt exceeded its 4096-token context, so `sys` failed closed + — model capacity, not a `sys` defect. The assistant prompt now shows the + concrete action-object shape (`{"action":"list_packages"}`) instead of only + `[...]` placeholders, which made the small model emit action objects. Gates: + `cargo fmt`, `clippy --workspace --all-targets -- -D warnings`, + `cargo test --workspace` 843 passed / 0 failed. - **VM-verified AI sandbox and unprivileged AI path (2026-09-28).** The first disposable-VM run of the newly built static `sys` (overlay on the golden installed image, `serial-boot-20260929T092147Z-6f88907a`) found and fixed two diff --git a/docs/HANDOFF.md b/docs/HANDOFF.md index d8243d8..a098094 100644 --- a/docs/HANDOFF.md +++ b/docs/HANDOFF.md @@ -25,6 +25,18 @@ commit it. It should be short and current; use `git log` for detail. call (`sys ask` with a real provider is still VM/live NotRun) but it closes the "connected-model path only tested with fake transports" gap. Gates: `cargo test --workspace` 841 passed / 0 failed. +- **VM live AI (2026-09-29, VM-VERIFIED):** from inside the installed guest, + the host's real `opencode` agent was reached through the loopback tunnel and + `sys ask` produced a valid plan (`live-agent`), with a hostile instruction + refused/downgraded (`live-agent-hostile-rejected`) + (`serial-boot-20260929T102703Z-ffd1ec25`). A real local model (Qwen2.5-1.5B + GGUF via `llama-server`) was also driven through the direct sandbox in both + scopes; it reached the model but failed closed on schema-noncompliant output. + The assistant prompt now shows a concrete action-object example + (`{"action":"list_packages"}`), which made the small model emit action + objects; a capable model is still required, and a 4096-token local context + can be exceeded by the one-shot correction turn. Source conversion and + tuned-kernel builds remain NotRun (multi-hour compiles). - **First VM test loop on the golden image (2026-09-28, VM-VERIFIED):** booted a fresh overlay of the installed `phoebus-headless-80g.qcow2` with the static `sys` from HEAD and found two real defects, both fixed. (1) Unprivileged diff --git a/docs/STATUS.md b/docs/STATUS.md index 52fb1de..6b34b17 100644 --- a/docs/STATUS.md +++ b/docs/STATUS.md @@ -98,7 +98,8 @@ Booting that recovery entry offline remains a manual gate. | Provider credential + endpoint boundary | yes | yes (unsafe key characters refused before curl config; userinfo endpoint refused; derived base64 token redacted) | no | n/a | | AI typed-action schema (exact, incl. unit-variant extra keys) | yes | yes (plan/step/investigate actions via `validate_action_value`; smuggled extra key refused) | no | n/a | | AI connected-model path over a real local stub | yes | yes (4 e2e cases: valid plan executes; request has no home path/credential; shell + smuggled key + malformed fail closed) | no (local stub, not a live model) | n/a | -| Live real-agent call (`opencode` backend, host) | yes | yes (`sys ask "list installed packages"` → valid plan + digest, exit 0; hostile shell/private-file instruction refused and downgraded; OpenCode tools disabled; no credential in the log) | host-level only (2026-09-28, local `opencode serve` + `opencode-go/deepseek-v4.1-flash`); VM/direct-provider live call remains NotRun | n/a | +| Live real-agent call (`opencode` backend) | yes | yes (`sys ask "list installed packages"` → valid plan + digest, exit 0; hostile shell/private-file instruction refused and downgraded; OpenCode tools disabled; no credential in the log) | yes — host direct (2026-09-28) and **from inside the installed VM through the loopback tunnel** (`serial-boot-20260929T102703Z-ffd1ec25`, `live-agent` + `live-agent-hostile-rejected`) with `opencode-go/deepseek-v4.1-flash` | n/a | +| Local model through the direct sandbox (VM) | yes | yes (real Qwen2.5-1.5B GGUF via `llama-server`; both sandbox scopes reached the model; schema-noncompliant output failed closed) | VM-run 2026-09-29: reached and failed closed; a 1.5B model is too weak for exact typed plans and its 4096-token context was exceeded by the correction turn — a capable model is required (a capable agent produces valid plans) | n/a | | Privilege derived only from the process identity | yes | yes (an environment variable cannot forge uid 0; the removed `PHOEBUS_ALLOW_UNPRIVILEGED_EXEC` regression test) | no | n/a | | AI OS sandbox (systemd hardened unit) | yes | yes (argv + sandboxed-send; system properties for root, user-manager subset for non-root) | yes (`serial-boot-20260929T092147Z-6f88907a`: `AI_ROOT_RC=0` system sandbox, `AI_USER_RC=0` `--user` sandbox; previously the user path hung on a polkit `manage-units` prompt) | no | | Unprivileged read-only AI plan completes | yes | yes (transaction recording best-effort for non-root; mandatory for privileged mutations) | yes (`serial-boot-20260929T092147Z-6f88907a`: `AI_NOSANDBOX_RC=0` as `phoebus`) | n/a | diff --git a/docs/VM-BOOT-VERIFICATION.md b/docs/VM-BOOT-VERIFICATION.md index b580159..9632e7c 100644 --- a/docs/VM-BOOT-VERIFICATION.md +++ b/docs/VM-BOOT-VERIFICATION.md @@ -45,9 +45,21 @@ the static `sys` built from HEAD; the base image was never modified. revision-changing refresh / strict AppArmor confinement still needs an external Store revision and an AppArmor-ready guest kernel. - **NotRun in this loop:** source conversion and tuned-kernel build/boot/fallback - (multi-hour compiles on the disposable image); a live direct-provider model in - the VM (no credential/GPU); recovery-entry BLS selection (manual gate); - NVIDIA. + (multi-hour compiles on the disposable image); recovery-entry BLS selection + (manual gate); NVIDIA. +- **Live AI in the VM (`serial-boot-20260929T102703Z-ffd1ec25`, driver PASS=6):** + from inside the guest, the host's real `opencode` agent was reached through the + loopback tunnel (`fwd.py`). `sys ask "list installed packages" --ai-backend + opencode --opencode-provider opencode-go --opencode-model deepseek-v4.1-flash` + returned a valid Assistant Plan + digest (`live-agent` PASS); a hostile + shell/private-file instruction was refused/downgraded + (`live-agent-hostile-rejected` PASS). Separately, a real local model + (Qwen2.5-1.5B GGUF via `llama-server`) was driven through the direct sandbox in + both scopes (`serial-boot-20260929T101322Z-b3d002ed`); it reached the model and + failed closed on schema-noncompliant output. The assistant prompt now shows a + concrete action-object example, which made the small model emit action objects; + a capable model is still required for valid plans, and a very small local + context can be exceeded by the one-shot correction turn. ## 2026-09-28 Headless dual-target golden install (Run 13) & detached boot proven diff --git a/installer-rs/crates/phoebus-ai/src/assistant.rs b/installer-rs/crates/phoebus-ai/src/assistant.rs index 78a755b..78cc6f8 100644 --- a/installer-rs/crates/phoebus-ai/src/assistant.rs +++ b/installer-rs/crates/phoebus-ai/src/assistant.rs @@ -81,10 +81,12 @@ inspect_candidate, check_backend, inspect_installation, list_packages, \ package_info, package_owners, package_dependencies, depclean, clean_cache, \ check_updates, apply_updates. -Each reply is ONE JSON object with version 1 and a `kind`: +Each reply is ONE JSON object with version 1 and a `kind`. An action is always \ +an object with an `action` field — for example {\"action\":\"list_packages\"} or \ +{\"action\":\"enable_service\",\"name\":\"sshd\"} — never a bare string. 1. Investigate (read-only evidence): \ -{\"version\":1,\"kind\":\"investigate\",\"requests\":[,...]}. \ +{\"version\":1,\"kind\":\"investigate\",\"requests\":[{\"action\":\"check_backend\",\"backend\":\"flatpak\"}]}. \ Use this to answer search, inventory, information and update-check requests from \ real evidence. Useful tools: search_packages (query, optional backend), \ discover_candidates, inspect_candidate, check_backend, inspect_installation, \ @@ -92,8 +94,9 @@ list_packages (optional backend/filter), package_info, check_updates \ (optional backend), package_owners, package_dependencies. Never request a \ mutating action as evidence. -2. Plan or answer: {\"version\":1,\"summary\":\"...\",\"actions\":[...],\ -\"rationale\":[{\"action\":\"...\",\"reason\":\"...\"}]}. Give every action a \ +2. Plan or answer: {\"version\":1,\"summary\":\"...\",\"actions\":\ +[{\"action\":\"list_packages\"}],\ +\"rationale\":[{\"action\":\"list_packages\",\"reason\":\"...\"}]}. Give every action a \ non-empty reason. For a pure question or a search summary, return an empty \ `actions` list and put the answer in `summary`, grounded ONLY in the evidence \ you were given: never invent package names, versions, descriptions, sizes, \