feat(ai): show a concrete action example; record VM live-agent and local-model evidence
Some checks are pending
ci / test (push) Waiting to run
Some checks are pending
ci / test (push) Waiting to run
This commit is contained in:
parent
685321f85b
commit
d96aae1cfa
5 changed files with 53 additions and 8 deletions
|
|
@ -5,6 +5,23 @@ this file is the curated, human-readable summary.
|
||||||
|
|
||||||
## Unreleased — Fast defaults with user-controlled source builds
|
## Unreleased — Fast defaults with user-controlled source builds
|
||||||
|
|
||||||
|
- **VM live AI: real capable agent, and a local-model prompt fix (2026-09-29).**
|
||||||
|
Booted the golden overlay and, from inside the guest, reached the host's real
|
||||||
|
`opencode` agent through the loopback tunnel: `sys ask "list installed
|
||||||
|
packages" --ai-backend opencode --opencode-provider opencode-go
|
||||||
|
--opencode-model deepseek-v4.1-flash` returned a valid Assistant Plan + digest
|
||||||
|
(`CASE live-agent PASS`), and a hostile shell/private-file instruction was
|
||||||
|
refused/downgraded (`CASE live-agent-hostile-rejected PASS`)
|
||||||
|
(`serial-boot-20260929T102703Z-ffd1ec25`). A real local model (Qwen2.5-1.5B
|
||||||
|
GGUF served by `llama-server`) was also driven through the direct sandbox in
|
||||||
|
both scopes; it reached the model, but the 1.5B model could not satisfy the
|
||||||
|
exact typed schema (bare-string action, then a missing `rationale` reason) and
|
||||||
|
the correction prompt exceeded its 4096-token context, so `sys` failed closed
|
||||||
|
— model capacity, not a `sys` defect. The assistant prompt now shows the
|
||||||
|
concrete action-object shape (`{"action":"list_packages"}`) instead of only
|
||||||
|
`[...]` placeholders, which made the small model emit action objects. Gates:
|
||||||
|
`cargo fmt`, `clippy --workspace --all-targets -- -D warnings`,
|
||||||
|
`cargo test --workspace` 843 passed / 0 failed.
|
||||||
- **VM-verified AI sandbox and unprivileged AI path (2026-09-28).** The first
|
- **VM-verified AI sandbox and unprivileged AI path (2026-09-28).** The first
|
||||||
disposable-VM run of the newly built static `sys` (overlay on the golden
|
disposable-VM run of the newly built static `sys` (overlay on the golden
|
||||||
installed image, `serial-boot-20260929T092147Z-6f88907a`) found and fixed two
|
installed image, `serial-boot-20260929T092147Z-6f88907a`) found and fixed two
|
||||||
|
|
|
||||||
|
|
@ -25,6 +25,18 @@ commit it. It should be short and current; use `git log` for detail.
|
||||||
call (`sys ask` with a real provider is still VM/live NotRun) but it closes
|
call (`sys ask` with a real provider is still VM/live NotRun) but it closes
|
||||||
the "connected-model path only tested with fake transports" gap. Gates:
|
the "connected-model path only tested with fake transports" gap. Gates:
|
||||||
`cargo test --workspace` 841 passed / 0 failed.
|
`cargo test --workspace` 841 passed / 0 failed.
|
||||||
|
- **VM live AI (2026-09-29, VM-VERIFIED):** from inside the installed guest,
|
||||||
|
the host's real `opencode` agent was reached through the loopback tunnel and
|
||||||
|
`sys ask` produced a valid plan (`live-agent`), with a hostile instruction
|
||||||
|
refused/downgraded (`live-agent-hostile-rejected`)
|
||||||
|
(`serial-boot-20260929T102703Z-ffd1ec25`). A real local model (Qwen2.5-1.5B
|
||||||
|
GGUF via `llama-server`) was also driven through the direct sandbox in both
|
||||||
|
scopes; it reached the model but failed closed on schema-noncompliant output.
|
||||||
|
The assistant prompt now shows a concrete action-object example
|
||||||
|
(`{"action":"list_packages"}`), which made the small model emit action
|
||||||
|
objects; a capable model is still required, and a 4096-token local context
|
||||||
|
can be exceeded by the one-shot correction turn. Source conversion and
|
||||||
|
tuned-kernel builds remain NotRun (multi-hour compiles).
|
||||||
- **First VM test loop on the golden image (2026-09-28, VM-VERIFIED):** booted a
|
- **First VM test loop on the golden image (2026-09-28, VM-VERIFIED):** booted a
|
||||||
fresh overlay of the installed `phoebus-headless-80g.qcow2` with the static
|
fresh overlay of the installed `phoebus-headless-80g.qcow2` with the static
|
||||||
`sys` from HEAD and found two real defects, both fixed. (1) Unprivileged
|
`sys` from HEAD and found two real defects, both fixed. (1) Unprivileged
|
||||||
|
|
|
||||||
|
|
@ -98,7 +98,8 @@ Booting that recovery entry offline remains a manual gate.
|
||||||
| Provider credential + endpoint boundary | yes | yes (unsafe key characters refused before curl config; userinfo endpoint refused; derived base64 token redacted) | no | n/a |
|
| Provider credential + endpoint boundary | yes | yes (unsafe key characters refused before curl config; userinfo endpoint refused; derived base64 token redacted) | no | n/a |
|
||||||
| AI typed-action schema (exact, incl. unit-variant extra keys) | yes | yes (plan/step/investigate actions via `validate_action_value`; smuggled extra key refused) | no | n/a |
|
| AI typed-action schema (exact, incl. unit-variant extra keys) | yes | yes (plan/step/investigate actions via `validate_action_value`; smuggled extra key refused) | no | n/a |
|
||||||
| AI connected-model path over a real local stub | yes | yes (4 e2e cases: valid plan executes; request has no home path/credential; shell + smuggled key + malformed fail closed) | no (local stub, not a live model) | n/a |
|
| AI connected-model path over a real local stub | yes | yes (4 e2e cases: valid plan executes; request has no home path/credential; shell + smuggled key + malformed fail closed) | no (local stub, not a live model) | n/a |
|
||||||
| Live real-agent call (`opencode` backend, host) | yes | yes (`sys ask "list installed packages"` → valid plan + digest, exit 0; hostile shell/private-file instruction refused and downgraded; OpenCode tools disabled; no credential in the log) | host-level only (2026-09-28, local `opencode serve` + `opencode-go/deepseek-v4.1-flash`); VM/direct-provider live call remains NotRun | n/a |
|
| Live real-agent call (`opencode` backend) | yes | yes (`sys ask "list installed packages"` → valid plan + digest, exit 0; hostile shell/private-file instruction refused and downgraded; OpenCode tools disabled; no credential in the log) | yes — host direct (2026-09-28) and **from inside the installed VM through the loopback tunnel** (`serial-boot-20260929T102703Z-ffd1ec25`, `live-agent` + `live-agent-hostile-rejected`) with `opencode-go/deepseek-v4.1-flash` | n/a |
|
||||||
|
| Local model through the direct sandbox (VM) | yes | yes (real Qwen2.5-1.5B GGUF via `llama-server`; both sandbox scopes reached the model; schema-noncompliant output failed closed) | VM-run 2026-09-29: reached and failed closed; a 1.5B model is too weak for exact typed plans and its 4096-token context was exceeded by the correction turn — a capable model is required (a capable agent produces valid plans) | n/a |
|
||||||
| Privilege derived only from the process identity | yes | yes (an environment variable cannot forge uid 0; the removed `PHOEBUS_ALLOW_UNPRIVILEGED_EXEC` regression test) | no | n/a |
|
| Privilege derived only from the process identity | yes | yes (an environment variable cannot forge uid 0; the removed `PHOEBUS_ALLOW_UNPRIVILEGED_EXEC` regression test) | no | n/a |
|
||||||
| AI OS sandbox (systemd hardened unit) | yes | yes (argv + sandboxed-send; system properties for root, user-manager subset for non-root) | yes (`serial-boot-20260929T092147Z-6f88907a`: `AI_ROOT_RC=0` system sandbox, `AI_USER_RC=0` `--user` sandbox; previously the user path hung on a polkit `manage-units` prompt) | no |
|
| AI OS sandbox (systemd hardened unit) | yes | yes (argv + sandboxed-send; system properties for root, user-manager subset for non-root) | yes (`serial-boot-20260929T092147Z-6f88907a`: `AI_ROOT_RC=0` system sandbox, `AI_USER_RC=0` `--user` sandbox; previously the user path hung on a polkit `manage-units` prompt) | no |
|
||||||
| Unprivileged read-only AI plan completes | yes | yes (transaction recording best-effort for non-root; mandatory for privileged mutations) | yes (`serial-boot-20260929T092147Z-6f88907a`: `AI_NOSANDBOX_RC=0` as `phoebus`) | n/a |
|
| Unprivileged read-only AI plan completes | yes | yes (transaction recording best-effort for non-root; mandatory for privileged mutations) | yes (`serial-boot-20260929T092147Z-6f88907a`: `AI_NOSANDBOX_RC=0` as `phoebus`) | n/a |
|
||||||
|
|
|
||||||
|
|
@ -45,9 +45,21 @@ the static `sys` built from HEAD; the base image was never modified.
|
||||||
revision-changing refresh / strict AppArmor confinement still needs an
|
revision-changing refresh / strict AppArmor confinement still needs an
|
||||||
external Store revision and an AppArmor-ready guest kernel.
|
external Store revision and an AppArmor-ready guest kernel.
|
||||||
- **NotRun in this loop:** source conversion and tuned-kernel build/boot/fallback
|
- **NotRun in this loop:** source conversion and tuned-kernel build/boot/fallback
|
||||||
(multi-hour compiles on the disposable image); a live direct-provider model in
|
(multi-hour compiles on the disposable image); recovery-entry BLS selection
|
||||||
the VM (no credential/GPU); recovery-entry BLS selection (manual gate);
|
(manual gate); NVIDIA.
|
||||||
NVIDIA.
|
- **Live AI in the VM (`serial-boot-20260929T102703Z-ffd1ec25`, driver PASS=6):**
|
||||||
|
from inside the guest, the host's real `opencode` agent was reached through the
|
||||||
|
loopback tunnel (`fwd.py`). `sys ask "list installed packages" --ai-backend
|
||||||
|
opencode --opencode-provider opencode-go --opencode-model deepseek-v4.1-flash`
|
||||||
|
returned a valid Assistant Plan + digest (`live-agent` PASS); a hostile
|
||||||
|
shell/private-file instruction was refused/downgraded
|
||||||
|
(`live-agent-hostile-rejected` PASS). Separately, a real local model
|
||||||
|
(Qwen2.5-1.5B GGUF via `llama-server`) was driven through the direct sandbox in
|
||||||
|
both scopes (`serial-boot-20260929T101322Z-b3d002ed`); it reached the model and
|
||||||
|
failed closed on schema-noncompliant output. The assistant prompt now shows a
|
||||||
|
concrete action-object example, which made the small model emit action objects;
|
||||||
|
a capable model is still required for valid plans, and a very small local
|
||||||
|
context can be exceeded by the one-shot correction turn.
|
||||||
|
|
||||||
## 2026-09-28 Headless dual-target golden install (Run 13) & detached boot proven
|
## 2026-09-28 Headless dual-target golden install (Run 13) & detached boot proven
|
||||||
|
|
||||||
|
|
|
||||||
|
|
@ -81,10 +81,12 @@ inspect_candidate, check_backend, inspect_installation, list_packages, \
|
||||||
package_info, package_owners, package_dependencies, depclean, clean_cache, \
|
package_info, package_owners, package_dependencies, depclean, clean_cache, \
|
||||||
check_updates, apply_updates.
|
check_updates, apply_updates.
|
||||||
|
|
||||||
Each reply is ONE JSON object with version 1 and a `kind`:
|
Each reply is ONE JSON object with version 1 and a `kind`. An action is always \
|
||||||
|
an object with an `action` field — for example {\"action\":\"list_packages\"} or \
|
||||||
|
{\"action\":\"enable_service\",\"name\":\"sshd\"} — never a bare string.
|
||||||
|
|
||||||
1. Investigate (read-only evidence): \
|
1. Investigate (read-only evidence): \
|
||||||
{\"version\":1,\"kind\":\"investigate\",\"requests\":[<read-only action>,...]}. \
|
{\"version\":1,\"kind\":\"investigate\",\"requests\":[{\"action\":\"check_backend\",\"backend\":\"flatpak\"}]}. \
|
||||||
Use this to answer search, inventory, information and update-check requests from \
|
Use this to answer search, inventory, information and update-check requests from \
|
||||||
real evidence. Useful tools: search_packages (query, optional backend), \
|
real evidence. Useful tools: search_packages (query, optional backend), \
|
||||||
discover_candidates, inspect_candidate, check_backend, inspect_installation, \
|
discover_candidates, inspect_candidate, check_backend, inspect_installation, \
|
||||||
|
|
@ -92,8 +94,9 @@ list_packages (optional backend/filter), package_info, check_updates \
|
||||||
(optional backend), package_owners, package_dependencies. Never request a \
|
(optional backend), package_owners, package_dependencies. Never request a \
|
||||||
mutating action as evidence.
|
mutating action as evidence.
|
||||||
|
|
||||||
2. Plan or answer: {\"version\":1,\"summary\":\"...\",\"actions\":[...],\
|
2. Plan or answer: {\"version\":1,\"summary\":\"...\",\"actions\":\
|
||||||
\"rationale\":[{\"action\":\"...\",\"reason\":\"...\"}]}. Give every action a \
|
[{\"action\":\"list_packages\"}],\
|
||||||
|
\"rationale\":[{\"action\":\"list_packages\",\"reason\":\"...\"}]}. Give every action a \
|
||||||
non-empty reason. For a pure question or a search summary, return an empty \
|
non-empty reason. For a pure question or a search summary, return an empty \
|
||||||
`actions` list and put the answer in `summary`, grounded ONLY in the evidence \
|
`actions` list and put the answer in `summary`, grounded ONLY in the evidence \
|
||||||
you were given: never invent package names, versions, descriptions, sizes, \
|
you were given: never invent package names, versions, descriptions, sizes, \
|
||||||
|
|
|
||||||
Loading…
Add table
Reference in a new issue