test(gui): drive the OpenCode AI step through the real UI
Some checks are pending
ci / test (push) Waiting to run

Add an 'opencode' harness mode with a guest-loopback relay (the tunnel
shape), keyboard selection of the OpenCode chip, UI entry of the server/
provider/model (and a 0600 server password), and assertions on the
frontend's secret-free state (tested, validated, applied). Record the VM
evidence and the remaining not-run full-install gate.
This commit is contained in:
Phoebus 2026-09-22 03:32:07 -07:00
parent 802b843bc9
commit 359f083bc2
7 changed files with 310 additions and 9 deletions

View file

@ -49,7 +49,20 @@ Format: Keep a Changelog. Dates are UTC.
server responses that echo credentials are not retained, and tests feed server responses that echo credentials are not retained, and tests feed
credential-bearing bodies and assert they do not appear in errors or state. credential-bearing bodies and assert they do not appear in errors or state.
### Verified
- **The graphical installer uses the OpenCode adapter.** In a disposable
graphical VM the wizard selected "OpenCode agent", entered the server,
provider and model through the UI, obtained a real model response (Test
connection) and a validated proposal (Connect and validate, green
"Validated"). The `opencode` harness run passed 17/17 and the GUI regression
25/25. A password-protected server was reached through authentication as
well. The full graphical installation beyond the AI gate is still not run.
### Fixed ### Fixed
- **A malformed model proposal now gets one bounded corrective retry** before
the strict parser rejects the interaction (the shipped DeepSeek models
occasionally keyed the change or rationale wrongly). The parser remains the
final authority; a retry that still fails is still rejected.
- **OpenCode client/probe defects found by testing:** the handwritten HTTP - **OpenCode client/probe defects found by testing:** the handwritten HTTP
reader (no total deadline, UTF-8-unsafe chunk fallback) was removed in favour reader (no total deadline, UTF-8-unsafe chunk fallback) was removed in favour
of the curl transport; the catalog is no longer parsed on a non-2xx status; a of the curl transport; the catalog is no longer parsed on a non-2xx status; a

View file

@ -23,9 +23,16 @@ commit it. It should be short and current; use `git log` for detail.
→ `insecure_transport`, exit 1, no request sent. → `insecure_transport`, exit 1, no request sent.
- 7 real-HTTP integration tests (status, chunked multibyte, oversize, - 7 real-HTTP integration tests (status, chunked multibyte, oversize,
redirect, disconnect, auth) and 2 env-gated live tests. redirect, disconnect, auth) and 2 env-gated live tests.
- **Not yet verified this session:** the full graphical install through - GUI acceptance in a disposable VM: `gui-driver.py opencode` passed 17/17
OpenCode in a VM, and a separate physical computer. The pinned manifest is (selected OpenCode, entered the endpoint/model through the UI, real model
`v1.18.31` while the tested binary is `1.18.32`. response + validated proposal); `gui-driver.py gui` passed 25/25.
ISO `sha256 4db98db3…`, static binary `sha256 72c2dcd0…`.
- **Not yet verified this session:** the full graphical **installation** through
OpenCode past the AI gate (disk/account/review/ERASE/install/detached boot),
and a separate physical computer. The pinned manifest is `v1.18.31` while the
tested binary is `1.18.32`. On this server use `deepseek-flash`; the model
must match the server's catalog (the GUI default `deepseek-chat` does not
exist on it).
- **Public repo:** <https://git.phoebus.club/phoebus/phoebus_os> - **Public repo:** <https://git.phoebus.club/phoebus/phoebus_os>
(remote `origin`, SSH port 2222). (remote `origin`, SSH port 2222).

View file

@ -101,7 +101,8 @@ still not VM-verified.
| `TargetRoot` symlink-escape guard | yes | yes | n/a | n/a | | `TargetRoot` symlink-escape guard | yes | yes | n/a | n/a |
| OpenCode server capability probe | yes | yes | real host probe + auth + policy | n/a | | OpenCode server capability probe | yes | yes | real host probe + auth + policy | n/a |
| OpenCode proposal adapter (real inference) | yes | yes | real model (`deepseek-v4-flash`) | n/a | | OpenCode proposal adapter (real inference) | yes | yes | real model (`deepseek-v4-flash`) | n/a |
| OpenCode in GUI / CLI / `sys ask` | yes | yes | bridge + `sys ask` live | n/a | | OpenCode in GUI / CLI / `sys ask` | yes | yes | GUI AI step + `sys ask` live | n/a |
| OpenCode full graphical install | yes | yes | AI gate passed; install not run | n/a |
| Mandatory AI requires a validated interaction | yes | yes | bridge + CLI | n/a | | Mandatory AI requires a validated interaction | yes | yes | bridge + CLI | n/a |
| GURU packaging (scoped keywords) | yes | yes | no (build/install unverified) | no | | GURU packaging (scoped keywords) | yes | yes | no (build/install unverified) | no |
| Release gates + atomic publish | yes | yes | no | n/a | | Release gates + atomic publish | yes | yes | no | n/a |

View file

@ -25,6 +25,31 @@ installer CLI and `sys ask`. Defects found while implementing and testing:
| O11 | The proposal prompt allowed the model to key the change by action name | prompt now requires exactly `action` + `value` with explicit examples | live bridge `ai_check` | | O11 | The proposal prompt allowed the model to key the change by action name | prompt now requires exactly `action` + `value` with explicit examples | live bridge `ai_check` |
| O12 | The GUI duplicate-session assertion accepted two pre-existing sessions | requires exactly one session before and after | harness | | O12 | The GUI duplicate-session assertion accepted two pre-existing sessions | requires exactly one session before and after | harness |
| O13 | GUI navigation was inferred from a surviving process/screenshot | the frontend emits secret-free `phoebus-state` lines; the harness asserts the current step and AI state | harness | | O13 | GUI navigation was inferred from a surviving process/screenshot | the frontend emits secret-free `phoebus-state` lines; the harness asserts the current step and AI state | harness |
| O14 | A model could return a malformed proposal with no second chance | one bounded corrective retry in `complete_proposal_at`/`complete_plan`, with the strict parser still final | `a_rejected_proposal_is_retried_once_with_a_correction`, `a_retry_that_still_fails_is_still_rejected` |
### Graphical OpenCode acceptance (VM verified)
`gui-driver.py opencode` on the built ISO (real model, no simulation):
- welcome → installer → AI step; selected **OpenCode agent** through the UI;
- typed the server URL, provider and model through the UI;
- **Test connection** obtained a real model response (green "Model responded");
- **Connect and validate** obtained and validated a proposal (green "Validated",
frontend state `aiConnected=true`);
- run `gui-opencode-20260922T102640Z-be168813`: **17/17 PASS**, exit 0; the full
GUI regression `gui-gui-20260922T102907Z-a9e5e824` was **25/25 PASS**.
- The model is reached through a guest-loopback relay to the host OpenCode
server (the SSH-tunnel shape), so the transport policy is satisfied.
- Authenticated server: the same harness with a password-protected server got
through authentication and a real Test connection; a flaky malformed proposal
was correctly rejected. Use `deepseek-flash` (reliable) over
`deepseek-v4-flash` (occasionally malformed).
Still **NOT RUN**: the full graphical installation (disk → account → review →
ERASE → install → detached boot) *through OpenCode*. The AI gate is now reached
and passed in the graphical VM; the remaining stages are the long package
installation. Two fresh-disk graphical installs and a separate physical machine
remain unverified.
OpenCode protocol/transport tests: 26 executor unit tests + 7 real-HTTP OpenCode protocol/transport tests: 26 executor unit tests + 7 real-HTTP
integration tests + 2 live tests (skipped offline, real inference when integration tests + 2 live tests (skipped offline, real inference when

View file

@ -85,12 +85,17 @@ ShellRoot {
+ " step=" + shell.step + " step=" + shell.step
+ " phase=" + shell.phase + " phase=" + shell.phase
+ " aiConnected=" + shell.aiConnected + " aiConnected=" + shell.aiConnected
+ " aiError=" + shell.aiError.replace(/\n/g, " ").slice(0, 200)); + " aiTested=" + shell.aiTested
+ " aiApplied=" + shell.aiApplied.length
+ " aiError=" + shell.aiError.replace(/\n/g, " ").slice(0, 200)
+ " aiTest=" + shell.aiTestMessage.replace(/\n/g, " ").slice(0, 200));
} }
onStepChanged: logState("step") onStepChanged: logState("step")
onPhaseChanged: logState("phase") onPhaseChanged: logState("phase")
onAiConnectedChanged: logState("ai") onAiConnectedChanged: logState("ai")
onAiErrorChanged: logState("error") onAiErrorChanged: logState("error")
onAiTestedChanged: logState("test")
onAiTestMessageChanged: logState("testmsg")
// QML `var` properties do not emit change signals for nested assignments, // QML `var` properties do not emit change signals for nested assignments,
// so bindings that read `cfg.*` would not update. Reassigning a clone // so bindings that read `cfg.*` would not update. Reassigning a clone

View file

@ -59,6 +59,15 @@ BOOT_TIMEOUT = int(os.environ.get("BOOT_TIMEOUT", "900"))
INSTALL_TIMEOUT = int(os.environ.get("INSTALL_TIMEOUT", "10800")) INSTALL_TIMEOUT = int(os.environ.get("INSTALL_TIMEOUT", "10800"))
SESSION_SETTLE = int(os.environ.get("SESSION_SETTLE", "20")) SESSION_SETTLE = int(os.environ.get("SESSION_SETTLE", "20"))
BACKEND_SETTLE = int(os.environ.get("BACKEND_SETTLE", "30")) BACKEND_SETTLE = int(os.environ.get("BACKEND_SETTLE", "30"))
# OpenCode GUI-driving knobs (test-only). Tab counts assume the focus order:
# Close, DeepSeek, OpenAI, Local, OpenCode, server, password, provider, model,
# Test, Connect, Back, Next.
OC_PORT = int(os.environ.get("OC_PORT", "4096"))
OC_TARGET_PORT = int(os.environ.get("OC_TARGET_PORT", "41999"))
OC_TAB_TO_CHIP = int(os.environ.get("OC_TAB_TO_CHIP", "5"))
OC_PROVIDER = os.environ.get("OC_PROVIDER", "deepseek")
OC_MODEL = os.environ.get("OC_MODEL", "deepseek-v4-flash")
OC_USE_PASSWORD = os.environ.get("OC_USE_PASSWORD", "0") in ("1", "yes", "true")
def _template_dir() -> Path: def _template_dir() -> Path:
@ -293,6 +302,8 @@ class Driver:
try: try:
if self.mode == "gui": if self.mode == "gui":
self.do_gui() self.do_gui()
elif self.mode == "opencode":
self.do_opencode()
elif self.mode == "install": elif self.mode == "install":
self.do_install() self.do_install()
elif self.mode == "boot": elif self.mode == "boot":
@ -537,7 +548,8 @@ class Driver:
self.child.expect("STATE-END", timeout=30) self.child.expect("STATE-END", timeout=30)
except (pexpect.TIMEOUT, pexpect.EOF): except (pexpect.TIMEOUT, pexpect.EOF):
return None return None
step = ai = None step = ai = tested = None
applied = 0
err = "" err = ""
for line in (self.child.before or "").splitlines(): for line in (self.child.before or "").splitlines():
if "phoebus-state" not in line: if "phoebus-state" not in line:
@ -548,12 +560,179 @@ class Driver:
match = re.search(r"aiConnected=(true|false)", line) match = re.search(r"aiConnected=(true|false)", line)
if match: if match:
ai = match.group(1) == "true" ai = match.group(1) == "true"
match = re.search(r"aiError=(.*)$", line) match = re.search(r"aiTested=(true|false)", line)
if match:
tested = match.group(1) == "true"
match = re.search(r"aiApplied=(\d+)", line)
if match:
applied = int(match.group(1))
match = re.search(r"aiError=(.*?) aiTest=", line)
if match: if match:
err = match.group(1).strip() err = match.group(1).strip()
if step is None: if step is None:
return None return None
return {"step": step, "aiConnected": bool(ai), "aiError": err} return {
"step": step,
"aiConnected": bool(ai),
"aiTested": bool(tested),
"aiApplied": applied,
"aiError": err,
}
def wait_state(self, predicate, timeout: int):
deadline = time.monotonic() + timeout
while time.monotonic() < deadline:
state = self._frontend_state()
if state is not None and predicate(state):
return state
time.sleep(2)
return None
def do_opencode(self) -> None:
"""Drive the OpenCode AI step through the real graphical UI.
The model is reached from the guest through a loopback relay (the same
shape as an SSH tunnel), so the connection satisfies the transport
policy while staying host-local or host+LAN only.
"""
self.spawn(attach_iso=True, boot_disk=False)
self.login_live()
welcome = self.poll_shell(
"echo WELCOME=$(pgrep -fc 'quickshell.*phoebus-welcome' 2>/dev/null || echo 0)",
r"WELCOME=([1-9][0-9]*)", timeout=int(os.environ.get("WELCOME_TIMEOUT", "180")),
label="welcome-window",
)
self.results.assert_true("welcome-autostarted", welcome is not None, "welcome did not appear")
self.set_wayland()
# Start the guest-local relay to the host OpenCode server.
self.shell(
f"curl -fsSL {SERVE_URL}/oc-forward.py -o /tmp/oc-forward.py && "
f"chmod 755 /tmp/oc-forward.py && "
f"setsid python3 /tmp/oc-forward.py {OC_PORT} 10.0.2.2 {OC_TARGET_PORT} "
f">/tmp/oc-forward.log 2>&1 < /dev/null & echo FWD-$(printf START)",
"FWD-START", timeout=60, label="forwarder-start",
)
ready = self.poll_shell(
f"curl -s -m 5 -o /dev/null -w 'HEALTH=%{{http_code}}' "
f"http://127.0.0.1:{OC_PORT}/global/health",
r"HEALTH=(200|401)", timeout=60, label="forwarder-ready",
)
self.results.assert_true(
"opencode-reachable-from-guest", ready is not None,
"the guest relay could not reach the OpenCode server",
)
self.sendkey("ret")
opened = self.poll_shell(
"echo INST=$(pgrep -fc 'quickshell.*phoebus-install' 2>/dev/null || echo 0)",
r"INST=([1-9][0-9]*)", timeout=int(os.environ.get("INSTALLER_OPEN_TIMEOUT", "90")),
label="launcher-opens-installer", record=False,
)
if opened is None:
self.results.fail("launcher-opens-installer", "Return did not open the installer")
return
self.results.pass_("launcher-opens-installer", "welcome action opened the installer")
time.sleep(BACKEND_SETTLE)
self.shell(
"pkill -f 'quickshell.*phoebus-wel[c]ome' || true; sleep 2; echo CLOSED",
"CLOSED", timeout=30, label="close-welcome",
)
# Navigate to the AI step (0 -> 1 -> 2).
self.sendkey("ctrl-right")
time.sleep(2)
self.sendkey("ctrl-right")
time.sleep(2)
state = self._frontend_state()
self.results.assert_true(
"ai-step-reached", state is not None and state["step"] == 2,
f"frontend state: {state}",
)
self.shot("oc-01-ai", "AI step", "Ctrl+Right twice", "the AI model step is shown")
# Select the "OpenCode agent" chip with the keyboard.
for _ in range(OC_TAB_TO_CHIP):
self.sendkey("tab")
time.sleep(0.3)
self.sendkey("ret")
time.sleep(1)
self.shot("oc-02-selected", "OpenCode selected", "Tab to the chip and Return",
"server, password, provider and model fields appear")
# Enter the connection through the UI. Focus is on the chip after
# selection; Tab order is server, password, provider, model, Test,
# Connect. Ctrl+A selects the existing default before typing.
def type_field(tabs: int, value: str, secret_file: str | None = None) -> None:
# `wtype` (Wayland) and QEMU `sendkey` (hardware) are two input
# paths; settle between them or the next key can truncate the text.
for _ in range(tabs):
self.sendkey("tab")
time.sleep(0.4)
self.sendkey("ctrl-a")
time.sleep(0.5)
if secret_file:
# The serial echo carries `$(cat ...)`, never the value.
self.shell(
f'wtype "$(cat {secret_file})" >/dev/null 2>&1; echo TYPED',
"TYPED", timeout=30, label="type-secret",
)
else:
self.shell(
f"wtype '{value}' >/dev/null 2>&1; echo TYPED",
"TYPED", timeout=30, label="type-field",
)
time.sleep(1.4)
# Clear the selection highlight so a screenshot shows the value.
self.sendkey("end")
time.sleep(0.3)
type_field(1, f"http://127.0.0.1:{OC_PORT}")
if OC_USE_PASSWORD:
self.shell(
f"curl -fsSL {SERVE_URL}/oc.pw -o /tmp/oc.pw && chmod 600 /tmp/oc.pw "
"&& echo PW-$(printf LOADED)",
"PW-LOADED", timeout=30, label="fetch-password",
)
type_field(1, "", secret_file="/tmp/oc.pw")
else:
self.sendkey("tab") # skip the empty password field
time.sleep(0.25)
type_field(1, OC_PROVIDER)
type_field(1, OC_MODEL)
self.shot("oc-02b-fields", "OpenCode fields", "typed server/provider/model",
"the fields hold the entered values")
self.sendkey("tab") # model -> Test
time.sleep(0.3)
self.sendkey("ret")
time.sleep(3)
self.shot("oc-03-test", "Test connection", "Return on Test connection",
"a real model response is obtained and validated")
tested = self.wait_state(lambda s: s["aiTested"], timeout=180)
self.results.assert_true(
"opencode-test-connection", tested is not None,
"Test connection did not obtain a valid model response",
)
# The next focus is the "Connect and validate" button.
self.sendkey("tab")
time.sleep(0.3)
self.sendkey("ret")
self.shot("oc-04-connect", "Connect and validate", "Return on Connect and validate",
"a validated proposal is applied; Next is enabled")
connected = self.wait_state(lambda s: s["aiConnected"], timeout=240)
self.results.assert_true(
"opencode-validated-proposal",
connected is not None,
f"Connect and validate did not validate a proposal: {connected}",
)
if connected is not None:
# A validated proposal may legitimately contain zero changes.
self.results.pass_(
"opencode-proposal-applied",
f"applied={connected['aiApplied']} change(s)",
)
self.results.pass_("opencode-gui-completed")
def _installer_alive(self) -> bool: def _installer_alive(self) -> bool:
assert self.child is not None assert self.child is not None
@ -683,7 +862,7 @@ class Driver:
def main() -> int: def main() -> int:
mode = sys.argv[1] if len(sys.argv) > 1 else "" mode = sys.argv[1] if len(sys.argv) > 1 else ""
if mode not in ("gui", "install", "boot"): if mode not in ("gui", "opencode", "install", "boot"):
print(__doc__) print(__doc__)
return 2 return 2
if mode in ("install", "boot") and not DISKS: if mode in ("install", "boot") and not DISKS:

View file

@ -0,0 +1,71 @@
#!/usr/bin/env python3
"""Guest-local TCP forwarder for OpenCode VM testing.
The Phoebus transport policy allows plain `http://` only to a **loopback**
destination. A VM guest cannot name the host's loopback directly, and the QEMU
gateway address (`10.0.2.2`) is not loopback. This forwarder is the same shape
as an SSH tunnel: the GUI connects to the guest's own loopback, and the
forwarder relays to the host's OpenCode server over the QEMU host gateway. The
connection therefore never leaves the machine pair.
oc-forward.py [listen_port] [target_host] [target_port]
It is test infrastructure, not a product component.
"""
from __future__ import annotations
import socket
import sys
import threading
LISTEN_HOST = "127.0.0.1"
def pipe(src: socket.socket, dst: socket.socket) -> None:
try:
while True:
data = src.recv(65536)
if not data:
break
dst.sendall(data)
except OSError:
pass
finally:
for sock in (src, dst):
try:
sock.shutdown(socket.SHUT_RDWR)
except OSError:
pass
def handle(client: socket.socket, target: tuple[str, int]) -> None:
try:
upstream = socket.create_connection(target)
except OSError:
client.close()
return
threading.Thread(target=pipe, args=(client, upstream), daemon=True).start()
threading.Thread(target=pipe, args=(upstream, client), daemon=True).start()
def main() -> int:
listen_port = int(sys.argv[1]) if len(sys.argv) > 1 else 4096
target_host = sys.argv[2] if len(sys.argv) > 2 else "10.0.2.2"
target_port = int(sys.argv[3]) if len(sys.argv) > 3 else 41999
server = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
server.setsockopt(socket.SOL_SOCKET, socket.SO_REUSEADDR, 1)
server.bind((LISTEN_HOST, listen_port))
server.listen(32)
print(f"oc-forward listening on {LISTEN_HOST}:{listen_port} -> {target_host}:{target_port}", flush=True)
while True:
try:
client, _ = server.accept()
except OSError:
break
threading.Thread(target=handle, args=(client, (target_host, target_port)), daemon=True).start()
return 0
if __name__ == "__main__":
sys.exit(main())