From 359f083bc2924b483fa632bcae44b6debab52a22 Mon Sep 17 00:00:00 2001 From: Phoebus Date: Tue, 22 Sep 2026 03:32:07 -0700 Subject: [PATCH] test(gui): drive the OpenCode AI step through the real UI Add an 'opencode' harness mode with a guest-loopback relay (the tunnel shape), keyboard selection of the OpenCode chip, UI entry of the server/ provider/model (and a 0600 server password), and assertions on the frontend's secret-free state (tested, validated, applied). Record the VM evidence and the remaining not-run full-install gate. --- docs/CHANGELOG.md | 13 ++ docs/HANDOFF.md | 13 +- docs/STATUS.md | 3 +- docs/VERIFICATION.md | 25 +++ .../live/desktop/phoebus-install/shell.qml | 7 +- installer/tests/vm/gui-driver.py | 187 +++++++++++++++++- installer/tests/vm/oc-forward.py | 71 +++++++ 7 files changed, 310 insertions(+), 9 deletions(-) create mode 100644 installer/tests/vm/oc-forward.py diff --git a/docs/CHANGELOG.md b/docs/CHANGELOG.md index 10266cc..fd67700 100644 --- a/docs/CHANGELOG.md +++ b/docs/CHANGELOG.md @@ -49,7 +49,20 @@ Format: Keep a Changelog. Dates are UTC. server responses that echo credentials are not retained, and tests feed credential-bearing bodies and assert they do not appear in errors or state. +### Verified +- **The graphical installer uses the OpenCode adapter.** In a disposable + graphical VM the wizard selected "OpenCode agent", entered the server, + provider and model through the UI, obtained a real model response (Test + connection) and a validated proposal (Connect and validate, green + "Validated"). The `opencode` harness run passed 17/17 and the GUI regression + 25/25. A password-protected server was reached through authentication as + well. The full graphical installation beyond the AI gate is still not run. + ### Fixed +- **A malformed model proposal now gets one bounded corrective retry** before + the strict parser rejects the interaction (the shipped DeepSeek models + occasionally keyed the change or rationale wrongly). The parser remains the + final authority; a retry that still fails is still rejected. - **OpenCode client/probe defects found by testing:** the handwritten HTTP reader (no total deadline, UTF-8-unsafe chunk fallback) was removed in favour of the curl transport; the catalog is no longer parsed on a non-2xx status; a diff --git a/docs/HANDOFF.md b/docs/HANDOFF.md index 2b9f673..4a7a7a9 100644 --- a/docs/HANDOFF.md +++ b/docs/HANDOFF.md @@ -23,9 +23,16 @@ commit it. It should be short and current; use `git log` for detail. → `insecure_transport`, exit 1, no request sent. - 7 real-HTTP integration tests (status, chunked multibyte, oversize, redirect, disconnect, auth) and 2 env-gated live tests. -- **Not yet verified this session:** the full graphical install through - OpenCode in a VM, and a separate physical computer. The pinned manifest is - `v1.18.31` while the tested binary is `1.18.32`. + - GUI acceptance in a disposable VM: `gui-driver.py opencode` passed 17/17 + (selected OpenCode, entered the endpoint/model through the UI, real model + response + validated proposal); `gui-driver.py gui` passed 25/25. + ISO `sha256 4db98db3…`, static binary `sha256 72c2dcd0…`. +- **Not yet verified this session:** the full graphical **installation** through + OpenCode past the AI gate (disk/account/review/ERASE/install/detached boot), + and a separate physical computer. The pinned manifest is `v1.18.31` while the + tested binary is `1.18.32`. On this server use `deepseek-flash`; the model + must match the server's catalog (the GUI default `deepseek-chat` does not + exist on it). - **Public repo:** (remote `origin`, SSH port 2222). diff --git a/docs/STATUS.md b/docs/STATUS.md index 565f21e..359d1d7 100644 --- a/docs/STATUS.md +++ b/docs/STATUS.md @@ -101,7 +101,8 @@ still not VM-verified. | `TargetRoot` symlink-escape guard | yes | yes | n/a | n/a | | OpenCode server capability probe | yes | yes | real host probe + auth + policy | n/a | | OpenCode proposal adapter (real inference) | yes | yes | real model (`deepseek-v4-flash`) | n/a | -| OpenCode in GUI / CLI / `sys ask` | yes | yes | bridge + `sys ask` live | n/a | +| OpenCode in GUI / CLI / `sys ask` | yes | yes | GUI AI step + `sys ask` live | n/a | +| OpenCode full graphical install | yes | yes | AI gate passed; install not run | n/a | | Mandatory AI requires a validated interaction | yes | yes | bridge + CLI | n/a | | GURU packaging (scoped keywords) | yes | yes | no (build/install unverified) | no | | Release gates + atomic publish | yes | yes | no | n/a | diff --git a/docs/VERIFICATION.md b/docs/VERIFICATION.md index 9b43528..aace219 100644 --- a/docs/VERIFICATION.md +++ b/docs/VERIFICATION.md @@ -25,6 +25,31 @@ installer CLI and `sys ask`. Defects found while implementing and testing: | O11 | The proposal prompt allowed the model to key the change by action name | prompt now requires exactly `action` + `value` with explicit examples | live bridge `ai_check` | | O12 | The GUI duplicate-session assertion accepted two pre-existing sessions | requires exactly one session before and after | harness | | O13 | GUI navigation was inferred from a surviving process/screenshot | the frontend emits secret-free `phoebus-state` lines; the harness asserts the current step and AI state | harness | +| O14 | A model could return a malformed proposal with no second chance | one bounded corrective retry in `complete_proposal_at`/`complete_plan`, with the strict parser still final | `a_rejected_proposal_is_retried_once_with_a_correction`, `a_retry_that_still_fails_is_still_rejected` | + +### Graphical OpenCode acceptance (VM verified) + +`gui-driver.py opencode` on the built ISO (real model, no simulation): + +- welcome → installer → AI step; selected **OpenCode agent** through the UI; +- typed the server URL, provider and model through the UI; +- **Test connection** obtained a real model response (green "Model responded"); +- **Connect and validate** obtained and validated a proposal (green "Validated", + frontend state `aiConnected=true`); +- run `gui-opencode-20260922T102640Z-be168813`: **17/17 PASS**, exit 0; the full + GUI regression `gui-gui-20260922T102907Z-a9e5e824` was **25/25 PASS**. +- The model is reached through a guest-loopback relay to the host OpenCode + server (the SSH-tunnel shape), so the transport policy is satisfied. +- Authenticated server: the same harness with a password-protected server got + through authentication and a real Test connection; a flaky malformed proposal + was correctly rejected. Use `deepseek-flash` (reliable) over + `deepseek-v4-flash` (occasionally malformed). + +Still **NOT RUN**: the full graphical installation (disk → account → review → +ERASE → install → detached boot) *through OpenCode*. The AI gate is now reached +and passed in the graphical VM; the remaining stages are the long package +installation. Two fresh-disk graphical installs and a separate physical machine +remain unverified. OpenCode protocol/transport tests: 26 executor unit tests + 7 real-HTTP integration tests + 2 live tests (skipped offline, real inference when diff --git a/installer/live/desktop/phoebus-install/shell.qml b/installer/live/desktop/phoebus-install/shell.qml index de3a0e0..7512ba8 100644 --- a/installer/live/desktop/phoebus-install/shell.qml +++ b/installer/live/desktop/phoebus-install/shell.qml @@ -85,12 +85,17 @@ ShellRoot { + " step=" + shell.step + " phase=" + shell.phase + " aiConnected=" + shell.aiConnected - + " aiError=" + shell.aiError.replace(/\n/g, " ").slice(0, 200)); + + " aiTested=" + shell.aiTested + + " aiApplied=" + shell.aiApplied.length + + " aiError=" + shell.aiError.replace(/\n/g, " ").slice(0, 200) + + " aiTest=" + shell.aiTestMessage.replace(/\n/g, " ").slice(0, 200)); } onStepChanged: logState("step") onPhaseChanged: logState("phase") onAiConnectedChanged: logState("ai") onAiErrorChanged: logState("error") + onAiTestedChanged: logState("test") + onAiTestMessageChanged: logState("testmsg") // QML `var` properties do not emit change signals for nested assignments, // so bindings that read `cfg.*` would not update. Reassigning a clone diff --git a/installer/tests/vm/gui-driver.py b/installer/tests/vm/gui-driver.py index 4ba6e9a..9ea8663 100644 --- a/installer/tests/vm/gui-driver.py +++ b/installer/tests/vm/gui-driver.py @@ -59,6 +59,15 @@ BOOT_TIMEOUT = int(os.environ.get("BOOT_TIMEOUT", "900")) INSTALL_TIMEOUT = int(os.environ.get("INSTALL_TIMEOUT", "10800")) SESSION_SETTLE = int(os.environ.get("SESSION_SETTLE", "20")) BACKEND_SETTLE = int(os.environ.get("BACKEND_SETTLE", "30")) +# OpenCode GUI-driving knobs (test-only). Tab counts assume the focus order: +# Close, DeepSeek, OpenAI, Local, OpenCode, server, password, provider, model, +# Test, Connect, Back, Next. +OC_PORT = int(os.environ.get("OC_PORT", "4096")) +OC_TARGET_PORT = int(os.environ.get("OC_TARGET_PORT", "41999")) +OC_TAB_TO_CHIP = int(os.environ.get("OC_TAB_TO_CHIP", "5")) +OC_PROVIDER = os.environ.get("OC_PROVIDER", "deepseek") +OC_MODEL = os.environ.get("OC_MODEL", "deepseek-v4-flash") +OC_USE_PASSWORD = os.environ.get("OC_USE_PASSWORD", "0") in ("1", "yes", "true") def _template_dir() -> Path: @@ -293,6 +302,8 @@ class Driver: try: if self.mode == "gui": self.do_gui() + elif self.mode == "opencode": + self.do_opencode() elif self.mode == "install": self.do_install() elif self.mode == "boot": @@ -537,7 +548,8 @@ class Driver: self.child.expect("STATE-END", timeout=30) except (pexpect.TIMEOUT, pexpect.EOF): return None - step = ai = None + step = ai = tested = None + applied = 0 err = "" for line in (self.child.before or "").splitlines(): if "phoebus-state" not in line: @@ -548,12 +560,179 @@ class Driver: match = re.search(r"aiConnected=(true|false)", line) if match: ai = match.group(1) == "true" - match = re.search(r"aiError=(.*)$", line) + match = re.search(r"aiTested=(true|false)", line) + if match: + tested = match.group(1) == "true" + match = re.search(r"aiApplied=(\d+)", line) + if match: + applied = int(match.group(1)) + match = re.search(r"aiError=(.*?) aiTest=", line) if match: err = match.group(1).strip() if step is None: return None - return {"step": step, "aiConnected": bool(ai), "aiError": err} + return { + "step": step, + "aiConnected": bool(ai), + "aiTested": bool(tested), + "aiApplied": applied, + "aiError": err, + } + + def wait_state(self, predicate, timeout: int): + deadline = time.monotonic() + timeout + while time.monotonic() < deadline: + state = self._frontend_state() + if state is not None and predicate(state): + return state + time.sleep(2) + return None + + def do_opencode(self) -> None: + """Drive the OpenCode AI step through the real graphical UI. + + The model is reached from the guest through a loopback relay (the same + shape as an SSH tunnel), so the connection satisfies the transport + policy while staying host-local or host+LAN only. + """ + self.spawn(attach_iso=True, boot_disk=False) + self.login_live() + welcome = self.poll_shell( + "echo WELCOME=$(pgrep -fc 'quickshell.*phoebus-welcome' 2>/dev/null || echo 0)", + r"WELCOME=([1-9][0-9]*)", timeout=int(os.environ.get("WELCOME_TIMEOUT", "180")), + label="welcome-window", + ) + self.results.assert_true("welcome-autostarted", welcome is not None, "welcome did not appear") + self.set_wayland() + + # Start the guest-local relay to the host OpenCode server. + self.shell( + f"curl -fsSL {SERVE_URL}/oc-forward.py -o /tmp/oc-forward.py && " + f"chmod 755 /tmp/oc-forward.py && " + f"setsid python3 /tmp/oc-forward.py {OC_PORT} 10.0.2.2 {OC_TARGET_PORT} " + f">/tmp/oc-forward.log 2>&1 < /dev/null & echo FWD-$(printf START)", + "FWD-START", timeout=60, label="forwarder-start", + ) + ready = self.poll_shell( + f"curl -s -m 5 -o /dev/null -w 'HEALTH=%{{http_code}}' " + f"http://127.0.0.1:{OC_PORT}/global/health", + r"HEALTH=(200|401)", timeout=60, label="forwarder-ready", + ) + self.results.assert_true( + "opencode-reachable-from-guest", ready is not None, + "the guest relay could not reach the OpenCode server", + ) + + self.sendkey("ret") + opened = self.poll_shell( + "echo INST=$(pgrep -fc 'quickshell.*phoebus-install' 2>/dev/null || echo 0)", + r"INST=([1-9][0-9]*)", timeout=int(os.environ.get("INSTALLER_OPEN_TIMEOUT", "90")), + label="launcher-opens-installer", record=False, + ) + if opened is None: + self.results.fail("launcher-opens-installer", "Return did not open the installer") + return + self.results.pass_("launcher-opens-installer", "welcome action opened the installer") + time.sleep(BACKEND_SETTLE) + self.shell( + "pkill -f 'quickshell.*phoebus-wel[c]ome' || true; sleep 2; echo CLOSED", + "CLOSED", timeout=30, label="close-welcome", + ) + + # Navigate to the AI step (0 -> 1 -> 2). + self.sendkey("ctrl-right") + time.sleep(2) + self.sendkey("ctrl-right") + time.sleep(2) + state = self._frontend_state() + self.results.assert_true( + "ai-step-reached", state is not None and state["step"] == 2, + f"frontend state: {state}", + ) + self.shot("oc-01-ai", "AI step", "Ctrl+Right twice", "the AI model step is shown") + + # Select the "OpenCode agent" chip with the keyboard. + for _ in range(OC_TAB_TO_CHIP): + self.sendkey("tab") + time.sleep(0.3) + self.sendkey("ret") + time.sleep(1) + self.shot("oc-02-selected", "OpenCode selected", "Tab to the chip and Return", + "server, password, provider and model fields appear") + + # Enter the connection through the UI. Focus is on the chip after + # selection; Tab order is server, password, provider, model, Test, + # Connect. Ctrl+A selects the existing default before typing. + def type_field(tabs: int, value: str, secret_file: str | None = None) -> None: + # `wtype` (Wayland) and QEMU `sendkey` (hardware) are two input + # paths; settle between them or the next key can truncate the text. + for _ in range(tabs): + self.sendkey("tab") + time.sleep(0.4) + self.sendkey("ctrl-a") + time.sleep(0.5) + if secret_file: + # The serial echo carries `$(cat ...)`, never the value. + self.shell( + f'wtype "$(cat {secret_file})" >/dev/null 2>&1; echo TYPED', + "TYPED", timeout=30, label="type-secret", + ) + else: + self.shell( + f"wtype '{value}' >/dev/null 2>&1; echo TYPED", + "TYPED", timeout=30, label="type-field", + ) + time.sleep(1.4) + # Clear the selection highlight so a screenshot shows the value. + self.sendkey("end") + time.sleep(0.3) + + type_field(1, f"http://127.0.0.1:{OC_PORT}") + if OC_USE_PASSWORD: + self.shell( + f"curl -fsSL {SERVE_URL}/oc.pw -o /tmp/oc.pw && chmod 600 /tmp/oc.pw " + "&& echo PW-$(printf LOADED)", + "PW-LOADED", timeout=30, label="fetch-password", + ) + type_field(1, "", secret_file="/tmp/oc.pw") + else: + self.sendkey("tab") # skip the empty password field + time.sleep(0.25) + type_field(1, OC_PROVIDER) + type_field(1, OC_MODEL) + self.shot("oc-02b-fields", "OpenCode fields", "typed server/provider/model", + "the fields hold the entered values") + self.sendkey("tab") # model -> Test + time.sleep(0.3) + self.sendkey("ret") + time.sleep(3) + self.shot("oc-03-test", "Test connection", "Return on Test connection", + "a real model response is obtained and validated") + tested = self.wait_state(lambda s: s["aiTested"], timeout=180) + self.results.assert_true( + "opencode-test-connection", tested is not None, + "Test connection did not obtain a valid model response", + ) + + # The next focus is the "Connect and validate" button. + self.sendkey("tab") + time.sleep(0.3) + self.sendkey("ret") + self.shot("oc-04-connect", "Connect and validate", "Return on Connect and validate", + "a validated proposal is applied; Next is enabled") + connected = self.wait_state(lambda s: s["aiConnected"], timeout=240) + self.results.assert_true( + "opencode-validated-proposal", + connected is not None, + f"Connect and validate did not validate a proposal: {connected}", + ) + if connected is not None: + # A validated proposal may legitimately contain zero changes. + self.results.pass_( + "opencode-proposal-applied", + f"applied={connected['aiApplied']} change(s)", + ) + self.results.pass_("opencode-gui-completed") def _installer_alive(self) -> bool: assert self.child is not None @@ -683,7 +862,7 @@ class Driver: def main() -> int: mode = sys.argv[1] if len(sys.argv) > 1 else "" - if mode not in ("gui", "install", "boot"): + if mode not in ("gui", "opencode", "install", "boot"): print(__doc__) return 2 if mode in ("install", "boot") and not DISKS: diff --git a/installer/tests/vm/oc-forward.py b/installer/tests/vm/oc-forward.py new file mode 100644 index 0000000..d80aa18 --- /dev/null +++ b/installer/tests/vm/oc-forward.py @@ -0,0 +1,71 @@ +#!/usr/bin/env python3 +"""Guest-local TCP forwarder for OpenCode VM testing. + +The Phoebus transport policy allows plain `http://` only to a **loopback** +destination. A VM guest cannot name the host's loopback directly, and the QEMU +gateway address (`10.0.2.2`) is not loopback. This forwarder is the same shape +as an SSH tunnel: the GUI connects to the guest's own loopback, and the +forwarder relays to the host's OpenCode server over the QEMU host gateway. The +connection therefore never leaves the machine pair. + + oc-forward.py [listen_port] [target_host] [target_port] + +It is test infrastructure, not a product component. +""" + +from __future__ import annotations + +import socket +import sys +import threading + +LISTEN_HOST = "127.0.0.1" + + +def pipe(src: socket.socket, dst: socket.socket) -> None: + try: + while True: + data = src.recv(65536) + if not data: + break + dst.sendall(data) + except OSError: + pass + finally: + for sock in (src, dst): + try: + sock.shutdown(socket.SHUT_RDWR) + except OSError: + pass + + +def handle(client: socket.socket, target: tuple[str, int]) -> None: + try: + upstream = socket.create_connection(target) + except OSError: + client.close() + return + threading.Thread(target=pipe, args=(client, upstream), daemon=True).start() + threading.Thread(target=pipe, args=(upstream, client), daemon=True).start() + + +def main() -> int: + listen_port = int(sys.argv[1]) if len(sys.argv) > 1 else 4096 + target_host = sys.argv[2] if len(sys.argv) > 2 else "10.0.2.2" + target_port = int(sys.argv[3]) if len(sys.argv) > 3 else 41999 + server = socket.socket(socket.AF_INET, socket.SOCK_STREAM) + server.setsockopt(socket.SOL_SOCKET, socket.SO_REUSEADDR, 1) + server.bind((LISTEN_HOST, listen_port)) + server.listen(32) + print(f"oc-forward listening on {LISTEN_HOST}:{listen_port} -> {target_host}:{target_port}", flush=True) + while True: + try: + client, _ = server.accept() + except OSError: + break + threading.Thread(target=handle, args=(client, (target_host, target_port)), daemon=True).start() + return 0 + + +if __name__ == "__main__": + sys.exit(main())