test(gui): drive the OpenCode AI step through the real UI
Some checks are pending
ci / test (push) Waiting to run
Some checks are pending
ci / test (push) Waiting to run
Add an 'opencode' harness mode with a guest-loopback relay (the tunnel shape), keyboard selection of the OpenCode chip, UI entry of the server/ provider/model (and a 0600 server password), and assertions on the frontend's secret-free state (tested, validated, applied). Record the VM evidence and the remaining not-run full-install gate.
This commit is contained in:
parent
802b843bc9
commit
359f083bc2
7 changed files with 310 additions and 9 deletions
|
|
@ -49,7 +49,20 @@ Format: Keep a Changelog. Dates are UTC.
|
|||
server responses that echo credentials are not retained, and tests feed
|
||||
credential-bearing bodies and assert they do not appear in errors or state.
|
||||
|
||||
### Verified
|
||||
- **The graphical installer uses the OpenCode adapter.** In a disposable
|
||||
graphical VM the wizard selected "OpenCode agent", entered the server,
|
||||
provider and model through the UI, obtained a real model response (Test
|
||||
connection) and a validated proposal (Connect and validate, green
|
||||
"Validated"). The `opencode` harness run passed 17/17 and the GUI regression
|
||||
25/25. A password-protected server was reached through authentication as
|
||||
well. The full graphical installation beyond the AI gate is still not run.
|
||||
|
||||
### Fixed
|
||||
- **A malformed model proposal now gets one bounded corrective retry** before
|
||||
the strict parser rejects the interaction (the shipped DeepSeek models
|
||||
occasionally keyed the change or rationale wrongly). The parser remains the
|
||||
final authority; a retry that still fails is still rejected.
|
||||
- **OpenCode client/probe defects found by testing:** the handwritten HTTP
|
||||
reader (no total deadline, UTF-8-unsafe chunk fallback) was removed in favour
|
||||
of the curl transport; the catalog is no longer parsed on a non-2xx status; a
|
||||
|
|
|
|||
|
|
@ -23,9 +23,16 @@ commit it. It should be short and current; use `git log` for detail.
|
|||
→ `insecure_transport`, exit 1, no request sent.
|
||||
- 7 real-HTTP integration tests (status, chunked multibyte, oversize,
|
||||
redirect, disconnect, auth) and 2 env-gated live tests.
|
||||
- **Not yet verified this session:** the full graphical install through
|
||||
OpenCode in a VM, and a separate physical computer. The pinned manifest is
|
||||
`v1.18.31` while the tested binary is `1.18.32`.
|
||||
- GUI acceptance in a disposable VM: `gui-driver.py opencode` passed 17/17
|
||||
(selected OpenCode, entered the endpoint/model through the UI, real model
|
||||
response + validated proposal); `gui-driver.py gui` passed 25/25.
|
||||
ISO `sha256 4db98db3…`, static binary `sha256 72c2dcd0…`.
|
||||
- **Not yet verified this session:** the full graphical **installation** through
|
||||
OpenCode past the AI gate (disk/account/review/ERASE/install/detached boot),
|
||||
and a separate physical computer. The pinned manifest is `v1.18.31` while the
|
||||
tested binary is `1.18.32`. On this server use `deepseek-flash`; the model
|
||||
must match the server's catalog (the GUI default `deepseek-chat` does not
|
||||
exist on it).
|
||||
- **Public repo:** <https://git.phoebus.club/phoebus/phoebus_os>
|
||||
(remote `origin`, SSH port 2222).
|
||||
|
||||
|
|
|
|||
|
|
@ -101,7 +101,8 @@ still not VM-verified.
|
|||
| `TargetRoot` symlink-escape guard | yes | yes | n/a | n/a |
|
||||
| OpenCode server capability probe | yes | yes | real host probe + auth + policy | n/a |
|
||||
| OpenCode proposal adapter (real inference) | yes | yes | real model (`deepseek-v4-flash`) | n/a |
|
||||
| OpenCode in GUI / CLI / `sys ask` | yes | yes | bridge + `sys ask` live | n/a |
|
||||
| OpenCode in GUI / CLI / `sys ask` | yes | yes | GUI AI step + `sys ask` live | n/a |
|
||||
| OpenCode full graphical install | yes | yes | AI gate passed; install not run | n/a |
|
||||
| Mandatory AI requires a validated interaction | yes | yes | bridge + CLI | n/a |
|
||||
| GURU packaging (scoped keywords) | yes | yes | no (build/install unverified) | no |
|
||||
| Release gates + atomic publish | yes | yes | no | n/a |
|
||||
|
|
|
|||
|
|
@ -25,6 +25,31 @@ installer CLI and `sys ask`. Defects found while implementing and testing:
|
|||
| O11 | The proposal prompt allowed the model to key the change by action name | prompt now requires exactly `action` + `value` with explicit examples | live bridge `ai_check` |
|
||||
| O12 | The GUI duplicate-session assertion accepted two pre-existing sessions | requires exactly one session before and after | harness |
|
||||
| O13 | GUI navigation was inferred from a surviving process/screenshot | the frontend emits secret-free `phoebus-state` lines; the harness asserts the current step and AI state | harness |
|
||||
| O14 | A model could return a malformed proposal with no second chance | one bounded corrective retry in `complete_proposal_at`/`complete_plan`, with the strict parser still final | `a_rejected_proposal_is_retried_once_with_a_correction`, `a_retry_that_still_fails_is_still_rejected` |
|
||||
|
||||
### Graphical OpenCode acceptance (VM verified)
|
||||
|
||||
`gui-driver.py opencode` on the built ISO (real model, no simulation):
|
||||
|
||||
- welcome → installer → AI step; selected **OpenCode agent** through the UI;
|
||||
- typed the server URL, provider and model through the UI;
|
||||
- **Test connection** obtained a real model response (green "Model responded");
|
||||
- **Connect and validate** obtained and validated a proposal (green "Validated",
|
||||
frontend state `aiConnected=true`);
|
||||
- run `gui-opencode-20260922T102640Z-be168813`: **17/17 PASS**, exit 0; the full
|
||||
GUI regression `gui-gui-20260922T102907Z-a9e5e824` was **25/25 PASS**.
|
||||
- The model is reached through a guest-loopback relay to the host OpenCode
|
||||
server (the SSH-tunnel shape), so the transport policy is satisfied.
|
||||
- Authenticated server: the same harness with a password-protected server got
|
||||
through authentication and a real Test connection; a flaky malformed proposal
|
||||
was correctly rejected. Use `deepseek-flash` (reliable) over
|
||||
`deepseek-v4-flash` (occasionally malformed).
|
||||
|
||||
Still **NOT RUN**: the full graphical installation (disk → account → review →
|
||||
ERASE → install → detached boot) *through OpenCode*. The AI gate is now reached
|
||||
and passed in the graphical VM; the remaining stages are the long package
|
||||
installation. Two fresh-disk graphical installs and a separate physical machine
|
||||
remain unverified.
|
||||
|
||||
OpenCode protocol/transport tests: 26 executor unit tests + 7 real-HTTP
|
||||
integration tests + 2 live tests (skipped offline, real inference when
|
||||
|
|
|
|||
|
|
@ -85,12 +85,17 @@ ShellRoot {
|
|||
+ " step=" + shell.step
|
||||
+ " phase=" + shell.phase
|
||||
+ " aiConnected=" + shell.aiConnected
|
||||
+ " aiError=" + shell.aiError.replace(/\n/g, " ").slice(0, 200));
|
||||
+ " aiTested=" + shell.aiTested
|
||||
+ " aiApplied=" + shell.aiApplied.length
|
||||
+ " aiError=" + shell.aiError.replace(/\n/g, " ").slice(0, 200)
|
||||
+ " aiTest=" + shell.aiTestMessage.replace(/\n/g, " ").slice(0, 200));
|
||||
}
|
||||
onStepChanged: logState("step")
|
||||
onPhaseChanged: logState("phase")
|
||||
onAiConnectedChanged: logState("ai")
|
||||
onAiErrorChanged: logState("error")
|
||||
onAiTestedChanged: logState("test")
|
||||
onAiTestMessageChanged: logState("testmsg")
|
||||
|
||||
// QML `var` properties do not emit change signals for nested assignments,
|
||||
// so bindings that read `cfg.*` would not update. Reassigning a clone
|
||||
|
|
|
|||
|
|
@ -59,6 +59,15 @@ BOOT_TIMEOUT = int(os.environ.get("BOOT_TIMEOUT", "900"))
|
|||
INSTALL_TIMEOUT = int(os.environ.get("INSTALL_TIMEOUT", "10800"))
|
||||
SESSION_SETTLE = int(os.environ.get("SESSION_SETTLE", "20"))
|
||||
BACKEND_SETTLE = int(os.environ.get("BACKEND_SETTLE", "30"))
|
||||
# OpenCode GUI-driving knobs (test-only). Tab counts assume the focus order:
|
||||
# Close, DeepSeek, OpenAI, Local, OpenCode, server, password, provider, model,
|
||||
# Test, Connect, Back, Next.
|
||||
OC_PORT = int(os.environ.get("OC_PORT", "4096"))
|
||||
OC_TARGET_PORT = int(os.environ.get("OC_TARGET_PORT", "41999"))
|
||||
OC_TAB_TO_CHIP = int(os.environ.get("OC_TAB_TO_CHIP", "5"))
|
||||
OC_PROVIDER = os.environ.get("OC_PROVIDER", "deepseek")
|
||||
OC_MODEL = os.environ.get("OC_MODEL", "deepseek-v4-flash")
|
||||
OC_USE_PASSWORD = os.environ.get("OC_USE_PASSWORD", "0") in ("1", "yes", "true")
|
||||
|
||||
|
||||
def _template_dir() -> Path:
|
||||
|
|
@ -293,6 +302,8 @@ class Driver:
|
|||
try:
|
||||
if self.mode == "gui":
|
||||
self.do_gui()
|
||||
elif self.mode == "opencode":
|
||||
self.do_opencode()
|
||||
elif self.mode == "install":
|
||||
self.do_install()
|
||||
elif self.mode == "boot":
|
||||
|
|
@ -537,7 +548,8 @@ class Driver:
|
|||
self.child.expect("STATE-END", timeout=30)
|
||||
except (pexpect.TIMEOUT, pexpect.EOF):
|
||||
return None
|
||||
step = ai = None
|
||||
step = ai = tested = None
|
||||
applied = 0
|
||||
err = ""
|
||||
for line in (self.child.before or "").splitlines():
|
||||
if "phoebus-state" not in line:
|
||||
|
|
@ -548,12 +560,179 @@ class Driver:
|
|||
match = re.search(r"aiConnected=(true|false)", line)
|
||||
if match:
|
||||
ai = match.group(1) == "true"
|
||||
match = re.search(r"aiError=(.*)$", line)
|
||||
match = re.search(r"aiTested=(true|false)", line)
|
||||
if match:
|
||||
tested = match.group(1) == "true"
|
||||
match = re.search(r"aiApplied=(\d+)", line)
|
||||
if match:
|
||||
applied = int(match.group(1))
|
||||
match = re.search(r"aiError=(.*?) aiTest=", line)
|
||||
if match:
|
||||
err = match.group(1).strip()
|
||||
if step is None:
|
||||
return None
|
||||
return {"step": step, "aiConnected": bool(ai), "aiError": err}
|
||||
return {
|
||||
"step": step,
|
||||
"aiConnected": bool(ai),
|
||||
"aiTested": bool(tested),
|
||||
"aiApplied": applied,
|
||||
"aiError": err,
|
||||
}
|
||||
|
||||
def wait_state(self, predicate, timeout: int):
|
||||
deadline = time.monotonic() + timeout
|
||||
while time.monotonic() < deadline:
|
||||
state = self._frontend_state()
|
||||
if state is not None and predicate(state):
|
||||
return state
|
||||
time.sleep(2)
|
||||
return None
|
||||
|
||||
def do_opencode(self) -> None:
|
||||
"""Drive the OpenCode AI step through the real graphical UI.
|
||||
|
||||
The model is reached from the guest through a loopback relay (the same
|
||||
shape as an SSH tunnel), so the connection satisfies the transport
|
||||
policy while staying host-local or host+LAN only.
|
||||
"""
|
||||
self.spawn(attach_iso=True, boot_disk=False)
|
||||
self.login_live()
|
||||
welcome = self.poll_shell(
|
||||
"echo WELCOME=$(pgrep -fc 'quickshell.*phoebus-welcome' 2>/dev/null || echo 0)",
|
||||
r"WELCOME=([1-9][0-9]*)", timeout=int(os.environ.get("WELCOME_TIMEOUT", "180")),
|
||||
label="welcome-window",
|
||||
)
|
||||
self.results.assert_true("welcome-autostarted", welcome is not None, "welcome did not appear")
|
||||
self.set_wayland()
|
||||
|
||||
# Start the guest-local relay to the host OpenCode server.
|
||||
self.shell(
|
||||
f"curl -fsSL {SERVE_URL}/oc-forward.py -o /tmp/oc-forward.py && "
|
||||
f"chmod 755 /tmp/oc-forward.py && "
|
||||
f"setsid python3 /tmp/oc-forward.py {OC_PORT} 10.0.2.2 {OC_TARGET_PORT} "
|
||||
f">/tmp/oc-forward.log 2>&1 < /dev/null & echo FWD-$(printf START)",
|
||||
"FWD-START", timeout=60, label="forwarder-start",
|
||||
)
|
||||
ready = self.poll_shell(
|
||||
f"curl -s -m 5 -o /dev/null -w 'HEALTH=%{{http_code}}' "
|
||||
f"http://127.0.0.1:{OC_PORT}/global/health",
|
||||
r"HEALTH=(200|401)", timeout=60, label="forwarder-ready",
|
||||
)
|
||||
self.results.assert_true(
|
||||
"opencode-reachable-from-guest", ready is not None,
|
||||
"the guest relay could not reach the OpenCode server",
|
||||
)
|
||||
|
||||
self.sendkey("ret")
|
||||
opened = self.poll_shell(
|
||||
"echo INST=$(pgrep -fc 'quickshell.*phoebus-install' 2>/dev/null || echo 0)",
|
||||
r"INST=([1-9][0-9]*)", timeout=int(os.environ.get("INSTALLER_OPEN_TIMEOUT", "90")),
|
||||
label="launcher-opens-installer", record=False,
|
||||
)
|
||||
if opened is None:
|
||||
self.results.fail("launcher-opens-installer", "Return did not open the installer")
|
||||
return
|
||||
self.results.pass_("launcher-opens-installer", "welcome action opened the installer")
|
||||
time.sleep(BACKEND_SETTLE)
|
||||
self.shell(
|
||||
"pkill -f 'quickshell.*phoebus-wel[c]ome' || true; sleep 2; echo CLOSED",
|
||||
"CLOSED", timeout=30, label="close-welcome",
|
||||
)
|
||||
|
||||
# Navigate to the AI step (0 -> 1 -> 2).
|
||||
self.sendkey("ctrl-right")
|
||||
time.sleep(2)
|
||||
self.sendkey("ctrl-right")
|
||||
time.sleep(2)
|
||||
state = self._frontend_state()
|
||||
self.results.assert_true(
|
||||
"ai-step-reached", state is not None and state["step"] == 2,
|
||||
f"frontend state: {state}",
|
||||
)
|
||||
self.shot("oc-01-ai", "AI step", "Ctrl+Right twice", "the AI model step is shown")
|
||||
|
||||
# Select the "OpenCode agent" chip with the keyboard.
|
||||
for _ in range(OC_TAB_TO_CHIP):
|
||||
self.sendkey("tab")
|
||||
time.sleep(0.3)
|
||||
self.sendkey("ret")
|
||||
time.sleep(1)
|
||||
self.shot("oc-02-selected", "OpenCode selected", "Tab to the chip and Return",
|
||||
"server, password, provider and model fields appear")
|
||||
|
||||
# Enter the connection through the UI. Focus is on the chip after
|
||||
# selection; Tab order is server, password, provider, model, Test,
|
||||
# Connect. Ctrl+A selects the existing default before typing.
|
||||
def type_field(tabs: int, value: str, secret_file: str | None = None) -> None:
|
||||
# `wtype` (Wayland) and QEMU `sendkey` (hardware) are two input
|
||||
# paths; settle between them or the next key can truncate the text.
|
||||
for _ in range(tabs):
|
||||
self.sendkey("tab")
|
||||
time.sleep(0.4)
|
||||
self.sendkey("ctrl-a")
|
||||
time.sleep(0.5)
|
||||
if secret_file:
|
||||
# The serial echo carries `$(cat ...)`, never the value.
|
||||
self.shell(
|
||||
f'wtype "$(cat {secret_file})" >/dev/null 2>&1; echo TYPED',
|
||||
"TYPED", timeout=30, label="type-secret",
|
||||
)
|
||||
else:
|
||||
self.shell(
|
||||
f"wtype '{value}' >/dev/null 2>&1; echo TYPED",
|
||||
"TYPED", timeout=30, label="type-field",
|
||||
)
|
||||
time.sleep(1.4)
|
||||
# Clear the selection highlight so a screenshot shows the value.
|
||||
self.sendkey("end")
|
||||
time.sleep(0.3)
|
||||
|
||||
type_field(1, f"http://127.0.0.1:{OC_PORT}")
|
||||
if OC_USE_PASSWORD:
|
||||
self.shell(
|
||||
f"curl -fsSL {SERVE_URL}/oc.pw -o /tmp/oc.pw && chmod 600 /tmp/oc.pw "
|
||||
"&& echo PW-$(printf LOADED)",
|
||||
"PW-LOADED", timeout=30, label="fetch-password",
|
||||
)
|
||||
type_field(1, "", secret_file="/tmp/oc.pw")
|
||||
else:
|
||||
self.sendkey("tab") # skip the empty password field
|
||||
time.sleep(0.25)
|
||||
type_field(1, OC_PROVIDER)
|
||||
type_field(1, OC_MODEL)
|
||||
self.shot("oc-02b-fields", "OpenCode fields", "typed server/provider/model",
|
||||
"the fields hold the entered values")
|
||||
self.sendkey("tab") # model -> Test
|
||||
time.sleep(0.3)
|
||||
self.sendkey("ret")
|
||||
time.sleep(3)
|
||||
self.shot("oc-03-test", "Test connection", "Return on Test connection",
|
||||
"a real model response is obtained and validated")
|
||||
tested = self.wait_state(lambda s: s["aiTested"], timeout=180)
|
||||
self.results.assert_true(
|
||||
"opencode-test-connection", tested is not None,
|
||||
"Test connection did not obtain a valid model response",
|
||||
)
|
||||
|
||||
# The next focus is the "Connect and validate" button.
|
||||
self.sendkey("tab")
|
||||
time.sleep(0.3)
|
||||
self.sendkey("ret")
|
||||
self.shot("oc-04-connect", "Connect and validate", "Return on Connect and validate",
|
||||
"a validated proposal is applied; Next is enabled")
|
||||
connected = self.wait_state(lambda s: s["aiConnected"], timeout=240)
|
||||
self.results.assert_true(
|
||||
"opencode-validated-proposal",
|
||||
connected is not None,
|
||||
f"Connect and validate did not validate a proposal: {connected}",
|
||||
)
|
||||
if connected is not None:
|
||||
# A validated proposal may legitimately contain zero changes.
|
||||
self.results.pass_(
|
||||
"opencode-proposal-applied",
|
||||
f"applied={connected['aiApplied']} change(s)",
|
||||
)
|
||||
self.results.pass_("opencode-gui-completed")
|
||||
|
||||
def _installer_alive(self) -> bool:
|
||||
assert self.child is not None
|
||||
|
|
@ -683,7 +862,7 @@ class Driver:
|
|||
|
||||
def main() -> int:
|
||||
mode = sys.argv[1] if len(sys.argv) > 1 else ""
|
||||
if mode not in ("gui", "install", "boot"):
|
||||
if mode not in ("gui", "opencode", "install", "boot"):
|
||||
print(__doc__)
|
||||
return 2
|
||||
if mode in ("install", "boot") and not DISKS:
|
||||
|
|
|
|||
71
installer/tests/vm/oc-forward.py
Normal file
71
installer/tests/vm/oc-forward.py
Normal file
|
|
@ -0,0 +1,71 @@
|
|||
#!/usr/bin/env python3
|
||||
"""Guest-local TCP forwarder for OpenCode VM testing.
|
||||
|
||||
The Phoebus transport policy allows plain `http://` only to a **loopback**
|
||||
destination. A VM guest cannot name the host's loopback directly, and the QEMU
|
||||
gateway address (`10.0.2.2`) is not loopback. This forwarder is the same shape
|
||||
as an SSH tunnel: the GUI connects to the guest's own loopback, and the
|
||||
forwarder relays to the host's OpenCode server over the QEMU host gateway. The
|
||||
connection therefore never leaves the machine pair.
|
||||
|
||||
oc-forward.py [listen_port] [target_host] [target_port]
|
||||
|
||||
It is test infrastructure, not a product component.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import socket
|
||||
import sys
|
||||
import threading
|
||||
|
||||
LISTEN_HOST = "127.0.0.1"
|
||||
|
||||
|
||||
def pipe(src: socket.socket, dst: socket.socket) -> None:
|
||||
try:
|
||||
while True:
|
||||
data = src.recv(65536)
|
||||
if not data:
|
||||
break
|
||||
dst.sendall(data)
|
||||
except OSError:
|
||||
pass
|
||||
finally:
|
||||
for sock in (src, dst):
|
||||
try:
|
||||
sock.shutdown(socket.SHUT_RDWR)
|
||||
except OSError:
|
||||
pass
|
||||
|
||||
|
||||
def handle(client: socket.socket, target: tuple[str, int]) -> None:
|
||||
try:
|
||||
upstream = socket.create_connection(target)
|
||||
except OSError:
|
||||
client.close()
|
||||
return
|
||||
threading.Thread(target=pipe, args=(client, upstream), daemon=True).start()
|
||||
threading.Thread(target=pipe, args=(upstream, client), daemon=True).start()
|
||||
|
||||
|
||||
def main() -> int:
|
||||
listen_port = int(sys.argv[1]) if len(sys.argv) > 1 else 4096
|
||||
target_host = sys.argv[2] if len(sys.argv) > 2 else "10.0.2.2"
|
||||
target_port = int(sys.argv[3]) if len(sys.argv) > 3 else 41999
|
||||
server = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
|
||||
server.setsockopt(socket.SOL_SOCKET, socket.SO_REUSEADDR, 1)
|
||||
server.bind((LISTEN_HOST, listen_port))
|
||||
server.listen(32)
|
||||
print(f"oc-forward listening on {LISTEN_HOST}:{listen_port} -> {target_host}:{target_port}", flush=True)
|
||||
while True:
|
||||
try:
|
||||
client, _ = server.accept()
|
||||
except OSError:
|
||||
break
|
||||
threading.Thread(target=handle, args=(client, (target_host, target_port)), daemon=True).start()
|
||||
return 0
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
sys.exit(main())
|
||||
Loading…
Add table
Reference in a new issue