fix(assistant): handle oversized search investigations
Some checks are pending
ci / test (push) Waiting to run
Some checks are pending
ci / test (push) Waiting to run
This commit is contained in:
parent
d479e4f45d
commit
a224bb72e8
8 changed files with 230 additions and 15 deletions
|
|
@ -5,6 +5,16 @@ this file is the curated, human-readable summary.
|
||||||
|
|
||||||
## Unreleased — Fast defaults with user-controlled source builds
|
## Unreleased — Fast defaults with user-controlled source builds
|
||||||
|
|
||||||
|
- **Bounded Assistant searches (2026-10-03).** A general search reply with more
|
||||||
|
than six typed read-only checks now runs the first six and tells the model
|
||||||
|
which checks were skipped, so it can continue on the next turn. The full
|
||||||
|
reply is validated before any check runs; a mutating action anywhere in the
|
||||||
|
batch is still refused. Informational answers with no actions now appear as
|
||||||
|
information in the desktop Assistant, with no approval button or staged
|
||||||
|
plan. The running VM answered the exact Balena Etcher package question with
|
||||||
|
the AppImage catalog lead instead of the previous `max 6` rejection, and
|
||||||
|
the final information card has no approval button.
|
||||||
|
|
||||||
- **Steady assistant display and missing-package result (2026-10-03).**
|
- **Steady assistant display and missing-package result (2026-10-03).**
|
||||||
Progress events now update a small activity list inside the current message
|
Progress events now update a small activity list inside the current message
|
||||||
instead of recreating the whole conversation and scrolling it on every
|
instead of recreating the whole conversation and scrolling it on every
|
||||||
|
|
|
||||||
|
|
@ -3,6 +3,26 @@
|
||||||
Rolling note between work sessions. **Update this file before you finish**, then
|
Rolling note between work sessions. **Update this file before you finish**, then
|
||||||
commit it. It should be short and current; use `git log` for detail.
|
commit it. It should be short and current; use `git log` for detail.
|
||||||
|
|
||||||
|
- **General Assistant search cap (2026-10-03):** The exact VM question `can you
|
||||||
|
find a package for balena etcher for me` previously failed with `too many
|
||||||
|
investigation requests in one turn (max 6)`. The general loop now validates
|
||||||
|
every typed action, executes only the first six read-only checks, and feeds
|
||||||
|
evidence plus a skipped-check note back to the model. The live VM answered
|
||||||
|
with an AppImage catalog lead and no verified installable package
|
||||||
|
(`/var/tmp/phoebus-vm/etcher-search-new2.png`). That run exposed a desktop
|
||||||
|
bridge issue: empty-action answers showed `Approve & Execute`. The bridge
|
||||||
|
now sends an information result with no plan ID or digest, and the service
|
||||||
|
does not stage an empty plan. Tests cover both behavior and the read-only
|
||||||
|
boundary. The final user-local static `sys` was installed in the VM (SHA-256
|
||||||
|
`6c7226433af167377d2002e2aba9682914e99691c501c61bb2c04681b4e586ee`).
|
||||||
|
Repeating the exact question returned an Information card, displayed its
|
||||||
|
read-only `sys` checks, and showed no approval button
|
||||||
|
(`/var/tmp/phoebus-vm/final-question-progress.png`). The VM's root-owned
|
||||||
|
binary remains older until a fresh install. A fresh ISO is still gated by
|
||||||
|
`make doctor` reporting 9 GiB free against its 20 GiB VM requirement. Rust
|
||||||
|
format, strict clippy, workspace tests, `make check`, and dynamic/static
|
||||||
|
release builds pass.
|
||||||
|
|
||||||
- **Assistant flicker and Etcher follow-up (2026-10-03):** Progress used to
|
- **Assistant flicker and Etcher follow-up (2026-10-03):** Progress used to
|
||||||
replace the entire QML conversation model and scroll the full view on every
|
replace the entire QML conversation model and scroll the full view on every
|
||||||
event. It now updates a fixed-height activity list inside the message card.
|
event. It now updates a fixed-height activity list inside the message card.
|
||||||
|
|
|
||||||
|
|
@ -48,6 +48,21 @@ disposable VM; `serial-driver.py recovery` now automates the same check.
|
||||||
|
|
||||||
## Matrix
|
## Matrix
|
||||||
|
|
||||||
|
General Assistant search bounds (2026-10-03): an oversized model investigation
|
||||||
|
is validated in full and only its first six read-only requests are executed;
|
||||||
|
the model receives the evidence and a clear note about the skipped checks.
|
||||||
|
Unit tests cover the cap and a mutating seventh request. In the running VM,
|
||||||
|
the exact `can you find a package for balena etcher for me` request returned a
|
||||||
|
package search summary with an AppImage catalog lead, instead of the previous
|
||||||
|
`max 6` error (`/var/tmp/phoebus-vm/etcher-search-new2.png`). That run also
|
||||||
|
showed an approval button for a no-action answer; the bridge now returns an
|
||||||
|
information card for such answers and does not stage an empty plan. The final
|
||||||
|
user-local static binary was installed in the VM (SHA-256
|
||||||
|
`6c7226433af167377d2002e2aba9682914e99691c501c61bb2c04681b4e586ee`).
|
||||||
|
The exact question then returned an Information card with visible read-only
|
||||||
|
checks and no approval button (`/var/tmp/phoebus-vm/final-question-progress.png`).
|
||||||
|
The root-owned installed binary remains older until a fresh ISO/install.
|
||||||
|
|
||||||
Assistant display and unavailable plan (2026-10-03): QML progress updates
|
Assistant display and unavailable plan (2026-10-03): QML progress updates
|
||||||
modify only the current card's fixed-height activity list; the conversation
|
modify only the current card's fixed-height activity list; the conversation
|
||||||
model is not replaced for every phase. The Mango Assistant window stays opaque
|
model is not replaced for every phase. The Mango Assistant window stays opaque
|
||||||
|
|
|
||||||
|
|
@ -92,7 +92,8 @@ real evidence. Useful tools: search_packages (query, optional backend), \
|
||||||
discover_candidates, inspect_candidate, check_backend, inspect_installation, \
|
discover_candidates, inspect_candidate, check_backend, inspect_installation, \
|
||||||
list_packages (optional backend/filter), package_info, check_updates \
|
list_packages (optional backend/filter), package_info, check_updates \
|
||||||
(optional backend), package_owners, package_dependencies. Never request a \
|
(optional backend), package_owners, package_dependencies. Never request a \
|
||||||
mutating action as evidence.
|
mutating action as evidence. Request at most six read-only actions per turn; \
|
||||||
|
you can ask for more evidence on the next turn.
|
||||||
|
|
||||||
2. Plan or answer: {\"version\":1,\"summary\":\"...\",\"actions\":\
|
2. Plan or answer: {\"version\":1,\"summary\":\"...\",\"actions\":\
|
||||||
[{\"action\":\"list_packages\"}],\
|
[{\"action\":\"list_packages\"}],\
|
||||||
|
|
@ -402,12 +403,15 @@ pub fn run_assistant(
|
||||||
|
|
||||||
match step {
|
match step {
|
||||||
AssistantStep::Investigate { requests } => {
|
AssistantStep::Investigate { requests } => {
|
||||||
if requests.len() > limits.max_requests_per_turn {
|
// The whole reply has already passed the typed, read-only
|
||||||
return Err(AiError::Rejected(format!(
|
// validator. Run only the bounded prefix and let the model
|
||||||
"too many investigation requests in one turn (max {})",
|
// request any omitted evidence on its next turn. An oversized
|
||||||
limits.max_requests_per_turn
|
// batch is a model mistake, not a user-facing failure.
|
||||||
)));
|
let requested_count = requests.len();
|
||||||
}
|
let requests: Vec<_> = requests
|
||||||
|
.into_iter()
|
||||||
|
.take(limits.max_requests_per_turn)
|
||||||
|
.collect();
|
||||||
let mut keys: Vec<String> = requests
|
let mut keys: Vec<String> = requests
|
||||||
.iter()
|
.iter()
|
||||||
.filter_map(|r| serde_json::to_string(r).ok())
|
.filter_map(|r| serde_json::to_string(r).ok())
|
||||||
|
|
@ -444,6 +448,16 @@ pub fn run_assistant(
|
||||||
payload.truncate(limits.max_evidence_bytes);
|
payload.truncate(limits.max_evidence_bytes);
|
||||||
payload.push_str("\n[evidence truncated]");
|
payload.push_str("\n[evidence truncated]");
|
||||||
}
|
}
|
||||||
|
if requested_count > requests.len() {
|
||||||
|
payload.push_str(&format!(
|
||||||
|
"\nOnly the first {} of {} requested checks ran. The remaining \
|
||||||
|
checks were not performed. Request any still needed evidence in \
|
||||||
|
another turn, with at most {} checks, or answer from the evidence above.",
|
||||||
|
requests.len(),
|
||||||
|
requested_count,
|
||||||
|
limits.max_requests_per_turn
|
||||||
|
));
|
||||||
|
}
|
||||||
chat.messages.push(ChatMessage {
|
chat.messages.push(ChatMessage {
|
||||||
role: "user".into(),
|
role: "user".into(),
|
||||||
content: format!("Evidence:\n{payload}"),
|
content: format!("Evidence:\n{payload}"),
|
||||||
|
|
@ -736,6 +750,97 @@ mod tests {
|
||||||
assert!(requests[1].contains("com.google.Chrome"));
|
assert!(requests[1].contains("com.google.Chrome"));
|
||||||
}
|
}
|
||||||
|
|
||||||
|
#[test]
|
||||||
|
fn oversized_investigation_runs_only_the_bounded_prefix() {
|
||||||
|
use crate::provider::{chat_response, ScriptedTransport};
|
||||||
|
let requests: Vec<_> = (0..7)
|
||||||
|
.map(|index| serde_json::json!({"action":"search_packages","query":format!("etcher{index}")}))
|
||||||
|
.collect();
|
||||||
|
let transport = ScriptedTransport::new(vec![
|
||||||
|
chat_response(
|
||||||
|
&serde_json::json!({"version":1,"kind":"investigate","requests":requests})
|
||||||
|
.to_string(),
|
||||||
|
),
|
||||||
|
chat_response(
|
||||||
|
r#"{"version":1,"summary":"No package found in the checked sources.","actions":[]}"#,
|
||||||
|
),
|
||||||
|
]);
|
||||||
|
let investigate = |requests: &[SystemAction]| -> Vec<Evidence> {
|
||||||
|
assert_eq!(requests.len(), 6);
|
||||||
|
let queries: Vec<_> = requests
|
||||||
|
.iter()
|
||||||
|
.map(|r| {
|
||||||
|
serde_json::to_value(r).unwrap()["query"]
|
||||||
|
.as_str()
|
||||||
|
.unwrap()
|
||||||
|
.to_string()
|
||||||
|
})
|
||||||
|
.collect();
|
||||||
|
assert_eq!(
|
||||||
|
queries,
|
||||||
|
(0..6)
|
||||||
|
.map(|index| format!("etcher{index}"))
|
||||||
|
.collect::<Vec<_>>()
|
||||||
|
);
|
||||||
|
requests
|
||||||
|
.iter()
|
||||||
|
.map(|r| Evidence {
|
||||||
|
request: r.describe(),
|
||||||
|
output: "[]".into(),
|
||||||
|
})
|
||||||
|
.collect()
|
||||||
|
};
|
||||||
|
let plan = run_assistant(
|
||||||
|
ProviderKind::Local,
|
||||||
|
"http://127.0.0.1:11434/v1/chat/completions",
|
||||||
|
"m",
|
||||||
|
&facts(),
|
||||||
|
"find etcher",
|
||||||
|
&transport,
|
||||||
|
&investigate,
|
||||||
|
&crate::install::Limits::default(),
|
||||||
|
)
|
||||||
|
.unwrap();
|
||||||
|
assert!(plan.actions.is_empty());
|
||||||
|
let sent = transport.requests();
|
||||||
|
assert_eq!(sent.len(), 2);
|
||||||
|
assert!(sent[1].contains("Only the first 6 of 7 requested checks ran"));
|
||||||
|
assert!(sent[1].contains("remaining checks were not performed"));
|
||||||
|
}
|
||||||
|
|
||||||
|
#[test]
|
||||||
|
fn oversized_investigation_still_rejects_a_mutation_after_the_limit() {
|
||||||
|
use crate::provider::{chat_response, ScriptedTransport};
|
||||||
|
let mut requests: Vec<_> = (0..6)
|
||||||
|
.map(|index| serde_json::json!({"action":"search_packages","query":format!("etcher{index}")}))
|
||||||
|
.collect();
|
||||||
|
requests.push(serde_json::json!({"action":"install_package","atom":"vim"}));
|
||||||
|
let transport = ScriptedTransport::new(vec![
|
||||||
|
chat_response(
|
||||||
|
&serde_json::json!({"version":1,"kind":"investigate","requests":requests})
|
||||||
|
.to_string(),
|
||||||
|
),
|
||||||
|
chat_response(
|
||||||
|
r#"{"version":1,"summary":"I can search without changing the system.","actions":[]}"#,
|
||||||
|
),
|
||||||
|
]);
|
||||||
|
let investigate = |_: &[SystemAction]| -> Vec<Evidence> {
|
||||||
|
panic!("no checks should run when any request is mutating")
|
||||||
|
};
|
||||||
|
let plan = run_assistant(
|
||||||
|
ProviderKind::Local,
|
||||||
|
"http://127.0.0.1:11434/v1/chat/completions",
|
||||||
|
"m",
|
||||||
|
&facts(),
|
||||||
|
"find etcher",
|
||||||
|
&transport,
|
||||||
|
&investigate,
|
||||||
|
&crate::install::Limits::default(),
|
||||||
|
)
|
||||||
|
.unwrap();
|
||||||
|
assert!(plan.actions.is_empty());
|
||||||
|
}
|
||||||
|
|
||||||
#[test]
|
#[test]
|
||||||
fn run_assistant_never_executes_a_mutating_investigation_request() {
|
fn run_assistant_never_executes_a_mutating_investigation_request() {
|
||||||
use crate::provider::{chat_response, ScriptedTransport};
|
use crate::provider::{chat_response, ScriptedTransport};
|
||||||
|
|
|
||||||
|
|
@ -267,6 +267,21 @@ fn plan_response(id: u64, kind: &str, plan: &PreparedPlan) -> BridgeResponse {
|
||||||
)
|
)
|
||||||
}
|
}
|
||||||
|
|
||||||
|
fn general_response(id: u64, plan: &PreparedPlan) -> BridgeResponse {
|
||||||
|
if plan.actions.is_empty() {
|
||||||
|
return BridgeResponse::ok(
|
||||||
|
id,
|
||||||
|
json!({
|
||||||
|
"kind": "info",
|
||||||
|
"state": "info",
|
||||||
|
"summary": plan.summary,
|
||||||
|
"recommendation": plan.recommendation,
|
||||||
|
}),
|
||||||
|
);
|
||||||
|
}
|
||||||
|
plan_response(id, "plan", plan)
|
||||||
|
}
|
||||||
|
|
||||||
pub fn handle_request(
|
pub fn handle_request(
|
||||||
exec: &Exec,
|
exec: &Exec,
|
||||||
root: &Path,
|
root: &Path,
|
||||||
|
|
@ -513,6 +528,7 @@ pub fn handle_request(
|
||||||
});
|
});
|
||||||
match response {
|
match response {
|
||||||
Ok(Ok(plan)) => {
|
Ok(Ok(plan)) => {
|
||||||
|
let informational = plan.actions.is_empty();
|
||||||
record_turn(
|
record_turn(
|
||||||
exec,
|
exec,
|
||||||
&conv_path,
|
&conv_path,
|
||||||
|
|
@ -520,16 +536,16 @@ pub fn handle_request(
|
||||||
timestamp_unix: now_unix(),
|
timestamp_unix: now_unix(),
|
||||||
// Structured plan summary, not the raw request.
|
// Structured plan summary, not the raw request.
|
||||||
query: plan.summary.clone(),
|
query: plan.summary.clone(),
|
||||||
response_kind: "plan".into(),
|
response_kind: if informational { "info" } else { "plan" }.into(),
|
||||||
application: None,
|
application: None,
|
||||||
chosen_candidate: None,
|
chosen_candidate: None,
|
||||||
alternatives: Vec::new(),
|
alternatives: Vec::new(),
|
||||||
plan_id: Some(plan.id.clone()),
|
plan_id: (!informational).then(|| plan.id.clone()),
|
||||||
plan_digest: Some(plan.digest.clone()),
|
plan_digest: (!informational).then(|| plan.digest.clone()),
|
||||||
rationale: plan.rationale.clone(),
|
rationale: plan.rationale.clone(),
|
||||||
},
|
},
|
||||||
);
|
);
|
||||||
plan_response(id, "plan", &plan)
|
general_response(id, &plan)
|
||||||
}
|
}
|
||||||
Ok(Err(e)) => BridgeResponse::err(id, e.to_string()),
|
Ok(Err(e)) => BridgeResponse::err(id, e.to_string()),
|
||||||
Err(e) => BridgeResponse::err(id, e),
|
Err(e) => BridgeResponse::err(id, e),
|
||||||
|
|
@ -1102,6 +1118,48 @@ mod tests {
|
||||||
assert_eq!(parse_install_phrase("why is audio broken"), None);
|
assert_eq!(parse_install_phrase("why is audio broken"), None);
|
||||||
}
|
}
|
||||||
|
|
||||||
|
#[test]
|
||||||
|
fn general_answer_without_actions_has_no_approval_controls() {
|
||||||
|
use phoebus_ai::provider::{chat_response, ProviderKind, ScriptedTransport};
|
||||||
|
|
||||||
|
let root = tempdir().unwrap();
|
||||||
|
let exec = make_test_exec(root.path());
|
||||||
|
let plans = PlanStore::with_dir(tempdir().unwrap().keep());
|
||||||
|
let transport = ScriptedTransport::new(vec![chat_response(
|
||||||
|
r#"{"version":1,"summary":"An AppImage catalog lead was found; no installable package was verified.","actions":[]}"#,
|
||||||
|
)]);
|
||||||
|
let ai = AiInstall {
|
||||||
|
provider: ProviderKind::Local,
|
||||||
|
endpoint: "http://127.0.0.1:11434/v1/chat/completions".into(),
|
||||||
|
model: "m".into(),
|
||||||
|
transport: &transport,
|
||||||
|
limits: phoebus_ai::install::Limits::default(),
|
||||||
|
};
|
||||||
|
let plan = service::plan_general(
|
||||||
|
&plans,
|
||||||
|
&exec,
|
||||||
|
root.path(),
|
||||||
|
"find Etcher",
|
||||||
|
&SystemFacts::default(),
|
||||||
|
&ai,
|
||||||
|
None,
|
||||||
|
)
|
||||||
|
.unwrap();
|
||||||
|
|
||||||
|
let response = general_response(5, &plan);
|
||||||
|
assert!(response.ok);
|
||||||
|
let result = response.result.unwrap();
|
||||||
|
assert_eq!(result["kind"], "info");
|
||||||
|
assert_eq!(result["state"], "info");
|
||||||
|
assert!(result["recommendation"]
|
||||||
|
.as_str()
|
||||||
|
.unwrap()
|
||||||
|
.contains("AppImage"));
|
||||||
|
assert!(result.get("prepared").is_none());
|
||||||
|
assert!(result.get("plan_id").is_none());
|
||||||
|
assert!(result.get("digest").is_none());
|
||||||
|
}
|
||||||
|
|
||||||
#[test]
|
#[test]
|
||||||
fn appimage_install_request_returns_catalog_information_without_a_plan() {
|
fn appimage_install_request_returns_catalog_information_without_a_plan() {
|
||||||
let root = tempdir().unwrap();
|
let root = tempdir().unwrap();
|
||||||
|
|
|
||||||
|
|
@ -1474,6 +1474,7 @@ fn run_ask(
|
||||||
}
|
}
|
||||||
};
|
};
|
||||||
|
|
||||||
|
let informational = plan.actions.is_empty();
|
||||||
record_turn(
|
record_turn(
|
||||||
exec,
|
exec,
|
||||||
&conv_path,
|
&conv_path,
|
||||||
|
|
@ -1482,12 +1483,12 @@ fn run_ask(
|
||||||
// Persist the plan's own summary, not the raw request: structured
|
// Persist the plan's own summary, not the raw request: structured
|
||||||
// context is sufficient for follow-ups and minimizes exposure.
|
// context is sufficient for follow-ups and minimizes exposure.
|
||||||
query: plan.summary.clone(),
|
query: plan.summary.clone(),
|
||||||
response_kind: "plan".into(),
|
response_kind: if informational { "info" } else { "plan" }.into(),
|
||||||
application: None,
|
application: None,
|
||||||
chosen_candidate: None,
|
chosen_candidate: None,
|
||||||
alternatives: Vec::new(),
|
alternatives: Vec::new(),
|
||||||
plan_id: Some(plan.id.clone()),
|
plan_id: (!informational).then(|| plan.id.clone()),
|
||||||
plan_digest: Some(plan.digest.clone()),
|
plan_digest: (!informational).then(|| plan.digest.clone()),
|
||||||
rationale: plan.rationale.clone(),
|
rationale: plan.rationale.clone(),
|
||||||
},
|
},
|
||||||
);
|
);
|
||||||
|
|
|
||||||
|
|
@ -882,6 +882,8 @@ pub fn plan_general(
|
||||||
integrity: String::new(),
|
integrity: String::new(),
|
||||||
};
|
};
|
||||||
|
|
||||||
|
if !plan.actions.is_empty() {
|
||||||
plans.save(&mut plan)?;
|
plans.save(&mut plan)?;
|
||||||
|
}
|
||||||
Ok(plan)
|
Ok(plan)
|
||||||
}
|
}
|
||||||
|
|
|
||||||
|
|
@ -1967,6 +1967,10 @@ fn plan_general_investigates_with_read_only_tools_then_answers() {
|
||||||
assert!(plan.actions.is_empty(), "an answer has no actions");
|
assert!(plan.actions.is_empty(), "an answer has no actions");
|
||||||
assert_eq!(plan.summary, "Two real candidates found.");
|
assert_eq!(plan.summary, "Two real candidates found.");
|
||||||
assert!(!plan.requires_root, "a read-only answer needs no elevation");
|
assert!(!plan.requires_root, "a read-only answer needs no elevation");
|
||||||
|
assert!(
|
||||||
|
store.load(&plan.id).is_err(),
|
||||||
|
"an answer must not stage a plan"
|
||||||
|
);
|
||||||
// Two model turns: investigate, then answer.
|
// Two model turns: investigate, then answer.
|
||||||
assert_eq!(transport.requests().len(), 2);
|
assert_eq!(transport.requests().len(), 2);
|
||||||
assert!(transport.requests()[1].contains("Package catalogue"));
|
assert!(transport.requests()[1].contains("Package catalogue"));
|
||||||
|
|
|
||||||
Loading…
Add table
Reference in a new issue