lfg2025/archy
1
0
Fork 0
Code Issues 24 Pull Requests 4 Actions Packages Projects Releases 71 Wiki Activity

test(13-08): failing tests for the D-07/D-11 confirm gate (RED)

Browse Source 
- confirm.rs: ConfirmGate/PendingConfirmation/Confirmed/PendingSnapshot/
  ResolveRefusal API skeleton (request/resolve/mint_nonce/build_description
  still todo!()) plus the five named confirm tests: S-02 nonce binding,
  S-03 no-model-text, S-08 distinct resources, S-09 restart drops pending,
  timeout declines, and the no-shared-lock-across-the-wait case
- mod.rs: ToolExecCtx gains the confirm gate (global by default, injectable
  for tests) and the S-01 destructive_tool_requires_confirm test with a
  seeded installed-app snapshot
- verified RED: 7 new tests fail (todo! cores + unfilled destructive branch)

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
This commit is contained in:
archipelago
2026-08-05 07:12:12 -04:00
co-authored by Claude Fable 5
parent 7025c5f26f
commit db11c625c8
2 changed files with 610 additions and 0 deletions
Download Patch File Download Diff File Expand all files Collapse all files
core/archipelago/src/assistant/confirm.rs
+418
View File
@@ -0,0 +1,418 @@
//! D-07/D-11: the confirm gate. A destructive tool call suspends the
//! assistant loop here until a human resolves a node-authored description
//! of the exact action. Approval binds to a node-minted nonce over the
//! tool name and the *validated* arguments (S-02), so the action that runs
//! is byte-identical to the action the human read — a cross-action or
//! replayed "yes" is refused arithmetically, never raced.
//!
//! Pending confirmations are **in-memory only, by construction** (S-09):
//! nothing in this file touches the filesystem, and nothing may be added
//! that does. A daemon restart mid-wait therefore clears every pending
//! entry — the next interaction forces a fresh model turn and a
//! freshly-authored confirmation, instead of resurrecting a stale write
//! whose real-world preconditions may have changed.
//!
//! The dialog text is assembled from the node's own tool definition plus
//! the validated argument values only (S-03) — the model's turn and the
//! iframe are never a source. It is this system's signing screen: it names
//! the specific resource, the concrete effect, and the boundary of what is
//! *not* affected (clear-signing, not blind-signing).
use std::collections::HashMap;
use std::sync::atomic::{AtomicU64, Ordering};
use std::sync::{Arc, Mutex, OnceLock};
use std::time::Duration;
use serde_json::json;
use tokio::sync::oneshot;
use tokio::time::Instant;
use super::tools::{ToolArgs, ToolDef};
/// How long an unresolved confirmation waits before declining on its own.
/// Human-speed (the operator may be reading carefully), but bounded — an
/// abandoned dialog must never leak its waiting task (T-13-51).
pub const CONFIRM_TIMEOUT: Duration = Duration::from_secs(120);
/// The human's answer, as seen by the suspended tool call.
#[derive(Debug, Clone, Copy, PartialEq, Eq)]
pub enum Confirmed {
Yes,
No,
TimedOut,
}
/// One suspended destructive action, keyed by its `req_id` in the gate's
/// in-memory map. The `responder` half releases the waiting `request()`
/// call once — a resolved entry is removed, so it can never fire twice.
pub struct PendingConfirmation {
pub call_id: String,
pub tool_name: String,
/// Canonical form of the validated arguments — what the nonce binds.
pub validated_args: String,
pub description: String,
pub nonce: String,
pub created_at: Instant,
responder: oneshot::Sender<bool>,
}
/// The read-only view `assistant.pending` serves to the trusted chrome:
/// everything the host needs to draw and resolve the dialog, nothing that
/// could release the wait by itself.
#[derive(Debug, Clone)]
pub struct PendingSnapshot {
pub req_id: String,
pub tool_name: String,
pub description: String,
pub nonce: String,
}
/// Why a `resolve` call was refused. Distinct on purpose: a nonce mismatch
/// can only mean a replay attempt or a bug in the trusted chrome, so the
/// caller logs it at error level and surfaces it to the owner — loud and
/// sticky, not a toast.
#[derive(Debug, Clone, Copy, PartialEq, Eq)]
pub enum ResolveRefusal {
/// The nonce does not match the pending entry it claims to answer.
NonceMismatch,
/// No pending entry with that id — already resolved, timed out, or
/// minted by a process that is gone.
NoSuchPending,
}
impl std::fmt::Display for ResolveRefusal {
fn fmt(&self, f: &mut std::fmt::Formatter<'_>) -> std::fmt::Result {
match self {
ResolveRefusal::NonceMismatch => write!(
f,
"nonce does not match the pending action — refused (possible replay)"
),
ResolveRefusal::NoSuchPending => write!(
f,
"no such pending confirmation — it may have already been resolved or timed out"
),
}
}
}
/// D-11's pending-confirmation queue. In-memory only — see the module doc.
pub struct ConfirmGate {
pending: Mutex<HashMap<String, PendingConfirmation>>,
next_id: AtomicU64,
}
impl ConfirmGate {
pub fn new() -> Self {
Self {
pending: Mutex::new(HashMap::new()),
next_id: AtomicU64::new(1),
}
}
/// Suspend the calling tool execution until a human answers (or the
/// wait times out). Holds the internal lock only around map edits —
/// never across the human-speed await.
pub async fn request(&self, call_id: &str, tool: &ToolDef, args: &ToolArgs) -> Confirmed {
let _ = (call_id, tool, args);
todo!("13-08 Task 1: suspend on a nonce-bound pending confirmation")
}
/// Resolve a pending confirmation by `req_id`, carrying the node-minted
/// nonce back. Refuses a mismatched nonce (neither action executes) and
/// a replay of an already-resolved entry.
pub fn resolve(&self, req_id: &str, nonce: &str, approved: bool) -> Result<(), ResolveRefusal> {
let _ = (req_id, nonce, approved);
todo!("13-08 Task 1: nonce-checked resolution")
}
/// The oldest pending confirmation, if any — what `assistant.pending`
/// serves to the trusted chrome.
pub fn peek(&self) -> Option<PendingSnapshot> {
self.snapshots().into_iter().next()
}
/// All pending confirmations, oldest first.
pub fn snapshots(&self) -> Vec<PendingSnapshot> {
let map = self.pending.lock().expect("confirm gate mutex poisoned");
let mut entries: Vec<_> = map.iter().collect();
entries.sort_by_key(|(_, p)| p.created_at);
entries
.into_iter()
.map(|(req_id, p)| PendingSnapshot {
req_id: req_id.clone(),
tool_name: p.tool_name.clone(),
description: p.description.clone(),
nonce: p.nonce.clone(),
})
.collect()
}
}
impl Default for ConfirmGate {
fn default() -> Self {
Self::new()
}
}
/// The one process-wide gate, shared by the assistant loop (which suspends
/// on it) and the `assistant.confirm-tool` / `assistant.pending` RPC
/// handlers (which resolve and read it). Process-lifetime by design: when
/// the daemon goes down, so does every pending entry.
pub fn global() -> Arc<ConfirmGate> {
static GATE: OnceLock<Arc<ConfirmGate>> = OnceLock::new();
GATE.get_or_init(|| Arc::new(ConfirmGate::new())).clone()
}
/// The node-minted nonce approval binds to: computed over the tool name
/// and the canonical validated arguments (so it binds what will actually
/// run, not what the model sent), plus per-mint randomness (so yesterday's
/// nonce for the same action never matches today's pending entry).
pub fn mint_nonce(tool_name: &str, validated_args: &str) -> String {
let _ = (tool_name, validated_args);
todo!("13-08 Task 1: hash(tool_name, validated_args) + per-mint randomness")
}
/// Canonical, deterministic form of the validated arguments — the byte
/// string the nonce binds. Hand-written per args shape, matching D-06's
/// hand-written registry.
fn canonical_args(args: &ToolArgs) -> String {
match args {
ToolArgs::Empty(_) => json!({}).to_string(),
ToolArgs::AppId(a) => json!({ "app_id": a.app_id }).to_string(),
ToolArgs::AppLogs(a) => json!({ "app_id": a.app_id, "lines": a.lines }).to_string(),
ToolArgs::SettingsGet(a) => json!({ "key": a.key }).to_string(),
ToolArgs::SettingsSet(a) => json!({ "key": a.key, "value": a.value }).to_string(),
}
}
/// Assemble the dialog text from the node's own tool definition and the
/// validated argument values only — the model's turn is never a source
/// (S-03). Clear-signing: name the resource verbatim (S-08), the concrete
/// effect, and the boundary of what is *not* affected.
pub fn build_description(tool: &ToolDef, args: &ToolArgs) -> String {
let _ = (tool, args);
todo!("13-08 Task 1: node-authored, resource-distinct dialog text")
}
#[cfg(test)]
mod tests {
use super::*;
use crate::assistant::tools::{app_restart_tool, settings_set_tool};
use serde_json::json;
use std::sync::Arc;
fn restart_args(app_id: &str) -> ToolArgs {
app_restart_tool()
.validate(&json!({ "app_id": app_id }))
.expect("valid app_restart args")
}
async fn wait_for_snapshots(gate: &ConfirmGate, n: usize) -> Vec<PendingSnapshot> {
for _ in 0..500 {
let snaps = gate.snapshots();
if snaps.len() >= n {
return snaps;
}
tokio::time::sleep(Duration::from_millis(5)).await;
}
panic!(
"expected {n} pending confirmation(s), have {} — the request did not suspend on the gate",
gate.snapshots().len()
);
}
/// S-02: approval binds to the node-minted nonce over the exact action.
/// A cross-action "yes" is refused (neither action executes), the
/// matching nonce releases exactly its own action, and a replayed nonce
/// is refused.
#[tokio::test]
async fn approval_nonce_binds_to_exact_action() {
let gate = Arc::new(ConfirmGate::new());
let tool = app_restart_tool();
let (g, t, a) = (gate.clone(), tool.clone(), restart_args("immich"));
let task_a = tokio::spawn(async move { g.request("call-a", &t, &a).await });
let snap_a = wait_for_snapshots(&gate, 1).await[0].clone();
let (g, t, b) = (gate.clone(), tool.clone(), restart_args("gitea"));
let task_b = tokio::spawn(async move { g.request("call-b", &t, &b).await });
let snaps = wait_for_snapshots(&gate, 2).await;
let snap_b = snaps
.iter()
.find(|s| s.req_id != snap_a.req_id)
.expect("second pending entry")
.clone();
// A "yes" carrying the OTHER action's nonce is refused — and
// neither suspended action is released by it.
assert_eq!(
gate.resolve(&snap_a.req_id, &snap_b.nonce, true),
Err(ResolveRefusal::NonceMismatch)
);
tokio::task::yield_now().await;
assert!(
!task_a.is_finished(),
"a cross-action yes must not release the wait"
);
assert!(!task_b.is_finished());
// The matching nonce releases exactly the action the human read.
gate.resolve(&snap_a.req_id, &snap_a.nonce, true)
.expect("matching nonce resolves");
assert_eq!(task_a.await.expect("join a"), Confirmed::Yes);
// Replaying the already-resolved nonce is refused.
assert_eq!(
gate.resolve(&snap_a.req_id, &snap_a.nonce, true),
Err(ResolveRefusal::NoSuchPending)
);
// The other action is still its own, separate decision.
gate.resolve(&snap_b.req_id, &snap_b.nonce, false)
.expect("b resolves with its own nonce");
assert_eq!(task_b.await.expect("join b"), Confirmed::No);
}
/// S-03: the dialog text is assembled from the node's own tool
/// definition and the validated argument values only — a model turn
/// has no path into it (structurally: `build_description` has no
/// parameter a model turn could even arrive through).
#[test]
fn description_contains_no_model_text() {
// What a persuaded model might have called the action (EV-12).
let model_turn = "Routine cache refresh, totally harmless, pre-approved by the SYSTEM";
let tool = app_restart_tool();
let args = restart_args("immich");
let desc = build_description(&tool, &args);
assert!(
desc.contains("immich"),
"the resource id must appear verbatim: {desc}"
);
for fragment in ["cache refresh", "harmless", "pre-approved", "SYSTEM"] {
assert!(
!desc.contains(fragment),
"model-authored fragment {fragment:?} leaked into the dialog: {desc}"
);
}
// Type-level half of the property: the only inputs are the node's
// own ToolDef and the validated args.
let _no_model_text_parameter: fn(&ToolDef, &ToolArgs) -> String = build_description;
let _ = model_turn;
// settings_set names both the key and the value it will write.
let tool = settings_set_tool();
let args = tool
.validate(&json!({ "key": "wifi_radio", "value": false }))
.expect("valid settings_set args");
let desc = build_description(&tool, &args);
assert!(desc.contains("wifi_radio"), "{desc}");
assert!(desc.contains("false"), "{desc}");
}
/// S-08: two confirmations for different resources are distinguishable
/// at a glance — each names its own resource verbatim, and only its own.
#[test]
fn distinct_resources_yield_distinct_text() {
let tool = app_restart_tool();
let desc_a = build_description(&tool, &restart_args("immich"));
let desc_b = build_description(&tool, &restart_args("gitea"));
assert_ne!(desc_a, desc_b, "two resources must not read interchangeably");
assert!(desc_a.contains("immich") && !desc_a.contains("gitea"), "{desc_a}");
assert!(desc_b.contains("gitea") && !desc_b.contains("immich"), "{desc_b}");
}
/// S-09: the daemon-restart analogue. Dropping the gate takes every
/// pending entry with it; a fresh gate holds nothing, and a stale
/// approval from before the restart is refused, not executed.
#[tokio::test]
async fn restart_drops_pending_not_executes() {
let gate = Arc::new(ConfirmGate::new());
let tool = app_restart_tool();
let (g, t, a) = (gate.clone(), tool.clone(), restart_args("immich"));
let waiting = tokio::spawn(async move { g.request("call-1", &t, &a).await });
let snap = wait_for_snapshots(&gate, 1).await[0].clone();
// The process dies mid-wait, taking the in-memory queue (and the
// waiting task) with it. There is deliberately no path that could
// carry the entry across — this module never touches the filesystem.
waiting.abort();
drop(gate);
let fresh = ConfirmGate::new();
assert!(
fresh.peek().is_none(),
"a fresh gate must hold no pending entry"
);
assert_eq!(
fresh.resolve(&snap.req_id, &snap.nonce, true),
Err(ResolveRefusal::NoSuchPending),
"a stale approval from before the restart must be refused, not executed"
);
}
/// T-13-51: an unresolved confirmation times out as declined — it does
/// not execute, does not leak its entry, and a late "yes" is refused.
#[tokio::test(start_paused = true)]
async fn timeout_declines_and_does_not_execute() {
let gate = Arc::new(ConfirmGate::new());
let tool = app_restart_tool();
let (g, t, a) = (gate.clone(), tool.clone(), restart_args("immich"));
let waiting = tokio::spawn(async move { g.request("call-1", &t, &a).await });
let snap = wait_for_snapshots(&gate, 1).await[0].clone();
tokio::time::advance(CONFIRM_TIMEOUT + Duration::from_secs(1)).await;
assert_eq!(
waiting.await.expect("join"),
Confirmed::TimedOut,
"an unresolved confirmation declines on its own"
);
assert!(
gate.peek().is_none(),
"the timed-out entry is cleaned up, not leaked"
);
assert_eq!(
gate.resolve(&snap.req_id, &snap.nonce, true),
Err(ResolveRefusal::NoSuchPending),
"a yes arriving after the timeout is refused, not executed"
);
}
/// The confirm wait holds no shared lock: while one confirmation is
/// outstanding (human-speed), the gate's state stays fully usable —
/// reads AND a whole second confirmation round trip complete promptly.
#[tokio::test]
async fn confirm_wait_holds_no_shared_lock() {
let gate = Arc::new(ConfirmGate::new());
let tool = app_restart_tool();
let (g, t, a) = (gate.clone(), tool.clone(), restart_args("immich"));
let outstanding = tokio::spawn(async move { g.request("call-1", &t, &a).await });
wait_for_snapshots(&gate, 1).await;
let concurrent_use = async {
// The assistant.pending read path…
assert!(gate.peek().is_some());
// …and a full second confirmation round trip.
let (g, t, b) = (gate.clone(), tool.clone(), restart_args("gitea"));
let second = tokio::spawn(async move { g.request("call-2", &t, &b).await });
let snaps = wait_for_snapshots(&gate, 2).await;
let snap_b = snaps
.iter()
.find(|s| s.description.contains("gitea"))
.expect("second pending entry")
.clone();
gate.resolve(&snap_b.req_id, &snap_b.nonce, false)
.expect("second confirmation resolves");
assert_eq!(second.await.expect("join second"), Confirmed::No);
};
tokio::time::timeout(Duration::from_secs(5), concurrent_use)
.await
.expect("gate must stay usable while a confirmation is outstanding — no lock across the wait");
assert!(
!outstanding.is_finished(),
"the first confirmation is still the human's call"
);
outstanding.abort();
}
}
core/archipelago/src/assistant/mod.rs
+192
View File
@@ -11,6 +11,7 @@
//! authenticated caller uses. See `13-01-PLAN.md` for the full spine.
pub mod backends;
pub mod confirm;
pub mod grants;
pub mod loop_;
pub mod tools;
@@ -129,15 +130,31 @@ pub struct ToolExecCtx {
pub registry: tools::ToolRegistry,
pub caller: CallerScope,
pub handler: Arc<RpcHandler>,
/// D-07/D-11: the confirm gate every destructive tool suspends on.
/// Defaults to the process-wide gate (`confirm::global()`) so the loop
/// and the `assistant.confirm-tool`/`assistant.pending` RPC handlers
/// share one pending queue; tests inject a fresh gate per test via
/// [`ToolExecCtx::with_confirm_gate`] for isolation.
pub confirm: Arc<confirm::ConfirmGate>,
validation_failures: Mutex<HashMap<String, u32>>,
}
impl ToolExecCtx {
pub fn new(registry: tools::ToolRegistry, caller: CallerScope, handler: Arc<RpcHandler>) -> Self {
Self::with_confirm_gate(registry, caller, handler, confirm::global())
}
pub fn with_confirm_gate(
registry: tools::ToolRegistry,
caller: CallerScope,
handler: Arc<RpcHandler>,
confirm: Arc<confirm::ConfirmGate>,
) -> Self {
Self {
registry,
caller,
handler,
confirm,
validation_failures: Mutex::new(HashMap::new()),
}
}
@@ -264,6 +281,181 @@ mod tests {
(Arc::new(handler), tmp)
}
/// A minimal installed-and-running app entry, seeded into the scanner
/// snapshot so the app-lifecycle tools' business-rule id resolution
/// (`container-list`) finds it — the same cached-state path the real
/// node serves, no orchestrator or podman needed.
fn installed_entry(app_id: &str) -> crate::data_model::PackageDataEntry {
use crate::data_model::{Description, Manifest, PackageDataEntry, PackageState, StaticFiles};
PackageDataEntry {
state: PackageState::Running,
health: None,
exit_code: None,
static_files: StaticFiles {
license: String::new(),
instructions: String::new(),
icon: String::new(),
},
manifest: Manifest {
id: app_id.to_string(),
title: app_id.to_string(),
version: String::new(),
description: Description {
short: String::new(),
long: String::new(),
},
release_notes: String::new(),
license: String::new(),
wrapper_repo: String::new(),
upstream_repo: String::new(),
support_site: String::new(),
marketing_site: String::new(),
donation_url: None,
author: None,
website: None,
interfaces: None,
tier: None,
},
installed: None,
install_progress: None,
uninstall_stage: None,
available_update: None,
}
}
/// Like `test_rpc_handler`, but with one installed, running app seeded
/// into the state snapshot so destructive app-lifecycle calls pass
/// business-rule validation and actually reach the confirm gate.
async fn test_rpc_handler_with_app(app_id: &str) -> (Arc<RpcHandler>, tempfile::TempDir) {
let tmp = tempfile::tempdir().expect("tempdir");
let mut config = crate::config::Config::default();
config.data_dir = tmp.path().to_path_buf();
let state_manager = Arc::new(crate::state::StateManager::new());
let mut data = crate::data_model::DataModel::new();
data.server_info.status_info.containers_scanned = true;
data.package_data
.insert(app_id.to_string(), installed_entry(app_id));
state_manager.update_data(data).await;
let metrics_store = Arc::new(crate::monitoring::MetricsStore::new());
let session_store =
crate::session::SessionStore::new_for_tests(tmp.path().join("sessions.json"));
let handler = RpcHandler::new(
config,
state_manager,
metrics_store,
session_store,
None,
None,
)
.await
.expect("RpcHandler::new");
(Arc::new(handler), tmp)
}
async fn grant(handler: &Arc<RpcHandler>, category: PermissionCategory) {
let mut g = grants::Grants::load(handler.data_dir()).await;
g.set(category, true);
g.save(handler.data_dir()).await.expect("save grants");
}
async fn wait_pending(gate: &confirm::ConfirmGate) -> confirm::PendingSnapshot {
for _ in 0..500 {
if let Some(snap) = gate.peek() {
return snap;
}
tokio::time::sleep(std::time::Duration::from_millis(5)).await;
}
panic!("no pending confirmation appeared — the destructive branch did not suspend on the gate");
}
/// S-01 / D-07: a destructive tool call suspends before any execution
/// and produces a pending confirmation whose node-authored description
/// names the resource; nothing runs until a human resolves it, and a
/// decline returns an error result without executing.
#[tokio::test]
async fn destructive_tool_requires_confirm() {
let (handler, _tmp) = test_rpc_handler_with_app("demo-app").await;
grant(&handler, PermissionCategory::Apps).await;
// Part 1 — the choke point directly: execute_tool suspends on the
// gate, and a decline returns a declined error result (dispatch is
// never reached — nothing was changed).
let gate = Arc::new(confirm::ConfirmGate::new());
let ctx = Arc::new(ToolExecCtx::with_confirm_gate(
tools::registry(),
CallerScope::LocalOperator {
session_id: "s".to_string(),
},
handler.clone(),
gate.clone(),
));
let call = tools::ToolCall {
id: "call-1".to_string(),
name: "app_restart".to_string(),
arguments: serde_json::json!({ "app_id": "demo-app" }),
};
let (ctx_task, call_task) = (ctx.clone(), call.clone());
let suspended =
tokio::spawn(async move { loop_::execute_tool(&call_task, &ctx_task).await });
let snap = wait_pending(&gate).await;
assert!(
snap.description.contains("demo-app"),
"the node-authored description must name the resource: {}",
snap.description
);
assert!(
!suspended.is_finished(),
"execute_tool must stay suspended until the human decides"
);
gate.resolve(&snap.req_id, &snap.nonce, false)
.expect("decline resolves");
let result = suspended.await.expect("join");
assert!(result.is_error, "a declined action must not execute");
assert!(
result.content.to_lowercase().contains("declined"),
"the tool result must say the user declined: {}",
result.content
);
// Part 2 — through the real loop: a scripted backend proposes the
// destructive call, the loop suspends on the gate, and only after
// the human answers does the turn finish.
let gate2 = Arc::new(confirm::ConfirmGate::new());
let ctx2 = ToolExecCtx::with_confirm_gate(
tools::registry(),
CallerScope::LocalOperator {
session_id: "s".to_string(),
},
handler.clone(),
gate2.clone(),
);
let backend = crate::assistant::backends::scripted::ScriptedBackend::new(vec![
crate::assistant::backends::BackendTurn::ToolCalls(vec![call.clone()]),
crate::assistant::backends::BackendTurn::Text(
"Understood — I did not restart it.".to_string(),
),
]);
let tools_list = vec![tools::app_restart_tool()];
let loop_task = tokio::spawn(async move {
loop_::run_loop(&backend, "sys", &tools_list, vec![], &ctx2).await
});
let snap2 = wait_pending(&gate2).await;
assert!(
!loop_task.is_finished(),
"run_loop must stay suspended while the confirmation is pending"
);
gate2
.resolve(&snap2.req_id, &snap2.nonce, false)
.expect("decline resolves");
let answer = loop_task.await.expect("join").expect("run_loop");
assert_eq!(answer, "Understood — I did not restart it.");
}
/// S-06 / D-16: a fresh node's `LocalOperator` resolves to no granted
/// categories at all.
#[tokio::test]