feat(13-14): the harness — offline, deterministic, per-backend, zero footprint on a node
assistant/evals.rs: a test-gated in-crate module (#![cfg(test)] here AND #[cfg(test)] pub mod evals; in mod.rs — never compiles into the shipped binary, asserted by a release-binary string grep). load_cases/case_by_id read the 18-case JSONL fixture by path; run_case drives the REAL run_loop/ execute_tool/ConfirmGate choke points end to end against a case's grants, seeded untrusted content, and scripted backend turns, returning a CaseOutcome that observes ToolCall/ToolResult/confirm-gate transitions in-process rather than inferring them from prose. evaluate_case asserts must_not_execute/must_not_claim at threshold zero (E-01's security and integrity halves) and confirmations/turns at exact match, every failure message naming the case id and the offending tool/term. Parameterized over the Backend trait (CountingBackend wraps any real Backend to measure turns used; a BudgetExhaustedStubBackend drives EV-17's S-12 stop-without-retry path) so scripted, Ollama, Claude or Routstr can all run the same 18 cases. report_by_backend/parity_requires_two_backends refuse to record a cross-backend parity pass from fewer than two backends (E-07). Live-backend runs are opt-in via ARCHY_EVAL_BACKENDS and #[ignore]d so a plain `cargo test` never touches the network. write_trace_jsonl writes one plain JSONL file per run under core/target/assistant-evals/ (gitignored build output) — no exporter, no collector, no listening port. All 18 cases pass against ScriptedBackend (23/23 assistant::evals:: tests); full crate suite 1258/1258; release binary contains zero eval-fixture strings; no phoenix/promptfoo/ragas/opentelemetry references anywhere in assistant/; no new CI job (ci.yml untouched — picked up by the existing `cargo test --all-features` step); zero new packages (T-13-SC). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
This commit is contained in:
co-authored by
Claude Fable 5
parent
d419141a86
commit
27aa5ccd10
@@ -13,6 +13,8 @@
|
||||
pub mod backends;
|
||||
pub mod confirm;
|
||||
pub mod egress;
|
||||
#[cfg(test)]
|
||||
pub mod evals;
|
||||
pub mod grants;
|
||||
pub mod history;
|
||||
pub mod loop_;
|
||||
|
||||
Reference in New Issue
Block a user