assistant/evals.rs: a test-gated in-crate module (#![cfg(test)] here AND
#[cfg(test)] pub mod evals; in mod.rs — never compiles into the shipped
binary, asserted by a release-binary string grep). load_cases/case_by_id
read the 18-case JSONL fixture by path; run_case drives the REAL run_loop/
execute_tool/ConfirmGate choke points end to end against a case's grants,
seeded untrusted content, and scripted backend turns, returning a
CaseOutcome that observes ToolCall/ToolResult/confirm-gate transitions
in-process rather than inferring them from prose. evaluate_case asserts
must_not_execute/must_not_claim at threshold zero (E-01's security and
integrity halves) and confirmations/turns at exact match, every failure
message naming the case id and the offending tool/term.
Parameterized over the Backend trait (CountingBackend wraps any real
Backend to measure turns used; a BudgetExhaustedStubBackend drives EV-17's
S-12 stop-without-retry path) so scripted, Ollama, Claude or Routstr can
all run the same 18 cases. report_by_backend/parity_requires_two_backends
refuse to record a cross-backend parity pass from fewer than two backends
(E-07). Live-backend runs are opt-in via ARCHY_EVAL_BACKENDS and #[ignore]d
so a plain `cargo test` never touches the network. write_trace_jsonl writes
one plain JSONL file per run under core/target/assistant-evals/ (gitignored
build output) — no exporter, no collector, no listening port.
All 18 cases pass against ScriptedBackend (23/23 assistant::evals:: tests);
full crate suite 1258/1258; release binary contains zero eval-fixture
strings; no phoenix/promptfoo/ragas/opentelemetry references anywhere in
assistant/; no new CI job (ci.yml untouched — picked up by the existing
`cargo test --all-features` step); zero new packages (T-13-SC).
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
