docs(13): routstr protocol findings + coverage matrix from live probe

13-ROUTSTR-FINDINGS.md records the routstr_probe run against all three
docs.routstr.com default relays: zero kind-38421 events and zero
#d=routstr-provider fallback events in a 30s window each, with all
three relay connections succeeding (ruling out a connectivity
failure as the explanation). Every claim is labelled OBSERVED or
DOCS-ONLY per the plan's scope note; RESEARCH assumption A2 is
recorded as neither confirmed nor refuted, risk unchanged.

COVERAGE.md's three former "INTEGRATE — UNCONFIRMED" rows (tool
calling, Cashu payment header, Nostr provider discovery) are
downgraded to explicit opt-outs with dated, evidenced reasons — zero
rows retain unconfirmed-integration status. The Gate section states
13-13 may not proceed directly and must open with a checkpoint:
decision, which 13-13-PLAN.md's Task 1 already is (proceed-observed /
proceed-docs-with-probe-first / defer-with-residual) — no edit to
13-13-PLAN.md was needed.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
This commit is contained in:
archipelago
2026-08-03 12:34:05 -04:00
co-authored by Claude Opus 5
parent ea90ef05a5
commit f8987d12f3
2 changed files with 111 additions and 16 deletions
@@ -0,0 +1,85 @@
# Routstr Protocol Findings — live probe (13-03)
**Date run:** 2026-08-03
**Probe:** `core/archipelago/examples/routstr_probe.rs` (commit `ea90ef05`), run by hand with
`cd core && ./target/debug/examples/routstr_probe` (equivalently `cargo run --example
routstr_probe`).
**Relays probed:** `wss://relay.damus.io`, `wss://relay.nostr.band`, `wss://nos.lol` — the
three default relays cited in `docs.routstr.com` (RESEARCH.md, `13-RESEARCH.md` "Routstr
chat-completions call shape").
**Filters used:** (1) kind `38421`, limit 50, 30s wait; (2) no kind restriction, `#d` tag
`routstr-provider`, limit 50, 30s wait — a fallback in case the kind number in the docs had
drifted.
**Result:** **NO LIVE PROVIDER OBSERVED.** Zero events matched either filter across all three
relays within the wait budget. No endpoint was discovered, so the capability-probe half
(`/v1/models`, `/`) never ran — there was nothing to point it at.
Every claim below is labelled **OBSERVED** (this probe run produced the evidence) or
**DOCS-ONLY** (still resting on `docs.routstr.com`, unconfirmed by this probe). No claim in
this document is invented; where nothing was seen, the verdict says so.
## Verbatim probe output
```
=== routstr_probe: live observation, read-only, spends nothing ===
relays: ["wss://relay.damus.io", "wss://relay.nostr.band", "wss://nos.lol"]
--- subscription 1: kind 38421 ---
(no events)
--- subscription 2: no kind filter, #d = "routstr-provider" (fallback, in case the kind number drifted) ---
(no events)
(no endpoint discovered and no --endpoint override given — skipping capability probe)
NO LIVE PROVIDER OBSERVED
```
(Reproduced in full — the output is 13 lines, nothing was trimmed.)
Process exit code: `0`. No relay-connect timeout warning was printed, so all three relay
connections succeeded within the 10s connect budget; the absence of events is not an artifact
of failed connections.
## Verdict table (per RESEARCH claim under test)
| # | Claim (as cited, `13-RESEARCH.md` "Routstr chat-completions call shape" / A2) | Observed value | Verdict |
|---|---|---|---|
| 1 | Provider-announcement event kind is `38421` | No event of kind 38421 seen on any of the 3 default relays in a 30s window | **NOT OBSERVED** |
| 2 | `d` tag value is `routstr-provider` | No event carrying `#d=routstr-provider` under any kind seen on any of the 3 default relays in a 30s window | **NOT OBSERVED** |
| 3 | Event content carries an `endpoints` field (http/onion) | No matching event at all — nothing to inspect | **NOT OBSERVED** |
| 4 | Event content carries a `models` field | No matching event at all — nothing to inspect | **NOT OBSERVED** |
| 5 | Event content carries a `pricing` field | No matching event at all — nothing to inspect | **NOT OBSERVED** |
| 6 | Payment header is `Authorization: Bearer cashuA…` and/or `X-Cashu:` | No provider endpoint was discovered, so no unauthenticated `GET`/`401`/`402` was ever issued — this probe never got the chance to see a provider name its own header | **NOT OBSERVED** |
| 7 | `POST /v1/chat/completions` is OpenAI-compatible, non-streaming as the primary mode | No endpoint discovered — never called | **NOT OBSERVED** |
| 8 | `tool_calls[].function.arguments` arrives as a JSON-encoded **string** (OpenAI convention, distinct from Ollama/Claude's parsed object) | No live chat-completions call was made (nothing to call it against) | **NOT OBSERVED** |
| 9 | Default relay list (`relay.damus.io`, `relay.nostr.band`, `nos.lol`) is a reasonable place to find providers | All three relays accepted the WebSocket connection (no connect-timeout warning) — the relays themselves are live and reachable, they simply carried no matching event during this window | **NOT OBSERVED** (relay reachability confirmed — **OBSERVED**; but that a Routstr provider announces there was not confirmed) |
**Summary: 0 of 9 claims confirmed, 0 differ, 9 not observed.** This is not evidence the docs
are wrong — it is evidence that no Routstr provider was actively announcing on these three
relays during this 60-second window on 2026-08-03. A young, actively-developed ecosystem with a
small provider count can plausibly have zero announcers online at any given moment; this result
does not distinguish "the docs are stale" from "nobody happened to be broadcasting right now."
## Assumption A2 (RESEARCH.md) — status after this probe
**OBSERVED (of the probe's own execution), DOCS-ONLY (of the protocol itself, unchanged):**
A2's original risk line ("if the docs site's content has drifted from the actual
`routstr-core` implementation, the Rust client's header names or the Nostr filter subscription
could be wrong on first integration attempt") is **neither confirmed nor refuted** by this
probe. A2 does **not hold** in the sense the RESEARCH.md recommendation intended — the
recommended live-relay test ran, but returned no data to validate or invalidate the cited
contract against. The Medium risk rating stands **unchanged and unretired**: `backends/routstr.rs`
(13-13) still cannot be written against an independently-observed contract, only against
`docs.routstr.com` citations, exactly the situation A2 warned about. 13-13's own Task 1
checkpoint (already present in `13-13-PLAN.md` as of this writing) is the correct place this
risk gets resolved into a decision, not this document.
## What this probe did NOT do (scope discipline, per the plan's `<network_probe_scope>`)
- Did not pay a Cashu token, build one, or import any wallet/ecash code.
- Did not send an authenticated request of any kind.
- Did not publish a Nostr event.
- Did not retry beyond the one clean 60-second run recorded above — a single run that
successfully connected to all three relays and returned zero matching events on both filters
is a complete, valid negative result, not an inconclusive one requiring more attempts.
@@ -4,11 +4,16 @@
> Produced at plan time (2026-08-03) for Phase 13, per D-04 ("Routstr is explicitly in scope
> at the user's request").
>
> **Confidence caveat, stated up front:** `13-RESEARCH.md` rates the Routstr protocol
> **MEDIUM** confidence — every row below is derived from `docs.routstr.com` and has **never
> been run against a live provider**. Open Question 3 asks for a spike; plan **13-03** is that
> spike. Rows marked `INTEGRATE — UNCONFIRMED` are ones this matrix cannot yet vouch for.
> **13-03 Task 2 rewrites this file from what the live relay/provider actually returns.**
> **Updated 2026-08-03 by 13-03 Task 2, from a live probe, not from docs.** `13-03` ran
> `core/archipelago/examples/routstr_probe.rs` against the three relays `docs.routstr.com`
> cites (`wss://relay.damus.io`, `wss://relay.nostr.band`, `wss://nos.lol`), subscribing for
> kind-38421 provider announcements plus a `#d=routstr-provider` fallback filter, for 30
> seconds each. All three relays accepted the connection; **zero matching events were found on
> either filter.** Full evidence and a per-claim verdict table are in
> `13-ROUTSTR-FINDINGS.md`. The three rows previously marked as integrate-but-not-yet-confirmed
> are downgraded below to an explicit opt-out with a dated reason, per this file's own prior
> instruction — nothing here is laundered into a confident integration on evidence this probe
> did not obtain.
## Scope note
@@ -21,9 +26,9 @@ API and Ollama's HTTP API are already partially in-tree (`mesh/listener/assist.r
| capability | decision | reason |
|---|---|---|
| `POST /v1/chat/completions` — non-streaming | INTEGRATE | The loop's only required call shape. Every turn where a tool may be emitted must be fully buffered (AI-SPEC §4b.2), so non-streaming is the primary mode, not a fallback. |
| Tool / function calling (`tools[]` request, `tool_calls[]` response) | INTEGRATE — UNCONFIRMED | Required for D-07 parity: the confirm gate must behave identically on Routstr. OpenAI-compat convention says `tool_calls[].function.arguments` is a JSON-encoded **string** (unlike Ollama/Claude's parsed object) — AI-SPEC §3 Pitfall 2. **Not confirmed against a live provider.** 13-03 must settle it before `backends/routstr.rs` is written. |
| Cashu payment attach (`Authorization: Bearer cashuA…` and/or `X-Cashu:`) | INTEGRATE — UNCONFIRMED | D-04/D-05 make paid inference the point of the integration. Two header spellings are documented; the spike determines which the live provider accepts, and the client must not guess. |
| Provider discovery over Nostr (kind `38421`) | INTEGRATE — UNCONFIRMED | D-04 says providers/models/prices are "discovered over Nostr". Reuses `nostr_discovery.rs::build_nostr_client` (Tor-aware). Event kind, `d` tag value and content schema are all cited-not-verified. |
| Tool / function calling (`tools[]` request, `tool_calls[]` response) | OPT-OUT | not observable — no live provider reachable on 2026-08-03 (probed all 3 default relays, kind 38421 + `#d=routstr-provider` fallback, 30s each, zero matching events — `13-ROUTSTR-FINDINGS.md`). 13-13 Task 1 (`checkpoint:decision`, already present in `13-13-PLAN.md`) decides whether to proceed against docs-only, add a fail-loud probe-first step, or defer this leg. |
| Cashu payment attach (`Authorization: Bearer cashuA…` and/or `X-Cashu:`) | OPT-OUT | not observable — no live provider reachable on 2026-08-03 (no endpoint was ever discovered to issue the 401/402 probe against — `13-ROUTSTR-FINDINGS.md` verdict row 6). 13-13 Task 1 decides the fallback path. |
| Provider discovery over Nostr (kind `38421`) | OPT-OUT | not observable — no live provider reachable on 2026-08-03 (zero kind-38421 events and zero `#d=routstr-provider` events across all 3 default relays in a 30s window each; all 3 relays connected successfully, so this is an absence of announcers, not a connectivity failure — `13-ROUTSTR-FINDINGS.md`). 13-13 Task 1 decides the fallback path. |
| Model listing (from the discovered provider event / `GET /v1/models`) | INTEGRATE | Routstr's model id is not a constant in this codebase — it comes from the provider. Without listing there is nothing to select. |
| Price listing (sats per model, from the provider event) | INTEGRATE | D-05's budget ceiling is arithmetic over a price. `auto_pay_token(…, price_sats)` cannot be called without one. |
| Provider selection strategy among multiple advertised providers | INTEGRATE | Explicitly delegated to Claude's discretion in CONTEXT.md. Implemented as: cheapest advertised price for the requested model that is affordable under the remaining `PaymentPolicy` budget, preferring an onion endpoint when Tor is up. |
@@ -37,14 +42,19 @@ API and Ollama's HTTP API are already partially in-tree (`mesh/listener/assist.r
## Opt-out audit
Every `OPT-OUT` row above carries a one-line reason. Six opt-outs, six reasons. No row is
marked INTEGRATE on confidence this matrix does not have — the three genuinely uncertain
capabilities are marked `INTEGRATE — UNCONFIRMED` rather than laundered into a clean
`INTEGRATE`.
Every opt-out row above carries a one-line reason. Nine opt-outs, nine reasons: the original
six plus the three rows 13-03 Task 2 downgraded from an integrate-but-not-yet-confirmed state
after a live probe found no reachable provider (2026-08-03). No row is marked as a confident
integration on confidence this matrix does not have — every claim that could not be observed
was downgraded with a dated, evidenced reason rather than left as an optimistic integration.
## Gate
`13-13` (Routstr backend + D-05 budget ceiling) **must not begin** until `13-03` has replaced
the three `UNCONFIRMED` rows with live-observed facts, or has recorded that no live provider
was reachable — in which case 13-13's own first task is a `checkpoint:decision` on whether to
ship a docs-only client or defer the Routstr leg of D-04 with a named residual.
`13-03` ran. No live provider was reachable (`13-ROUTSTR-FINDINGS.md`). Per this file's own
prior instruction, that means **13-13 may not proceed directly** — its first task must be a
`checkpoint:decision`. This is already true of `13-13-PLAN.md` as written: Task 1 is
`type="checkpoint:decision" gate="blocking"` with exactly the three options this situation
calls for (`proceed-observed`, `proceed-docs-with-probe-first`, `defer-with-residual`), and its
own acceptance criteria require it to read this `## Gate` section and quote it. No edit to
`13-13-PLAN.md` was needed or made by this plan — 13-03's job was to produce the evidence that
checkpoint reads, not to alter the checkpoint itself.