docs(13): routstr protocol findings + coverage matrix from live probe
13-ROUTSTR-FINDINGS.md records the routstr_probe run against all three docs.routstr.com default relays: zero kind-38421 events and zero #d=routstr-provider fallback events in a 30s window each, with all three relay connections succeeding (ruling out a connectivity failure as the explanation). Every claim is labelled OBSERVED or DOCS-ONLY per the plan's scope note; RESEARCH assumption A2 is recorded as neither confirmed nor refuted, risk unchanged. COVERAGE.md's three former "INTEGRATE — UNCONFIRMED" rows (tool calling, Cashu payment header, Nostr provider discovery) are downgraded to explicit opt-outs with dated, evidenced reasons — zero rows retain unconfirmed-integration status. The Gate section states 13-13 may not proceed directly and must open with a checkpoint: decision, which 13-13-PLAN.md's Task 1 already is (proceed-observed / proceed-docs-with-probe-first / defer-with-residual) — no edit to 13-13-PLAN.md was needed. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
This commit is contained in:
co-authored by
Claude Opus 5
parent
ea90ef05a5
commit
f8987d12f3
+85
@@ -0,0 +1,85 @@
|
||||
# Routstr Protocol Findings — live probe (13-03)
|
||||
|
||||
**Date run:** 2026-08-03
|
||||
**Probe:** `core/archipelago/examples/routstr_probe.rs` (commit `ea90ef05`), run by hand with
|
||||
`cd core && ./target/debug/examples/routstr_probe` (equivalently `cargo run --example
|
||||
routstr_probe`).
|
||||
**Relays probed:** `wss://relay.damus.io`, `wss://relay.nostr.band`, `wss://nos.lol` — the
|
||||
three default relays cited in `docs.routstr.com` (RESEARCH.md, `13-RESEARCH.md` "Routstr
|
||||
chat-completions call shape").
|
||||
**Filters used:** (1) kind `38421`, limit 50, 30s wait; (2) no kind restriction, `#d` tag
|
||||
`routstr-provider`, limit 50, 30s wait — a fallback in case the kind number in the docs had
|
||||
drifted.
|
||||
**Result:** **NO LIVE PROVIDER OBSERVED.** Zero events matched either filter across all three
|
||||
relays within the wait budget. No endpoint was discovered, so the capability-probe half
|
||||
(`/v1/models`, `/`) never ran — there was nothing to point it at.
|
||||
|
||||
Every claim below is labelled **OBSERVED** (this probe run produced the evidence) or
|
||||
**DOCS-ONLY** (still resting on `docs.routstr.com`, unconfirmed by this probe). No claim in
|
||||
this document is invented; where nothing was seen, the verdict says so.
|
||||
|
||||
## Verbatim probe output
|
||||
|
||||
```
|
||||
=== routstr_probe: live observation, read-only, spends nothing ===
|
||||
relays: ["wss://relay.damus.io", "wss://relay.nostr.band", "wss://nos.lol"]
|
||||
|
||||
--- subscription 1: kind 38421 ---
|
||||
(no events)
|
||||
|
||||
--- subscription 2: no kind filter, #d = "routstr-provider" (fallback, in case the kind number drifted) ---
|
||||
(no events)
|
||||
|
||||
(no endpoint discovered and no --endpoint override given — skipping capability probe)
|
||||
|
||||
NO LIVE PROVIDER OBSERVED
|
||||
```
|
||||
|
||||
(Reproduced in full — the output is 13 lines, nothing was trimmed.)
|
||||
|
||||
Process exit code: `0`. No relay-connect timeout warning was printed, so all three relay
|
||||
connections succeeded within the 10s connect budget; the absence of events is not an artifact
|
||||
of failed connections.
|
||||
|
||||
## Verdict table (per RESEARCH claim under test)
|
||||
|
||||
| # | Claim (as cited, `13-RESEARCH.md` "Routstr chat-completions call shape" / A2) | Observed value | Verdict |
|
||||
|---|---|---|---|
|
||||
| 1 | Provider-announcement event kind is `38421` | No event of kind 38421 seen on any of the 3 default relays in a 30s window | **NOT OBSERVED** |
|
||||
| 2 | `d` tag value is `routstr-provider` | No event carrying `#d=routstr-provider` under any kind seen on any of the 3 default relays in a 30s window | **NOT OBSERVED** |
|
||||
| 3 | Event content carries an `endpoints` field (http/onion) | No matching event at all — nothing to inspect | **NOT OBSERVED** |
|
||||
| 4 | Event content carries a `models` field | No matching event at all — nothing to inspect | **NOT OBSERVED** |
|
||||
| 5 | Event content carries a `pricing` field | No matching event at all — nothing to inspect | **NOT OBSERVED** |
|
||||
| 6 | Payment header is `Authorization: Bearer cashuA…` and/or `X-Cashu:` | No provider endpoint was discovered, so no unauthenticated `GET`/`401`/`402` was ever issued — this probe never got the chance to see a provider name its own header | **NOT OBSERVED** |
|
||||
| 7 | `POST /v1/chat/completions` is OpenAI-compatible, non-streaming as the primary mode | No endpoint discovered — never called | **NOT OBSERVED** |
|
||||
| 8 | `tool_calls[].function.arguments` arrives as a JSON-encoded **string** (OpenAI convention, distinct from Ollama/Claude's parsed object) | No live chat-completions call was made (nothing to call it against) | **NOT OBSERVED** |
|
||||
| 9 | Default relay list (`relay.damus.io`, `relay.nostr.band`, `nos.lol`) is a reasonable place to find providers | All three relays accepted the WebSocket connection (no connect-timeout warning) — the relays themselves are live and reachable, they simply carried no matching event during this window | **NOT OBSERVED** (relay reachability confirmed — **OBSERVED**; but that a Routstr provider announces there was not confirmed) |
|
||||
|
||||
**Summary: 0 of 9 claims confirmed, 0 differ, 9 not observed.** This is not evidence the docs
|
||||
are wrong — it is evidence that no Routstr provider was actively announcing on these three
|
||||
relays during this 60-second window on 2026-08-03. A young, actively-developed ecosystem with a
|
||||
small provider count can plausibly have zero announcers online at any given moment; this result
|
||||
does not distinguish "the docs are stale" from "nobody happened to be broadcasting right now."
|
||||
|
||||
## Assumption A2 (RESEARCH.md) — status after this probe
|
||||
|
||||
**OBSERVED (of the probe's own execution), DOCS-ONLY (of the protocol itself, unchanged):**
|
||||
A2's original risk line ("if the docs site's content has drifted from the actual
|
||||
`routstr-core` implementation, the Rust client's header names or the Nostr filter subscription
|
||||
could be wrong on first integration attempt") is **neither confirmed nor refuted** by this
|
||||
probe. A2 does **not hold** in the sense the RESEARCH.md recommendation intended — the
|
||||
recommended live-relay test ran, but returned no data to validate or invalidate the cited
|
||||
contract against. The Medium risk rating stands **unchanged and unretired**: `backends/routstr.rs`
|
||||
(13-13) still cannot be written against an independently-observed contract, only against
|
||||
`docs.routstr.com` citations, exactly the situation A2 warned about. 13-13's own Task 1
|
||||
checkpoint (already present in `13-13-PLAN.md` as of this writing) is the correct place this
|
||||
risk gets resolved into a decision, not this document.
|
||||
|
||||
## What this probe did NOT do (scope discipline, per the plan's `<network_probe_scope>`)
|
||||
|
||||
- Did not pay a Cashu token, build one, or import any wallet/ecash code.
|
||||
- Did not send an authenticated request of any kind.
|
||||
- Did not publish a Nostr event.
|
||||
- Did not retry beyond the one clean 60-second run recorded above — a single run that
|
||||
successfully connected to all three relays and returned zero matching events on both filters
|
||||
is a complete, valid negative result, not an inconclusive one requiring more attempts.
|
||||
+26
-16
@@ -4,11 +4,16 @@
|
||||
> Produced at plan time (2026-08-03) for Phase 13, per D-04 ("Routstr is explicitly in scope
|
||||
> at the user's request").
|
||||
>
|
||||
> **Confidence caveat, stated up front:** `13-RESEARCH.md` rates the Routstr protocol
|
||||
> **MEDIUM** confidence — every row below is derived from `docs.routstr.com` and has **never
|
||||
> been run against a live provider**. Open Question 3 asks for a spike; plan **13-03** is that
|
||||
> spike. Rows marked `INTEGRATE — UNCONFIRMED` are ones this matrix cannot yet vouch for.
|
||||
> **13-03 Task 2 rewrites this file from what the live relay/provider actually returns.**
|
||||
> **Updated 2026-08-03 by 13-03 Task 2, from a live probe, not from docs.** `13-03` ran
|
||||
> `core/archipelago/examples/routstr_probe.rs` against the three relays `docs.routstr.com`
|
||||
> cites (`wss://relay.damus.io`, `wss://relay.nostr.band`, `wss://nos.lol`), subscribing for
|
||||
> kind-38421 provider announcements plus a `#d=routstr-provider` fallback filter, for 30
|
||||
> seconds each. All three relays accepted the connection; **zero matching events were found on
|
||||
> either filter.** Full evidence and a per-claim verdict table are in
|
||||
> `13-ROUTSTR-FINDINGS.md`. The three rows previously marked as integrate-but-not-yet-confirmed
|
||||
> are downgraded below to an explicit opt-out with a dated reason, per this file's own prior
|
||||
> instruction — nothing here is laundered into a confident integration on evidence this probe
|
||||
> did not obtain.
|
||||
|
||||
## Scope note
|
||||
|
||||
@@ -21,9 +26,9 @@ API and Ollama's HTTP API are already partially in-tree (`mesh/listener/assist.r
|
||||
| capability | decision | reason |
|
||||
|---|---|---|
|
||||
| `POST /v1/chat/completions` — non-streaming | INTEGRATE | The loop's only required call shape. Every turn where a tool may be emitted must be fully buffered (AI-SPEC §4b.2), so non-streaming is the primary mode, not a fallback. |
|
||||
| Tool / function calling (`tools[]` request, `tool_calls[]` response) | INTEGRATE — UNCONFIRMED | Required for D-07 parity: the confirm gate must behave identically on Routstr. OpenAI-compat convention says `tool_calls[].function.arguments` is a JSON-encoded **string** (unlike Ollama/Claude's parsed object) — AI-SPEC §3 Pitfall 2. **Not confirmed against a live provider.** 13-03 must settle it before `backends/routstr.rs` is written. |
|
||||
| Cashu payment attach (`Authorization: Bearer cashuA…` and/or `X-Cashu:`) | INTEGRATE — UNCONFIRMED | D-04/D-05 make paid inference the point of the integration. Two header spellings are documented; the spike determines which the live provider accepts, and the client must not guess. |
|
||||
| Provider discovery over Nostr (kind `38421`) | INTEGRATE — UNCONFIRMED | D-04 says providers/models/prices are "discovered over Nostr". Reuses `nostr_discovery.rs::build_nostr_client` (Tor-aware). Event kind, `d` tag value and content schema are all cited-not-verified. |
|
||||
| Tool / function calling (`tools[]` request, `tool_calls[]` response) | OPT-OUT | not observable — no live provider reachable on 2026-08-03 (probed all 3 default relays, kind 38421 + `#d=routstr-provider` fallback, 30s each, zero matching events — `13-ROUTSTR-FINDINGS.md`). 13-13 Task 1 (`checkpoint:decision`, already present in `13-13-PLAN.md`) decides whether to proceed against docs-only, add a fail-loud probe-first step, or defer this leg. |
|
||||
| Cashu payment attach (`Authorization: Bearer cashuA…` and/or `X-Cashu:`) | OPT-OUT | not observable — no live provider reachable on 2026-08-03 (no endpoint was ever discovered to issue the 401/402 probe against — `13-ROUTSTR-FINDINGS.md` verdict row 6). 13-13 Task 1 decides the fallback path. |
|
||||
| Provider discovery over Nostr (kind `38421`) | OPT-OUT | not observable — no live provider reachable on 2026-08-03 (zero kind-38421 events and zero `#d=routstr-provider` events across all 3 default relays in a 30s window each; all 3 relays connected successfully, so this is an absence of announcers, not a connectivity failure — `13-ROUTSTR-FINDINGS.md`). 13-13 Task 1 decides the fallback path. |
|
||||
| Model listing (from the discovered provider event / `GET /v1/models`) | INTEGRATE | Routstr's model id is not a constant in this codebase — it comes from the provider. Without listing there is nothing to select. |
|
||||
| Price listing (sats per model, from the provider event) | INTEGRATE | D-05's budget ceiling is arithmetic over a price. `auto_pay_token(…, price_sats)` cannot be called without one. |
|
||||
| Provider selection strategy among multiple advertised providers | INTEGRATE | Explicitly delegated to Claude's discretion in CONTEXT.md. Implemented as: cheapest advertised price for the requested model that is affordable under the remaining `PaymentPolicy` budget, preferring an onion endpoint when Tor is up. |
|
||||
@@ -37,14 +42,19 @@ API and Ollama's HTTP API are already partially in-tree (`mesh/listener/assist.r
|
||||
|
||||
## Opt-out audit
|
||||
|
||||
Every `OPT-OUT` row above carries a one-line reason. Six opt-outs, six reasons. No row is
|
||||
marked INTEGRATE on confidence this matrix does not have — the three genuinely uncertain
|
||||
capabilities are marked `INTEGRATE — UNCONFIRMED` rather than laundered into a clean
|
||||
`INTEGRATE`.
|
||||
Every opt-out row above carries a one-line reason. Nine opt-outs, nine reasons: the original
|
||||
six plus the three rows 13-03 Task 2 downgraded from an integrate-but-not-yet-confirmed state
|
||||
after a live probe found no reachable provider (2026-08-03). No row is marked as a confident
|
||||
integration on confidence this matrix does not have — every claim that could not be observed
|
||||
was downgraded with a dated, evidenced reason rather than left as an optimistic integration.
|
||||
|
||||
## Gate
|
||||
|
||||
`13-13` (Routstr backend + D-05 budget ceiling) **must not begin** until `13-03` has replaced
|
||||
the three `UNCONFIRMED` rows with live-observed facts, or has recorded that no live provider
|
||||
was reachable — in which case 13-13's own first task is a `checkpoint:decision` on whether to
|
||||
ship a docs-only client or defer the Routstr leg of D-04 with a named residual.
|
||||
`13-03` ran. No live provider was reachable (`13-ROUTSTR-FINDINGS.md`). Per this file's own
|
||||
prior instruction, that means **13-13 may not proceed directly** — its first task must be a
|
||||
`checkpoint:decision`. This is already true of `13-13-PLAN.md` as written: Task 1 is
|
||||
`type="checkpoint:decision" gate="blocking"` with exactly the three options this situation
|
||||
calls for (`proceed-observed`, `proceed-docs-with-probe-first`, `defer-with-residual`), and its
|
||||
own acceptance criteria require it to read this `## Gate` section and quote it. No edit to
|
||||
`13-13-PLAN.md` was needed or made by this plan — 13-03's job was to produce the evidence that
|
||||
checkpoint reads, not to alter the checkpoint itself.
|
||||
|
||||
Reference in New Issue
Block a user