archy/docs/incident-2026-09-15-minibits-already-redeemed.md

# Incident — 2026-09-15: Minibits Cashu claim stuck retrying an already-redeemed token

## Report

User: "The cashu server is unable to get it's tokens from nostr on
[affected node]" — clarified as the Cashu **client wallet**
(Minibits `@minibits.cash` Lightning-address receive flow), not a mint
server. UI showed: *"a payment arrived but couldn't be redeemed yet (1)"*.

## Root cause

`wallet::minibits::claim_and_redeem` (`core/archipelago/src/wallet/minibits.rs`)
polls Nostr relays for NIP-04-encrypted Cashu tokens sent to the node's
`@minibits.cash` address, decrypts them, and redeems them at the mint. A
token that fails to redeem is kept in `MinibitsState.pending_claims` and
retried on the next poll — by design, so a *transient* failure (mint briefly
down, decrypt hiccup) never drops real money.

But one queued claim had already been redeemed (mint error **11001 "Token
Already Spent"** — most likely double-delivered by the relay, or redeemed
by an earlier run before a crash lost track of it). That's a *terminal*
condition, not a transient one: the code didn't distinguish the two, so it
retried the same dead claim every ~6 seconds forever:

```
WARN archipelago::wallet::ecash: Failed to swap proofs from mint https://mint.minibits.cash/Bitcoin:
     This ecash has already been redeemed — it can't be claimed twice.: {"code":11001,"detail":"Token Already Spent"}
WARN archipelago::wallet::minibits: Minibits claim decrypted but failed to redeem (...); will retry next poll
```

Confirmed via `sudo journalctl -u archipelago.service` on the affected node,
and via `/var/lib/archipelago/wallet/minibits.json`, which had exactly one
`pending_claims` entry. Each poll also unconditionally queried all three
`CLAIM_RELAY_URLS` (`relay.minibits.cash`, `relay.damus.io`, `nos.lol`)
instead of the primary relay only, adding needless churn and leaking the
wallet's Nostr pubkey to two relays it didn't need to touch — `relay.damus.io`
was additionally failing NIP-42 auth / 503ing on every poll.

**No funds were at risk** — an already-redeemed token has zero remaining
value. The only symptom was a permanently stuck "couldn't be redeemed yet"
banner and wasted relay connections.

### Why this had already been "fixed" once and came back

This exact bug (terminal-11001 handling + relay-query reduction) was fixed
on 2026-09-09 on branch `feat/minibits-lnurl-receive` (commits `4e410d7`,
`489995c`) and pushed to `origin`. **That branch was never merged into
`main`.** `main` carries its own, independently-diverged rewrite of
`minibits.rs` that never got those two hardening fixes. The affected node
OTA'd to `1.8.16-alpha` (built from `main`) earlier on 2026-09-15, so the bug
resurfaced on the first replayed/double-delivered claim after that update.

## Fix

Two parts:

### 1. Immediate unstick (affected node, operational, no code change)

- Backed up `/var/lib/archipelago/wallet/minibits.json`.
- Stopped `archipelago.service`, emptied `pending_claims` (`[]`) in the
  state file, restarted the service.
- Verified via `journalctl` that polling resumed cleanly with no further
  "already been redeemed" warnings.

### 2. Code fix, ported into `main`

- **`core/archipelago/src/wallet/mint_client.rs`**: exposed the existing
  NUT error-code-11001 translation as a public constant,
  `ALREADY_REDEEMED_MSG`, so other modules can recognize it without
  duplicating the string.
- **`core/archipelago/src/wallet/minibits.rs`**:
  - Added `is_already_redeemed(&anyhow::Error) -> bool`, checking the error
    chain for `ALREADY_REDEEMED_MSG`.
  - In the claim redeem loop, a redeem failure matching
    `is_already_redeemed` is now dropped (logged at `info!`, not retried)
    instead of being pushed back onto `pending_claims`. Every other failure
    still retries next poll, unchanged.
  - `fetch_relay_dms` now connects to `RELAY_URL` (the Minibits relay)
    alone first via `try_connect_relay`, and only adds the two public
    fallback relays (`relay.damus.io`, `nos.lol`) if that primary relay is
    unreachable. Also paginates the DM fetch (200/page, capped at 5 pages)
    so a backlog larger than one page can't silently strand older DMs
    behind an un-advanced watermark.

Deliberately **not** ported from the unmerged branch: its `STATE_LOCK`
skip-if-busy guard and per-claim attempt-count backstop. `main`'s existing
`MINIBITS_STATE_LOCK` already fully serializes claim polls (blocks rather
than skips — a different but equally valid way to close the same race), and
an attempt-count backstop would have required reshaping the `PendingClaim`
enum for marginal extra protection beyond what the 11001 fix already covers.

## Verification

- `cargo build -p archipelago` — clean, no new warnings.
- `cargo test -p archipelago --bin archipelago wallet::minibits` — existing
  suite still green (see PR/commit for the run).
- Live on the affected node: claim poll loop confirmed quiet post-unstick
  (only `relay.minibits.cash` connects logged, no redeem-failure warnings).

## Lesson (recorded in memory)

A fix that lives only on an unmerged feature branch is not a fix that's
actually deployed. Before trusting a memory or changelog claim that
something "shipped," check which branch the running/released build was
built from (`git log <branch>..main` / `main..<branch>`) rather than
assuming a pushed branch was merged.