archipelagoandClaude Opus 5
d15cd58d7f
Demo images / Build & push demo images (push) Successful in 3m34s
Rotating LND's macaroons was an SSH-only script, which in practice meant it did not happen — while a macaroon is a bearer token with no revocation and no expiry, so anything that ever read one keeps the ability to spend until they are replaced. Settings → Lightning credentials now does it behind the node password, shows a step checklist, and refuses to report success unless it has confirmed the node identity and channel census are unchanged. Three findings from performing a real rotation on a dev node, each fixed here: 1. BTCPay was left holding a dead credential, silently. Its connection string carries the macaroon INLINE (LND's datadir is owned by its container subuid, so btcpay cannot bind-mount the file), and the daemon only regenerates that secret when LND's TLS cert thumbprint changes — which macaroon rotation does not touch. Result: btcpay up, LND up, both healthy, every Lightning payment failing, nothing anywhere saying why. 2. Rewriting the secret is not enough to fix it. `secret_env_hash` makes the change visible as env drift, but the reconcile loop runs `ExistingOnly` at boot AND periodically, and there it deliberately leaves running restart-sensitive apps untouched — observed once per tick for half an hour on the dev node. So this reuses FED-07's `credential_rotated` carve-out via a new default-no-op `ContainerOrchestrator::mark_credential_rotated`, on the same reasoning: restart sensitivity protects apps that are working, and this one is working only in appearance. The shell script cannot reach an in-process flag, so it removes the container and lets desired-state recovery rebuild it. 3. LND stayed locked forever on a loaded node. The unlocker is only served after channel.db/graph.db/wallet.db open, measured at 2m38s on a box running 30 containers; the unlock helper gave up at ~60s. That is not a harmless retry — reconcile records the post-start hook as failed, restarts LND, and the slow open begins again, so the wallet never opens and every LND-dependent app stays broken. The not-ready budget is now ~10 minutes; a genuinely wrong password still exits on the first pass via `all_rejected`. Safety properties worth not regressing: - No macaroon content in any response, error, log line or the polled progress feed — digests and byte counts only. - Rotation unlocks via a new `unlock_existing_wallet_no_wipe`, so there is no code path from "rotate my credentials" to `recreate_wallet_destructively`. A wallet whose password this node lacks fails the rotation with the wallet intact. - Channels are compared as active+inactive totals, not `num_active_channels`, which legitimately dips after any restart while peers reconnect. - Backup verified by file count before anything is deleted. Verified: cargo check + fmt clean, 6 new unit tests and the 6 existing container::lnd tests pass, vue-tsc clean, and the built bundle contains the three new RPC method names (the frontend build can silently no-op). Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
167 lines
8.2 KiB
Markdown
167 lines
8.2 KiB
Markdown
# Rotating this node's Lightning credentials
|
|
|
|
A Lightning macaroon is a **bearer token**: whoever holds one can spend from the
|
|
node's wallet. There is no revocation list and no expiry. If a macaroon is ever
|
|
read by something you do not control — a leaked endpoint, a screenshot, a phone
|
|
that has since been lost, an app that ran a version with a published
|
|
vulnerability — that ability persists until the macaroons are rotated.
|
|
|
|
Rotation is therefore a **routine operator action**, not an emergency procedure.
|
|
Two paths do the same work:
|
|
|
|
| Path | Use when |
|
|
|---|---|
|
|
| **Dashboard** — Settings → *Lightning credentials* | Normal case. Password-confirmed, shows progress, repairs BTCPay for you. |
|
|
| **`scripts/security/rotate-lnd-macaroon.sh`** | No dashboard reachable, or you want a detect-only report. |
|
|
|
|
## What rotation actually does
|
|
|
|
LND derives every macaroon it issues from a root key in `macaroons.db`. Remove
|
|
that root key plus the issued `*.macaroon` files, restart, and LND mints a fresh
|
|
root key and a fresh set of macaroons when the wallet unlocks. Every macaroon
|
|
issued before that moment — including any an attacker holds — stops verifying.
|
|
|
|
## Why your funds and channels survive
|
|
|
|
Macaroons are bearer tokens, not keys. Coins live in `wallet.db` and channel
|
|
state in `channel.db`; channels are secured by the node's identity and channel
|
|
keys, none of which are derived from the macaroon root key. Neither database is
|
|
opened, moved or deleted.
|
|
|
|
Both paths **prove** this rather than asserting it: they record the node's
|
|
identity pubkey and its channel census before rotating, and refuse to report
|
|
success if either differs afterwards.
|
|
|
|
Two details in that check are deliberate and should not be "tightened":
|
|
|
|
- **Channels are compared as a total, not as `num_active_channels`.** The active
|
|
count only counts channels whose peer is currently online, so it legitimately
|
|
dips for minutes after *any* restart while peers reconnect. Asserting on it
|
|
alone would abort a perfectly healthy rotation.
|
|
- **`wallet.db` is not compared byte-for-byte.** btcwallet records chain-sync
|
|
progress inside it, so the file changes on every start. Asserting byte-identity
|
|
would fire a frightening false alarm on a completely healthy rotation.
|
|
|
|
## What it never does
|
|
|
|
- No macaroon **content** reaches a response, an error, a log line, or the
|
|
progress feed the dashboard polls. Everything reported is a SHA-256 digest or a
|
|
byte count — enough to prove the material changed without disclosing it to
|
|
whoever is reading the screen.
|
|
- No path from "rotate my credentials" to "delete my wallet". LND's boot path
|
|
self-heals a wallet no candidate password can open by wiping and recreating it;
|
|
correct for an unattended boot, catastrophic here. Rotation unlocks through
|
|
`container::lnd::unlock_existing_wallet_no_wipe`, so a wallet whose password
|
|
this node does not hold surfaces as a **failed rotation** with the wallet
|
|
intact.
|
|
|
|
## The BTCPay coupling — the part that bites
|
|
|
|
**BTCPay Server keeps its own inline copy of the admin macaroon**, and it cannot
|
|
self-heal. LND's data directory is owned by its container's mapped uid, so BTCPay
|
|
cannot bind-mount the macaroon file (EACCES across the userns boundary). The
|
|
connection string therefore carries the macaroon as hex:
|
|
|
|
```
|
|
type=lnd-rest;server=https://lnd:8080/;macaroon=<hex>;certthumbprint=<hex>
|
|
```
|
|
|
|
delivered as the `btcpay-lnd-connection` secret file. Rotate the macaroons and
|
|
that copy becomes a dead credential. Nothing notices on its own, because the
|
|
daemon only regenerates this secret when LND's **TLS cert thumbprint** changes —
|
|
and macaroon rotation does not touch the cert.
|
|
|
|
The resulting state is the dangerous one: **BTCPay is up, LND is up, both report
|
|
healthy, and every Lightning invoice BTCPay tries to create fails.**
|
|
|
|
Repair needs two things, and one without the other is cosmetic:
|
|
|
|
1. **Rewrite the secret** (`container::lnd::rewrite_btcpay_lnd_connection_secret`).
|
|
This is what makes the change visible: `secret_env_hash` is derived from the
|
|
resolved secret contents, so a changed file reads as label drift on the
|
|
running container.
|
|
2. **Recreate the container.** `btcpay-server` is on the restart-sensitive list,
|
|
and the reconcile loop runs in `ExistingOnly` mode *always* — boot and
|
|
periodic alike — where env drift on a restart-sensitive app is detected and
|
|
then deliberately skipped. Rewriting the secret alone therefore changes
|
|
nothing that is running. Observed directly on a development node, once per
|
|
tick, for half an hour:
|
|
|
|
```
|
|
container drift detected during boot reconcile; leaving running
|
|
restart-sensitive app untouched app_id=btcpay-server
|
|
```
|
|
|
|
The dashboard path calls
|
|
`ContainerOrchestrator::mark_credential_rotated("btcpay-server")`, which is
|
|
the flag the drift check consults to override restart-sensitivity. It is the
|
|
same carve-out FED-07 added for the Fedimint gateway, and the reasoning is
|
|
identical: restart sensitivity protects apps that are *working*, and this one
|
|
is working only in appearance.
|
|
|
|
**The shell script cannot set that in-process flag**, so it does the equivalent
|
|
from outside: it deletes the secret (the daemon regenerates it within a tick),
|
|
then removes the `btcpay-server` container so the orchestrator's own
|
|
desired-state recovery rebuilds it around unchanged data. That recovery is what
|
|
makes this safe rather than a hand-rolled remove-and-run — it fires because the
|
|
app is still installed and was in the last running-containers snapshot. The
|
|
script then prints the commands to confirm it actually happened, because a
|
|
failure here is invisible.
|
|
|
|
## Slow nodes: the unlock budget
|
|
|
|
LND opens `channel.db`, `graph.db` and `wallet.db` before it serves the unlocker
|
|
at all, and on a busy node that is genuinely slow — **2m38s measured on a box
|
|
running 30 containers**. The unlock helper used to give up after ~60s, which on
|
|
such a node could never succeed.
|
|
|
|
That timeout was not a harmless retry. Reconcile records the post-start hook as
|
|
failed, restarts LND, and the slow database open starts over: a restart loop that
|
|
leaves the wallet permanently locked and every LND-dependent app (BTCPay's
|
|
internal node included) broken, on exactly the nodes least able to afford it.
|
|
|
|
The not-ready budget is now ~10 minutes (`UNLOCK_NOT_READY_ATTEMPTS`). Waiting
|
|
longer costs nothing, because a genuinely wrong password still exits on the first
|
|
pass through the candidate list — the `all_rejected` fast path is untouched.
|
|
|
|
## Verifying a rotation
|
|
|
|
The dashboard shows all of this. From a shell:
|
|
|
|
```bash
|
|
# 1. Fingerprint changed (digest only — never print the macaroon)
|
|
sudo sha256sum /var/lib/archipelago/lnd/data/chain/bitcoin/mainnet/admin.macaroon
|
|
|
|
# 2. Same node, same channels
|
|
podman exec lnd lncli --network=mainnet getinfo \
|
|
| python3 -c 'import json,sys; d=json.load(sys.stdin); print(d["identity_pubkey"], \
|
|
d["num_active_channels"] + d["num_inactive_channels"], d["num_pending_channels"])'
|
|
|
|
# 3. BTCPay is carrying the CURRENT macaroon, not the rotated-out one
|
|
CUR=$(sudo od -An -v -tx1 /var/lib/archipelago/lnd/data/chain/bitcoin/mainnet/admin.macaroon | tr -d ' \n')
|
|
SEC=$(sudo sed -n 's/.*macaroon=\([0-9a-f]*\).*/\1/p' /var/lib/archipelago/secrets/btcpay-lnd-connection)
|
|
[ "$CUR" = "$SEC" ] && echo "current" || echo "STALE — BTCPay's Lightning is broken"
|
|
|
|
# 4. BTCPay was actually recreated (a silent failure looks like success)
|
|
podman inspect btcpay-server --format '{{.Created}}'
|
|
```
|
|
|
|
Check 3 is the one people skip, and it is the one that fails.
|
|
|
|
## Afterwards
|
|
|
|
- **Re-pair every wallet app**, Zeus most importantly. Open the Lightning app in
|
|
the dashboard and scan the pairing QR again; it serves the new macaroon.
|
|
- **Delete the backup once re-pairing is done.** Both paths back the old material
|
|
up to `/var/lib/archipelago/lnd/macaroon-rotation-<stamp>` (0700) so a mistake
|
|
is recoverable. That directory holds the **old root key** and is still
|
|
sensitive: `sudo rm -rf <path>`.
|
|
|
|
## Related
|
|
|
|
- `docs/security/BITCOIN-RPC-PROXY-EXPOSURE.md` — the leak that first made
|
|
rotation necessary, and the operator decision not to rotate the fleet for it.
|
|
- `scripts/security/rotate-lnd-macaroon.sh` — the shell path, including its
|
|
ordering guard (it refuses to rotate on a binary that still leaks
|
|
`/lnd-connect-info`, since the new macaroon would leak within seconds).
|