archipelagoandClaude Opus 5
cfa6c6cb0d
Demo images / Build & push demo images (push) Successful in 3m55s
Reviewing the rotation against what this dev node actually did to LND today —
25 restarts, most of them automatic — surfaced a race the code did not defend
against. Between "stop LND" and "start LND" the rotation owns a stopped
container whose credential material is being deleted, and two background actors
step in there unasked: the health monitor restarts any container it finds
stopped, and the reconciler starts one whose unit is enabled.
Either brings LND back up mid-deletion. LND re-mints macaroons.db on unlock, so
the deletion loop would race a live process writing that file, or "succeed"
against material that had already been regenerated — and the operator would be
told they had rotated while the old root key was still in service. That is the
one outcome this feature exists to make impossible.
It now holds `app_ops::op_lock("lnd")` for the whole rotation. That is the lock
both actors already consult (`lifecycle_op_in_flight`; the health monitor
reaches it through `lifecycle_op_covers_container`), and it additionally
serialises against the package.start/stop/restart workers, so "Restart" on
Lightning mid-rotation queues instead of interleaving. A rotation requested
while one of those is in flight fails fast with a short explanation rather than
waiting silently behind an operation that may itself take minutes.
Deliberately NOT the `user-stopped` marker `recreate_wallet_destructively` uses
for its own window. That marker is a file on disk: a rotation that died between
marking and clearing would leave Lightning suppressed permanently, fixable only
by finding and editing JSON on the node. A lock guard releases when it drops, on
every path including a panic.
Also mocks the three RPCs in mock-backend.js, so the Settings section can be
driven end-to-end without a node — the dev preview otherwise shows only a load
error. The mock advances one step per poll rather than on a timer, which is
deterministic and makes every intermediate state observable.
Verified: cargo check + fmt clean, 6/6 rotation tests, 12/12 component tests,
mock-rpc-parity unchanged (its 2 failures are the in-flight Reticulum panel, not
this), and the three RPCs driven against the live mock through the full arc —
idle → started → 7 steps → ok with the channel count preserved, plus both
password-rejection paths.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
192 lines
9.7 KiB
Markdown
192 lines
9.7 KiB
Markdown
# Rotating this node's Lightning credentials
|
|
|
|
A Lightning macaroon is a **bearer token**: whoever holds one can spend from the
|
|
node's wallet. There is no revocation list and no expiry. If a macaroon is ever
|
|
read by something you do not control — a leaked endpoint, a screenshot, a phone
|
|
that has since been lost, an app that ran a version with a published
|
|
vulnerability — that ability persists until the macaroons are rotated.
|
|
|
|
Rotation is therefore a **routine operator action**, not an emergency procedure.
|
|
Two paths do the same work:
|
|
|
|
| Path | Use when |
|
|
|---|---|
|
|
| **Dashboard** — Settings → *Lightning credentials* | Normal case. Password-confirmed, shows progress, repairs BTCPay for you. |
|
|
| **`scripts/security/rotate-lnd-macaroon.sh`** | No dashboard reachable, or you want a detect-only report. |
|
|
|
|
## What rotation actually does
|
|
|
|
LND derives every macaroon it issues from a root key in `macaroons.db`. Remove
|
|
that root key plus the issued `*.macaroon` files, restart, and LND mints a fresh
|
|
root key and a fresh set of macaroons when the wallet unlocks. Every macaroon
|
|
issued before that moment — including any an attacker holds — stops verifying.
|
|
|
|
## Why your funds and channels survive
|
|
|
|
Macaroons are bearer tokens, not keys. Coins live in `wallet.db` and channel
|
|
state in `channel.db`; channels are secured by the node's identity and channel
|
|
keys, none of which are derived from the macaroon root key. Neither database is
|
|
opened, moved or deleted.
|
|
|
|
Both paths **prove** this rather than asserting it: they record the node's
|
|
identity pubkey and its channel census before rotating, and refuse to report
|
|
success if either differs afterwards.
|
|
|
|
Two details in that check are deliberate and should not be "tightened":
|
|
|
|
- **Channels are compared as a total, not as `num_active_channels`.** The active
|
|
count only counts channels whose peer is currently online, so it legitimately
|
|
dips for minutes after *any* restart while peers reconnect. Asserting on it
|
|
alone would abort a perfectly healthy rotation.
|
|
- **`wallet.db` is not compared byte-for-byte.** btcwallet records chain-sync
|
|
progress inside it, so the file changes on every start. Asserting byte-identity
|
|
would fire a frightening false alarm on a completely healthy rotation.
|
|
|
|
## What it never does
|
|
|
|
- No macaroon **content** reaches a response, an error, a log line, or the
|
|
progress feed the dashboard polls. Everything reported is a SHA-256 digest or a
|
|
byte count — enough to prove the material changed without disclosing it to
|
|
whoever is reading the screen.
|
|
- No path from "rotate my credentials" to "delete my wallet". LND's boot path
|
|
self-heals a wallet no candidate password can open by wiping and recreating it;
|
|
correct for an unattended boot, catastrophic here. Rotation unlocks through
|
|
`container::lnd::unlock_existing_wallet_no_wipe`, so a wallet whose password
|
|
this node does not hold surfaces as a **failed rotation** with the wallet
|
|
intact.
|
|
|
|
## Nothing else may touch LND mid-rotation
|
|
|
|
Between "stop LND" and "start LND" the rotation owns a stopped container whose
|
|
credential material is being deleted. Two background actors would step in there
|
|
unasked: the **health monitor** restarts any container it finds stopped, and the
|
|
**reconciler** starts one whose unit is enabled. Either brings LND back up
|
|
mid-deletion — and LND re-mints `macaroons.db` on unlock, so the deletion loop
|
|
would race a live process writing that file, or "succeed" against material that
|
|
had already been regenerated. The operator would be told they had rotated while
|
|
the old root key was still in service.
|
|
|
|
The rotation therefore holds `app_ops::op_lock("lnd")` for its whole duration.
|
|
That is the lock both actors already consult (`lifecycle_op_in_flight`, reached
|
|
in the health monitor via `lifecycle_op_covers_container`), and it also
|
|
serialises against the `package.start`/`stop`/`restart` workers, so an operator
|
|
hitting "Restart" on Lightning mid-rotation queues rather than interleaving. A
|
|
rotation requested while one of those is running fails fast with a short
|
|
explanation instead of waiting silently.
|
|
|
|
Deliberately **not** the `user-stopped` marker that `recreate_wallet_destructively`
|
|
uses for its own window: that marker is a file on disk, so a rotation that died
|
|
between marking and clearing would leave Lightning suppressed *permanently* —
|
|
fixable only by finding and editing JSON on the node. The lock guard releases
|
|
when it drops, on every path including a panic.
|
|
|
|
## The BTCPay coupling — the part that bites
|
|
|
|
**BTCPay Server keeps its own inline copy of the admin macaroon**, and it cannot
|
|
self-heal. LND's data directory is owned by its container's mapped uid, so BTCPay
|
|
cannot bind-mount the macaroon file (EACCES across the userns boundary). The
|
|
connection string therefore carries the macaroon as hex:
|
|
|
|
```
|
|
type=lnd-rest;server=https://lnd:8080/;macaroon=<hex>;certthumbprint=<hex>
|
|
```
|
|
|
|
delivered as the `btcpay-lnd-connection` secret file. Rotate the macaroons and
|
|
that copy becomes a dead credential. Nothing notices on its own, because the
|
|
daemon only regenerates this secret when LND's **TLS cert thumbprint** changes —
|
|
and macaroon rotation does not touch the cert.
|
|
|
|
The resulting state is the dangerous one: **BTCPay is up, LND is up, both report
|
|
healthy, and every Lightning invoice BTCPay tries to create fails.**
|
|
|
|
Repair needs two things, and one without the other is cosmetic:
|
|
|
|
1. **Rewrite the secret** (`container::lnd::rewrite_btcpay_lnd_connection_secret`).
|
|
This is what makes the change visible: `secret_env_hash` is derived from the
|
|
resolved secret contents, so a changed file reads as label drift on the
|
|
running container.
|
|
2. **Recreate the container.** `btcpay-server` is on the restart-sensitive list,
|
|
and the reconcile loop runs in `ExistingOnly` mode *always* — boot and
|
|
periodic alike — where env drift on a restart-sensitive app is detected and
|
|
then deliberately skipped. Rewriting the secret alone therefore changes
|
|
nothing that is running. Observed directly on a development node, once per
|
|
tick, for half an hour:
|
|
|
|
```
|
|
container drift detected during boot reconcile; leaving running
|
|
restart-sensitive app untouched app_id=btcpay-server
|
|
```
|
|
|
|
The dashboard path calls
|
|
`ContainerOrchestrator::mark_credential_rotated("btcpay-server")`, which is
|
|
the flag the drift check consults to override restart-sensitivity. It is the
|
|
same carve-out FED-07 added for the Fedimint gateway, and the reasoning is
|
|
identical: restart sensitivity protects apps that are *working*, and this one
|
|
is working only in appearance.
|
|
|
|
**The shell script cannot set that in-process flag**, so it does the equivalent
|
|
from outside: it deletes the secret (the daemon regenerates it within a tick),
|
|
then removes the `btcpay-server` container so the orchestrator's own
|
|
desired-state recovery rebuilds it around unchanged data. That recovery is what
|
|
makes this safe rather than a hand-rolled remove-and-run — it fires because the
|
|
app is still installed and was in the last running-containers snapshot. The
|
|
script then prints the commands to confirm it actually happened, because a
|
|
failure here is invisible.
|
|
|
|
## Slow nodes: the unlock budget
|
|
|
|
LND opens `channel.db`, `graph.db` and `wallet.db` before it serves the unlocker
|
|
at all, and on a busy node that is genuinely slow — **2m38s measured on a box
|
|
running 30 containers**. The unlock helper used to give up after ~60s, which on
|
|
such a node could never succeed.
|
|
|
|
That timeout was not a harmless retry. Reconcile records the post-start hook as
|
|
failed, restarts LND, and the slow database open starts over: a restart loop that
|
|
leaves the wallet permanently locked and every LND-dependent app (BTCPay's
|
|
internal node included) broken, on exactly the nodes least able to afford it.
|
|
|
|
The not-ready budget is now ~10 minutes (`UNLOCK_NOT_READY_ATTEMPTS`). Waiting
|
|
longer costs nothing, because a genuinely wrong password still exits on the first
|
|
pass through the candidate list — the `all_rejected` fast path is untouched.
|
|
|
|
## Verifying a rotation
|
|
|
|
The dashboard shows all of this. From a shell:
|
|
|
|
```bash
|
|
# 1. Fingerprint changed (digest only — never print the macaroon)
|
|
sudo sha256sum /var/lib/archipelago/lnd/data/chain/bitcoin/mainnet/admin.macaroon
|
|
|
|
# 2. Same node, same channels
|
|
podman exec lnd lncli --network=mainnet getinfo \
|
|
| python3 -c 'import json,sys; d=json.load(sys.stdin); print(d["identity_pubkey"], \
|
|
d["num_active_channels"] + d["num_inactive_channels"], d["num_pending_channels"])'
|
|
|
|
# 3. BTCPay is carrying the CURRENT macaroon, not the rotated-out one
|
|
CUR=$(sudo od -An -v -tx1 /var/lib/archipelago/lnd/data/chain/bitcoin/mainnet/admin.macaroon | tr -d ' \n')
|
|
SEC=$(sudo sed -n 's/.*macaroon=\([0-9a-f]*\).*/\1/p' /var/lib/archipelago/secrets/btcpay-lnd-connection)
|
|
[ "$CUR" = "$SEC" ] && echo "current" || echo "STALE — BTCPay's Lightning is broken"
|
|
|
|
# 4. BTCPay was actually recreated (a silent failure looks like success)
|
|
podman inspect btcpay-server --format '{{.Created}}'
|
|
```
|
|
|
|
Check 3 is the one people skip, and it is the one that fails.
|
|
|
|
## Afterwards
|
|
|
|
- **Re-pair every wallet app**, Zeus most importantly. Open the Lightning app in
|
|
the dashboard and scan the pairing QR again; it serves the new macaroon.
|
|
- **Delete the backup once re-pairing is done.** Both paths back the old material
|
|
up to `/var/lib/archipelago/lnd/macaroon-rotation-<stamp>` (0700) so a mistake
|
|
is recoverable. That directory holds the **old root key** and is still
|
|
sensitive: `sudo rm -rf <path>`.
|
|
|
|
## Related
|
|
|
|
- `docs/security/BITCOIN-RPC-PROXY-EXPOSURE.md` — the leak that first made
|
|
rotation necessary, and the operator decision not to rotate the fleet for it.
|
|
- `scripts/security/rotate-lnd-macaroon.sh` — the shell path, including its
|
|
ordering guard (it refuses to rotate on a binary that still leaks
|
|
`/lnd-connect-info`, since the new macaroon would leak within seconds).
|