archy/docs/security/LND-MACAROON-ROTATION.md

# Rotating this node's Lightning credentials

A Lightning macaroon is a **bearer token**: whoever holds one can spend from the
node's wallet. There is no revocation list and no expiry. If a macaroon is ever
read by something you do not control — a leaked endpoint, a screenshot, a phone
that has since been lost, an app that ran a version with a published
vulnerability — that ability persists until the macaroons are rotated.

Rotation is therefore a **routine operator action**, not an emergency procedure.
Two paths do the same work:

| Path | Use when |
|---|---|
| **Dashboard** — Settings → *Lightning credentials* | Normal case. Password-confirmed, shows progress, repairs BTCPay for you. |
| **`scripts/security/rotate-lnd-macaroon.sh`** | No dashboard reachable, or you want a detect-only report. |

## What rotation actually does

LND derives every macaroon it issues from a root key in `macaroons.db`. Remove
that root key plus the issued `*.macaroon` files, restart, and LND mints a fresh
root key and a fresh set of macaroons when the wallet unlocks. Every macaroon
issued before that moment — including any an attacker holds — stops verifying.

## Why your funds and channels survive

Macaroons are bearer tokens, not keys. Coins live in `wallet.db` and channel
state in `channel.db`; channels are secured by the node's identity and channel
keys, none of which are derived from the macaroon root key. Neither database is
opened, moved or deleted.

Both paths **prove** this rather than asserting it: they record the node's
identity pubkey and its channel census before rotating, and refuse to report
success if either differs afterwards.

Two details in that check are deliberate and should not be "tightened":

- **Channels are compared as a total, not as `num_active_channels`.** The active
  count only counts channels whose peer is currently online, so it legitimately
  dips for minutes after *any* restart while peers reconnect. Asserting on it
  alone would abort a perfectly healthy rotation.
- **`wallet.db` is not compared byte-for-byte.** btcwallet records chain-sync
  progress inside it, so the file changes on every start. Asserting byte-identity
  would fire a frightening false alarm on a completely healthy rotation.

## What it never does

- No macaroon **content** reaches a response, an error, a log line, or the
  progress feed the dashboard polls. Everything reported is a SHA-256 digest or a
  byte count — enough to prove the material changed without disclosing it to
  whoever is reading the screen.
- No path from "rotate my credentials" to "delete my wallet". LND's boot path
  self-heals a wallet no candidate password can open by wiping and recreating it;
  correct for an unattended boot, catastrophic here. Rotation unlocks through
  `container::lnd::unlock_existing_wallet_no_wipe`, so a wallet whose password
  this node does not hold surfaces as a **failed rotation** with the wallet
  intact.

## The BTCPay coupling — the part that bites

**BTCPay Server keeps its own inline copy of the admin macaroon**, and it cannot
self-heal. LND's data directory is owned by its container's mapped uid, so BTCPay
cannot bind-mount the macaroon file (EACCES across the userns boundary). The
connection string therefore carries the macaroon as hex:

```
type=lnd-rest;server=https://lnd:8080/;macaroon=<hex>;certthumbprint=<hex>
```

delivered as the `btcpay-lnd-connection` secret file. Rotate the macaroons and
that copy becomes a dead credential. Nothing notices on its own, because the
daemon only regenerates this secret when LND's **TLS cert thumbprint** changes —
and macaroon rotation does not touch the cert.

The resulting state is the dangerous one: **BTCPay is up, LND is up, both report
healthy, and every Lightning invoice BTCPay tries to create fails.**

Repair needs two things, and one without the other is cosmetic:

1. **Rewrite the secret** (`container::lnd::rewrite_btcpay_lnd_connection_secret`).
   This is what makes the change visible: `secret_env_hash` is derived from the
   resolved secret contents, so a changed file reads as label drift on the
   running container.
2. **Recreate the container.** `btcpay-server` is on the restart-sensitive list,
   and the reconcile loop runs in `ExistingOnly` mode *always* — boot and
   periodic alike — where env drift on a restart-sensitive app is detected and
   then deliberately skipped. Rewriting the secret alone therefore changes
   nothing that is running. Observed directly on a development node, once per
   tick, for half an hour:

   ```
   container drift detected during boot reconcile; leaving running
   restart-sensitive app untouched app_id=btcpay-server
   ```

   The dashboard path calls
   `ContainerOrchestrator::mark_credential_rotated("btcpay-server")`, which is
   the flag the drift check consults to override restart-sensitivity. It is the
   same carve-out FED-07 added for the Fedimint gateway, and the reasoning is
   identical: restart sensitivity protects apps that are *working*, and this one
   is working only in appearance.

**The shell script cannot set that in-process flag**, so it does the equivalent
from outside: it deletes the secret (the daemon regenerates it within a tick),
then removes the `btcpay-server` container so the orchestrator's own
desired-state recovery rebuilds it around unchanged data. That recovery is what
makes this safe rather than a hand-rolled remove-and-run — it fires because the
app is still installed and was in the last running-containers snapshot. The
script then prints the commands to confirm it actually happened, because a
failure here is invisible.

## Slow nodes: the unlock budget

LND opens `channel.db`, `graph.db` and `wallet.db` before it serves the unlocker
at all, and on a busy node that is genuinely slow — **2m38s measured on a box
running 30 containers**. The unlock helper used to give up after ~60s, which on
such a node could never succeed.

That timeout was not a harmless retry. Reconcile records the post-start hook as
failed, restarts LND, and the slow database open starts over: a restart loop that
leaves the wallet permanently locked and every LND-dependent app (BTCPay's
internal node included) broken, on exactly the nodes least able to afford it.

The not-ready budget is now ~10 minutes (`UNLOCK_NOT_READY_ATTEMPTS`). Waiting
longer costs nothing, because a genuinely wrong password still exits on the first
pass through the candidate list — the `all_rejected` fast path is untouched.

## Verifying a rotation

The dashboard shows all of this. From a shell:

```bash
# 1. Fingerprint changed (digest only — never print the macaroon)
sudo sha256sum /var/lib/archipelago/lnd/data/chain/bitcoin/mainnet/admin.macaroon

# 2. Same node, same channels
podman exec lnd lncli --network=mainnet getinfo \
  | python3 -c 'import json,sys; d=json.load(sys.stdin); print(d["identity_pubkey"], \
      d["num_active_channels"] + d["num_inactive_channels"], d["num_pending_channels"])'

# 3. BTCPay is carrying the CURRENT macaroon, not the rotated-out one
CUR=$(sudo od -An -v -tx1 /var/lib/archipelago/lnd/data/chain/bitcoin/mainnet/admin.macaroon | tr -d ' \n')
SEC=$(sudo sed -n 's/.*macaroon=\([0-9a-f]*\).*/\1/p' /var/lib/archipelago/secrets/btcpay-lnd-connection)
[ "$CUR" = "$SEC" ] && echo "current" || echo "STALE — BTCPay's Lightning is broken"

# 4. BTCPay was actually recreated (a silent failure looks like success)
podman inspect btcpay-server --format '{{.Created}}'
```

Check 3 is the one people skip, and it is the one that fails.

## Afterwards

- **Re-pair every wallet app**, Zeus most importantly. Open the Lightning app in
  the dashboard and scan the pairing QR again; it serves the new macaroon.
- **Delete the backup once re-pairing is done.** Both paths back the old material
  up to `/var/lib/archipelago/lnd/macaroon-rotation-<stamp>` (0700) so a mistake
  is recoverable. That directory holds the **old root key** and is still
  sensitive: `sudo rm -rf <path>`.

## Related

- `docs/security/BITCOIN-RPC-PROXY-EXPOSURE.md` — the leak that first made
  rotation necessary, and the operator decision not to rotate the fleet for it.
- `scripts/security/rotate-lnd-macaroon.sh` — the shell path, including its
  ordering guard (it refuses to rotate on a binary that still leaks
  `/lnd-connect-info`, since the new macaroon would leak within seconds).