Rotating LND's macaroons was an SSH-only script, which in practice meant it did
not happen — while a macaroon is a bearer token with no revocation and no expiry,
so anything that ever read one keeps the ability to spend until they are
replaced. Settings → Lightning credentials now does it behind the node password,
shows a step checklist, and refuses to report success unless it has confirmed the
node identity and channel census are unchanged.
Three findings from performing a real rotation on a dev node, each fixed here:
1. BTCPay was left holding a dead credential, silently. Its connection string
carries the macaroon INLINE (LND's datadir is owned by its container subuid,
so btcpay cannot bind-mount the file), and the daemon only regenerates that
secret when LND's TLS cert thumbprint changes — which macaroon rotation does
not touch. Result: btcpay up, LND up, both healthy, every Lightning payment
failing, nothing anywhere saying why.
2. Rewriting the secret is not enough to fix it. `secret_env_hash` makes the
change visible as env drift, but the reconcile loop runs `ExistingOnly` at
boot AND periodically, and there it deliberately leaves running
restart-sensitive apps untouched — observed once per tick for half an hour on
the dev node. So this reuses FED-07's `credential_rotated` carve-out via a new
default-no-op `ContainerOrchestrator::mark_credential_rotated`, on the same
reasoning: restart sensitivity protects apps that are working, and this one is
working only in appearance. The shell script cannot reach an in-process flag,
so it removes the container and lets desired-state recovery rebuild it.
3. LND stayed locked forever on a loaded node. The unlocker is only served after
channel.db/graph.db/wallet.db open, measured at 2m38s on a box running 30
containers; the unlock helper gave up at ~60s. That is not a harmless retry —
reconcile records the post-start hook as failed, restarts LND, and the slow
open begins again, so the wallet never opens and every LND-dependent app stays
broken. The not-ready budget is now ~10 minutes; a genuinely wrong password
still exits on the first pass via `all_rejected`.
Safety properties worth not regressing:
- No macaroon content in any response, error, log line or the polled progress
feed — digests and byte counts only.
- Rotation unlocks via a new `unlock_existing_wallet_no_wipe`, so there is no
code path from "rotate my credentials" to `recreate_wallet_destructively`. A
wallet whose password this node lacks fails the rotation with the wallet intact.
- Channels are compared as active+inactive totals, not `num_active_channels`,
which legitimately dips after any restart while peers reconnect.
- Backup verified by file count before anything is deleted.
Verified: cargo check + fmt clean, 6 new unit tests and the 6 existing
container::lnd tests pass, vue-tsc clean, and the built bundle contains the three
new RPC method names (the frontend build can silently no-op).
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
BITCOIN-RPC-PROXY-EXPOSURE.md's port claims verify against code (Bitcoin RPC on
127.0.0.1:8332, the bitcoin-ui proxy on 127.0.0.1:8334). But its incident
narrative named a specific node's LAN address (192.168.63.240, five times) on a
subnet the earlier 192.168.1.x sweep did not cover. Replaced with the RFC 5737
documentation address 192.0.2.240. The incident content — the exposure, the
probes, the fix — is unchanged and remains a legitimate public security record.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The repo is source code and guidelines only. Nothing about how Archipelago's
own fleet is run, or how the team works, stays in it.
Untracked (kept on disk, gitignored) — 250 files:
- .planning/ (199) and loop/ — internal development process
- fleet operations tooling that targets specific nodes: deploy-to-target,
deploy-tailscale, deploy-config-defaults, setup-target-dev, setup-aiui-server,
setup-https-dev, debug-frontend, node-profile, fleet-fips-pair/unpair,
image-recipe/sync-from-live.sh
- image-recipe/INTEGRATION-GUIDE.md and docs/multinode-testing-plan.md, both of
which are live-server workflow and fleet node inventories
- the Phase 10 on-node verification and evidence records, which cite .planning/
as their evidence base
KEY-05-ENTROPY-ENFORCEMENT.md was initially moved out with the other Phase 10
docs and then put back: it is cited as normative rationale from ten places in
the codebase, including core/clippy.toml, which bans rand::thread_rng and
points at it for the reason. That makes it a guideline, not an internal record.
Node names removed from source (48 occurrences across comments, manifests and
test fixtures): archi-dev-box, archy-x250*, shorty-s, framework-pt,
zaza-optiplex, archi-thinkpad. Comments keep the engineering context and the
date, which is what carried the meaning; the machine name did not.
Three of those were live test values rather than comments and were replaced
with valid stand-ins, not prose: two mDNS hostnames and a mesh peer name.
An earlier pass substituted "a test node" into a hostname assertion, producing
an invalid hostname; caught and fixed as test-node.local.
Wipe mechanism: .local-only/manifest.txt inventories every local-only path and
.local-only/wipe.sh deletes them on one confirmation, refusing to touch
anything git still tracks. Both are themselves untracked, so the public repo
does not carry a map of internal filenames.
Verified: cargo check -p archipelago --all-features clean; archipelago-container
75/75 tests pass; appOrigin vitest 7/7; audit-secrets 5/5; every relative link
in tracked markdown resolves (0 broken).
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Controlled test on archi-dev-box with operator approval. The daemon was
stopped first so the reconciler could not repair the state before the
re-exposure was confirmed — without a confirmed 200, the later 401 would
be consistent with the state never having been broken at all.
1. stale conf installed + container restarted -> POST /bitcoin-rpc/
returned 200 with a real block height and Allow-Origin: *
2. daemon started 20:00:36, nothing else touched
3. 20:02:19 reconcile rendered the conf and logged the expected warn
line naming bitcoin-ui/archy-bitcoin-ui, then restarted it
4. POST -> 401, Allow-Origin origin-scoped
5. conf byte-identical to the pre-test known-good, container healthy
Both halves are now proven on real hardware: a05956c4's template (the
gate works) and f6b5245b's delivery path (the gate reaches a container
the reconciler had been skipping).
Also records the operator's decision AGAINST credential rotation — no
macaroon, no Bitcoin RPC password — with the trade it accepts stated
plainly, so it is not silently re-litigated later.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The node is closed and verified 401 with origin-scoped CORS. But an
unrelated bitcoin-ui rebuild at 18:36 cleared the stale conf before the
reconcile fix was deployed at 19:06, so the 401 proves a05956c4's
template and NOT the delivery path f6b5245b adds.
Window 14 closed (exposure gone, verified). Window 15 opened for the
delivery path, which is deployed but never exercised — bitcoin-ui is
still in the uninstall marker, so this node depends on that untested
path the next time its config has to change.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The half that landed correctly (LND, clean 401) made the half that did
not harder to notice, because the first check an operator would run
returns a pass. Records the probes, the three-fact root cause, and the
pass condition for re-probing a node.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
