feat(lnd): rotate Lightning macaroons from the dashboard, and stop stranding BTCPay
Demo images / Build & push demo images (push) Successful in 3m34s
Demo images / Build & push demo images (push) Successful in 3m34s
Rotating LND's macaroons was an SSH-only script, which in practice meant it did not happen — while a macaroon is a bearer token with no revocation and no expiry, so anything that ever read one keeps the ability to spend until they are replaced. Settings → Lightning credentials now does it behind the node password, shows a step checklist, and refuses to report success unless it has confirmed the node identity and channel census are unchanged. Three findings from performing a real rotation on a dev node, each fixed here: 1. BTCPay was left holding a dead credential, silently. Its connection string carries the macaroon INLINE (LND's datadir is owned by its container subuid, so btcpay cannot bind-mount the file), and the daemon only regenerates that secret when LND's TLS cert thumbprint changes — which macaroon rotation does not touch. Result: btcpay up, LND up, both healthy, every Lightning payment failing, nothing anywhere saying why. 2. Rewriting the secret is not enough to fix it. `secret_env_hash` makes the change visible as env drift, but the reconcile loop runs `ExistingOnly` at boot AND periodically, and there it deliberately leaves running restart-sensitive apps untouched — observed once per tick for half an hour on the dev node. So this reuses FED-07's `credential_rotated` carve-out via a new default-no-op `ContainerOrchestrator::mark_credential_rotated`, on the same reasoning: restart sensitivity protects apps that are working, and this one is working only in appearance. The shell script cannot reach an in-process flag, so it removes the container and lets desired-state recovery rebuild it. 3. LND stayed locked forever on a loaded node. The unlocker is only served after channel.db/graph.db/wallet.db open, measured at 2m38s on a box running 30 containers; the unlock helper gave up at ~60s. That is not a harmless retry — reconcile records the post-start hook as failed, restarts LND, and the slow open begins again, so the wallet never opens and every LND-dependent app stays broken. The not-ready budget is now ~10 minutes; a genuinely wrong password still exits on the first pass via `all_rejected`. Safety properties worth not regressing: - No macaroon content in any response, error, log line or the polled progress feed — digests and byte counts only. - Rotation unlocks via a new `unlock_existing_wallet_no_wipe`, so there is no code path from "rotate my credentials" to `recreate_wallet_destructively`. A wallet whose password this node lacks fails the rotation with the wallet intact. - Channels are compared as active+inactive totals, not `num_active_channels`, which legitimately dips after any restart while peers reconnect. - Backup verified by file count before anything is deleted. Verified: cargo check + fmt clean, 6 new unit tests and the 6 existing container::lnd tests pass, vue-tsc clean, and the built bundle contains the three new RPC method names (the frontend build can silently no-op). Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
This commit is contained in:
archipelago
co-authored by
Claude Opus 5
Claude Opus 5
parent
b7e57ca9cf
commit
d15cd58d7f
@@ -1,5 +1,11 @@
|
||||
# Changelog
|
||||
|
||||
## Unreleased
|
||||
|
||||
- **You can now replace your Lightning connection keys from Settings, without touching a terminal.** The tokens wallet apps like Zeus use to reach your node are bearer keys: anything that has ever seen one can spend from your node until they are replaced, and there is no way to cancel one individually. Replacing them was previously a script you had to SSH in and run, which in practice meant it never happened. Settings → Lightning credentials now shows when yours were issued, which node they belong to and how many channels must survive, then does the whole job behind your node password — with a step-by-step progress list, and a refusal to call it a success unless it has confirmed your node identity and every channel came back. Your coins and channels are not touched: nothing is closed, and the wallet is never re-created. Afterwards you re-pair Zeus by scanning the Lightning app's QR code again.
|
||||
- **Replacing those keys no longer silently breaks BTCPay Server.** BTCPay holds its own copy of the key, and that copy cannot repair itself — so a node that replaced its keys ended up with BTCPay running, healthy, and unable to take a single Lightning payment, with nothing anywhere saying why. The dashboard now updates BTCPay's copy as part of the run and restarts it around its existing data, and the Settings screen warns you if it finds a node already stuck in that state. The command-line script fixes the same gap.
|
||||
- **Lightning stops getting stuck locked on a busy node.** Lightning opens its databases before it will accept the password that unlocks the wallet, and on a loaded node that took nearly three minutes — longer than the node was willing to wait. Giving up restarted Lightning, which started the slow open again, so the wallet stayed locked forever and everything depending on it stayed broken. The node now waits as long as it takes. A genuinely wrong password still fails immediately.
|
||||
|
||||
## v1.7.126-alpha (2026-08-07)
|
||||
|
||||
- **The most important fix in this release: the update button could take you backwards onto a version withdrawn for a security hole.** BTCPay Server published 2.4.2 to close a flaw that was being actively exploited — a way past two-factor authentication. Nodes that had already moved to 2.4.2 were then shown an "Update" button offering 2.3.9, the very release being withdrawn, and taking it would have rolled the node back onto the vulnerable version. The cause was that the node only asked whether the two version numbers differed, never which was newer, so any stale record anywhere could present a rollback as an upgrade. It now refuses to offer a lower version as an update, so a stale record fails safe instead of becoming a trap. BTCPay itself is on 2.4.2, and every place that still named the old version — including the fallback installer, which would have installed it outright — has been corrected.
|
||||
|
||||
@@ -146,6 +146,12 @@ impl RpcHandler {
|
||||
"lnd.seed-backup-status" => self.handle_lnd_seed_backup_status().await,
|
||||
"lnd.seed-reveal" => self.handle_lnd_seed_reveal(params).await,
|
||||
"lnd.seed-backup-ack" => self.handle_lnd_seed_backup_ack().await,
|
||||
// Lightning credential rotation. `rotate` re-verifies the node
|
||||
// password and returns immediately; the work runs in the
|
||||
// background and the UI polls `-progress`.
|
||||
"lnd.macaroon-status" => self.handle_lnd_macaroon_status().await,
|
||||
"lnd.rotate-macaroons" => self.handle_lnd_rotate_macaroons(params).await,
|
||||
"lnd.macaroon-rotation-progress" => self.handle_lnd_macaroon_rotation_progress().await,
|
||||
|
||||
// Multi-identity management
|
||||
"identity.list" => self.handle_identity_list(params).await,
|
||||
|
||||
@@ -0,0 +1,821 @@
|
||||
//! LND macaroon rotation, driven from the dashboard.
|
||||
//!
|
||||
//! A macaroon is a bearer token: whoever holds it can spend from this node's
|
||||
//! Lightning wallet. Anything that ever read one — a leaked endpoint, a shared
|
||||
//! screenshot, a paired phone that has since been lost, a BTCPay instance that
|
||||
//! ran a version with a published vulnerability — keeps that ability until the
|
||||
//! macaroons are rotated. Rotation is therefore a routine operator action, and
|
||||
//! before this module the only way to perform it was to SSH into the node and
|
||||
//! run `scripts/security/rotate-lnd-macaroon.sh` by hand.
|
||||
//!
|
||||
//! ## What rotation actually does
|
||||
//!
|
||||
//! LND derives every macaroon it issues from a root key in `macaroons.db`.
|
||||
//! Remove that root key and the issued macaroon files, restart, and LND mints a
|
||||
//! fresh root key and a fresh set of macaroons on unlock. Every previously
|
||||
//! issued macaroon — including any an attacker holds — stops verifying.
|
||||
//!
|
||||
//! ## Why funds and channels survive
|
||||
//!
|
||||
//! Macaroons are bearer tokens, not keys. Coins live in `wallet.db` and channel
|
||||
//! state in `channel.db`; channels are secured by the node's identity and
|
||||
//! channel keys, none of which derive from the macaroon root key. This code
|
||||
//! never opens, moves or deletes either database. What it does instead is
|
||||
//! *prove* they survived: it records the node's identity pubkey and channel
|
||||
//! census before rotating and refuses to report success if either changed.
|
||||
//!
|
||||
//! Deliberately NOT asserted: that `wallet.db` is byte-identical. btcwallet
|
||||
//! records chain-sync progress inside it, so the file legitimately changes on
|
||||
//! every start — asserting byte-identity would fire a frightening false alarm
|
||||
//! on a completely healthy rotation.
|
||||
//!
|
||||
//! ## What it never does
|
||||
//!
|
||||
//! No macaroon *content* is read into a response, an error, a log line or the
|
||||
//! progress feed the UI polls. Everything reported is a SHA-256 digest or a
|
||||
//! byte count — enough to prove the material changed without disclosing it to
|
||||
//! whoever is looking at the screen.
|
||||
//!
|
||||
//! It also cannot reach LND's destructive wallet-recovery path: the restart
|
||||
//! unlocks via `unlock_existing_wallet_no_wipe`, so a wallet whose password
|
||||
//! this node does not hold surfaces as a failed rotation, never as a wipe.
|
||||
|
||||
use crate::api::rpc::RpcHandler;
|
||||
use anyhow::{Context, Result};
|
||||
use serde::Serialize;
|
||||
use std::sync::{Arc, Mutex, OnceLock};
|
||||
|
||||
use super::LND_REST_BASE_URL;
|
||||
|
||||
/// LND's mainnet macaroon directory. 0700 and owned by the container's mapped
|
||||
/// uid, so every read/write below goes through `sudo -n`.
|
||||
const LND_MAINNET_DIR: &str = "/var/lib/archipelago/lnd/data/chain/bitcoin/mainnet";
|
||||
|
||||
/// Quadlet service for the core LND app. Older nodes run LND as a plain podman
|
||||
/// container with no unit; `restart_lnd` handles both.
|
||||
const LND_SERVICE: &str = "lnd.service";
|
||||
const LND_CONTAINER: &str = "lnd";
|
||||
|
||||
/// Where the orchestrator materialises app secrets. Hardcoded to match
|
||||
/// `prod_orchestrator`'s own default rather than derived from `config.data_dir`
|
||||
/// — writing the BTCPay connection string anywhere the orchestrator does not
|
||||
/// read it would be worse than not writing it at all, because it would look
|
||||
/// like it worked.
|
||||
const SECRETS_DIR: &str = "/var/lib/archipelago/secrets";
|
||||
|
||||
/// Longest we wait for LND to mint a fresh `admin.macaroon` after the restart.
|
||||
/// Generous on purpose: LND opens its databases before serving anything, which
|
||||
/// is minutes on a loaded node.
|
||||
const MACAROON_WAIT_SECS: u64 = 900;
|
||||
|
||||
// ── Progress the UI polls ────────────────────────────────────────────────────
|
||||
|
||||
#[derive(Debug, Clone, Copy, PartialEq, Eq, Serialize)]
|
||||
#[serde(rename_all = "lowercase")]
|
||||
enum StepState {
|
||||
Pending,
|
||||
Running,
|
||||
Done,
|
||||
Failed,
|
||||
/// Ran, decided there was nothing to do, and said so. Distinct from `Done`
|
||||
/// so "BTCPay has no internal Lightning node" never reads as "BTCPay was
|
||||
/// reconnected".
|
||||
Skipped,
|
||||
}
|
||||
|
||||
#[derive(Debug, Clone, Serialize)]
|
||||
struct RotationStep {
|
||||
key: &'static str,
|
||||
label: &'static str,
|
||||
state: StepState,
|
||||
detail: Option<String>,
|
||||
}
|
||||
|
||||
/// Ordered because the UI renders it as a checklist and an operator watching a
|
||||
/// credential rotation should be able to see exactly how far it got.
|
||||
const STEPS: &[(&str, &str)] = &[
|
||||
(
|
||||
"preflight",
|
||||
"Check LND is healthy and record what must survive",
|
||||
),
|
||||
("backup", "Back up the current macaroon material"),
|
||||
("stop", "Stop Lightning"),
|
||||
("remove", "Remove the old root key and issued macaroons"),
|
||||
("start", "Start Lightning and unlock the wallet"),
|
||||
("verify", "Confirm the node and its channels are unchanged"),
|
||||
("btcpay", "Reconnect BTCPay Server to the new credentials"),
|
||||
];
|
||||
|
||||
#[derive(Debug, Clone, Serialize)]
|
||||
pub(crate) struct RotationProgress {
|
||||
running: bool,
|
||||
/// `None` while running, then the verdict. Split from `running` so the UI
|
||||
/// can tell "in progress" from "finished and failed".
|
||||
ok: Option<bool>,
|
||||
started_at: Option<String>,
|
||||
finished_at: Option<String>,
|
||||
error: Option<String>,
|
||||
steps: Vec<RotationStep>,
|
||||
/// Where the old material was copied. Still secret — it is the old root key
|
||||
/// — so the UI tells the operator to delete it once clients are re-paired.
|
||||
backup_path: Option<String>,
|
||||
identity_pubkey: Option<String>,
|
||||
channels_before: Option<u32>,
|
||||
channels_after: Option<u32>,
|
||||
/// Digest of the freshly minted admin macaroon. A digest, never the token.
|
||||
new_admin_macaroon_sha256: Option<String>,
|
||||
}
|
||||
|
||||
impl Default for RotationProgress {
|
||||
fn default() -> Self {
|
||||
Self {
|
||||
running: false,
|
||||
ok: None,
|
||||
started_at: None,
|
||||
finished_at: None,
|
||||
error: None,
|
||||
steps: STEPS
|
||||
.iter()
|
||||
.map(|(key, label)| RotationStep {
|
||||
key,
|
||||
label,
|
||||
state: StepState::Pending,
|
||||
detail: None,
|
||||
})
|
||||
.collect(),
|
||||
backup_path: None,
|
||||
identity_pubkey: None,
|
||||
channels_before: None,
|
||||
channels_after: None,
|
||||
new_admin_macaroon_sha256: None,
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
impl RotationProgress {
|
||||
fn set(&mut self, key: &str, state: StepState, detail: Option<String>) {
|
||||
if let Some(step) = self.steps.iter_mut().find(|s| s.key == key) {
|
||||
step.state = state;
|
||||
if detail.is_some() {
|
||||
step.detail = detail;
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
/// One rotation at a time, process-wide. Two concurrent rotations would race on
|
||||
/// the same files with LND stopped underneath them.
|
||||
fn progress() -> &'static Mutex<RotationProgress> {
|
||||
static PROGRESS: OnceLock<Mutex<RotationProgress>> = OnceLock::new();
|
||||
PROGRESS.get_or_init(|| Mutex::new(RotationProgress::default()))
|
||||
}
|
||||
|
||||
fn with_progress<F: FnOnce(&mut RotationProgress)>(f: F) {
|
||||
if let Ok(mut guard) = progress().lock() {
|
||||
f(&mut guard);
|
||||
}
|
||||
}
|
||||
|
||||
fn snapshot() -> RotationProgress {
|
||||
progress()
|
||||
.lock()
|
||||
.map(|g| g.clone())
|
||||
.unwrap_or_else(|e| e.into_inner().clone())
|
||||
}
|
||||
|
||||
// ── Host helpers ─────────────────────────────────────────────────────────────
|
||||
|
||||
/// `sudo -n <args>`, capturing output. Non-interactive: a node whose sudoers
|
||||
/// does not permit this fails loudly here rather than hanging on a prompt.
|
||||
async fn sudo(args: &[&str]) -> Result<std::process::Output> {
|
||||
let mut cmd = tokio::process::Command::new("sudo");
|
||||
cmd.arg("-n").args(args);
|
||||
cmd.output()
|
||||
.await
|
||||
.with_context(|| format!("sudo -n {}", args.join(" ")))
|
||||
}
|
||||
|
||||
async fn sudo_ok(args: &[&str]) -> Result<()> {
|
||||
let out = sudo(args).await?;
|
||||
if !out.status.success() {
|
||||
anyhow::bail!(
|
||||
"sudo {} exited {}: {}",
|
||||
args.join(" "),
|
||||
out.status,
|
||||
String::from_utf8_lossy(&out.stderr).trim()
|
||||
);
|
||||
}
|
||||
Ok(())
|
||||
}
|
||||
|
||||
/// SHA-256 of a root-owned file, or `None` if it isn't there. Only ever the
|
||||
/// digest — the file's bytes never enter this process.
|
||||
async fn digest_as_root(path: &str) -> Option<String> {
|
||||
let out = sudo(&["sha256sum", path]).await.ok()?;
|
||||
if !out.status.success() {
|
||||
return None;
|
||||
}
|
||||
String::from_utf8_lossy(&out.stdout)
|
||||
.split_whitespace()
|
||||
.next()
|
||||
.map(str::to_string)
|
||||
}
|
||||
|
||||
/// Every file rotation replaces: the issued macaroons plus the root key they
|
||||
/// derive from.
|
||||
///
|
||||
/// Enumerated with `sudo find` rather than a shell glob for a reason worth
|
||||
/// keeping: the directory is 0700 owned by the container's mapped uid, so a
|
||||
/// glob evaluated by this (unprivileged) process expands to nothing. It would
|
||||
/// silently make both the backup and the removal no-ops while every surrounding
|
||||
/// step still reported success.
|
||||
async fn macaroon_files() -> Result<Vec<String>> {
|
||||
let out = sudo(&[
|
||||
"find",
|
||||
LND_MAINNET_DIR,
|
||||
"-maxdepth",
|
||||
"1",
|
||||
"(",
|
||||
"-name",
|
||||
"*.macaroon",
|
||||
"-o",
|
||||
"-name",
|
||||
"macaroons.db",
|
||||
")",
|
||||
])
|
||||
.await?;
|
||||
if !out.status.success() {
|
||||
anyhow::bail!(
|
||||
"listing macaroon material failed: {}",
|
||||
String::from_utf8_lossy(&out.stderr).trim()
|
||||
);
|
||||
}
|
||||
Ok(String::from_utf8_lossy(&out.stdout)
|
||||
.lines()
|
||||
.map(str::trim)
|
||||
.filter(|l| !l.is_empty())
|
||||
.map(str::to_string)
|
||||
.collect())
|
||||
}
|
||||
|
||||
/// True when LND is managed by a generated Quadlet unit on this node. Nodes
|
||||
/// predating the Quadlet migration run a bare podman container instead, and
|
||||
/// stopping the wrong way there means either a no-op or an orphan.
|
||||
async fn lnd_has_quadlet_unit() -> bool {
|
||||
crate::container::quadlet::is_active(LND_SERVICE).await
|
||||
|| tokio::process::Command::new("systemctl")
|
||||
.args(["--user", "cat", LND_SERVICE])
|
||||
.output()
|
||||
.await
|
||||
.map(|o| o.status.success())
|
||||
.unwrap_or(false)
|
||||
}
|
||||
|
||||
async fn stop_lnd() -> Result<()> {
|
||||
if lnd_has_quadlet_unit().await {
|
||||
return crate::container::quadlet::stop_service(LND_SERVICE)
|
||||
.await
|
||||
.context("stopping lnd.service");
|
||||
}
|
||||
podman_scoped(&["stop", LND_CONTAINER]).await
|
||||
}
|
||||
|
||||
async fn start_lnd() -> Result<()> {
|
||||
if lnd_has_quadlet_unit().await {
|
||||
return crate::container::quadlet::enable_now(LND_SERVICE)
|
||||
.await
|
||||
.context("starting lnd.service");
|
||||
}
|
||||
podman_scoped(&["start", LND_CONTAINER]).await
|
||||
}
|
||||
|
||||
/// `podman` inside a transient user scope, matching how the orchestrator and
|
||||
/// health monitor drive rootless containers (keeps it out of the archipelago
|
||||
/// service's cgroup, so an archipelago restart doesn't take LND with it).
|
||||
async fn podman_scoped(args: &[&str]) -> Result<()> {
|
||||
let out = tokio::process::Command::new("systemd-run")
|
||||
.args(["--user", "--scope", "--quiet", "--collect", "podman"])
|
||||
.args(args)
|
||||
.output()
|
||||
.await
|
||||
.with_context(|| format!("systemd-run --user --scope podman {}", args.join(" ")))?;
|
||||
if !out.status.success() {
|
||||
anyhow::bail!(
|
||||
"podman {} failed: {}",
|
||||
args.join(" "),
|
||||
String::from_utf8_lossy(&out.stderr).trim()
|
||||
);
|
||||
}
|
||||
Ok(())
|
||||
}
|
||||
|
||||
// ── LND facts ────────────────────────────────────────────────────────────────
|
||||
|
||||
/// Identity and channel census — the two things that must be identical either
|
||||
/// side of a rotation.
|
||||
#[derive(Debug, Clone, Copy, Default)]
|
||||
struct LndCensus {
|
||||
channels_open: u32,
|
||||
channels_pending: u32,
|
||||
}
|
||||
|
||||
fn lnd_client() -> Result<reqwest::Client> {
|
||||
reqwest::Client::builder()
|
||||
.no_proxy()
|
||||
.timeout(std::time::Duration::from_secs(15))
|
||||
// LND serves its own self-signed cert on loopback; the macaroon, not
|
||||
// the certificate, is what authenticates this call.
|
||||
.danger_accept_invalid_certs(true)
|
||||
.build()
|
||||
.context("building LND REST client")
|
||||
}
|
||||
|
||||
/// `getinfo` using whatever macaroon is on disk right now. Returns the identity
|
||||
/// pubkey and census, or an error describing why LND could not answer.
|
||||
async fn read_census() -> Result<(String, LndCensus)> {
|
||||
let macaroon = super::read_lnd_admin_macaroon()
|
||||
.await
|
||||
.context("reading LND admin macaroon")?;
|
||||
let resp = lnd_client()?
|
||||
.get(format!("{LND_REST_BASE_URL}/v1/getinfo"))
|
||||
.header("Grpc-Metadata-macaroon", hex::encode(&macaroon))
|
||||
.send()
|
||||
.await
|
||||
.context("LND is not answering on its REST port")?;
|
||||
let body: serde_json::Value = resp
|
||||
.json()
|
||||
.await
|
||||
.context("LND returned a response that is not JSON")?;
|
||||
let pubkey = body
|
||||
.get("identity_pubkey")
|
||||
.and_then(|v| v.as_str())
|
||||
.map(str::to_string)
|
||||
.ok_or_else(|| {
|
||||
anyhow::anyhow!(
|
||||
"LND did not report an identity — it is most likely still starting or locked ({})",
|
||||
body.get("message")
|
||||
.and_then(|m| m.as_str())
|
||||
.unwrap_or("no detail")
|
||||
)
|
||||
})?;
|
||||
let num = |k: &str| body.get(k).and_then(|v| v.as_u64()).unwrap_or(0) as u32;
|
||||
Ok((
|
||||
pubkey,
|
||||
LndCensus {
|
||||
// Active + inactive, summed deliberately. `num_active_channels`
|
||||
// counts channels whose peer is currently online, so it legitimately
|
||||
// dips for minutes after ANY restart while peers reconnect —
|
||||
// asserting on it alone would abort a perfectly healthy rotation.
|
||||
// The total number of channels held is the real safety property.
|
||||
channels_open: num("num_active_channels") + num("num_inactive_channels"),
|
||||
channels_pending: num("num_pending_channels"),
|
||||
},
|
||||
))
|
||||
}
|
||||
|
||||
/// Poll until LND answers `getinfo` with a fresh macaroon, or the budget runs
|
||||
/// out. Used after the restart, so "not ready yet" is the expected case for
|
||||
/// most of the wait.
|
||||
async fn wait_for_serving(deadline: std::time::Instant) -> Result<(String, LndCensus)> {
|
||||
let mut last = String::from("LND did not become reachable");
|
||||
while std::time::Instant::now() < deadline {
|
||||
match read_census().await {
|
||||
Ok(v) => return Ok(v),
|
||||
Err(e) => last = format!("{e:#}"),
|
||||
}
|
||||
tokio::time::sleep(std::time::Duration::from_secs(5)).await;
|
||||
}
|
||||
anyhow::bail!("timed out waiting for LND to serve again: {last}")
|
||||
}
|
||||
|
||||
// ── Status ───────────────────────────────────────────────────────────────────
|
||||
|
||||
impl RpcHandler {
|
||||
/// Read-only picture of this node's Lightning credentials: when they were
|
||||
/// issued, what depends on them, and whether anything is already out of
|
||||
/// step. Never returns macaroon content.
|
||||
pub(in crate::api::rpc) async fn handle_lnd_macaroon_status(
|
||||
&self,
|
||||
) -> Result<serde_json::Value> {
|
||||
let admin_path = format!("{LND_MAINNET_DIR}/admin.macaroon");
|
||||
let installed = digest_as_root(&admin_path).await;
|
||||
|
||||
// `stat -c %y` on the macaroon is when LND last minted it, which is the
|
||||
// one date an operator actually wants ("am I still carrying credentials
|
||||
// from before that incident?").
|
||||
let issued_at = match sudo(&["stat", "-c", "%y", &admin_path]).await {
|
||||
Ok(out) if out.status.success() => Some(
|
||||
String::from_utf8_lossy(&out.stdout)
|
||||
.trim()
|
||||
.chars()
|
||||
.take(19)
|
||||
.collect::<String>(),
|
||||
),
|
||||
_ => None,
|
||||
};
|
||||
|
||||
let (identity_pubkey, census, lnd_error) = match read_census().await {
|
||||
Ok((pk, c)) => (Some(pk), Some(c), None),
|
||||
Err(e) => (None, None, Some(format!("{e:#}"))),
|
||||
};
|
||||
|
||||
// Whether BTCPay's inline copy still matches. `None` = BTCPay has no
|
||||
// internal Lightning node configured, which is a normal state and not a
|
||||
// problem to report.
|
||||
let btcpay_current = crate::container::lnd::btcpay_lnd_connection_is_current(
|
||||
std::path::Path::new(SECRETS_DIR),
|
||||
)
|
||||
.await;
|
||||
|
||||
Ok(serde_json::json!({
|
||||
"installed": installed.is_some(),
|
||||
"admin_macaroon_sha256": installed,
|
||||
"issued_at": issued_at,
|
||||
"identity_pubkey": identity_pubkey,
|
||||
"channels_open": census.map(|c| c.channels_open),
|
||||
"channels_pending": census.map(|c| c.channels_pending),
|
||||
"lnd_error": lnd_error,
|
||||
"btcpay_uses_internal_lnd": btcpay_current.is_some(),
|
||||
"btcpay_credential_current": btcpay_current,
|
||||
"rotation": snapshot(),
|
||||
}))
|
||||
}
|
||||
|
||||
/// Start a rotation. Password-gated and asynchronous.
|
||||
///
|
||||
/// Password-gated because invalidating every credential a wallet app holds
|
||||
/// is an operator action, and a session cookie only proves a browser was
|
||||
/// once logged in — the same reasoning as `node.rotate-identity` and TOTP
|
||||
/// setup, which both re-verify.
|
||||
///
|
||||
/// Asynchronous because the work takes minutes (LND's databases have to
|
||||
/// close and reopen); the HTTP request returns immediately and the UI polls
|
||||
/// `lnd.macaroon-rotation-progress`.
|
||||
pub(in crate::api::rpc) async fn handle_lnd_rotate_macaroons(
|
||||
self: &Arc<Self>,
|
||||
params: Option<serde_json::Value>,
|
||||
) -> Result<serde_json::Value> {
|
||||
let password = params
|
||||
.as_ref()
|
||||
.and_then(|p| p.get("password"))
|
||||
.and_then(|v| v.as_str())
|
||||
.unwrap_or("");
|
||||
if password.is_empty() {
|
||||
anyhow::bail!("Node password required to rotate Lightning credentials");
|
||||
}
|
||||
if !self.auth_manager.verify_password(password).await? {
|
||||
anyhow::bail!("Password verification failed");
|
||||
}
|
||||
|
||||
// Claim the slot and publish a fresh feed in one critical section, so a
|
||||
// second click cannot observe a half-reset progress object.
|
||||
{
|
||||
let mut guard = progress()
|
||||
.lock()
|
||||
.map_err(|_| anyhow::anyhow!("rotation state poisoned"))?;
|
||||
if guard.running {
|
||||
anyhow::bail!("A macaroon rotation is already running on this node");
|
||||
}
|
||||
*guard = RotationProgress {
|
||||
running: true,
|
||||
started_at: Some(chrono::Utc::now().to_rfc3339()),
|
||||
..Default::default()
|
||||
};
|
||||
}
|
||||
|
||||
let orchestrator = self.orchestrator.clone();
|
||||
tokio::spawn(async move {
|
||||
let outcome = run_rotation(orchestrator).await;
|
||||
with_progress(|p| {
|
||||
p.running = false;
|
||||
p.finished_at = Some(chrono::Utc::now().to_rfc3339());
|
||||
match &outcome {
|
||||
Ok(()) => p.ok = Some(true),
|
||||
Err(e) => {
|
||||
p.ok = Some(false);
|
||||
p.error = Some(format!("{e:#}"));
|
||||
}
|
||||
}
|
||||
});
|
||||
match outcome {
|
||||
Ok(()) => tracing::info!("LND macaroon rotation completed"),
|
||||
Err(e) => {
|
||||
tracing::error!(error = %format!("{e:#}"), "LND macaroon rotation failed")
|
||||
}
|
||||
}
|
||||
});
|
||||
|
||||
Ok(serde_json::json!({ "status": "started" }))
|
||||
}
|
||||
|
||||
/// Poll the running (or last) rotation.
|
||||
pub(in crate::api::rpc) async fn handle_lnd_macaroon_rotation_progress(
|
||||
&self,
|
||||
) -> Result<serde_json::Value> {
|
||||
Ok(serde_json::to_value(snapshot())?)
|
||||
}
|
||||
}
|
||||
|
||||
// ── The rotation itself ──────────────────────────────────────────────────────
|
||||
|
||||
async fn run_rotation(
|
||||
orchestrator: Option<Arc<dyn crate::container::ContainerOrchestrator>>,
|
||||
) -> Result<()> {
|
||||
// 1. Preflight — establish what must survive, while LND can still be asked.
|
||||
with_progress(|p| p.set("preflight", StepState::Running, None));
|
||||
let files = macaroon_files().await?;
|
||||
if files.is_empty() {
|
||||
with_progress(|p| p.set("preflight", StepState::Failed, None));
|
||||
anyhow::bail!(
|
||||
"no macaroon material found in {LND_MAINNET_DIR} — nothing to rotate, and \
|
||||
restarting Lightning for no reason would be a pointless outage"
|
||||
);
|
||||
}
|
||||
let (pubkey_before, census_before) = read_census().await.context(
|
||||
"refusing to rotate: LND is not answering, so there would be no baseline to prove your \
|
||||
channels survived. Start Lightning, wait for it to sync, and try again",
|
||||
)?;
|
||||
with_progress(|p| {
|
||||
p.identity_pubkey = Some(pubkey_before.clone());
|
||||
p.channels_before = Some(census_before.channels_open);
|
||||
p.set(
|
||||
"preflight",
|
||||
StepState::Done,
|
||||
Some(format!(
|
||||
"{} channel(s) open, {} pending — these must be identical afterwards",
|
||||
census_before.channels_open, census_before.channels_pending
|
||||
)),
|
||||
);
|
||||
});
|
||||
|
||||
// 2. Back up, so a mistake is recoverable. Verified by count: a backup that
|
||||
// silently copied nothing is the one failure that makes the deletion
|
||||
// below unrecoverable.
|
||||
with_progress(|p| p.set("backup", StepState::Running, None));
|
||||
let stamp = chrono::Utc::now().format("%Y%m%dT%H%M%SZ").to_string();
|
||||
let backup = format!("/var/lib/archipelago/lnd/macaroon-rotation-{stamp}");
|
||||
sudo_ok(&["mkdir", "-p", &backup]).await?;
|
||||
sudo_ok(&["chmod", "700", &backup]).await?;
|
||||
for f in &files {
|
||||
sudo_ok(&["cp", "-a", f, &backup])
|
||||
.await
|
||||
.with_context(|| format!("backing up {f} — aborting before any deletion"))?;
|
||||
}
|
||||
let backed_up = sudo(&["find", &backup, "-maxdepth", "1", "-type", "f"])
|
||||
.await?
|
||||
.stdout;
|
||||
let backed_up = String::from_utf8_lossy(&backed_up)
|
||||
.lines()
|
||||
.filter(|l| !l.trim().is_empty())
|
||||
.count();
|
||||
if backed_up != files.len() {
|
||||
with_progress(|p| p.set("backup", StepState::Failed, None));
|
||||
anyhow::bail!(
|
||||
"backup incomplete — {backed_up} of {} files in {backup}. Refusing to delete anything",
|
||||
files.len()
|
||||
);
|
||||
}
|
||||
with_progress(|p| {
|
||||
p.backup_path = Some(backup.clone());
|
||||
p.set(
|
||||
"backup",
|
||||
StepState::Done,
|
||||
Some(format!("{backed_up} file(s) copied to {backup}")),
|
||||
);
|
||||
});
|
||||
|
||||
// 3. Stop.
|
||||
with_progress(|p| p.set("stop", StepState::Running, None));
|
||||
stop_lnd().await.context("stopping LND")?;
|
||||
with_progress(|p| p.set("stop", StepState::Done, None));
|
||||
|
||||
// 4. Remove the credential material — and only now, with a verified backup.
|
||||
with_progress(|p| p.set("remove", StepState::Running, None));
|
||||
for f in &files {
|
||||
sudo_ok(&["rm", "-f", f])
|
||||
.await
|
||||
.with_context(|| format!("removing {f} — restore from {backup}"))?;
|
||||
}
|
||||
with_progress(|p| {
|
||||
p.set(
|
||||
"remove",
|
||||
StepState::Done,
|
||||
Some(format!("{} file(s) removed", files.len())),
|
||||
)
|
||||
});
|
||||
|
||||
// 5. Start, and unlock. The unlock is explicit rather than left to the next
|
||||
// reconcile tick: LND does not mint macaroons until the wallet opens, so
|
||||
// without this the rotation would sit waiting for a file that cannot
|
||||
// appear. `_no_wipe` keeps the destructive recovery path out of reach.
|
||||
with_progress(|p| p.set("start", StepState::Running, None));
|
||||
start_lnd().await.with_context(|| {
|
||||
format!("starting LND after removing its macaroons — old material is in {backup}")
|
||||
})?;
|
||||
crate::container::lnd::unlock_existing_wallet_no_wipe()
|
||||
.await
|
||||
.with_context(|| format!("unlocking the wallet — old material is in {backup}"))?;
|
||||
|
||||
let deadline = std::time::Instant::now() + std::time::Duration::from_secs(MACAROON_WAIT_SECS);
|
||||
let admin_path = format!("{LND_MAINNET_DIR}/admin.macaroon");
|
||||
let mut new_digest = None;
|
||||
while std::time::Instant::now() < deadline {
|
||||
if let Some(d) = digest_as_root(&admin_path).await {
|
||||
new_digest = Some(d);
|
||||
break;
|
||||
}
|
||||
tokio::time::sleep(std::time::Duration::from_secs(5)).await;
|
||||
}
|
||||
let Some(new_digest) = new_digest else {
|
||||
with_progress(|p| p.set("start", StepState::Failed, None));
|
||||
anyhow::bail!(
|
||||
"LND did not mint a new admin.macaroon within {} minutes. The old material is intact \
|
||||
in {backup} — restore it there and investigate before retrying",
|
||||
MACAROON_WAIT_SECS / 60
|
||||
);
|
||||
};
|
||||
with_progress(|p| {
|
||||
p.new_admin_macaroon_sha256 = Some(new_digest.clone());
|
||||
p.set(
|
||||
"start",
|
||||
StepState::Done,
|
||||
Some("Lightning is up with freshly minted credentials".into()),
|
||||
);
|
||||
});
|
||||
|
||||
// 6. Verify the things that must NOT have changed.
|
||||
with_progress(|p| p.set("verify", StepState::Running, None));
|
||||
let (pubkey_after, census_after) = wait_for_serving(deadline).await.with_context(|| {
|
||||
format!("verifying the node after rotation — old material is in {backup}")
|
||||
})?;
|
||||
with_progress(|p| p.channels_after = Some(census_after.channels_open));
|
||||
if pubkey_after != pubkey_before {
|
||||
with_progress(|p| p.set("verify", StepState::Failed, None));
|
||||
anyhow::bail!(
|
||||
"NODE IDENTITY CHANGED — this is not the same Lightning node. Old material is in \
|
||||
{backup}. Do not use this node until you understand why"
|
||||
);
|
||||
}
|
||||
if census_after.channels_open != census_before.channels_open
|
||||
|| census_after.channels_pending != census_before.channels_pending
|
||||
{
|
||||
with_progress(|p| p.set("verify", StepState::Failed, None));
|
||||
anyhow::bail!(
|
||||
"channel count changed ({} open/{} pending before, {} open/{} pending after). Old \
|
||||
material is in {backup}",
|
||||
census_before.channels_open,
|
||||
census_before.channels_pending,
|
||||
census_after.channels_open,
|
||||
census_after.channels_pending
|
||||
);
|
||||
}
|
||||
with_progress(|p| {
|
||||
p.set(
|
||||
"verify",
|
||||
StepState::Done,
|
||||
Some(format!(
|
||||
"same node, same {} channel(s)",
|
||||
census_after.channels_open
|
||||
)),
|
||||
)
|
||||
});
|
||||
|
||||
// 7. BTCPay. Its connection string embeds the macaroon inline and cannot
|
||||
// self-heal — see `rewrite_btcpay_lnd_connection_secret`. Left undone,
|
||||
// the node looks healthy while every Lightning invoice BTCPay creates
|
||||
// fails, which is precisely the failure this step exists to prevent.
|
||||
with_progress(|p| p.set("btcpay", StepState::Running, None));
|
||||
match crate::container::lnd::rewrite_btcpay_lnd_connection_secret(std::path::Path::new(
|
||||
SECRETS_DIR,
|
||||
))
|
||||
.await
|
||||
{
|
||||
Ok(true) => {
|
||||
// Writing the secret is only half of it. btcpay-server is
|
||||
// restart-sensitive, so reconcile sees the drift and deliberately
|
||||
// leaves the running container alone — which would strand it on the
|
||||
// dead macaroon indefinitely. This is the flag that overrides that,
|
||||
// and without it this whole step is cosmetic.
|
||||
match &orchestrator {
|
||||
Some(orch) => {
|
||||
orch.mark_credential_rotated("btcpay-server").await;
|
||||
with_progress(|p| {
|
||||
p.set(
|
||||
"btcpay",
|
||||
StepState::Done,
|
||||
Some(
|
||||
"Connection string updated. BTCPay restarts itself within a \
|
||||
minute or two to pick it up."
|
||||
.into(),
|
||||
),
|
||||
)
|
||||
});
|
||||
}
|
||||
// Only reachable in builds without an orchestrator (tests). Say
|
||||
// what is left for a human rather than implying it is handled.
|
||||
None => with_progress(|p| {
|
||||
p.set(
|
||||
"btcpay",
|
||||
StepState::Skipped,
|
||||
Some(
|
||||
"Connection string updated, but no orchestrator is available to \
|
||||
restart BTCPay — restart it yourself to pick up the new credentials."
|
||||
.into(),
|
||||
),
|
||||
)
|
||||
}),
|
||||
}
|
||||
}
|
||||
Ok(false) => with_progress(|p| {
|
||||
p.set(
|
||||
"btcpay",
|
||||
StepState::Skipped,
|
||||
Some("No internal Lightning node is configured for BTCPay on this node.".into()),
|
||||
)
|
||||
}),
|
||||
// Not fatal: the macaroons ARE rotated by this point, and reporting the
|
||||
// whole rotation as failed would be a lie that invites a needless retry.
|
||||
// Say exactly what is left undone instead.
|
||||
Err(e) => {
|
||||
tracing::warn!(error = %format!("{e:#}"), "btcpay connection string not updated after macaroon rotation");
|
||||
with_progress(|p| {
|
||||
p.set(
|
||||
"btcpay",
|
||||
StepState::Failed,
|
||||
Some(format!(
|
||||
"Your macaroons ARE rotated, but BTCPay's stored copy could not be \
|
||||
updated, so its Lightning payments will fail until it is: {e:#}"
|
||||
)),
|
||||
)
|
||||
})
|
||||
}
|
||||
}
|
||||
|
||||
Ok(())
|
||||
}
|
||||
|
||||
#[cfg(test)]
|
||||
mod tests {
|
||||
use super::*;
|
||||
|
||||
#[test]
|
||||
fn progress_starts_with_every_step_pending() {
|
||||
let p = RotationProgress::default();
|
||||
assert_eq!(p.steps.len(), STEPS.len());
|
||||
assert!(p.steps.iter().all(|s| s.state == StepState::Pending));
|
||||
assert!(!p.running);
|
||||
assert!(p.ok.is_none());
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn set_updates_only_the_named_step() {
|
||||
let mut p = RotationProgress::default();
|
||||
p.set("stop", StepState::Done, Some("stopped".into()));
|
||||
let stop = p.steps.iter().find(|s| s.key == "stop").unwrap();
|
||||
assert_eq!(stop.state, StepState::Done);
|
||||
assert_eq!(stop.detail.as_deref(), Some("stopped"));
|
||||
assert!(p
|
||||
.steps
|
||||
.iter()
|
||||
.filter(|s| s.key != "stop")
|
||||
.all(|s| s.state == StepState::Pending));
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn set_on_an_unknown_step_is_a_no_op_not_a_panic() {
|
||||
let mut p = RotationProgress::default();
|
||||
p.set("not-a-step", StepState::Failed, None);
|
||||
assert!(p.steps.iter().all(|s| s.state == StepState::Pending));
|
||||
}
|
||||
|
||||
/// A detail is informational; passing `None` must not wipe one already set,
|
||||
/// or a later state transition would erase the explanation the operator is
|
||||
/// reading.
|
||||
#[test]
|
||||
fn set_without_a_detail_keeps_the_existing_one() {
|
||||
let mut p = RotationProgress::default();
|
||||
p.set("btcpay", StepState::Running, Some("working".into()));
|
||||
p.set("btcpay", StepState::Done, None);
|
||||
let step = p.steps.iter().find(|s| s.key == "btcpay").unwrap();
|
||||
assert_eq!(step.state, StepState::Done);
|
||||
assert_eq!(step.detail.as_deref(), Some("working"));
|
||||
}
|
||||
|
||||
/// The serialized shape is a UI contract: the frontend renders `state`
|
||||
/// as a lowercase discriminant.
|
||||
#[test]
|
||||
fn step_states_serialize_lowercase() {
|
||||
let json = serde_json::to_string(&StepState::Skipped).unwrap();
|
||||
assert_eq!(json, "\"skipped\"");
|
||||
}
|
||||
|
||||
/// Macaroon *content* must never reach the progress feed the UI polls.
|
||||
#[test]
|
||||
fn progress_carries_digests_not_tokens() {
|
||||
let mut p = RotationProgress::default();
|
||||
p.new_admin_macaroon_sha256 = Some("a".repeat(64));
|
||||
let json = serde_json::to_string(&p).unwrap();
|
||||
assert!(json.contains("new_admin_macaroon_sha256"));
|
||||
assert!(!json.contains("macaroon_hex"));
|
||||
}
|
||||
}
|
||||
@@ -1,5 +1,6 @@
|
||||
mod channels;
|
||||
mod info;
|
||||
mod macaroons;
|
||||
mod payments;
|
||||
mod seed_backup;
|
||||
mod wallet;
|
||||
|
||||
@@ -22,6 +22,11 @@ const WALLET_PASSWORD_SECRET: &str = "/var/lib/archipelago/secrets/lnd-wallet-pa
|
||||
/// never use it, and the login-path migration rotates away from it.
|
||||
const LEGACY_WALLET_PASSWORD: &str = "hellohello";
|
||||
|
||||
/// How many one-second passes `unlock_existing_wallet_via_rest` will make while
|
||||
/// LND's unlocker is still not listening (~10 minutes). See the comment at the
|
||||
/// retry loop for why this is measured in minutes rather than seconds.
|
||||
const UNLOCK_NOT_READY_ATTEMPTS: u32 = 600;
|
||||
|
||||
#[derive(Debug, Clone)]
|
||||
pub struct EnsurePaths {
|
||||
pub data_dir: PathBuf,
|
||||
@@ -345,7 +350,18 @@ async fn unlock_existing_wallet_via_rest() -> Result<bool> {
|
||||
// *actively rejected* (invalid passphrase), retrying can't help — fail fast
|
||||
// with a clear message instead of hanging the boot path for 60s+ (the wallet
|
||||
// was created with a password this node doesn't have → migration/recovery).
|
||||
for _ in 0..60 {
|
||||
//
|
||||
// The not-ready budget is deliberately generous. LND opens channel.db,
|
||||
// graph.db and wallet.db before it starts serving the unlocker at all, and
|
||||
// on a busy node that is genuinely slow — observed at 2m38s on a box running
|
||||
// 30 containers, where a 60s budget could never succeed. Timing out here is
|
||||
// not a harmless retry: reconcile records the post-start hook as failed,
|
||||
// which restarts LND, which starts the slow database open over again. The
|
||||
// result is a restart loop that leaves the wallet permanently locked and
|
||||
// every LND-dependent app (BTCPay's internal node included) broken, on
|
||||
// exactly the nodes least able to afford it. Waiting longer costs nothing —
|
||||
// a wrong password still exits on the first pass via `all_rejected`.
|
||||
for _ in 0..UNLOCK_NOT_READY_ATTEMPTS {
|
||||
let mut all_rejected = true;
|
||||
for pw in &candidates {
|
||||
match try_unlock_once(&client, pw).await {
|
||||
@@ -364,7 +380,28 @@ async fn unlock_existing_wallet_via_rest() -> Result<bool> {
|
||||
}
|
||||
tokio::time::sleep(std::time::Duration::from_secs(1)).await;
|
||||
}
|
||||
anyhow::bail!("LND wallet unlock timed out waiting for the unlocker to become ready")
|
||||
anyhow::bail!(
|
||||
"LND wallet unlock timed out after ~{}s waiting for the unlocker to become ready",
|
||||
UNLOCK_NOT_READY_ATTEMPTS
|
||||
)
|
||||
}
|
||||
|
||||
/// Unlock an existing wallet WITHOUT the destructive fallback.
|
||||
///
|
||||
/// `ensure_wallet_initialized` wipes and recreates a wallet no candidate
|
||||
/// password can open — correct for a boot path that must self-heal, and exactly
|
||||
/// wrong for macaroon rotation, which restarts LND against a wallet the operator
|
||||
/// still wants. Rotation calls this instead, so there is no code path from
|
||||
/// "rotate my credentials" to "delete my wallet": a rejected password surfaces
|
||||
/// as an error the caller reports, never as a wipe.
|
||||
pub(crate) async fn unlock_existing_wallet_no_wipe() -> Result<()> {
|
||||
match unlock_existing_wallet().await? {
|
||||
true => Ok(()),
|
||||
false => anyhow::bail!(
|
||||
"LND rejected every candidate wallet password — refusing to touch the wallet. \
|
||||
The wallet is intact and still locked; its password is not one this node holds."
|
||||
),
|
||||
}
|
||||
}
|
||||
|
||||
/// Current LND wallet state via the unauthenticated `/v1/state` endpoint
|
||||
@@ -718,29 +755,73 @@ const BTCPAY_LND_CONNECTION_SECRET: &str = "btcpay-lnd-connection";
|
||||
/// btcpay's secret_env entry is `optional`, so it simply starts without an
|
||||
/// internal Lightning node and picks it up on a later reconcile tick.
|
||||
/// Rewrites when the pinned cert thumbprint no longer matches (LND TLS cert
|
||||
/// rotation). Macaroon rotation without cert rotation is not auto-detected
|
||||
/// rotation). Macaroon rotation without cert rotation is not auto-detected here
|
||||
/// (reading the macaroon needs sudo; probing it every tick is not worth the
|
||||
/// churn) — delete the secret file once to force regeneration.
|
||||
/// churn) — the rotation path calls `rewrite_btcpay_lnd_connection_secret`
|
||||
/// instead, and deleting the secret file also forces regeneration.
|
||||
pub async fn ensure_btcpay_lnd_connection_secret(secrets_dir: &std::path::Path) -> Result<()> {
|
||||
build_btcpay_lnd_connection_secret(secrets_dir, false)
|
||||
.await
|
||||
.map(|_| ())
|
||||
}
|
||||
|
||||
/// Rewrite the BTCPay→LND connection secret unconditionally, ignoring the
|
||||
/// cert-thumbprint fast path.
|
||||
///
|
||||
/// Rotating LND's macaroons invalidates the one embedded in this secret, and it
|
||||
/// is embedded *inline* rather than referenced by path — LND's datadir is owned
|
||||
/// by its container subuid, so btcpay cannot bind-mount the file and the string
|
||||
/// cannot self-heal. Nothing else notices: the TLS cert is untouched by macaroon
|
||||
/// rotation, so `ensure_…` takes its fast path forever and BTCPay keeps
|
||||
/// presenting a credential LND no longer honours. A node in that state looks
|
||||
/// entirely healthy — btcpay is up, LND is up — while every Lightning invoice it
|
||||
/// tries to create fails.
|
||||
///
|
||||
/// Writing the new value makes the change *visible*: `secret_env_hash` is
|
||||
/// derived from the resolved secret contents, so a changed file reads as label
|
||||
/// drift on the running container. It is not sufficient on its own — btcpay is
|
||||
/// restart-sensitive, and boot reconcile deliberately leaves running
|
||||
/// restart-sensitive apps untouched on drift. The caller must also call
|
||||
/// `ContainerOrchestrator::mark_credential_rotated("btcpay-server")`, which is
|
||||
/// the carve-out for exactly this case: a container that is up and healthy while
|
||||
/// holding a credential that no longer works. The orchestrator's own recreate
|
||||
/// path then rebuilds it around an unchanged data directory. No teardown here,
|
||||
/// deliberately — a hand-rolled remove-and-run is the anti-pattern CLAUDE.md
|
||||
/// names.
|
||||
///
|
||||
/// Returns `false` when LND isn't provisioned enough to derive a value.
|
||||
pub async fn rewrite_btcpay_lnd_connection_secret(secrets_dir: &std::path::Path) -> Result<bool> {
|
||||
build_btcpay_lnd_connection_secret(secrets_dir, true).await
|
||||
}
|
||||
|
||||
/// Shared body. `force` skips the "already pins the current cert" fast path.
|
||||
/// Returns whether a value was written.
|
||||
async fn build_btcpay_lnd_connection_secret(
|
||||
secrets_dir: &std::path::Path,
|
||||
force: bool,
|
||||
) -> Result<bool> {
|
||||
let cert_path = format!("{DEFAULT_DATA_DIR}/tls.cert");
|
||||
let pem = match fs::read_to_string(&cert_path).await {
|
||||
Ok(s) => s,
|
||||
Err(_) => return Ok(()), // LND not installed/provisioned yet
|
||||
Err(_) => return Ok(false), // LND not installed/provisioned yet
|
||||
};
|
||||
let thumbprint = cert_sha256_thumbprint(&pem).context("computing LND tls.cert thumbprint")?;
|
||||
|
||||
let target = secrets_dir.join(BTCPAY_LND_CONNECTION_SECRET);
|
||||
// Fast path (no sudo): existing secret already pins the current cert.
|
||||
if let Ok(existing) = fs::read_to_string(&target).await {
|
||||
if !existing.trim().is_empty() && existing.contains(&format!("certthumbprint={thumbprint}"))
|
||||
{
|
||||
return Ok(());
|
||||
if !force {
|
||||
if let Ok(existing) = fs::read_to_string(&target).await {
|
||||
if !existing.trim().is_empty()
|
||||
&& existing.contains(&format!("certthumbprint={thumbprint}"))
|
||||
{
|
||||
return Ok(false);
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
let macaroon_path = format!("{DEFAULT_DATA_DIR}/data/chain/bitcoin/mainnet/admin.macaroon");
|
||||
if !file_exists_as_root(&macaroon_path).await {
|
||||
return Ok(()); // wallet not created yet; next tick retries
|
||||
return Ok(false); // wallet not created yet; next tick retries
|
||||
}
|
||||
let macaroon = read_file_as_root(&macaroon_path).await?;
|
||||
let value = format!(
|
||||
@@ -749,7 +830,30 @@ pub async fn ensure_btcpay_lnd_connection_secret(secrets_dir: &std::path::Path)
|
||||
thumbprint
|
||||
);
|
||||
crate::container::secrets::write_secret_file(&target, &value)
|
||||
.context("writing btcpay-lnd-connection secret")
|
||||
.context("writing btcpay-lnd-connection secret")?;
|
||||
Ok(true)
|
||||
}
|
||||
|
||||
/// Does the on-disk BTCPay connection secret still carry the macaroon LND is
|
||||
/// currently issuing? `None` when there is nothing to compare — no secret file
|
||||
/// (BTCPay has no internal node configured) or no macaroon (LND unprovisioned).
|
||||
///
|
||||
/// Compares only hex text that is already on this host; the value is never
|
||||
/// logged, returned over RPC, or placed in an error.
|
||||
pub(crate) async fn btcpay_lnd_connection_is_current(
|
||||
secrets_dir: &std::path::Path,
|
||||
) -> Option<bool> {
|
||||
let target = secrets_dir.join(BTCPAY_LND_CONNECTION_SECRET);
|
||||
let existing = fs::read_to_string(&target).await.ok()?;
|
||||
let embedded = existing
|
||||
.split("macaroon=")
|
||||
.nth(1)?
|
||||
.split(';')
|
||||
.next()?
|
||||
.to_string();
|
||||
let macaroon_path = format!("{DEFAULT_DATA_DIR}/data/chain/bitcoin/mainnet/admin.macaroon");
|
||||
let current = read_file_as_root(&macaroon_path).await.ok()?;
|
||||
Some(embedded.eq_ignore_ascii_case(&hex::encode(current)))
|
||||
}
|
||||
|
||||
/// SHA256 over the DER certificate body (matches
|
||||
|
||||
@@ -4449,6 +4449,18 @@ impl ContainerOrchestrator for ProdContainerOrchestrator {
|
||||
ContainerState::Unknown(s) => format!("unknown:{s}"),
|
||||
})
|
||||
}
|
||||
|
||||
async fn mark_credential_rotated(&self, app_id: &str) {
|
||||
self.credential_rotated
|
||||
.lock()
|
||||
.await
|
||||
.insert(app_id.to_string());
|
||||
tracing::info!(
|
||||
app_id = %app_id,
|
||||
"a credential this app consumes was rotated — its running container will be recreated \
|
||||
on the next drift check even though the app is restart-sensitive"
|
||||
);
|
||||
}
|
||||
}
|
||||
|
||||
/// Compare a manifest's command against a live container's, over the
|
||||
|
||||
@@ -73,4 +73,21 @@ pub trait ContainerOrchestrator: Send + Sync {
|
||||
|
||||
/// Coarse health summary: "healthy", "unhealthy", "starting", "paused", "unknown".
|
||||
async fn health(&self, app_id: &str) -> Result<String>;
|
||||
|
||||
/// Declare that a credential this app consumes has just been rotated, so
|
||||
/// the running container is now holding an invalid one.
|
||||
///
|
||||
/// Restart-sensitivity normally protects apps like `btcpay-server` from
|
||||
/// being recreated on drift — correct when the running container is
|
||||
/// working, and exactly wrong when it is working only in appearance. After
|
||||
/// an LND macaroon rotation, BTCPay is up and healthy while every Lightning
|
||||
/// operation it attempts fails against a credential LND no longer honours;
|
||||
/// leaving it untouched perpetuates the breakage rather than protecting
|
||||
/// anything. This is the same carve-out FED-07 uses for the Fedimint
|
||||
/// gateway, reached from the RPC layer instead of from inside a reconcile.
|
||||
///
|
||||
/// Consumed by the next drift check, which recreates the container around
|
||||
/// its unchanged data directory, ports and volumes. Default no-op: an
|
||||
/// orchestrator without restart-sensitivity has nothing to override.
|
||||
async fn mark_credential_rotated(&self, _app_id: &str) {}
|
||||
}
|
||||
|
||||
@@ -134,12 +134,21 @@ The operator's call, recorded here so it is not silently re-litigated: **no LND
|
||||
rotation, and no Bitcoin RPC password rotation.** The reasoning was that there is no
|
||||
evidence of exploitation and the vulnerability is being closed rather than lived with.
|
||||
|
||||
`scripts/security/rotate-lnd-macaroon.sh` stays in the tree as a tool. It has been
|
||||
exercised in detect mode only, and has never rotated anything on any node. Its ordering
|
||||
`scripts/security/rotate-lnd-macaroon.sh` stays in the tree as a tool. Its ordering
|
||||
guard (refuses to rotate on a binary lacking the fix) remains the right shape for whenever
|
||||
rotation is wanted — including for the Bitcoin RPC password, which has no equivalent tool
|
||||
yet.
|
||||
|
||||
**Amended 2026-08-08.** This section said the script "has never rotated anything on any
|
||||
node"; that is no longer true. A rotation was performed on a development node while
|
||||
responding to the BTCPay Server advisory (that node had been running an affected
|
||||
`btcpayserver:2.3.9`), and it exposed a gap the script did not cover: BTCPay's inline copy
|
||||
of the macaroon was left stranded, so its Lightning payments failed silently while both
|
||||
apps reported healthy. Rotation is now a first-class, password-confirmed dashboard action
|
||||
that repairs that copy as part of the run — see
|
||||
[`LND-MACAROON-ROTATION.md`](LND-MACAROON-ROTATION.md). The fleet decision recorded above
|
||||
is unchanged: no fleet-wide rotation for this leak.
|
||||
|
||||
What this decision accepts: any macaroon or RPC password read through either hole before
|
||||
it was closed stays valid. That is a deliberate, informed trade, not an oversight.
|
||||
|
||||
|
||||
@@ -0,0 +1,166 @@
|
||||
# Rotating this node's Lightning credentials
|
||||
|
||||
A Lightning macaroon is a **bearer token**: whoever holds one can spend from the
|
||||
node's wallet. There is no revocation list and no expiry. If a macaroon is ever
|
||||
read by something you do not control — a leaked endpoint, a screenshot, a phone
|
||||
that has since been lost, an app that ran a version with a published
|
||||
vulnerability — that ability persists until the macaroons are rotated.
|
||||
|
||||
Rotation is therefore a **routine operator action**, not an emergency procedure.
|
||||
Two paths do the same work:
|
||||
|
||||
| Path | Use when |
|
||||
|---|---|
|
||||
| **Dashboard** — Settings → *Lightning credentials* | Normal case. Password-confirmed, shows progress, repairs BTCPay for you. |
|
||||
| **`scripts/security/rotate-lnd-macaroon.sh`** | No dashboard reachable, or you want a detect-only report. |
|
||||
|
||||
## What rotation actually does
|
||||
|
||||
LND derives every macaroon it issues from a root key in `macaroons.db`. Remove
|
||||
that root key plus the issued `*.macaroon` files, restart, and LND mints a fresh
|
||||
root key and a fresh set of macaroons when the wallet unlocks. Every macaroon
|
||||
issued before that moment — including any an attacker holds — stops verifying.
|
||||
|
||||
## Why your funds and channels survive
|
||||
|
||||
Macaroons are bearer tokens, not keys. Coins live in `wallet.db` and channel
|
||||
state in `channel.db`; channels are secured by the node's identity and channel
|
||||
keys, none of which are derived from the macaroon root key. Neither database is
|
||||
opened, moved or deleted.
|
||||
|
||||
Both paths **prove** this rather than asserting it: they record the node's
|
||||
identity pubkey and its channel census before rotating, and refuse to report
|
||||
success if either differs afterwards.
|
||||
|
||||
Two details in that check are deliberate and should not be "tightened":
|
||||
|
||||
- **Channels are compared as a total, not as `num_active_channels`.** The active
|
||||
count only counts channels whose peer is currently online, so it legitimately
|
||||
dips for minutes after *any* restart while peers reconnect. Asserting on it
|
||||
alone would abort a perfectly healthy rotation.
|
||||
- **`wallet.db` is not compared byte-for-byte.** btcwallet records chain-sync
|
||||
progress inside it, so the file changes on every start. Asserting byte-identity
|
||||
would fire a frightening false alarm on a completely healthy rotation.
|
||||
|
||||
## What it never does
|
||||
|
||||
- No macaroon **content** reaches a response, an error, a log line, or the
|
||||
progress feed the dashboard polls. Everything reported is a SHA-256 digest or a
|
||||
byte count — enough to prove the material changed without disclosing it to
|
||||
whoever is reading the screen.
|
||||
- No path from "rotate my credentials" to "delete my wallet". LND's boot path
|
||||
self-heals a wallet no candidate password can open by wiping and recreating it;
|
||||
correct for an unattended boot, catastrophic here. Rotation unlocks through
|
||||
`container::lnd::unlock_existing_wallet_no_wipe`, so a wallet whose password
|
||||
this node does not hold surfaces as a **failed rotation** with the wallet
|
||||
intact.
|
||||
|
||||
## The BTCPay coupling — the part that bites
|
||||
|
||||
**BTCPay Server keeps its own inline copy of the admin macaroon**, and it cannot
|
||||
self-heal. LND's data directory is owned by its container's mapped uid, so BTCPay
|
||||
cannot bind-mount the macaroon file (EACCES across the userns boundary). The
|
||||
connection string therefore carries the macaroon as hex:
|
||||
|
||||
```
|
||||
type=lnd-rest;server=https://lnd:8080/;macaroon=<hex>;certthumbprint=<hex>
|
||||
```
|
||||
|
||||
delivered as the `btcpay-lnd-connection` secret file. Rotate the macaroons and
|
||||
that copy becomes a dead credential. Nothing notices on its own, because the
|
||||
daemon only regenerates this secret when LND's **TLS cert thumbprint** changes —
|
||||
and macaroon rotation does not touch the cert.
|
||||
|
||||
The resulting state is the dangerous one: **BTCPay is up, LND is up, both report
|
||||
healthy, and every Lightning invoice BTCPay tries to create fails.**
|
||||
|
||||
Repair needs two things, and one without the other is cosmetic:
|
||||
|
||||
1. **Rewrite the secret** (`container::lnd::rewrite_btcpay_lnd_connection_secret`).
|
||||
This is what makes the change visible: `secret_env_hash` is derived from the
|
||||
resolved secret contents, so a changed file reads as label drift on the
|
||||
running container.
|
||||
2. **Recreate the container.** `btcpay-server` is on the restart-sensitive list,
|
||||
and the reconcile loop runs in `ExistingOnly` mode *always* — boot and
|
||||
periodic alike — where env drift on a restart-sensitive app is detected and
|
||||
then deliberately skipped. Rewriting the secret alone therefore changes
|
||||
nothing that is running. Observed directly on a development node, once per
|
||||
tick, for half an hour:
|
||||
|
||||
```
|
||||
container drift detected during boot reconcile; leaving running
|
||||
restart-sensitive app untouched app_id=btcpay-server
|
||||
```
|
||||
|
||||
The dashboard path calls
|
||||
`ContainerOrchestrator::mark_credential_rotated("btcpay-server")`, which is
|
||||
the flag the drift check consults to override restart-sensitivity. It is the
|
||||
same carve-out FED-07 added for the Fedimint gateway, and the reasoning is
|
||||
identical: restart sensitivity protects apps that are *working*, and this one
|
||||
is working only in appearance.
|
||||
|
||||
**The shell script cannot set that in-process flag**, so it does the equivalent
|
||||
from outside: it deletes the secret (the daemon regenerates it within a tick),
|
||||
then removes the `btcpay-server` container so the orchestrator's own
|
||||
desired-state recovery rebuilds it around unchanged data. That recovery is what
|
||||
makes this safe rather than a hand-rolled remove-and-run — it fires because the
|
||||
app is still installed and was in the last running-containers snapshot. The
|
||||
script then prints the commands to confirm it actually happened, because a
|
||||
failure here is invisible.
|
||||
|
||||
## Slow nodes: the unlock budget
|
||||
|
||||
LND opens `channel.db`, `graph.db` and `wallet.db` before it serves the unlocker
|
||||
at all, and on a busy node that is genuinely slow — **2m38s measured on a box
|
||||
running 30 containers**. The unlock helper used to give up after ~60s, which on
|
||||
such a node could never succeed.
|
||||
|
||||
That timeout was not a harmless retry. Reconcile records the post-start hook as
|
||||
failed, restarts LND, and the slow database open starts over: a restart loop that
|
||||
leaves the wallet permanently locked and every LND-dependent app (BTCPay's
|
||||
internal node included) broken, on exactly the nodes least able to afford it.
|
||||
|
||||
The not-ready budget is now ~10 minutes (`UNLOCK_NOT_READY_ATTEMPTS`). Waiting
|
||||
longer costs nothing, because a genuinely wrong password still exits on the first
|
||||
pass through the candidate list — the `all_rejected` fast path is untouched.
|
||||
|
||||
## Verifying a rotation
|
||||
|
||||
The dashboard shows all of this. From a shell:
|
||||
|
||||
```bash
|
||||
# 1. Fingerprint changed (digest only — never print the macaroon)
|
||||
sudo sha256sum /var/lib/archipelago/lnd/data/chain/bitcoin/mainnet/admin.macaroon
|
||||
|
||||
# 2. Same node, same channels
|
||||
podman exec lnd lncli --network=mainnet getinfo \
|
||||
| python3 -c 'import json,sys; d=json.load(sys.stdin); print(d["identity_pubkey"], \
|
||||
d["num_active_channels"] + d["num_inactive_channels"], d["num_pending_channels"])'
|
||||
|
||||
# 3. BTCPay is carrying the CURRENT macaroon, not the rotated-out one
|
||||
CUR=$(sudo od -An -v -tx1 /var/lib/archipelago/lnd/data/chain/bitcoin/mainnet/admin.macaroon | tr -d ' \n')
|
||||
SEC=$(sudo sed -n 's/.*macaroon=\([0-9a-f]*\).*/\1/p' /var/lib/archipelago/secrets/btcpay-lnd-connection)
|
||||
[ "$CUR" = "$SEC" ] && echo "current" || echo "STALE — BTCPay's Lightning is broken"
|
||||
|
||||
# 4. BTCPay was actually recreated (a silent failure looks like success)
|
||||
podman inspect btcpay-server --format '{{.Created}}'
|
||||
```
|
||||
|
||||
Check 3 is the one people skip, and it is the one that fails.
|
||||
|
||||
## Afterwards
|
||||
|
||||
- **Re-pair every wallet app**, Zeus most importantly. Open the Lightning app in
|
||||
the dashboard and scan the pairing QR again; it serves the new macaroon.
|
||||
- **Delete the backup once re-pairing is done.** Both paths back the old material
|
||||
up to `/var/lib/archipelago/lnd/macaroon-rotation-<stamp>` (0700) so a mistake
|
||||
is recoverable. That directory holds the **old root key** and is still
|
||||
sensitive: `sudo rm -rf <path>`.
|
||||
|
||||
## Related
|
||||
|
||||
- `docs/security/BITCOIN-RPC-PROXY-EXPOSURE.md` — the leak that first made
|
||||
rotation necessary, and the operator decision not to rotate the fleet for it.
|
||||
- `scripts/security/rotate-lnd-macaroon.sh` — the shell path, including its
|
||||
ordering guard (it refuses to rotate on a binary that still leaks
|
||||
`/lnd-connect-info`, since the new macaroon would leak within seconds).
|
||||
@@ -1158,6 +1158,69 @@ class RPCClient {
|
||||
})
|
||||
}
|
||||
|
||||
/** This node's Lightning credential state. Digests and counts only — the
|
||||
* backend never returns macaroon content, so nothing here is sensitive. */
|
||||
async lndMacaroonStatus(): Promise<LndMacaroonStatus> {
|
||||
return this.call({ method: 'lnd.macaroon-status', timeout: 30000 })
|
||||
}
|
||||
|
||||
/** Begin a macaroon rotation. Returns as soon as the job is accepted; the
|
||||
* work takes minutes (LND has to close and reopen its databases), so poll
|
||||
* `lndMacaroonRotationProgress` for the outcome. */
|
||||
async lndRotateMacaroons(password: string): Promise<{ status: string }> {
|
||||
return this.call({
|
||||
method: 'lnd.rotate-macaroons',
|
||||
params: { password },
|
||||
timeout: 30000,
|
||||
})
|
||||
}
|
||||
|
||||
async lndMacaroonRotationProgress(): Promise<LndRotationProgress> {
|
||||
return this.call({ method: 'lnd.macaroon-rotation-progress' })
|
||||
}
|
||||
}
|
||||
|
||||
export type RotationStepState = 'pending' | 'running' | 'done' | 'failed' | 'skipped'
|
||||
|
||||
export interface LndRotationStep {
|
||||
key: string
|
||||
label: string
|
||||
state: RotationStepState
|
||||
detail: string | null
|
||||
}
|
||||
|
||||
export interface LndRotationProgress {
|
||||
running: boolean
|
||||
/** null while running, then the verdict. Lets the UI tell "in progress"
|
||||
* apart from "finished and failed". */
|
||||
ok: boolean | null
|
||||
started_at: string | null
|
||||
finished_at: string | null
|
||||
error: string | null
|
||||
steps: LndRotationStep[]
|
||||
/** Holds the OLD root key, so it is still secret. The UI tells the operator
|
||||
* to delete it once every wallet app has been re-paired. */
|
||||
backup_path: string | null
|
||||
identity_pubkey: string | null
|
||||
channels_before: number | null
|
||||
channels_after: number | null
|
||||
new_admin_macaroon_sha256: string | null
|
||||
}
|
||||
|
||||
export interface LndMacaroonStatus {
|
||||
installed: boolean
|
||||
admin_macaroon_sha256: string | null
|
||||
/** When LND last minted these credentials, host local time. */
|
||||
issued_at: string | null
|
||||
identity_pubkey: string | null
|
||||
channels_open: number | null
|
||||
channels_pending: number | null
|
||||
/** Why LND could not be asked, when it could not. */
|
||||
lnd_error: string | null
|
||||
btcpay_uses_internal_lnd: boolean
|
||||
/** null when BTCPay has no internal Lightning node — an absence, not a fault. */
|
||||
btcpay_credential_current: boolean | null
|
||||
rotation: LndRotationProgress
|
||||
}
|
||||
|
||||
export const rpcClient = new RPCClient()
|
||||
|
||||
@@ -0,0 +1,350 @@
|
||||
<script setup lang="ts">
|
||||
import { computed, onMounted, onUnmounted, ref } from 'vue'
|
||||
import { rpcClient, type LndMacaroonStatus, type LndRotationProgress } from '@/api/rpc-client'
|
||||
|
||||
// A Lightning macaroon is a bearer token: whoever holds one can spend from this
|
||||
// node's wallet. Rotating them is the only way to take that ability back from
|
||||
// anything that has seen one — a lost phone, a shared screenshot, an app that
|
||||
// ran a version with a published vulnerability. Until now that meant SSHing in
|
||||
// and running a script, which in practice meant it did not happen.
|
||||
//
|
||||
// The screen is deliberately fact-first. Before anyone clicks the button they
|
||||
// can see when the credentials were issued, which node they belong to, how many
|
||||
// channels must survive, and whether anything on this node is already out of
|
||||
// step — because "will this close my channels?" is the question that stops
|
||||
// people rotating, and the honest answer is on the page.
|
||||
|
||||
const status = ref<LndMacaroonStatus | null>(null)
|
||||
const loading = ref(true)
|
||||
const loadError = ref('')
|
||||
|
||||
const showConfirm = ref(false)
|
||||
const password = ref('')
|
||||
const submitting = ref(false)
|
||||
const confirmError = ref('')
|
||||
|
||||
let poll: ReturnType<typeof setInterval> | null = null
|
||||
|
||||
const rotation = computed<LndRotationProgress | null>(() => status.value?.rotation ?? null)
|
||||
const isRunning = computed(() => rotation.value?.running === true)
|
||||
|
||||
/** A finished rotation, successful or not. `ok` is null while running. */
|
||||
const finished = computed(
|
||||
() => rotation.value !== null && !rotation.value.running && rotation.value.ok !== null,
|
||||
)
|
||||
|
||||
/** BTCPay embeds a copy of the macaroon inline and cannot self-heal, so it is
|
||||
* the one dependency that can silently fall out of step. `false` is the state
|
||||
* worth shouting about; `null` just means BTCPay has no internal node. */
|
||||
const btcpayStale = computed(() => status.value?.btcpay_credential_current === false)
|
||||
|
||||
async function load() {
|
||||
try {
|
||||
status.value = await rpcClient.lndMacaroonStatus()
|
||||
loadError.value = ''
|
||||
// Poll only while there is something to watch, so an idle Settings tab
|
||||
// isn't waking the node every few seconds.
|
||||
if (status.value.rotation.running) startPolling()
|
||||
else stopPolling()
|
||||
} catch (e) {
|
||||
loadError.value = e instanceof Error ? e.message : String(e)
|
||||
} finally {
|
||||
loading.value = false
|
||||
}
|
||||
}
|
||||
|
||||
function startPolling() {
|
||||
if (poll) return
|
||||
poll = setInterval(load, 4000)
|
||||
}
|
||||
|
||||
function stopPolling() {
|
||||
if (poll) {
|
||||
clearInterval(poll)
|
||||
poll = null
|
||||
}
|
||||
}
|
||||
|
||||
function openConfirm() {
|
||||
password.value = ''
|
||||
confirmError.value = ''
|
||||
showConfirm.value = true
|
||||
}
|
||||
|
||||
function closeConfirm() {
|
||||
showConfirm.value = false
|
||||
password.value = ''
|
||||
confirmError.value = ''
|
||||
}
|
||||
|
||||
async function rotate() {
|
||||
submitting.value = true
|
||||
confirmError.value = ''
|
||||
try {
|
||||
await rpcClient.lndRotateMacaroons(password.value)
|
||||
closeConfirm()
|
||||
startPolling()
|
||||
await load()
|
||||
} catch (e) {
|
||||
confirmError.value = e instanceof Error ? e.message : String(e)
|
||||
} finally {
|
||||
submitting.value = false
|
||||
password.value = ''
|
||||
}
|
||||
}
|
||||
|
||||
function stepIcon(state: string): string {
|
||||
switch (state) {
|
||||
case 'done':
|
||||
return '✓'
|
||||
case 'failed':
|
||||
return '✕'
|
||||
case 'skipped':
|
||||
return '–'
|
||||
case 'running':
|
||||
return '…'
|
||||
default:
|
||||
return '·'
|
||||
}
|
||||
}
|
||||
|
||||
function stepClass(state: string): string {
|
||||
switch (state) {
|
||||
case 'done':
|
||||
return 'text-emerald-400'
|
||||
case 'failed':
|
||||
return 'text-red-400'
|
||||
case 'skipped':
|
||||
return 'text-white/40'
|
||||
case 'running':
|
||||
return 'text-orange-300'
|
||||
default:
|
||||
return 'text-white/30'
|
||||
}
|
||||
}
|
||||
|
||||
/** First 16 characters is plenty to compare two digests by eye, and keeps the
|
||||
* line readable on a phone. */
|
||||
function shortHash(h: string | null): string {
|
||||
return h ? `${h.slice(0, 16)}…` : '—'
|
||||
}
|
||||
|
||||
onMounted(load)
|
||||
onUnmounted(stopPolling)
|
||||
</script>
|
||||
|
||||
<template>
|
||||
<div class="mb-6">
|
||||
<h3 class="text-base font-medium text-white/90 mb-1">Lightning credentials</h3>
|
||||
<p class="text-sm text-white/60 mb-4">
|
||||
Wallet apps like Zeus connect to this node using a Lightning credential — a
|
||||
token that lets them spend. Rotating replaces every one of them, so anything
|
||||
that copied an old token can no longer use it. Your coins and channels are
|
||||
not touched: the node keeps its identity and no channel is closed.
|
||||
</p>
|
||||
|
||||
<div v-if="loading" class="text-sm text-white/50">Checking…</div>
|
||||
|
||||
<div
|
||||
v-else-if="loadError"
|
||||
class="p-3 bg-white/5 border border-white/10 rounded-lg text-sm text-white/70"
|
||||
>
|
||||
Could not read the Lightning credential state: {{ loadError }}
|
||||
</div>
|
||||
|
||||
<div
|
||||
v-else-if="!status?.installed"
|
||||
class="p-3 bg-white/5 border border-white/10 rounded-lg text-sm text-white/70"
|
||||
>
|
||||
Lightning is not set up on this node yet, so there are no credentials to
|
||||
rotate. Install the Lightning app first.
|
||||
</div>
|
||||
|
||||
<div v-else class="space-y-4">
|
||||
<!-- What exists right now -->
|
||||
<dl class="grid grid-cols-1 sm:grid-cols-2 gap-3 text-sm">
|
||||
<div>
|
||||
<dt class="text-white/50 text-xs">Issued</dt>
|
||||
<dd class="text-white/80">{{ status.issued_at || 'unknown' }}</dd>
|
||||
</div>
|
||||
<div>
|
||||
<dt class="text-white/50 text-xs">Credential fingerprint</dt>
|
||||
<dd class="text-white/80 font-mono text-xs break-all">
|
||||
{{ shortHash(status.admin_macaroon_sha256) }}
|
||||
</dd>
|
||||
</div>
|
||||
<div>
|
||||
<dt class="text-white/50 text-xs">Channels that must survive</dt>
|
||||
<dd class="text-white/80">
|
||||
<template v-if="status.channels_open !== null">
|
||||
{{ status.channels_open }} open<span v-if="status.channels_pending">
|
||||
, {{ status.channels_pending }} pending</span
|
||||
>
|
||||
</template>
|
||||
<span v-else class="text-white/50">not readable — Lightning is not answering</span>
|
||||
</dd>
|
||||
</div>
|
||||
<div>
|
||||
<dt class="text-white/50 text-xs">Node identity</dt>
|
||||
<dd class="text-white/80 font-mono text-xs break-all">
|
||||
{{ status.identity_pubkey ? `${status.identity_pubkey.slice(0, 16)}…` : '—' }}
|
||||
</dd>
|
||||
</div>
|
||||
</dl>
|
||||
|
||||
<!-- Lightning has to be answering for a rotation to be verifiable at all,
|
||||
so this is a blocker rather than a footnote. -->
|
||||
<div
|
||||
v-if="status.lnd_error && !isRunning"
|
||||
class="p-3 bg-orange-500/10 border border-orange-500/30 rounded-lg text-sm text-orange-100/90"
|
||||
>
|
||||
<p class="font-medium mb-1">Lightning is not answering right now.</p>
|
||||
<p class="text-orange-100/70">
|
||||
Rotation is blocked until it is: without a reading from before the
|
||||
change there is no way to prove afterwards that your channels came
|
||||
back. Wait for Lightning to finish starting and reload this page.
|
||||
</p>
|
||||
<p class="text-xs text-orange-100/50 mt-2 font-mono break-all">{{ status.lnd_error }}</p>
|
||||
</div>
|
||||
|
||||
<!-- The failure this whole feature exists to prevent. -->
|
||||
<div
|
||||
v-if="btcpayStale"
|
||||
class="p-3 bg-red-500/10 border border-red-500/30 rounded-lg text-sm text-red-100/90"
|
||||
>
|
||||
<p class="font-medium mb-1">BTCPay Server is holding an old Lightning credential.</p>
|
||||
<p class="text-red-100/70">
|
||||
BTCPay keeps its own copy of the credential, and the copy it has no
|
||||
longer works — so its Lightning payments will fail even though both
|
||||
apps look healthy. Rotating now repairs this as part of the run.
|
||||
</p>
|
||||
</div>
|
||||
|
||||
<!-- Progress. Shown while running and kept afterwards, because the
|
||||
verdict ("same node, same channels") is the reassurance the operator
|
||||
came here for. -->
|
||||
<div v-if="rotation && (isRunning || finished)" class="p-3 bg-white/5 border border-white/10 rounded-lg">
|
||||
<p class="text-sm font-medium text-white/80 mb-2">
|
||||
<span v-if="isRunning">Rotating…</span>
|
||||
<span v-else-if="rotation.ok" class="text-emerald-400">Rotation complete</span>
|
||||
<span v-else class="text-red-400">Rotation failed</span>
|
||||
</p>
|
||||
<ul class="space-y-1.5">
|
||||
<li v-for="step in rotation.steps" :key="step.key" class="text-sm">
|
||||
<span class="font-mono mr-2" :class="stepClass(step.state)">{{
|
||||
stepIcon(step.state)
|
||||
}}</span>
|
||||
<span :class="step.state === 'pending' ? 'text-white/40' : 'text-white/80'">{{
|
||||
step.label
|
||||
}}</span>
|
||||
<p v-if="step.detail" class="ml-6 text-xs text-white/50">{{ step.detail }}</p>
|
||||
</li>
|
||||
</ul>
|
||||
|
||||
<p v-if="rotation.error" class="mt-3 text-xs text-red-300/90 break-words">
|
||||
{{ rotation.error }}
|
||||
</p>
|
||||
|
||||
<div v-if="finished && rotation.ok" class="mt-3 space-y-2 text-xs text-white/60">
|
||||
<p class="text-white/80">
|
||||
Re-pair anything that connects to this node — Zeus most importantly.
|
||||
Open the Lightning app and scan its pairing QR again; it serves the
|
||||
new credential.
|
||||
</p>
|
||||
<p v-if="rotation.backup_path">
|
||||
The old credentials were backed up on the node so a mistake is
|
||||
recoverable. That backup is still sensitive. Once every app is
|
||||
re-paired, delete it:
|
||||
<code class="block mt-1 px-2 py-1 bg-black/30 rounded font-mono break-all"
|
||||
>sudo rm -rf {{ rotation.backup_path }}</code
|
||||
>
|
||||
</p>
|
||||
</div>
|
||||
</div>
|
||||
|
||||
<button
|
||||
:disabled="isRunning || !!status.lnd_error"
|
||||
class="w-full flex items-center justify-center gap-2 px-4 py-2 rounded-lg glass-button glass-button-warning font-medium disabled:opacity-50 disabled:cursor-not-allowed"
|
||||
@click="openConfirm"
|
||||
>
|
||||
<svg class="w-5 h-5" fill="none" stroke="currentColor" viewBox="0 0 24 24">
|
||||
<path
|
||||
stroke-linecap="round"
|
||||
stroke-linejoin="round"
|
||||
stroke-width="2"
|
||||
d="M4 4v5h.582m15.356 2A8.001 8.001 0 004.582 9m0 0H9m11 11v-5h-.581m0 0a8.003 8.003 0 01-15.357-2m15.357 2H15"
|
||||
/>
|
||||
</svg>
|
||||
<span>{{ isRunning ? 'Rotating…' : 'Rotate Lightning credentials' }}</span>
|
||||
</button>
|
||||
</div>
|
||||
</div>
|
||||
|
||||
<!-- Confirmation. Teleported to body: a glass-panel ancestor creates a
|
||||
transform context that would trap a position:fixed backdrop. -->
|
||||
<Teleport to="body">
|
||||
<div
|
||||
v-if="showConfirm"
|
||||
class="fixed inset-0 z-50 flex items-center justify-center p-4 bg-black/60 backdrop-blur-md"
|
||||
@click.self="closeConfirm"
|
||||
@keydown.escape="closeConfirm"
|
||||
>
|
||||
<div
|
||||
class="glass-card p-6 max-w-md w-full"
|
||||
role="dialog"
|
||||
aria-modal="true"
|
||||
aria-labelledby="rotate-macaroon-title"
|
||||
>
|
||||
<h3 id="rotate-macaroon-title" class="text-lg font-semibold text-white mb-2">
|
||||
Rotate Lightning credentials
|
||||
</h3>
|
||||
<div class="text-sm text-white/70 space-y-2 mb-4">
|
||||
<p>
|
||||
<strong class="text-white/90">What changes:</strong> every app paired
|
||||
with this node stops working until you re-pair it. Zeus and any other
|
||||
remote wallet will need to scan a fresh pairing code.
|
||||
</p>
|
||||
<p>
|
||||
<strong class="text-white/90">What does not:</strong> your coins and
|
||||
your channels. The node keeps its identity, nothing is closed, and
|
||||
this run refuses to report success unless it has confirmed both.
|
||||
</p>
|
||||
<p>
|
||||
Lightning restarts as part of this, which takes a few minutes on a
|
||||
busy node. Payments cannot be sent or received during that window.
|
||||
</p>
|
||||
</div>
|
||||
<form class="space-y-4" @submit.prevent="rotate">
|
||||
<label class="block">
|
||||
<span class="text-xs text-white/60">Confirm with your node password</span>
|
||||
<input
|
||||
v-model="password"
|
||||
type="password"
|
||||
required
|
||||
autocomplete="current-password"
|
||||
class="mt-1 w-full px-3 py-2 rounded-lg bg-white/10 text-white border border-white/20 focus:border-orange-500 focus:ring-1 focus:ring-orange-500"
|
||||
placeholder="Node password"
|
||||
/>
|
||||
</label>
|
||||
<p v-if="confirmError" class="text-sm text-red-400 break-words">{{ confirmError }}</p>
|
||||
<div class="flex gap-3">
|
||||
<button
|
||||
type="button"
|
||||
class="flex-1 px-4 py-2 rounded-lg glass-button font-medium"
|
||||
@click="closeConfirm"
|
||||
>
|
||||
Cancel
|
||||
</button>
|
||||
<button
|
||||
type="submit"
|
||||
:disabled="submitting || !password"
|
||||
class="flex-1 px-4 py-2 rounded-lg bg-orange-500 text-white font-medium hover:bg-orange-600 disabled:opacity-50 disabled:cursor-not-allowed transition-colors"
|
||||
>
|
||||
{{ submitting ? 'Starting…' : 'Rotate' }}
|
||||
</button>
|
||||
</div>
|
||||
</form>
|
||||
</div>
|
||||
</div>
|
||||
</Teleport>
|
||||
</template>
|
||||
@@ -6,6 +6,7 @@ import AIDataAccessSection from '@/views/settings/AIDataAccessSection.vue'
|
||||
import WebhookSection from '@/views/settings/WebhookSection.vue'
|
||||
import TelemetrySection from '@/views/settings/TelemetrySection.vue'
|
||||
import NodeCertificateSection from '@/views/settings/NodeCertificateSection.vue'
|
||||
import LightningCredentialsSection from '@/views/settings/LightningCredentialsSection.vue'
|
||||
import BackupSection from '@/views/settings/BackupSection.vue'
|
||||
import SystemDangerZone from '@/views/settings/SystemDangerZone.vue'
|
||||
</script>
|
||||
@@ -18,6 +19,7 @@ import SystemDangerZone from '@/views/settings/SystemDangerZone.vue'
|
||||
<WebhookSection />
|
||||
<TelemetrySection />
|
||||
<NodeCertificateSection />
|
||||
<LightningCredentialsSection />
|
||||
<BackupSection />
|
||||
<SystemDangerZone />
|
||||
</template>
|
||||
|
||||
@@ -268,6 +268,57 @@ if [ -n "$FAIL" ]; then
|
||||
die "rotation verification FAILED:$FAIL — old material is in $BACKUP"
|
||||
fi
|
||||
|
||||
# ── BTCPay's inline copy ──────────────────────────────────────────────
|
||||
# BTCPay reaches the internal LND node with a connection string that carries
|
||||
# the macaroon INLINE as hex, not as a file path: LND's datadir is owned by its
|
||||
# container's mapped uid, so btcpay cannot bind-mount the file. That copy is
|
||||
# therefore now a dead credential, and nothing else will notice — the daemon
|
||||
# only regenerates this secret when LND's TLS *cert* thumbprint changes, which
|
||||
# macaroon rotation does not touch. The node keeps looking healthy (btcpay up,
|
||||
# LND up) while every Lightning invoice BTCPay tries to create fails.
|
||||
#
|
||||
# Deleting the secret file gets the daemon to regenerate it from the new
|
||||
# macaroon on its next reconcile tick. That is necessary but NOT sufficient, and
|
||||
# the difference matters: the RUNNING container still holds the dead value, and
|
||||
# the periodic reconciler only ever runs in `ExistingOnly` mode, where env drift
|
||||
# on a restart-sensitive app (btcpay-server is one) is detected and then
|
||||
# deliberately skipped — "leaving running restart-sensitive app untouched". So
|
||||
# the container has to be recreated on purpose. The dashboard path
|
||||
# (Settings → Lightning credentials) does this itself by flagging the app as
|
||||
# credential-rotated; a shell script cannot reach that in-process flag, so it
|
||||
# removes the container instead and lets the orchestrator's own desired-state
|
||||
# recovery rebuild it around unchanged data, ports and volumes.
|
||||
#
|
||||
# Nothing is printed but a path — never the value.
|
||||
BTCPAY_SECRET="/var/lib/archipelago/secrets/btcpay-lnd-connection"
|
||||
BTCPAY_NOTE=no
|
||||
if sudo test -f "$BTCPAY_SECRET"; then
|
||||
if sudo rm -f "$BTCPAY_SECRET"; then
|
||||
say
|
||||
say "btcpay : removed its stale connection string ($BTCPAY_SECRET)."
|
||||
say " The daemon regenerates it from the new macaroon within a minute."
|
||||
BTCPAY_NOTE=yes
|
||||
if podman container exists btcpay-server 2>/dev/null; then
|
||||
say " Recreating btcpay-server so it stops using the dead one."
|
||||
podman stop btcpay-server >/dev/null 2>&1 || true
|
||||
if podman rm -f btcpay-server >/dev/null 2>&1; then
|
||||
say " Removed; the orchestrator rebuilds it around its existing"
|
||||
say " data (it was running, so desired-state recovery restores it)."
|
||||
else
|
||||
say " ⚠ could not remove btcpay-server. Its Lightning payments will"
|
||||
say " fail until it is recreated."
|
||||
BTCPAY_NOTE=warn
|
||||
fi
|
||||
fi
|
||||
else
|
||||
say
|
||||
say "btcpay : ⚠ could not remove $BTCPAY_SECRET. BTCPay is still holding"
|
||||
say " the OLD macaroon, so its Lightning payments will fail until"
|
||||
say " that file is deleted and btcpay-server is recreated."
|
||||
BTCPAY_NOTE=warn
|
||||
fi
|
||||
fi
|
||||
|
||||
say
|
||||
say "✅ Rotated. Every macaroon issued before now no longer verifies."
|
||||
say
|
||||
@@ -278,6 +329,13 @@ say " and scan the new pairing QR; it serves the new macaroon."
|
||||
say
|
||||
say " Your funds and channels are untouched: the node kept its identity and"
|
||||
say " no channel was closed."
|
||||
if [ "${BTCPAY_NOTE:-no}" != no ]; then
|
||||
say
|
||||
say " CONFIRM BTCPAY CAME BACK. A silent failure here looks identical to success:"
|
||||
say " btcpay stays up and healthy while every Lightning payment it tries fails."
|
||||
say " podman inspect btcpay-server --format '{{.Created}}' # should be just now"
|
||||
say " sudo test -f $BTCPAY_SECRET && echo regenerated"
|
||||
fi
|
||||
say
|
||||
say " Once every client is re-paired, delete the backup — it holds the OLD"
|
||||
say " root key, which is still sensitive:"
|
||||
|
||||
Reference in New Issue
Block a user