Device panel gets MeshCore RF fields (frequency MHz, bandwidth kHz, SF, CR)
wired to mesh.configure's validated lora_radio_params; shown only for
MeshCore radios, all-empty leaves the radio's flashed settings untouched.
Human units in the UI, firmware field units on the wire. Per operator
direction the panel's static stat rows duplicating editable fields (name,
region, channel) are dropped — the editable control is the display. Stale
'adjust via a MeshCore client' hints updated: the node programs the radio now.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Archy set no radio params on Meshcore devices — SF/CR/BW/freq lived only in
device firmware, so a radio on the wrong preset could not be fixed from the
node (it hears RF energy but demodulates nothing; the fleet hit exactly this
on an RNode: docs/RETICULUM-TRANSPORT-PROGRESS.md item 4).
- protocol.rs: build_set_radio_params — wire format verified against the
MeshCore companion firmware handler (CMD_SET_RADIO_PARAMS=11:
[11][freq:u32 LE MHz*1000][bw:u32 LE kHz*1000][sf][cr]); byte-layout test
pinned to the Portugal preset 869.618 MHz / 62.5 kHz / SF8 / CR8.
- serial.rs: MeshcoreDevice::set_radio_params (device reboots on OK).
- MeshConfig.lora_radio_params: Option — None (default) leaves the radio
untouched, so only explicitly-configured deployments are affected.
- session.rs: provision on connect, gated on a persisted last-applied marker
(SELF_INFO offsets shift across firmware versions) + the same attempt cap
as region/channel so a refusing radio never reboot-loops.
- mesh.configure RPC: lora_radio_params arm with firmware-range validation.
mesh tests: 114/114 pass.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Register "pine" as a manifest-driven stack (like netbird) across every
lifecycle site so install/stop/start/restart/uninstall/crash-recovery treat
the launcher + two Wyoming engines as one app:
- stacks.rs: pine_stack_app_ids() + install_pine_stack() (orchestrator-first,
adopt existing, else bail — no hardcoded fallback)
- install.rs: dispatch package_id == "pine" to the stack installer
- app_ops.rs: canonical stack-member table + owning_package STACKS
- dependencies.rs: startup_order + needs_archy_net
- config.rs: all_container_names
- crash_recovery.rs: auto-start members + a StackRecoverySpec (anchor
pine-whisper) so a genuinely-installed stack self-heals but orphan debris
does not crash-loop
- docker_packages.rs: exclude the engines from the apps list
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
RPC failures logged only the top-level anyhow context, so a peer-files
failure produced exactly "RPC error on content.browse-peer: Failed to
connect to peer" with the real cause discarded. That made it impossible
to tell a dead peer from a slow Tor circuit from a FIPS resolve failure
without reproducing by hand.
Switch the log line to `{:#}` so the whole context chain is rendered.
The client-facing message still uses `{}` through sanitize_error_message,
so no internal detail is leaked to callers.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
vpn.peer-config returned the config exactly as stored at creation time, so
after a node moves networks the reused companion peer's QR encodes the old
location's address (seen on .116: Endpoint = 10.125.9.0 from the previous
LAN) and the tunnel can never connect. Rewrite the Endpoint line with the
node's current address before rendering the QR, and persist it back so the
downloadable .conf matches. Endpoint detection is shared with
vpn.create-peer via current_wg_endpoint_host().
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Backend: tor.list-services now hides services whose name maps to a
known-but-uninstalled catalog app (aliases covered: bitcoin/knots/core,
electrumx/electrs, btcpay, mempool). The node's own service, the relay,
and custom user services always show.
ISO: first-boot tor setup no longer pre-creates hidden services for six
apps that may never be installed — only the node's own onion is baked.
It also only seeds services.json/torrc on FIRST boot instead of
clobbering backend-managed services on every boot.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Chromium's SingletonLock is a symlink encoding <hostname>-<pid>. After a
node rename changes the OS hostname, the lock left from the previous
boot reads as "another computer" holding the profile; Chromium refuses
to start, --noerrdialogs hides the only dialog, and the kiosk
black-screens forever.
Two layers:
- kiosk launcher removes Singleton{Lock,Cookie,Socket} before every
Chromium start (any prior owner is dead at that point — pkill at
session start / loop respawn). Ships via ISO splice AND the binary's
include_str! self-heal, so existing nodes get it with the next OTA.
- the rename handler clears the lock immediately as a hostname
side-effect, covering nodes still running a pre-fix launcher.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
auth.setup only wrote the web password hash to user.json, so the
archipelago system user kept the image default password ("archipelago")
on console/SSH even after the user picked a real password during
onboarding. Reuse the existing usermod-based sync (already used by
auth.changePassword's alsoChangeSsh path) at setup time, best-effort so
a failure can never break onboarding.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Every Lightning setup (Open a Shop, Accept Payments, Run a Lightning Node)
gains a guided path to a working channel:
- New "Fund Your Bitcoin Wallet" step, gated on the blockchain finishing
its initial sync: while syncing it shows a live progress bar with a
counting-down time-remaining estimate; once synced it shows the on-chain
balance and a "Fund Wallet" button that opens the receive modal with a
fresh address, QR, and the Zeus channel limits (min 150,000 / max
1,500,000 sats) noted.
- When the sync finishes while a Lightning setup is mid-flight, a toast
pops with a "Finish setup" link straight back to the wizard (toasts now
support action links). If several setups are in flight, one is chosen —
the shared steps complete the lightning part of any of them.
- Channel steps are Zeus-branded (logo + copy) and land on the Lightning
Channels screen, which now carries an "Open a channel with Zeus" card
that prefills the open-channel modal with the Olympus peer URI, 150k
sats, and private-channel checked. "Get Zeus" links to zeusln.com.
- Setup completion now actually requires walking the manual steps (fund,
open channel, configure) — previously any step whose app was installed
was silently auto-ticked and running apps marked the whole goal done.
- Completion CTAs now go to the app you just set up ("Go to my shop
(BTCPay)" etc.) instead of the generic services list; iframe apps open
in the on-top app overlay.
- On-chain send modal gains a "Send all funds" sweep toggle, backed by
LND's send_all on the backend (amount no longer required when sweeping).
- Demo/mock: bitcoin.getinfo now returns the block_height/sync_progress
contract the UI reads and simulates a ~90s IBD ramp per visitor so the
timer, toast, and fund flow can all be demoed live; channel list data
fixed (status/channel_point/liquidity totals — the panel previously
crashed on the missing status field); sendcoins supports send_all.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Orchestrator-installed stacks (indeedhub, btcpay, netbird, immich,
mempool) sat at "Preparing… 5%" for the entire install — the
install_stack_via_orchestrator path never called set_install_phase /
set_install_progress after the initial Preparing, so a 7-image,
multi-gigabyte indeedhub pull gave the user zero feedback ("just says
preparing for ages"). The legacy installers already report phases and
the frontend already interpolates X-of-N across the PullingImage band;
the new path simply never fed it.
Now: PullingImage phase up front, an X-of-N counter plus a per-member
"Installing postgres (2 of 7)…" message as each member installs, and
the legacy installers' truthful PostInstall→Done settle at the end.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The full backup carried identity/lnd_aezeed.enc but NOT the per-node
wallet secret that encrypts it (secrets/lnd-wallet-password), so a
restored node could never decrypt its Lightning seed. The secrets dir
now rides the backup (archive v3; the .bak itself is
passphrase-encrypted). Restore gains a staging-presence guard so
restoring an older archive without a secrets dir can no longer displace
and delete the node's live secrets — covered by two new tests.
Backups can now also be downloaded through the browser: a session-gated
GET /api/blob/backup/<id> endpoint (under the existing nginx /api/blob
prefix, so no fleet nginx changes) plus a Download button next to
Verify / USB / Restore / Delete in Settings → Backup & Restore.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Two confirmed fresh-node failure modes made 'install LND' fail on the
first try and succeed on the second:
- The dependency gate failed fast when Bitcoin was still mid-install
(image pulling, container not created yet): DepProbe.existing only
covered 'podman ps -a', so an in-flight dependency looked identical to
'not installed at all'. The probe now also reports packages in
Installing/Updating state, and the gate waits up to 30 minutes for an
in-flight dependency install to finish (surfacing 'Waiting for X to
finish installing…' on the card) before its normal 3-minute
started-but-not-running window.
- Image pulls had a hard 300s per-URL wall clock, which killed
slow-but-progressing pulls of large images (LND on fresh-node WiFi);
the retry then succeeded off the warm layer cache. Pulls are now
stall-aware: killed only after 3 minutes with no podman output AND no
byte movement in the staging dir, with a 30-minute absolute ceiling.
Dead registries still fail in ~3 minutes.
Adds dep-gate unit tests for the in-flight-install wait and the
neither-installed-nor-installing fail-fast.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
server.set-name ran hostnamectl but left Debian's 127.0.1.1 line on the
old name (breaking sudo's 'unable to resolve host' and hostname -f) and
waited on avahi to notice the change by itself. Sync /etc/hosts and
avahi-set-host-name (fallback: try-restart avahi-daemon) right after a
successful rename, so http(s)://<hostname>.local works immediately —
best-effort, never blocks the rename.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
'Add Service' only worked for the 19 apps in the hardcoded
known_service_port map — everything else failed with 'Unknown app'.
tor.create-service and tor.toggle-app now fall back to the app's live
launch address from the scanner state, so any manifest-driven app gets
a .onion without a per-app entry (the static map still wins for
protocol services like bitcoin, which must expose 8333, not a UI port).
toggle-app also no longer writes a broken port-0 HiddenServicePort for
unknown apps.
Every successful package.install now auto-creates the app's hidden
service (detached, best-effort, retries while the scanner derives the
port; skips protocol services and existing entries).
The demo mock gains stateful tor.create-service/tor.delete-service so
the demo round-trips instead of erroring.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
barkd (Ark wallet daemon, pinned 0.3.0, checksum-verified release binary)
packaged as an installable app; thin HTTP bridge in wallet/ark_client.rs
mirrors the fedimint_client pattern — the bark SDK stays out of the node
binary. wallet.ark-* RPCs cover status/balance/address/send/invoice/
board/offboard/history/configure; Ark movements merge into the unified
ecash history (kind="ark") and spendable Ark sats into total_sats.
Signet defaults (Second's public Ark server) until Ark matures.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The optimistic package_data entry painted at install-spawn survived
failure as a 'Stopped' tile for an app with zero footprint (netbird on
the fresh ISO). On INSTALL FAIL with no container/user-stop marker for
the app, the entry and the pre-saved dynamic app config are removed and
an error notification is pushed; failures that left a real container
keep today's behavior (visible exited state for retry/diagnosis).
Install suite 39/39.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
write_bitcoin_conf silently produced an unpruned (~810GB and growing)
config regardless of disk: the framework node got a full-archive chain
on a 205GB volume. Volumes under 1.2TB now get prune = 25% of the
volume, clamped to [550MB, 100GB]; bigger volumes keep full archive;
unknown df results stay unpruned (don't surprise big iron). Framework
node switched live to prune=50000.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Plugging in a LoRa radio now pops a global two-step setup modal on any
page: step 1 shows the ACTUAL detected board (25 board SVGs vendored
from the Meshtastic web-flasher, GPL-3.0; matched by USB product string
on native-USB boards, vid:pid heuristics for bridge chips, animated
fallback otherwise); step 2 configures with Archipelago presets — the
LoRa region pre-selected from the node's location (new lat/lon→region
mapping), channel 'archipelago', node name, and an advanced firmware
pin. Options adapt per protocol: region+channel program Meshtastic;
MeshCore shows its community frequency plan (firmware owns RF); RNode
defers to the Reticulum daemon config.
Backend: mesh.configure now accepts lora_region (validated against the
driver's region table) and device_kind (probe pin); mesh.status returns
the persisted values plus per-port USB identity (sysfs vid/pid/product)
for board identification. The Mesh Device tab gains a full settings
form (region with duty-cycle/legality hints, firmware pin, channel,
name, identity broadcast) replacing the read-only panel; the old
Mesh-page-only onboarding modal is superseded by the global flow.
Mock backend: stateful mesh config so the dev UI demos the full
detect→configure round-trip (disable mesh → modal fires with Heltec V3).
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Root cause of the fresh-install triple failure (LND needing ~5 install
attempts, Bitcoin UI bitcoin-rpc 502, backend getblockchaininfo errors):
the legacy conf writer set rpcallowip but never rpcbind, so bitcoind
bound RPC to the container's loopback only — every dial over the
container network (LND, bitcoin-ui) and the rootless port-forward was
refused. LND exited instantly, the 60s post-start poll read that as
INSTALL CRASH, and the wallet (and its seed backup) never got created.
- write_bitcoin_conf + ensure_bitcoin_rpc_config now emit/ensure
rpcbind=0.0.0.0 (host exposure unchanged: publish stays 127.0.0.1)
- new install gate: bitcoin-dependent apps (lnd/electrs/electrumx/
mempool-electrs/btcpay) wait up to 3 min for an authenticated
getblockchaininfo to answer before their container starts, with
INSTALL WAIT progress lines; on timeout the install fails with an
actionable message instead of a crash-looping container
Verified live on the fresh-ISO framework node: after adding rpcbind and
restarting bitcoin-knots, LND connected, created its wallet, baked
macaroons, and settled into 'Waiting for chain backend to finish sync'.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Invites carried no trust level, and both accept_invite and the
peer-joined handler hardcoded TrustLevel::Trusted — so 'Invite a Peer'
(Observer) federated both sides as fully Trusted.
- invite codes now carry a 'trust' field (legacy codes parse as Trusted)
- federation.invite accepts trust_level: trusted|observer
- accept_invite assigns the invite's level on the acceptor side
- peer-joined resolves the granted level authoritatively by matching the
acceptor's echoed invite token against our stored outgoing invites;
the peer's own unsigned claim is honored only as a downgrade, so no
escalation is possible
- discovery/connection-request approvals mint Observer invites directly
(the post-hoc demotion in handshake.rs stays as a legacy safety net)
- asymmetry self-heal re-asserts at the locally-held level
Tests: observer threading + legacy default-trusted parse.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
telemetry.fleet-status only read the telemetry-fleet/ collector inbox —
an opt-in anonymous-telemetry pipeline that federated peers never write
to — so Fleet showed 'No nodes reporting' even with a healthy federation.
Build the fleet list from nodes.json instead: every TRUSTED federated
node is mapped from its last_state sync snapshot (cpu/mem/disk, uptime,
apps, peer count) into the FleetNode shape the UI expects, keyed by DID.
Observer ('peer') and untrusted nodes are excluded by design. Collector
reports are still merged in for non-federated telemetry nodes.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Web5ConnectedNodes sent { request_id } but the backend only read
params.id, so accepting/rejecting a connection request always failed
with 'Missing required parameter: id'. Frontend now sends id; backend
accepts both spellings for older deployed UIs.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
- aezeed words are now captured at BOTH wallet-init paths and stored
encrypted (Argon2 + ChaCha20-Poly1305, per-node wallet secret) at
identity/lnd_aezeed.enc; ack marker cleared when a wallet is recreated
- lnd.init-wallet-from-seed was posting seed_entropy to /v1/initwallet,
which is a GenSeed field — the wallet was never actually derived from
the master seed; now GenSeed(entropy) → InitWallet(words)
- new RPCs: lnd.seed-backup-status / lnd.seed-reveal (password + 2FA
gated, same as seed.reveal via shared verify_reveal_auth) /
lnd.seed-backup-ack
- LND app detail page: Lightning-seed card with first-launch backup
prompt, tap-to-reveal modal, 'I've backed it up' acknowledgment
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Kiosk (backgrounds + FIPS mismatch, one root cause each):
- kiosk service now also waits for the web-ui asset swap (probes a large
bg asset, bounded 60s) — it launched mid-rsync and CSS background 404s
are never refetched, hence blank backgrounds on first boot
- FipsNetworkCard polls every 15s instead of fetching once on mount
- PWA updates auto-apply on the kiosk (localStorage kiosk flag) — an
unattended kiosk could never click 'Update Now' and ran stale bundles
LND App Store:
- catalog icon pointed at lnd.svg which doesn't exist (lnd.png does);
store card also retries the sibling extension before the placeholder
- image pulls retry once after fast transient failures (<30s) — fresh
first-boot networks reset connections; slow failures aren't retried
so the UI never waits longer than before
Mesh detection:
- mesh.status now always reports device_present + detected_devices
(previously only when the service wasn't running, so the UI couldn't
tell 'no radio' from 'radio present, session connecting')
- Mesh page shows 'Mesh device detected' badge + 'Device detected —
connecting' status for present-but-not-connected radios
- the 99-mesh-radio.rules udev rule was never staged into the ISO tree
(installer searched for it and found nothing) — ship it at the media
root and install it from install-to-disk.sh too
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
- image: create archipelago user with dialout so the backend can open
/dev/ttyUSB*/ttyACM* LoRa radios; doctor Fix 14 heals existing nodes
(verified live on the .65 RC install — device detected after the fix)
- Web5Federation.vue called nonexistent federation.list (Unknown method);
now uses federation.list-nodes + federation.list-pending-requests with
a last-seen-based online count
- sanitizer: let app-install dependency errors through — 'LND requires a
running Bitcoin node' was masked as 'Check server logs', so install
failures on fresh nodes looked random
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
- new lnd.lightning-history RPC: settled invoices (incoming) + succeeded
payments (outgoing) normalized to the wallet-transaction shape; on-chain
history stays in lnd.gettransactions
- Home merges on-chain + lightning + ecash history into one sorted list;
every entry now carries kind (onchain/lightning/cashu/fedimint)
- modal renders a rail badge for non-onchain rows instead of a bogus
confirmation count, and only on-chain rows link into mempool
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The tx1138 Gitea was retired as a release server 2026-06-13 and its registry
frontend is fully dead (500 on every /v2 manifest read, observed 2026-07-10).
Nothing may reference it anymore:
- registry.rs: no longer a default registry, no longer force-enabled on
load; saved configs are stripped of it on load (same one-time migration
treatment as the decommissioned Hetzner mirror), with a regression test.
- image_policy.rs: removed from TRUSTED_REGISTRIES — refs through the dead
host are now refused at the pull site (rejection test added).
- api/handler: dropped the legacy catalog-proxy fallback URL.
- .gitmodules: indeedhub submodule repointed to the OVH Gitea.
- scripts, image-recipe, app-catalog data, neode-ui strings, docs, and all
test fixtures repointed to 146.59.87.168:3000 (or neutral example hosts).
- image-versions.sh: ARCHY_REGISTRY_FALLBACK emptied (guarded consumers
skip it); reconcile-containers.sh candidate guard hardened.
The only remaining occurrences of the host string are the strip/reject
enforcement paths and their regression tests — the code that guarantees it
is never used again.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Channel open fee control, channels tab in wallet settings, mempool tx
link, and the channel-open async peer-connect race fix. Conflict in
LightningChannels.vue resolved to the branch's extracted
LightningChannelsPanel component, with main's Teleport-to-body modal
fixes and glass-button-warning class (8256fde1) ported into the panel
so they aren't lost. lnd tests 16/16, LightningChannels vitest green.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
24fd97ed cascades on package.start/restart, but a reconcile pass that
recreates or starts a bitcoin backend (desired-state recovery, repair
recreate, boot InstallMissing) also moves the RPC address behind a
running lnd's back — §C 'restart lnd after ANY bitcoin recreate'.
After the pass, dependents from the shared address_caching_dependents
table (moved to app_ops as the single source of truth) that sat
untouched (NoOp) and aren't user-stopped are restarted under their op
lock; a dependent the pass itself (re)created already resolved the
fresh address and is left alone.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
- Fee selector on channel open: Standard (~6 blocks) / Medium (~3) /
Fast (next block) presets, or custom target confirmations / sat/vB;
backend validates and passes target_conf / sat_per_vbyte to LND
- Extract channel management into LightningChannelsPanel, reused by the
LND channels page and a new Channels tab in the Wallet Settings modal
- Funding tx link on each channel card opens the node's mempool app at
/tx/<txid>
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
- Connect to peer synchronously (perm=false, 30s timeout) and check the
result before opening the channel; previously the connect was fired
with perm=true (queued in background, returns immediately) and its
result ignored, so the open failed with 'peer is not online'
- Let LND channel/peer errors through the error sanitizer so the UI
shows the real cause instead of 'Check server logs'
- Add private (unannounced) channel option to backend and UI — some
peers (e.g. Olympus/ZEUS LSP) reject public channel opens
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
load_manifests() only ran at startup, so a manifest published via the
signed catalog sat dormant until the next service restart (found while
shipping the strfry manifest fixes: package.check-updates refreshed the
cache but the orchestrator kept rendering the old overlay).
- refresh_catalog now reports whether the cached bytes actually changed
(write_cache skips identical rewrites), so unchanged hourly polls
don't churn the manifest map.
- New ContainerOrchestrator::reload_manifests (default no-op); prod
delegates to load_manifests — the rebuild is atomic under the state
write lock, and the reconciler's durable user-stopped/uninstalled
marker filters make a mid-session reload equivalent to the restart
path that already runs on every boot.
- package.check-updates reloads on change and reports catalog_changed +
manifests_reloaded; the hourly update-scheduler tick (and its startup
refresh) do the same.
Tests: app_catalog 5/5 (new write_cache changed-detection test),
reconcile 16/16, knows_app 1/1.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
package.install routed to the manifest-driven orchestrator only for a
hardcoded per-app allowlist; every other app fell through to the legacy
flow, which ignores manifests entirely and creates a bare container with
no ports or volumes (strfry crash-looped this way on .228, 2026-07-09).
New ContainerOrchestrator::knows_app(app_id) (default false; prod checks
its loaded-manifest map, disk + signed-catalog overlay). The install gate
is now allowlist OR knows_app — no per-app Rust for manifest-driven apps,
matching the packaging invariant. Apps without a manifest keep the legacy
flow including the container-exists adopt block; the unknown-app_id →
legacy fallback inside the orchestrator branch is unchanged.
Tests: knows_app_reflects_loaded_manifests + install suite 37/37.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The reconciler doesn't take the RPC layer's per-app FIFO op lock (known
limit of 891cbba4): between a restart worker's stop and start halves it saw
the mempool frontend "missing", repair-recreated it behind systemd's back,
killed the worker's fresh container 11s after start, and left the unit down
for ~3.5 min until the next heal — gate test 123 measured exactly that
window (.228 iteration 3, 2026-07-09).
New crate::app_ops module owns the op-lock registry + stack member table
(runtime.rs and dependencies.rs now delegate) so the reconciler can probe
lifecycle_op_in_flight(app_id) — covering both the app's own key and its
owning stack package — and skip that app for the cycle. The ownership-sweep
podman restart gets the same guard. Health monitor is a follow-up.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
lnd resolves the bitcoin RPC address once at startup and never re-resolves;
when bitcoin-core/knots restarts (new container IP) lnd spins on "dial tcp
<old-ip>:8332: no route to host" and its own RPC wedges until lnd restarts
(gate lnd getinfo test 75 failed every iteration on .228, 2026-07-09;
hardening plan §C cascade item). electrumx is already restarted by the
version-switch path — lnd was the only dependent left stranded.
After a successful package.start/restart of a bitcoin backend, restart
running, not-user-stopped address-caching dependents under their own app-op
lock (lock order backend→dependent only, so no deadlock).
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
package.stop/restart on orchestrator-managed stacks (immich, indeedhub,
btcpay, netbird, mempool) addressed live CONTAINER names; unknown app ids
fell through to raw podman stop/start, racing systemd's --rm cleanup of the
quadlet unit — the container vanished on stop and the start half found
nothing (immich RESTART FAIL + indeedhub stuck restarting, gate 2026-07-09).
Now stack ops resolve member APP ids (reverse start order for stop) so the
orchestrator drives the quadlet .service; user-stop markers are written for
member app ids too (covers absent containers), and the orchestrator stop
fan-out treats an already-missing container as stopped.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
/var/log/archipelago/container-installs.log has been 0 bytes since April:
the service sandbox (ProtectSystem=strict class) leaves /var/log
read-only for the unit, the open() fails, and install_log() drops every
line by design (fire-and-forget). These START/STOP/RESTART/FAIL
breadcrumbs are the primary forensic trail for gate failures, so mirror
every line to tracing (journald) unconditionally and keep the file
append as best-effort for hosts where the path is writable.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
package.start/stop/restart reply immediately and run their multi-container
sequences in spawned tasks with no coordination. Back-to-back RPCs on the
same app interleave those sequences: gate run D (.228, 2026-07-09 04:28)
fired stop→start→restart on mempool; the start's member bring-up raced the
still-running stop's member shutdown — archy-mempool-db was stopped 2s
after it started and the stack finished with ZERO containers (tests 95/123/
124). A user double-clicking restart in the UI can do the same.
Workers now take a per-app tokio Mutex (fair/FIFO, keyed by the normalized
orchestrator app id) as their first await, so queued ops run in RPC arrival
order and the final state matches the last request. The RPC reply stays
immediate.
Known limit: container lists are still computed at RPC time (pre-lock), so
a stop landing mid-start can still return 'No containers found' — it can no
longer interleave/corrupt, only error. Reconciler/health-monitor restarts
don't take the lock yet (separate paths, same candidate follow-up).
Also from run D, filed separately: jellyfin/vaultwarden (legacy containers
on a quadlet node) stop fine but the package state sticks at 'stopping' —
'Failed to stop jellyfin.service: Unit not loaded' error path suspected.
Tests: runtime 9/9; archipelago builds clean.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The dd61a204 bind hardening published Bitcoin RPC on the archy-net gateway
10.89.0.1 — but rootlessport binds in the HOST netns, where that address
does not exist. First real deploy (.228, 2026-07-09) crash-looped
bitcoin-knots AND bitcoin-core the moment the gate's stop→start regenerated
the unit from the re-signed catalog: 'rootlessport listen tcp
10.89.0.1:8332: bind: cannot assign requested address', restart counter 132.
Hand-edits to the unit don't survive — the orchestrator regenerates it from
the signed catalog manifest within seconds.
- New archipelago_container::manifest::host_can_bind_publish_ip(): empty/
wildcard/loopback accepted without probing, anything else ephemeral-bind
probed. Applied at all three publish paths — quadlet from_manifest
(PublishPort), podman API create (host_ip), and the legacy -p string
table loop — each dropping the publish with a warn instead of taking the
container down. This neutralizes the bad binds already in the SIGNED
catalog, so nodes recover on binary deploy alone (no ceremony needed).
- Remove the gateway publishes from bitcoin-knots/-core manifests and the
legacy config.rs tables: verified on .228 that every in-node consumer
(lnd, btcpay/nbxplorer, fedimint, mempool-api) dials the container's
archy-net alias directly (bitcoin-knots:8332) and lnd uses RPC polling
(no ZMQ) — the gateway publish had zero consumers. Loopback-only RPC/ZMQ
(the approved LAN lockdown) stands; P2P 8333 stays public.
Catalog still ships the gateway binds until the next signing ceremony
regenerates it from these manifests; the guard makes that non-urgent.
Tests: container crate 65/65 (2 new guard tests), quadlet 40/40.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Second instance of the stop→start destruction class from tonight's gate
runs (.228): package.stop of a multi-container stack marks every member
user-stopped and quadlet-removes their containers; package.start's fallback
then resurrected only the bare package id, leaving the other members absent
AND user-stopped — indeedhub lost all 7 containers this way (mempool needed
its umbrella alias for the same reason).
- New stack_member_app_ids map (real manifest app ids, dependency order) —
used by the start fallback when a stack has no live containers, and by
package.restart instead of hard-failing "No containers found".
- orchestrator_uninstall_app_ids grows the missing indeedhub arm (same
reconciler-ghost rationale as the immich arm).
- do_orchestrator_package_start no longer aborts the whole sequence when one
member's unknown-app-id fallback fails (the #73/#74 wish) — collects and
continues.
NOTE: extends the known §G per-app-map debt (tracker updated) — the proper
home for stack membership is a manifest field; parked post-tag by design.
Tests: package:: 48/48 incl. new fallback + indeedhub-uninstall coverage.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
USER DECISION 2026-07-08: accept breaking external wallets pointed at
nodeIP:8332. The 0.0.0.0 publish exposed auth-only RPC (and unauthenticated
ZMQ 28332/28333 on the legacy path) to the whole LAN.
- New `bind` field on manifest port mappings (validated as an IP; the same
host port may repeat with distinct binds). Rendered as
PublishPort=<ip>:<host>:<container> in quadlet units and host_ip in the
podman API create — unbound ports render byte-identical to before, so no
fleet-wide false-drift wave.
- bitcoin-knots/-core manifests publish 8332 on 127.0.0.1 + 10.89.0.1 only.
In-node consumers (lnd, fedimint-gateway, btcpay/nbxplorer) are unaffected:
they dial host.archipelago / host.containers.internal, which the
orchestrator pins to the archy-net gateway 10.89.0.1. P2P 8333 stays public.
- Legacy config.rs port strings get the same treatment incl. ZMQ.
Tests: new quadlet bind-render + manifest bind-validation tests;
container:: suite 167/167, archipelago-container 63/63.
DEPLOY NOTE: PublishPort strings change for bitcoin containers → one
planned recreate per node; restart lnd afterwards (it caches the backend
IP — see the backend-recreate cascade tracker item). Catalog manifests for
bitcoin apps must be regenerated + re-signed for catalog-covered nodes.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Found by the .228 lifecycle gate (2026-07-08, first quadlet-mode run):
stop→start on the legacy `mempool` umbrella app id destroyed the mempool
deployment. Under quadlet, package.stop removes the containers; package.start
then failed with `unknown app_id: mempool` because load_manifests drops the
umbrella manifest whenever the three split-stack members are loaded — leaving
nothing to recreate from and the stack down.
Two fixes:
- prod_orchestrator start/stop/restart now resolve the historical
`mempool`/`mempool-web` id to the split members (archy-mempool-db,
mempool-api, archy-mempool-web) whenever the umbrella manifest was dropped —
the same alias install already implements ("installing mempool assembles the
split stack"). Restart falls back to start for members whose container/unit
is gone. Legacy umbrella-only nodes are unaffected (alias inactive while the
umbrella manifest is live).
- is_missing_container_error (3 sites) now recognizes podman 5.x's
`no such object` inspect phrasing, so a genuinely absent container is
classified as missing instead of surfacing as a hard inspect error.
Tests: 2 new alias tests + 2 classifier tests; prod_orchestrator suite 61/61
green.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
- mesh: verify_signature accepts a v2 preimage (t,v,ts,seq) alongside
legacy v1 (t,v,ts); signed_with_seq() is the v2 sender path, not yet
wired — senders stay v1 until the fleet verifies v2 (receivers
hard-drop bad sigs, so flipping send-side first would break
mixed-fleet alerts). Tests: v2 verify, v2 seq-tamper rejection,
v1 sign-then-set-seq compat.
- mesh listener: malformed radio-supplied DID shorter than the
'did🔑' prefix can no longer panic advert_name (slice -> .get()).
- auth: the pre-setup password123 dev login and the constant itself are
now #[cfg(debug_assertions)] — no release binary carries the bypass,
whatever its runtime config says.
- orchestrator: canned host-facts under #[cfg(test)] — awaiting real
subprocesses under tokio's paused test clock deadlocks against
auto-advanced timers (the old blocking detection only worked by never
yielding).
- drop two now-unused std::process::Command imports left by 4c75bb3d.
Tests: mesh 110/110 (incl. 2 new), api 68/68, container 159/159,
archipelago-container check clean.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
extract_client_ip took X-Real-IP/X-Forwarded-For from any request, so
a client talking to the backend directly (the FIPS peer listener, or
any non-proxy path) could rotate a fake IP per request and never trip
the login rate limiter. The accept loop now records the TCP peer
address in request extensions, and forwarded headers are honored only
when the connection itself is from loopback — where nginx overwrites
X-Real-IP with the real client address. Direct connections bucket
under their socket IP.
§C of the 1.8.0 hardening plan; 3 new unit tests cover the
loopback/direct/no-header matrix.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Catalog- and manifest-supplied image refs reached pull_image without
ever passing the RPC boundary's validator — a malicious catalog entry
or manifest could pull from an arbitrary registry. The allowlist now
lives in container::image_policy (the RPC check delegates to it) and
both orchestrator pull sites (install_fresh and
ensure_resolved_source_available) refuse refs that fail it.
The shared policy accepts trusted-registry refs and registry-less
Docker Hub shorthand (grafana/grafana etc., used by 8 shipped
manifests — a registry-less ref cannot name an attacker host), and
rejects explicit non-allowlisted hosts, shell metacharacters, and
malformed refs. §A of the 1.8.0 hardening plan.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Every production process spawn reachable from a tokio worker now uses
tokio::process: the install path's podman-port probe, the dependencies
disk check, factory-reset restart, config host-IP detection, the
orchestrator's host-facts helpers (resolve_dynamic_env and its call
sites made async to carry it through), and AutoRuntime's podman/docker
probes.
The FIPS transport probe is the special case: is_available() is a sync
trait method called from async route(), so instead of blocking ~50ms
on systemctl per stale-cache hit it now serves the cached value and
refreshes on a background thread (stale-while-revalidate) — bounded
staleness, zero stalled workers.
§C of the 1.8.0 hardening plan; container/transport/config/package
suites green.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
