Same kiosk.service conflict resolution as the previous merge (branch carried
rebased duplicates of the kiosk commits): CPUQuota=200% + main's memory tuning.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Conflict resolution: keep the branch's CPUQuota=200% (75% was the proven
cause of choppy HDMI audio — the file's own comment says so; the 75% on main
was an accidental regression bundled into an unrelated UI commit) and keep
main's newer MemoryMax=2800M/MemoryHigh=2200M tuning.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The tx1138 Gitea was retired as a release server 2026-06-13 and its registry
frontend is fully dead (500 on every /v2 manifest read, observed 2026-07-10).
Nothing may reference it anymore:
- registry.rs: no longer a default registry, no longer force-enabled on
load; saved configs are stripped of it on load (same one-time migration
treatment as the decommissioned Hetzner mirror), with a regression test.
- image_policy.rs: removed from TRUSTED_REGISTRIES — refs through the dead
host are now refused at the pull site (rejection test added).
- api/handler: dropped the legacy catalog-proxy fallback URL.
- .gitmodules: indeedhub submodule repointed to the OVH Gitea.
- scripts, image-recipe, app-catalog data, neode-ui strings, docs, and all
test fixtures repointed to 146.59.87.168:3000 (or neutral example hosts).
- image-versions.sh: ARCHY_REGISTRY_FALLBACK emptied (guarded consumers
skip it); reconcile-containers.sh candidate guard hardened.
The only remaining occurrences of the host string are the strip/reject
enforcement paths and their regression tests — the code that guarantees it
is never used again.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Debian's podman only Recommends catatonit; nodes installed before
install-podman.sh gained the dependency fail any init-enabled container
deploy (observed on shorty-s deploying sites via Portainer). Doctor
Fix 13 installs it via apt when absent.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
A store on a node whose LND has only unannounced channels mints BOLT11
invoices with no route hints when lightningPrivateRouteHints is off, and
external wallets fail with "no way to pay this invoice" (hit on shorty-s
2026-07-10 with a Blink payer). Hints cost nothing on public channels, so
the doctor now flips the flag on for every store on each run.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The netns-egress repair only stopped/started containers, which reuses the
existing netns — its holders (aardvark-dns, podman pause) survive, so a
missing pasta tap was never rebuilt. On shorty-s (2026-07-10) this cycled all
35 containers every timer run for ~an hour, bouncing bitcoind/LND/BTCPay the
whole time, with 'egress still broken after cycle' after every pass.
Now the cycle kills the netns holders, runs podman system migrate, and clears
/run/user/<uid>/containers/networks so the first container start recreates
pasta + aardvark-dns from scratch (the sequence that actually fixed the node).
A failure counter (/var/lib/archipelago/doctor-netns-cycle-failures) stops the
fleet-wide cycling after 3 consecutive failed rebuilds until egress is
observed healthy again.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Two btcpay-server manifest gaps found on shorty-s (2026-07-10):
- BTCPAY_PLUGINDIR=/datadir/Plugins: the image default is container-local,
so every container recreate silently wiped installed plugins (the
reinstall-the-nostr-plugin-after-every-restart symptom, plus a soft-restart
loop while an install was pending).
- BTCPAY_BTCLIGHTNING via optional secret_env btcpay-lnd-connection: the
manifest never wired the internal LND node. The secret is generated by the
daemon (see previous commit); nodes without LND skip it.
releases/app-catalog.json regenerated (embeds the manifest; origin-wins
overlay means nodes only pick this up from the published catalog). Signature
stripped by regeneration — run scripts/sign-catalog.sh before publishing.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Materialise /var/lib/archipelago/secrets/btcpay-lnd-connection
(type=lnd-rest;server=https://lnd:8080/;macaroon=<hex>;certthumbprint=<sha256>)
in ensure_app_secrets before btcpay-server env resolution.
The macaroon travels inline as hex because LND's datadir is owned by its
container subuid (100999) — bind-mounting it into btcpay's userns EACCESes.
Reads the macaroon via the existing read_file_as_root sudo fallback; pins the
TLS cert by DER SHA256 thumbprint and rewrites on cert rotation. No-op while
LND is absent (btcpay's secret_env entry is optional), and generation errors
log-and-continue so btcpay never fails to start over this.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
A secret_env entry with optional: true is skipped when its secret file is
missing, unreadable, or empty, instead of failing the whole resolution and
taking the app down. Needed for per-node integrations like btcpay's
internal-LND connection string: nodes without LND must still run btcpay.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Found live: after receiving a TollGate Cashu payment into the wallet,
the balance correctly showed the new sats (wallet.ecash-balance), but
the Transactions modal said "no transactions yet." Home.vue's
loadWeb5Status() only ever fetched lnd.gettransactions — ecash and
Fedimint activity (wallet.ecash-history, which already unifies both)
was never wired into walletTransactions at all, so no Cashu or
Fedimint receive could ever show up there regardless of how long you
waited.
Maps EcashTransaction into the existing (LND-shaped) WalletTransaction
interface rather than widening that type, and merges+sorts both lists
by timestamp.
tollgate-wrt keeps its own separate Cashu wallet on the router
(/etc/tollgate/wallet.db) — customer payments land there, never in
this node's own wallet. Its Lightning auto-payout is configured
independently in /etc/tollgate/identities.json, which is easy to
leave misconfigured or pointed at a placeholder address (found live:
both "owner" and "developer" identities on this deployment pointed at
the same unconfigured tollgate@minibits.cash default).
Rather than depend on getting that Lightning payout config right,
tollgate_sweep::sweep_once() periodically (every 5 min, via SSH)
checks the router's `tollgate wallet balance`, and if nonzero, runs
`tollgate wallet drain cashu` and receives the resulting token(s)
straight into the local wallet via the same path the "Receive ecash"
UI uses (wallet::ecash::receive_token) — bypassing Lightning payout
entirely. A no-op (Ok(0)) if no router is configured or it doesn't
have TollGate installed.
Found live: after fixing DHCP, the user could join the archipelago SSID,
saw a splash page, clicked "Continue", and got straight to the internet —
no payment step at all. NoDogSplash was serving its own bundled generic
click-to-continue splash page, whose "Continue" button calls NDS's
built-in auth handler directly and authorizes the client unconditionally.
TollGate's actual payment UI — a React SPA with a Cashu/QR token entry
flow — was already sitting on disk at
/etc/tollgate/tollgate-captive-portal-site (staged by the .ipk's data
payload during install), just never wired up as NoDogSplash's webroot.
install_captive_portal_symlink() mirrors upstream's own
90-tollgate-captive-portal-symlink uci-defaults script exactly: swap
/etc/nodogsplash/htdocs for a symlink to the real portal directory,
backing up any existing real directory first. Confirmed live that
setting `option webroot` directly instead (rather than the symlink
swap) makes NoDogSplash 500 on every request for reasons not fully
understood — the symlink approach is what's actually shipped/tested
upstream, so that's what this uses.
Also restores `authenticated_users 'allow all'` (the stock package
default our from-scratch nodogsplash.main section never carried over)
for correctness, even though this router's default-ACCEPT FORWARD
policy happens to make an empty list behave the same.
Found live again, in a different shape: after a later round of service
restarts (dnsmasq restart while debugging), br-tollgate desynced a
second time — but this time netifd's own status reported the interface
up with 192.168.99.1 assigned, while `ip -4 addr show br-tollgate` was
genuinely empty at the kernel level. dnsmasq logged "DHCP packet
received on br-tollgate which has no address" and silently dropped
every DISCOVER — clients associated to the archipelago SSID fine but
never got an IP.
A single blind ifdown/ifup (the previous fix) isn't trustworthy against
this netifd race — replace it with a loop that checks the actual kernel
address after each cycle and retries up to 5 times, failing loudly
(rather than silently leaving DHCP broken) if it never converges.
Found live: new clients on the archipelago SSID couldn't get an IP
address at all. configure()'s users_to_router rebuild replaced the
nodogsplash package's stock default list (DNS, DHCP, SSH/Telnet to the
router) with only our two TollGate-specific ports (2121, 2050) —
dropping `allow udp port 67`, so DHCP DISCOVER from an unauthenticated
client hit ndsRTR's default REJECT before ever reaching dnsmasq.
Carries over DNS (53) and DHCP (67/udp) from the stock default —
without them a client can't get an IP or resolve anything before
authenticating. Deliberately does not carry over SSH/Telnet (22/23):
the stock default exposes router shell access to every unauthenticated
device on the network, which isn't an appropriate default for a public
pay-as-you-go hotspot.
Two more issues found deploying the previous two commits live:
1. The nodogsplash package's own uci-defaults populate an anonymous
@nodogsplash[0] section pointed at br-lan (see install_and_stop's
doc comment). NoDogSplash runs one gateway instance per config
section, so this ran alongside our own nodogsplash.main instead of
being superseded by it — silently re-gating br-lan. configure() now
deletes it.
2. After `network restart` + `wifi down/up`, netifd intermittently
loses the race to claim br-tollgate as the wifi vif attaches
(reports up:false, DEVICE_CLAIM_FAILED) even though the bridge
device and member interface both exist correctly. NoDogSplash
refuses to start against an interface netifd hasn't brought up.
restart_services() now explicitly cycles just the tollgate
interface (ifdown/ifup) after the wifi restart.
Deploying the previous commit's fix live exposed a real bug: nodogsplash's
OpenWrt package postinst auto-enables and starts the service immediately
on install, using its stock default config — gatewayinterface=br-lan.
Since NoDogSplash only touches IPv4 iptables, this silently cut IPv4
(not IPv6) connectivity for anything on br-lan, including the admin
management box plugged into this router's LAN port, for the window
between install and our own configure step.
Split nodogsplash provisioning into install_and_stop() (runs first,
closes that window immediately) and configure() (runs after
wifi::provision_ssid has created br-tollgate, so gatewayinterface is
pointed at the isolated tollgate bridge instead of br-lan).
tollgate-wrt has no firewall/netfilter code of its own (confirmed via
strings on the binary and the upstream Go source) — it delegates all
MAC authorization and gate open/close to NoDogSplash via ndsctl.
Upstream's package declares +nodogsplash as a hard dependency, but
neither of our install paths pull it in: the opkg fast path only
resolves deps against a real feed, and the raw .ipk-extraction
fallback (used whenever the package isn't in a feed, and always on
ApkNative) skips dependency resolution entirely. Result: tollgate-wrt
ran, accepted payments, and tracked balances, but never actually
blocked unpaid clients — the static firewall zone just forwarded
everyone to WAN unconditionally.
Also fixes the config.json mismatch: tollgate-wrt reads
/etc/tollgate/config.json exclusively, never the tollgate.main.* UCI
keys we were writing — so mint/price changes through this project's
UI silently had no effect on what the daemon actually advertised.
- tollgate/nodogsplash.rs: install nodogsplash, configure it to gate
the dedicated br-tollgate bridge, open the pre-auth walled-garden
ports (2121 payment, 2050 portal).
- tollgate/wifi.rs: bind network.tollgate to a stable br-tollgate
bridge device (NoDogSplash needs a known interface name — the
driver-assigned name of a bare wifi vif isn't guaranteed); disable
IPv6 RA/DHCPv6 on it (NoDogSplash only manages IPv4 iptables, so
IPv6 would let clients bypass the portal entirely).
- tollgate/config.rs: apply_daemon_config() merges pricing/mint into
the real config.json instead of (only) UCI.
- opkg.rs: generic install_package() for standard feed packages under
either PkgManager mode.
- router.rs: upload_file() (SCP) for non-UCI config files.
Routes ALSA's "default" device through PulseAudio/PipeWire. Was a manual,
untracked fix living only on archy-x250-exp — a reprovision or fresh image
would silently lose it, same failure mode that already bit the GPU-flags
and CPUQuota fixes. Wired into deploy-to-target.sh (both the primary --live
path and the .198/.253 secondary-copy path) and deploy-tailscale.sh, mirroring
the existing 99-mesh-radio.rules udev sync pattern (diff-check, copy if changed).
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Both had regressed back to the pre-2026-06-28-incident state: GPU_FLAGS used
--enable-gpu-rasterization on any node with a GPU, and CPUQuota was 75%.
On archy-x250-exp (Intel HD 5500 under X11) this spins a dedicated GPU
process at 55-92% CPU (falls back to software compositing anyway), and the
75% cgroup quota throttled the kiosk unit ~81% of the time (nr_throttled
4856/6005) — the exact CPU-starvation signature documented as the choppy-
audio root cause. Restores the validated fix: --in-process-gpu,
--num-raster-threads=1, GpuRasterization disabled via --disable-features,
CPUQuota=200%. Verified on archy-x250-exp: no separate gpu-process, throttling
dropped to ~3% (nr_throttled 14/503).
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
The launcher's sudo -u archipelago invocation set DISPLAY/HOME but not
XDG_RUNTIME_DIR, so Chromium's audio backend couldn't find the PipeWire-Pulse
socket at /run/user/<uid>/pulse/native. It silently fell back to raw ALSA
"default", which also failed ("Connection refused"), producing no HDMI audio
at all with no visible error since --noerrdialogs suppresses it.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Routes ALSA's "default" device through PulseAudio/PipeWire. Was a manual,
untracked fix living only on archy-x250-exp — a reprovision or fresh image
would silently lose it, same failure mode that already bit the GPU-flags
and CPUQuota fixes. Wired into deploy-to-target.sh (both the primary --live
path and the .198/.253 secondary-copy path) and deploy-tailscale.sh, mirroring
the existing 99-mesh-radio.rules udev sync pattern (diff-check, copy if changed).
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Both had regressed back to the pre-2026-06-28-incident state: GPU_FLAGS used
--enable-gpu-rasterization on any node with a GPU, and CPUQuota was 75%.
On archy-x250-exp (Intel HD 5500 under X11) this spins a dedicated GPU
process at 55-92% CPU (falls back to software compositing anyway), and the
75% cgroup quota throttled the kiosk unit ~81% of the time (nr_throttled
4856/6005) — the exact CPU-starvation signature documented as the choppy-
audio root cause. Restores the validated fix: --in-process-gpu,
--num-raster-threads=1, GpuRasterization disabled via --disable-features,
CPUQuota=200%. Verified on archy-x250-exp: no separate gpu-process, throttling
dropped to ~3% (nr_throttled 14/503).
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
The launcher's sudo -u archipelago invocation set DISPLAY/HOME but not
XDG_RUNTIME_DIR, so Chromium's audio backend couldn't find the PipeWire-Pulse
socket at /run/user/<uid>/pulse/native. It silently fell back to raw ALSA
"default", which also failed ("Connection refused"), producing no HDMI audio
at all with no visible error since --noerrdialogs suppresses it.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
