diff --git a/docs/troubleshooting.md b/docs/troubleshooting.md index 075fcd10..c024c189 100644 --- a/docs/troubleshooting.md +++ b/docs/troubleshooting.md @@ -1,6 +1,6 @@ # Archipelago Troubleshooting Guide -This guide covers the 20 most common issues you may encounter with Archipelago, along with diagnostic commands and solutions. +This guide covers the most common issues you may encounter with Archipelago, along with diagnostic commands and solutions. ## Connection & Access @@ -111,9 +111,12 @@ df -h /var/lib/archipelago **Solutions**: - If the image is missing: reinstall the app from the Marketplace -- If disk is full: run disk cleanup from Settings, or manually `podman system prune` +- If disk is full: run disk cleanup from the **Server** page (`/server`), or manually `podman system prune` - If the container exits immediately: check logs for the root cause (usually missing config or permissions) -- Restart podman: `sudo systemctl restart podman` +- Restart the Podman socket. Archipelago runs **rootless** Podman as the + `archipelago` user, so this is a `--user` unit — `sudo systemctl restart podman` + would restart the unrelated root socket: + `systemctl --user restart podman.socket` ### 6. App shows "unhealthy" status @@ -146,20 +149,34 @@ podman logs --tail 100 | grep -i error # Check Bitcoin logs podman logs bitcoin-knots --tail 50 -# Check if Bitcoin is connected to peers -podman exec bitcoin-knots bitcoin-cli -datadir=/data getpeerinfo | grep -c '"addr"' +# Check if Bitcoin is connected to peers. +# The datadir inside the container is /home/bitcoin/.bitcoin, and the RPC +# credentials live in the generated /tmp/rpc.conf (the manifest's entrypoint +# writes it from the BITCOIN_RPC_USER/BITCOIN_RPC_PASS secrets) — bitcoin-cli +# needs both flags or it can't authenticate. +podman exec bitcoin-knots bitcoin-cli \ + -datadir=/home/bitcoin/.bitcoin -conf=/tmp/rpc.conf \ + getpeerinfo | grep -c '"addr"' # Check sync progress -podman exec bitcoin-knots bitcoin-cli -datadir=/data getblockchaininfo | grep -E "blocks|headers|verificationprogress" +podman exec bitcoin-knots bitcoin-cli \ + -datadir=/home/bitcoin/.bitcoin -conf=/tmp/rpc.conf \ + getblockchaininfo | grep -E "blocks|headers|verificationprogress" ``` **Solutions**: - Initial sync takes 1-7 days depending on hardware — be patient - Ensure the server has a stable internet connection -- Check disk space: Bitcoin requires 600GB+ for full chain +- Check disk space. The manifest picks the mode from the disk it's given: under + 1000 GB it runs **pruned** (`-prune=550`, a few GB); at 1000 GB or more it runs + a full `-txindex=1` archival node, which needs 600 GB+ and growing - If stuck: restart the container `podman restart bitcoin-knots` - If peers = 0: check firewall allows port 8333 outbound -- Add manual peers: edit bitcoin.conf to add `addnode=` entries +- Editing `bitcoin.conf` in the data directory has **no effect** — the + entrypoint runs bitcoind with an explicit `-conf=/tmp/rpc.conf` and logs + "ignoring legacy datadir bitcoin.conf". Flags come from the app manifest, so + persistent changes belong there (and, for catalog-covered apps, in the signed + catalog entry that overrides the on-disk manifest) ### 8. LND won't connect to Bitcoin @@ -254,7 +271,9 @@ df -h / - Ensure at least 2GB free disk space - If update hangs: wait 10 minutes, then restart the backend - Do NOT power off during an update — this can corrupt the system -- If system is in a bad state after failed update: boot from the USB installer and select "Repair" +- If the system is in a bad state after a failed update, recover over SSH — the + USB installer has no repair mode (its boot menu offers only "Install + Archipelago", "Install Archipelago (verbose)" and "Boot from local disk") ### 12. Server won't boot after update @@ -263,8 +282,11 @@ df -h / **Solutions**: - Wait 5 minutes — the first boot after update may take longer - If still unresponsive: connect a monitor/keyboard to check boot messages -- Try the recovery mode: boot from USB installer and select "Repair" -- As a last resort: reflash the USB and restore from backup +- If it's a bootloader problem rather than a disk problem, boot the USB and pick + "Boot from local disk" to chainload the installed system +- As a last resort: reinstall from USB and restore from backup. The installer is + interactive — it asks for the target disk and requires typing `yes` — so + booting it does not by itself destroy the existing install --- @@ -312,21 +334,35 @@ xrandr --query 2>/dev/null || echo "No display server" **Symptoms**: Settings shows "Tor: Not configured" or the .onion address is missing +Tor is **not** a container — it's the host's Debian `tor` package, running as +`debian-tor`. Archipelago never touches it directly: it stages a torrc and asks +`archipelago-tor-helper` (a `.path` unit watching +`/var/lib/archipelago/tor-config/tor-action`) to install it and restart Tor. + **Diagnosis**: ```bash -# Check Tor container -podman ps --filter "name=tor" -podman logs tor --tail 20 +# Check the host Tor service and the helper that drives it +sudo systemctl status tor +sudo journalctl -u archipelago-tor-helper --since "10 minutes ago" -# Check if Tor hostname file exists -cat /var/lib/archipelago/tor/hidden_service/hostname 2>/dev/null +# Is the SOCKS port up? (this is the liveness check the backend itself uses) +nc -z 127.0.0.1 9050 && echo "Tor SOCKS OK" + +# The readable hostname copy the backend actually reads +cat /var/lib/archipelago/tor-hostnames/archipelago + +# The hidden-service dir itself (root-owned 0700 — needs sudo) +sudo cat /var/lib/tor/hidden_service_archipelago/hostname 2>/dev/null \ + || sudo cat /var/lib/archipelago/tor/hidden_service_archipelago/hostname ``` **Solutions**: - Tor takes 30-60 seconds to bootstrap — wait and refresh -- If Tor container is stopped: start it from the Apps page -- Check that the Tor data directory exists and has correct permissions -- Restart Tor: `podman restart tor` +- If `/var/lib/archipelago/tor-hostnames/archipelago` is missing but the + hidden-service dir has a `hostname`, the readable copy didn't sync — the + helper's `sync-hostnames` action rewrites it +- Check that the Tor data directory exists and is owned by `debian-tor` +- Restart Tor: `sudo systemctl restart tor` ### 16. Peers can't reach my node @@ -334,11 +370,11 @@ cat /var/lib/archipelago/tor/hidden_service/hostname 2>/dev/null **Diagnosis**: ```bash -# Check if Tor is running (needed for peer connectivity) -podman ps --filter "name=tor" +# Check if Tor is running (the fallback transport for peer connectivity) +sudo systemctl status tor # Check your Tor address -cat /var/lib/archipelago/tor/hidden_service/hostname +cat /var/lib/archipelago/tor-hostnames/archipelago # Test connectivity from the server side curl -s http://localhost:5678/rpc/v1 \ @@ -347,10 +383,13 @@ curl -s http://localhost:5678/rpc/v1 \ ``` **Solutions**: -- Ensure Tor is running (required for peer-to-peer communication) +- Tor is the last-resort transport, not the only one: peering prefers mesh + radio, then LAN, then FIPS, and only falls back to Tor. A peer stuck on + "unreachable" with Tor healthy usually means the higher transports are all + down too — check the FIPS anchor first - Tor circuits can be slow — connections may take 30+ seconds - Share your correct .onion address with peers -- Both nodes must have Tor running and be on the same federation +- Both nodes must be on the same federation ### 17. DNS resolution issues @@ -370,7 +409,7 @@ podman exec bitcoin-knots nslookup seed.bitcoin.sipa.be ``` **Solutions**: -- Configure DNS from Settings > Network: try Cloudflare (1.1.1.1) or Google (8.8.8.8) +- Configure DNS from the **Server** page (`/server`): try Cloudflare (1.1.1.1) or Google (8.8.8.8) - If using custom DNS, verify the server addresses are correct - Restart networking: `sudo systemctl restart systemd-resolved` @@ -418,7 +457,7 @@ podman system df ``` **Solutions**: -- Run disk cleanup from Settings +- Run disk cleanup from the **Server** page (`/server`) - Remove unused app data: `podman system prune -a` (WARNING: removes all stopped containers and unused images) - Move Bitcoin data to external drive if chain data is too large - Check for large log files: `du -sh /var/log/*/ | sort -rh` @@ -560,7 +599,8 @@ If the system is completely unresponsive: 1. **Power cycle**: Hold power button for 10 seconds, then turn back on 2. **Wait 5 minutes**: Services take time to start, especially if containers need to recover 3. **SSH in**: If web UI is down but SSH works, restart services manually -4. **USB recovery**: Boot from the Archipelago USB installer and select "Repair" +4. **Chainload the installed system**: Boot the Archipelago USB and pick "Boot + from local disk" — this rules out a broken bootloader 5. **Clean install + restore**: As last resort, do a fresh install and restore from backup ### Collecting Diagnostic Information