Compare commits

..
103 Commits
Author SHA1 Message Date
archipelago cedfbb2b07 docs: record public release verification and storage follow-up 2026-10-05 19:52:07 -04:00
archipelago 7ae812e3f6 chore: publish verified 1.9.0-alpha OTA and app catalog 2026-10-05 19:13:08 -04:00
archipelago bc386965da docs: require FIPS for distributed media streaming 2026-10-05 18:59:49 -04:00
archipelago 7abd04a7f6 docs: record corrected signed OTA reboot acceptance
Demo images / Build & push demo images (push) Failing after 39s
2026-10-05 18:56:17 -04:00
archipelago 441733646f chore: stage signed 1.9.0-alpha release metadata 2026-10-05 18:50:26 -04:00
archipelago ccf823590a fix: reject stale first-boot scripts in OTA release payloads 2026-10-05 18:44:46 -04:00
archipelago d9775ac144 docs: require headless node qualification and IndeeHub app delivery 2026-10-05 18:25:18 -04:00
archipelago 0925c58821 docs: track Fleet metrics and secure FIPS performance work 2026-10-05 18:22:51 -04:00
archipelago 2d27f9c475 docs: track node availability and navigation latency follow-ups 2026-10-05 18:21:02 -04:00
archipelago 868e46fac9 docs: queue companion Fleet and AIUI setup follow-ups 2026-10-05 18:18:58 -04:00
archipelago 2aa77d1b7b docs: record installer acceptance and post-release work 2026-10-05 18:18:11 -04:00
archipelago a6b9e7ab49 fix: preserve apps and diagnostics when first-boot setup retries 2026-10-05 17:21:08 -04:00
archipelago a902cc84fe test: qualify built demo images through mobile and upload flows 2026-10-05 16:48:52 -04:00
archipelago 157c9ec055 fix: keep public demo online during optional upstream DNS outages 2026-10-05 16:15:41 -04:00
archipelago 415826f0a6 fix: package the qualified UI in release ISOs 2026-10-05 16:02:48 -04:00
archipelago 69857c4ace fix: require explicit dispatch for public demo redeployment 2026-10-05 15:42:46 -04:00
archipelago e6e46a1427 fix: build demo AIUI from the reviewed source revision 2026-10-05 15:34:41 -04:00
archipelago e0b2181ae9 fix: isolate NPM upstream TLS sessions across public domains 2026-10-05 15:26:28 -04:00
archipelago 446fa7b7fd fix: retain management guard through legacy runtime install and rollback 2026-10-05 15:08:42 -04:00
archipelago ba8b1f29b2 docs: record accepted companion and Yaya deployment 2026-10-05 14:44:03 -04:00
archipelago b8266c2872 fix: support resumable Cloud uploads in the public demo 2026-10-05 14:41:28 -04:00
archipelago 5aa74d0513 fix: qualify mobile Cloud viewer and companion downloads 2026-10-05 14:41:08 -04:00
archipelago daac47cac4 fix: harden node upgrades and prepare 1.9.0-alpha 2026-10-05 12:43:49 -04:00
archipelago 138a541d01 docs: make ngit canonical and gate mirror publication 2026-10-05 11:47:12 -04:00
archipelago 833c939220 docs: make alpha status and funds risk prominent 2026-10-05 10:26:10 -04:00
archipelago 2c1bcacf0a Test Nostr encryption compatibility and forged relay messages
Demo images / Build & push demo images (push) Failing after 52s
2026-10-05 10:00:41 -04:00
archipelago f1d0092e57 Validate app owner keys and match identity picker whitespace rules 2026-10-05 09:31:46 -04:00
archipelago 7dfb0e0013 Merge opt-in app owner identity placeholder
nevent1qqs9d76qm6f5xj2vrtjfnkqz5exrc8r0s9zev4f672kqyd0wjh7wwvqpz3mhxue69uhhyetvv9ujumn8d96zuer9wcx2tvaw
2026-10-05 09:29:18 -04:00
archipelago 775d7b9877 Merge Nostr 0.44 security fixes
nevent1qqsgj7l3gewl20m6xxxeu89zsmjy9prvc9g8ggkr6cysa5p9tc3hc2gpz3mhxue69uhhyetvv9ujumn8d96zuer9wcyr3wqc
2026-10-05 09:29:08 -04:00
TheCryptoDonkey c18ebd7f5b fix: update nostr to 0.44.7 and nostr-relay-pool to 0.44.3
Patch releases within 0.44 that clear eleven RustSec advisories against
the versions in the lock: RUSTSEC-2026-0216, -0219, -0224 to -0232.
They cover NIP-04 and NIP-44 decryption panics and resource exhaustion,
Debug output exposing NIP-46 and NIP-60 credentials, and the relay pool's
handling of unverified events. No other package changes.
2026-10-03 11:11:06 +02:00
TheCryptoDonkey 494d248356 feat: add NODE_IDENTITY_PUBKEYS derived-env placeholder
Lets an app grant the node's users owner rights, e.g. a Blossom server's
allowed uploaders. The value is the Nostr keys of the identities the app
identity picker offers for NIP-07 signing, chosen by the same rule as
NostrIdentityPicker.vue, so the node's own appliance identity is never
included. It is resolved only for manifests that template it, and an
empty set is an error rather than an empty owner list.

identity.list now shares its is_node test with the new helper.
2026-10-03 11:10:29 +02:00
archipelago 3acefecc24 Serve the shared Mempool explorer through the Angor indexer 2026-10-01 16:03:19 -04:00
archipelago 19c49c6605 Remove remaining blur from transaction filter container 2026-10-01 14:48:51 -04:00
archipelago d6e0c142c6 Return retryable payment status errors and record NPM release gate 2026-10-01 14:24:24 -04:00
archipelago 57729f8e18 Record candidate deployment and corrected payment evidence 2026-10-01 12:15:41 -04:00
archipelago 4fdadad89d Record final build status and live resource checks 2026-10-01 10:46:44 -04:00
archipelago f4d3455496 Fix paid-file recovery, app lifecycle regressions and wallet controls
Demo images / Build & push demo images (push) Failing after 1m10s
2026-10-01 10:31:55 -04:00
archipelago 227174e541 docs: close 1.8.22 publication with verified Git and ngit assets 2026-10-01 06:19:12 -04:00
archipelago 2e72b38778 release: publish verified signed 1.8.22 OTA and app catalog 2026-10-01 06:15:32 -04:00
archipelago 0be7aee49d docs: record final 1.8.22 OTA and ISO acceptance 2026-09-30 19:50:01 -04:00
archipelago 6d5f3ffb85 fix: select NPM admin port regardless of binding order
Demo images / Build & push demo images (push) Failing after 34s
2026-09-30 18:24:35 -04:00
archipelago d1bc1273d4 fix: replace cached container doctor before ISO first boot 2026-09-30 17:52:30 -04:00
archipelago 96fb5a4f19 fix: prevent stale snapshots resurrecting orphaned dashboards
Demo images / Build & push demo images (push) Failing after 36s
2026-09-30 17:45:18 -04:00
archipelago f91c1f33db fix: keep apps running when network diagnostics fail 2026-09-30 17:34:11 -04:00
archipelago 02b840f2d1 chore: prepare 1.8.22-alpha release candidate
Demo images / Build & push demo images (push) Failing after 36s
2026-09-30 16:45:43 -04:00
archipelago f992780957 fix: probe Angor IPv4 health endpoint inside the actual image 2026-09-30 16:40:16 -04:00
archipelago c82c1eee98 fix: prevent NPM tunnel collisions and false app health restarts 2026-09-30 16:30:40 -04:00
archipelago 2992443d5d docs: record verified NPM tunnel port conflict and node repair 2026-09-30 16:12:04 -04:00
archipelago 259c353147 Clear completed candidate deployment instructions 2026-09-30 13:46:50 -04:00
archipelago 1724ea05d1 Show readiness reason first and record live dashboard acceptance
Demo images / Build & push demo images (push) Failing after 45s
2026-09-30 13:46:16 -04:00
archipelago c1e20a71ae Check companion dashboards and omit headless UI waiting messages
Demo images / Build & push demo images (push) Failing after 37s
2026-09-30 13:28:37 -04:00
archipelago bf56956790 Record UI acceptance and remaining normal-startup release gate 2026-09-30 12:57:35 -04:00
archipelago 2f1a3ade07 Promote runtime manifests before starting app reconciliation 2026-09-30 12:50:46 -04:00
archipelago ef8254272c Name waiting apps, align card actions, and update Angor icon
Demo images / Build & push demo images (push) Failing after 39s
2026-09-30 12:43:53 -04:00
archipelago d50be13232 Normalize Mempool frontend aliases in restored app inventory 2026-09-30 12:41:15 -04:00
archipelago 439b55a236 Use manifest names for new services and document release acceptance
Demo images / Build & push demo images (push) Failing after 38s
2026-09-30 12:20:29 -04:00
archipelago 5ab65f7581 Preserve apostrophes in Quadlet commands and record funded acceptance 2026-09-30 12:08:35 -04:00
archipelago 169bf77de6 Add headless Angor services and shared-index install guard
Demo images / Build & push demo images (push) Failing after 43s
2026-09-30 11:52:19 -04:00
archipelago 7c4169867c docs: consolidate release scope and record migration recovery checks 2026-09-30 10:52:41 -04:00
archipelago acf544500f fix(apps): preserve state across runtime repairs and restore Gitea SSH 2026-09-30 10:46:38 -04:00
archipelago 7d767c8cb0 fix(catalog): gate network migration manifests on backup support 2026-09-30 10:08:56 -04:00
archipelago eb3ccfa00b Merge branch 'fix/gitea-portainer-20260930' 2026-09-30 09:57:49 -04:00
archipelago eda28c4cd6 fix(portainer): repair same-node Git routing with recoverable network migration 2026-09-30 09:57:25 -04:00
archipelago d69e845216 Merge remote-tracking branch 'origin/main'
Demo images / Build & push demo images (push) Failing after 1m10s
2026-09-30 09:32:20 -04:00
archipelago dc962c53b0 docs: record live lifecycle acceptance and next release blockers 2026-09-30 09:31:18 -04:00
archipelago 6ac26f637c fix(apps): preserve lifecycle state and wait for usable launch endpoints 2026-09-30 09:10:30 -04:00
archipelago 27d81e956d fix(installer): ship all app build contexts and refresh GitWorkshop dependencies 2026-09-30 09:09:21 -04:00
archipelago eb39391223 fix(ui): keep Bitcoin version choices readable in kiosk 2026-09-30 09:09:21 -04:00
chaum b02ba4100d Merge pull request 'fix(files): save purchased files atomically with rootless ownership' (#162) from fix/filebrowser-purchase-filing into main 2026-09-30 12:58:36 +00:00
chaum 3daea6623b Merge pull request 'fix(ecash): prevent paid-download replay and read failures after charging' (#161) from fix/ecash-paid-download-v2-keyset into main 2026-09-30 12:58:33 +00:00
archipelago d42f448e31 docs: close verified 1.8.21 OTA and ISO publication 2026-09-30 07:46:05 -04:00
archipelago 1566f1bb00 docs: record tested paid-download PRs for next release 2026-09-30 07:34:18 -04:00
archipelago 0677924a64 Merge current main and make purchase filing atomic under concurrent writes 2026-09-30 07:26:51 -04:00
archipelago 971d477795 Merge current main and harden paid-download delivery 2026-09-30 07:25:47 -04:00
archipelago f12042f194 docs: track X250 kiosk Bitcoin version selector regression 2026-09-30 07:01:57 -04:00
archipelago e7cf336665 chore: publish release v1.8.21-alpha
Demo images / Build & push demo images (push) Failing after 37s
2026-09-30 05:55:13 -04:00
archipelago 8ca20de82e release: prepare signed 1.8.21-alpha OTA 2026-09-30 05:51:16 -04:00
archipelago 1fa654cb6a docs: record 1.8.21 artifact and two-node release acceptance 2026-09-30 05:39:26 -04:00
archipelago c993d9dd0d fix(lnd): require observed Bitcoin lifecycle change before dependency restart 2026-09-30 05:16:17 -04:00
archipelago 33d2b3ce60 fix(containers): preserve graceful shutdown through Quadlet and prepare 1.8.21 2026-09-30 04:59:30 -04:00
archipelago c7ce35bd43 chore: publish release v1.8.20-alpha
Demo images / Build & push demo images (push) Failing after 1m31s
2026-09-30 04:32:43 -04:00
archipelago ad1d71a462 Prepare signed v1.8.20-alpha release and record operator acceptance 2026-09-30 04:27:02 -04:00
ssmithxandClaude Opus 5.5 33477f284b fix(files): file purchased content into FileBrowser folders again
Every paid download logged "filing into filebrowser/Music/... failed
(non-fatal): Permission denied". The purchase played in-app but never
appeared in Files. FileBrowser's folders belong to its rootless container
range (host uid 100000, mode 755). This service is host uid 1000, outside
that range, so it can read them but not create files in them.

New container::filebrowser::save_new_file:
- Writes directly when the folder allows it.
- Otherwise writes through `podman unshare`, where that uid range is
  ours: to a temp file, then chowned to the folder's owner, set to 0644,
  and hard-linked into place. FileBrowser never sees a partial file and an
  existing file is never replaced. A missing folder is created and given
  its parent's owner. No sudo.
- Keeps the "name (2).ext" de-duplication the RPC did inline.

Checked the unshare script on amishparadise in a scratch folder owned
like FileBrowser's: new folder + file OK, owner/mode right, no clobber,
no temp file left, and the service can read the result.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-09-29 22:36:31 +00:00
ssmithxandClaude Opus 5.5 03e38d1ca3 test: regression tests for the paid-download fixes
- mint_client: a stub mint shows swap() sends the full v2 keyset id when
  given a cashuB short id, and leaves complete v1/v2 ids unchanged.
- fips::dial: the single-delivery decisions are now small functions
  (fips_answer_is_final, fips_retryable). Tests cover them and, against a
  silent local peer, check that a single-delivery request isn't resent
  after a timeout while an ordinary one still is.
- content_server: an unreadable paid file returns Unavailable before the
  payment gate runs, and a readable one still returns 402. Also covers
  ensure_readable's grant/reopen behaviour. The podman grant is replaced
  by a refusal under cfg(test) so results don't depend on the host.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-09-29 20:12:16 +00:00
archipelago bded929812 Record validated OTA candidate and remaining release gates 2026-09-29 15:47:22 -04:00
archipelago 3612458e86 Handle empty recorded calls in install regression assertion 2026-09-29 15:38:49 -04:00
archipelago 8d9fad1749 Require Bitcoin version and pruning selection from the App Store 2026-09-29 15:37:05 -04:00
archipelago d25ed492c9 Keep Bitcoin pruning explanation below desktop install controls 2026-09-29 15:27:00 -04:00
archipelago 1f9abefc35 Isolate backend tests from live node wallets and services 2026-09-29 15:15:51 -04:00
archipelago b634f41a1c Complete paid-file caching and deliver LND waiting UI to existing nodes 2026-09-29 14:59:08 -04:00
archipelago 0f85f588fb Fix Cashu file redemption and Bitcoin-dependent wallet readiness 2026-09-29 14:42:44 -04:00
ssmithxandClaude Opus 5.5 e5fc99d66c fix(content): never charge for a file the seller can't serve or replay a spent token
After the keyset-id fix, a Minibits paid download still failed and the
buyer lost the sats. What happened, 2026-09-29, amishparadise:

1. The seller redeemed the token, then failed to read the file. It was a
   FileBrowser upload owned by the container subuid (100999) with mode
   0640. The handler mapped that Err to 404.
2. The buyer's FIPS dial treats 404 as "fall back to Tor" and resent the
   request with the same, now spent, token. The seller answered 402, and
   the buyer showed "seller doesn't accept your Cashu mint".

Fixes:
- serve_content checks the file is readable before the paid gate. If it
  isn't, it grants read with `podman unshare chmod a+r`, which matches
  the other shared files. If that also fails it returns Unavailable (503)
  without taking payment.
- The content handler returns 500 on internal errors and logs them,
  instead of a silent 404.
- New PeerRequest::single_delivery(), used for the paid download: the
  FIPS answer is final, FIPS retries only when it never connected, and
  there's no Tor replay once the request may have been delivered.
- The buyer shows the seller's error text for non-402 failures.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-09-29 18:42:20 +00:00
ssmithxandClaude Opus 5.5 8b74803290 fix(ecash): repair short v2 keyset ids on every swap, not just receive
Paid cloud downloads paid with Minibits ecash were always rejected. The
buyer sends a cashuB token, which carries NUT-02 v2 keyset ids in their
8-byte short form. Minibits rotated its active keyset to a v2 id, and the
seller's verify_and_receive_payment called MintClient::swap directly,
skipping the short->full id repair that only receive_token applied. The
mint answered 422 ("ID length invalid"). The buyer then showed the
misleading "seller doesn't accept your Cashu mint" hint.

Move the repair into swap() so every caller is covered: payment verify,
streaming gate, send change, and cross-mint swaps.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-09-29 15:47:23 +00:00
archipelago 540639d2c1 chore: publish release v1.8.19-alpha 2026-09-28 13:21:51 -04:00
archipelago 562871b1ce chore: prepare signed release v1.8.19-alpha 2026-09-28 13:18:34 -04:00
archipelago cca3f8bfcd docs: include AIUI packaging fix in v1.8.19 release notes
Demo images / Build & push demo images (push) Failing after 44s
2026-09-28 13:13:31 -04:00
archipelago 89c08be712 fix(aiui): match host color scheme for iframe transparency
Demo images / Build & push demo images (push) Failing after 56s
2026-09-28 12:37:40 -04:00
archipelago b4ecf86c13 chore: stage v1.8.19-alpha version bump 2026-09-28 12:33:59 -04:00
archipelago b14fe78306 fix(aiui): expose host wallpaper and package fresh production builds 2026-09-28 12:32:18 -04:00
archipelago 3f0c1038c3 docs: add v1.8.19 release notes to settings 2026-09-28 12:23:57 -04:00
archipelago 1fbefce6df docs: add v1.8.19-alpha release notes 2026-09-28 12:23:05 -04:00
archipelago 63cb68451a fix(aiui): keep embedded chat background transparent 2026-09-22 05:04:59 -04:00
archipelago 17cfebbe26 chore: publish release v1.8.18-alpha 2026-09-20 11:49:39 -04:00
294 changed files with 26063 additions and 2561 deletions
+7 -1
View File
@@ -4,7 +4,13 @@
# Allow neode-ui (frontend + mock backend + docker configs) # Allow neode-ui (frontend + mock backend + docker configs)
!neode-ui/ !neode-ui/
# Allow demo assets (AIUI pre-built dist) !aiui/
aiui/**/node_modules
aiui/**/dist
aiui/**/.turbo
aiui/**/.build-aiui-last-*
# Allow curated demo assets
!demo/ !demo/
# Allow the Bitcoin UI + ElectrumX UI mock shells (served from /docker/*) # Allow the Bitcoin UI + ElectrumX UI mock shells (served from /docker/*)
+6 -1
View File
@@ -17,6 +17,9 @@ on:
branches: [main] branches: [main]
paths: paths:
- 'neode-ui/**' - 'neode-ui/**'
- 'aiui/**'
- 'scripts/build-aiui.sh'
- '.dockerignore'
- 'docker-compose.demo.yml' - 'docker-compose.demo.yml'
- '.gitea/workflows/demo-images.yml' - '.gitea/workflows/demo-images.yml'
workflow_dispatch: workflow_dispatch:
@@ -65,10 +68,12 @@ jobs:
push: true push: true
build-args: | build-args: |
VITE_DEMO=1 VITE_DEMO=1
SOURCE_REVISION=${{ github.sha }}
tags: | tags: |
${{ vars.DEMO_REGISTRY }}/archy-demo-web:demo ${{ vars.DEMO_REGISTRY }}/archy-demo-web:demo
${{ vars.DEMO_REGISTRY }}/archy-demo-web:${{ github.sha }} ${{ vars.DEMO_REGISTRY }}/archy-demo-web:${{ github.sha }}
- name: Trigger Portainer redeploy - name: Trigger Portainer redeploy
if: ${{ success() && secrets.PORTAINER_WEBHOOK != '' }} # Source pushes prepare images; public deployment is an explicit post-release action.
if: ${{ success() && github.event_name == 'workflow_dispatch' && secrets.PORTAINER_WEBHOOK != '' }}
run: curl -fsS -X POST "${{ secrets.PORTAINER_WEBHOOK }}" run: curl -fsS -X POST "${{ secrets.PORTAINER_WEBHOOK }}"
+6 -1
View File
@@ -17,6 +17,9 @@ on:
branches: [main] branches: [main]
paths: paths:
- 'neode-ui/**' - 'neode-ui/**'
- 'aiui/**'
- 'scripts/build-aiui.sh'
- '.dockerignore'
- 'docker-compose.demo.yml' - 'docker-compose.demo.yml'
- '.github/workflows/demo-images.yml' - '.github/workflows/demo-images.yml'
workflow_dispatch: workflow_dispatch:
@@ -65,10 +68,12 @@ jobs:
push: true push: true
build-args: | build-args: |
VITE_DEMO=1 VITE_DEMO=1
SOURCE_REVISION=${{ github.sha }}
tags: | tags: |
${{ vars.DEMO_REGISTRY }}/archy-demo-web:demo ${{ vars.DEMO_REGISTRY }}/archy-demo-web:demo
${{ vars.DEMO_REGISTRY }}/archy-demo-web:${{ github.sha }} ${{ vars.DEMO_REGISTRY }}/archy-demo-web:${{ github.sha }}
- name: Trigger Portainer redeploy - name: Trigger Portainer redeploy
if: ${{ success() && secrets.PORTAINER_WEBHOOK != '' }} # Source pushes prepare images; public deployment is an explicit post-release action.
if: ${{ success() && github.event_name == 'workflow_dispatch' && secrets.PORTAINER_WEBHOOK != '' }}
run: curl -fsS -X POST "${{ secrets.PORTAINER_WEBHOOK }}" run: curl -fsS -X POST "${{ secrets.PORTAINER_WEBHOOK }}"
+50
View File
@@ -21,3 +21,53 @@ While its status is OPEN:
This priority comes from the user's explicit instruction on 2026-09-15. It remains This priority comes from the user's explicit instruction on 2026-09-15. It remains
in effect across sessions until the documented acceptance criteria are met or the in effect across sessions until the documented acceptance criteria are met or the
user explicitly changes it. user explicitly changes it.
## Unit tests on a live node
Run backend unit tests through `scripts/test-backend-isolated.sh`. Do not run
unrestricted `cargo test` on a node with installed apps: older mocked-runtime
tests still reached real service commands. The runner isolates wallet data,
service buses, container storage, networking, and process IDs. Compilation with
`cargo test --no-run` is safe. Keep separately authorized live checks explicit.
## Active release regression checklist
Before resuming release work, read
`docs/post-1.8.22-regressions-20261001.md` and retain its unfinished tasks.
The operator requested that every reported issue be tracked, fixed and tested
before another OTA/ISO. Keep source/unit-test results separate from actual-node
acceptance. In particular, paid-file recovery must not send another payment,
and app cleanup must preserve wallets, persistent data and uninstall decisions.
Do not mark the new paid-file incident resolved merely because the earlier
Framework LND startup incident was closed.
## Gitea and ngit mirror parity
Nostr Git (`ngit`) is the canonical contribution and review platform. Gitea
(`origin`) mirrors accepted code on `main` and release tags. Both are required
publication mirrors; duplicate PRs and proposal branches on Gitea are not required.
For every change, including fixes and release preparation:
- Review and merge once. Push the exact same resulting commits to both mirrors;
never independently squash, rebase or merge the same change on each platform.
- Open new contributions and PRs on ngit; review and merge there, then mirror the
exact accepted main commits to Gitea. Record the ngit proposal and resulting
merge commit in the release ledger. Existing Gitea PRs must be reviewed and
explicitly linked to their ngit replacement or accepted result before closing;
do not abandon contributions or mark unmerged changes as merged. PR numbers,
reviews and discussions remain platform-specific; matching Git refs does not
prove their synchronization.
- Push main and release tags to both mirrors. Preserve commit history
and annotated tag objects/signatures. Do not resolve drift by force pushing,
deleting remote refs, or rewriting published history without explicit approval.
- After publishing source, run `python3 scripts/check-git-mirrors.py --local`.
Include each additional shared branch or release tag with repeated `--ref`
arguments (full `refs/heads/...` or `refs/tags/...` names).
- Before OTA, catalog or ISO publication, require matching reviewed local and
remote main and release tag refs, and record ngit PR dispositions in the
release acceptance ledger. A failed push, unavailable mirror, missing ref or
mismatch blocks publication; never describe a partial push as synchronized.
Run `--all` for a complete advertised branch/tag audit; a main-only pass must
never be described as full historical mirror parity. Proposal-only branches
may intentionally differ. Existing unrelated drift
must be inventoried explicitly rather than silently overwritten.
+5 -2
View File
@@ -11,8 +11,8 @@ android {
applicationId = "com.archipelago.app" applicationId = "com.archipelago.app"
minSdk = 26 minSdk = 26
targetSdk = 35 targetSdk = 35
versionCode = 52 versionCode = 54
versionName = "0.5.32" versionName = "0.5.34"
vectorDrawables { vectorDrawables {
useSupportLibrary = true useSupportLibrary = true
@@ -142,6 +142,9 @@ tasks.matching {
}.configureEach { dependsOn("buildRustArm64") } }.configureEach { dependsOn("buildRustArm64") }
dependencies { dependencies {
testImplementation("junit:junit:4.13.2")
testImplementation("com.squareup.okhttp3:mockwebserver:4.12.0")
testImplementation("org.robolectric:robolectric:4.14.1")
val composeBom = platform("androidx.compose:compose-bom:2024.05.00") val composeBom = platform("androidx.compose:compose-bom:2024.05.00")
implementation(composeBom) implementation(composeBom)
@@ -0,0 +1,179 @@
package com.archipelago.app.ui.screens
import android.app.Activity
import android.content.Intent
import android.net.Uri
import android.provider.DocumentsContract
import android.webkit.CookieManager
import android.webkit.DownloadListener
import android.webkit.URLUtil
import android.widget.Toast
import androidx.activity.compose.rememberLauncherForActivityResult
import androidx.activity.result.contract.ActivityResultContracts
import androidx.compose.foundation.layout.Column
import androidx.compose.material3.AlertDialog
import androidx.compose.material3.LinearProgressIndicator
import androidx.compose.material3.Text
import androidx.compose.material3.TextButton
import androidx.compose.runtime.*
import androidx.compose.ui.platform.LocalContext
import kotlinx.coroutines.*
import okhttp3.Call
import okhttp3.HttpUrl.Companion.toHttpUrlOrNull
import okhttp3.OkHttpClient
import okhttp3.Request
import java.io.IOException
import java.io.OutputStream
import java.util.concurrent.TimeUnit
internal data class WebDownload(val url: String, val userAgent: String, val cookies: String, val name: String, val mime: String)
/** Only the starting origin receives its WebView cookies, even across redirects. */
internal fun streamWebDownload(
download: WebDownload,
output: OutputStream,
client: OkHttpClient,
onCall: (Call) -> Unit = {},
checkCancelled: () -> Unit = {},
onProgress: (Long, Long) -> Unit = { _, _ -> },
): Long {
val transport = client.newBuilder().followRedirects(false).followSslRedirects(false).build()
val original = download.url.toHttpUrlOrNull() ?: throw IOException("Unsupported download link")
var url = original
var redirects = 0
while (true) {
checkCancelled()
if (url.username.isNotEmpty() || url.password.isNotEmpty()) throw IOException("Unsupported download link")
val request = Request.Builder().url(url).header("User-Agent", download.userAgent)
if (url.scheme == original.scheme && url.host == original.host && url.port == original.port && download.cookies.isNotBlank()) {
request.header("Cookie", download.cookies)
}
val call = transport.newCall(request.build())
onCall(call)
call.execute().use { response ->
if (response.code in listOf(301, 302, 303, 307, 308)) {
if (++redirects > 5) throw IOException("Too many download redirects")
val next = response.header("Location")?.let { url.resolve(it) } ?: throw IOException("Invalid download redirect")
if (url.isHttps && !next.isHttps) throw IOException("Insecure download redirect blocked")
url = next
} else {
if (response.code == 401 || response.code == 403) throw IOException("Sign in to the node again, then retry the download")
if (!response.isSuccessful) throw IOException("Download failed (HTTP ${response.code})")
if (response.header("Content-Type")?.substringBefore(';')?.trim()?.lowercase() == "text/html" &&
download.mime != "text/html" && !download.name.endsWith(".html", true) && !download.name.endsWith(".htm", true)) {
throw IOException("Sign in to the node again, then retry the download")
}
val body = response.body ?: throw IOException("The download was empty")
val total = body.contentLength()
var written = 0L
body.byteStream().use { input ->
val buffer = ByteArray(64 * 1024)
var lastUpdate = 0L
while (true) {
checkCancelled()
val count = input.read(buffer)
if (count == -1) break
output.write(buffer, 0, count)
written += count
val now = System.nanoTime()
if (now - lastUpdate > 100_000_000L) { onProgress(written, total); lastUpdate = now }
}
}
if (total >= 0 && written != total) throw IOException("Download interrupted; please retry")
onProgress(written, total)
return written
}
}
}
}
/** Uses the system Save dialog: no broad storage permission and no external browser login. */
@Composable
internal fun rememberWebViewDownloads(): DownloadListener {
val context = LocalContext.current
val scope = rememberCoroutineScope()
var pending by remember { mutableStateOf<WebDownload?>(null) }
var active by remember { mutableStateOf<WebDownload?>(null) }
var progress by remember { mutableStateOf<Pair<Long, Long>>(0L to -1L) }
var failure by remember { mutableStateOf<String?>(null) }
var job by remember { mutableStateOf<Job?>(null) }
val currentCall = remember { java.util.concurrent.atomic.AtomicReference<Call?>(null) }
val client = remember {
OkHttpClient.Builder().followRedirects(false).followSslRedirects(false)
.connectTimeout(20, TimeUnit.SECONDS).readTimeout(60, TimeUnit.SECONDS).build()
}
fun cancel() { job?.cancel(); currentCall.getAndSet(null)?.cancel() }
DisposableEffect(Unit) { onDispose { currentCall.getAndSet(null)?.cancel() } }
val save = rememberLauncherForActivityResult(ActivityResultContracts.StartActivityForResult()) { result ->
val download = pending
pending = null
val uri = result.data?.data
if (result.resultCode != Activity.RESULT_OK || uri == null || download == null) return@rememberLauncherForActivityResult
job = scope.launch {
active = download
progress = 0L to -1L
var complete = false
try {
withContext(Dispatchers.IO) {
val task = currentCoroutineContext()
context.contentResolver.openOutputStream(uri, "w")?.use { output ->
streamWebDownload(download, output, client,
onCall = { call -> currentCall.set(call); if (!task.isActive) call.cancel() },
checkCancelled = { task.ensureActive() },
onProgress = { done, total -> scope.launch { progress = done to total } })
} ?: throw IOException("Unable to open the selected destination")
}
complete = true
Toast.makeText(context, "Download complete: ${download.name}", Toast.LENGTH_LONG).show()
} catch (error: CancellationException) {
throw error
} catch (error: Exception) {
if (currentCoroutineContext().isActive) {
// Do not expose authenticated URLs or request headers in UI/logs.
failure = when {
error is javax.net.ssl.SSLException -> "The server certificate could not be verified."
error is IOException && error.message?.startsWith("Sign in") == true -> error.message
else -> "Download failed. Check your connection and available storage, then try again."
}
}
} finally {
currentCall.getAndSet(null)?.cancel()
if (!complete) withContext(NonCancellable + Dispatchers.IO) {
// This URI was newly created by ACTION_CREATE_DOCUMENT; never remove an existing user file.
runCatching { DocumentsContract.deleteDocument(context.contentResolver, uri) }
}
active = null
job = null
}
}
}
if (active != null) {
AlertDialog(onDismissRequest = {}, title = { Text("Downloading") }, text = {
Column {
Text(active!!.name)
if (progress.second > 0) LinearProgressIndicator(progress = (progress.first.toFloat() / progress.second).coerceIn(0f, 1f))
else LinearProgressIndicator()
}
}, confirmButton = {}, dismissButton = { TextButton(onClick = { cancel() }) { Text("Cancel") } })
}
failure?.let { message ->
AlertDialog(onDismissRequest = { failure = null }, title = { Text("Download unavailable") },
text = { Text(message) }, confirmButton = { TextButton(onClick = { failure = null }) { Text("OK") } })
}
return DownloadListener { url, userAgent, disposition, mimeType, _ ->
if (active != null || pending != null) {
Toast.makeText(context, "Finish or cancel the current download first", Toast.LENGTH_SHORT).show()
} else if (url.toHttpUrlOrNull() == null) {
failure = "This download link is not supported. Open the file from Cloud and try again."
} else {
val mime = mimeType?.substringBefore(';')?.takeIf { it.contains('/') } ?: "application/octet-stream"
val name = URLUtil.guessFileName(url, disposition, mime).replace(Regex("[\\\\/\\p{Cntrl}]"), "_").take(180).ifBlank { "download" }
pending = WebDownload(url, userAgent ?: "Archipelago Companion", CookieManager.getInstance().getCookie(url).orEmpty(), name, mime)
try {
save.launch(Intent(Intent.ACTION_CREATE_DOCUMENT).apply {
addCategory(Intent.CATEGORY_OPENABLE); type = mime; putExtra(Intent.EXTRA_TITLE, name)
})
} catch (_: Exception) { pending = null; failure = "No file-saving app is available on this device." }
}
}
}
@@ -0,0 +1,99 @@
package com.archipelago.app.ui.screens
import android.content.Context
import android.content.ContextWrapper
import android.graphics.Color
import android.view.View
import android.view.ViewGroup
import android.webkit.WebChromeClient
import android.widget.FrameLayout
import androidx.activity.ComponentActivity
import androidx.activity.OnBackPressedCallback
import androidx.compose.runtime.Composable
import androidx.compose.runtime.DisposableEffect
import androidx.compose.runtime.remember
import androidx.compose.ui.platform.LocalContext
import androidx.core.view.ViewCompat
import androidx.core.view.WindowCompat
import androidx.core.view.WindowInsetsCompat
import androidx.core.view.WindowInsetsControllerCompat
private fun Context.fullscreenActivity(): ComponentActivity? = when (this) {
is ComponentActivity -> this
is ContextWrapper -> baseContext.takeIf { it !== this }?.fullscreenActivity()
else -> null
}
/** Hosts Chromium's custom fullscreen view without replacing or reloading its WebView. */
internal class WebViewFullscreen(private val activity: ComponentActivity?) {
private var overlay: FrameLayout? = null
private var callback: WebChromeClient.CustomViewCallback? = null
private var back: OnBackPressedCallback? = null
private var visibleBars = 0
private var originalBehavior = 0
fun show(view: View?, onHidden: WebChromeClient.CustomViewCallback?) {
val owner = activity
// A second enter must not detach the active video or strand its callback.
if (owner == null || owner.isFinishing || owner.isDestroyed || view == null ||
view.parent != null || overlay != null
) {
onHidden?.onCustomViewHidden()
return
}
val decor = owner.window.decorView as? ViewGroup
if (decor == null) { onHidden?.onCustomViewHidden(); return }
val controller = WindowCompat.getInsetsController(owner.window, decor)
val insets = ViewCompat.getRootWindowInsets(decor)
visibleBars = 0
if (insets?.isVisible(WindowInsetsCompat.Type.statusBars()) != false) {
visibleBars = visibleBars or WindowInsetsCompat.Type.statusBars()
}
if (insets?.isVisible(WindowInsetsCompat.Type.navigationBars()) != false) {
visibleBars = visibleBars or WindowInsetsCompat.Type.navigationBars()
}
originalBehavior = controller.systemBarsBehavior
val host = FrameLayout(owner).apply {
setBackgroundColor(Color.BLACK)
keepScreenOn = true
addView(view, FrameLayout.LayoutParams(-1, -1))
}
overlay = host
callback = onHidden
decor.addView(host, ViewGroup.LayoutParams(-1, -1))
controller.systemBarsBehavior = WindowInsetsControllerCompat.BEHAVIOR_SHOW_TRANSIENT_BARS_BY_SWIPE
controller.hide(WindowInsetsCompat.Type.systemBars())
back = object : OnBackPressedCallback(true) {
override fun handleOnBackPressed() = hide()
}.also { owner.onBackPressedDispatcher.addCallback(it) }
view.requestFocus()
}
fun hide() {
val host = overlay ?: return
// Clear first: Chromium may synchronously call onHideCustomView again.
overlay = null
val notify = callback
callback = null
back?.remove()
back = null
host.keepScreenOn = false
host.removeAllViews()
(host.parent as? ViewGroup)?.removeView(host)
activity?.let { owner ->
val controller = WindowCompat.getInsetsController(owner.window, owner.window.decorView)
controller.systemBarsBehavior = originalBehavior
controller.hide(WindowInsetsCompat.Type.systemBars())
if (visibleBars != 0) controller.show(visibleBars)
}
notify?.onCustomViewHidden()
}
}
@Composable
internal fun rememberWebViewFullscreen(): WebViewFullscreen {
val context = LocalContext.current
val fullscreen = remember(context) { WebViewFullscreen(context.fullscreenActivity()) }
DisposableEffect(fullscreen) { onDispose { fullscreen.hide() } }
return fullscreen
}
@@ -611,6 +611,8 @@ fun WebViewScreen(
// before surfacing the error page: the mesh tunnel works from anywhere. // before surfacing the error page: the mesh tunnel works from anywhere.
meshFallbackUrl: String? = null, meshFallbackUrl: String? = null,
) { ) {
val fullscreen = rememberWebViewFullscreen()
val downloads = rememberWebViewDownloads()
var isLoading by remember { mutableStateOf(true) } var isLoading by remember { mutableStateOf(true) }
// First kiosk load (often over the FIPS mesh) gets the full branded // First kiosk load (often over the FIPS mesh) gets the full branded
// loader; later navigations keep just the slim top progress bar. // loader; later navigations keep just the slim top progress bar.
@@ -913,6 +915,7 @@ fun WebViewScreen(
cookieManager.setAcceptThirdPartyCookies(this, true) cookieManager.setAcceptThirdPartyCookies(this, true)
applyArchipelagoSettings() applyArchipelagoSettings()
setDownloadListener(downloads)
settings.apply { settings.apply {
setSupportMultipleWindows(true) // enables onCreateWindow for window.open setSupportMultipleWindows(true) // enables onCreateWindow for window.open
// Let JS open windows without a synchronous user-gesture // Let JS open windows without a synchronous user-gesture
@@ -1181,6 +1184,12 @@ fun WebViewScreen(
} }
webChromeClient = object : WebChromeClient() { webChromeClient = object : WebChromeClient() {
override fun onShowCustomView(view: android.view.View?, callback: CustomViewCallback?) {
fullscreen.show(view, callback)
}
override fun onHideCustomView() = fullscreen.hide()
override fun onProgressChanged(view: WebView?, newProgress: Int) { override fun onProgressChanged(view: WebView?, newProgress: Int) {
loadProgress = newProgress loadProgress = newProgress
} }
@@ -1546,6 +1555,8 @@ private fun InAppBrowser(
appName: String? = null, appName: String? = null,
onClose: () -> Unit, onClose: () -> Unit,
) { ) {
val fullscreen = rememberWebViewFullscreen()
val downloads = rememberWebViewDownloads()
val context = LocalContext.current val context = LocalContext.current
// Same-node check across BOTH node addresses (LAN + mesh ULA) — see the // Same-node check across BOTH node addresses (LAN + mesh ULA) — see the
// kiosk's isSameNode; a mismatch here bounced app links to the browser. // kiosk's isSameNode; a mismatch here bounced app links to the browser.
@@ -1643,6 +1654,7 @@ private fun InAppBrowser(
CookieManager.getInstance().setAcceptThirdPartyCookies(this, true) CookieManager.getInstance().setAcceptThirdPartyCookies(this, true)
applyArchipelagoSettings() applyArchipelagoSettings()
setDownloadListener(downloads)
// Node apps (BTCPay invoices, LND, Portainer tokens) are // Node apps (BTCPay invoices, LND, Portainer tokens) are
// served over plain HTTP too — same dead-clipboard trap. // served over plain HTTP too — same dead-clipboard trap.
addClipboardBridge() addClipboardBridge()
@@ -1662,6 +1674,12 @@ private fun InAppBrowser(
) )
webChromeClient = object : WebChromeClient() { webChromeClient = object : WebChromeClient() {
override fun onShowCustomView(view: android.view.View?, callback: CustomViewCallback?) {
fullscreen.show(view, callback)
}
override fun onHideCustomView() = fullscreen.hide()
override fun onProgressChanged(view: WebView?, newProgress: Int) { override fun onProgressChanged(view: WebView?, newProgress: Int) {
progress = newProgress progress = newProgress
} }
@@ -0,0 +1,88 @@
package com.archipelago.app.ui.screens
import okhttp3.OkHttpClient
import okhttp3.ResponseBody.Companion.toResponseBody
import okhttp3.mockwebserver.MockResponse
import okhttp3.mockwebserver.MockWebServer
import okio.Buffer
import org.junit.Assert.*
import org.junit.Test
import java.io.ByteArrayOutputStream
import java.io.IOException
import java.util.concurrent.CancellationException
class WebViewDownloadsTest {
private fun spec(url: String) = WebDownload(url, "test-agent", "session=test-only", "file.bin", "application/octet-stream")
@Test fun authenticatedDownloadWritesExactBytesAndReportsCompletion() {
MockWebServer().use { server ->
val bytes = ByteArray(256 * 1024 + 13) { (it % 251).toByte() }
server.enqueue(MockResponse().setBody(Buffer().write(bytes)))
val out = ByteArrayOutputStream()
var progress = 0L to 0L
assertEquals(bytes.size.toLong(), streamWebDownload(spec(server.url("/file").toString()), out, OkHttpClient(), onProgress = { done, total -> progress = done to total }))
assertArrayEquals(bytes, out.toByteArray())
assertEquals(bytes.size.toLong() to bytes.size.toLong(), progress)
assertEquals("session=test-only", server.takeRequest().getHeader("Cookie"))
}
}
@Test fun sameOriginRedirectKeepsSessionButCrossOriginNeverReceivesIt() {
MockWebServer().use { first -> MockWebServer().use { second ->
second.enqueue(MockResponse().setBody("final"))
first.enqueue(MockResponse().setResponseCode(302).addHeader("Location", "/relative"))
first.enqueue(MockResponse().setResponseCode(307).addHeader("Location", second.url("/target")))
val out = ByteArrayOutputStream()
streamWebDownload(spec(first.url("/start").toString()), out, OkHttpClient())
assertEquals("final", out.toString())
assertEquals("session=test-only", first.takeRequest().getHeader("Cookie"))
assertEquals("session=test-only", first.takeRequest().getHeader("Cookie"))
assertNull(second.takeRequest().getHeader("Cookie"))
} }
}
@Test fun authenticationFailureDoesNotSaveErrorBody() {
MockWebServer().use { server ->
server.enqueue(MockResponse().setResponseCode(401).setBody("login required"))
val out = ByteArrayOutputStream()
val error = assertThrows(IOException::class.java) { streamWebDownload(spec(server.url("/").toString()), out, OkHttpClient()) }
assertTrue(error.message!!.startsWith("Sign in"))
assertEquals(0, out.size())
}
}
@Test fun redirectsAreBoundedAndUnsafeSchemesAreRejected() {
MockWebServer().use { server ->
repeat(6) { server.enqueue(MockResponse().setResponseCode(302).addHeader("Location", "/loop")) }
assertThrows(IOException::class.java) { streamWebDownload(spec(server.url("/loop").toString()), ByteArrayOutputStream(), OkHttpClient()) }
assertEquals(6, server.requestCount)
}
for (url in listOf("file:///etc/passwd", "data:text/plain,test", "blob:test")) {
assertThrows(IOException::class.java) { streamWebDownload(spec(url), ByteArrayOutputStream(), OkHttpClient()) }
}
}
@Test fun cancellationAndDestinationFailureAreNotReportedAsComplete() {
MockWebServer().use { server ->
server.enqueue(MockResponse().setBody("bytes"))
assertThrows(CancellationException::class.java) { streamWebDownload(spec(server.url("/").toString()), ByteArrayOutputStream(), OkHttpClient(), checkCancelled = { throw CancellationException() }) }
assertEquals(0, server.requestCount)
var progressCalled = false
val out = object : java.io.OutputStream() { override fun write(b: Int) { throw IOException("disk full") } }
assertThrows(IOException::class.java) { streamWebDownload(spec(server.url("/").toString()), out, OkHttpClient(), onProgress = { _, _ -> progressCalled = true }) }
assertFalse(progressCalled)
}
}
@Test fun tlsDowngradeAndLoginHtmlAreRejected() {
var requests = 0
val client = OkHttpClient.Builder().addInterceptor { chain ->
requests++
okhttp3.Response.Builder().request(chain.request()).protocol(okhttp3.Protocol.HTTP_1_1)
.code(302).message("redirect").header("Location", "http://example.test/file").body("".toResponseBody(null)).build()
}.build()
assertThrows(IOException::class.java) { streamWebDownload(spec("https://example.test/file"), ByteArrayOutputStream(), client) }
assertEquals(1, requests)
MockWebServer().use { server ->
server.enqueue(MockResponse().addHeader("Content-Type", "Text/HTML; charset=utf-8").setBody("<html>Sign in</html>"))
val out = ByteArrayOutputStream()
assertThrows(IOException::class.java) { streamWebDownload(spec(server.url("/file").toString()), out, OkHttpClient()) }
assertEquals(0, out.size())
}
}
}
@@ -0,0 +1,71 @@
package com.archipelago.app.ui.screens
import android.view.View
import android.widget.FrameLayout
import androidx.activity.ComponentActivity
import org.junit.Assert.*
import org.junit.Test
import org.junit.runner.RunWith
import org.robolectric.Robolectric
import org.robolectric.RobolectricTestRunner
import org.robolectric.annotation.Config
@RunWith(RobolectricTestRunner::class)
@Config(manifest = Config.NONE, sdk = [28, 35])
class WebViewFullscreenTest {
@Test fun backExitsFullscreenWithoutFinishingActivityAndNotifiesOnce() {
val lifecycle = Robolectric.buildActivity(ComponentActivity::class.java).setup()
try {
val activity = lifecycle.get()
val fullscreen = WebViewFullscreen(activity)
val video = View(activity)
var hidden = 0
fullscreen.show(video) { hidden++ }
assertNotNull(video.parent)
activity.onBackPressedDispatcher.onBackPressed()
assertNull(video.parent)
assertFalse(activity.isFinishing)
assertEquals(1, hidden)
fullscreen.hide()
assertEquals(1, hidden)
} finally { lifecycle.pause().stop().destroy() }
}
@Test fun duplicateRequestPreservesActiveViewAndCanReenterAfterExit() {
val lifecycle = Robolectric.buildActivity(ComponentActivity::class.java).setup()
try {
val activity = lifecycle.get()
val fullscreen = WebViewFullscreen(activity)
val first = View(activity)
val second = View(activity)
var firstHidden = 0
var secondHidden = 0
fullscreen.show(first) { firstHidden++ }
fullscreen.show(second) { secondHidden++ }
assertNotNull(first.parent)
assertNull(second.parent)
assertEquals(0, firstHidden)
assertEquals(1, secondHidden)
fullscreen.hide()
fullscreen.show(second) { secondHidden++ }
assertNotNull(second.parent)
fullscreen.hide()
assertEquals(1, firstHidden)
assertEquals(2, secondHidden)
} finally { lifecycle.pause().stop().destroy() }
}
@Test fun rejectsOwnedViewWithoutReparentingAndHandlesUnavailableActivity() {
val lifecycle = Robolectric.buildActivity(ComponentActivity::class.java).setup()
try {
val activity = lifecycle.get()
val video = View(activity)
val owner = FrameLayout(activity).apply { addView(video) }
var hidden = 0
WebViewFullscreen(activity).show(video) { hidden++ }
assertSame(owner, video.parent)
WebViewFullscreen(null).show(null) { hidden++ }
assertEquals(2, hidden)
} finally { lifecycle.pause().stop().destroy() }
}
}
+78 -1
View File
@@ -1,6 +1,83 @@
# Changelog # Changelog
## Unreleased ## v1.9.0-alpha (2026-10-05)
Unpublished release candidate; qualification is still in progress.
- Keep Cuprate and NetBird supporting components out of app listings and consolidate BTCPay Server under Commerce.
- Default on-chain sends, channel opens and cooperative closes to a dynamic next-block fee target, preserving explicit slower and custom choices.
- Add reviewed fee-bump quotes, explicit budgets and durable operation tracking for supported wallet transactions.
- Preserve Nginx Proxy Manager storage, same-node upstream connectivity, certificates and access controls through managed migrations.
- Restrict public management access while retaining configured public apps and ACME certificate validation.
- Serve the Mempool explorer on the Angor indexer origin alongside its API.
- Include the self-contained LoRa flashing tool and explicit board selection in update and installer payloads.
- Preserve paid-file Lightning entitlements across restarts and recover settled invoices from LND. Retry delivery without paying again and retain purchased files in the owned cache.
- Return explicit payment-status errors with safe retry guidance when verification is unavailable.
- Keep upload progress on its original screen, show completion there or notify on other screens, and cancel active and queued uploads.
- Resume interrupted uploads while the app remains open, preserve the original destination, and verify saved file contents before reporting completion.
- Provision a unique private File Browser login on each node while keeping Cloud sign-in automatic and preserving existing accounts and files.
- Update Nostr dependencies to reject forged relay events and oversized encrypted messages; preserve native signing and encryption compatibility.
- Allow apps to opt in to a validated public-key list of user identities without granting signing access.
- Make transaction filters transparent and horizontally scrollable on mobile.
- Keep Immich internal services out of My Apps, avoid false recovery states for healthy stacks, and allow removal of retired catalog apps.
- Repair the redundant managed Portainer network override that can prevent startup, preserving custom overrides and persistent state.
- Offer Standard, Medium, Fast and custom fees when cooperatively closing Lightning channels.
- Add a clear-search icon to My Apps, Services and the App Store on desktop and mobile.
- Keep Angor Indexer and the optional Angor Relay in the signed app catalog. Full indexing requires a synced, unpruned Bitcoin node and its indexing dependencies.
## v1.8.22-alpha (2026-09-30)
- Fixed Nginx Proxy Manager launch readiness choosing a proxy listener instead of its admin port after container recreation.
- Network diagnostic failures no longer stop all apps or rebuild shared container networking.
- Prevented orphaned companion dashboards from repeatedly reinstalling themselves after their backend app was removed.
- Fixed companion dashboard builds still referencing a retired image registry.
- Fixed Angor Indexer health checks choosing IPv6 localhost for an IPv4 listener and unnecessarily restarting the working service.
- Prevented false app restarts by probing each published port at its actual bind address; Nginx Proxy Manager now checks its internal admin API.
- Added a backed-up migration for the recognized legacy Nginx Proxy Manager tunnel/LND port conflict in both OTA and ISO startup paths.
- Checked Bitcoin and Electrum companion dashboards instead of backend protocol ports, preserving dashboard access during initial sync.
- Removed web-interface waiting messages from headless services such as Phoenixd and clarified which interface is unavailable for launchable apps.
- Finished runtime app-file promotion before manifest loading, preventing startup catalog refresh from forgetting disk-only apps.
- Named the app in compact readiness messages and kept app-card actions aligned at the bottom.
- Removed duplicate Mempool cards caused by frontend container aliases in restored inventory.
- Kept installed apps visible through restarts and hard refreshes, and delayed app launches until their web interface is ready.
- Made Bitcoin version selection readable and usable in the ThinkPad kiosk, above the pruning settings.
- Restored GitWorkshop build files in installation/update payloads and made slow image-pull progress clearer.
- Fixed same-node Gitea access from Portainer, with persistent runtime migration, state backups and recovery after failed restarts.
- Preserved Gitea configuration and SSH operation during fresh setup and upgrades.
- Improved paid-file delivery, saved-file permissions and repeat-download compatibility; verified Tor-only payment with change, rejection refunds and free repeat downloads.
- Added a headless Angor Indexer service using the existing Mempool/ElectrumX stack, and an optional separate Angor relay.
- Prevented manifest command arguments containing apostrophes from being corrupted in generated services.
## v1.8.21-alpha (2026-09-30)
- Fixed Bitcoin and other containers being forcibly stopped after ten seconds during managed updates and restarts.
- Existing installations now receive the same graceful shutdown allowance as new containers, without restarting apps just to apply this setting.
- Prevented unnecessary Lightning restarts when Bitcoin has stayed running; dependency restarts now require an observed Bitcoin container change.
- Includes the Cashu payment, optional Bitcoin pruning, Lightning readiness, and explorer improvements from 1.8.20.
## v1.8.20-alpha (2026-09-29)
- Fixed Cashu file payments rejected despite a shared mint, and preserved the payment amount when mint fees reduce change.
- Payment failures now report whether a refund actually succeeded; missing files and unsupported payment methods are rejected before charging.
- Improved saving paid files into Files and reopening purchases without paying again.
- Bitcoin Core and Knots installation offers optional pruning on larger disks, using the same settings as automatic pruning.
- Fixed false missing-port checks that unnecessarily restarted Bitcoin and LND; recovery now respects managed shutdown timeouts.
- LND explains when it is waiting for Bitcoin installation, startup, or sync, without treating normal synchronization as a restart-worthy failure.
- Bitcoin startup messages explain block-index loading without exposing raw RPC errors, and Lightning keeps known balances clearly marked during outages.
- Changed the public transaction-explorer default to mempool.space while preserving local explorers and custom choices.
## v1.8.19-alpha (2026-09-28)
- Fixed the embedded AIUI chat page painting a second background and dark scrim over Archy’s dashboard background.
- Embedded AIUI now stays transparent so the dashboard background appears once.
- AIUI background fixes are now included reliably in OTA updates and fresh installations.
## v1.8.18-alpha (2026-09-18) ## v1.8.18-alpha (2026-09-18)
+38 -1
View File
@@ -64,12 +64,49 @@ App submissions must:
## Pull requests ## Pull requests
1. Open one focused PR per behavior or documentation change. Contributions, PRs and reviews live on **ngit**. Clone the canonical repository:
```text
nostr://npub1w3sqdkrhn0gyuvsex32effzgnfpyde6qrrc4u467flg5e9txh4wsfn5vjg/relay.ngit.dev/archy
```
Gitea is a conventional Git mirror of accepted `main` commits and release tags.
You do not need to open a duplicate Gitea PR. Existing Gitea contributions will
be reviewed and linked to their ngit replacement or accepted result before
closure.
1. Open one focused ngit PR per behavior or documentation change.
2. Explain what changed, why it changed, and how it was verified. 2. Explain what changed, why it changed, and how it was verified.
3. Include screenshots for UI changes. 3. Include screenshots for UI changes.
4. Link relevant issues or docs. 4. Link relevant issues or docs.
5. Keep generated catalog changes in sync with manifest changes. 5. Keep generated catalog changes in sync with manifest changes.
### Maintainer publication gate
Merge once through the ngit contribution workflow, then push the exact same
accepted commits to Gitea. Do not independently merge or squash on each mirror.
Publish identical release tag objects, including annotations and signatures.
After pushing main, verify:
```bash
python3 scripts/check-git-mirrors.py --local
```
Before publishing release artifacts, also check the actual release tag:
```bash
python3 scripts/check-git-mirrors.py --local --ref refs/tags/v1.9.0-alpha
```
Use the release's actual tag name. Missing refs, inaccessible mirrors or differing
object IDs block publication. Record the ngit PR disposition and resulting merge
commit in the release acceptance ledger. Resolve drift deliberately; do not
force-push or delete published history without explicit approval.
The checker is read-only. `--all` audits every advertised branch and tag; ngit
proposal branches may intentionally differ from Gitea. A main-only pass proves
only main parity, and no Git ref check verifies PR discussions or review state.
Suggested commit format: Suggested commit format:
```text ```text
+2
View File
@@ -1,5 +1,7 @@
# Archipelago # Archipelago
> **Alpha testing:** Archipelago is experimental software. Any funds you put on it are at your own risk.
> Self-sovereign Bitcoin node OS and manifest-driven app platform. > Self-sovereign Bitcoin node OS and manifest-driven app platform.
Archipelago is a bootable personal server OS for Bitcoin infrastructure, Archipelago is a bootable personal server OS for Bitcoin infrastructure,
+3 -2
View File
@@ -46,13 +46,14 @@ interface RateBucket {
const rateBuckets = new Map<string, RateBucket>() const rateBuckets = new Map<string, RateBucket>()
// Clean up stale buckets every 5 minutes // Vite imports this module during builds too; cleanup must not keep the
// process alive once compilation has finished.
setInterval(() => { setInterval(() => {
const now = Date.now() const now = Date.now()
for (const [key, bucket] of rateBuckets) { for (const [key, bucket] of rateBuckets) {
if (now > bucket.resetAt) rateBuckets.delete(key) if (now > bucket.resetAt) rateBuckets.delete(key)
} }
}, 5 * 60_000) }, 5 * 60_000).unref()
function getClientIp(req: IncomingMessage): string { function getClientIp(req: IncomingMessage): string {
return req.socket.remoteAddress ?? 'unknown' return req.socket.remoteAddress ?? 'unknown'
+1
View File
@@ -33,6 +33,7 @@ const PWA_CACHE_VERSION = '2'
// Only embedded when explicitly requested via ?embedded param // Only embedded when explicitly requested via ?embedded param
const _embeddedFlag = new URLSearchParams(window.location.search).has('embedded') const _embeddedFlag = new URLSearchParams(window.location.search).has('embedded')
;(window as unknown as Record<string, unknown>).__AIUI_EMBEDDED__ = _embeddedFlag ;(window as unknown as Record<string, unknown>).__AIUI_EMBEDDED__ = _embeddedFlag
document.documentElement.classList.toggle('aiui-embedded', _embeddedFlag)
const router = createRouter({ const router = createRouter({
history: createWebHistory(import.meta.env.BASE_URL), history: createWebHistory(import.meta.env.BASE_URL),
+5 -5
View File
@@ -2,13 +2,13 @@
<div <div
class="h-full flex flex-col relative overflow-hidden transition-colors duration-300" class="h-full flex flex-col relative overflow-hidden transition-colors duration-300"
:class="[]" :class="[]"
:style="isDark :style="isEmbedded
? { background: '#000 url(' + bgImageUrl + ') center center / cover no-repeat fixed' } ? { background: 'transparent' }
: isEmbedded : isDark
? { background: 'transparent' } ? { background: '#000 url(' + bgImageUrl + ') center center / cover no-repeat fixed' }
: { backgroundColor: '#f5f4f1' }" : { backgroundColor: '#f5f4f1' }"
> >
<div v-if="isDark" class="absolute inset-0 pointer-events-none bg-black/20" /> <div v-if="isDark && !isEmbedded" class="absolute inset-0 pointer-events-none bg-black/20" />
<!-- Desktop layout --> <!-- Desktop layout -->
<div <div
+15 -6
View File
@@ -57,12 +57,8 @@ body {
width: 100%; width: 100%;
height: 100%; height: 100%;
overflow: hidden; overflow: hidden;
/* Every page paints its own explicit background (bg-[#0a0a0a] / bg-[#faf9f6]) /* Standalone canvas fallback. Embedded mode overrides this below so
EXCEPT the embedded Chat page, which intentionally goes transparent so Archy's wallpaper remains visible through the iframe. */
Archy's own dark chrome can show behind it (Chat.vue's iframe host). With
no background-color here, "transparent" fell through to the browser's
default white canvas instead. Match the theme's own dark/light default so
nothing above this ever needs to guess. */
background-color: #0a0a0a; background-color: #0a0a0a;
} }
@@ -70,6 +66,19 @@ html.light body {
background-color: #faf9f6; background-color: #faf9f6;
} }
/* The host owns the wallpaper when AIUI is embedded. The document canvas
must be transparent too, otherwise it hides the host behind ChatPage. */
html.aiui-embedded {
/* Match Archy's dark canvas scheme. Browsers otherwise give an iframe
with a different scheme an opaque canvas despite transparent CSS. */
color-scheme: dark;
}
html.aiui-embedded,
html.aiui-embedded body {
background: transparent;
}
/* ===== DARK MODE GLASSMORPHISM — from Archy ===== */ /* ===== DARK MODE GLASSMORPHISM — from Archy ===== */
@layer components { @layer components {
+32 -3
View File
@@ -436,13 +436,13 @@
{ {
"id": "nginx-proxy-manager", "id": "nginx-proxy-manager",
"title": "Nginx Proxy Manager", "title": "Nginx Proxy Manager",
"version": "2.12.1", "version": "2.14.0",
"description": "Reverse proxy with SSL. Beautiful web interface for managing proxies. On a node, this manages its admin UI and upstream configuration — the proxy's own :80/:443 listeners are not published (the node's web server owns those ports).", "description": "Reverse proxy with SSL. Beautiful web interface for managing proxies. The node's public web server forwards configured domains through this service, preserving its access lists, certificates and custom routes.",
"icon": "/assets/img/app-icons/nginx.svg", "icon": "/assets/img/app-icons/nginx.svg",
"author": "Nginx Proxy Manager", "author": "Nginx Proxy Manager",
"category": "networking", "category": "networking",
"tier": "optional", "tier": "optional",
"dockerImage": "source.archipelago-foundation.org/lfg2025/nginx-proxy-manager:latest", "dockerImage": "source.archipelago-foundation.org/lfg2025/nginx-proxy-manager@sha256:8b91afcca90f5f2a7b2b8937999824f623c8a8748ae8013a1c9bf94f62177f08",
"repoUrl": "https://github.com/NginxProxyManager/nginx-proxy-manager" "repoUrl": "https://github.com/NginxProxyManager/nginx-proxy-manager"
}, },
{ {
@@ -644,6 +644,35 @@
"/var/lib/archipelago/vaultwarden:/data" "/var/lib/archipelago/vaultwarden:/data"
] ]
} }
},
{
"id": "angor-indexer",
"title": "Angor Indexer",
"version": "1.0.2",
"description": "Bitcoin indexer endpoint for Angor with the existing Mempool explorer. Reuses this node’s Mempool and Electrum index; requires a synced, unpruned Bitcoin node. Add this service’s address as the custom indexer in Angor settings. A relay is optional and installed separately.",
"dockerImage": "source.archipelago-foundation.org/chaum/angor-indexer:1.0.2",
"author": "Angor / Archipelago",
"requires": [
"Mempool API",
"Unpruned Bitcoin"
],
"category": "money",
"tier": "optional",
"icon": "/assets/img/app-icons/angor-green.png",
"repoUrl": "https://github.com/block-core/angor"
},
{
"id": "angor-relay",
"title": "Angor Relay",
"version": "1.1.2",
"description": "Optional dedicated Nostr relay for Angor project metadata. Separate storage and access settings keep the node’s internal relay private. Add this service’s address to Angor’s relay settings; use WSS for browser clients.",
"dockerImage": "source.archipelago-foundation.org/chaum/angor-relay:1.1.2",
"author": "Angor / Archipelago",
"requires": [],
"category": "nostr",
"tier": "optional",
"icon": "/assets/img/app-icons/angor-green.png",
"repoUrl": "https://github.com/hoytech/strfry"
} }
] ]
} }
+109
View File
@@ -0,0 +1,109 @@
# Angor Indexer
Mainnet indexer endpoint for Angor, serving the existing Mempool explorer at
the same origin. The service reuses this node's Mempool frontend/backend and
Electrum index instead of creating another explorer or blockchain database.
An unpruned, fully synced Bitcoin node is required. Installing against a pruned
node must show the existing archival-node requirement; it must never silently
unprune or replace its Bitcoin data.
## Connect Angor
Install **Angor Indexer** in the store. Its API appears under **Services**.
In Angor settings, use `http://<node-address>:8998/` as the custom indexer origin.
The `/health` endpoint reports readiness against Mempool's indexed block height;
it returns 503 while that backend is unavailable. Index building may take time.
Browser clients require a reachable HTTPS origin with a trusted certificate.
Configure your HTTPS reverse proxy to forward to port 8998, then use that HTTPS
origin in Angor. Do not disable browser TLS checks. The API supports both
`/api/v1/` and `/api/` paths, transaction broadcast, and CORS without cookies.
This endpoint intentionally exposes public blockchain queries and transaction
broadcast through the app gate without dashboard-cookie login. It has no Bitcoin
RPC password, wallet keys, or persistent wallet data. The backend stays on the
managed container network; its private port does not become publicly exposed.
You can change network access using the node's normal access controls.
## Relay
A relay is optional. Angor can continue using its configured external relays.
Install **Angor Relay** separately to host project metadata locally, then add
`ws://<node-address>:8091/` in Angor, or a trusted `wss://` proxy origin for browser
clients. Its storage and configuration are separate from the node's internal
relay; installing or uninstalling it does not change the internal relay.
## Verify the complete client flow
The root URL opens the Mempool explorer. `/health` and fee
estimates establish API availability; they do not prove that project discovery,
address history, or browser CORS works. Test a known funded project's address
history, its original Nostr announcement, the Explore page, and project details
in the actual Angor client. A certificate alone does not establish public routing.
Keep existing discovery relays when adding a new relay. A new relay has no
historical project data and does not automatically replicate other relays.
Even with existing relays, an empty Explore page can be a client discovery
failure: Angor Hub v2.0.0 was observed to stop after a batch whose announcements
all failed on-chain validation. The same failure reproduced with our indexer
and Angor's public indexer. Do not bypass the funding transaction's event-ID
commitment or substitute an unsigned announcement to make a project appear.
For opt-in read-only browser acceptance, install the frontend test dependencies
and Playwright Chromium, then run:
```sh
ANGOR_TEST_INDEXER=https://indexer.example.com/ \
ANGOR_TEST_RELAY=wss://relay.example.com/ \
ANGOR_TEST_RELAYS='["wss://relay.angor.io","wss://relay.example.com/"]' \
node tests/lifecycle/angor-public-browser.cjs
```
The relay under test must already contain the known original public project
announcement documented in the test. The test does not import events, send
funds, change your browser profile, or disable TLS verification. It checks the
funding transaction/event commitment and real browser discovery and details.
Relay signed writes, invalid-signature rejection, persistence, full node sync,
and proxy upgrade/renewal tests remain separate acceptance requirements.
## Packaging
Build the pinned image with:
```
podman build -t source.archipelago-foundation.org/chaum/angor-indexer:1.0.2 apps/angor-indexer/container
```
The image runs as UID 101 with a read-only root filesystem and no capabilities.
Only temporary nginx state is writable. Runtime DNS is read from resolv.conf so
Mempool recreation does not require editing IP addresses or restarting this app.
No app-specific Rust installer is required.
Source documentation: [Angor's official deployment guide](https://github.com/block-core/angor/blob/869dd43cf38332dd7128a284a6bf4c1cac44c1a7/docker/DEPLOY-INDEXER-AND-RELAY.md).
The app icon is based on [Angor’s dark-mode app icon](https://angor.io/images/app-icon-dark-mode.png), retrieved 2026-09-30. At the operator’s request, the outer corners use the same green as the background. The built-in imagegen edit preserved the black mark and filled the square green; the project asset is `neode-ui/public/assets/img/app-icons/angor-green.png`.
Tests and release acceptance are recorded in the next-release checklist. The
health probe establishes backend availability, not a guarantee that every
address query is indexed at the latest Bitcoin tip.
Install Mempool Explorer first. The declarative `install_prerequisites` check
refuses a new adapter installation if its Mempool API component is absent, before
creating an installed-app record. It does not install or resync Bitcoin for you.
## Explorer on the public indexer origin
The linked official deployment guide exposes **Mempool frontend and API together**
on the public indexer URL. It uses standard Mempool images and requires no custom
Angor fork or `ANGOR_ENABLED` flag.
The operator now requires that same browser experience: opening the configured
indexer domain must show the existing Mempool explorer, while Angor API requests
continue working on that origin. Reuse the existing Mempool stack, including its
live WebSocket feed; do not install a second explorer or blockchain database.
**Candidate 1.0.2:** `/` and frontend paths proxy to the existing Mempool
frontend; `/api/`, `/api/v1/`, `/health` and the WebSocket feed retain their
indexer routes. Version 1.0.1 served only service JSON at `/`. The candidate
remains pending deployment/release acceptance, which must cover assets and deep links,
desktop/mobile rendering, WebSocket updates, API/CORS/broadcast, trusted HTTPS,
restart/upgrade and management-access isolation before documenting it as shipped.
+6
View File
@@ -0,0 +1,6 @@
FROM docker.io/library/nginx:1.31.3-alpine@sha256:1d40e3eb3bf4f138de1d67193f2aa5309fcaf343eb5ffadbf5e9439de1eb1ebb
COPY nginx.conf /etc/angor-nginx.conf.template
COPY entrypoint.sh /usr/local/bin/angor-indexer
USER 101:101
EXPOSE 8080
ENTRYPOINT ["/usr/local/bin/angor-indexer"]
+12
View File
@@ -0,0 +1,12 @@
#!/bin/sh
set -eu
# Resolve through the container runtime's DNS, including after dependency
# recreation. Never bake a container IP into the indexer endpoint.
DNS_RESOLVER=$(awk '/^nameserver[[:space:]]/ {print $2; exit}' /etc/resolv.conf)
case "$DNS_RESOLVER" in
''|*[!0-9a-fA-F.:]*) echo 'Container DNS resolver is unavailable' >&2; exit 1 ;;
esac
case "$DNS_RESOLVER" in *:*) DNS_RESOLVER="[$DNS_RESOLVER]" ;; esac
export DNS_RESOLVER
envsubst '${DNS_RESOLVER}' < /etc/angor-nginx.conf.template > /tmp/nginx.conf
exec nginx -c /tmp/nginx.conf -g 'daemon off;'
+81
View File
@@ -0,0 +1,81 @@
worker_processes 1;
pid /tmp/nginx.pid;
error_log /dev/stderr warn;
events { worker_connections 512; }
http {
access_log off;
server_tokens off;
client_body_temp_path /tmp/client_temp;
proxy_temp_path /tmp/proxy_temp;
fastcgi_temp_path /tmp/fastcgi_temp;
uwsgi_temp_path /tmp/uwsgi_temp;
scgi_temp_path /tmp/scgi_temp;
resolver ${DNS_RESOLVER} valid=10s ipv6=off;
upstream mempool_backend {
zone mempool_backend 64k;
server mempool-api:8999 resolve;
}
upstream mempool_frontend {
zone mempool_frontend 64k;
server mempool:8080 resolve;
}
map $http_upgrade $angor_connection_upgrade {
default upgrade;
'' close;
}
server {
listen 8080;
client_max_body_size 4m;
proxy_connect_timeout 5s;
proxy_read_timeout 60s;
proxy_send_timeout 30s;
proxy_http_version 1.1;
proxy_set_header Host $host;
proxy_set_header Connection $angor_connection_upgrade;
proxy_set_header Upgrade $http_upgrade;
proxy_set_header Authorization "";
proxy_set_header Cookie "";
proxy_hide_header Access-Control-Allow-Origin;
add_header Access-Control-Allow-Origin '*' always;
add_header Access-Control-Allow-Methods 'GET, HEAD, POST, OPTIONS' always;
add_header Access-Control-Allow-Headers 'Content-Type' always;
add_header Cache-Control 'no-store' always;
if ($request_method = OPTIONS) { return 204; }
# Mempool's backend uses /api/v1. Match its frontend's shorter /api
# surface too, without doubling already-versioned Angor URLs.
rewrite ^/api/(?!v1/)(.*)$ /api/v1/$1 last;
# Readiness checks the indexing backend, not this gateway's process.
location = /health {
limit_except GET { deny all; }
proxy_pass http://mempool_backend/api/v1/blocks/tip/height;
proxy_intercept_errors on;
error_page 500 502 503 504 =503 @waiting;
}
location @waiting {
default_type application/json;
return 503 '{"status":"waiting","message":"Waiting for Bitcoin and Mempool indexing"}\n';
}
location ~ ^/api/(v1/)?tx$ {
limit_except GET POST { deny all; }
proxy_pass http://mempool_backend;
}
location = /api/v1/ws {
limit_except GET { deny all; }
proxy_read_timeout 600s;
proxy_send_timeout 600s;
proxy_pass http://mempool_backend;
}
location /api/ {
limit_except GET { deny all; }
proxy_pass http://mempool_backend;
}
# Share the already-installed explorer; no second frontend or index DB.
# Its SPA handles transaction/block deep links and static assets.
location / {
limit_except GET { deny all; }
proxy_pass http://mempool_frontend;
proxy_intercept_errors on;
error_page 500 502 503 504 =503 @waiting;
}
}
}
+73
View File
@@ -0,0 +1,73 @@
app:
id: angor-indexer
name: Angor Indexer
version: 1.0.2
description: Bitcoin indexer endpoint for Angor with the existing Mempool explorer.
Reuses this node’s Mempool
and Electrum index; requires a synced, unpruned Bitcoin node. Add this service’s
address as the custom indexer in Angor settings. A relay is optional and installed
separately.
category: money
install_prerequisites:
- mempool
- mempool-api
upstream:
kind: github
repo: block-core/angor
container:
image: source.archipelago-foundation.org/chaum/angor-indexer:1.0.2
pull_policy: if-not-present
network: archy-net
dependencies:
- app_id: mempool
version: '>=3.0.0'
- app_id: mempool-api
version: '>=3.0.0'
- bitcoin:archival
resources:
cpu_limit: 1
memory_limit: 128Mi
disk_limit: 128Mi
security:
capabilities: []
readonly_root: true
no_new_privileges: true
user: 101
network_policy: isolated
ports:
- host: 8998
container: 8080
protocol: tcp
bind: 127.0.0.1
auth: open
auth_rationale: Public Bitcoin chain-data API and validated transaction broadcast for Angor clients; no wallet keys or node RPC credentials are exposed. Browser cookie login would break machine clients.
interfaces:
main:
name: Angor Indexer API
description: Use this origin as Angor’s custom mainnet indexer URL, or open it
to view the existing Mempool explorer. HTTPS is required for browser clients.
type: api
port: 8998
protocol: http
path: /
health_check:
type: http
endpoint: http://127.0.0.1:8080
path: /health
interval: 30s
timeout: 8s
retries: 3
bitcoin_integration:
rpc_access: none
sync_required: true
pruning_support: false
metadata:
icon: /assets/img/app-icons/angor-green.png
tier: optional
repo: https://github.com/block-core/angor
features:
- Angor mainnet API
- Mempool explorer on the same origin
- Reuses existing Mempool indexing
- No separate blockchain database
- Optional independent relay
+33
View File
@@ -0,0 +1,33 @@
# Angor Relay
Optional standalone strfry relay for Angor's public project metadata. See
[Angor Indexer setup](../angor-indexer/README.md) for client URLs and HTTPS/WSS.
The gate exposes port 8091 for Nostr clients. strfry validates event signatures;
this is a public relay, not a private messaging archive. It mounts only
`/var/lib/archipelago/angor-relay` and its separate configuration directory.
It never opens, reconfigures or shares the node's internal strfry database.
For a public domain, proxy HTTPS to node port **8091**, enable WebSocket upgrade,
and add `wss://your-relay-domain/` in Angor. Test both NIP-11 metadata (send
`Accept: application/nostr+json`) and a real Nostr subscription over WSS. An
Archipelago login page at this domain is a routing failure, not relay readiness.
New relays start without project history. Keep existing discovery relays alongside
yours until the needed original signed announcements and metadata are available
locally. Relays do not automatically synchronize. Any history import must retain
the original event IDs and signatures; verify funded projects against their
on-chain commitments. A working WebSocket with zero stored events is not proof
that the client's project discovery works. See the indexer README's browser test.
The configuration is seeded only when absent, preserving operator changes.
Stop the service before making a consistent backup of its event database.
Ordinary start/restart/recreation preserves both mounts. Use the standard app
lifecycle; do not manually recreate a systemd-managed container.
## Image provenance
Mirrored from `docker.io/dockurr/strfry:1.1.2`, upstream manifest digest
`sha256:e81d238db13507f6ef24c49d47cd0b0ea58ff207961f10581fa2a7c901054df4`.
The public Angor policy is supplied by this app's own configuration; it does not
reuse the internal relay's event whitelist.
+223
View File
@@ -0,0 +1,223 @@
app:
id: angor-relay
name: Angor Relay
version: 1.1.2
upstream:
kind: github
repo: hoytech/strfry
description: Optional dedicated Nostr relay for Angor project metadata. Separate
storage and access settings keep the node’s internal relay private. Add this service’s
address to Angor’s relay settings; use WSS for browser clients.
container:
image: source.archipelago-foundation.org/chaum/angor-relay:1.1.2
pull_policy: if-not-present
dependencies:
- storage: 5Gi
resources:
cpu_limit: 1
memory_limit: 512Mi
disk_limit: 5Gi
security:
capabilities: []
readonly_root: true
no_new_privileges: true
seccomp_profile: default
network_policy: isolated
apparmor_profile: nostr-relay
ports:
- host: 8091
container: 7777
protocol: tcp
bind: 127.0.0.1
auth: open
auth_rationale: Dedicated public Nostr relay for Angor project metadata; strfry verifies event signatures. It has separate storage from the private node relay and no wallet or node credentials.
volumes:
- type: bind
source: /var/lib/archipelago/angor-relay
target: /app/strfry-db
options:
- rw
- type: bind
source: /var/lib/archipelago/angor-relay-config/angor-relay.conf
target: /etc/strfry.conf
options:
- ro
files:
- path: /var/lib/archipelago/angor-relay-config/angor-relay.conf
overwrite: false
content: |
##
## Default strfry config
##
# Directory that contains the strfry LMDB database (restart required)
db = "./strfry-db/"
dbParams {
# Maximum number of threads/processes that can simultaneously have LMDB transactions open (restart required)
maxreaders = 256
# Size of mmap() to use when loading LMDB (default is 10TB, does *not* correspond to disk-space used) (restart required)
mapsize = 10995116277760
# Disables read-ahead when accessing the LMDB mapping. Reduces IO activity when DB size is larger than RAM. (restart required)
noReadAhead = false
}
events {
# Maximum size of normalised JSON, in bytes
maxEventSize = 65536
# Events newer than this will be rejected
rejectEventsNewerThanSeconds = 900
# Events older than this will be rejected
rejectEventsOlderThanSeconds = 94608000
# Ephemeral events older than this will be rejected
rejectEphemeralEventsOlderThanSeconds = 60
# Ephemeral events will be deleted from the DB when older than this
ephemeralEventsLifetimeSeconds = 300
# Maximum number of tags allowed
maxNumTags = 2000
# Maximum size for tag values, in bytes
maxTagValSize = 1024
}
relay {
# Interface to listen on. Use 0.0.0.0 to listen on all interfaces (restart required)
bind = "0.0.0.0"
# Port to open for the nostr websocket protocol (restart required)
port = 7777
# Set OS-limit on maximum number of open files/sockets (if 0, don't attempt to set) (restart required)
nofiles = 0
# HTTP header that contains the client's real IP, before reverse proxying (ie x-real-ip) (MUST be all lower-case)
realIpHeader = ""
info {
# NIP-11: Name of this server. Short/descriptive (< 30 characters)
name = "Angor Relay"
# NIP-11: Detailed information about relay, free-form
description = "Dedicated public relay for Angor project metadata."
# NIP-11: Administrative nostr pubkey, for contact purposes
pubkey = ""
# NIP-11: Alternative administrative contact (email, website, etc)
contact = ""
# NIP-11: URL pointing to an image to be used as an icon for the relay
icon = ""
# List of supported lists as JSON array, or empty string to use default. Example: "[1,2]"
nips = ""
}
# Maximum accepted incoming websocket frame size (should be larger than max event) (restart required)
maxWebsocketPayloadSize = 131072
# Maximum number of filters allowed in a REQ
maxReqFilterSize = 200
# Websocket-level PING message frequency (should be less than any reverse proxy idle timeouts) (restart required)
autoPingSeconds = 55
# If TCP keep-alive should be enabled (detect dropped connections to upstream reverse proxy)
enableTcpKeepalive = false
# How much uninterrupted CPU time a REQ query should get during its DB scan
queryTimesliceBudgetMicroseconds = 10000
# Maximum records that can be returned per filter
maxFilterLimit = 500
# Maximum number of subscriptions (concurrent REQs) a connection can have open at any time
maxSubsPerConnection = 20
writePolicy {
# If non-empty, path to an executable script that implements the writePolicy plugin logic
plugin = ""
}
compression {
# Use permessage-deflate compression if supported by client. Reduces bandwidth, but slight increase in CPU (restart required)
enabled = true
# Maintain a sliding window buffer for each connection. Improves compression, but uses more memory (restart required)
slidingWindow = true
}
logging {
# Dump all incoming messages
dumpInAll = false
# Dump all incoming EVENT messages
dumpInEvents = false
# Dump all incoming REQ/CLOSE messages
dumpInReqs = false
# Log performance metrics for initial REQ database scans
dbScanPerf = false
# Log reason for invalid event rejection? Can be disabled to silence excessive logging
invalidEvents = true
}
numThreads {
# Ingester threads: route incoming requests, validate events/sigs (restart required)
ingester = 3
# reqWorker threads: Handle initial DB scan for events (restart required)
reqWorker = 3
# reqMonitor threads: Handle filtering of new events (restart required)
reqMonitor = 3
# negentropy threads: Handle negentropy protocol messages (restart required)
negentropy = 2
}
negentropy {
# Support negentropy protocol messages
enabled = true
# Maximum records that sync will process before returning an error
maxSyncEvents = 1000000
}
}
health_check:
type: http
endpoint: http://127.0.0.1:7777
path: /health
interval: 30s
timeout: 5s
retries: 3
nostr_integration:
relay_type: public
monetization_enabled: false
category: nostr
interfaces:
main:
name: Angor Relay
description: Nostr WebSocket endpoint; use ws:// for LAN or wss:// through your
HTTPS domain.
type: api
port: 8091
protocol: http
path: /
metadata:
icon: /assets/img/app-icons/angor-green.png
tier: optional
repo: https://github.com/hoytech/strfry
features:
- Angor project metadata
- Separate from the node relay
- Persistent Nostr event storage
+1 -1
View File
@@ -54,7 +54,7 @@ app:
if [ -n "$RPC_TXRELAY_AUTH" ]; then if [ -n "$RPC_TXRELAY_AUTH" ]; then
RPC_TXRELAY_FLAGS="$RPC_TXRELAY_FLAGS -rpcauth=$RPC_TXRELAY_AUTH -rpcwhitelist=txrelay:sendrawtransaction,submitpackage,testmempoolaccept,getmempoolinfo,getrawmempool,getmempoolentry,getnetworkinfo,getblockchaininfo,getblockcount,getblockhash,getblock,getblockheader,getrawtransaction,gettxout,gettxspendingprevout,decoderawtransaction,decodescript,estimatesmartfee,uptime,ping,getconnectioncount,getpeerinfo,getindexinfo,getdeploymentinfo,getchaintips"; RPC_TXRELAY_FLAGS="$RPC_TXRELAY_FLAGS -rpcauth=$RPC_TXRELAY_AUTH -rpcwhitelist=txrelay:sendrawtransaction,submitpackage,testmempoolaccept,getmempoolinfo,getrawmempool,getmempoolentry,getnetworkinfo,getblockchaininfo,getblockcount,getblockhash,getblock,getblockheader,getrawtransaction,gettxout,gettxspendingprevout,decoderawtransaction,decodescript,estimatesmartfee,uptime,ping,getconnectioncount,getpeerinfo,getindexinfo,getdeploymentinfo,getchaintips";
fi; fi;
if [ "${DISK_GB_VALUE:-0}" -lt 1000 ]; then if [ "${BITCOIN_PRUNE:-0}" = "1" ] || [ "${DISK_GB_VALUE:-0}" -lt 1000 ]; then
exec "$BITCOIND" -datadir=/home/bitcoin/.bitcoin -conf="$RPC_CONF" -allowignoredconf=1 -printtoconsole=0 -server=1 -prune=50000 -rpcallowip=0.0.0.0/0 -rpcbind=0.0.0.0:8332 -listen=1 -bind=0.0.0.0:8333 -dbcache=1024 -par=0 -maxconnections=125 $RPC_HEADROOM $RPC_TXRELAY_FLAGS; exec "$BITCOIND" -datadir=/home/bitcoin/.bitcoin -conf="$RPC_CONF" -allowignoredconf=1 -printtoconsole=0 -server=1 -prune=50000 -rpcallowip=0.0.0.0/0 -rpcbind=0.0.0.0:8332 -listen=1 -bind=0.0.0.0:8333 -dbcache=1024 -par=0 -maxconnections=125 $RPC_HEADROOM $RPC_TXRELAY_FLAGS;
else else
exec "$BITCOIND" -datadir=/home/bitcoin/.bitcoin -conf="$RPC_CONF" -allowignoredconf=1 -printtoconsole=0 -server=1 -txindex=1 -rpcallowip=0.0.0.0/0 -rpcbind=0.0.0.0:8332 -listen=1 -bind=0.0.0.0:8333 -dbcache=4096 -par=0 -maxconnections=125 $RPC_HEADROOM $RPC_TXRELAY_FLAGS; exec "$BITCOIND" -datadir=/home/bitcoin/.bitcoin -conf="$RPC_CONF" -allowignoredconf=1 -printtoconsole=0 -server=1 -txindex=1 -rpcallowip=0.0.0.0/0 -rpcbind=0.0.0.0:8332 -listen=1 -bind=0.0.0.0:8333 -dbcache=4096 -par=0 -maxconnections=125 $RPC_HEADROOM $RPC_TXRELAY_FLAGS;
+1 -1
View File
@@ -60,7 +60,7 @@ app:
if [ -n "$RPC_TXRELAY_AUTH" ]; then if [ -n "$RPC_TXRELAY_AUTH" ]; then
RPC_TXRELAY_FLAGS="$RPC_TXRELAY_FLAGS -rpcauth=$RPC_TXRELAY_AUTH -rpcwhitelist=txrelay:sendrawtransaction,submitpackage,testmempoolaccept,getmempoolinfo,getrawmempool,getmempoolentry,getnetworkinfo,getblockchaininfo,getblockcount,getblockhash,getblock,getblockheader,getrawtransaction,gettxout,gettxspendingprevout,decoderawtransaction,decodescript,estimatesmartfee,uptime,ping,getconnectioncount,getpeerinfo,getindexinfo,getdeploymentinfo,getchaintips"; RPC_TXRELAY_FLAGS="$RPC_TXRELAY_FLAGS -rpcauth=$RPC_TXRELAY_AUTH -rpcwhitelist=txrelay:sendrawtransaction,submitpackage,testmempoolaccept,getmempoolinfo,getrawmempool,getmempoolentry,getnetworkinfo,getblockchaininfo,getblockcount,getblockhash,getblock,getblockheader,getrawtransaction,gettxout,gettxspendingprevout,decoderawtransaction,decodescript,estimatesmartfee,uptime,ping,getconnectioncount,getpeerinfo,getindexinfo,getdeploymentinfo,getchaintips";
fi; fi;
if [ "${DISK_GB_VALUE:-0}" -lt 1000 ]; then if [ "${BITCOIN_PRUNE:-0}" = "1" ] || [ "${DISK_GB_VALUE:-0}" -lt 1000 ]; then
exec "$BITCOIND" -datadir=/home/bitcoin/.bitcoin -conf="$RPC_CONF" -allowignoredconf=1 -printtoconsole=0 -server=1 -prune=50000 -rpcallowip=0.0.0.0/0 -rpcbind=0.0.0.0:8332 -listen=1 -bind=0.0.0.0:8333 -dbcache=2048 -par=0 -maxconnections=125 $RPC_HEADROOM $RPC_TXRELAY_FLAGS; exec "$BITCOIND" -datadir=/home/bitcoin/.bitcoin -conf="$RPC_CONF" -allowignoredconf=1 -printtoconsole=0 -server=1 -prune=50000 -rpcallowip=0.0.0.0/0 -rpcbind=0.0.0.0:8332 -listen=1 -bind=0.0.0.0:8333 -dbcache=2048 -par=0 -maxconnections=125 $RPC_HEADROOM $RPC_TXRELAY_FLAGS;
else else
exec "$BITCOIND" -datadir=/home/bitcoin/.bitcoin -conf="$RPC_CONF" -allowignoredconf=1 -printtoconsole=0 -server=1 -txindex=1 -rpcallowip=0.0.0.0/0 -rpcbind=0.0.0.0:8332 -listen=1 -bind=0.0.0.0:8333 -dbcache=4096 -par=0 -maxconnections=125 $RPC_HEADROOM $RPC_TXRELAY_FLAGS; exec "$BITCOIND" -datadir=/home/bitcoin/.bitcoin -conf="$RPC_CONF" -allowignoredconf=1 -printtoconsole=0 -server=1 -txindex=1 -rpcallowip=0.0.0.0/0 -rpcbind=0.0.0.0:8332 -listen=1 -bind=0.0.0.0:8333 -dbcache=4096 -par=0 -maxconnections=125 $RPC_HEADROOM $RPC_TXRELAY_FLAGS;
+15 -8
View File
@@ -15,6 +15,9 @@ app:
image: source.archipelago-foundation.org/lfg2025/gitea:1.27.3 image: source.archipelago-foundation.org/lfg2025/gitea:1.27.3
pull_policy: if-not-present pull_policy: if-not-present
# Preserve repositories, database, keys and configuration during runtime repairs.
backup_before_runtime_change: true
dependencies: dependencies:
# Source history, LFS objects, release artifacts and OCI layers all share # Source history, LFS objects, release artifacts and OCI layers all share
# this persistent store. 500Mi was only suitable for an empty demo node. # this persistent store. 500Mi was only suitable for an empty demo node.
@@ -25,7 +28,7 @@ app:
disk_limit: 50Gi disk_limit: 50Gi
security: security:
capabilities: [CHOWN, FOWNER, SETUID, SETGID, DAC_OVERRIDE, NET_BIND_SERVICE] capabilities: [CHOWN, FOWNER, SETUID, SETGID, DAC_OVERRIDE, NET_BIND_SERVICE, SYS_CHROOT]
readonly_root: false readonly_root: false
no_new_privileges: false no_new_privileges: false
network_policy: bridge network_policy: bridge
@@ -62,6 +65,17 @@ app:
target: /etc/gitea target: /etc/gitea
options: [rw] options: [rw]
# Seed a fresh installation with the same origin advertised by the app gate.
# Existing app.ini (including custom HTTPS/domain settings) is never replaced.
files:
- path: /var/lib/archipelago/gitea/data/gitea/conf/app.ini
overwrite: false
content: |
[server]
DOMAIN = {{HOST_IP}}
SSH_DOMAIN = {{HOST_IP}}
ROOT_URL = http://{{HOST_IP}}:3001/
environment: environment:
- GITEA__database__DB_TYPE=sqlite3 - GITEA__database__DB_TYPE=sqlite3
- GITEA__server__SSH_PORT=2222 - GITEA__server__SSH_PORT=2222
@@ -106,10 +120,3 @@ app:
- Issue tracking and pull requests - Issue tracking and pull requests
- CI/CD via Gitea Actions - CI/CD via Gitea Actions
- Lightweight SQLite deployment - Lightweight SQLite deployment
nginx_proxy:
listen: 3000
proxy_pass: http://127.0.0.1:3001
extra_headers:
- proxy_hide_header X-Frame-Options
- proxy_hide_header Content-Security-Policy
+24 -8
View File
@@ -1,20 +1,23 @@
app: app:
id: nginx-proxy-manager id: nginx-proxy-manager
name: Nginx Proxy Manager name: Nginx Proxy Manager
version: 2.12.1 version: 2.14.0
upstream: upstream:
kind: github kind: github
repo: NginxProxyManager/nginx-proxy-manager repo: NginxProxyManager/nginx-proxy-manager
description: >- description: >-
Reverse proxy with SSL. Beautiful web interface for managing proxies. Reverse proxy with SSL. Beautiful web interface for managing proxies.
On a node, this manages its admin UI and upstream configuration — the The node's public web server forwards configured domains through this
proxy's own :80/:443 listeners are not published (the node's web server service, preserving its access lists, certificates and custom routes.
owns those ports). backup_before_runtime_change: true
container: container:
image: source.archipelago-foundation.org/lfg2025/nginx-proxy-manager:latest image: source.archipelago-foundation.org/lfg2025/nginx-proxy-manager@sha256:8b91afcca90f5f2a7b2b8937999824f623c8a8748ae8013a1c9bf94f62177f08
pull_policy: if-not-present pull_policy: if-not-present
network: pasta # Rootless pasta copies the LAN IP, preventing requests back to this node.
# Retain the old pasta host gateway used by saved NPM upstreams, plus
# host.containers.internal. This subnet stays inside the private rootless namespace.
network: slirp4netns:allow_host_loopback=true,cidr=169.254.1.0/24
dependencies: dependencies:
- storage: 1Gi - storage: 1Gi
@@ -49,6 +52,17 @@ app:
Nginx Proxy Manager enforces its own admin account on every page; Nginx Proxy Manager enforces its own admin account on every page;
the initial setup wizard also has to answer before any account exists. the initial setup wizard also has to answer before any account exists.
- host: 8088
container: 80
protocol: tcp
bind: 127.0.0.1
auth: local
- host: 8444
container: 443
protocol: tcp
bind: 127.0.0.1
auth: local
volumes: volumes:
- type: bind - type: bind
source: /var/lib/archipelago/nginx-proxy-manager source: /var/lib/archipelago/nginx-proxy-manager
@@ -64,9 +78,11 @@ app:
environment: [] environment: []
# Probe the admin API inside the container, independent of optional
# tunnel listeners. This also verifies the Node backend is ready.
health_check: health_check:
type: tcp type: http
endpoint: localhost:81 endpoint: http://127.0.0.1:81/api/
interval: 30s interval: 30s
timeout: 5s timeout: 5s
retries: 3 retries: 3
+8
View File
@@ -14,8 +14,16 @@ app:
container: container:
image: source.archipelago-foundation.org/lfg2025/portainer:2.45.0 image: source.archipelago-foundation.org/lfg2025/portainer:2.45.0
pull_policy: if-not-present pull_policy: if-not-present
# Portainer fetches Git sources and images from services on this same node.
# Rootless pasta copies the host LAN address into its namespace, so a LAN
# URL points back at Portainer itself. Give it a private address with the
# supported rootless slirp backend; public app URLs still traverse the gate.
network: slirp4netns
data_uid: "1000:1000" data_uid: "1000:1000"
# Snapshot state before an upgrade recreates this app with new networking.
backup_before_runtime_change: true
dependencies: dependencies:
- storage: 1Gi - storage: 1Gi
+5 -5
View File
@@ -104,7 +104,7 @@ dependencies = [
[[package]] [[package]]
name = "archipelago" name = "archipelago"
version = "1.8.18-alpha" version = "1.9.0-alpha"
dependencies = [ dependencies = [
"anyhow", "anyhow",
"archipelago-container", "archipelago-container",
@@ -3636,9 +3636,9 @@ dependencies = [
[[package]] [[package]]
name = "nostr" name = "nostr"
version = "0.44.2" version = "0.44.7"
source = "registry+https://github.com/rust-lang/crates.io-index" source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "3aa5e3b6a278ed061835fe1ee293b71641e6bf8b401cfe4e1834bbf4ef0a34e1" checksum = "c7d3d987ea7078dc36947cde532637c472a229426702e4331dd7667325378bd9"
dependencies = [ dependencies = [
"aes", "aes",
"base64 0.22.1", "base64 0.22.1",
@@ -3681,9 +3681,9 @@ dependencies = [
[[package]] [[package]]
name = "nostr-relay-pool" name = "nostr-relay-pool"
version = "0.44.0" version = "0.44.3"
source = "registry+https://github.com/rust-lang/crates.io-index" source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "4b1073ccfbaea5549fb914a9d52c68dab2aecda61535e5143dd73e95445a804b" checksum = "c85c54d6ca9aae4ae2bf19a7663ba9db5f45f783f1d24aff55f006386b8b99a1"
dependencies = [ dependencies = [
"async-utility", "async-utility",
"async-wsocket", "async-wsocket",
+1 -1
View File
@@ -1,6 +1,6 @@
[package] [package]
name = "archipelago" name = "archipelago"
version = "1.8.18-alpha" version = "1.9.0-alpha"
edition = "2021" edition = "2021"
license.workspace = true license.workspace = true
description = "Archipelago Bitcoin Node OS - Native backend" description = "Archipelago Bitcoin Node OS - Native backend"
+192 -53
View File
@@ -74,7 +74,7 @@ impl ApiHandler {
let invoice_hash = headers let invoice_hash = headers
.get("x-invoice-hash") .get("x-invoice-hash")
.and_then(|v| v.to_str().ok()) .and_then(|v| v.to_str().ok())
.map(|s| s.to_string()) .map(|s| s.to_ascii_lowercase())
.or_else(|| { .or_else(|| {
headers headers
.get("x-onchain-address") .get("x-onchain-address")
@@ -98,6 +98,46 @@ impl ApiHandler {
None => false, None => false,
}; };
// Payment settlement is verified on the seller even when no status
// poll preceded this download (e.g. direct payment from another node).
let requires_payment = if !owner_session && headers.contains_key("x-invoice-hash") {
content_server::load_catalog(&config.data_dir)
.await?
.items
.iter()
.any(|item| {
item.id == content_id
&& matches!(item.access, content_server::AccessControl::Paid { .. })
})
} else {
false
};
if requires_payment {
if let Some(hash) = headers.get("x-invoice-hash").and_then(|v| v.to_str().ok()) {
if hash.len() != 64 || !hash.bytes().all(|c| c.is_ascii_hexdigit()) {
return Ok(build_response(
StatusCode::BAD_REQUEST,
"text/plain",
hyper::Body::from("Invalid payment hash"),
));
}
if let Err(error) = self
.rpc_handler
.settle_content_invoice(hash, content_id)
.await
{
tracing::warn!("Cannot verify peer-file invoice settlement: {error:#}");
return Ok(build_response(
StatusCode::SERVICE_UNAVAILABLE,
"application/json",
hyper::Body::from(
r#"{"error":"Payment verification is temporarily unavailable. Retry the download without paying again."}"#,
),
));
}
}
}
// Parse Range header for streaming support // Parse Range header for streaming support
let range = headers let range = headers
.get("range") .get("range")
@@ -162,11 +202,33 @@ impl ApiHandler {
r#"{"error":"This file is shared with the host's federation peers only. Federate with that node (exchange invites) so it recognizes you, then try again."}"#, r#"{"error":"This file is shared with the host's federation peers only. Federate with that node (exchange invites) so it recognizes you, then try again."}"#,
), ),
)), )),
Ok(content_server::ServeResult::NotFound) | Err(_) => Ok(build_response( Ok(content_server::ServeResult::Unavailable) => Ok(build_response(
StatusCode::SERVICE_UNAVAILABLE,
"application/json",
hyper::Body::from(
r#"{"error":"The seller's node can't read this file right now. This request did not redeem an ecash payment."}"#,
),
)),
Ok(content_server::ServeResult::RangeNotSatisfiable(total)) => Ok(Response::builder()
.status(StatusCode::RANGE_NOT_SATISFIABLE)
.header("Content-Range", format!("bytes */{total}"))
.body(hyper::Body::empty())
.unwrap()),
Ok(content_server::ServeResult::NotFound) => Ok(build_response(
StatusCode::NOT_FOUND, StatusCode::NOT_FOUND,
"text/plain", "text/plain",
hyper::Body::from("Content not found"), hyper::Body::from("Content not found"),
)), )),
// Not a 404: a paid request may already have been charged by the
// time this fails, and "not found" hid the real error entirely.
Err(e) => {
tracing::error!("Serving content {content_id} failed: {e:#}");
Ok(build_response(
StatusCode::INTERNAL_SERVER_ERROR,
"text/plain",
hyper::Body::from("Failed to serve content"),
))
}
} }
} }
@@ -227,7 +289,13 @@ impl ApiHandler {
.await .await
{ {
Ok((bolt11, payment_hash)) if !payment_hash.is_empty() => { Ok((bolt11, payment_hash)) if !payment_hash.is_empty() => {
crate::content_invoice::record_pending(&payment_hash, content_id, price_sats).await; crate::content_invoice::record_pending(
&self.config.data_dir,
&payment_hash,
content_id,
price_sats,
)
.await?;
let body = serde_json::json!({ let body = serde_json::json!({
"bolt11": bolt11, "bolt11": bolt11,
"payment_hash": payment_hash, "payment_hash": payment_hash,
@@ -268,52 +336,10 @@ impl ApiHandler {
&self, &self,
path: &str, path: &str,
) -> Result<Response<hyper::Body>> { ) -> Result<Response<hyper::Body>> {
let rest = path.strip_prefix("/content/").unwrap_or(""); Ok(invoice_status_response(path, |hash, id| async move {
let (content_id, payment_hash) = match rest.split_once("/invoice-status/") { self.rpc_handler.settle_content_invoice(&hash, &id).await
Some((id, hash)) => (id, hash), })
None => { .await)
return Ok(build_response(
StatusCode::BAD_REQUEST,
"text/plain",
hyper::Body::from("Invalid request"),
))
}
};
if content_id.is_empty() || !is_valid_app_id(content_id) || payment_hash.is_empty() {
return Ok(build_response(
StatusCode::BAD_REQUEST,
"text/plain",
hyper::Body::from("Invalid request"),
));
}
// The hash must be one we issued for exactly this content item.
match crate::content_invoice::lookup(payment_hash).await {
Some((cid, _)) if cid == content_id => {}
_ => {
return Ok(build_response(
StatusCode::NOT_FOUND,
"application/json",
hyper::Body::from(r#"{"error":"Unknown invoice"}"#),
))
}
}
// Already paid? Otherwise ask our LND and persist the result.
let mut paid = crate::content_invoice::is_paid_for(payment_hash, content_id).await;
if !paid {
if let Ok(true) = self.rpc_handler.invoice_is_settled(payment_hash).await {
crate::content_invoice::mark_paid(payment_hash).await;
paid = true;
}
}
let body = serde_json::json!({ "paid": paid });
Ok(build_response(
StatusCode::OK,
"application/json",
hyper::Body::from(serde_json::to_vec(&body).unwrap_or_default()),
))
} }
/// Seller side (#46): issue a fresh on-chain address for a paid catalog item /// Seller side (#46): issue a fresh on-chain address for a paid catalog item
@@ -367,7 +393,13 @@ impl ApiHandler {
match self.rpc_handler.new_onchain_address().await { match self.rpc_handler.new_onchain_address().await {
Ok(address) if !address.is_empty() => { Ok(address) if !address.is_empty() => {
crate::content_invoice::record_pending(&address, content_id, price_sats).await; crate::content_invoice::record_pending(
&self.config.data_dir,
&address,
content_id,
price_sats,
)
.await?;
let body = serde_json::json!({ let body = serde_json::json!({
"address": address, "address": address,
"amount_sats": price_sats, "amount_sats": price_sats,
@@ -417,7 +449,7 @@ impl ApiHandler {
)); ));
} }
// The address must be one we issued for exactly this content item. // The address must be one we issued for exactly this content item.
let price = match crate::content_invoice::lookup(address).await { let price = match crate::content_invoice::lookup(&self.config.data_dir, address).await? {
Some((cid, price)) if cid == content_id => price, Some((cid, price)) if cid == content_id => price,
_ => { _ => {
return Ok(build_response( return Ok(build_response(
@@ -428,10 +460,11 @@ impl ApiHandler {
} }
}; };
let mut paid = crate::content_invoice::is_paid_for(address, content_id).await; let mut paid =
crate::content_invoice::is_paid_for(&self.config.data_dir, address, content_id).await;
if !paid { if !paid {
if let Ok(true) = self.rpc_handler.onchain_received(address, price).await { if let Ok(true) = self.rpc_handler.onchain_received(address, price).await {
crate::content_invoice::mark_paid(address).await; crate::content_invoice::mark_paid(&self.config.data_dir, address).await?;
paid = true; paid = true;
} }
} }
@@ -509,3 +542,109 @@ impl ApiHandler {
} }
} }
} }
/// Keep invalid input and an unavailable wallet inside the HTTP protocol so
/// buyers can retry delivery without treating a dropped socket as lost payment.
async fn invoice_status_response<F, Fut>(path: &str, settle: F) -> Response<hyper::Body>
where
F: FnOnce(String, String) -> Fut,
Fut: std::future::Future<Output = Result<bool>>,
{
let parsed = path
.strip_prefix("/content/")
.and_then(|rest| rest.split_once("/invoice-status/"))
.filter(|(id, hash)| {
!id.is_empty()
&& is_valid_app_id(id)
&& hash.len() == 64
&& hash.bytes().all(|c| c.is_ascii_hexdigit())
});
let Some((id, hash)) = parsed else {
return build_response(
StatusCode::BAD_REQUEST,
"application/json",
hyper::Body::from(r#"{"error":"Invalid content ID or payment hash"}"#),
);
};
match settle(hash.to_ascii_lowercase(), id.to_owned()).await {
Ok(paid) => build_response(
StatusCode::OK,
"application/json",
hyper::Body::from(serde_json::json!({"paid": paid}).to_string()),
),
Err(_) => {
tracing::warn!("Peer-file payment status verification is temporarily unavailable");
let mut response = build_response(
StatusCode::SERVICE_UNAVAILABLE,
"application/json",
hyper::Body::from(
r#"{"error":"Payment verification is temporarily unavailable. Retry without paying again."}"#,
),
);
response.headers_mut().insert(
hyper::header::RETRY_AFTER,
hyper::header::HeaderValue::from_static("5"),
);
response
}
}
}
#[cfg(test)]
mod invoice_status_tests {
use super::*;
#[tokio::test]
async fn malformed_requests_do_not_query_the_wallet() {
for path in [
"/bad",
"/content//invoice-status/aa",
"/content/file/invoice-status/aa",
"/content/file/invoice-status/",
"/content/file/invoice-status/not-a-hash",
] {
let response = invoice_status_response(path, |_, _| async {
panic!("Invalid request reached wallet");
#[allow(unreachable_code)]
Ok(false)
})
.await;
assert_eq!(response.status(), StatusCode::BAD_REQUEST);
assert_eq!(response.headers()["content-type"], "application/json");
let body = hyper::body::to_bytes(response.into_body()).await.unwrap();
assert!(
serde_json::from_slice::<serde_json::Value>(&body).unwrap()["error"].is_string()
);
}
}
#[tokio::test]
async fn settlement_results_and_failures_have_explicit_http_responses() {
let hash = "AB".repeat(32);
let path = format!("/content/file/invoice-status/{hash}");
for paid in [false, true] {
let response = invoice_status_response(&path, |hash, id| async move {
assert_eq!(hash, "ab".repeat(32));
assert_eq!(id, "file");
Ok(paid)
})
.await;
assert_eq!(response.status(), StatusCode::OK);
let body = hyper::body::to_bytes(response.into_body()).await.unwrap();
assert_eq!(
serde_json::from_slice::<serde_json::Value>(&body).unwrap()["paid"],
paid
);
}
let response = invoice_status_response(&path, |_, _| async {
anyhow::bail!("private wallet details must not escape")
})
.await;
assert_eq!(response.status(), StatusCode::SERVICE_UNAVAILABLE);
assert_eq!(response.headers()["retry-after"], "5");
let body = hyper::body::to_bytes(response.into_body()).await.unwrap();
let text = String::from_utf8(body.to_vec()).unwrap();
assert!(text.contains("without paying again"));
assert!(!text.contains("private wallet"));
}
}
+13
View File
@@ -138,6 +138,19 @@ impl ApiHandler {
cors_origin: &str, cors_origin: &str,
) -> Result<Response<hyper::Body>> { ) -> Result<Response<hyper::Body>> {
let suffix = path.strip_prefix("/proxy/lnd").unwrap_or("/"); let suffix = path.strip_prefix("/proxy/lnd").unwrap_or("/");
if suffix == "/archy-status" {
return Ok(Response::builder()
.status(StatusCode::OK)
.header("Content-Type", "application/json")
.header("Cache-Control", "no-store")
.header("Access-Control-Allow-Origin", cors_origin)
.header("Access-Control-Allow-Credentials", "true")
.header("Vary", "Origin")
.body(hyper::Body::from(
rpc.handle_lnd_readiness().await.to_string(),
))?);
}
let url = format!("{LND_REST_BASE_URL}{suffix}"); let url = format!("{LND_REST_BASE_URL}{suffix}");
// LND REST serves a self-signed cert and requires the admin macaroon. // LND REST serves a self-signed cert and requires the admin macaroon.
// A bare reqwest::get() uses the default client, which rejects the // A bare reqwest::get() uses the default client, which rejects the
+1 -1
View File
@@ -136,7 +136,7 @@ impl RpcHandler {
/// ~30% of UI calls error out even though the node is perfectly healthy. /// ~30% of UI calls error out even though the node is perfectly healthy.
/// With retry + backoff, the UI sees a uniform slow-but-successful /// With retry + backoff, the UI sees a uniform slow-but-successful
/// response instead of intermittent failures. /// response instead of intermittent failures.
async fn bitcoin_rpc_call<T: serde::de::DeserializeOwned>( pub(in crate::api::rpc) async fn bitcoin_rpc_call<T: serde::de::DeserializeOwned>(
&self, &self,
client: &reqwest::Client, client: &reqwest::Client,
method: &str, method: &str,
+252 -152
View File
@@ -22,9 +22,9 @@ const FILE_CATALOG_PROTOCOL: &str = "https://archipelago.dev/protocols/file-cata
/// Best-effort reclaim of an ecash payment token that was minted but the sale /// Best-effort reclaim of an ecash payment token that was minted but the sale
/// didn't complete (seller unreachable or couldn't redeem it), so the buyer /// didn't complete (seller unreachable or couldn't redeem it), so the buyer
/// doesn't lose the value. For Fedimint the spender can reissue its own /// doesn't lose the value. For Fedimint the spender can reissue its own
/// un-redeemed notes; for Cashu the proofs are received back. Fails silently if /// un-redeemed notes; for Cashu the proofs are received back. Report the actual
/// the seller already claimed the token (then the value is genuinely gone). /// recovered amount, or explicitly say when a refund could not be confirmed.
async fn reclaim_spent_ecash(data_dir: &std::path::Path, token: &str, backend: &str) { async fn reclaim_spent_ecash(data_dir: &std::path::Path, token: &str, backend: &str) -> String {
let res = match backend { let res = match backend {
"fedimint" => crate::wallet::fedimint_client::reissue_into_any(data_dir, token) "fedimint" => crate::wallet::fedimint_client::reissue_into_any(data_dir, token)
.await .await
@@ -32,16 +32,120 @@ async fn reclaim_spent_ecash(data_dir: &std::path::Path, token: &str, backend: &
_ => ecash::receive_token(data_dir, token).await, _ => ecash::receive_token(data_dir, token).await,
}; };
match res { match res {
Ok(sats) => tracing::info!( Ok(sats) => {
"paid download: reclaimed {sats} sats of unspent {backend} ecash after a failed sale" tracing::info!("paid download: reclaimed {sats} sats after failed sale");
), format!("Refunded {sats} sats to your wallet.")
Err(e) => tracing::warn!( }
"paid download: could not reclaim {backend} ecash (the peer may have already \ Err(e) => {
claimed it): {e:#}" tracing::warn!("paid download: refund not confirmed: {e}");
), "Your refund could not be confirmed. The seller may have received the payment. Do not pay again until this is checked.".to_string()
}
} }
} }
/// Only pass through the peer's bounded, printable explanation; refund status
/// is always determined locally and must never come from the peer's wording.
fn seller_error_message(status: reqwest::StatusCode, body: &str) -> String {
let reason = serde_json::from_str::<serde_json::Value>(body)
.ok()
.and_then(|v| v.get("error").and_then(|e| e.as_str()).map(str::to_owned));
match reason {
Some(reason) if !reason.trim().is_empty() => {
let clean: String = reason
.chars()
.filter(|c| !c.is_control())
.take(240)
.collect();
format!("Seller response ({status}): {clean}")
}
_ => format!("Peer returned an error ({status})."),
}
}
/// Keep first purchases and cached repeats compatible with both existing clients.
fn paid_content_response(bytes: &[u8], mime: &str, paid_sats: u64) -> serde_json::Value {
use base64::Engine;
let data = base64::engine::general_purpose::STANDARD.encode(bytes);
serde_json::json!({
"data": data, "data_base64": data,
"size": bytes.len(), "size_bytes": bytes.len(),
"mime_type": mime, "paid_sats": paid_sats, "owned": true,
})
}
// Resolve known purchases BEFORE any mint/spend. Missing bytes or an unreadable
// index require recovery; neither is authorization to charge the buyer again.
async fn existing_paid_content(
data_dir: &std::path::Path,
onion: &str,
content_id: &str,
filename: Option<&str>,
) -> Result<Option<serde_json::Value>> {
let owned = crate::content_owned::list_owned_checked(data_dir)
.await
.context("Could not verify previous purchases; no new payment was sent")?;
let Some(item) = owned.iter().find(|o| {
o.onion == onion
&& (o.content_id == content_id
|| filename.is_some_and(|f| {
!f.is_empty() && o.filename.trim_start_matches('/') == f.trim_start_matches('/')
}))
}) else {
return Ok(None);
};
let (mime, bytes) = crate::content_owned::read_owned(data_dir, &item.onion, &item.content_id)
.await.context("This purchase is recorded, but its cached file is unavailable. No new payment was sent. Restore the cached file or contact the seller.")?;
let mut response = paid_content_response(&bytes, &mime, 0);
response["already_owned"] = serde_json::json!(true);
response["filename"] = serde_json::json!(item.filename);
Ok(Some(response))
}
// Updated clients open the persisted file through the Range-capable HTTP
// endpoint. Avoid putting two base64 copies of a large video in a JSON reply.
// Keep older clients compatible until both sides have upgraded.
fn invoice_download_response(bytes: &[u8], mime: &str, cache_only: bool) -> serde_json::Value {
if cache_only {
serde_json::json!({ "owned": true, "mime_type": mime, "size_bytes": bytes.len() })
} else {
paid_content_response(bytes, mime, 0)
}
}
/// File purchases through an atomic no-clobber write in Files' own namespace.
async fn file_purchase_in_files(
data_dir: &std::path::Path,
filename: &str,
mime: &str,
bytes: &[u8],
) -> Result<String> {
let folder = if mime.starts_with("image/") || mime.starts_with("video/") {
"Photos"
} else if mime.starts_with("audio/") {
"Music"
} else {
"Documents"
};
let root = data_dir.join("filebrowser");
anyhow::ensure!(
tokio::fs::metadata(&root).await?.is_dir(),
"Files storage is unavailable"
);
let name = std::path::Path::new(filename)
.file_name()
.and_then(|n| n.to_str())
.filter(|n| !n.is_empty())
.unwrap_or("download");
let path =
crate::container::filebrowser::save_new_file(&root.join(folder), name, bytes).await?;
Ok(format!(
"{folder}/{}",
path.file_name()
.and_then(|n| n.to_str())
.context("Invalid Files name")?
))
}
impl RpcHandler { impl RpcHandler {
/// List content I'm sharing. /// List content I'm sharing.
pub(super) async fn handle_content_list_mine(&self) -> Result<serde_json::Value> { pub(super) async fn handle_content_list_mine(&self) -> Result<serde_json::Value> {
@@ -441,43 +545,15 @@ impl RpcHandler {
// by exact (onion, content_id) and by (onion, filename) — the latter // by exact (onion, content_id) and by (onion, filename) — the latter
// catches duplicate ids pointing at the same file on the same // catches duplicate ids pointing at the same file on the same
// seller. The owned copy is served from the local cache instead. // seller. The owned copy is served from the local cache instead.
if let Some(cached) = existing_paid_content(
&self.config.data_dir,
onion,
content_id,
params.get("filename").and_then(|v| v.as_str()),
)
.await?
{ {
let filename = params.get("filename").and_then(|v| v.as_str()); return Ok(cached);
let owned = crate::content_owned::list_owned(&self.config.data_dir).await;
let already = owned.iter().find(|o| {
o.onion == onion
&& (o.content_id == content_id
|| filename.is_some_and(|f| {
!f.is_empty()
&& o.filename.trim_start_matches('/') == f.trim_start_matches('/')
}))
});
if let Some(o) = already {
tracing::info!(
onion,
content_id,
owned_as = %o.content_id,
"paid download: already owned — serving cached copy, NOT paying again"
);
if let Some((mime, bytes)) =
crate::content_owned::read_owned(&self.config.data_dir, &o.onion, &o.content_id)
.await
{
use base64::Engine;
return Ok(serde_json::json!({
"owned": true,
"already_owned": true,
"filename": o.filename,
"mime_type": mime,
"size_bytes": bytes.len(),
"paid_sats": 0,
"data_base64":
base64::engine::general_purpose::STANDARD.encode(&bytes),
}));
}
// Cache record exists but bytes are gone — fall through and
// repurchase rather than stranding the user.
}
} }
// `method` pins the backend the user confirmed in the UI ("cashu" | // `method` pins the backend the user confirmed in the UI ("cashu" |
@@ -545,31 +621,30 @@ impl RpcHandler {
let path = format!("/content/{}", content_id); let path = format!("/content/{}", content_id);
// Surface a real reason instead of the generic sanitized error (#30): // Surface a real reason instead of the generic sanitized error (#30):
// the dial already tries FIPS/mesh then falls back to Tor, so a failure // A bearer token must not be replayed after an ambiguous delivery.
// here means the peer is genuinely unreachable on both transports. // A transport error can mean the seller received it without replying.
let (response, transport) = match crate::fips::dial::PeerRequest::new( let (response, transport) =
fips_npub.as_deref(), match crate::fips::dial::PeerRequest::new(fips_npub.as_deref(), onion, &path)
onion, .service(crate::settings::transport::PeerService::PeerFiles)
&path, .header("X-Federation-DID", local_did)
) .header("X-Payment-Token", token_str.clone())
.service(crate::settings::transport::PeerService::PeerFiles) .single_delivery()
.header("X-Federation-DID", local_did) .timeout(std::time::Duration::from_secs(900))
.header("X-Payment-Token", token_str.clone()) .send_get()
.timeout(std::time::Duration::from_secs(900)) .await
.send_get() {
.await Ok(v) => v,
{ Err(e) => {
Ok(v) => v, tracing::warn!("paid peer download dial failed for {}: {:#}", onion, e);
Err(e) => { // The token was already minted/spent — reclaim it so the buyer
tracing::warn!("paid peer download dial failed for {}: {:#}", onion, e); // doesn't lose the value when the seller was simply unreachable.
// The token was already minted/spent — reclaim it so the buyer let refund =
// doesn't lose the value when the seller was simply unreachable. reclaim_spent_ecash(&self.config.data_dir, &token_str, used_backend).await;
reclaim_spent_ecash(&self.config.data_dir, &token_str, used_backend).await; return Ok(serde_json::json!({
return Ok(serde_json::json!({ "error": format!("The purchase could not be completed. {refund}")
"error": "Could not reach the peer over mesh or Tor — it may be offline. Your ecash was refunded to your wallet. Please try again." }));
})); }
} };
};
// Record which transport actually reached the peer (B14). // Record which transport actually reached the peer (B14).
if let Err(e) = crate::federation::record_peer_transport( if let Err(e) = crate::federation::record_peer_transport(
&self.config.data_dir, &self.config.data_dir,
@@ -583,25 +658,17 @@ impl RpcHandler {
} }
if response.status() == reqwest::StatusCode::PAYMENT_REQUIRED { if response.status() == reqwest::StatusCode::PAYMENT_REQUIRED {
// Payment was rejected by the seller. Surface the most likely cause // A 402 can mean mint validation, network failure, underpayment,
// per backend — for ecash both sides must share a redemption network // or an unaccepted mint. Do not invent a mint-mismatch diagnosis.
// (a Cashu mint, or a Fedimint federation).
let body = response.text().await.unwrap_or_default(); let body = response.text().await.unwrap_or_default();
tracing::warn!( tracing::warn!(
"paid download: seller {onion} rejected {used_backend} payment of {price_sats} sats: {body}" "paid download: seller {onion} rejected {used_backend} payment of {price_sats} sats: {body}"
); );
// Seller couldn't redeem the token — reclaim it so the buyer keeps // Seller couldn't redeem the token — reclaim it so the buyer keeps
// their funds (the spent-but-unredeemed-notes case the user hit). // their funds (the spent-but-unredeemed-notes case the user hit).
reclaim_spent_ecash(&self.config.data_dir, &token_str, used_backend).await; let refund = reclaim_spent_ecash(&self.config.data_dir, &token_str, used_backend).await;
let hint = match used_backend {
"fedimint" => "the seller isn't in the same Fedimint federation as you",
_ => "the seller doesn't accept your Cashu mint",
};
return Ok(serde_json::json!({ return Ok(serde_json::json!({
"error": format!( "error": format!("The seller could not verify the payment. {refund}")
"Payment rejected by the seller — {hint}. Your ecash was refunded to \
your wallet. Try the other ecash type, or use a shared mint/federation."
)
})); }));
} }
@@ -609,9 +676,9 @@ impl RpcHandler {
let status = response.status(); let status = response.status();
let body = response.text().await.unwrap_or_default(); let body = response.text().await.unwrap_or_default();
tracing::warn!("paid download: seller {onion} returned {status}: {body}"); tracing::warn!("paid download: seller {onion} returned {status}: {body}");
reclaim_spent_ecash(&self.config.data_dir, &token_str, used_backend).await; let refund = reclaim_spent_ecash(&self.config.data_dir, &token_str, used_backend).await;
return Ok(serde_json::json!({ return Ok(serde_json::json!({
"error": format!("Peer returned an error ({status}). Your ecash was refunded to your wallet.") "error": format!("{} {refund}", seller_error_message(status, &body))
})); }));
} }
@@ -625,10 +692,17 @@ impl RpcHandler {
.filter(|s| !s.is_empty()) .filter(|s| !s.is_empty())
.unwrap_or_else(|| "application/octet-stream".to_string()); .unwrap_or_else(|| "application/octet-stream".to_string());
let bytes = response let bytes = match response.bytes().await {
.bytes() Ok(bytes) => bytes,
.await Err(error) => {
.context("Failed to read response body")?; tracing::warn!("paid download: response body failed: {error}");
let refund =
reclaim_spent_ecash(&self.config.data_dir, &token_str, used_backend).await;
return Ok(serde_json::json!({
"error": format!("The file transfer was interrupted after payment was sent. {refund}")
}));
}
};
// Persist the purchase so it "stays unlocked" for this buyer: cache the // Persist the purchase so it "stays unlocked" for this buyer: cache the
// bytes + metadata keyed by (onion, content_id). The gallery then renders // bytes + metadata keyed by (onion, content_id). The gallery then renders
@@ -658,63 +732,21 @@ impl RpcHandler {
tracing::warn!("paid download: failed to cache purchased content (non-fatal): {e:#}"); tracing::warn!("paid download: failed to cache purchased content (non-fatal): {e:#}");
} }
// Auto-file the purchase into the user's Files area (2026-07-22): // The durable purchased-content cache above is primary. A Files copy
// Photos for images/video, Music for audio, Documents otherwise — // remains optional: a stopped FileBrowser must not undo a paid download.
// same buckets the Cloud view uses. The in-app viewer still plays let filed =
// from the purchase cache; this makes the file ALSO show up where file_purchase_in_files(&self.config.data_dir, &filename, &mime_type, &bytes).await;
// files live, on every device, without relying on a browser match filed {
// download. Best-effort: never fail a paid download over it. Ok(path) => tracing::info!("paid download: filed into Files/{path}"),
{ Err(error) => tracing::warn!(
let folder = if mime_type.starts_with("image/") || mime_type.starts_with("video/") { "paid download: optional Files copy failed; purchase cache retained: {error}"
"Photos" ),
} else if mime_type.starts_with("audio/") {
"Music"
} else {
"Documents"
};
let base = std::path::Path::new(&filename)
.file_name()
.and_then(|n| n.to_str())
.unwrap_or("download")
.to_string();
let dir = self.config.data_dir.join("filebrowser").join(folder);
if let Err(e) = tokio::fs::create_dir_all(&dir).await {
tracing::warn!("paid download: cannot create {}: {e}", dir.display());
} else {
// Don't clobber an existing file of the same name: "x.jpg"
// → "x (2).jpg" etc.
let mut target = dir.join(&base);
let (stem, ext) = match base.rsplit_once('.') {
Some((s, e)) if !s.is_empty() => (s.to_string(), format!(".{e}")),
_ => (base.clone(), String::new()),
};
let mut n = 2;
while target.exists() {
target = dir.join(format!("{stem} ({n}){ext}"));
n += 1;
}
match tokio::fs::write(&target, &bytes).await {
Ok(()) => tracing::info!("paid download: filed into {}", target.display()),
Err(e) => tracing::warn!(
"paid download: filing into {} failed (non-fatal): {e}",
target.display()
),
}
}
} }
use base64::Engine;
let encoded = base64::engine::general_purpose::STANDARD.encode(&bytes);
tracing::info!("paid download: received {} bytes from {onion} (paid {price_sats} sats via {used_backend})", bytes.len()); tracing::info!("paid download: received {} bytes from {onion} (paid {price_sats} sats via {used_backend})", bytes.len());
Ok(serde_json::json!({ let mut result = paid_content_response(&bytes, &mime_type, price_sats);
"data": encoded, result["ecash_backend"] = serde_json::json!(used_backend);
"size": bytes.len(), Ok(result)
"paid_sats": price_sats,
"ecash_backend": used_backend,
"mime_type": mime_type,
"owned": true,
}))
} }
/// Buyer side (#46): ask the selling node to mint a Lightning invoice for a /// Buyer side (#46): ask the selling node to mint a Lightning invoice for a
@@ -856,10 +888,29 @@ impl RpcHandler {
if !is_valid_v3_onion(onion) { if !is_valid_v3_onion(onion) {
return Err(anyhow::anyhow!("Invalid v3 onion address")); return Err(anyhow::anyhow!("Invalid v3 onion address"));
} }
if payment_hash.is_empty() || !payment_hash.chars().all(|c| c.is_ascii_hexdigit()) { if payment_hash.len() != 64 || !payment_hash.chars().all(|c| c.is_ascii_hexdigit()) {
return Err(anyhow::anyhow!("Invalid payment_hash")); return Err(anyhow::anyhow!("Invalid payment_hash"));
} }
let cache_only = params
.get("cache_only")
.and_then(|v| v.as_bool())
.unwrap_or(false);
if let Some((mime, bytes)) =
crate::content_owned::read_owned(&self.config.data_dir, onion, content_id).await
{
return Ok(invoice_download_response(&bytes, &mime, cache_only));
}
// Older sellers only mark settlement during status polling. Always
// perform that handshake before requesting bytes; retries never pay.
// The download gate remains authoritative: a file may have become
// free, and newer sellers verify directly if status polling fails.
let _ = self
.handle_content_invoice_status(Some(serde_json::json!({
"onion": onion, "content_id": content_id, "payment_hash": payment_hash,
})))
.await;
let (data, _) = self.state_manager.get_snapshot().await; let (data, _) = self.state_manager.get_snapshot().await;
let local_did = crate::identity::did_key_from_pubkey_hex(&data.server_info.pubkey)?; let local_did = crate::identity::did_key_from_pubkey_hex(&data.server_info.pubkey)?;
let fips_npub = crate::federation::fips_npub_for_onion(&self.config.data_dir, onion).await; let fips_npub = crate::federation::fips_npub_for_onion(&self.config.data_dir, onion).await;
@@ -898,7 +949,7 @@ impl RpcHandler {
if response.status() == reqwest::StatusCode::PAYMENT_REQUIRED { if response.status() == reqwest::StatusCode::PAYMENT_REQUIRED {
return Ok(serde_json::json!({ return Ok(serde_json::json!({
"error": "Seller has not registered this payment yet — wait for settlement and retry." "error": "The seller has not confirmed access yet. Retry the download without paying again."
})); }));
} }
if !response.status().is_success() { if !response.status().is_success() {
@@ -907,16 +958,45 @@ impl RpcHandler {
})); }));
} }
let mime = response
.headers()
.get(reqwest::header::CONTENT_TYPE)
.and_then(|v| v.to_str().ok())
.unwrap_or("application/octet-stream")
.split(';')
.next()
.unwrap_or("application/octet-stream")
.to_string();
let bytes = response let bytes = response
.bytes() .bytes()
.await .await
.context("Failed to read response body")?; .context("Paid file transfer interrupted; retry the download without paying again")?;
use base64::Engine; let filename = params
let encoded = base64::engine::general_purpose::STANDARD.encode(&bytes); .get("filename")
Ok(serde_json::json!({ .and_then(|v| v.as_str())
"data": encoded, .unwrap_or(content_id);
"size": bytes.len(), crate::content_owned::record_purchase(
})) &self.config.data_dir,
onion,
content_id,
filename,
&mime,
&bytes,
params
.get("price_sats")
.and_then(|v| v.as_u64())
.unwrap_or(0),
"lightning",
&chrono::Utc::now().to_rfc3339(),
)
.await
.context("Paid file could not be saved; retry the download without paying again")?;
if let Err(error) =
file_purchase_in_files(&self.config.data_dir, filename, &mime, &bytes).await
{
tracing::warn!("Lightning purchase cached; optional Files copy failed: {error:#}");
}
Ok(invoice_download_response(&bytes, &mime, cache_only))
} }
/// Buyer side (#46): ask the seller for a fresh on-chain address to pay. /// Buyer side (#46): ask the seller for a fresh on-chain address to pay.
@@ -1387,3 +1467,23 @@ impl RpcHandler {
} }
} }
} }
#[cfg(test)]
#[path = "content_tests.rs"]
mod tests;
#[cfg(test)]
mod invoice_delivery_response_tests {
use super::*;
#[test]
fn cached_delivery_avoids_base64_but_keeps_old_clients_compatible() {
let cached = invoice_download_response(b"paid bytes", "video/mp4", true);
assert_eq!(cached["owned"], true);
assert_eq!(cached["size_bytes"], 10);
assert!(cached.get("data").is_none());
assert!(cached.get("data_base64").is_none());
let legacy = invoice_download_response(b"paid bytes", "video/mp4", false);
assert_eq!(legacy["data"], "cGFpZCBieXRlcw==");
assert_eq!(legacy["data"], legacy["data_base64"]);
}
}
@@ -0,0 +1,138 @@
use super::*;
#[test]
fn first_and_cached_paid_downloads_have_the_same_client_payload_contract() {
use base64::Engine;
for paid in [0, 1] {
let response = paid_content_response(&[0, 255, 123], "application/octet-stream", paid);
assert_eq!(response["data"], response["data_base64"]);
assert_eq!(
base64::engine::general_purpose::STANDARD
.decode(response["data"].as_str().unwrap())
.unwrap(),
[0, 255, 123]
);
assert_eq!(response["size"], 3);
assert_eq!(response["size_bytes"], 3);
assert_eq!(response["paid_sats"], paid);
assert_eq!(response["owned"], true);
}
}
#[tokio::test]
async fn files_copy_routes_media_and_sanitizes_the_filename() {
let dir = tempfile::tempdir().unwrap();
tokio::fs::create_dir(dir.path().join("filebrowser"))
.await
.unwrap();
for (mime, folder) in [
("image/png", "Photos"),
("video/mp4", "Photos"),
("audio/mpeg", "Music"),
("text/plain", "Documents"),
] {
let relative = file_purchase_in_files(dir.path(), "../name #?.bin", mime, b"paid")
.await
.unwrap();
assert!(relative.starts_with(&format!("{folder}/name #?")));
assert_eq!(
tokio::fs::read(dir.path().join("filebrowser").join(relative))
.await
.unwrap(),
b"paid"
);
}
}
#[tokio::test]
async fn unavailable_files_storage_is_reported_without_creating_a_fake_installation() {
let dir = tempfile::tempdir().unwrap();
assert!(
file_purchase_in_files(dir.path(), "name", "text/plain", b"bytes")
.await
.is_err()
);
assert!(!dir.path().join("filebrowser").exists());
}
#[test]
fn seller_errors_are_bounded_printable_and_identified_as_peer_text() {
let status = reqwest::StatusCode::SERVICE_UNAVAILABLE;
let message = seller_error_message(status, r#"{"error":"Cannot read file\n\u0000"}"#);
assert!(message.starts_with("Seller response (503"));
assert!(message.ends_with("Cannot read file"));
assert!(!message.contains('\n') && !message.contains('\0'));
let body = serde_json::json!({"error": "é".repeat(1000)}).to_string();
assert!(seller_error_message(status, &body).chars().count() < 300);
for body in ["not JSON", r#"{"error": 7}"#, r#"{"error":" "}"#] {
assert_eq!(
seller_error_message(status, body),
"Peer returned an error (503 Service Unavailable)."
);
}
}
#[tokio::test]
async fn known_purchase_never_becomes_a_new_spend_when_cache_or_index_is_unavailable() {
let dir = tempfile::tempdir().unwrap();
assert!(
existing_paid_content(dir.path(), "seller.onion", "id", None)
.await
.unwrap()
.is_none()
);
crate::content_owned::record_purchase(
dir.path(),
"seller.onion",
"id",
"file.txt",
"text/plain",
b"paid",
1,
"cashu",
"now",
)
.await
.unwrap();
for (id, filename) in [("id", None), ("duplicate-id", Some("/file.txt"))] {
let cached = existing_paid_content(dir.path(), "seller.onion", id, filename)
.await
.unwrap()
.unwrap();
assert_eq!(cached["paid_sats"], 0);
assert_eq!(cached["already_owned"], true);
assert_eq!(cached["data"], "cGFpZA==");
}
assert!(
existing_paid_content(dir.path(), "different.onion", "id", None)
.await
.unwrap()
.is_none()
);
tokio::fs::remove_file(dir.path().join("purchased-content/seller.onion/id"))
.await
.unwrap();
assert!(
existing_paid_content(dir.path(), "seller.onion", "id", None)
.await
.unwrap_err()
.to_string()
.contains("No new payment")
);
tokio::fs::write(dir.path().join("purchased-content/owned.json"), b"damaged")
.await
.unwrap();
assert!(
existing_paid_content(dir.path(), "seller.onion", "other-id", None)
.await
.unwrap_err()
.to_string()
.contains("no new payment")
);
assert_eq!(
tokio::fs::read(dir.path().join("purchased-content/owned.json"))
.await
.unwrap(),
b"damaged"
);
}
@@ -132,6 +132,9 @@ impl RpcHandler {
"lnd.newaddress" => self.handle_lnd_newaddress().await, "lnd.newaddress" => self.handle_lnd_newaddress().await,
"lnd.sendcoins" => self.handle_lnd_sendcoins(params).await, "lnd.sendcoins" => self.handle_lnd_sendcoins(params).await,
"lnd.estimatefee" => self.handle_lnd_estimatefee(params).await, "lnd.estimatefee" => self.handle_lnd_estimatefee(params).await,
"lnd.bump-quote" => self.handle_lnd_bump_quote(params).await,
"lnd.bump-submit" => self.handle_lnd_bump_submit(params).await,
"lnd.bump-status" => self.handle_lnd_bump_status(params).await,
"lnd.createinvoice" => self.handle_lnd_createinvoice(params).await, "lnd.createinvoice" => self.handle_lnd_createinvoice(params).await,
"lnd.invoicestatus" => self.handle_lnd_invoicestatus(params).await, "lnd.invoicestatus" => self.handle_lnd_invoicestatus(params).await,
"lnd.payinvoice" => self.handle_lnd_payinvoice(params).await, "lnd.payinvoice" => self.handle_lnd_payinvoice(params).await,
@@ -1,6 +1,8 @@
use super::*; use super::*;
use crate::api::rpc::RpcHandler; use crate::api::rpc::RpcHandler;
use crate::identity_manager::{IdentityManager, IdentityProfile, IdentityPurpose}; use crate::identity_manager::{
is_node_identity, IdentityManager, IdentityProfile, IdentityPurpose,
};
use crate::network::did_dht; use crate::network::did_dht;
use anyhow::{Context, Result}; use anyhow::{Context, Result};
use nostr_sdk::ToBech32; use nostr_sdk::ToBech32;
@@ -38,7 +40,7 @@ impl RpcHandler {
.into_iter() .into_iter()
.map(|id| { .map(|id| {
let is_default = default_id.as_deref() == Some(&id.id); let is_default = default_id.as_deref() == Some(&id.id);
let is_node = !node_pubkey_hex.is_empty() && id.pubkey_hex == node_pubkey_hex; let is_node = is_node_identity(&id, &node_pubkey_hex);
let (nostr_pubkey, nostr_npub) = if is_node { let (nostr_pubkey, nostr_npub) = if is_node {
( (
node_nostr_hex.clone().or(id.nostr_pubkey), node_nostr_hex.clone().or(id.nostr_pubkey),
+108 -26
View File
@@ -272,28 +272,8 @@ impl RpcHandler {
.and_then(|v| v.as_bool()) .and_then(|v| v.as_bool())
.unwrap_or(false); .unwrap_or(false);
// Fee control: either a confirmation target or an explicit fee rate // Omitted fees target the next block; explicit slower/custom choices win.
let target_conf = params.get("target_conf").and_then(|v| v.as_i64()); let (target_conf, sat_per_vbyte) = super::fee_policy::fee_options(&params)?;
let sat_per_vbyte = params.get("sat_per_vbyte").and_then(|v| v.as_i64());
if target_conf.is_some() && sat_per_vbyte.is_some() {
return Err(anyhow::anyhow!(
"Invalid fee parameters: specify either target_conf or sat_per_vbyte, not both"
));
}
if let Some(tc) = target_conf {
if !(1..=1008).contains(&tc) {
return Err(anyhow::anyhow!(
"Invalid target_conf: must be between 1 and 1008 blocks"
));
}
}
if let Some(rate) = sat_per_vbyte {
if !(1..=5000).contains(&rate) {
return Err(anyhow::anyhow!(
"Invalid sat_per_vbyte: must be between 1 and 5000"
));
}
}
info!( info!(
peer = pubkey, peer = pubkey,
@@ -473,6 +453,7 @@ impl RpcHandler {
)); ));
} }
let fee_query = close_channel_fee_query(&params)?;
let force = params let force = params
.get("force") .get("force")
.and_then(|v| v.as_bool()) .and_then(|v| v.as_bool())
@@ -498,13 +479,11 @@ impl RpcHandler {
.build() .build()
.context("Failed to create streaming HTTP client")?; .context("Failed to create streaming HTTP client")?;
let url = format!( let url = format!("{LND_REST_BASE_URL}/v1/channels/{}/{}", parts[0], parts[1]);
"{LND_REST_BASE_URL}/v1/channels/{}/{}?force={}",
parts[0], parts[1], force
);
let mut resp = client let mut resp = client
.delete(&url) .delete(&url)
.query(&fee_query)
.header("Grpc-Metadata-macaroon", &macaroon_hex) .header("Grpc-Metadata-macaroon", &macaroon_hex)
.send() .send()
.await .await
@@ -572,3 +551,106 @@ impl RpcHandler {
} }
} }
} }
/// LND's CloseChannel REST endpoint takes fee selection as query parameters.
/// With neither parameter LND uses a lax target; keep legacy clients on our
/// explicit next-block target rather than silently accepting that default.
fn close_channel_fee_query(params: &serde_json::Value) -> Result<Vec<(&'static str, String)>> {
let force = match params.get("force") {
None | Some(serde_json::Value::Null) => false,
Some(value) => value
.as_bool()
.ok_or_else(|| anyhow::anyhow!("force must be a boolean"))?,
};
let integer = |key: &str, max: u64| -> Result<Option<u64>> {
match params.get(key) {
None | Some(serde_json::Value::Null) => Ok(None),
Some(value) => {
let n = value
.as_u64()
.ok_or_else(|| anyhow::anyhow!("{key} must be a positive whole number"))?;
anyhow::ensure!((1..=max).contains(&n), "{key} must be between 1 and {max}");
Ok(Some(n))
}
}
};
let target = integer("target_conf", 1008)?;
let rate = integer("sat_per_vbyte", 5000)?;
anyhow::ensure!(
target.is_none() || rate.is_none(),
"Specify either target_conf or sat_per_vbyte, not both"
);
anyhow::ensure!(
!force || (target.is_none() && rate.is_none()),
"Closing fee selection requires a cooperative close"
);
let mut query = vec![("force", force.to_string())];
if !force {
if let Some(rate) = rate {
query.push(("sat_per_vbyte", rate.to_string()));
} else {
query.push((
"target_conf",
target
.unwrap_or(super::fee_policy::DEFAULT_TARGET as u64)
.to_string(),
));
}
}
Ok(query)
}
#[cfg(test)]
mod close_fee_tests {
use super::*;
#[test]
fn close_fee_query_forwards_presets_custom_and_legacy_default() {
for target in [1, 3, 6, 1008] {
assert_eq!(
close_channel_fee_query(&serde_json::json!({"target_conf":target})).unwrap(),
vec![
("force", "false".into()),
("target_conf", target.to_string())
]
);
}
for rate in [1, 25, 5000] {
let query =
close_channel_fee_query(&serde_json::json!({"sat_per_vbyte":rate})).unwrap();
let request = reqwest::Client::new()
.delete("http://localhost/v1/channels/test/0")
.query(&query)
.build()
.unwrap();
assert_eq!(request.method(), reqwest::Method::DELETE);
assert_eq!(
request.url().query(),
Some(format!("force=false&sat_per_vbyte={rate}").as_str())
);
}
assert_eq!(
close_channel_fee_query(&serde_json::json!({})).unwrap(),
vec![("force", "false".into()), ("target_conf", "1".into())]
);
assert_eq!(
close_channel_fee_query(&serde_json::json!({"force":true})).unwrap(),
vec![("force", "true".into())]
);
}
#[test]
fn malformed_or_conflicting_close_fees_fail_before_wallet_access() {
for params in [
serde_json::json!({"target_conf":1,"sat_per_vbyte":2}),
serde_json::json!({"force":true,"target_conf":1}),
serde_json::json!({"force":"false"}),
serde_json::json!({"target_conf":0}),
serde_json::json!({"target_conf":1009}),
serde_json::json!({"sat_per_vbyte":5001}),
serde_json::json!({"sat_per_vbyte":-1}),
serde_json::json!({"sat_per_vbyte":1.5}),
serde_json::json!({"sat_per_vbyte":"25"}),
] {
assert!(close_channel_fee_query(&params).is_err(), "{params}");
}
}
}
@@ -0,0 +1,835 @@
//! WalletKit BumpFee is CPFP for new wallet outputs, RBF only for sweeper inputs.
//! Never feed an ordinary payment input to it and call that a replacement.
use super::LND_REST_BASE_URL;
use crate::api::rpc::RpcHandler;
use anyhow::{bail, ensure, Context, Result};
use serde::{Deserialize, Serialize};
use serde_json::{json, Value};
use std::{collections::HashMap, path::Path, sync::LazyLock};
use tokio::{io::AsyncWriteExt, sync::Mutex};
static QUOTES: LazyLock<Mutex<HashMap<String, Quote>>> = LazyLock::new(Default::default);
// Serialize check/register/persist across dashboard clients. The create_new receipt
// additionally survives process restarts and prevents retries of ambiguous results.
static SUBMIT: Mutex<()> = Mutex::const_new(());
const QUOTE_SECONDS: u64 = 60;
#[derive(Clone, Debug, Serialize, Deserialize, PartialEq)]
struct Plan {
txid: String,
method: String,
input_txid: String,
input_index: u32,
parent_txid: String,
recipient_sats: u64,
rate_sat_vb: u64,
current_fee_sats: u64,
additional_fee_sats: u64,
total_fee_sats: u64,
budget_sats: u64,
input_sats: u64,
parent_vsize: u64,
sweep_vsize_bound: u64,
tip: String,
}
#[derive(Clone, Serialize, Deserialize)]
struct Quote {
quote_id: String,
expires_at: u64,
custom_rate: Option<u64>,
#[serde(flatten)]
plan: Plan,
}
#[derive(Serialize, Deserialize)]
struct Operation {
quote: Quote,
status: String,
message: String,
}
fn now() -> u64 {
std::time::SystemTime::now()
.duration_since(std::time::UNIX_EPOCH)
.unwrap_or_default()
.as_secs()
}
fn number(v: &Value) -> Result<u64> {
v.as_u64()
.or_else(|| v.as_str().and_then(|s| s.parse().ok()))
.context("Missing or invalid wallet amount")
}
fn txid_param(p: &Value) -> Result<String> {
let s = p["txid"].as_str().context("Missing transaction ID")?;
ensure!(
s.len() == 64 && s.bytes().all(|c| c.is_ascii_hexdigit()),
"Invalid transaction ID"
);
Ok(s.to_ascii_lowercase())
}
fn btc_sats(v: &Value) -> Result<u64> {
let n = v.as_f64().context("Missing Bitcoin fee")? * 100_000_000.0;
ensure!(
n.is_finite() && n >= 0.0 && n <= 2_100_000_000_000_000.0,
"Invalid Bitcoin fee"
);
Ok(n.round() as u64)
}
fn outpoint_matches(v: &Value, txid: &str, index: u32) -> bool {
v["txid_str"].as_str() == Some(txid) && v["output_index"].as_u64() == Some(index as u64)
}
fn array<'a>(v: &'a Value, key: &str) -> Result<&'a Vec<Value>> {
v[key]
.as_array()
.with_context(|| format!("Missing wallet field: {key}"))
}
fn sweep_size(output: &Value) -> Result<u64> {
// One native input, one wallet taproot output, including signature rounding.
match output["output_type"].as_str() {
Some("SCRIPT_TYPE_WITNESS_V1_TAPROOT") => Ok(112),
Some("SCRIPT_TYPE_WITNESS_V0_PUBKEY_HASH") => Ok(123),
_ => bail!("This output type is not supported for fee bumping yet"),
}
}
fn fee_budget(
rate: u64,
parent_size: u64,
parent_fee: u64,
size: u64,
old_fee: u64,
relay: u64,
input: u64,
) -> Result<u64> {
ensure!(
(1..=5000).contains(&rate),
"Fee rate must be a whole number from 1 to 5000 sat/vB"
);
ensure!(
parent_size <= 100_000 && size <= 100_000 && relay <= 5000,
"Unsupported package size or relay fee"
);
let required = rate * (parent_size + size);
let mut budget = required.saturating_sub(parent_fee).max(relay * size);
if old_fee > 0 {
budget = budget.max(old_fee + relay * size + 1);
}
ensure!(budget > old_fee, "Choose a higher fee rate");
// Conservative dust buffer; never attach unrelated wallet inputs to fund fees.
ensure!(
budget.checked_add(1000).is_some_and(|v| v <= input),
"Not enough wallet change for this fee; choose a lower rate"
);
Ok(budget)
}
async fn lnd(
client: &reqwest::Client,
macaroon: &str,
path: &str,
body: Option<Value>,
) -> Result<Value> {
let url = format!("{LND_REST_BASE_URL}{path}");
let req = match body {
Some(v) => client.post(url).json(&v),
None => client.get(url),
};
let response = req
.header("Grpc-Metadata-macaroon", macaroon)
.send()
.await?;
let status = response.status();
let value: Value = response.json().await.context("Invalid LND response")?;
ensure!(
status.is_success() && value.get("code").is_none(),
"{}",
value["message"].as_str().unwrap_or("LND request failed")
);
Ok(value)
}
fn validate_quote(quote: &Quote, fresh: &Plan, timestamp: u64) -> Result<()> {
ensure!(
quote.expires_at > timestamp && quote.plan == *fresh,
"Transaction or fees changed; review a fresh quote"
);
Ok(())
}
fn bump_body(plan: &Plan) -> Value {
json!({"outpoint":{"txid_str":plan.input_txid,"output_index":plan.input_index},
"sat_per_vbyte":plan.rate_sat_vb.to_string(), "budget":plan.budget_sats.to_string(),
"deadline_delta":1, "immediate":true})
}
async fn reserve(path: &Path, op: &Operation) -> Result<()> {
let parent = path.parent().context("Invalid operation path")?;
tokio::fs::create_dir_all(parent).await?;
let mut f = tokio::fs::OpenOptions::new()
.write(true)
.create_new(true)
.mode(0o600)
.open(path)
.await
.context("A bump already exists for this transaction; check its status")?;
f.write_all(&serde_json::to_vec(op)?).await?;
f.sync_all().await?;
// Sync directory entry too: a crash must not make a submitted operation vanish.
tokio::fs::File::open(parent).await?.sync_all().await?;
Ok(())
}
// Only a recorded Archy CPFP with one owned input and no external outputs may
// be folded into a payment. Labels and a fee-sized delta alone are not evidence.
fn fee_child_matches(tx: &Value, plan: &Plan) -> bool {
let input = format!("{}:{}", plan.input_txid, plan.input_index);
let amount = tx["amount"]
.as_i64()
.or_else(|| tx["amount"].as_str()?.parse().ok());
let fee = number(&tx["total_fees"])
.ok()
.and_then(|n| i64::try_from(n).ok());
tx["tx_hash"]
.as_str()
.is_some_and(|id| id.len() == 64 && id.bytes().all(|c| c.is_ascii_hexdigit()))
&& amount
.zip(fee)
.is_some_and(|(amount, fee)| fee > 0 && amount == -fee)
&& tx["previous_outpoints"].as_array().is_some_and(|inputs| {
inputs.len() == 1
&& inputs[0]["outpoint"] == input
&& inputs[0]["is_our_output"] == true
})
&& tx["output_details"].as_array().is_some_and(|outputs| {
!outputs.is_empty() && outputs.iter().all(|o| o["is_our_address"] == true)
})
}
impl RpcHandler {
pub(super) async fn group_fee_bump_history(
&self,
raw: &[Value],
normalized: &mut Vec<Value>,
client: &reqwest::Client,
) {
let mut hidden = std::collections::HashSet::new();
for parent in normalized.iter_mut() {
if parent["direction"] != "outgoing" {
continue;
}
let Some(id) = parent["tx_hash"].as_str().map(str::to_owned) else {
continue;
};
if id.len() != 64 || !id.bytes().all(|c| c.is_ascii_hexdigit()) {
continue;
}
let path = self
.config
.data_dir
.join("wallet/fee-bumps")
.join(format!("{id}.json"));
let Ok(bytes) = tokio::fs::read(path).await else {
continue;
};
let Ok(op) = serde_json::from_slice::<Operation>(&bytes) else {
continue;
};
let plan = &op.quote.plan;
if plan.method != "cpfp"
|| plan.txid != id
|| plan.parent_txid != id
|| plan.input_txid != id
{
continue;
}
let candidates: Vec<_> = raw
.iter()
.filter(|tx| fee_child_matches(tx, plan))
.collect();
let mut active = Vec::new();
for child in &candidates {
let child_id = child["tx_hash"].as_str().unwrap();
if child["num_confirmations"].as_i64().unwrap_or(0) > 0
|| self
.bitcoin_rpc_call::<Value>(client, "getmempoolentry", &[json!(child_id)])
.await
.is_ok()
{
active.push(*child);
}
}
// Ambiguous or unavailable chain state must not hide wallet history.
if active.len() != 1 {
continue;
}
let current = active[0];
parent["bump_fee_sats"] = json!(number(&current["total_fees"]).unwrap());
parent["fee_bump_txid"] = current["tx_hash"].clone();
parent["fee_bump_confirmations"] = current["num_confirmations"].clone();
parent["fee_bump_history"] = json!(candidates.iter().map(|child| {
let child_id = child["tx_hash"].as_str().unwrap();
hidden.insert(child_id.to_owned());
json!({"tx_hash":child_id,"fee_sats":number(&child["total_fees"]).unwrap(),
"status":if child["tx_hash"] != current["tx_hash"] { "replaced" }
else if child["num_confirmations"].as_i64().unwrap_or(0) > 0 { "confirmed" } else { "mempool" }})
}).collect::<Vec<_>>());
}
normalized.retain(|tx| !tx["tx_hash"].as_str().is_some_and(|id| hidden.contains(id)));
}
async fn bump_plan(&self, txid: &str, custom_rate: Option<u64>) -> Result<Plan> {
let (client, macaroon) = self.lnd_client().await?;
let info = lnd(&client, &macaroon, "/v1/getinfo", None).await?;
ensure!(
info["synced_to_chain"] == true,
"Wait for the wallet to finish syncing"
);
let version = info["version"]
.as_str()
.context("LND version is unavailable")?;
let mut parts = version.trim_start_matches('v').split('.');
let major: u32 = parts
.next()
.unwrap_or("")
.parse()
.context("Invalid LND version")?;
let minor: u32 = parts
.next()
.unwrap_or("")
.parse()
.context("Invalid LND version")?;
ensure!(
major > 0 || minor >= 21,
"This fee-bump interface requires LND 0.21 or newer"
);
let history = lnd(&client, &macaroon, "/v1/transactions", None).await?;
let txs = array(&history, "transactions")?;
let tx = txs
.iter()
.find(|t| t["tx_hash"] == txid)
.context("Transaction is not in this wallet")?;
ensure!(
tx["num_confirmations"].as_i64() == Some(0),
"This transaction is no longer pending"
);
let entry: Value = self
.bitcoin_rpc_call(&client, "getmempoolentry", &[json!(txid)])
.await
.context("Transaction is not currently in the node's mempool")?;
ensure!(
number(&entry["descendantcount"])? == 1,
"This transaction already has a child; open the child's Bump options instead"
);
let pending = lnd(&client, &macaroon, "/v2/wallet/sweeps/pending", None).await?;
let sweeps = array(&pending, "pending_sweeps")?;
let published = lnd(
&client,
&macaroon,
"/v2/wallet/sweeps?verbose=false&start_height=-1",
None,
)
.await?;
let is_sweep = published["transaction_ids"]["transaction_ids"]
.as_array()
.is_some_and(|ids| ids.iter().any(|id| id == txid));
let outputs = array(tx, "output_details")?;
ensure!(
tx["amount"]
.as_str()
.and_then(|v| v.parse::<i64>().ok())
.or_else(|| tx["amount"].as_i64())
.is_some_and(|v| v < 0),
"Bump is available for outgoing payments and wallet fee sweeps"
);
let (
method,
input_txid,
input_index,
input_sats,
parent_txid,
parent_size,
parent_fee,
old_fee,
size,
recipient_sats,
) = if is_sweep {
// Only a simple wallet CPFP sweep is replaceable here. Anchor/HTLC,
// batched sweeps and arbitrary signed payments need different previews.
let raw: Value = self
.bitcoin_rpc_call(&client, "getrawtransaction", &[json!(txid), json!(true)])
.await?;
let inputs = array(&raw, "vin")?;
ensure!(
inputs.len() == 1 && outputs.len() == 1 && outputs[0]["is_our_address"] == true,
"RBF for batched or channel sweeps is not supported here yet"
);
let input_txid = inputs[0]["txid"]
.as_str()
.context("Missing sweep input")?
.to_string();
let index = u32::try_from(number(&inputs[0]["vout"])?)?;
ensure!(
sweeps.len() == 1 && outpoint_matches(&sweeps[0]["outpoint"], &input_txid, index),
"RBF is unavailable while other wallet sweeps are active"
);
let parent = txs
.iter()
.find(|t| t["tx_hash"] == input_txid)
.context("Sweep parent is unavailable")?;
let parent_output = array(parent, "output_details")?
.iter()
.find(|o| {
number(&o["output_index"]).ok() == Some(index as u64)
&& o["is_our_address"] == true
})
.context("RBF requires a wallet-owned change input")?;
let parent_entry: Value = self
.bitcoin_rpc_call(&client, "getmempoolentry", &[json!(input_txid)])
.await
.context("Only unconfirmed CPFP sweep replacements are supported here")?;
ensure!(
number(&parent_entry["ancestorcount"])? == 1
&& number(&parent_entry["descendantcount"])? == 2,
"Complex sweep package cannot be quoted safely"
);
let recipients = recipient_amount(parent)?;
(
"rbf",
input_txid.clone(),
index,
number(&parent_output["amount"])?,
input_txid,
number(&parent_entry["vsize"])?,
btc_sats(&parent_entry["fees"]["base"])?,
btc_sats(&entry["fees"]["base"])?,
sweep_size(parent_output)?.max(number(&entry["vsize"])?),
recipients,
)
} else {
ensure!(
sweeps.is_empty(),
"Another wallet sweep is active; wait for it before creating a CPFP bump"
);
ensure!(
number(&entry["ancestorcount"])? == 1,
"Fee bumping a chain of unconfirmed payments is not supported yet"
);
let unspent = lnd(
&client,
&macaroon,
"/v2/wallet/utxos",
Some(json!({"unconfirmed_only":true})),
)
.await?;
let utxos = array(&unspent, "utxos")?;
let leases = lnd(
&client,
&macaroon,
"/v2/wallet/utxos/leases",
Some(json!({})),
)
.await?;
let locked = array(&leases, "locked_utxos")?;
let output = outputs
.iter()
.filter(|o| o["is_our_address"] == true && sweep_size(o).is_ok())
.filter(|o| {
number(&o["output_index"]).ok().is_some_and(|i| {
utxos
.iter()
.any(|u| outpoint_matches(&u["outpoint"], txid, i as u32))
&& !locked
.iter()
.any(|u| outpoint_matches(&u["outpoint"], txid, i as u32))
})
})
.max_by_key(|o| number(&o["amount"]).unwrap_or(0))
.context(
"RBF is unavailable for this payment. CPFP needs spendable wallet-owned change",
)?;
let index = u32::try_from(number(&output["output_index"])?)?;
let available: Value = self
.bitcoin_rpc_call(
&client,
"gettxout",
&[json!(txid), json!(index), json!(true)],
)
.await?;
ensure!(
available.is_object()
&& number(&available["confirmations"])? == 0
&& btc_sats(&available["value"])? == number(&output["amount"])?,
"Change is no longer available"
);
(
"cpfp",
txid.to_string(),
index,
number(&output["amount"])?,
txid.to_string(),
number(&entry["vsize"])?,
btc_sats(&entry["fees"]["base"])?,
0,
sweep_size(output)?,
recipient_amount(tx)?,
)
};
let mempool: Value = self
.bitcoin_rpc_call(&client, "getmempoolinfo", &[])
.await?;
let relay = btc_sats(&mempool["incrementalrelayfee"])?
.div_ceil(1000)
.max(1);
let floor = btc_sats(&mempool["mempoolminfee"])?
.max(btc_sats(&mempool["minrelaytxfee"])?)
.div_ceil(1000)
.max(1);
let rate = match custom_rate {
Some(rate) => {
ensure!(
rate >= floor,
"Custom rate is below the current mempool minimum"
);
rate
}
None => {
let estimate = lnd(&client, &macaroon, "/v2/wallet/estimatefee/1", None).await?;
number(&estimate["sat_per_kw"])?.div_ceil(250).max(floor)
}
};
let budget = fee_budget(
rate,
parent_size,
parent_fee,
size,
old_fee,
relay.max(floor),
input_sats,
)?;
let tip: String = self
.bitcoin_rpc_call(&client, "getbestblockhash", &[])
.await?;
Ok(Plan {
txid: txid.to_string(),
method: method.into(),
input_txid,
input_index,
parent_txid,
recipient_sats,
rate_sat_vb: rate,
current_fee_sats: parent_fee + old_fee,
additional_fee_sats: budget - old_fee,
total_fee_sats: parent_fee + budget,
budget_sats: budget,
input_sats,
parent_vsize: parent_size,
sweep_vsize_bound: size,
tip,
})
}
pub(in crate::api::rpc) async fn handle_lnd_bump_quote(
&self,
params: Option<Value>,
) -> Result<Value> {
let p = params.unwrap_or_default();
let txid = txid_param(&p)?;
let path = self
.config
.data_dir
.join("wallet/fee-bumps")
.join(format!("{txid}.json"));
ensure!(
!path.try_exists()?,
"A bump was already submitted for this transaction. Check its status"
);
let custom = p
.get("sat_per_vbyte")
.map(|v| v.as_u64().context("Custom rate must be a whole number"))
.transpose()?;
if let Some(rate) = custom {
ensure!(
(1..=5000).contains(&rate),
"Custom rate must be 1–5000 sat/vB"
);
}
let plan = self.bump_plan(&txid, custom).await?;
let quote = Quote {
quote_id: uuid::Uuid::new_v4().to_string(),
expires_at: now() + QUOTE_SECONDS,
custom_rate: custom,
plan,
};
let mut quotes = QUOTES.lock().await;
quotes.retain(|_, q| q.expires_at > now());
ensure!(quotes.len() < 128, "Too many fee quotes; try again shortly");
quotes.insert(quote.quote_id.clone(), quote.clone());
Ok(serde_json::to_value(quote)?)
}
pub(in crate::api::rpc) async fn handle_lnd_bump_submit(
&self,
params: Option<Value>,
) -> Result<Value> {
let p = params.unwrap_or_default();
let txid = txid_param(&p)?;
let _guard = SUBMIT.lock().await;
let path = self
.config
.data_dir
.join("wallet/fee-bumps")
.join(format!("{txid}.json"));
if path.try_exists()? {
return self
.handle_lnd_bump_status(Some(json!({"txid":txid})))
.await;
}
let id = p["quote_id"]
.as_str()
.context("A reviewed fee quote is required")?;
let quote = QUOTES
.lock()
.await
.get(id)
.cloned()
.context("Quote expired; review the fee again")?;
ensure!(
quote.plan.txid == txid && quote.expires_at > now(),
"Quote expired; review the fee again"
);
let fresh = self.bump_plan(&txid, quote.custom_rate).await?;
validate_quote(&quote, &fresh, now())?;
let op = Operation {
quote: quote.clone(),
status: "unknown".into(),
message: "Submission recorded; checking the wallet. Do not submit another bump.".into(),
};
reserve(&path, &op).await?;
QUOTES.lock().await.remove(id);
let (client, macaroon) = self.lnd_client().await?;
// At a one-block deadline LND may spend ALL this explicitly previewed
// budget. It is always below input value, so no extra funding is requested.
let result = lnd(
&client,
&macaroon,
"/v2/wallet/bumpfee",
Some(bump_body(&fresh)),
)
.await;
// Keep the write-ahead record even for an RPC error: a lost response can
// conceal an accepted bump. Status reconciles from wallet/mempool evidence.
match result {
Ok(_) => Ok(
json!({"status":"registered", "message":"Bump registered with the wallet. Waiting for broadcast.", "quote":quote}),
),
Err(_) => Ok(
json!({"status":"unknown", "message":"The wallet response was not confirmed. Check status; do not submit again.", "quote":quote}),
),
}
}
pub(in crate::api::rpc) async fn handle_lnd_bump_status(
&self,
params: Option<Value>,
) -> Result<Value> {
let txid = txid_param(&params.unwrap_or_default())?;
let path = self
.config
.data_dir
.join("wallet/fee-bumps")
.join(format!("{txid}.json"));
let bytes = match tokio::fs::read(path).await {
Ok(b) => b,
Err(e) if e.kind() == std::io::ErrorKind::NotFound => {
return Ok(json!({"status":"none"}))
}
Err(e) => return Err(e.into()),
};
let op: Operation = serde_json::from_slice(&bytes)
.context("Bump receipt needs recovery; do not resubmit")?;
let (client, macaroon) = self.lnd_client().await?;
let history = lnd(&client, &macaroon, "/v1/transactions", None).await?;
let plan = &op.quote.plan;
let input = format!("{}:{}", plan.input_txid, plan.input_index);
let mut candidates: Vec<&Value> = array(&history, "transactions")?
.iter()
.filter(|t| {
t["tx_hash"] != txid
&& t["output_details"].as_array().is_some_and(|outputs| {
!outputs.is_empty() && outputs.iter().all(|o| o["is_our_address"] == true)
})
&& t["previous_outpoints"]
.as_array()
.is_some_and(|inputs| inputs.iter().any(|i| i["outpoint"] == input))
})
.collect();
candidates.sort_by_key(|t| std::cmp::Reverse(number(&t["time_stamp"]).unwrap_or(0)));
for t in candidates {
let id = t["tx_hash"]
.as_str()
.context("Missing bump transaction ID")?;
let confirmed = t["num_confirmations"].as_i64().unwrap_or(0) > 0;
let accepted = if confirmed {
false
} else {
self.bitcoin_rpc_call::<Value>(&client, "getmempoolentry", &[json!(id)])
.await
.is_ok()
};
if confirmed || accepted {
return Ok(json!({"status":if confirmed {"confirmed"} else {"mempool"},
"message":if confirmed {"Fee bump confirmed."} else {"Fee bump accepted in the node's mempool; awaiting confirmation."},
"bump_txid":id,"confirmations":t["num_confirmations"],"actual_sweep_fee_sats":number(&t["total_fees"])?,"quote":op.quote}));
}
}
let pending = lnd(&client, &macaroon, "/v2/wallet/sweeps/pending", None).await?;
let registered = array(&pending, "pending_sweeps")?.iter().any(|s| {
outpoint_matches(&s["outpoint"], &plan.input_txid, plan.input_index)
&& number(&s["budget"]).ok() == Some(plan.budget_sats)
&& number(&s["requested_sat_per_vbyte"]).ok() == Some(plan.rate_sat_vb)
});
Ok(
json!({"status":if registered {"registered"} else {"unknown"},
"message":if registered {"Bump registered; waiting for a verified broadcast."} else {"Submission outcome is unknown. Do not submit again; check wallet status."}, "quote":op.quote}),
)
}
}
fn recipient_amount(tx: &Value) -> Result<u64> {
array(tx, "output_details")?
.iter()
.filter(|o| o["is_our_address"] == false)
.try_fold(0u64, |sum, o| {
sum.checked_add(number(&o["amount"])?)
.context("Recipient amount overflow")
})
}
#[cfg(test)]
mod tests {
use super::*;
fn sample_plan() -> Plan {
serde_json::from_value(json!({"txid":"a","method":"cpfp","input_txid":"a","input_index":0,"parent_txid":"a","recipient_sats":161650,"rate_sat_vb":3,"current_fee_sats":144,"additional_fee_sats":618,"total_fee_sats":762,"budget_sats":618,"input_sats":21126,"parent_vsize":142,"sweep_vsize_bound":112,"tip":"tip"})).unwrap()
}
#[test]
fn history_requires_owned_simple_fee_only_child() {
let plan = sample_plan();
let tx = json!({"tx_hash":"b".repeat(64),"amount":"-200","total_fees":"200",
"previous_outpoints":[{"outpoint":"a:0","is_our_output":true}],
"output_details":[{"is_our_address":true}]});
assert!(fee_child_matches(&tx, &plan));
for bad in [
json!({"amount":"-201"}),
json!({"amount":"200"}),
json!({"total_fees":"0"}),
json!({"previous_outpoints":[{"outpoint":"a:1","is_our_output":true}]}),
json!({"previous_outpoints":[{"outpoint":"a:0","is_our_output":false}]}),
json!({"previous_outpoints":[{"outpoint":"a:0","is_our_output":true},{"outpoint":"c:0","is_our_output":true}]}),
json!({"output_details":[{"is_our_address":false}]}),
json!({"output_details":[]}),
json!({"tx_hash":"../../invalid"}),
] {
let mut changed = tx.clone();
for (key, value) in bad.as_object().unwrap() {
changed[key] = value.clone();
}
assert!(!fee_child_matches(&changed, &plan), "{bad}");
}
}
#[test]
fn stale_quotes_cannot_silently_change_approved_fee_or_transaction() {
let plan = sample_plan();
let q = Quote {
quote_id: "q".into(),
expires_at: 100,
custom_rate: None,
plan: plan.clone(),
};
assert!(validate_quote(&q, &plan, 99).is_ok());
assert!(validate_quote(&q, &plan, 100).is_err());
let mut changed = plan.clone();
changed.budget_sats += 1;
assert!(validate_quote(&q, &changed, 99).is_err());
changed = plan.clone();
changed.input_index += 1;
assert!(validate_quote(&q, &changed, 99).is_err());
changed = plan.clone();
changed.recipient_sats -= 1;
assert!(validate_quote(&q, &changed, 99).is_err());
changed = plan.clone();
changed.tip = "new block".into();
assert!(validate_quote(&q, &changed, 99).is_err());
}
#[test]
fn mutation_always_has_explicit_budget_and_does_not_send_a_second_payment() {
assert_eq!(
bump_body(&sample_plan()),
json!({"outpoint":{"txid_str":"a","output_index":0},"sat_per_vbyte":"3","budget":"618","deadline_delta":1,"immediate":true})
);
let mut rbf = sample_plan();
rbf.method = "rbf".into();
rbf.txid = "child".into();
// RBF uses the already-registered input, not the child's output.
assert_eq!(bump_body(&rbf)["outpoint"]["txid_str"], "a");
}
#[test]
fn outpoint_ownership_and_recipient_exclude_wallet_change() {
assert!(outpoint_matches(
&json!({"txid_str":"a","output_index":2}),
"a",
2
));
assert!(!outpoint_matches(
&json!({"txid_str":"b","output_index":2}),
"a",
2
));
assert!(!outpoint_matches(
&json!({"txid_str":"a","output_index":3}),
"a",
2
));
assert_eq!(recipient_amount(&json!({"output_details":[{"is_our_address":true,"amount":"21126"},{"is_our_address":false,"amount":"161650"}]})).unwrap(), 161650);
assert!(recipient_amount(
&json!({"output_details":[{"is_our_address":false,"amount":"bad"}]})
)
.is_err());
}
#[test]
fn cpfp_budget_covers_parent_and_preserves_change() {
assert_eq!(fee_budget(3, 142, 144, 112, 0, 1, 21126).unwrap(), 618);
assert!(fee_budget(5000, 142, 144, 112, 0, 1, 21126).is_err());
assert!(fee_budget(0, 142, 144, 112, 0, 1, 21126).is_err());
}
#[test]
fn rbf_pays_incremental_relay_cost_and_counts_only_extra_cost() {
let fee = fee_budget(3, 142, 144, 112, 650, 1, 21126).unwrap();
assert_eq!(fee, 763);
assert_eq!(fee - 650, 113);
}
#[test]
fn unsupported_outputs_and_malformed_ids_fail_closed() {
assert!(sweep_size(&json!({"output_type":"SCRIPT_TYPE_WITNESS_V0_SCRIPT_HASH"})).is_err());
assert!(txid_param(&json!({"txid":"../../file"})).is_err());
assert!(number(&json!(-1)).is_err());
assert!(btc_sats(&json!(-0.1)).is_err());
assert_eq!(btc_sats(&json!(0.00000650)).unwrap(), 650);
}
#[tokio::test]
async fn receipt_prevents_duplicate_submission_after_restart() {
let dir = std::env::temp_dir().join(uuid::Uuid::new_v4().to_string());
let path = dir.join("receipt.json");
let plan: Plan = serde_json::from_value(json!({"txid":"a","method":"cpfp","input_txid":"a","input_index":0,"parent_txid":"a","recipient_sats":1000,"rate_sat_vb":3,"current_fee_sats":144,"additional_fee_sats":618,"total_fee_sats":762,"budget_sats":618,"input_sats":21126,"parent_vsize":142,"sweep_vsize_bound":112,"tip":"tip"})).unwrap();
let op = Operation {
quote: Quote {
quote_id: "q".into(),
expires_at: now() + 60,
custom_rate: None,
plan,
},
status: "unknown".into(),
message: "pending".into(),
};
reserve(&path, &op).await.unwrap();
assert!(reserve(&path, &op).await.is_err());
let restored: Operation =
serde_json::from_slice(&tokio::fs::read(&path).await.unwrap()).unwrap();
assert_eq!(restored.quote.plan.budget_sats, 618);
tokio::fs::remove_dir_all(dir).await.unwrap();
}
}
@@ -0,0 +1,120 @@
//! Explicit on-chain fee choices retain priority; omitted choices target the next block.
use anyhow::{ensure, Context, Result};
use serde_json::Value;
pub(super) const DEFAULT_TARGET: i64 = 1;
pub(super) fn estimated_sat_per_vbyte(value: &Value) -> Result<u64> {
let per_kw = value["sat_per_kw"]
.as_u64()
.or_else(|| value["sat_per_kw"].as_str().and_then(|s| s.parse().ok()))
.context("Next-block fee estimate is unavailable")?;
let rate = per_kw.div_ceil(250);
ensure!(
(1..=5000).contains(&rate),
"Next-block fee estimate is outside supported bounds; choose an explicit fee"
);
Ok(rate)
}
pub(super) fn fee_options(params: &Value) -> Result<(Option<i64>, Option<i64>)> {
let integer = |key: &str, max: i64| -> Result<Option<i64>> {
match params.get(key) {
None | Some(Value::Null) => Ok(None),
Some(value) => {
let n = value
.as_i64()
.with_context(|| format!("{key} must be a positive whole number"))?;
ensure!((1..=max).contains(&n), "{key} must be between 1 and {max}");
Ok(Some(n))
}
}
};
let target = integer("target_conf", 1008)?;
let rate = integer("sat_per_vbyte", 5000)?;
ensure!(
target.is_none() || rate.is_none(),
"Specify either target_conf or sat_per_vbyte, not both"
);
Ok((
if rate.is_none() {
Some(target.unwrap_or(DEFAULT_TARGET))
} else {
None
},
rate,
))
}
#[cfg(test)]
mod tests {
use super::*;
use serde_json::json;
#[test]
fn estimates_round_up_and_missing_or_extreme_estimates_fail_closed() {
assert_eq!(
estimated_sat_per_vbyte(&json!({"sat_per_kw":"501"})).unwrap(),
3
);
assert_eq!(
estimated_sat_per_vbyte(&json!({"sat_per_kw":250})).unwrap(),
1
);
for v in [
json!({}),
json!({"sat_per_kw":0}),
json!({"sat_per_kw":-1}),
json!({"sat_per_kw":1250001}),
] {
assert!(estimated_sat_per_vbyte(&v).is_err());
}
}
#[test]
fn next_block_default_preserves_explicit_slower_and_custom_choices() {
assert_eq!(fee_options(&json!({})).unwrap(), (Some(1), None));
assert_eq!(
fee_options(&json!({"target_conf":null})).unwrap(),
(Some(1), None)
);
for target in [1, 3, 6, 144, 1008] {
assert_eq!(
fee_options(&json!({"target_conf":target})).unwrap(),
(Some(target), None)
);
}
for rate in [1, 17, 5000] {
assert_eq!(
fee_options(&json!({"sat_per_vbyte":rate})).unwrap(),
(None, Some(rate))
);
}
}
#[test]
fn malformed_explicit_fees_never_silently_become_fast() {
for value in [
json!(0),
json!(-1),
json!(1.5),
json!("6"),
json!(true),
json!({}),
json!(1009),
] {
assert!(fee_options(&json!({"target_conf":value})).is_err());
}
for value in [
json!(0),
json!(-1),
json!(1.5),
json!("6"),
json!(true),
json!(5001),
] {
assert!(fee_options(&json!({"sat_per_vbyte":value})).is_err());
}
assert!(fee_options(&json!({"target_conf":1,"sat_per_vbyte":2})).is_err());
}
}
+84
View File
@@ -109,7 +109,50 @@ fn checked_balances(
)) ))
} }
fn bitcoin_wait_state(
installed: bool,
running: bool,
fresh: bool,
ibd: Option<bool>,
) -> (&'static str, &'static str) {
if !installed {
("waiting_install", "Waiting for Bitcoin to be installed")
} else if !running {
("waiting_start", "Waiting for Bitcoin to start")
} else if !fresh || ibd.is_none() {
("waiting_start", "Waiting for Bitcoin to start")
} else if ibd == Some(true) {
("waiting_sync", "Waiting for Bitcoin to sync")
} else {
("bitcoin_ready", "Bitcoin is ready")
}
}
impl RpcHandler { impl RpcHandler {
pub(crate) async fn handle_lnd_readiness(&self) -> serde_json::Value {
let (data, _) = self.state_manager.get_snapshot().await;
if !data.server_info.status_info.containers_scanned {
return serde_json::json!({"state":"checking", "message":"Checking Bitcoin availability"});
}
let nodes: Vec<_> = ["bitcoin-core", "bitcoin-knots", "bitcoin"]
.iter()
.filter_map(|id| data.package_data.get(*id))
.collect();
let installed = !nodes.is_empty();
let running = nodes
.iter()
.any(|p| p.state == crate::data_model::PackageState::Running);
let bitcoin = crate::bitcoin_status::get_bitcoin_status().await;
let ibd = bitcoin
.blockchain_info
.as_ref()
.and_then(|v| v.get("initialblockdownload"))
.and_then(|v| v.as_bool());
let (state, message) =
bitcoin_wait_state(installed, running, bitcoin.ok && !bitcoin.stale, ibd);
serde_json::json!({"state": state, "message": message})
}
pub(in crate::api::rpc) async fn handle_lnd_getinfo(&self) -> Result<serde_json::Value> { pub(in crate::api::rpc) async fn handle_lnd_getinfo(&self) -> Result<serde_json::Value> {
let macaroon_bytes = read_lnd_admin_macaroon().await?; let macaroon_bytes = read_lnd_admin_macaroon().await?;
let macaroon_hex = hex::encode(&macaroon_bytes); let macaroon_hex = hex::encode(&macaroon_bytes);
@@ -419,3 +462,44 @@ mod tests {
assert!(!is_valid_identity_pubkey(&"g".repeat(66))); assert!(!is_valid_identity_pubkey(&"g".repeat(66)));
} }
} }
#[cfg(test)]
mod dependency_readiness_tests {
use super::bitcoin_wait_state;
#[test]
fn waiting_states_cover_install_start_sync_outage_and_recovery() {
assert_eq!(
bitcoin_wait_state(false, false, false, None).0,
"waiting_install"
);
assert_eq!(
bitcoin_wait_state(true, false, false, None).0,
"waiting_start"
);
assert_eq!(
bitcoin_wait_state(true, true, false, None).0,
"waiting_start"
);
assert_eq!(
bitcoin_wait_state(true, true, true, Some(true)).0,
"waiting_sync"
);
assert_eq!(
bitcoin_wait_state(true, true, true, Some(false)).0,
"bitcoin_ready"
);
// Previously synced cached information must not hide a current outage.
assert_eq!(
bitcoin_wait_state(true, true, false, Some(false)).0,
"waiting_start"
);
assert_eq!(
bitcoin_wait_state(true, true, true, None).0,
"waiting_start"
);
assert_eq!(
bitcoin_wait_state(true, true, true, Some(false)).0,
"bitcoin_ready"
);
}
}
+64 -1
View File
@@ -1,4 +1,6 @@
mod channels; mod channels;
mod fee_bump;
mod fee_policy;
mod info; mod info;
mod macaroons; mod macaroons;
mod payments; mod payments;
@@ -133,12 +135,36 @@ async fn stream_lnd_transactions(sm: &crate::state::StateManager) -> Result<()>
/// RPC-unreachable and locked-wallet states are deliberately NOT handled /// RPC-unreachable and locked-wallet states are deliberately NOT handled
/// here — container-down is crash-recovery's job, and unlocking needs the /// here — container-down is crash-recovery's job, and unlocking needs the
/// operator. /// operator.
fn bitcoin_ready_for_lnd_watchdog(status: &crate::bitcoin_status::BitcoinNodeStatus) -> bool {
status.ok
&& !status.stale
&& status.age_ms < 30_000
&& status
.blockchain_info
.as_ref()
.and_then(|v| v.get("initialblockdownload"))
.and_then(|v| v.as_bool())
== Some(false)
}
pub(crate) fn spawn_lnd_health_watchdog() { pub(crate) fn spawn_lnd_health_watchdog() {
tokio::spawn(async move { tokio::spawn(async move {
let mut bad_minutes: u32 = 0; let mut bad_minutes: u32 = 0;
let mut last_restart: Option<tokio::time::Instant> = None; let mut last_restart: Option<tokio::time::Instant> = None;
let mut last_height: Option<u64> = None;
loop { loop {
tokio::time::sleep(std::time::Duration::from_secs(60)).await; tokio::time::sleep(std::time::Duration::from_secs(60)).await;
// Initial Bitcoin sync, warmup, and outages are dependencies to
// wait for, never evidence that LND is wedged. Do not accumulate
// restart pressure during a days-long initial block download.
let bitcoin = crate::bitcoin_status::get_bitcoin_status().await;
if !bitcoin_ready_for_lnd_watchdog(&bitcoin)
|| crate::app_ops::lifecycle_op_in_flight("lnd")
{
bad_minutes = 0;
last_height = None;
continue;
}
let Ok(bytes) = read_lnd_admin_macaroon().await else { let Ok(bytes) = read_lnd_admin_macaroon().await else {
bad_minutes = 0; // no LND on this node (or not set up yet) bad_minutes = 0; // no LND on this node (or not set up yet)
continue; continue;
@@ -161,6 +187,10 @@ pub(crate) fn spawn_lnd_health_watchdog() {
bad_minutes = 0; // down/locked — not the wedge signature bad_minutes = 0; // down/locked — not the wedge signature
continue; continue;
}; };
if !resp.status().is_success() {
bad_minutes = 0;
continue;
}
let Ok(info) = resp.json::<serde_json::Value>().await else { let Ok(info) = resp.json::<serde_json::Value>().await else {
bad_minutes = 0; bad_minutes = 0;
continue; continue;
@@ -182,7 +212,12 @@ pub(crate) fn spawn_lnd_health_watchdog() {
.get("num_pending_channels") .get("num_pending_channels")
.and_then(|v| v.as_u64()) .and_then(|v| v.as_u64())
.unwrap_or(0); .unwrap_or(0);
let wedged = !synced || (channels > 0 && peers == 0); let height = info.get("block_height").and_then(|v| v.as_u64());
let progressing = height
.zip(last_height)
.is_some_and(|(now, before)| now > before);
last_height = height;
let wedged = !progressing && (!synced || (channels > 0 && peers == 0));
if !wedged { if !wedged {
bad_minutes = 0; bad_minutes = 0;
continue; continue;
@@ -239,3 +274,31 @@ impl RpcHandler {
Ok((client, macaroon_hex)) Ok((client, macaroon_hex))
} }
} }
#[cfg(test)]
mod watchdog_dependency_tests {
use super::bitcoin_ready_for_lnd_watchdog;
use crate::bitcoin_status::BitcoinNodeStatus;
use serde_json::json;
#[test]
fn initial_sync_warmup_outage_stale_and_unknown_never_trigger_lnd_restart() {
let mut status = BitcoinNodeStatus::default();
assert!(!bitcoin_ready_for_lnd_watchdog(&status));
status.ok = true;
status.blockchain_info = Some(json!({"initialblockdownload":true}));
assert!(!bitcoin_ready_for_lnd_watchdog(&status));
status.blockchain_info = Some(json!({"initialblockdownload":false}));
assert!(bitcoin_ready_for_lnd_watchdog(&status));
status.stale = true;
assert!(!bitcoin_ready_for_lnd_watchdog(&status));
status.stale = false;
status.ok = false;
assert!(!bitcoin_ready_for_lnd_watchdog(&status));
status.ok = true;
status.age_ms = 30_000;
assert!(!bitcoin_ready_for_lnd_watchdog(&status));
status.age_ms = 0;
status.blockchain_info = Some(json!({}));
assert!(!bitcoin_ready_for_lnd_watchdog(&status));
}
}
@@ -402,6 +402,9 @@ impl RpcHandler {
})); }));
} }
self.group_fee_bump_history(raw_txs, &mut transactions, &client)
.await;
// Sort by timestamp descending (most recent first) // Sort by timestamp descending (most recent first)
transactions.sort_by(|a, b| { transactions.sort_by(|a, b| {
let ta = a.get("time_stamp").and_then(|v| v.as_i64()).unwrap_or(0); let ta = a.get("time_stamp").and_then(|v| v.as_i64()).unwrap_or(0);
+154 -37
View File
@@ -124,28 +124,8 @@ impl RpcHandler {
return Err(anyhow::anyhow!("Invalid Bitcoin address format")); return Err(anyhow::anyhow!("Invalid Bitcoin address format"));
} }
// Fee control: either a confirmation target or an explicit fee rate // Omitted fees target the next block; explicit slower/custom choices win.
let target_conf = params.get("target_conf").and_then(|v| v.as_i64()); let (target_conf, sat_per_vbyte) = super::fee_policy::fee_options(&params)?;
let sat_per_vbyte = params.get("sat_per_vbyte").and_then(|v| v.as_i64());
if target_conf.is_some() && sat_per_vbyte.is_some() {
return Err(anyhow::anyhow!(
"Invalid fee parameters: specify either target_conf or sat_per_vbyte, not both"
));
}
if let Some(tc) = target_conf {
if !(1..=1008).contains(&tc) {
return Err(anyhow::anyhow!(
"Invalid target_conf: must be between 1 and 1008 blocks"
));
}
}
if let Some(rate) = sat_per_vbyte {
if !(1..=5000).contains(&rate) {
return Err(anyhow::anyhow!(
"Invalid sat_per_vbyte: must be between 1 and 5000"
));
}
}
info!( info!(
addr = addr, addr = addr,
@@ -238,15 +218,12 @@ impl RpcHandler {
if !(546..=21_000_000 * 100_000_000).contains(&amount) { if !(546..=21_000_000 * 100_000_000).contains(&amount) {
return Err(anyhow::anyhow!("Invalid amount")); return Err(anyhow::anyhow!("Invalid amount"));
} }
let target_conf = params let (target_conf, custom_rate) = super::fee_policy::fee_options(&params)?;
.get("target_conf") anyhow::ensure!(
.and_then(|v| v.as_i64()) custom_rate.is_none(),
.unwrap_or(6); "Fee estimation requires a confirmation target"
if !(1..=1008).contains(&target_conf) { );
return Err(anyhow::anyhow!( let target_conf = target_conf.unwrap_or(super::fee_policy::DEFAULT_TARGET);
"Invalid target_conf: must be between 1 and 1008 blocks"
));
}
let (client, macaroon_hex) = self.lnd_client().await?; let (client, macaroon_hex) = self.lnd_client().await?;
@@ -453,6 +430,56 @@ impl RpcHandler {
Ok(settled) Ok(settled)
} }
/// Verify against LND at download time, rather than relying on a browser
/// having polled first. The memo/amount also recover pre-upgrade in-memory
/// entitlements after restart; unrelated invoices never unlock a file.
pub(crate) async fn settle_content_invoice(
&self,
hash: &str,
content_id: &str,
) -> Result<bool> {
anyhow::ensure!(
hash.len() == 64 && hash.bytes().all(|c| c.is_ascii_hexdigit()),
"Invalid payment hash"
);
let hash = hash.to_ascii_lowercase();
let existing = crate::content_invoice::lookup(&self.config.data_dir, &hash).await?;
if let Some((id, _)) = &existing {
if id != content_id {
return Ok(false);
}
}
if crate::content_invoice::is_paid_for(&self.config.data_dir, &hash, content_id).await {
return Ok(true);
}
let (client, macaroon_hex) = self.lnd_client().await?;
let response = client
.get(format!("{LND_REST_BASE_URL}/v1/invoice/{hash}"))
.header("Grpc-Metadata-macaroon", &macaroon_hex)
.send()
.await?;
if response.status() == reqwest::StatusCode::NOT_FOUND {
return Ok(false);
}
let body: serde_json::Value = response.error_for_status()?.json().await?;
let Some(price) = content_invoice_amount(&body, content_id) else {
return Ok(false);
};
if existing
.as_ref()
.is_some_and(|(_, expected)| *expected != price)
{
return Ok(false);
}
crate::content_invoice::record_pending(&self.config.data_dir, &hash, content_id, price)
.await?;
let settled = content_invoice_fully_settled(&body, price);
if settled {
crate::content_invoice::mark_paid(&self.config.data_dir, &hash).await?;
}
Ok(settled)
}
/// Generate a fresh on-chain receive address (seller side, #46). /// Generate a fresh on-chain receive address (seller side, #46).
pub(crate) async fn new_onchain_address(&self) -> Result<String> { pub(crate) async fn new_onchain_address(&self) -> Result<String> {
let (client, macaroon_hex) = self.lnd_client().await?; let (client, macaroon_hex) = self.lnd_client().await?;
@@ -732,10 +759,24 @@ impl RpcHandler {
total_amount += amount; total_amount += amount;
} }
let sat_per_vbyte = params let (_, explicit_rate) = super::fee_policy::fee_options(&serde_json::json!({
.get("fee_rate_sat_per_vbyte") "sat_per_vbyte": params.get("fee_rate_sat_per_vbyte")
.and_then(|v| v.as_u64()) }))?;
.unwrap_or(10); let (client, macaroon_hex) = self.lnd_client().await?;
let sat_per_vbyte = if let Some(rate) = explicit_rate {
rate as u64
} else {
let response = client
.get(format!("{LND_REST_BASE_URL}/v2/wallet/estimatefee/1"))
.header("Grpc-Metadata-macaroon", &macaroon_hex)
.send()
.await
.context("Cannot estimate the next-block fee")?
.error_for_status()
.context("Next-block fee estimate rejected")?;
let estimate: serde_json::Value = response.json().await?;
super::fee_policy::estimated_sat_per_vbyte(&estimate)?
};
info!( info!(
total_amount = total_amount, total_amount = total_amount,
@@ -743,8 +784,6 @@ impl RpcHandler {
"Creating PSBT for hardware wallet signing" "Creating PSBT for hardware wallet signing"
); );
let (client, macaroon_hex) = self.lnd_client().await?;
let fund_body = serde_json::json!({ let fund_body = serde_json::json!({
"raw": { "raw": {
"outputs": lnd_outputs, "outputs": lnd_outputs,
@@ -1444,3 +1483,81 @@ mod tests {
assert!(s.contains("[LND_REST_UNREACHABLE]"), "got: {s}"); assert!(s.contains("[LND_REST_UNREACHABLE]"), "got: {s}");
} }
} }
// LND REST uses decimal strings for int64 fields. Match the complete seller
// memo, not a substring supplied by a buyer or an arbitrary settled invoice.
fn json_u64(value: &serde_json::Value) -> Option<u64> {
value.as_u64().or_else(|| value.as_str()?.parse().ok())
}
fn content_invoice_fully_settled(body: &serde_json::Value, price: u64) -> bool {
let settled = match body.get("state").and_then(|v| v.as_str()) {
Some(state) => state == "SETTLED",
None => body.get("settled").and_then(|v| v.as_bool()) == Some(true),
};
settled
&& price > 0
&& body
.get("amt_paid_sat")
.and_then(json_u64)
.is_some_and(|paid| paid >= price)
}
fn content_invoice_amount(body: &serde_json::Value, content_id: &str) -> Option<u64> {
if body.get("memo")?.as_str()? != format!("Archipelago peer file {content_id}") {
return None;
}
body.get("value").and_then(json_u64).filter(|v| *v > 0)
}
#[cfg(test)]
mod peer_file_invoice_tests {
use super::*;
#[test]
fn settlement_requires_terminal_state_and_full_amount() {
for state in ["OPEN", "ACCEPTED", "CANCELED", "unknown"] {
assert!(!content_invoice_fully_settled(
&serde_json::json!({"state":state,"settled":true,"amt_paid_sat":"100"}),
7
));
}
for amount in [
serde_json::json!(6),
serde_json::json!("-1"),
serde_json::json!(null),
serde_json::json!("bad"),
] {
assert!(!content_invoice_fully_settled(
&serde_json::json!({"state":"SETTLED","amt_paid_sat":amount}),
7
));
}
for amount in [serde_json::json!(7), serde_json::json!("8")] {
assert!(content_invoice_fully_settled(
&serde_json::json!({"state":"SETTLED","amt_paid_sat":amount}),
7
));
}
assert!(content_invoice_fully_settled(
&serde_json::json!({"settled":true,"amt_paid_sat":"7"}),
7
));
assert!(!content_invoice_fully_settled(
&serde_json::json!({"state":"SETTLED","amt_paid_sat":"7"}),
0
));
}
#[test]
fn legacy_recovery_requires_exact_file_memo_and_positive_amount() {
let invoice = serde_json::json!({"memo":"Archipelago peer file file-1", "value":"7"});
assert_eq!(content_invoice_amount(&invoice, "file-1"), Some(7));
assert_eq!(content_invoice_amount(&invoice, "file-2"), None);
for value in [
serde_json::json!("-1"),
serde_json::json!(0),
serde_json::json!("bad"),
] {
let mut invalid = invoice.clone();
invalid["value"] = value;
assert_eq!(content_invoice_amount(&invalid, "file-1"), None);
}
}
}
@@ -221,6 +221,10 @@ impl RpcHandler {
params: Option<serde_json::Value>, params: Option<serde_json::Value>,
) -> Result<serde_json::Value> { ) -> Result<serde_json::Value> {
let params = params.ok_or_else(|| anyhow::anyhow!("Missing params"))?; let params = params.ok_or_else(|| anyhow::anyhow!("Missing params"))?;
if let Some(job) = self.flash_job.read().await.as_ref() {
anyhow::ensure!(job.snapshot().await.done,
"A firmware flash is in progress; wait before reconnecting or changing radio settings");
}
let mut config = mesh::load_config(&self.config.data_dir).await?; let mut config = mesh::load_config(&self.config.data_dir).await?;
@@ -325,7 +329,16 @@ impl RpcHandler {
{ {
let service_arc = Arc::clone(&self.mesh_service); let service_arc = Arc::clone(&self.mesh_service);
let config_for_apply = config.clone(); let config_for_apply = config.clone();
let flash_jobs = Arc::clone(&self.flash_job);
tokio::spawn(async move { tokio::spawn(async move {
// Serialize against flash registration. If a flash started
// after this RPC saved settings, its completion applies them.
let flash_guard = flash_jobs.read().await;
if let Some(job) = flash_guard.as_ref() {
if !job.snapshot().await.done {
return;
}
}
let mut service = service_arc.write().await; let mut service = service_arc.write().await;
if let Some(svc) = service.as_mut() { if let Some(svc) = service.as_mut() {
if let Err(e) = svc.configure(config_for_apply).await { if let Err(e) = svc.configure(config_for_apply).await {
+3 -1
View File
@@ -110,7 +110,8 @@ impl RpcHandler {
// `mesh.probe-device` call (e.g. the hot-swap modal's own re-probe) // `mesh.probe-device` call (e.g. the hot-swap modal's own re-probe)
// from opening the identical port at the same time and corrupting // from opening the identical port at the same time and corrupting
// both operations' handshakes. // both operations' handshakes.
if let Some(job) = self.flash_job.read().await.as_ref() { let flash_guard = self.flash_job.read().await;
if let Some(job) = flash_guard.as_ref() {
anyhow::ensure!( anyhow::ensure!(
job.snapshot().await.done, job.snapshot().await.done,
"A firmware flash is in progress — refusing to probe the serial port until it finishes" "A firmware flash is in progress — refusing to probe the serial port until it finishes"
@@ -131,6 +132,7 @@ impl RpcHandler {
} }
} }
let probe = mesh::listener::probe_device(&path).await?; let probe = mesh::listener::probe_device(&path).await?;
drop(flash_guard);
Ok(serde_json::to_value(probe)?) Ok(serde_json::to_value(probe)?)
} }
@@ -89,6 +89,15 @@ impl RpcHandler {
match handler.handle_package_install(params).await { match handler.handle_package_install(params).await {
Ok(_) => { Ok(_) => {
info!("package.install {}: complete", package_id_spawn); info!("package.install {}: complete", package_id_spawn);
for id in [&package_id_spawn, &format!("archy-{}", package_id_spawn)] {
crate::crash_recovery::clear_user_uninstalled(&handler.config.data_dir, id)
.await;
}
crate::crash_recovery::mark_installed(
&handler.config.data_dir,
&package_id_spawn,
)
.await;
// The install pipeline has verified the container is up // The install pipeline has verified the container is up
// and healthy (see install.rs post-start exit check). // and healthy (see install.rs post-start exit check).
// Kick the scanner first so the fresh manifest (with // Kick the scanner first so the fresh manifest (with
@@ -184,17 +193,20 @@ impl RpcHandler {
// phase is cleared (None) so no stale InstallPhase // phase is cleared (None) so no stale InstallPhase
// lingers on the card. // lingers on the card.
let err_msg = format!("Install failed: {:#}", e); let err_msg = format!("Install failed: {:#}", e);
let (mut data, _) = handler.state_manager.get_snapshot().await; handler
if let Some(entry) = data.package_data.get_mut(&package_id_spawn) { .state_manager
entry.state = PackageState::Stopped; .mutate_data(|data| {
entry.install_progress = Some(crate::data_model::InstallProgress { if let Some(entry) = data.package_data.get_mut(&package_id_spawn) {
size: 0, entry.state = PackageState::Stopped;
downloaded: 0, entry.install_progress = Some(crate::data_model::InstallProgress {
phase: None, size: 0,
message: Some(err_msg), downloaded: 0,
}); phase: None,
handler.state_manager.update_data(data).await; message: Some(err_msg),
} });
}
})
.await;
} }
} }
}); });
@@ -252,6 +264,11 @@ impl RpcHandler {
match handler.handle_package_uninstall(params).await { match handler.handle_package_uninstall(params).await {
Ok(_) => { Ok(_) => {
info!("package.uninstall {}: complete", package_id_spawn); info!("package.uninstall {}: complete", package_id_spawn);
for id in [&package_id_spawn, &format!("archy-{}", package_id_spawn)] {
crate::crash_recovery::mark_user_uninstalled(&handler.config.data_dir, id)
.await;
crate::crash_recovery::clear_installed(&handler.config.data_dir, id).await;
}
// Inner handler already removed the package entry on // Inner handler already removed the package entry on
// success. Nothing more to do here. // success. Nothing more to do here.
} }
@@ -382,52 +399,56 @@ impl RpcHandler {
/// call, but fires before the spawn so the UI sees it immediately. /// call, but fires before the spawn so the UI sees it immediately.
async fn flip_to_installing(state_manager: &StateManager, package_id: &str) { async fn flip_to_installing(state_manager: &StateManager, package_id: &str) {
use crate::data_model::{Description, Manifest, PackageDataEntry, StaticFiles}; use crate::data_model::{Description, Manifest, PackageDataEntry, StaticFiles};
let (mut data, _) = state_manager.get_snapshot().await; state_manager
let entry = data .mutate_data(|data| {
.package_data let entry = data
.entry(package_id.to_string()) .package_data
.or_insert_with(|| PackageDataEntry { .entry(package_id.to_string())
state: PackageState::Installing, .or_insert_with(|| PackageDataEntry {
health: None, ui_ready: None,
exit_code: None, state: PackageState::Installing,
static_files: StaticFiles { health: None,
license: String::new(), exit_code: None,
instructions: String::new(), static_files: StaticFiles {
// Leave icon empty during the transient Installing window: license: String::new(),
// hardcoding `<id>.png` is wrong for ~half our apps (many use instructions: String::new(),
// `.svg` / `.webp`), producing a broken-image flicker until // Leave icon empty during the transient Installing window:
// the scanner refreshes the entry. The frontend's `icon` // hardcoding `<id>.png` is wrong for ~half our apps (many use
// computed falls through to `curatedMap.get(id)?.icon` which // `.svg` / `.webp`), producing a broken-image flicker until
// has the correct extensions for known apps. // the scanner refreshes the entry. The frontend's `icon`
icon: String::new(), // computed falls through to `curatedMap.get(id)?.icon` which
}, // has the correct extensions for known apps.
manifest: Manifest { icon: String::new(),
id: package_id.to_string(), },
title: package_id.to_string(), manifest: Manifest {
version: String::new(), id: package_id.to_string(),
description: Description { title: package_id.to_string(),
short: "Installing...".to_string(), version: String::new(),
long: String::new(), description: Description {
}, short: "Installing...".to_string(),
release_notes: String::new(), long: String::new(),
license: String::new(), },
wrapper_repo: String::new(), release_notes: String::new(),
upstream_repo: String::new(), license: String::new(),
support_site: String::new(), wrapper_repo: String::new(),
marketing_site: String::new(), upstream_repo: String::new(),
donation_url: None, support_site: String::new(),
author: None, marketing_site: String::new(),
website: None, donation_url: None,
interfaces: None, author: None,
tier: None, website: None,
}, interfaces: None,
installed: None, tier: None,
install_progress: None, },
uninstall_stage: None, installed: None,
available_update: None, install_progress: None,
}); uninstall_stage: None,
entry.state = PackageState::Installing; available_update: None,
state_manager.update_data(data).await; });
entry.ui_ready = Some(false);
entry.state = PackageState::Installing;
})
.await;
} }
/// True when the failed install still has a real footprint: any container /// True when the failed install still has a real footprint: any container
@@ -485,20 +506,23 @@ async fn remove_entry_with_notification(
id_prefix: &str, id_prefix: &str,
message: &str, message: &str,
) { ) {
let (mut data, _) = handler.state_manager.get_snapshot().await; handler
data.package_data.remove(package_id); .state_manager
data.notifications.push(crate::data_model::Notification { .mutate_data(|data| {
id: format!("{id_prefix}-{package_id}"), data.package_data.remove(package_id);
level: crate::data_model::NotificationLevel::Error, data.notifications.push(crate::data_model::Notification {
title: format!("Could not install {package_id}"), id: format!("{id_prefix}-{package_id}"),
message: message.to_string(), level: crate::data_model::NotificationLevel::Error,
timestamp: chrono::Utc::now().to_rfc3339(), title: format!("Could not install {package_id}"),
app_id: Some(package_id.to_string()), message: message.to_string(),
}); timestamp: chrono::Utc::now().to_rfc3339(),
while data.notifications.len() > 20 { app_id: Some(package_id.to_string()),
data.notifications.remove(0); });
} while data.notifications.len() > 20 {
handler.state_manager.update_data(data).await; data.notifications.remove(0);
}
})
.await;
} }
/// Flip an existing entry's state and return the pre-flip value (or None if /// Flip an existing entry's state and return the pre-flip value (or None if
@@ -508,18 +532,23 @@ async fn flip_package_state(
package_id: &str, package_id: &str,
new_state: PackageState, new_state: PackageState,
) -> Option<PackageState> { ) -> Option<PackageState> {
let (mut data, _) = state_manager.get_snapshot().await; state_manager
let prev = data.package_data.get(package_id).map(|e| e.state.clone()); .mutate_data(|data| {
if let Some(entry) = data.package_data.get_mut(package_id) { let prev = data.package_data.get(package_id).map(|e| e.state.clone());
entry.state = new_state; if let Some(entry) = data.package_data.get_mut(package_id) {
state_manager.update_data(data).await; if new_state != PackageState::Running {
} else { entry.ui_ready = Some(false);
warn!( }
"flip_package_state: no entry for {} — cannot flip", entry.state = new_state;
package_id } else {
); warn!(
} "flip_package_state: no entry for {} — cannot flip",
prev package_id
);
}
prev
})
.await
} }
/// Set state unconditionally (no-op if entry no longer exists). /// Set state unconditionally (no-op if entry no longer exists).
@@ -528,13 +557,18 @@ async fn set_package_state(
package_id: &str, package_id: &str,
new_state: PackageState, new_state: PackageState,
) { ) {
let (mut data, _) = state_manager.get_snapshot().await; state_manager
if let Some(entry) = data.package_data.get_mut(package_id) { .mutate_data(|data| {
if entry.state != new_state { if let Some(entry) = data.package_data.get_mut(package_id) {
entry.state = new_state; if entry.state != new_state {
state_manager.update_data(data).await; if new_state != PackageState::Running {
} entry.ui_ready = Some(false);
} }
entry.state = new_state;
}
}
})
.await
} }
/// Set state and clear the uninstall_stage label. Used when an uninstall /// Set state and clear the uninstall_stage label. Used when an uninstall
@@ -545,12 +579,17 @@ async fn set_package_state_and_clear_uninstall_stage(
package_id: &str, package_id: &str,
new_state: PackageState, new_state: PackageState,
) { ) {
let (mut data, _) = state_manager.get_snapshot().await; state_manager
if let Some(entry) = data.package_data.get_mut(package_id) { .mutate_data(|data| {
entry.state = new_state; if let Some(entry) = data.package_data.get_mut(package_id) {
entry.uninstall_stage = None; if new_state != PackageState::Running {
state_manager.update_data(data).await; entry.ui_ready = Some(false);
} }
entry.state = new_state;
entry.uninstall_stage = None;
}
})
.await
} }
/// Kick the container scanner to run immediately and wait for it to finish /// Kick the container scanner to run immediately and wait for it to finish
@@ -985,28 +985,26 @@ pub(super) async fn get_app_config(
) )
} }
"nginx-proxy-manager" => { "nginx-proxy-manager" => {
let storage = crate::container::npm::resolve_storage().await?;
let admin_port = allocator let admin_port = allocator
.allocate_or_get(app_id, 8081, 81) .allocate_or_get(app_id, 8081, 81)
.await .await
.unwrap_or(8081); .unwrap_or(8081);
let http_port = allocator let http_port = allocator
.allocate_or_get("nginx-proxy-manager-http", 8084, 80) .allocate_or_get("nginx-proxy-manager-http", 8088, 80)
.await .await
.unwrap_or(8084); .unwrap_or(8088);
let https_port = allocator let https_port = allocator
.allocate_or_get("nginx-proxy-manager-https", 8444, 443) .allocate_or_get("nginx-proxy-manager-https", 8444, 443)
.await .await
.unwrap_or(8444); .unwrap_or(8444);
( (
vec![ vec![
format!("{}:81", admin_port), format!("127.0.0.1:{}:81", admin_port),
format!("{}:80", http_port), format!("127.0.0.1:{}:80", http_port),
format!("{}:443", https_port), format!("127.0.0.1:{}:443", https_port),
],
vec![
"/var/lib/archipelago/nginx-proxy-manager/data:/data".to_string(),
"/var/lib/archipelago/nginx-proxy-manager/letsencrypt:/etc/letsencrypt".to_string(),
], ],
storage.bind_mounts(),
vec![], vec![],
None, None,
None, None,
@@ -22,6 +22,18 @@ const ARCHIVAL_BITCOIN_DEPENDENCY: &str = "bitcoin:archival";
/// hardcoded id list below — a new app just declares the dependency instead /// hardcoded id list below — a new app just declares the dependency instead
/// of needing a code change here. /// of needing a code change here.
fn manifest_declares_archival_bitcoin(package_id: &str) -> bool { fn manifest_declares_archival_bitcoin(package_id: &str) -> bool {
// Registry-only apps need the same guard as OTA-bundled manifests. Honor
// the verified catalog's effective manifest before the disk fallback.
if let Some((_, value)) = crate::container::app_catalog::catalog_manifest_values()
.into_iter()
.find(|(id, _)| id == package_id)
{
if let Some(manifest) =
crate::container::app_catalog::catalog_manifest_overlay(package_id, value)
{
return dependency_list_declares_archival_bitcoin(&manifest.app.dependencies);
}
}
for apps_dir in manifest_apps_dirs() { for apps_dir in manifest_apps_dirs() {
let path = apps_dir.join(package_id).join("manifest.yml"); let path = apps_dir.join(package_id).join("manifest.yml");
let Ok(contents) = std::fs::read_to_string(&path) else { let Ok(contents) = std::fs::read_to_string(&path) else {
@@ -1055,6 +1067,14 @@ mod tests {
// edit to `requires_unpruned_bitcoin`. // edit to `requires_unpruned_bitcoin`.
assert!(manifest_declares_archival_bitcoin("electrumx")); assert!(manifest_declares_archival_bitcoin("electrumx"));
assert!(manifest_declares_archival_bitcoin("mempool")); assert!(manifest_declares_archival_bitcoin("mempool"));
let angor = archipelago_container::AppManifest::parse(include_str!(concat!(
env!("CARGO_MANIFEST_DIR"),
"/../../apps/angor-indexer/manifest.yml"
)))
.unwrap();
assert!(dependency_list_declares_archival_bitcoin(
&angor.app.dependencies
));
// An app whose manifest exists but never declares the marker. // An app whose manifest exists but never declares the marker.
assert!(!manifest_declares_archival_bitcoin("bitcoin-knots")); assert!(!manifest_declares_archival_bitcoin("bitcoin-knots"));
// An id with no manifest on disk at all. // An id with no manifest on disk at all.
+36 -227
View File
@@ -326,6 +326,10 @@ impl RpcHandler {
// an older version pins it so install_fresh resolves that image and the // an older version pins it so install_fresh resolves that image and the
// update badge stays suppressed. See docs/bitcoin-multi-version-design.md. // update badge stays suppressed. See docs/bitcoin-multi-version-design.md.
if matches!(package_id, "bitcoin-core" | "bitcoin-knots") { if matches!(package_id, "bitcoin-core" | "bitcoin-knots") {
if let Some(value) = params.get("prune") {
let prune = value.as_bool().context("prune must be a boolean")?;
crate::settings::bitcoin_storage::save(&self.config.data_dir, prune).await?;
}
if let Some(version) = params.get("version").and_then(|v| v.as_str()) { if let Some(version) = params.get("version").and_then(|v| v.as_str()) {
persist_install_version_selection(package_id, version).await; persist_install_version_selection(package_id, version).await;
} }
@@ -541,7 +545,7 @@ impl RpcHandler {
// Keep legacy install flow as default while migration is in progress. // Keep legacy install flow as default while migration is in progress.
if orchestrator_managed { if orchestrator_managed {
let orchestrator_app_id = orchestrator_install_app_id(package_id); let orchestrator_app_id = orchestrator_install_app_id(package_id);
self.set_install_phase(package_id, InstallPhase::CreatingContainer) self.set_install_phase(package_id, InstallPhase::PreparingApp)
.await; .await;
install_log(&format!( install_log(&format!(
"INSTALL ORCH: {} — attempting orchestrator install as {}", "INSTALL ORCH: {} — attempting orchestrator install as {}",
@@ -569,6 +573,9 @@ impl RpcHandler {
"message": format!("Package {} installed and started", package_id) "message": format!("Package {} installed and started", package_id)
})); }));
} }
Err(e) if e.downcast_ref::<crate::container::prod_orchestrator::InstallPrerequisiteError>().is_some() => {
return Err(super::dependencies::DependencyGateError(e.to_string()).into());
}
Err(e) if is_unknown_app_id_error(&e) => { Err(e) if is_unknown_app_id_error(&e) => {
info!( info!(
"Install {}: orchestrator has no manifest mapping yet, falling back to legacy installer", "Install {}: orchestrator has no manifest mapping yet, falling back to legacy installer",
@@ -686,7 +693,11 @@ impl RpcHandler {
// These standalone web UIs have repeatedly lost host listeners // These standalone web UIs have repeatedly lost host listeners
// under Podman's rootless pasta backend while staying healthy internally. // under Podman's rootless pasta backend while staying healthy internally.
// Use slirp4netns/rootlessport for this standalone web UI. // Use slirp4netns/rootlessport for this standalone web UI.
run_args.push("--network=slirp4netns:allow_host_loopback=true"); run_args.push(if package_id == "nginx-proxy-manager" {
"--network=slirp4netns:allow_host_loopback=true,cidr=169.254.1.0/24"
} else {
"--network=slirp4netns:allow_host_loopback=true"
});
} else if needs_archy_net(package_id) { } else if needs_archy_net(package_id) {
// Create archy-net if it doesn't exist (idempotent — "already exists" is fine) // Create archy-net if it doesn't exist (idempotent — "already exists" is fine)
match tokio::process::Command::new("podman") match tokio::process::Command::new("podman")
@@ -1561,88 +1572,8 @@ autopilot.active=false\n",
super::pine_ha::restart_home_assistant_if_running().await; super::pine_ha::restart_home_assistant_if_running().await;
} }
} }
if package_id == "filebrowser" { // File Browser credentials are provisioned and verified before the
// Generate a random password (32 bytes, hex-encoded) // server starts. Never attempt a default-password change after launch.
let mut buf = [0u8; 32];
rand::RngCore::fill_bytes(&mut rand::rngs::OsRng, &mut buf);
let password = hex::encode(buf);
let client = match reqwest::Client::builder()
.timeout(std::time::Duration::from_secs(10))
.build()
{
Ok(c) => c,
Err(e) => {
tracing::warn!("Failed to create HTTP client for FileBrowser hook: {}", e);
return;
}
};
// Retry loop: FileBrowser may take time to initialize its SQLite database
let mut password_changed = false;
for attempt in 0..6u32 {
let delay = if attempt == 0 { 5 } else { 10 };
tokio::time::sleep(std::time::Duration::from_secs(delay)).await;
// Try to log in with default credentials
let login_res = client
.post("http://127.0.0.1:8083/api/login")
.json(&serde_json::json!({"username": "admin", "password": "admin"}))
.send()
.await;
let token = match login_res {
Ok(resp) if resp.status().is_success() => match resp.text().await {
Ok(t) => t.trim_matches('"').to_string(),
Err(_) => continue,
},
_ => {
debug!("FileBrowser not ready (attempt {}/6)", attempt + 1);
continue;
}
};
// Change admin password
let change_res = client
.put("http://127.0.0.1:8083/api/users/1")
.header("X-Auth", &token)
.json(&serde_json::json!({"password": password}))
.send()
.await;
match change_res {
Ok(resp) if resp.status().is_success() => {
let secret_dir = "/var/lib/archipelago/secrets/filebrowser";
if let Err(e) = tokio::fs::create_dir_all(secret_dir).await {
tracing::warn!("Failed to create filebrowser secrets dir: {}", e);
}
let pw_path = format!("{}/password", secret_dir);
if let Err(e) = tokio::fs::write(&pw_path, &password).await {
tracing::warn!("Failed to write filebrowser password: {}", e);
}
// Set restrictive permissions on the password file
#[cfg(unix)]
{
use std::os::unix::fs::PermissionsExt;
let _ = std::fs::set_permissions(
&pw_path,
std::fs::Permissions::from_mode(0o600),
);
}
info!("FileBrowser admin password secured (default credentials replaced)");
password_changed = true;
break;
}
_ => continue,
}
}
if !password_changed {
tracing::warn!(
"FileBrowser password could not be changed after 6 attempts — \
default credentials (admin/admin) remain active"
);
}
}
// Auto-configure Tor hidden service for protocol services (LND, ElectrumX, Bitcoin) // Auto-configure Tor hidden service for protocol services (LND, ElectrumX, Bitcoin)
{ {
@@ -1695,32 +1626,10 @@ autopilot.active=false\n",
patch_indeedhub_nostr_provider().await; patch_indeedhub_nostr_provider().await;
} }
// Gitea: keep it on its native host port (3001). The UI opens Gitea // Gitea owns its public URL and security settings in app.ini, including
// in a new tab on that direct port so absolute asset URLs must be // values chosen in its first-run setup. Do not rewrite operator values
// rooted at the host port rather than Archipelago's /app/gitea/ path. // or claim success from best-effort grep/sed commands. The app gate
if package_id == "gitea" { // fronts its declared HTTP port and handles frame headers separately.
let _ = tokio::fs::remove_file("/etc/nginx/conf.d/gitea-iframe.conf").await;
// Set ROOT_URL to the direct launch route so links/assets stay
// anchored under the same origin Gitea is launched from.
let host_ip = &self.config.host_ip;
let _ = tokio::process::Command::new("podman")
.args(["exec", "gitea", "sh", "-c",
&format!("grep -q ROOT_URL /data/gitea/conf/app.ini && sed -i 's|ROOT_URL.*|ROOT_URL = http://{}:3001/|' /data/gitea/conf/app.ini || true", host_ip)])
.output()
.await;
// Also ensure X_FRAME_OPTIONS is empty so Gitea doesn't send the header
let _ = tokio::process::Command::new("podman")
.args(["exec", "gitea", "sh", "-c",
"grep -q X_FRAME_OPTIONS /data/gitea/conf/app.ini && sed -i 's|X_FRAME_OPTIONS.*|X_FRAME_OPTIONS =|' /data/gitea/conf/app.ini || sed -i '/^\\[security\\]/a X_FRAME_OPTIONS =' /data/gitea/conf/app.ini"])
.output()
.await;
info!(
"Gitea: ROOT_URL set to http://{}:3001/, X_FRAME_OPTIONS cleared",
host_ip
);
}
if package_id == "nextcloud" { if package_id == "nextcloud" {
let host_ip = &self.config.host_ip; let host_ip = &self.config.host_ip;
@@ -1927,10 +1836,10 @@ autopilot.active=false\n",
} }
pub(in crate::api::rpc) async fn handle_filebrowser_token(&self) -> Result<serde_json::Value> { pub(in crate::api::rpc) async fn handle_filebrowser_token(&self) -> Result<serde_json::Value> {
let secret_path = "/var/lib/archipelago/secrets/filebrowser/password"; let credentials = crate::container::filebrowser::cloud_credentials(std::path::Path::new(
let password = tokio::fs::read_to_string(secret_path) "/var/lib/archipelago/secrets/filebrowser",
.await ))
.unwrap_or_else(|_| "admin".to_string()); .await?;
let client = reqwest::Client::builder() let client = reqwest::Client::builder()
.timeout(std::time::Duration::from_secs(10)) .timeout(std::time::Duration::from_secs(10))
@@ -1939,7 +1848,7 @@ autopilot.active=false\n",
let resp = client let resp = client
.post("http://127.0.0.1:8083/api/login") .post("http://127.0.0.1:8083/api/login")
.json(&serde_json::json!({"username": "admin", "password": password})) .json(&serde_json::json!({"username": credentials.username, "password": credentials.password}))
.send() .send()
.await .await
.context("Failed to connect to FileBrowser")?; .context("Failed to connect to FileBrowser")?;
@@ -1969,17 +1878,16 @@ autopilot.active=false\n",
super::validation::validate_app_id(app_id)?; super::validation::validate_app_id(app_id)?;
if app_id == "filebrowser" { if app_id == "filebrowser" {
let password = let credentials = crate::container::filebrowser::cloud_credentials(
tokio::fs::read_to_string("/var/lib/archipelago/secrets/filebrowser/password") std::path::Path::new("/var/lib/archipelago/secrets/filebrowser"),
.await )
.map(|p| p.trim().to_string()) .await?;
.unwrap_or_else(|_| "admin".to_string());
return Ok(serde_json::json!({ return Ok(serde_json::json!({
"title": "File Browser credentials", "title": "File Browser credentials",
"description": "Use these credentials when File Browser asks you to sign in.", "description": "Use these credentials when File Browser asks you to sign in.",
"credentials": [ "credentials": [
{ "label": "Username", "value": "admin" }, { "label": "Username", "value": credentials.username },
{ "label": "Password", "value": password, "sensitive": true } { "label": "Password", "value": credentials.password, "sensitive": true }
] ]
})); }));
} }
@@ -2049,25 +1957,8 @@ fn parse_setup_token(lines: &[&str]) -> Option<String> {
} }
async fn cleanup_stale_package_ports(package_id: &str) { async fn cleanup_stale_package_ports(package_id: &str) {
match package_id { // Never kill by port: another app or the management gate may own it.
"grafana" => cleanup_stale_pasta_port("3000").await, crate::container::ghost_reaper::reap_for_app(package_id).await;
"homeassistant" | "home-assistant" => cleanup_stale_pasta_port("8123").await,
"searxng" => cleanup_stale_pasta_port("8888").await,
"uptime-kuma" => cleanup_stale_pasta_port("3002").await,
"gitea" => {
cleanup_stale_pasta_port("3001").await;
cleanup_stale_pasta_port("2222").await;
cleanup_stale_pasta_port("3000").await;
}
"nginx-proxy-manager" => {
cleanup_stale_pasta_port("8081").await;
cleanup_stale_pasta_port("8084").await;
cleanup_stale_pasta_port("8444").await;
}
"nextcloud" => cleanup_stale_pasta_port("8085").await,
"portainer" => cleanup_stale_pasta_port("9000").await,
_ => {}
}
} }
fn install_command_tail( fn install_command_tail(
@@ -2192,93 +2083,11 @@ async fn cleanup_start_conflict(package_id: &str, stderr: &str) -> bool {
return true; return true;
} }
match package_id { if stderr.contains("pasta failed") || stderr.contains("address already in use") {
"grafana" crate::container::ghost_reaper::reap_for_app(package_id).await;
if stderr.contains("pasta failed") || stderr.contains("address already in use") => return true;
{
cleanup_stale_pasta_port("3000").await;
true
}
"homeassistant" | "home-assistant"
if stderr.contains("pasta failed") || stderr.contains("address already in use") =>
{
cleanup_stale_pasta_port("8123").await;
true
}
"searxng"
if stderr.contains("pasta failed") || stderr.contains("address already in use") =>
{
cleanup_stale_pasta_port("8888").await;
true
}
"uptime-kuma"
if stderr.contains("pasta failed") || stderr.contains("address already in use") =>
{
cleanup_stale_pasta_port("3002").await;
true
}
"gitea" if stderr.contains("pasta failed") || stderr.contains("address already in use") => {
cleanup_stale_pasta_port("3001").await;
cleanup_stale_pasta_port("2222").await;
cleanup_stale_pasta_port("3000").await;
true
}
"nginx-proxy-manager"
if stderr.contains("pasta failed") || stderr.contains("address already in use") =>
{
cleanup_stale_pasta_port("8081").await;
cleanup_stale_pasta_port("8084").await;
cleanup_stale_pasta_port("8444").await;
true
}
"nextcloud"
if stderr.contains("pasta failed") || stderr.contains("address already in use") =>
{
cleanup_stale_pasta_port("8085").await;
true
}
"portainer"
if stderr.contains("pasta failed") || stderr.contains("address already in use") =>
{
cleanup_stale_pasta_port("9000").await;
true
}
_ => false,
} }
} false
async fn cleanup_stale_pasta_port(port: &str) {
// NEVER kill our own process. The daemon holds catalog app ports over
// IPv6 (the mesh app-port relay), so a blunt `fuser -k <port>/tcp` would
// terminate archipelago itself mid-install — installs failed and apps
// vanished on a test node 2026-07-27. Kill every listener on the port
// EXCEPT our PID (and our process group), leaving the relay/daemon alive.
let self_pid = std::process::id();
let kill_listener = format!(
"ss -ltnp 'sport = :{port}' 2>/dev/null | sed -n 's/.*pid=\\([0-9]*\\).*/\\1/p' | \
while read p; do [ \"$p\" = \"{self_pid}\" ] || kill \"$p\" 2>/dev/null; done || true",
);
let _ = tokio::process::Command::new("sh")
.args(["-c", &kill_listener])
.output()
.await;
// sudo fuser -k, but exclude our own PID: fuser prints the PIDs holding
// the port; kill each except self. (`fuser -k` has no exclusion flag.)
let fuser_kill = format!(
"for p in $(sudo fuser {port}/tcp 2>/dev/null); do [ \"$p\" = \"{self_pid}\" ] || sudo kill \"$p\" 2>/dev/null; done || true",
);
let _ = tokio::process::Command::new("sh")
.args(["-c", &fuser_kill])
.output()
.await;
let pattern = format!("pasta.*{}", port);
let _ = tokio::process::Command::new("pkill")
.args(["-f", &pattern])
.output()
.await;
tokio::time::sleep(std::time::Duration::from_secs(1)).await;
} }
async fn repair_nextcloud_permissions() { async fn repair_nextcloud_permissions() {
@@ -14,20 +14,23 @@ impl RpcHandler {
/// the rare case where the pull stream actually parses, but podman /// the rare case where the pull stream actually parses, but podman
/// almost never emits parseable progress on a piped stderr. /// almost never emits parseable progress on a piped stderr.
pub(super) async fn set_install_progress(&self, package_id: &str, downloaded: u64, size: u64) { pub(super) async fn set_install_progress(&self, package_id: &str, downloaded: u64, size: u64) {
let (mut data, _rev) = self.state_manager.get_snapshot().await; self.state_manager
let entry = data .mutate_data(|data| {
.package_data let entry = data
.entry(package_id.to_string()) .package_data
.or_insert_with(|| create_installing_entry(package_id)); .entry(package_id.to_string())
entry.state = PackageState::Installing; .or_insert_with(|| create_installing_entry(package_id));
let existing_phase = entry.install_progress.as_ref().and_then(|p| p.phase); entry.ui_ready = Some(false);
entry.install_progress = Some(InstallProgress { entry.state = PackageState::Installing;
size, let existing_phase = entry.install_progress.as_ref().and_then(|p| p.phase);
downloaded, entry.install_progress = Some(InstallProgress {
phase: existing_phase, size,
message: None, downloaded,
}); phase: existing_phase,
self.state_manager.update_data(data).await; message: None,
});
})
.await;
} }
/// Set the install pipeline phase and broadcast. This is the /// Set the install pipeline phase and broadcast. This is the
@@ -35,76 +38,86 @@ impl RpcHandler {
/// percentage and a user-facing label. Byte counters are retained /// percentage and a user-facing label. Byte counters are retained
/// for the rare case podman emits parseable progress. /// for the rare case podman emits parseable progress.
pub(super) async fn set_install_phase(&self, package_id: &str, phase: InstallPhase) { pub(super) async fn set_install_phase(&self, package_id: &str, phase: InstallPhase) {
let (mut data, _rev) = self.state_manager.get_snapshot().await; self.state_manager
let entry = data .mutate_data(|data| {
.package_data let entry = data
.entry(package_id.to_string()) .package_data
.or_insert_with(|| create_installing_entry(package_id)); .entry(package_id.to_string())
// Preparing / PullingImage / CreatingContainer / StartingContainer / .or_insert_with(|| create_installing_entry(package_id));
// WaitingHealthy / PostInstall all map to the Installing state. // Preparing / PullingImage / CreatingContainer / StartingContainer /
// Updates use Updating state — the wrapper has already flipped // WaitingHealthy / PostInstall all map to the Installing state.
// state to Updating, so don't clobber it. // Updates use Updating state — the wrapper has already flipped
if entry.state != PackageState::Updating { // state to Updating, so don't clobber it.
entry.state = PackageState::Installing; if entry.state != PackageState::Updating {
} entry.ui_ready = Some(false);
let (size, downloaded) = entry entry.state = PackageState::Installing;
.install_progress }
.as_ref() let (size, downloaded) = entry
.map(|p| (p.size, p.downloaded)) .install_progress
.unwrap_or((0, 0)); .as_ref()
entry.install_progress = Some(InstallProgress { .map(|p| (p.size, p.downloaded))
size, .unwrap_or((0, 0));
downloaded, entry.install_progress = Some(InstallProgress {
phase: Some(phase), size,
message: None, downloaded,
}); phase: Some(phase),
self.state_manager.update_data(data).await; message: None,
});
})
.await;
} }
/// Set a user-facing install status message (e.g. "Waiting for Bitcoin /// Set a user-facing install status message (e.g. "Waiting for Bitcoin
/// to start…") without disturbing the current phase/byte counters. /// to start…") without disturbing the current phase/byte counters.
pub(super) async fn set_install_message(&self, package_id: &str, message: &str) { pub(super) async fn set_install_message(&self, package_id: &str, message: &str) {
let (mut data, _rev) = self.state_manager.get_snapshot().await; self.state_manager
let entry = data .mutate_data(|data| {
.package_data let entry = data
.entry(package_id.to_string()) .package_data
.or_insert_with(|| create_installing_entry(package_id)); .entry(package_id.to_string())
if entry.state != PackageState::Updating { .or_insert_with(|| create_installing_entry(package_id));
entry.state = PackageState::Installing; if entry.state != PackageState::Updating {
} entry.ui_ready = Some(false);
let (size, downloaded, phase) = entry entry.state = PackageState::Installing;
.install_progress }
.as_ref() let (size, downloaded, phase) = entry
.map(|p| (p.size, p.downloaded, p.phase)) .install_progress
.unwrap_or((0, 0, None)); .as_ref()
entry.install_progress = Some(InstallProgress { .map(|p| (p.size, p.downloaded, p.phase))
size, .unwrap_or((0, 0, None));
downloaded, entry.install_progress = Some(InstallProgress {
phase, size,
message: Some(message.to_string()), downloaded,
}); phase,
self.state_manager.update_data(data).await; message: Some(message.to_string()),
});
})
.await;
} }
/// Clear install progress after pull completes or fails. /// Clear install progress after pull completes or fails.
pub(super) async fn clear_install_progress(&self, package_id: &str) { pub(super) async fn clear_install_progress(&self, package_id: &str) {
let (mut data, _rev) = self.state_manager.get_snapshot().await; self.state_manager
if let Some(entry) = data.package_data.get_mut(package_id) { .mutate_data(|data| {
entry.install_progress = None; if let Some(entry) = data.package_data.get_mut(package_id) {
} entry.install_progress = None;
self.state_manager.update_data(data).await; }
})
.await;
} }
/// Set the uninstall stage label so the UI can show what's happening /// Set the uninstall stage label so the UI can show what's happening
/// instead of a generic spinner. Each call broadcasts a state change /// instead of a generic spinner. Each call broadcasts a state change
/// — call sparingly (one per pipeline phase, not per container). /// — call sparingly (one per pipeline phase, not per container).
pub(super) async fn set_uninstall_stage(&self, package_id: &str, stage: &str) { pub(super) async fn set_uninstall_stage(&self, package_id: &str, stage: &str) {
let (mut data, _rev) = self.state_manager.get_snapshot().await; self.state_manager
if let Some(entry) = data.package_data.get_mut(package_id) { .mutate_data(|data| {
entry.uninstall_stage = Some(stage.to_string()); if let Some(entry) = data.package_data.get_mut(package_id) {
entry.state = crate::data_model::PackageState::Removing; entry.uninstall_stage = Some(stage.to_string());
} entry.state = crate::data_model::PackageState::Removing;
self.state_manager.update_data(data).await; }
})
.await;
} }
/// Update install progress (static method for use in async closures). /// Update install progress (static method for use in async closures).
@@ -114,25 +127,28 @@ impl RpcHandler {
downloaded: u64, downloaded: u64,
total: u64, total: u64,
) { ) {
let (mut data, _rev) = state_manager.get_snapshot().await; state_manager
let entry = data .mutate_data(|data| {
.package_data let entry = data
.entry(package_id.to_string()) .package_data
.or_insert_with(|| create_installing_entry(package_id)); .entry(package_id.to_string())
let existing_phase = entry.install_progress.as_ref().and_then(|p| p.phase); .or_insert_with(|| create_installing_entry(package_id));
entry.install_progress = Some(InstallProgress { let existing_phase = entry.install_progress.as_ref().and_then(|p| p.phase);
size: total, entry.install_progress = Some(InstallProgress {
downloaded, size: total,
phase: existing_phase, downloaded,
message: None, phase: existing_phase,
}); message: None,
state_manager.update_data(data).await; });
})
.await;
} }
} }
/// Create a minimal PackageDataEntry for a package being installed. /// Create a minimal PackageDataEntry for a package being installed.
fn create_installing_entry(package_id: &str) -> PackageDataEntry { fn create_installing_entry(package_id: &str) -> PackageDataEntry {
PackageDataEntry { PackageDataEntry {
ui_ready: None,
state: PackageState::Installing, state: PackageState::Installing,
health: None, health: None,
exit_code: None, exit_code: None,
+256 -125
View File
@@ -1431,10 +1431,9 @@ async fn repair_before_package_start(container_name: &str) {
// published port and the data-dir file locks, so the replacement either // published port and the data-dir file locks, so the replacement either
// fails to bind (`address already in use`) or starts and dies on the // fails to bind (`address already in use`) or starts and dies on the
// lock — and `Restart=always` loops it there forever. Ordered before // lock — and `Restart=always` loops it there forever. Ordered before
// the port cleanup below: killing the owner is what actually frees the // starting the replacement. A port sweep cannot distinguish a ghost
// port, and the port sweep alone cannot tell a ghost from a live app. // from the dashboard gate or another live app and must never kill it.
crate::container::ghost_reaper::reap_for_app(container_name).await; crate::container::ghost_reaper::reap_for_app(container_name).await;
cleanup_runtime_host_ports(container_name).await;
} }
async fn wait_before_package_start(container_name: &str) { async fn wait_before_package_start(container_name: &str) {
@@ -1577,42 +1576,110 @@ async fn repair_netbird_network() {
} }
async fn repair_nginx_proxy_manager_container() { async fn repair_nginx_proxy_manager_container() {
repair_nginx_proxy_manager_dirs().await; // Quadlet owns managed containers; its backed-up reconciliation applies
if !nginx_proxy_manager_has_legacy_admin_port().await { // port and mount changes. Never remove a systemd-owned container here.
cleanup_nginx_proxy_manager_ports().await; if crate::container::quadlet::unit_exists("nginx-proxy-manager").await {
return; return;
} }
// Serialize repair so a second caller cannot overlap a replacement.
install_log( static REPAIR: tokio::sync::Mutex<()> = tokio::sync::Mutex::const_new(());
"START REPAIR: nginx-proxy-manager - recreating stale container using host port 8081", let _repair = REPAIR.lock().await;
) if !nginx_proxy_manager_has_legacy_admin_port().await {
.await; return;
let _ = podman_control(&["rm", "-f", "nginx-proxy-manager"]).await; }
cleanup_nginx_proxy_manager_ports().await; if let Err(error) = repair_legacy_nginx_proxy_manager().await {
if let Err(err) = recreate_nginx_proxy_manager_container().await { tracing::warn!(error = %error, "NPM legacy repair failed; persistent state preserved");
tracing::warn!(error = %err, "failed to recreate stale nginx-proxy-manager container");
} }
} }
async fn repair_nginx_proxy_manager_dirs() { const NPM_PREVIOUS_CONTAINER: &str = "archy-npm-upgrade-previous";
let _ = tokio::process::Command::new("sudo")
.args([ async fn restore_failed_npm_repair() -> Result<()> {
"mkdir", let removed = podman_control(&["rm", "-f", "--ignore", "nginx-proxy-manager"]).await?;
"-p", anyhow::ensure!(
"/var/lib/archipelago/nginx-proxy-manager/data/letsencrypt-acme-challenge/.well-known/acme-challenge", removed.status.success(),
"/var/lib/archipelago/nginx-proxy-manager/letsencrypt", "cannot remove failed NPM replacement; previous container retained"
]) );
.output() let renamed =
.await; podman_control(&["rename", NPM_PREVIOUS_CONTAINER, "nginx-proxy-manager"]).await?;
let _ = tokio::process::Command::new("sudo") anyhow::ensure!(
.args([ renamed.status.success(),
"chown", "cannot restore previous NPM container name"
"-R", );
"1000:1000", let started = podman_control(&["start", "nginx-proxy-manager"]).await?;
"/var/lib/archipelago/nginx-proxy-manager", anyhow::ensure!(
]) started.status.success(),
.output() "previous NPM container restored but failed to start"
.await; );
Ok(())
}
async fn repair_legacy_nginx_proxy_manager() -> Result<()> {
let previous = podman_control(&["container", "exists", NPM_PREVIOUS_CONTAINER]).await?;
anyhow::ensure!(previous.status.code() == Some(1),
"NPM previous-container slot is occupied or cannot be inspected; preserve it and review interrupted repair before proceeding");
let inspection = podman_control(&[
"inspect",
"nginx-proxy-manager",
"--format",
"{{json .Config.Env}}",
])
.await?;
anyhow::ensure!(
inspection.status.success(),
"cannot preserve NPM environment before repair"
);
let environment: Vec<String> =
serde_json::from_slice(&inspection.stdout).context("invalid original NPM environment")?;
let environment = npm_repair_environment(&environment)?;
let storage = crate::container::npm::resolve_storage().await?;
let mut manifest: archipelago_container::AppManifest = serde_yaml::from_str(include_str!(
"../../../../../../apps/nginx-proxy-manager/manifest.yml"
))?;
storage.apply(&mut manifest)?;
let stopped = podman_control(&["stop", "--time", "30", "nginx-proxy-manager"]).await?;
anyhow::ensure!(
stopped.status.success(),
"could not stop NPM for a consistent backup"
);
if let Err(error) = crate::container::migration_backup::snapshot(
&manifest,
std::path::Path::new("/var/lib/archipelago"),
None,
)
.await
{
let _ = podman_control(&["start", "nginx-proxy-manager"]).await;
return Err(error);
}
// Keep the original runtime definition for rollback, including its operator
// options. Never delete it before the replacement has become ready.
let renamed = podman_control(&["rename", "nginx-proxy-manager", NPM_PREVIOUS_CONTAINER]).await;
if !renamed.as_ref().is_ok_and(|out| out.status.success()) {
let _ = podman_control(&["start", "nginx-proxy-manager"]).await;
anyhow::bail!("could not retain legacy NPM runtime; state backup preserved, inspect both container names before retrying");
}
let replacement = async {
recreate_nginx_proxy_manager_container(&storage, &environment).await?;
anyhow::ensure!(
wait_for_runtime_host_port("nginx-proxy-manager", 8081, 180).await,
"replacement NPM admin listener did not become ready"
);
Ok::<_, anyhow::Error>(())
}
.await;
if let Err(error) = replacement {
restore_failed_npm_repair()
.await
.context("restoring previous NPM after replacement failure")?;
return Err(error);
}
let removed = podman_control(&["rm", NPM_PREVIOUS_CONTAINER]).await?;
anyhow::ensure!(
removed.status.success(),
"replacement ready but previous NPM cleanup failed; rollback container retained"
);
Ok(())
} }
async fn nginx_proxy_manager_has_legacy_admin_port() -> bool { async fn nginx_proxy_manager_has_legacy_admin_port() -> bool {
@@ -1647,36 +1714,75 @@ async fn nginx_proxy_manager_has_legacy_admin_port() -> bool {
ports.contains(":81->81/tcp") || ports.contains(":8443->443/tcp") ports.contains(":81->81/tcp") || ports.contains(":8443->443/tcp")
} }
async fn recreate_nginx_proxy_manager_container() -> Result<()> { fn npm_repair_environment(values: &[String]) -> Result<Vec<(String, String)>> {
tokio::process::Command::new("sudo") values.iter().map(|value| {
.args([ let (key, value) = value.split_once('=').context("invalid NPM environment entry")?;
"mkdir", anyhow::ensure!(!key.is_empty() && key.chars().all(|c| c.is_ascii_alphanumeric() || c == '_'),
"-p", "unsupported NPM environment name; original container preserved");
"/var/lib/archipelago/nginx-proxy-manager/data/letsencrypt-acme-challenge/.well-known/acme-challenge", anyhow::ensure!(!value.contains(['\n', '\r', '\0']),
"/var/lib/archipelago/nginx-proxy-manager/letsencrypt", "NPM environment requires explicit migration of a multiline value; original container preserved");
]) Ok((key.to_owned(), value.to_owned()))
.output() }).collect()
.await }
.context("failed to create nginx-proxy-manager data directories")?;
let _ = tokio::process::Command::new("sudo")
.args([
"chown",
"-R",
"1000:1000",
"/var/lib/archipelago/nginx-proxy-manager",
])
.output()
.await;
let image = crate::container::image_versions::pinned_image_for_app("nginx-proxy-manager") struct NpmRepairEnvironmentFile(std::path::PathBuf);
.unwrap_or_else(|| "docker.io/jc21/nginx-proxy-manager:latest".to_string());
impl NpmRepairEnvironmentFile {
fn create(environment: &[(String, String)]) -> Result<Self> {
use std::io::Write;
use std::os::unix::fs::OpenOptionsExt;
let path = std::env::temp_dir().join(format!(".archy-npm-env-{}", uuid::Uuid::new_v4()));
let mut file = std::fs::OpenOptions::new()
.write(true)
.create_new(true)
.mode(0o600)
.open(&path)?;
let guard = Self(path);
for (key, value) in environment {
writeln!(file, "{key}={value}")?;
}
file.sync_all()?;
Ok(guard)
}
}
impl Drop for NpmRepairEnvironmentFile {
fn drop(&mut self) {
let _ = std::fs::remove_file(&self.0);
}
}
async fn recreate_nginx_proxy_manager_container(
storage: &crate::container::npm::Storage,
environment: &[(String, String)],
) -> Result<()> {
// Existing directories and ownership came from the active runtime. Never
// create a second database tree or recursively rewrite data permissions.
for directory in [&storage.data, &storage.certificates] {
anyhow::ensure!(
std::path::Path::new(directory).is_dir(),
"NPM persistent directory is missing"
);
}
// This repair changes connectivity, not NPM's application version. Keep
// the exact old image so rollback never starts an older binary against a
// database that an incidental mutable-tag update may have migrated.
let image_output =
podman_control(&["inspect", NPM_PREVIOUS_CONTAINER, "--format", "{{.Image}}"]).await?;
anyhow::ensure!(
image_output.status.success(),
"cannot resolve original NPM image for repair"
);
let image = String::from_utf8(image_output.stdout)?.trim().to_string();
anyhow::ensure!(!image.is_empty(), "original NPM image is missing");
let mut args = vec![ let mut args = vec![
"run".to_string(), "run".to_string(),
"-d".to_string(), "-d".to_string(),
"--name".to_string(), "--name".to_string(),
"nginx-proxy-manager".to_string(), "nginx-proxy-manager".to_string(),
"--restart=unless-stopped".to_string(), "--restart=unless-stopped".to_string(),
"--network=slirp4netns:allow_host_loopback=true".to_string(), "--network=slirp4netns:allow_host_loopback=true,cidr=169.254.1.0/24".to_string(),
"--cap-drop=ALL".to_string(), "--cap-drop=ALL".to_string(),
"--security-opt=no-new-privileges:true".to_string(), "--security-opt=no-new-privileges:true".to_string(),
"--pids-limit=4096".to_string(), "--pids-limit=4096".to_string(),
@@ -1684,24 +1790,32 @@ async fn recreate_nginx_proxy_manager_container() -> Result<()> {
args.extend(get_app_capabilities("nginx-proxy-manager")); args.extend(get_app_capabilities("nginx-proxy-manager"));
args.extend([ args.extend([
"-p".to_string(), "-p".to_string(),
"8081:81".to_string(), "127.0.0.1:8081:81".to_string(),
"-p".to_string(), "-p".to_string(),
"8084:80".to_string(), "127.0.0.1:8088:80".to_string(),
"-p".to_string(), "-p".to_string(),
"8444:443".to_string(), "127.0.0.1:8444:443".to_string(),
"-v".to_string(), "-v".to_string(),
"/var/lib/archipelago/nginx-proxy-manager/data:/data".to_string(), format!("{}:/data", storage.data),
"-v".to_string(), "-v".to_string(),
"/var/lib/archipelago/nginx-proxy-manager/letsencrypt:/etc/letsencrypt".to_string(), format!("{}:/etc/letsencrypt", storage.certificates),
"--memory".to_string(), "--memory".to_string(),
get_memory_limit("nginx-proxy-manager").to_string(), get_memory_limit("nginx-proxy-manager").to_string(),
"--cpus=2".to_string(), "--cpus=2".to_string(),
]); ]);
args.extend(get_health_check_args("nginx-proxy-manager", "")); args.extend(get_health_check_args("nginx-proxy-manager", ""));
// Keep values out of argv/logs AND out of Podman's host environment
// (a container's PATH or LD_PRELOAD must never alter the host command).
let env_file = NpmRepairEnvironmentFile::create(environment)?;
args.extend([
"--env-file".to_string(),
env_file.0.to_string_lossy().into_owned(),
]);
args.push(image); args.push(image);
let refs = args.iter().map(String::as_str).collect::<Vec<_>>(); let mut command = tokio::process::Command::new("podman");
let output = podman_control(&refs).await?; command.args(&args);
let output = command_with_timeout(command, Duration::from_secs(120), "NPM replacement").await?;
if !output.status.success() { if !output.status.success() {
anyhow::bail!( anyhow::bail!(
"podman run nginx-proxy-manager failed: {}", "podman run nginx-proxy-manager failed: {}",
@@ -1769,7 +1883,7 @@ fn runtime_host_ports(container_name: &str) -> Vec<u16> {
"vaultwarden" => vec![8082], "vaultwarden" => vec![8082],
"gitea" => vec![3001, 2222, 3000], "gitea" => vec![3001, 2222, 3000],
"nextcloud" => vec![8085], "nextcloud" => vec![8085],
"nginx-proxy-manager" => vec![8081, 8084, 8444], "nginx-proxy-manager" => vec![8081, 8088, 8444],
_ => Vec::new(), _ => Vec::new(),
}; };
ports ports
@@ -1780,7 +1894,7 @@ fn with_legacy_extra_ports(container_name: &str, mut ports: Vec<u16>) -> Vec<u16
ports.push(3000); ports.push(3000);
} }
if container_name == "nginx-proxy-manager" { if container_name == "nginx-proxy-manager" {
for port in [8084, 8444] { for port in [8088, 8444] {
if !ports.contains(&port) { if !ports.contains(&port) {
ports.push(port); ports.push(port);
} }
@@ -1812,6 +1926,9 @@ fn manifest_host_ports(container_name: &str) -> Vec<u16> {
pub(super) fn manifest_apps_dirs() -> Vec<std::path::PathBuf> { pub(super) fn manifest_apps_dirs() -> Vec<std::path::PathBuf> {
let mut dirs = Vec::new(); let mut dirs = Vec::new();
if let Some(root) = std::env::var_os("ARCHIPELAGO_APPS_DIR") {
dirs.push(root.into());
}
if let Ok(manifest_dir) = std::env::var("CARGO_MANIFEST_DIR") { if let Ok(manifest_dir) = std::env::var("CARGO_MANIFEST_DIR") {
dirs.push(Path::new(&manifest_dir).join("../../apps")); dirs.push(Path::new(&manifest_dir).join("../../apps"));
} }
@@ -1842,6 +1959,7 @@ async fn wait_for_runtime_host_port(container_name: &str, port: u16, timeout_sec
loop { loop {
let ready = match container_name { let ready = match container_name {
"uptime-kuma" => http_host_port_ready(port, "/").await, "uptime-kuma" => http_host_port_ready(port, "/").await,
"nginx-proxy-manager" => http_host_port_ready(port, "/api/").await,
_ => tokio::net::TcpStream::connect(("127.0.0.1", port)) _ => tokio::net::TcpStream::connect(("127.0.0.1", port))
.await .await
.is_ok(), .is_ok(),
@@ -2032,51 +2150,10 @@ async fn cleanup_start_conflict(container_name: &str, stderr: &str) {
return; return;
} }
let ports = runtime_host_ports(container_name); // Only reap processes proven to belong to an absent container. The app
if !ports.is_empty() { // gate shares the app's port on other addresses and lives in this daemon;
cleanup_ports(&ports).await; // killing port owners (or matching argv with pkill) kills the dashboard.
return; crate::container::ghost_reaper::reap_for_app(container_name).await;
}
}
async fn cleanup_runtime_host_ports(container_name: &str) {
let ports = runtime_host_ports(container_name);
if !ports.is_empty() {
cleanup_ports(&ports).await;
}
}
async fn cleanup_nginx_proxy_manager_ports() {
cleanup_ports(&[8081, 8084, 8444]).await;
}
async fn cleanup_ports(ports: &[u16]) {
for port in ports {
cleanup_stale_pasta_port(&port.to_string()).await;
}
}
async fn cleanup_stale_pasta_port(port: &str) {
let kill_listener = format!(
"ss -ltnp 'sport = :{}' 2>/dev/null | sed -n 's/.*pid=\\([0-9]*\\).*/\\1/p' | xargs -r kill 2>/dev/null || true",
port
);
let _ = tokio::process::Command::new("sh")
.args(["-c", &kill_listener])
.output()
.await;
let pattern = format!("pasta.*{}", port);
let _ = tokio::process::Command::new("pkill")
.args(["-f", &pattern])
.output()
.await;
let pattern = format!("rootlessport.*{}", port);
let _ = tokio::process::Command::new("pkill")
.args(["-f", &pattern])
.output()
.await;
tokio::time::sleep(std::time::Duration::from_secs(1)).await;
} }
pub(super) fn is_missing_companion_ok(name: &str, stderr: &str) -> bool { pub(super) fn is_missing_companion_ok(name: &str, stderr: &str) -> bool {
@@ -2095,13 +2172,16 @@ async fn flip_package_state(
package_id: &str, package_id: &str,
transitional: PackageState, transitional: PackageState,
) -> Option<PackageState> { ) -> Option<PackageState> {
let (mut data, _) = state_manager.get_snapshot().await; state_manager
let prev = data.package_data.get(package_id).map(|e| e.state.clone()); .mutate_data(|data| {
if let Some(entry) = data.package_data.get_mut(package_id) { let prev = data.package_data.get(package_id).map(|e| e.state.clone());
entry.state = transitional; if let Some(entry) = data.package_data.get_mut(package_id) {
state_manager.update_data(data).await; entry.ui_ready = Some(false);
} entry.state = transitional;
prev }
prev
})
.await
} }
/// Write the package entry's final state. No-op if the entry has since /// Write the package entry's final state. No-op if the entry has since
@@ -2111,13 +2191,18 @@ async fn set_package_state(
package_id: &str, package_id: &str,
new_state: PackageState, new_state: PackageState,
) { ) {
let (mut data, _) = state_manager.get_snapshot().await; state_manager
if let Some(entry) = data.package_data.get_mut(package_id) { .mutate_data(|data| {
if entry.state != new_state { if let Some(entry) = data.package_data.get_mut(package_id) {
entry.state = new_state; if entry.state != new_state {
state_manager.update_data(data).await; if new_state != PackageState::Running {
} entry.ui_ready = Some(false);
} }
entry.state = new_state;
}
}
})
.await
} }
pub(super) async fn reconcile_companions_for(package_id: &str) { pub(super) async fn reconcile_companions_for(package_id: &str) {
@@ -2183,8 +2268,54 @@ pub(super) fn orchestrator_uninstall_app_ids(package_id: &str) -> Vec<String> {
#[cfg(test)] #[cfg(test)]
mod tests { mod tests {
#[test]
fn npm_environment_backup_is_private_exact_and_removed_on_drop() {
use std::os::unix::fs::PermissionsExt;
let values = vec![
"DB_PASSWORD=fixture=a b".to_string(),
"PATH=/container/only".to_string(),
];
let parsed = super::npm_repair_environment(&values).unwrap();
let host_path = std::env::var_os("PATH");
let path = {
let file = super::NpmRepairEnvironmentFile::create(&parsed).unwrap();
assert_eq!(
std::fs::metadata(&file.0).unwrap().permissions().mode() & 0o777,
0o600
);
assert_eq!(
std::fs::read_to_string(&file.0).unwrap(),
"DB_PASSWORD=fixture=a b\nPATH=/container/only\n"
);
assert_eq!(std::env::var_os("PATH"), host_path);
file.0.clone()
};
assert!(!path.exists());
for value in [
"INVALID",
"=empty key",
"KEY=value\nINJECTED=true",
"--env=value",
] {
assert!(super::npm_repair_environment(&[value.to_string()]).is_err());
}
}
use super::*; use super::*;
#[tokio::test]
async fn port_conflict_cleanup_preserves_live_host_listener() {
// The previous ss|kill sweep terminated the daemon's app gate on a
// restart. Keep a real listening socket owned by this test process.
let listener = tokio::net::TcpListener::bind("127.0.0.2:2342")
.await
.unwrap();
let addr = listener.local_addr().unwrap();
cleanup_start_conflict("photoprism", "address already in use").await;
let client = tokio::net::TcpStream::connect(addr).await.unwrap();
let _connection = listener.accept().await.unwrap();
drop(client);
}
#[test] #[test]
fn missing_container_classifier_covers_podman5_phrasings() { fn missing_container_classifier_covers_podman5_phrasings() {
// Regression (.228 gate 2026-07-08): podman 5.x `inspect` on a missing // Regression (.228 gate 2026-07-08): podman 5.x `inspect` on a missing
@@ -153,8 +153,18 @@ impl RpcHandler {
let default = app_catalog::catalog_default_version(app_id); let default = app_catalog::catalog_default_version(app_id);
let cfg = version_config::read(app_id); let cfg = version_config::read(app_id);
let installed = installed_version(app_id).await; let installed = installed_version(app_id).await;
let bitcoin_prune = if matches!(app_id, "bitcoin-core" | "bitcoin-knots") {
Some(
crate::settings::bitcoin_storage::load(&self.config.data_dir)
.await?
.prune,
)
} else {
None
};
Ok(serde_json::json!({ Ok(serde_json::json!({
"bitcoinPrune": bitcoin_prune,
"id": app_id, "id": app_id,
"supportsVersions": supports_versions(app_id), "supportsVersions": supports_versions(app_id),
"default": default, "default": default,
+22 -14
View File
@@ -150,23 +150,31 @@ async fn flip_to_transitional(
app_id: &str, app_id: &str,
transitional: PackageState, transitional: PackageState,
) -> Option<PackageState> { ) -> Option<PackageState> {
let (mut data, _) = state_manager.get_snapshot().await; state_manager
let prev = data.package_data.get(app_id).map(|e| e.state.clone()); .mutate_data(|data| {
if let Some(entry) = data.package_data.get_mut(app_id) { let prev = data.package_data.get(app_id).map(|e| e.state.clone());
entry.state = transitional; if let Some(entry) = data.package_data.get_mut(app_id) {
state_manager.update_data(data).await; entry.ui_ready = Some(false);
} entry.state = transitional;
prev }
prev
})
.await
} }
/// Set the entry's state to `new_state`. No-ops if the entry has since been /// Set the entry's state to `new_state`. No-ops if the entry has since been
/// removed (e.g. uninstall ran concurrently). /// removed (e.g. uninstall ran concurrently).
async fn set_state(state_manager: &StateManager, app_id: &str, new_state: PackageState) { async fn set_state(state_manager: &StateManager, app_id: &str, new_state: PackageState) {
let (mut data, _) = state_manager.get_snapshot().await; state_manager
if let Some(entry) = data.package_data.get_mut(app_id) { .mutate_data(|data| {
if entry.state != new_state { if let Some(entry) = data.package_data.get_mut(app_id) {
entry.state = new_state; if entry.state != new_state {
state_manager.update_data(data).await; if new_state != PackageState::Running {
} entry.ui_ready = Some(false);
} }
entry.state = new_state;
}
}
})
.await
} }
+3
View File
@@ -114,6 +114,9 @@ impl PortMap {
/// there. /// there.
fn apps_dirs() -> Vec<PathBuf> { fn apps_dirs() -> Vec<PathBuf> {
let mut dirs = Vec::new(); let mut dirs = Vec::new();
if let Some(root) = std::env::var_os("ARCHIPELAGO_APPS_DIR") {
dirs.push(root.into());
}
if let Ok(manifest_dir) = std::env::var("CARGO_MANIFEST_DIR") { if let Ok(manifest_dir) = std::env::var("CARGO_MANIFEST_DIR") {
dirs.push(PathBuf::from(manifest_dir).join("../../apps")); dirs.push(PathBuf::from(manifest_dir).join("../../apps"));
} }
+48 -3
View File
@@ -144,6 +144,34 @@ pub fn shared_status() -> Arc<RwLock<GateStatus>> {
.clone() .clone()
} }
static REFRESH_KICK: std::sync::LazyLock<tokio::sync::Notify> =
std::sync::LazyLock::new(tokio::sync::Notify::new);
static REFRESH_REV: std::sync::LazyLock<tokio::sync::watch::Sender<u64>> =
std::sync::LazyLock::new(|| tokio::sync::watch::channel(0).0);
/// Installation must not wait for the minute sweep before becoming reachable.
/// Wait for a completed sweep, bounded if shutdown/startup prevents one.
pub async fn refresh_now() {
let mut completed = REFRESH_REV.subscribe();
REFRESH_KICK.notify_one();
let _ = tokio::time::timeout(std::time::Duration::from_secs(3), completed.changed()).await;
}
pub fn port_claimed(status: &GateStatus, port: u16) -> bool {
let mut external = false;
let mut tor = false;
for (claimed_port, address) in &status.claimed {
if *claimed_port != port {
continue;
}
if let Ok(ip) = address.parse::<IpAddr>() {
tor |= ip == GATE_TOR_UPSTREAM;
external |= !ip.is_loopback();
}
}
external && tor
}
/// Run the gate. Returns only on shutdown. /// Run the gate. Returns only on shutdown.
pub async fn run( pub async fn run(
gate: Arc<AppGate>, gate: Arc<AppGate>,
@@ -162,11 +190,12 @@ pub async fn run(
loop { loop {
tokio::select! { tokio::select! {
_ = interval.tick() => { _ = interval.tick() => {}
sweep(&gate, &status, &mut held, &shutdown_rx).await; _ = REFRESH_KICK.notified() => {}
}
_ = shutdown_rx.changed() => return, _ = shutdown_rx.changed() => return,
} }
sweep(&gate, &status, &mut held, &shutdown_rx).await;
REFRESH_REV.send_modify(|revision| *revision = revision.wrapping_add(1));
} }
} }
@@ -461,3 +490,19 @@ mod tests {
assert!(!status.is_fully_enforced()); assert!(!status.is_fully_enforced());
} }
} }
#[cfg(test)]
mod readiness_tests {
use super::*;
#[test]
fn readiness_requires_external_and_tor_claims_for_the_same_port() {
let mut status = GateStatus::default();
assert!(!port_claimed(&status, 3001));
status.claimed.push((3001, "127.0.0.2".into()));
assert!(!port_claimed(&status, 3001));
status.claimed.push((3002, "192.0.2.10".into()));
assert!(!port_claimed(&status, 3001));
status.claimed.push((3001, "192.0.2.10".into()));
assert!(port_claimed(&status, 3001));
}
}
+1
View File
@@ -322,6 +322,7 @@ async fn eval_rpc_handler() -> (Arc<RpcHandler>, tempfile::TempDir) {
fn installed_entry(app_id: &str) -> crate::data_model::PackageDataEntry { fn installed_entry(app_id: &str) -> crate::data_model::PackageDataEntry {
use crate::data_model::{Description, Manifest, PackageDataEntry, PackageState, StaticFiles}; use crate::data_model::{Description, Manifest, PackageDataEntry, PackageState, StaticFiles};
PackageDataEntry { PackageDataEntry {
ui_ready: None,
state: PackageState::Running, state: PackageState::Running,
health: None, health: None,
exit_code: None, exit_code: None,
+1
View File
@@ -1069,6 +1069,7 @@ mod tests {
Description, Manifest, PackageDataEntry, PackageState, StaticFiles, Description, Manifest, PackageDataEntry, PackageState, StaticFiles,
}; };
PackageDataEntry { PackageDataEntry {
ui_ready: None,
state: PackageState::Running, state: PackageState::Running,
health: None, health: None,
exit_code: None, exit_code: None,
+23 -1
View File
@@ -100,7 +100,11 @@ fn friendly_transient_error(has_cached_state: bool, err_msg: &str) -> String {
.trim() .trim()
.trim_end_matches('.'); .trim_end_matches('.');
let lower = detail.to_lowercase(); let lower = detail.to_lowercase();
let state = if lower.contains("verifying blocks") { let state = if lower.contains("loading block index") {
Some("loading its block index. This can take a while after installation or restart")
} else if lower.contains("replaying blocks") {
Some("checking saved blocks before startup completes")
} else if lower.contains("verifying blocks") {
Some("verifying blocks after restart") Some("verifying blocks after restart")
} else if lower.contains("connection reset") { } else if lower.contains("connection reset") {
Some("starting up and not yet accepting RPC connections") Some("starting up and not yet accepting RPC connections")
@@ -340,3 +344,21 @@ mod tests {
assert!(msg.len() < 260); assert!(msg.len() < 260);
} }
} }
#[cfg(test)]
mod startup_message_tests {
#[test]
fn loading_block_index_is_explained_without_rpc_error_dump() {
for cached in [false, true] {
let message = super::friendly_transient_error(
cached,
r#"getblockchaininfo: Bitcoin RPC returned 500 Internal Server Error: {"error":{"code":-28,"message":"Loading block index…"}}"#,
);
assert!(message.contains("loading its block index"));
for raw in ["500", "-28", "Detail:", "getblockchaininfo", "{", "RPC"] {
assert!(!message.contains(raw));
}
assert_eq!(message.contains("last known state"), cached);
}
}
}
+132 -18
View File
@@ -138,6 +138,119 @@ const NGINX_FEDIMINT_NEW: &str = " sub_filter_types text/css application/
const NGINX_FEDIMINT_SNIPPET_ANCHOR: &str = "proxy_pass http://127.0.0.1:8175/;"; const NGINX_FEDIMINT_SNIPPET_ANCHOR: &str = "proxy_pass http://127.0.0.1:8175/;";
const NGINX_FEDIMINT_SNIPPET_INSERT: &str = "proxy_pass http://127.0.0.1:8175/;\n proxy_set_header Accept-Encoding \"\";\n sub_filter_types text/css application/javascript application/json;\n sub_filter_once off;\n sub_filter 'href=\"/' 'href=\"/app/fedimint/';\n sub_filter 'src=\"/' 'src=\"/app/fedimint/';\n sub_filter \"href='/\" \"href='/app/fedimint/\";\n sub_filter \"src='/\" \"src='/app/fedimint/\";\n sub_filter 'url(\"/' 'url(\"/app/fedimint/';\n sub_filter \"url('/\" \"url('/app/fedimint/\";\n sub_filter '</head>' '<script src=\"/nostr-provider.js\"></script></head>';"; const NGINX_FEDIMINT_SNIPPET_INSERT: &str = "proxy_pass http://127.0.0.1:8175/;\n proxy_set_header Accept-Encoding \"\";\n sub_filter_types text/css application/javascript application/json;\n sub_filter_once off;\n sub_filter 'href=\"/' 'href=\"/app/fedimint/';\n sub_filter 'src=\"/' 'src=\"/app/fedimint/';\n sub_filter \"href='/\" \"href='/app/fedimint/\";\n sub_filter \"src='/\" \"src='/app/fedimint/\";\n sub_filter 'url(\"/' 'url(\"/app/fedimint/';\n sub_filter \"url('/\" \"url('/app/fedimint/\";\n sub_filter '</head>' '<script src=\"/nostr-provider.js\"></script></head>';";
/// Finish manifest promotion before constructing the orchestrator or starting
/// catalog refresh/reconciliation. Replacing the app tree in the background
/// could let a reload observe its temporary empty state and forget disk-only apps.
pub async fn ensure_runtime_assets_ready() {
// Install the guard before any startup path can reload an older dashboard
// vhost. The canonical OTA/ISO config contains the same guard inline.
if Path::new(NGINX_CONF_PATH).exists() || Path::new(NGINX_ENABLED_CONF_PATH).exists() {
match host_sudo(&[
"python3",
"-c",
include_str!("../../../scripts/dashboard-public-guard.py"),
])
.await
{
Ok(status) if status.success() => debug!("Dashboard public source guard verified"),
Ok(status) => warn!("Dashboard public source guard needs attention: {status}"),
Err(error) => warn!("Dashboard public source guard could not run: {error}"),
}
}
match run_runtime_assets().await {
Ok(changed) if changed => info!("Runtime assets synchronized from OTA payload"),
Ok(_) => debug!("No OTA runtime payload to synchronize"),
Err(e) => warn!("Runtime asset bootstrap failed (non-fatal): {:#}", e),
}
// A binary-only qualification or OTA rollback can precede the matching
// script payload. Install the exact embedded helper before Quadlet
// reconciliation can introduce its required ExecStartPre command.
if let Err(error) = write_root_if_needed(
"/opt/archipelago/scripts/filebrowser-credentials.py",
include_str!("../../../scripts/filebrowser-credentials.py"),
)
.await
{
warn!("File Browser credential helper installation failed: {error:#}");
}
if let Err(error) = run_npm_bridge_bootstrap().await {
warn!("NPM public routing bootstrap needs attention: {error:#}");
}
// Repair the narrowly recognized legacy NPM tunnel override before app
// reconciliation. The embedded script ships in both OTA and ISO binaries.
// It preserves native wallet services and refuses unknown custom routing.
match tokio::process::Command::new("python3")
.arg("-c")
.arg(include_str!("../../../scripts/repair-npm-tunnel.py"))
.output()
.await
{
Ok(output) if output.status.success() => {
if !output.stdout.is_empty() {
info!("{}", String::from_utf8_lossy(&output.stdout).trim());
}
}
Ok(output) => warn!(
"NPM tunnel migration needs attention: {}",
String::from_utf8_lossy(&output.stderr).trim()
),
Err(error) => warn!("NPM tunnel migration could not run: {error}"),
}
match run_apps_dir_repair().await {
Ok(true) => {
info!("Populated /opt/archipelago/apps from installer copy at /etc/archipelago/apps")
}
Ok(false) => debug!("/opt/archipelago/apps already populated (or no installer copy)"),
Err(e) => warn!("Apps dir repair failed (non-fatal): {:#}", e),
}
}
async fn run_npm_bridge_bootstrap() -> Result<()> {
if !Path::new("/opt/archipelago/scripts").is_dir() {
return Ok(());
}
let mut units_changed = false;
for (path, content) in [
(
"/opt/archipelago/scripts/npm-public-bridge.py",
include_str!("../../../scripts/npm-public-bridge.py"),
),
(
"/opt/archipelago/scripts/dashboard-public-guard.py",
include_str!("../../../scripts/dashboard-public-guard.py"),
),
(
"/etc/systemd/system/archipelago-npm-bridge.service",
include_str!("../../../image-recipe/configs/archipelago-npm-bridge.service"),
),
(
"/etc/systemd/system/archipelago-npm-bridge.timer",
include_str!("../../../image-recipe/configs/archipelago-npm-bridge.timer"),
),
] {
let changed = write_root_if_needed(path, content).await?;
units_changed |= changed && (path.ends_with(".service") || path.ends_with(".timer"));
}
if units_changed {
anyhow::ensure!(
host_sudo(&["systemctl", "daemon-reload"]).await?.success(),
"NPM bridge daemon reload failed"
);
}
anyhow::ensure!(
host_sudo(&[
"systemctl",
"enable",
"--now",
"archipelago-npm-bridge.timer"
])
.await?
.success(),
"NPM bridge timer could not start"
);
Ok(())
}
/// Entry point called from main startup. Never returns an error to the caller — /// Entry point called from main startup. Never returns an error to the caller —
/// failing to bootstrap host artifacts must not prevent the backend from serving. /// failing to bootstrap host artifacts must not prevent the backend from serving.
pub async fn ensure_doctor_installed() { pub async fn ensure_doctor_installed() {
@@ -146,11 +259,6 @@ pub async fn ensure_doctor_installed() {
Ok(false) => debug!("No stale Archipelago dev-mode service override found"), Ok(false) => debug!("No stale Archipelago dev-mode service override found"),
Err(e) => warn!("Service override repair failed (non-fatal): {:#}", e), Err(e) => warn!("Service override repair failed (non-fatal): {:#}", e),
} }
match run_runtime_assets().await {
Ok(changed) if changed => info!("Runtime assets synchronized from OTA payload"),
Ok(_) => debug!("No OTA runtime payload to synchronize"),
Err(e) => warn!("Runtime asset bootstrap failed (non-fatal): {:#}", e),
}
match run().await { match run().await {
Ok(changed) if changed => info!("Doctor artifacts synchronized with binary"), Ok(changed) if changed => info!("Doctor artifacts synchronized with binary"),
Ok(_) => debug!("Doctor artifacts already in sync"), Ok(_) => debug!("Doctor artifacts already in sync"),
@@ -168,13 +276,6 @@ pub async fn ensure_doctor_installed() {
Ok(false) => debug!("No stale bitcoin.conf found"), Ok(false) => debug!("No stale bitcoin.conf found"),
Err(e) => warn!("Bitcoin RPC repair failed (non-fatal): {:#}", e), Err(e) => warn!("Bitcoin RPC repair failed (non-fatal): {:#}", e),
} }
match run_apps_dir_repair().await {
Ok(true) => {
info!("Populated /opt/archipelago/apps from installer copy at /etc/archipelago/apps")
}
Ok(false) => debug!("/opt/archipelago/apps already populated (or no installer copy)"),
Err(e) => warn!("Apps dir repair failed (non-fatal): {:#}", e),
}
match run_tor_helper_sync().await { match run_tor_helper_sync().await {
Ok(true) => info!("tor-helper.sh synchronized with binary"), Ok(true) => info!("tor-helper.sh synchronized with binary"),
Ok(false) => debug!("tor-helper.sh already current"), Ok(false) => debug!("tor-helper.sh already current"),
@@ -395,17 +496,28 @@ async fn run_runtime_assets() -> Result<bool> {
if nginx_src.exists() { if nginx_src.exists() {
let src_s = nginx_src.to_string_lossy().to_string(); let src_s = nginx_src.to_string_lossy().to_string();
let status = host_sudo(&[ let status = host_sudo(&[
"install", "python3",
"-m", "-c",
"644", include_str!("../../../scripts/dashboard-public-guard.py"),
"--install",
&src_s, &src_s,
"/etc/nginx/sites-available/archipelago",
]) ])
.await .await
.context("install nginx-archipelago.conf")?; .context("install guarded nginx-archipelago.conf")?;
if !status.success() { if !status.success() {
anyhow::bail!("install nginx-archipelago.conf exited with {}", status); anyhow::bail!("install nginx-archipelago.conf exited with {}", status);
} }
let acme_status = host_sudo(&[
"python3",
"-c",
include_str!("../../../scripts/npm-public-bridge.py"),
"--acme-only",
])
.await?;
anyhow::ensure!(
acme_status.success(),
"active NPM ACME root migration failed"
);
changed = true; changed = true;
} }
@@ -422,6 +534,8 @@ async fn run_runtime_assets() -> Result<bool> {
"archipelago-doctor.service", "archipelago-doctor.service",
"archipelago-doctor.timer", "archipelago-doctor.timer",
"archipelago-host-secrets-audit.service", "archipelago-host-secrets-audit.service",
"archipelago-npm-bridge.service",
"archipelago-npm-bridge.timer",
] { ] {
let src = configs.join(unit); let src = configs.join(unit);
if src.exists() { if src.exists() {
@@ -449,7 +563,7 @@ async fn run_runtime_assets() -> Result<bool> {
// or directory"). Skipped when byte-identical; a running daemon is // or directory"). Skipped when byte-identical; a running daemon is
// unaffected (install replaces the inode) and picks the new binary up // unaffected (install replaces the inode) and picks the new binary up
// on its next spawn. // on its next spawn.
for tool in ["archy-reticulum-daemon", "archy-rnodeconf"] { for tool in ["archy-reticulum-daemon", "archy-rnodeconf", "archy-esptool"] {
let src = runtime_dir.join("radio-tools").join(tool); let src = runtime_dir.join("radio-tools").join(tool);
if !src.exists() { if !src.exists() {
continue; continue;
+193 -1
View File
@@ -102,6 +102,33 @@ pub struct AppCatalogEntry {
/// `docs/registry-manifest-design.md`. /// `docs/registry-manifest-design.md`.
#[serde(default, skip_serializing_if = "Option::is_none")] #[serde(default, skip_serializing_if = "Option::is_none")]
pub manifest: Option<serde_json::Value>, pub manifest: Option<serde_json::Value>,
/// Backward-compatible catalog rollout: old daemons ignore these and keep
/// the base manifest. New daemons choose only variants they can safely apply.
#[serde(default, skip_serializing_if = "Vec::is_empty")]
pub manifest_variants: Vec<CatalogManifestVariant>,
}
#[derive(Debug, Clone, Serialize, Deserialize)]
pub struct CatalogManifestVariant {
pub requires: Vec<String>,
pub manifest: serde_json::Value,
}
fn selected_manifest(entry: AppCatalogEntry) -> Option<serde_json::Value> {
// Never let an unknown future requirement become an unsafe partial match.
for variant in entry.manifest_variants.into_iter().rev() {
if !variant.requires.is_empty()
&& variant.requires.iter().all(|capability| {
matches!(
capability.as_str(),
"runtime-migration-backup-v1" | "npm-legacy-host-gateway-v1"
)
})
{
return Some(variant.manifest);
}
}
entry.manifest
} }
/// One selectable version in an app's `versions[]` list. The catalog carries a /// One selectable version in an app's `versions[]` list. The catalog carries a
@@ -234,7 +261,7 @@ pub fn catalog_manifest_values() -> Vec<(String, serde_json::Value)> {
load_catalog() load_catalog()
.apps .apps
.into_iter() .into_iter()
.filter_map(|(id, e)| e.manifest.map(|m| (id, m))) .filter_map(|(id, e)| selected_manifest(e).map(|m| (id, m)))
.collect() .collect()
} }
@@ -443,6 +470,12 @@ pub struct CatalogRefresh {
/// changed. Best-effort: a fetch failure leaves the existing cache untouched /// changed. Best-effort: a fetch failure leaves the existing cache untouched
/// (origin-always-wins; updates simply aren't refreshed this cycle). /// (origin-always-wins; updates simply aren't refreshed this cycle).
pub async fn refresh_catalog(data_dir: &Path) -> anyhow::Result<CatalogRefresh> { pub async fn refresh_catalog(data_dir: &Path) -> anyhow::Result<CatalogRefresh> {
// Explicit operator-only qualification of a signed candidate on selected
// nodes. Never change fleet mirrors or fall back to an older public catalog
// while a candidate is selected. Normal signature enforcement still applies.
if let Some(path) = std::env::var_os("ARCHY_APP_CATALOG_CANDIDATE") {
return refresh_candidate_catalog(data_dir, Path::new(&path)).await;
}
let mirrors = crate::update::load_mirrors(data_dir) let mirrors = crate::update::load_mirrors(data_dir)
.await .await
.unwrap_or_default(); .unwrap_or_default();
@@ -489,6 +522,49 @@ pub async fn refresh_catalog(data_dir: &Path) -> anyhow::Result<CatalogRefresh>
Err(last_err.unwrap_or_else(|| anyhow::anyhow!("no catalog mirrors reachable"))) Err(last_err.unwrap_or_else(|| anyhow::anyhow!("no catalog mirrors reachable")))
} }
async fn refresh_candidate_catalog(data_dir: &Path, path: &Path) -> anyhow::Result<CatalogRefresh> {
anyhow::ensure!(
path.is_absolute(),
"candidate catalog path must be absolute"
);
let metadata = tokio::fs::metadata(path)
.await
.context("inspect candidate catalog")?;
anyhow::ensure!(
metadata.is_file() && metadata.len() <= 4 * 1024 * 1024,
"candidate catalog must be a file no larger than 4 MiB"
);
let body = tokio::fs::read_to_string(path)
.await
.context("read candidate catalog")?;
anyhow::ensure!(
body.len() <= 4 * 1024 * 1024,
"candidate catalog exceeds 4 MiB"
);
let raw: serde_json::Value = serde_json::from_str(&body)?;
anyhow::ensure!(
matches!(
crate::trust::verify_detached(&raw)?,
crate::trust::SignatureStatus::Verified { anchored: true, .. }
),
"candidate catalog requires a signature anchored to the release root"
);
let catalog: AppCatalog = serde_json::from_value(raw)?;
let changed = write_cache(data_dir, &body)?;
if changed {
*CACHE.lock().unwrap() = None;
}
info!(
apps = catalog.apps.len(),
changed,
"app-catalog: using explicitly selected signed candidate; public catalog refresh paused"
);
Ok(CatalogRefresh {
apps: catalog.apps.len(),
changed,
})
}
async fn fetch_one(client: &reqwest::Client, url: &str) -> anyhow::Result<(AppCatalog, String)> { async fn fetch_one(client: &reqwest::Client, url: &str) -> anyhow::Result<(AppCatalog, String)> {
let resp = client.get(url).send().await?; let resp = client.get(url).send().await?;
if !resp.status().is_success() { if !resp.status().is_success() {
@@ -557,6 +633,122 @@ fn write_cache(data_dir: &Path, body: &str) -> anyhow::Result<bool> {
mod tests { mod tests {
use super::*; use super::*;
fn signed_candidate(key_byte: u8) -> serde_json::Value {
// Same test anchor as trust::signed_doc tests; never a production key.
let anchor = ed25519_dalek::SigningKey::from_bytes(&[7u8; 32]);
std::env::set_var(
"ARCHY_RELEASE_ROOT_PUBKEY",
hex::encode(anchor.verifying_key().to_bytes()),
);
let key = ed25519_dalek::SigningKey::from_bytes(&[key_byte; 32]);
let mut value = serde_json::json!({"schema":1,"apps":{"demo":{"version":"2"}},"future_field":{"retain":true}});
let (sig, did) = crate::trust::signed_doc::sign_detached(&key, &value).unwrap();
value["signature"] = sig.into();
value["signed_by"] = did.into();
value
}
#[tokio::test]
async fn candidate_catalog_preserves_signed_bytes_and_is_idempotent() {
let dir = tempfile::tempdir().unwrap();
let path = dir.path().join("candidate.json");
let body = serde_json::to_string_pretty(&signed_candidate(7)).unwrap();
std::fs::write(&path, &body).unwrap();
let first = refresh_candidate_catalog(dir.path(), &path).await.unwrap();
assert!(first.changed);
assert_eq!(first.apps, 1);
assert_eq!(
std::fs::read_to_string(dir.path().join(APP_CATALOG_FILE)).unwrap(),
body
);
assert!(
!refresh_candidate_catalog(dir.path(), &path)
.await
.unwrap()
.changed
);
}
#[tokio::test]
async fn rejected_candidate_never_replaces_previous_catalog() {
let dir = tempfile::tempdir().unwrap();
let path = dir.path().join("candidate.json");
let previous = "previous cached bytes";
write_cache(dir.path(), previous).unwrap();
let mut tampered = signed_candidate(7);
tampered["apps"]["demo"]["version"] = "tampered".into();
for body in [
"malformed".into(),
r#"{"schema":1,"apps":{}}"#.into(),
signed_candidate(11).to_string(),
tampered.to_string(),
" ".repeat(4 * 1024 * 1024 + 1),
] {
std::fs::write(&path, body).unwrap();
assert!(refresh_candidate_catalog(dir.path(), &path).await.is_err());
assert_eq!(
std::fs::read_to_string(dir.path().join(APP_CATALOG_FILE)).unwrap(),
previous
);
}
std::fs::remove_file(&path).unwrap();
assert!(refresh_candidate_catalog(dir.path(), &path).await.is_err());
assert!(
refresh_candidate_catalog(dir.path(), Path::new("relative.json"))
.await
.is_err()
);
assert_eq!(
std::fs::read_to_string(dir.path().join(APP_CATALOG_FILE)).unwrap(),
previous
);
}
#[test]
fn catalog_migration_variant_is_compatible_with_old_and_future_daemons() {
let raw = serde_json::json!({
"version": "2.45.0", "manifest": {"app": {"id": "portainer", "container": {}}},
"manifest_variants": [{"requires": ["runtime-migration-backup-v1"],
"manifest": {"app": {"id": "portainer", "container": {"network": "slirp4netns"}, "backup_before_runtime_change": true}}}]
});
#[derive(Deserialize)]
struct OldEntry {
manifest: serde_json::Value,
}
let old: OldEntry = serde_json::from_value(raw.clone()).unwrap();
assert!(old.manifest["app"]["container"].get("network").is_none());
let current: AppCatalogEntry = serde_json::from_value(raw.clone()).unwrap();
let chosen = selected_manifest(current).unwrap();
assert_eq!(chosen["app"]["container"]["network"], "slirp4netns");
assert_eq!(chosen["app"]["backup_before_runtime_change"], true);
let mut future = raw;
future["manifest_variants"][0]["requires"]
.as_array_mut()
.unwrap()
.push(serde_json::json!("unknown-next-capability"));
let chosen = selected_manifest(serde_json::from_value(future).unwrap()).unwrap();
assert!(chosen["app"]["container"].get("network").is_none());
}
#[test]
fn npm_gateway_variant_requires_explicit_runtime_support() {
let mut raw = serde_json::json!({
"version": "2.12.1", "manifest": {"network":"pasta"},
"manifest_variants": [{"requires":["runtime-migration-backup-v1", "npm-legacy-host-gateway-v1"],
"manifest":{"network":"slirp4netns:allow_host_loopback=true,cidr=169.254.1.0/24"}}]
});
assert_eq!(
selected_manifest(serde_json::from_value(raw.clone()).unwrap()).unwrap()["network"],
"slirp4netns:allow_host_loopback=true,cidr=169.254.1.0/24"
);
// A runtime missing any required capability must retain the base.
raw["manifest_variants"][0]["requires"][1] = serde_json::json!("unknown-gateway-v2");
assert_eq!(
selected_manifest(serde_json::from_value(raw).unwrap()).unwrap()["network"],
"pasta"
);
}
#[test] #[test]
fn parses_and_ignores_unknown_fields() { fn parses_and_ignores_unknown_fields() {
let json = r#"{ let json = r#"{
+40 -6
View File
@@ -103,6 +103,15 @@ pub fn companions_for(package_id: &str) -> &'static [CompanionSpec] {
} }
} }
/// Missing companion UIs are provisioned here, never by snapshot recovery.
/// A stale running-container snapshot must not resurrect an orphaned UI.
pub fn is_companion_app(app_id: &str) -> bool {
ALL_COMPANIONS
.iter()
.flat_map(|specs| specs.iter())
.any(|spec| spec.image_base == app_id)
}
/// Every companion this build knows how to provision. Kept beside /// Every companion this build knows how to provision. Kept beside
/// `companions_for` — a new companion must be added to both, or the reaper /// `companions_for` — a new companion must be added to both, or the reaper
/// will not recognise it as one of ours and will leave it running forever. /// will not recognise it as one of ours and will leave it running forever.
@@ -313,7 +322,7 @@ async fn image_id(image_ref: &str) -> Option<String> {
/// should reference (`localhost/<base>:latest` for build, registry /// should reference (`localhost/<base>:latest` for build, registry
/// URL for pull). /// URL for pull).
async fn ensure_image_present(spec: &CompanionSpec) -> Result<String> { async fn ensure_image_present(spec: &CompanionSpec) -> Result<String> {
let local_image = format!("localhost/{}:latest", spec.image_base); let mut local_image = format!("localhost/{}:latest", spec.image_base);
let local_image_compat = format!("localhost/{}:local", spec.image_base); let local_image_compat = format!("localhost/{}:local", spec.image_base);
let registry_image = format!("{}/{}:latest", COMPANION_REGISTRY, spec.image_base); let registry_image = format!("{}/{}:latest", COMPANION_REGISTRY, spec.image_base);
@@ -322,11 +331,13 @@ async fn ensure_image_present(spec: &CompanionSpec) -> Result<String> {
for dir in spec.build_dir_candidates { for dir in spec.build_dir_candidates {
let dockerfile = PathBuf::from(dir).join("Dockerfile"); let dockerfile = PathBuf::from(dir).join("Dockerfile");
if fs::try_exists(&dockerfile).await.unwrap_or(false) { if fs::try_exists(&dockerfile).await.unwrap_or(false) {
// `:local` is a deliberate manual override — never auto-rebuild it. // Older installers and self-update create :local themselves. It
// must receive source updates too; treating it as a permanent
// manual override silently kept the old LND UI after an OTA.
if image_exists(&local_image_compat).await { if image_exists(&local_image_compat).await {
return Ok(local_image_compat); local_image = local_image_compat.clone();
} }
// Reuse the auto-built `:latest` only when the build context has NOT // Reuse either local tag only when the build context has NOT
// changed since it was built. Without this staleness check an // changed since it was built. Without this staleness check an
// already-present image is reused forever, so edits to the baked-in // already-present image is reused forever, so edits to the baked-in
// context (Dockerfile, nginx.conf, …) never reach the node — this is // context (Dockerfile, nginx.conf, …) never reach the node — this is
@@ -849,20 +860,43 @@ async fn needs_repair(spec: &CompanionSpec) -> Result<bool> {
if !matches_known_shape { if !matches_known_shape {
return Ok(true); return Ok(true);
} }
if on_disk.contains(&local_image) && !on_disk.contains(&local_image_compat) { if let Some(image) = managed_local_image(spec, &on_disk) {
for dir in spec.build_dir_candidates { for dir in spec.build_dir_candidates {
let dockerfile = PathBuf::from(dir).join("Dockerfile"); let dockerfile = PathBuf::from(dir).join("Dockerfile");
if fs::try_exists(&dockerfile).await.unwrap_or(false) { if fs::try_exists(&dockerfile).await.unwrap_or(false) {
// Conservative on any timeout/error inside: reuse the cache. // Conservative on any timeout/error inside: reuse the cache.
return Ok(context_is_newer_than_image(dir, &local_image).await); return Ok(context_is_newer_than_image(dir, &image).await);
} }
} }
} }
Ok(false) Ok(false)
} }
fn managed_local_image(spec: &CompanionSpec, unit: &str) -> Option<String> {
["latest", "local"]
.iter()
.map(|tag| format!("localhost/{}:{tag}", spec.image_base))
.find(|image| build_unit(spec, image).render() == unit)
}
#[cfg(test)] #[cfg(test)]
mod tests { mod tests {
#[test]
fn legacy_installer_local_tag_is_checked_for_source_updates_like_latest() {
for spec in ALL_COMPANIONS.iter().flat_map(|group| group.iter()) {
for tag in ["local", "latest"] {
let image = format!("localhost/{}:{tag}", spec.image_base);
let unit = build_unit(spec, &image).render();
assert_eq!(managed_local_image(spec, &unit), Some(image));
}
let registry = format!("{}/{}:latest", COMPANION_REGISTRY, spec.image_base);
assert_eq!(
managed_local_image(spec, &build_unit(spec, &registry).render()),
None
);
}
}
use super::*; use super::*;
fn names(specs: &[&'static CompanionSpec]) -> Vec<&'static str> { fn names(specs: &[&'static CompanionSpec]) -> Vec<&'static str> {
+479 -34
View File
@@ -3,8 +3,9 @@
use anyhow::Result; use anyhow::Result;
use archipelago_container::{ use archipelago_container::{
ContainerRuntime as ContainerRuntimeTrait, ContainerState, PodmanClient, ContainerRuntime as ContainerRuntimeTrait, ContainerState, ContainerStatus, PodmanClient,
}; };
use futures_util::StreamExt;
use std::collections::HashMap; use std::collections::HashMap;
use std::sync::Arc; use std::sync::Arc;
use tracing::{debug, info}; use tracing::{debug, info};
@@ -15,6 +16,19 @@ use crate::data_model::{
PackageDataEntry, PackageState, ServiceStatus, StaticFiles, PackageDataEntry, PackageState, ServiceStatus, StaticFiles,
}; };
/// One displayed package for each known container/manifest alias. Keep the
/// stopped-app restoration path in agreement with live-container discovery.
fn canonical_package_id(name: &str) -> &str {
match name.strip_prefix("archy-").unwrap_or(name) {
"immich_server" | "immich-server" => "immich",
"immich-postgres" => "immich_postgres",
"immich-redis" => "immich_redis",
"mempool-web" | "mempool-frontend" => "mempool",
"btcpay" | "btcpayserver" => "btcpay-server",
name => name,
}
}
pub struct DockerPackageScanner { pub struct DockerPackageScanner {
runtime: Arc<dyn ContainerRuntimeTrait>, runtime: Arc<dyn ContainerRuntimeTrait>,
} }
@@ -25,8 +39,15 @@ impl DockerPackageScanner {
} }
/// Scan Docker containers and convert to package data /// Scan Docker containers and convert to package data
pub async fn scan_containers(&self) -> Result<HashMap<String, PackageDataEntry>> { pub async fn scan_containers(
let containers = self.runtime.list_containers().await?; &self,
data_dir: &std::path::Path,
cached: &HashMap<String, PackageDataEntry>,
) -> Result<HashMap<String, PackageDataEntry>> {
let mut containers = self.runtime.list_containers().await?;
let installed = crate::crash_recovery::load_installed_apps(data_dir).await;
let uninstalled = crate::crash_recovery::load_user_uninstalled(data_dir).await;
restore_absent_installed(&mut containers, &installed, &uninstalled);
debug!("Found {} containers", containers.len()); debug!("Found {} containers", containers.len());
@@ -91,24 +112,8 @@ impl DockerPackageScanner {
debug!("Found {} UI containers", ui_containers.len()); debug!("Found {} UI containers", ui_containers.len());
for container in containers { for container in containers {
// Extract app ID from container name // Use the same alias mapping as stopped-app restoration.
// Support both archy-* containers (docker-compose) and plain names (manual) let app_id = canonical_package_id(&container.name).to_owned();
let app_id = if container.name.starts_with("archy-") {
container
.name
.strip_prefix("archy-")
.unwrap_or(&container.name)
.to_string()
} else {
// Use the container name as-is for manually started containers
container.name.clone()
};
// Normalize multi-container app IDs to their canonical names
let app_id = match app_id.as_str() {
"immich_server" => "immich".to_string(),
_ => app_id,
};
// Skip backend services (databases, APIs, etc.) // Skip backend services (databases, APIs, etc.)
if excluded_services.contains(&app_id.as_str()) { if excluded_services.contains(&app_id.as_str()) {
@@ -139,6 +144,18 @@ impl DockerPackageScanner {
continue; continue;
} }
if container.id.is_empty() {
if let Some(previous) = cached.get(&app_id) {
let mut held = previous.clone();
held.state = PackageState::Stopped;
held.ui_ready = Some(false);
held.health = None;
held.exit_code = None;
packages.insert(app_id.clone(), held);
continue;
}
}
// Get metadata for this app // Get metadata for this app
let metadata = get_app_metadata(&app_id); let metadata = get_app_metadata(&app_id);
// Manifest-owned metadata (icon) wins over the static table: the // Manifest-owned metadata (icon) wins over the static table: the
@@ -158,13 +175,11 @@ impl DockerPackageScanner {
} else { } else {
// Prefer the known web UI port over arbitrary first binding // Prefer the known web UI port over arbitrary first binding
// (for example Gitea exposes SSH on 2222 before web on 3001). // (for example Gitea exposes SSH on 2222 before web on 3001).
let candidate = if uses_allocated_launch_port(&app_id) { let candidate = package_launch_candidate(
extract_lan_address(&container.ports) &app_id,
.or_else(|| PodmanClient::lan_address_for(&app_id)) &container.ports,
} else { PodmanClient::lan_address_for(&app_id),
PodmanClient::lan_address_for(&app_id) );
.or_else(|| extract_lan_address(&container.ports))
};
reachable_lan_address(&app_id, candidate).await reachable_lan_address(&app_id, candidate).await
}; };
@@ -179,14 +194,22 @@ impl DockerPackageScanner {
let tor_address = read_tor_address(&app_id).await; let tor_address = read_tor_address(&app_id).await;
// Extract actual version from container image tag // Extract actual version from container image tag
let running_version = image_versions::extract_version_from_image(&container.image); let running_version = if container.id.is_empty() {
String::new() // Absence cannot establish the installed image version.
} else {
image_versions::extract_version_from_image(&container.image)
};
// Decoupled from the binary OTA: prefer the remote app catalog, // Decoupled from the binary OTA: prefer the remote app catalog,
// falling back to the image-versions.sh pin when uncovered/offline. // falling back to the image-versions.sh pin when uncovered/offline.
let available_update = let available_update = if container.id.is_empty() {
crate::container::app_catalog::available_update_for_app(&app_id, &container.image); None
} else {
crate::container::app_catalog::available_update_for_app(&app_id, &container.image)
};
let package = PackageDataEntry { let package = PackageDataEntry {
ui_ready: Some(false),
state: package_state.clone(), state: package_state.clone(),
health: container.health.clone(), health: container.health.clone(),
exit_code: if package_state == PackageState::Exited { exit_code: if package_state == PackageState::Exited {
@@ -283,10 +306,259 @@ impl DockerPackageScanner {
); );
} }
let probes: Vec<_> = packages
.iter()
.filter_map(|(id, pkg)| {
if pkg.state != PackageState::Running {
return None;
}
let url = pkg
.installed
.as_ref()?
.interface_addresses
.get("main")?
.lan_address
.clone()?;
Some((id.clone(), url))
})
.collect();
let mut results = futures_util::stream::iter(
probes
.into_iter()
.map(|(id, url)| async move { (id, launch_http_ready(&url).await) }),
)
.buffer_unordered(8);
while let Some((id, ready)) = results.next().await {
if let Some(pkg) = packages.get_mut(&id) {
pkg.ui_ready = Some(ready);
}
}
// HTTP on loopback can precede the LAN/Tor listener after install.
let port_map = crate::appgate::identity::build_port_map();
let gated: Vec<_> = packages
.iter()
.filter_map(|(id, pkg)| {
if pkg.ui_ready != Some(true) {
return None;
}
let url = pkg
.installed
.as_ref()?
.interface_addresses
.get("main")?
.lan_address
.as_deref()?;
let port = launch_url_port(url)?;
port_map
.gated(port)
.filter(|gate| gate.declared)
.map(|_| (id.clone(), port))
})
.collect();
if !gated.is_empty() {
use crate::appgate::listener::{port_claimed, refresh_now, shared_status};
let status = shared_status();
let needs_refresh = {
let current = status.read().await;
gated.iter().any(|(_, port)| !port_claimed(&current, *port))
};
if needs_refresh {
refresh_now().await;
}
let current = status.read().await;
for (id, port) in gated {
if !port_claimed(&current, port) {
packages.get_mut(&id).unwrap().ui_ready = Some(false);
}
}
}
Ok(packages) Ok(packages)
} }
} }
/// Quadlet removes containers during ordinary stops/restarts. Rebuild installed
/// entries even on the daemon's first scan; a runtime absence is not uninstall.
fn restore_absent_installed(
containers: &mut Vec<ContainerStatus>,
installed: &std::collections::HashSet<String>,
uninstalled: &std::collections::HashSet<String>,
) {
let mut present: std::collections::HashSet<String> = containers
.iter()
.map(|c| canonical_package_id(&c.name).to_owned())
.collect();
let removed: std::collections::HashSet<_> = uninstalled
.iter()
.map(|id| canonical_package_id(id))
.collect();
for name in installed {
let id = canonical_package_id(name);
if removed.contains(id) || !present.insert(id.to_owned()) {
continue;
}
containers.push(ContainerStatus {
id: String::new(),
name: id.to_owned(),
state: ContainerState::Stopped,
health: None,
exit_code: None,
started_at: None,
image: String::new(),
created: String::new(),
ports: Vec::new(),
lan_address: None,
});
}
}
/// Probe the actual loopback upstream, not the app gate's login page. A bound
/// TCP socket alone can still reset requests or serve a startup 503.
async fn launch_http_ready(candidate: &str) -> bool {
let Ok(mut url) = reqwest::Url::parse(candidate) else {
return false;
};
if !matches!(url.scheme(), "http" | "https") {
return false;
}
if url.set_host(Some("127.0.0.1")).is_err() {
return false;
}
static CLIENT: std::sync::OnceLock<reqwest::Client> = std::sync::OnceLock::new();
let client = CLIENT.get_or_init(|| {
reqwest::Client::builder()
.no_proxy()
.timeout(std::time::Duration::from_secs(2))
.redirect(reqwest::redirect::Policy::none())
// Self-signed local app certificates are normal. This client only
// contacts loopback and never sends credentials or follows redirects.
.danger_accept_invalid_certs(true)
.build()
.expect("local readiness client")
});
match client.get(url).send().await {
Ok(response) => matches!(response.status().as_u16(), 200..=399 | 401 | 403),
Err(_) => false,
}
}
#[cfg(test)]
mod lifecycle_regression_tests {
use super::*;
use tokio::io::{AsyncReadExt, AsyncWriteExt};
#[test]
fn btcpay_aliases_share_one_package_without_promoting_dependencies() {
for name in ["btcpay", "btcpayserver", "btcpay-server", "archy-btcpay"] {
assert_eq!(canonical_package_id(name), "btcpay-server");
}
assert_eq!(canonical_package_id("archy-btcpay-db"), "btcpay-db");
assert_eq!(canonical_package_id("archy-nbxplorer"), "nbxplorer");
}
#[test]
fn immich_dependency_aliases_share_the_hidden_component_ids() {
for id in [
"immich-postgres",
"immich_postgres",
"archy-immich-postgres",
] {
assert_eq!(canonical_package_id(id), "immich_postgres");
}
assert_eq!(canonical_package_id("immich-redis"), "immich_redis");
assert_eq!(canonical_package_id("immich-server"), "immich");
}
#[test]
fn registry_survives_empty_runtime_and_deduplicates_aliases() {
let installed = ["archy-gitea", "gitea", "immich_server", "archy-removed"]
.into_iter()
.map(str::to_owned)
.collect();
let removed = ["removed".to_owned()].into_iter().collect();
let mut containers = Vec::new();
restore_absent_installed(&mut containers, &installed, &removed);
assert_eq!(containers.len(), 2);
assert!(containers
.iter()
.all(|c| c.state == ContainerState::Stopped));
containers[0].state = ContainerState::Running;
restore_absent_installed(&mut containers, &installed, &removed);
assert_eq!(containers.len(), 2);
assert_eq!(containers[0].state, ContainerState::Running);
}
#[test]
fn mempool_frontend_inventory_alias_does_not_create_a_second_package() {
let installed = ["mempool", "archy-mempool-web", "mempool-web"]
.into_iter()
.map(str::to_owned)
.collect();
let mut containers = Vec::new();
restore_absent_installed(&mut containers, &installed, &Default::default());
assert_eq!(containers.len(), 1);
assert_eq!(containers[0].name, "mempool");
containers[0].id = "live-frontend".into();
containers[0].state = ContainerState::Running;
restore_absent_installed(&mut containers, &installed, &Default::default());
assert_eq!(containers.len(), 1);
assert_eq!(containers[0].id, "live-frontend");
assert_eq!(containers[0].state, ContainerState::Running);
assert_eq!(canonical_package_id("archy-mempool-web"), "mempool");
assert_eq!(canonical_package_id("mempool-api"), "mempool-api");
containers.clear();
restore_absent_installed(
&mut containers,
&installed,
&["mempool".into()].into_iter().collect(),
);
assert!(containers.is_empty());
}
#[tokio::test]
async fn readiness_rejects_startup_errors_and_accepts_auth_and_redirects() {
for (status, expected) in [
(200, true),
(302, true),
(401, true),
(403, true),
(404, false),
(500, false),
(502, false),
(503, false),
] {
let listener = tokio::net::TcpListener::bind("127.0.0.1:0").await.unwrap();
let port = listener.local_addr().unwrap().port();
let task = tokio::spawn(async move {
let (mut stream, _) = listener.accept().await.unwrap();
let mut buf = [0; 2048];
let n = stream.read(&mut buf).await.unwrap();
assert!(String::from_utf8_lossy(&buf[..n]).starts_with("GET /start HTTP/1.1"));
stream.write_all(format!("HTTP/1.1 {status} Test\r\nContent-Length: 0\r\nConnection: close\r\n\r\n").as_bytes()).await.unwrap();
});
assert_eq!(
launch_http_ready(&format!("http://localhost:{port}/start")).await,
expected,
"status {status}"
);
task.await.unwrap();
}
}
#[tokio::test]
async fn readiness_rejects_tcp_accept_without_http() {
let listener = tokio::net::TcpListener::bind("127.0.0.1:0").await.unwrap();
let port = listener.local_addr().unwrap().port();
let task = tokio::spawn(async move {
let (stream, _) = listener.accept().await.unwrap();
drop(stream);
});
assert!(!launch_http_ready(&format!("http://localhost:{port}/")).await);
task.await.unwrap();
assert!(!launch_http_ready(&format!("http://localhost:{port}/")).await);
assert!(!launch_http_ready("file:///tmp/test").await);
}
}
struct AppMetadata { struct AppMetadata {
title: String, title: String,
description: String, description: String,
@@ -731,14 +1003,20 @@ fn extract_lan_address(ports: &[String]) -> Option<String> {
let mut first_candidate = None; let mut first_candidate = None;
for port_str in ports { for port_str in ports {
// Parse port strings like "0.0.0.0:18443->18443/tcp" or "0.0.0.0:18443-18444->18443-18444/tcp" // Parse port strings like "0.0.0.0:18443->18443/tcp" or "0.0.0.0:18443-18444->18443-18444/tcp"
let Some(public_part) = port_str.split("->").next() else { let Some((public_part, _)) = port_str.split_once("->") else {
continue; continue;
}; };
let Some(port_part) = public_part.split(':').nth(1) else { let Some((_, port_part)) = public_part.rsplit_once(':') else {
continue; continue;
}; };
// Extract just the first port if it's a range (e.g., "18443-18444" -> "18443") // Extract just the first port if it's a range (e.g., "18443-18444" -> "18443")
let host_port = port_part.split('-').next().unwrap_or(port_part); let host_port = port_part.split('-').next().unwrap_or(port_part);
let Ok(host_port) = host_port.parse::<u16>() else {
continue;
};
if host_port == 0 {
continue;
}
let candidate = format!("http://localhost:{}", host_port); let candidate = format!("http://localhost:{}", host_port);
if first_candidate.is_none() { if first_candidate.is_none() {
first_candidate = Some(candidate.clone()); first_candidate = Some(candidate.clone());
@@ -856,6 +1134,46 @@ fn companion_lan_address(app_id: &str) -> Option<String> {
} }
} }
/// Companion dashboards remain usable while their backend is syncing. Never
/// probe a Bitcoin RPC or Electrum protocol socket as dashboard readiness.
fn package_launch_candidate(
app_id: &str,
ports: &[String],
known: Option<String>,
) -> Option<String> {
if let Some(companion) = companion_lan_address(app_id) {
return Some(companion);
}
if app_id == "nginx-proxy-manager" {
// 80/443 serve users' proxy hosts; only container port 81 serves the
// admin UI. Podman's binding order is unstable across recreation.
// Resolve its actual host allocation rather than guessing the first
// HTTP port or hardcoding the default host port 8081.
let admin_ports: Vec<String> = ports
.iter()
.filter(|port| {
port.split_once("->")
.is_some_and(|(_, target)| target == "81/tcp")
})
.cloned()
.collect();
// With published bindings, a missing admin mapping is not evidence
// that some unrelated service on the default host port is this UI.
return extract_lan_address(&admin_ports).or_else(|| {
if ports.is_empty() {
known
} else {
None
}
});
}
if uses_allocated_launch_port(app_id) {
extract_lan_address(ports).or(known)
} else {
known.or_else(|| extract_lan_address(ports))
}
}
fn uses_allocated_launch_port(app_id: &str) -> bool { fn uses_allocated_launch_port(app_id: &str) -> bool {
matches!( matches!(
app_id, app_id,
@@ -940,7 +1258,134 @@ mod tor_service_name_tests {
#[cfg(test)] #[cfg(test)]
mod extract_lan_address_tests { mod extract_lan_address_tests {
use super::extract_lan_address; use super::{extract_lan_address, package_launch_candidate};
#[test]
fn companion_dashboard_wins_over_backend_protocol_ports() {
for id in ["bitcoin", "bitcoin-core", "bitcoin-knots"] {
assert_eq!(
package_launch_candidate(
id,
&["127.0.0.1:8332->8332/tcp".into()],
Some("http://localhost:8332".into())
)
.as_deref(),
Some("http://localhost:8334")
);
}
for id in ["electrumx", "electrs", "mempool-electrs"] {
assert_eq!(
package_launch_candidate(
id,
&["127.0.0.1:50001->50001/tcp".into()],
Some("http://localhost:50001".into())
)
.as_deref(),
Some("http://localhost:50002")
);
}
assert_eq!(
package_launch_candidate(
"filebrowser",
&["127.0.0.1:19080->80/tcp".into()],
Some("http://localhost:8080".into())
)
.as_deref(),
Some("http://localhost:19080")
);
}
#[test]
fn npm_admin_launch_is_independent_of_proxy_binding_order() {
let mappings = [
"10.77.0.2:18081->80/tcp",
"10.77.0.2:18443->443/tcp",
"127.0.0.1:8081->81/tcp",
];
for order in [
[0, 1, 2],
[0, 2, 1],
[1, 0, 2],
[1, 2, 0],
[2, 0, 1],
[2, 1, 0],
] {
let ports: Vec<String> = order.iter().map(|&i| mappings[i].into()).collect();
assert_eq!(
package_launch_candidate(
"nginx-proxy-manager",
&ports,
Some("http://localhost:8081/".into())
)
.as_deref(),
Some("http://localhost:8081")
);
}
}
#[test]
fn npm_admin_launch_respects_host_allocation_and_ipv6_bindings() {
for binding in ["127.0.0.1", "0.0.0.0", "[::1]", "[::]"] {
let ports = vec![
"10.77.0.2:18081->80/tcp".into(),
format!("{binding}:28081->81/tcp"),
];
assert_eq!(
package_launch_candidate(
"nginx-proxy-manager",
&ports,
Some("http://localhost:8081/".into())
)
.as_deref(),
Some("http://localhost:28081")
);
}
let proxies = vec![
"10.77.0.2:18081->80/tcp".into(),
"10.77.0.2:18443->443/tcp".into(),
];
assert_eq!(
package_launch_candidate("nginx-proxy-manager", &proxies, None),
None
);
assert_eq!(
package_launch_candidate(
"nginx-proxy-manager",
&proxies,
Some("http://localhost:8081/".into())
),
None
);
}
#[test]
fn npm_without_port_information_uses_declared_admin_url() {
assert_eq!(
package_launch_candidate(
"nginx-proxy-manager",
&[],
Some("http://localhost:8081/".into())
)
.as_deref(),
Some("http://localhost:8081/")
);
}
#[test]
fn malformed_published_ports_do_not_become_launch_urls() {
for port in [
"81/tcp",
"127.0.0.1:bad->81/tcp",
"[::1]:0->81/tcp",
"[::]:65536->81/tcp",
"127.0.0.1:8081->81/udp",
] {
assert_eq!(
package_launch_candidate("nginx-proxy-manager", &[port.into()], None),
None
);
}
}
#[test] #[test]
fn skips_ssh_port_when_web_port_is_published() { fn skips_ssh_port_when_web_port_is_published() {
+567 -3
View File
@@ -5,7 +5,7 @@
//! starting the container with `--config /data/.filebrowser.json`. //! starting the container with `--config /data/.filebrowser.json`.
use anyhow::{Context, Result}; use anyhow::{Context, Result};
use std::path::PathBuf; use std::path::{Path, PathBuf};
use tokio::fs; use tokio::fs;
use crate::update::host_sudo; use crate::update::host_sudo;
@@ -17,6 +17,84 @@ pub const DEFAULT_CONFIG_PATH: &str = "/var/lib/archipelago/filebrowser-data/.fi
const DEFAULT_CONFIG_JSON: &str = const DEFAULT_CONFIG_JSON: &str =
"{\"port\":80,\"baseURL\":\"\",\"address\":\"0.0.0.0\",\"database\":\"/data/filebrowser.db\",\"root\":\"/srv\",\"log\":\"stdout\"}\n"; "{\"port\":80,\"baseURL\":\"\",\"address\":\"0.0.0.0\",\"database\":\"/data/filebrowser.db\",\"root\":\"/srv\",\"log\":\"stdout\"}\n";
/// One atomically published record shared by setup, Cloud and credentials UI.
/// Deliberately has no Debug implementation: the password must never be logged.
#[derive(serde::Deserialize)]
pub struct CloudCredentials {
pub schema: u32,
pub username: String,
pub password: String,
}
pub async fn cloud_credentials(directory: &Path) -> Result<CloudCredentials> {
let path = directory.join("credentials.json");
match fs::read(&path).await {
Ok(bytes) => {
let value: CloudCredentials = serde_json::from_slice(&bytes)
.context("Invalid private File Browser credential record")?;
let suffix = value.username.strip_prefix("archy-").unwrap_or("");
anyhow::ensure!(
value.schema == 1
&& suffix.len() == 32
&& suffix.bytes().all(|c| c.is_ascii_hexdigit())
&& value.password.len() == 64
&& value.password.bytes().all(|c| c.is_ascii_hexdigit()),
"Invalid managed File Browser credentials"
);
Ok(value)
}
Err(error) if error.kind() == std::io::ErrorKind::NotFound => {
// Compatibility during staged upgrades only. Never invent admin/admin
// when a secret is missing; the pre-start provisioner repairs legacy DBs.
let password = fs::read_to_string(directory.join("password"))
.await
.context("File Browser secure Cloud login has not been provisioned")?;
let password = password.trim().to_owned();
anyhow::ensure!(
!password.is_empty() && password != "admin",
"File Browser default credentials must be migrated before Cloud login"
);
Ok(CloudCredentials {
schema: 0,
username: "admin".into(),
password,
})
}
Err(error) => Err(error).context("Cannot read private File Browser credentials"),
}
}
/// Prepare a stopped server using the same helper used by Quadlet and the ISO.
pub async fn prepare_credentials(
paths: &EnsurePaths,
secret_dir: &Path,
image: &str,
runtime: &str,
) -> Result<()> {
let output = tokio::process::Command::new("python3")
.args([
"-c",
include_str!("../../../../scripts/filebrowser-credentials.py"),
"--image",
image,
"--runtime",
runtime,
"--data-dir",
&paths.data_dir.to_string_lossy(),
"--srv-root",
&paths.srv_root.to_string_lossy(),
"--secrets-dir",
&secret_dir.to_string_lossy(),
])
.kill_on_drop(true)
.output()
.await
.context("Running File Browser credential setup")?;
anyhow::ensure!(output.status.success(),
"File Browser secure login setup failed; existing state and private rollback backup retained");
Ok(())
}
#[derive(Debug, Clone)] #[derive(Debug, Clone)]
pub struct EnsurePaths { pub struct EnsurePaths {
pub srv_root: PathBuf, pub srv_root: PathBuf,
@@ -82,7 +160,18 @@ async fn create_dir_all_or_sudo(path: &std::path::Path) -> Result<()> {
async fn write_config_atomically(paths: &EnsurePaths) -> Result<()> { async fn write_config_atomically(paths: &EnsurePaths) -> Result<()> {
let tmp = paths.config_path.with_extension("tmp"); let tmp = paths.config_path.with_extension("tmp");
match fs::write(&tmp, DEFAULT_CONFIG_JSON).await { let legacy = paths.data_dir.join("database.db").exists();
let canonical = paths.data_dir.join("filebrowser.db").exists();
anyhow::ensure!(
!(legacy && canonical),
"Multiple File Browser databases need explicit config selection"
);
let config = if legacy {
DEFAULT_CONFIG_JSON.replace("/data/filebrowser.db", "/data/database.db")
} else {
DEFAULT_CONFIG_JSON.to_string()
};
match fs::write(&tmp, &config).await {
Ok(()) => { Ok(()) => {
fs::rename(&tmp, &paths.config_path) fs::rename(&tmp, &paths.config_path)
.await .await
@@ -99,7 +188,7 @@ async fn write_config_atomically(paths: &EnsurePaths) -> Result<()> {
let script = format!( let script = format!(
"set -eu\ncat > '{}' <<'FILEBROWSERCONF'\n{}FILEBROWSERCONF\n", "set -eu\ncat > '{}' <<'FILEBROWSERCONF'\n{}FILEBROWSERCONF\n",
shell_quote(&paths.config_path.to_string_lossy()), shell_quote(&paths.config_path.to_string_lossy()),
DEFAULT_CONFIG_JSON config
); );
let status = host_sudo(&["sh", "-lc", &script]) let status = host_sudo(&["sh", "-lc", &script])
.await .await
@@ -117,10 +206,257 @@ fn shell_quote(s: &str) -> String {
s.replace('\'', "'\\''") s.replace('\'', "'\\''")
} }
/// Save a complete purchase without overwriting any existing directory entry.
/// Both host and rootless-namespace paths publish with a no-clobber hard link.
pub async fn save_new_file(dir: &Path, name: &str, bytes: &[u8]) -> Result<PathBuf> {
save_new_file_with(dir, name, bytes, write_via_userns).await
}
fn validate_filename(name: &str) -> Result<()> {
anyhow::ensure!(
!name.is_empty()
&& name != "."
&& name != ".."
&& !name.contains(['/', '\\', '\0'])
&& name.len() <= 255,
"Invalid purchased filename"
);
Ok(())
}
async fn save_new_file_with<F, Fut>(
dir: &Path,
name: &str,
bytes: &[u8],
fallback: F,
) -> Result<PathBuf>
where
F: FnOnce(PathBuf, String, Vec<u8>) -> Fut,
Fut: std::future::Future<Output = Result<PathBuf>>,
{
validate_filename(name)?;
// Never follow a user-created destination directory symlink.
match fs::symlink_metadata(dir).await {
Ok(meta) => anyhow::ensure!(meta.is_dir(), "Files destination is not a directory"),
Err(error) if error.kind() == std::io::ErrorKind::NotFound => {}
Err(error) => return Err(error.into()),
}
save_after_direct_result(
write_direct(dir, name, bytes).await,
dir,
name,
bytes,
fallback,
)
.await
}
async fn save_after_direct_result<F, Fut>(
result: std::io::Result<PathBuf>,
dir: &Path,
name: &str,
bytes: &[u8],
fallback: F,
) -> Result<PathBuf>
where
F: FnOnce(PathBuf, String, Vec<u8>) -> Fut,
Fut: std::future::Future<Output = Result<PathBuf>>,
{
match result {
Ok(path) => Ok(path),
Err(error) if error.kind() == std::io::ErrorKind::PermissionDenied => {
fallback(dir.to_owned(), name.to_owned(), bytes.to_vec())
.await
.context("Saving purchase in Files user namespace")
}
Err(error) => Err(error).context("Saving purchase in Files"),
}
}
fn numbered_name(name: &str, attempt: usize) -> String {
if attempt == 1 {
return name.to_owned();
}
match name.rsplit_once('.') {
Some((stem, extension)) if !stem.is_empty() => format!("{stem} ({attempt}).{extension}"),
_ => format!("{name} ({attempt})"),
}
}
struct PendingFile(PathBuf);
impl Drop for PendingFile {
fn drop(&mut self) {
let _ = std::fs::remove_file(&self.0);
}
}
async fn write_direct(dir: &Path, name: &str, bytes: &[u8]) -> std::io::Result<PathBuf> {
use std::os::unix::fs::PermissionsExt;
use tokio::io::AsyncWriteExt;
fs::create_dir_all(dir).await?;
let temp_path = dir.join(format!(".archy-saving-{}", uuid::Uuid::new_v4()));
let mut file = fs::OpenOptions::new()
.write(true)
.create_new(true)
.mode(0o600)
.open(&temp_path)
.await?;
let temp = PendingFile(temp_path);
file.write_all(bytes).await?;
file.set_permissions(std::fs::Permissions::from_mode(0o644))
.await?;
file.sync_all().await?;
for attempt in 1..=100 {
let target = dir.join(numbered_name(name, attempt));
match fs::hard_link(&temp.0, &target).await {
Ok(()) => return Ok(target),
Err(error) if error.kind() == std::io::ErrorKind::AlreadyExists => continue,
Err(error) => return Err(error),
}
}
Err(std::io::Error::new(
std::io::ErrorKind::AlreadyExists,
"Too many existing copies; purchase cache retained",
))
}
// Positional arguments carry all user-controlled text. mktemp prevents temp-name
// collisions; ln -T refuses files, symlinks and directories, including races.
const WRITE_VIA_USERNS: &str = r#"set -eu
dir=$1
name=$2
expected=$3
[ ! -L "$dir" ] || exit 1
if [ ! -d "$dir" ]; then
mkdir -p -- "$dir"
chown --reference="$(dirname -- "$dir")" -- "$dir"
fi
tmp=$(mktemp "$dir/.archy-saving.XXXXXXXXXX")
trap 'rm -f -- "$tmp"' EXIT HUP INT TERM
cat > "$tmp"
[ "$(wc -c < "$tmp")" -eq "$expected" ] || exit 1
chown --reference="$dir" -- "$tmp"
chmod 0644 -- "$tmp"
sync -f -- "$tmp"
stem=$name
ext=
case "$name" in
*.*) prefix=${name%.*}; if [ -n "$prefix" ]; then stem=$prefix; ext=.${name##*.}; fi ;;
esac
n=1
while [ "$n" -le 100 ]; do
candidate=$name
if [ "$n" -gt 1 ]; then candidate="$stem ($n)$ext"; fi
dst="$dir/$candidate"
if ln -T -- "$tmp" "$dst" 2>/dev/null; then
printf '%s' "$candidate"
exit 0
fi
# A conflict may be a dangling symlink; never follow it or overwrite it.
if [ ! -e "$dst" ] && [ ! -L "$dst" ]; then exit 1; fi
n=$((n + 1))
done
exit 1
"#;
async fn write_via_userns(dir: PathBuf, name: String, bytes: Vec<u8>) -> Result<PathBuf> {
use tokio::io::AsyncWriteExt;
let mut child = tokio::process::Command::new("podman")
.args(["unshare", "sh", "-c", WRITE_VIA_USERNS, "sh"])
.arg(&dir)
.arg(&name)
.arg(bytes.len().to_string())
.kill_on_drop(true)
.stdin(std::process::Stdio::piped())
.stdout(std::process::Stdio::piped())
.stderr(std::process::Stdio::piped())
.spawn()
.context("Starting Files namespace writer")?;
let mut stdin = child.stdin.take().context("Files writer stdin missing")?;
let operation = async {
let fed = stdin.write_all(&bytes).await;
drop(stdin);
let output = child.wait_with_output().await?;
anyhow::ensure!(
output.status.success(),
"Files namespace writer failed: {}",
output.status
);
fed.context("Sending purchase bytes to Files")?;
let chosen =
String::from_utf8(output.stdout).context("Files writer returned an invalid name")?;
validate_filename(&chosen)?;
anyhow::ensure!(
(1..=100).any(|n| numbered_name(&name, n) == chosen),
"Files writer returned an unexpected name"
);
Ok(dir.join(chosen))
};
tokio::time::timeout(std::time::Duration::from_secs(120), operation)
.await
.context("Files namespace writer timed out")?
}
#[cfg(test)] #[cfg(test)]
mod tests { mod tests {
use super::*; use super::*;
#[tokio::test]
async fn cloud_credentials_use_unique_record_and_never_default_password() {
let dir = tempfile::tempdir().unwrap();
assert!(cloud_credentials(dir.path()).await.is_err());
fs::write(dir.path().join("password"), "admin")
.await
.unwrap();
assert!(cloud_credentials(dir.path()).await.is_err());
fs::write(dir.path().join("password"), "legacy-unique-password")
.await
.unwrap();
assert_eq!(cloud_credentials(dir.path()).await.unwrap().schema, 0);
let value = serde_json::json!({"schema":1,"username":format!("archy-{}", "a".repeat(32)),"password":"b".repeat(64)});
fs::write(dir.path().join("credentials.json"), value.to_string())
.await
.unwrap();
let loaded = cloud_credentials(dir.path()).await.unwrap();
assert_eq!(loaded.username, value["username"].as_str().unwrap());
assert_eq!(loaded.password, value["password"].as_str().unwrap());
fs::write(dir.path().join("credentials.json"), "{}")
.await
.unwrap();
assert!(
cloud_credentials(dir.path()).await.is_err(),
"damaged managed record must not fall back to old credentials"
);
}
#[tokio::test]
async fn missing_config_preserves_legacy_database_and_refuses_ambiguity() {
let tmp = tempfile::tempdir().unwrap();
let paths = EnsurePaths {
srv_root: tmp.path().join("srv"),
data_dir: tmp.path().join("data"),
config_path: tmp.path().join("data/.filebrowser.json"),
};
fs::create_dir_all(&paths.data_dir).await.unwrap();
fs::write(paths.data_dir.join("database.db"), b"legacy fixture")
.await
.unwrap();
ensure_config(&paths).await.unwrap();
let config: serde_json::Value =
serde_json::from_slice(&fs::read(&paths.config_path).await.unwrap()).unwrap();
assert_eq!(config["database"], "/data/database.db");
fs::remove_file(&paths.config_path).await.unwrap();
fs::write(paths.data_dir.join("filebrowser.db"), b"other fixture")
.await
.unwrap();
assert!(ensure_config(&paths).await.is_err());
assert!(!paths.config_path.exists());
assert_eq!(
fs::read(paths.data_dir.join("database.db")).await.unwrap(),
b"legacy fixture"
);
}
#[tokio::test] #[tokio::test]
async fn ensure_config_creates_dirs_and_file() { async fn ensure_config_creates_dirs_and_file() {
let tmp = tempfile::TempDir::new().unwrap(); let tmp = tempfile::TempDir::new().unwrap();
@@ -152,3 +488,231 @@ mod tests {
assert_eq!(second, EnsureOutcome::Unchanged); assert_eq!(second, EnsureOutcome::Unchanged);
} }
} }
#[cfg(test)]
mod purchase_write_tests {
use super::*;
use std::{
collections::HashSet,
os::unix::fs::{symlink, PermissionsExt},
};
fn no_temps(dir: &Path) {
assert!(std::fs::read_dir(dir).unwrap().all(|e| !e
.unwrap()
.file_name()
.to_string_lossy()
.starts_with(".archy-saving")));
}
#[tokio::test]
async fn direct_write_uses_complete_bytes_and_preserves_originals() {
let dir = tempfile::tempdir().unwrap();
fs::write(dir.path().join("song.mp3"), b"original")
.await
.unwrap();
let target = save_new_file(dir.path(), "song.mp3", b"new").await.unwrap();
assert_eq!(target.file_name().unwrap(), "song (2).mp3");
assert_eq!(fs::read(target).await.unwrap(), b"new");
assert_eq!(
fs::read(dir.path().join("song.mp3")).await.unwrap(),
b"original"
);
no_temps(dir.path());
}
#[tokio::test]
async fn simultaneous_saves_publish_unique_complete_files() {
let dir = tempfile::tempdir().unwrap();
let mut tasks = Vec::new();
for n in 0..24u8 {
let dir = dir.path().to_owned();
tasks.push(tokio::spawn(async move {
let bytes = vec![n; 32768];
let path = save_new_file(&dir, "same.bin", &bytes).await.unwrap();
assert_eq!(fs::read(&path).await.unwrap(), bytes);
path
}));
}
let mut paths = HashSet::new();
for task in tasks {
assert!(paths.insert(task.await.unwrap()));
}
assert_eq!(paths.len(), 24);
no_temps(dir.path());
}
#[tokio::test]
async fn existing_directories_and_dangling_symlinks_are_conflicts() {
let dir = tempfile::tempdir().unwrap();
fs::create_dir(dir.path().join("name")).await.unwrap();
symlink("missing", dir.path().join("name (2)")).unwrap();
let path = save_new_file(dir.path(), "name", b"new").await.unwrap();
assert_eq!(path.file_name().unwrap(), "name (3)");
assert!(dir.path().join("name").is_dir());
assert!(fs::symlink_metadata(dir.path().join("name (2)"))
.await
.unwrap()
.is_symlink());
no_temps(dir.path());
}
#[tokio::test]
async fn invalid_names_and_symlink_destination_are_refused() {
let dir = tempfile::tempdir().unwrap();
for name in [
"",
".",
"..",
"../escape",
"/absolute",
"a/b",
"a\\b",
"a\0b",
] {
assert!(save_new_file(dir.path(), name, b"bytes").await.is_err());
}
let outside = tempfile::tempdir().unwrap();
symlink(outside.path(), dir.path().join("Music")).unwrap();
assert!(save_new_file(&dir.path().join("Music"), "song", b"bytes")
.await
.is_err());
assert_eq!(std::fs::read_dir(outside.path()).unwrap().count(), 0);
}
#[tokio::test]
async fn collision_limit_preserves_all_files_and_cleans_temporary_data() {
let dir = tempfile::tempdir().unwrap();
for n in 1..=100 {
fs::write(dir.path().join(numbered_name("a.txt", n)), b"keep")
.await
.unwrap();
}
assert!(save_new_file(dir.path(), "a.txt", b"new").await.is_err());
for n in 1..=100 {
assert_eq!(
fs::read(dir.path().join(numbered_name("a.txt", n)))
.await
.unwrap(),
b"keep"
);
}
no_temps(dir.path());
}
#[tokio::test]
async fn permission_fallback_is_exercised_without_skipping_as_root() {
let dir = tempfile::tempdir().unwrap();
let result = save_after_direct_result(
Err(std::io::ErrorKind::PermissionDenied.into()),
dir.path(),
"a",
b"abc",
|dir, name, bytes| async move {
assert_eq!(bytes, b"abc");
Ok(dir.join(name))
},
)
.await
.unwrap();
assert_eq!(result, dir.path().join("a"));
assert!(save_after_direct_result(
Err(std::io::ErrorKind::PermissionDenied.into()),
dir.path(),
"a",
b"abc",
|_, _, _| async { anyhow::bail!("namespace unavailable") }
)
.await
.unwrap_err()
.to_string()
.contains("namespace"));
assert!(save_after_direct_result(
Err(std::io::ErrorKind::StorageFull.into()),
dir.path(),
"a",
b"abc",
|_, _, _| async { panic!("disk full must not trigger permission fallback") }
)
.await
.is_err());
}
async fn run_script(
dir: &Path,
name: &str,
bytes: &[u8],
expected: usize,
) -> std::process::Output {
use tokio::io::AsyncWriteExt;
let mut child = tokio::process::Command::new("sh")
.args(["-c", WRITE_VIA_USERNS, "sh"])
.arg(dir)
.arg(name)
.arg(expected.to_string())
.stdin(std::process::Stdio::piped())
.stdout(std::process::Stdio::piped())
.stderr(std::process::Stdio::piped())
.spawn()
.unwrap();
let mut input = child.stdin.take().unwrap();
input.write_all(bytes).await.unwrap();
drop(input);
child.wait_with_output().await.unwrap()
}
#[tokio::test]
async fn namespace_script_preserves_names_bytes_modes_and_existing_entries() {
let dir = tempfile::tempdir().unwrap();
let folder = dir.path().join("Music");
let name = "song ' $() ; #.mp3";
for n in 1..=2 {
let output = run_script(&folder, name, b"abc", 3).await;
assert!(
output.status.success(),
"{}",
String::from_utf8_lossy(&output.stderr)
);
let chosen = String::from_utf8(output.stdout).unwrap();
assert_eq!(chosen, numbered_name(name, n));
let path = folder.join(chosen);
assert_eq!(fs::read(&path).await.unwrap(), b"abc");
assert_eq!(
fs::metadata(path).await.unwrap().permissions().mode() & 0o777,
0o644
);
}
no_temps(&folder);
}
#[tokio::test]
async fn namespace_script_refuses_truncated_input_and_cleans_up() {
let dir = tempfile::tempdir().unwrap();
let output = run_script(dir.path(), "never.bin", b"partial", 100).await;
assert!(!output.status.success());
assert!(!dir.path().join("never.bin").exists());
no_temps(dir.path());
}
#[tokio::test]
async fn namespace_script_does_not_link_inside_existing_directory() {
let dir = tempfile::tempdir().unwrap();
fs::create_dir(dir.path().join("name")).await.unwrap();
symlink("missing", dir.path().join("name (2)")).unwrap();
let output = run_script(dir.path(), "name", b"abc", 3).await;
assert!(output.status.success());
assert_eq!(output.stdout, b"name (3)");
assert_eq!(
std::fs::read_dir(dir.path().join("name")).unwrap().count(),
0
);
no_temps(dir.path());
}
#[test]
fn names_keep_extensions_and_dotfiles() {
assert_eq!(numbered_name("a.tar.gz", 2), "a.tar (2).gz");
assert_eq!(numbered_name(".hidden", 2), ".hidden (2)");
assert_eq!(numbered_name("README", 2), "README (2)");
}
}
+104
View File
@@ -89,18 +89,74 @@ bitcoind.estimatemode=ECONOMICAL\n"
Ok(EnsureOutcome::Written) Ok(EnsureOutcome::Written)
} }
/// Bitcoin can accept TCP while returning RPC_IN_WARMUP for many minutes.
/// Unlocking LND then triggers its short chain-backend timeout and a restart loop.
/// Leave the wallet intact and locked; the next reconciliation retries readiness.
async fn bitcoin_rpc_ready() -> bool {
let (user, password) = crate::bitcoin_rpc::bitcoin_rpc_credentials().await;
let client = match reqwest::Client::builder()
.no_proxy()
.timeout(std::time::Duration::from_secs(5))
.build()
{
Ok(client) => client,
Err(_) => return false,
};
let response = client.post(crate::constants::BITCOIN_RPC_URL)
.basic_auth(user, Some(password))
.json(&serde_json::json!({"jsonrpc":"1.0","id":"lnd-readiness","method":"getblockchaininfo","params":[]}))
.send().await;
match response {
Ok(response) if response.status().is_success() => response
.json::<serde_json::Value>()
.await
.is_ok_and(|value| bitcoin_readiness_response(&value)),
_ => false,
}
}
fn bitcoin_readiness_response(value: &serde_json::Value) -> bool {
value.get("error").is_none_or(|e| e.is_null())
&& value
.pointer("/result/blocks")
.and_then(|v| v.as_u64())
.is_some()
&& value
.pointer("/result/initialblockdownload")
.and_then(|v| v.as_bool())
.is_some()
}
pub async fn ensure_wallet_initialized() -> Result<()> { pub async fn ensure_wallet_initialized() -> Result<()> {
let admin_macaroon = "/var/lib/archipelago/lnd/data/chain/bitcoin/mainnet/admin.macaroon"; let admin_macaroon = "/var/lib/archipelago/lnd/data/chain/bitcoin/mainnet/admin.macaroon";
let wallet_db = "/var/lib/archipelago/lnd/data/chain/bitcoin/mainnet/wallet.db"; let wallet_db = "/var/lib/archipelago/lnd/data/chain/bitcoin/mainnet/wallet.db";
if file_exists_as_root(wallet_db).await { if file_exists_as_root(wallet_db).await {
// GetInfo can wait for Bitcoin sync even though the wallet is already
// unlocked. State RPC stays available during that normal startup phase.
let client = reqwest::Client::builder()
.no_proxy()
.timeout(std::time::Duration::from_secs(5))
.danger_accept_invalid_certs(true)
.build()?;
if wallet_is_unlocked(wallet_state(&client).await.as_deref()) {
return Ok(());
}
if file_exists_as_root(admin_macaroon).await && lnd_getinfo_ready(admin_macaroon).await { if file_exists_as_root(admin_macaroon).await && lnd_getinfo_ready(admin_macaroon).await {
return Ok(()); return Ok(());
} }
if !bitcoin_rpc_ready().await {
tracing::debug!("[lnd] waiting for Bitcoin RPC readiness before wallet unlock");
return Ok(());
}
unlock_existing_wallet_no_wipe().await?; unlock_existing_wallet_no_wipe().await?;
wait_for_admin_macaroon(admin_macaroon).await?; wait_for_admin_macaroon(admin_macaroon).await?;
return Ok(()); return Ok(());
} }
if !bitcoin_rpc_ready().await {
tracing::debug!("[lnd] waiting for Bitcoin RPC readiness before wallet initialization");
return Ok(());
}
init_wallet_via_rest().await?; init_wallet_via_rest().await?;
wait_for_admin_macaroon(admin_macaroon).await wait_for_admin_macaroon(admin_macaroon).await
} }
@@ -258,6 +314,9 @@ async fn unlock_existing_wallet_via_rest() -> Result<bool> {
// exactly the nodes least able to afford it. Waiting longer costs nothing — // exactly the nodes least able to afford it. Waiting longer costs nothing —
// a wrong password still exits on the first pass via `all_rejected`. // a wrong password still exits on the first pass via `all_rejected`.
for _ in 0..UNLOCK_NOT_READY_ATTEMPTS { for _ in 0..UNLOCK_NOT_READY_ATTEMPTS {
if wallet_is_unlocked(wallet_state(&client).await.as_deref()) {
return Ok(true);
}
let mut all_rejected = true; let mut all_rejected = true;
for pw in &candidates { for pw in &candidates {
match try_unlock_once(&client, pw).await { match try_unlock_once(&client, pw).await {
@@ -294,6 +353,10 @@ pub(crate) async fn unlock_existing_wallet_no_wipe() -> Result<()> {
} }
} }
fn wallet_is_unlocked(state: Option<&str>) -> bool {
matches!(state, Some("UNLOCKED" | "RPC_ACTIVE" | "SERVER_ACTIVE"))
}
/// Current LND wallet state via the unauthenticated `/v1/state` endpoint /// Current LND wallet state via the unauthenticated `/v1/state` endpoint
/// (NON_EXISTING / LOCKED / UNLOCKED / RPC_ACTIVE / …). None if unreachable. /// (NON_EXISTING / LOCKED / UNLOCKED / RPC_ACTIVE / …). None if unreachable.
async fn wallet_state(client: &reqwest::Client) -> Option<String> { async fn wallet_state(client: &reqwest::Client) -> Option<String> {
@@ -1089,3 +1152,44 @@ mod tests {
.is_empty()); .is_empty());
} }
} }
#[cfg(test)]
mod bitcoin_readiness_tests {
use super::bitcoin_readiness_response;
use serde_json::json;
#[test]
fn only_usable_bitcoin_rpc_allows_wallet_unlock() {
for response in [
json!({}),
json!({"error":{"code":-28,"message":"Loading block index"},"result":null}),
json!({"result":{"blocks":null}}),
] {
assert!(!bitcoin_readiness_response(&response));
}
// Initial sync is supported by LND. Loading the database is not.
for ibd in [true, false] {
assert!(bitcoin_readiness_response(
&json!({"result":{"blocks":100,"initialblockdownload":ibd},"error":null})
));
}
}
}
#[cfg(test)]
mod syncing_wallet_state_tests {
#[test]
fn an_unlocked_wallet_waiting_for_chain_sync_is_never_unlocked_again() {
for state in ["UNLOCKED", "RPC_ACTIVE", "SERVER_ACTIVE"] {
assert!(super::wallet_is_unlocked(Some(state)));
}
for state in [
None,
Some("LOCKED"),
Some("NON_EXISTING"),
Some("WAITING_TO_START"),
Some("unknown"),
] {
assert!(!super::wallet_is_unlocked(state));
}
}
}
@@ -0,0 +1,254 @@
//! Consistent, private snapshots for declaratively opted-in runtime migrations.
use anyhow::{bail, Context, Result};
use archipelago_container::AppManifest;
use std::os::unix::fs::PermissionsExt;
use std::path::{Path, PathBuf};
pub fn enabled(manifest: &AppManifest) -> Result<bool> {
match manifest.app.extensions.get("backup_before_runtime_change") {
None => Ok(false),
Some(value) => value
.as_bool()
.context("backup_before_runtime_change must be boolean"),
}
}
fn relative_sources(manifest: &AppManifest, data_dir: &Path) -> Result<Vec<PathBuf>> {
let mut sources = Vec::new();
for volume in &manifest.app.volumes {
if volume.options.iter().any(|v| v == "ro") || volume.volume_type == "tmpfs" {
continue;
}
// A runtime socket is a connection, not application state.
if volume.source == "/run/user/1000/podman/podman.sock" {
continue;
}
if volume.volume_type != "bind" {
bail!("runtime migration backup requires bind-mounted persistent state");
}
let path = Path::new(&volume.source);
let relative = path
.strip_prefix(data_dir)
.context("runtime migration state must be inside the node data directory")?;
if relative.starts_with("migration-backups") {
bail!("migration backup cannot include its own archive directory");
}
if relative.as_os_str().is_empty()
|| relative
.components()
.any(|c| !matches!(c, std::path::Component::Normal(_)))
{
bail!("invalid runtime migration state path");
}
sources.push(relative.to_path_buf());
}
sources.sort();
sources.dedup();
let mut roots: Vec<PathBuf> = Vec::new();
for source in sources {
if !roots.iter().any(|root| source.starts_with(root)) {
roots.push(source);
}
}
if roots.is_empty() {
bail!("runtime migration backup has no persistent state mounts");
}
Ok(roots)
}
/// Caller must gracefully stop the app before this function, and resume the old
/// service if it fails. No source files are changed or deleted by this operation.
pub async fn snapshot(
manifest: &AppManifest,
data_dir: &Path,
previous_unit: Option<&[u8]>,
) -> Result<PathBuf> {
let mut command = tokio::process::Command::new("podman");
command.args(["unshare", "tar"]);
snapshot_with_command(manifest, data_dir, previous_unit, command).await
}
async fn snapshot_with_command(
manifest: &AppManifest,
data_dir: &Path,
previous_unit: Option<&[u8]>,
mut command: tokio::process::Command,
) -> Result<PathBuf> {
let sources = relative_sources(manifest, data_dir)?;
let canonical_root = tokio::fs::canonicalize(data_dir).await?;
for source in &sources {
let path = data_dir.join(source);
if tokio::fs::symlink_metadata(&path)
.await?
.file_type()
.is_symlink()
{
bail!("runtime migration state mount is a symlink; explicit backup required");
}
let canonical = tokio::fs::canonicalize(&path).await?;
if !canonical.starts_with(&canonical_root) {
bail!("runtime migration state path resolves outside node data directory");
}
}
let root = data_dir.join("migration-backups");
tokio::fs::create_dir_all(&root).await?;
tokio::fs::set_permissions(&root, std::fs::Permissions::from_mode(0o700)).await?;
let dir = root.join(uuid::Uuid::new_v4().to_string());
tokio::fs::create_dir(&dir).await?;
tokio::fs::set_permissions(&dir, std::fs::Permissions::from_mode(0o700)).await?;
if let Some(unit) = previous_unit {
let path = dir.join("previous.container");
tokio::fs::write(&path, unit).await?;
tokio::fs::set_permissions(&path, std::fs::Permissions::from_mode(0o600)).await?;
tokio::fs::File::open(&path).await?.sync_all().await?;
}
let partial = dir.join("state.tar.partial");
let archive = dir.join("state.tar");
let output = command
.args([
"--create",
"--numeric-owner",
"--acls",
"--xattrs",
"--file",
])
.arg(&partial)
.arg("--directory")
.arg(data_dir)
.arg("--")
.args(&sources)
.output()
.await
.context("start rootless migration snapshot")?;
if !output.status.success() {
// No tar stderr in public logs: it can contain private filenames.
let _ = tokio::fs::remove_file(&partial).await;
bail!("persistent-state snapshot failed; original state was left intact");
}
tokio::fs::set_permissions(&partial, std::fs::Permissions::from_mode(0o600)).await?;
tokio::fs::File::open(&partial).await?.sync_all().await?;
tokio::fs::rename(&partial, &archive).await?;
let metadata = serde_json::json!({"app": manifest.app.id, "version": manifest.app.version,
"network": manifest.app.container.network, "capabilities": manifest.app.security.capabilities, "sources": sources});
tokio::fs::write(
dir.join("metadata.json"),
serde_json::to_vec_pretty(&metadata)?,
)
.await?;
tokio::fs::File::open(&dir).await?.sync_all().await?;
Ok(archive)
}
#[cfg(test)]
mod tests {
use super::*;
fn portainer() -> AppManifest {
AppManifest::parse(include_str!("../../../../apps/portainer/manifest.yml")).unwrap()
}
#[tokio::test]
async fn stopped_state_archive_round_trips_database_compose_and_old_unit() {
let dir = tempfile::tempdir().unwrap();
let state = dir.path().join("portainer");
tokio::fs::create_dir_all(state.join("compose"))
.await
.unwrap();
tokio::fs::write(state.join("portainer.db"), b"fixture database")
.await
.unwrap();
tokio::fs::write(state.join("compose/stack.yml"), b"services: {}\n")
.await
.unwrap();
let mut m = portainer();
m.app.volumes[0].source = state.display().to_string();
m.app.volumes[1].source = state.join("compose").display().to_string();
let archive = snapshot_with_command(
&m,
dir.path(),
Some(b"old unit"),
tokio::process::Command::new("tar"),
)
.await
.unwrap();
assert_eq!(
std::fs::metadata(&archive).unwrap().permissions().mode() & 0o777,
0o600
);
assert_eq!(
tokio::fs::read(archive.parent().unwrap().join("previous.container"))
.await
.unwrap(),
b"old unit"
);
let restored = tempfile::tempdir().unwrap();
assert!(tokio::process::Command::new("tar")
.arg("-xf")
.arg(archive)
.arg("-C")
.arg(restored.path())
.status()
.await
.unwrap()
.success());
assert_eq!(
tokio::fs::read(restored.path().join("portainer/portainer.db"))
.await
.unwrap(),
b"fixture database"
);
assert_eq!(
tokio::fs::read(restored.path().join("portainer/compose/stack.yml"))
.await
.unwrap(),
b"services: {}\n"
);
assert_eq!(
tokio::fs::read(state.join("portainer.db")).await.unwrap(),
b"fixture database"
);
}
#[tokio::test]
async fn failed_snapshot_never_publishes_archive_or_changes_original_state() {
let dir = tempfile::tempdir().unwrap();
let state = dir.path().join("portainer");
tokio::fs::create_dir_all(state.join("compose"))
.await
.unwrap();
tokio::fs::write(state.join("portainer.db"), b"unchanged")
.await
.unwrap();
let mut m = portainer();
m.app.volumes[0].source = state.display().to_string();
m.app.volumes[1].source = state.join("compose").display().to_string();
assert!(
snapshot_with_command(&m, dir.path(), None, tokio::process::Command::new("false"))
.await
.is_err()
);
assert_eq!(
tokio::fs::read(state.join("portainer.db")).await.unwrap(),
b"unchanged"
);
for entry in std::fs::read_dir(dir.path().join("migration-backups")).unwrap() {
assert!(!entry.unwrap().path().join("state.tar").exists());
}
}
#[test]
fn backup_covers_all_portainer_state_once_and_excludes_runtime_socket() {
let m = portainer();
assert!(enabled(&m).unwrap());
assert_eq!(
relative_sources(&m, Path::new("/var/lib/archipelago")).unwrap(),
vec![PathBuf::from("portainer")]
);
}
#[test]
fn backup_refuses_unknown_state_locations_instead_of_silently_omitting_them() {
let mut m = portainer();
m.app.volumes[0].source = "/other/operator/state".into();
assert!(relative_sources(&m, Path::new("/var/lib/archipelago")).is_err());
m.app.volumes[0].source = "/var/lib/archipelago/../secret".into();
assert!(relative_sources(&m, Path::new("/var/lib/archipelago")).is_err());
}
}
+2
View File
@@ -12,6 +12,8 @@ pub mod hooks;
pub mod image_policy; pub mod image_policy;
pub mod image_versions; pub mod image_versions;
pub mod lnd; pub mod lnd;
pub mod migration_backup;
pub mod npm;
pub mod prod_orchestrator; pub mod prod_orchestrator;
pub mod quadlet; pub mod quadlet;
pub mod registry; pub mod registry;
+115
View File
@@ -0,0 +1,115 @@
//! Preserve NPM's active persistent mounts across installer and runtime paths.
use anyhow::{bail, Context, Result};
use archipelago_container::AppManifest;
use serde::Deserialize;
use std::path::Path;
use std::time::Duration;
#[derive(Debug, Deserialize)]
pub struct Storage {
pub data: String,
pub certificates: String,
}
impl Storage {
pub fn bind_mounts(&self) -> Vec<String> {
vec![
format!("{}:/data", self.data),
format!("{}:/etc/letsencrypt", self.certificates),
]
}
pub fn apply(&self, manifest: &mut AppManifest) -> Result<()> {
for (target, source) in [
("/data", &self.data),
("/etc/letsencrypt", &self.certificates),
] {
let matching: Vec<_> = manifest
.app
.volumes
.iter_mut()
.filter(|volume| volume.target == target)
.collect();
if matching.len() != 1 {
bail!("NPM requires one persistent mount per storage target");
}
let volume = matching.into_iter().next().unwrap();
if volume.volume_type != "bind" {
bail!("NPM persistent state requires bind mounts");
}
volume.source.clone_from(source);
}
Ok(())
}
}
fn parse_storage(bytes: &[u8]) -> Result<Storage> {
let storage: Storage =
serde_json::from_slice(bytes).context("invalid NPM storage resolution")?;
for value in [&storage.data, &storage.certificates] {
if !Path::new(value).is_absolute() || value.chars().any(|c| c.is_control() || c == ':') {
bail!("invalid NPM persistent storage path");
}
}
Ok(storage)
}
pub async fn resolve_storage() -> Result<Storage> {
// Verify live mounts/database before any caller can stop or recreate NPM.
// Remember only validated mount paths so later recreation, after the inspect
// record is removed, still uses the operator's original storage.
let mut command = tokio::process::Command::new("python3");
command
.args([
"-c",
include_str!("../../../../scripts/npm-public-bridge.py"),
"--resolve",
"--remember",
"--prepare-realip",
])
.kill_on_drop(true);
let output = tokio::time::timeout(Duration::from_secs(75), command.output())
.await
.context("NPM storage resolution timed out; existing state preserved")??;
if !output.status.success() {
bail!("NPM storage resolution failed; inspect mount/database ambiguity or permissions before migration");
}
parse_storage(&output.stdout)
}
#[cfg(test)]
mod tests {
use super::*;
#[test]
fn resolved_mounts_preserve_custom_and_legacy_paths() {
for data in [
"/var/lib/archipelago/nginx-proxy-manager/data",
"/srv/operator npm",
] {
let bytes = serde_json::to_vec(
&serde_json::json!({"data": data, "certificates": "/srv/certificates"}),
)
.unwrap();
let storage = parse_storage(&bytes).unwrap();
assert_eq!(
storage.bind_mounts(),
[
format!("{data}:/data"),
"/srv/certificates:/etc/letsencrypt".into(),
]
);
}
}
#[test]
fn invalid_resolution_cannot_become_a_container_mount() {
for data in ["relative", "/srv/data:ro", "/srv/data\nother"] {
let bytes =
serde_json::to_vec(&serde_json::json!({"data": data, "certificates": "/certs"}))
.unwrap();
assert!(parse_storage(&bytes).is_err());
}
assert!(parse_storage(b"{}").is_err());
}
}
File diff suppressed because it is too large Load Diff
+550 -27
View File
@@ -68,6 +68,10 @@ pub enum NetworkMode {
/// Rootless slirp4netns networking. Podman rejects network aliases with /// Rootless slirp4netns networking. Podman rejects network aliases with
/// this mode, so render only Network=slirp4netns. /// this mode, so render only Network=slirp4netns.
Slirp4netns, Slirp4netns,
/// Permit explicit host aliases as well as LAN upstreams for NPM.
Slirp4netnsHostLoopback,
/// Preserve existing NPM upstreams using pasta's former host gateway.
Slirp4netnsLegacyGateway,
/// Rootless pasta networking. This is more reliable than slirp4netns for /// Rootless pasta networking. This is more reliable than slirp4netns for
/// host port forwarding on long-running web apps. /// host port forwarding on long-running web apps.
Pasta, Pasta,
@@ -184,6 +188,7 @@ pub struct QuadletUnit {
pub no_new_privileges: bool, pub no_new_privileges: bool,
pub cpu_quota: Option<u32>, pub cpu_quota: Option<u32>,
pub restart_policy: RestartPolicy, pub restart_policy: RestartPolicy,
pub stop_grace_secs: Option<u64>,
} }
impl QuadletUnit { impl QuadletUnit {
@@ -216,6 +221,10 @@ impl QuadletUnit {
let _ = writeln!(s, "[Container]"); let _ = writeln!(s, "[Container]");
let _ = writeln!(s, "ContainerName={}", self.name); let _ = writeln!(s, "ContainerName={}", self.name);
let _ = writeln!(s, "Image={}", self.image); let _ = writeln!(s, "Image={}", self.image);
let grace = self
.stop_grace_secs
.unwrap_or_else(|| archipelago_container::runtime::stop_grace_secs_for(&self.name));
let _ = writeln!(s, "StopTimeout={grace}");
// Pull=never: companions are pre-pulled or built. A missing image // Pull=never: companions are pre-pulled or built. A missing image
// must surface as a unit start failure, not a silent retry storm. // must surface as a unit start failure, not a silent retry storm.
let _ = writeln!(s, "Pull=never"); let _ = writeln!(s, "Pull=never");
@@ -224,6 +233,15 @@ impl QuadletUnit {
NetworkMode::Host => { NetworkMode::Host => {
let _ = writeln!(s, "Network=host"); let _ = writeln!(s, "Network=host");
} }
NetworkMode::Slirp4netnsHostLoopback => {
let _ = writeln!(s, "Network=slirp4netns:allow_host_loopback=true");
}
NetworkMode::Slirp4netnsLegacyGateway => {
let _ = writeln!(
s,
"Network=slirp4netns:allow_host_loopback=true,cidr=169.254.1.0/24"
);
}
NetworkMode::Slirp4netns => { NetworkMode::Slirp4netns => {
let _ = writeln!(s, "Network=slirp4netns"); let _ = writeln!(s, "Network=slirp4netns");
} }
@@ -343,6 +361,26 @@ impl QuadletUnit {
} }
let _ = writeln!(s); let _ = writeln!(s);
let _ = writeln!(s, "[Service]"); let _ = writeln!(s, "[Service]");
if self.name == "filebrowser" {
// Runs while the managed server is stopped, before it can expose
// a default account. The helper verifies real login and root access.
let mut argv = vec![
"/usr/bin/python3".to_string(),
"/opt/archipelago/scripts/filebrowser-credentials.py".to_string(),
"--image".to_string(),
self.image.clone(),
];
for (target, flag) in [("/data", "--data-dir"), ("/srv", "--srv-root")] {
if let Some(mount) = self
.bind_mounts
.iter()
.find(|m| m.container == Path::new(target))
{
argv.extend([flag.to_string(), mount.host.display().to_string()]);
}
}
let _ = writeln!(s, "ExecStartPre={}", shell_join(&argv));
}
// Dependency-gated apps may legitimately keep their container entrypoint // Dependency-gated apps may legitimately keep their container entrypoint
// in a wait loop before the actual daemon binds ports. Fedimint waits // in a wait loop before the actual daemon binds ports. Fedimint waits
// for Bitcoin IBD to finish before execing fedimintd; systemd's default // for Bitcoin IBD to finish before execing fedimintd; systemd's default
@@ -350,6 +388,15 @@ impl QuadletUnit {
// the unit stuck in deactivating. Health/status remains app-level state, // the unit stuck in deactivating. Health/status remains app-level state,
// not a systemd start gate. // not a systemd start gate.
let _ = writeln!(s, "TimeoutStartSec=0"); let _ = writeln!(s, "TimeoutStartSec=0");
let _ = writeln!(s, "TimeoutStopSec={}", grace.saturating_add(15));
// Stop explicitly before Quadlet's generated `podman rm -f`. The
// existing container may still carry Podman's old 10-second default;
// StopTimeout alone only protects containers created after migration.
let _ = writeln!(s, "ExecStop=");
let _ = writeln!(
s,
"ExecStop=/usr/bin/podman stop --ignore --time={grace} --cidfile=%t/%N.cid"
);
// Restart policy + 10s backoff. RestartSec keeps a crash-loop // Restart policy + 10s backoff. RestartSec keeps a crash-loop
// from saturating the journal. Companions: Always. Backends: // from saturating the journal. Companions: Always. Backends:
// OnFailure (clean stops stay stopped). // OnFailure (clean stops stay stopped).
@@ -376,7 +423,10 @@ fn shell_join(parts: &[String]) -> String {
.iter() .iter()
.map(|p| { .map(|p| {
let p = p.replace(['\r', '\n'], " ").replace('%', "%%"); let p = p.replace(['\r', '\n'], " ").replace('%', "%%");
if p.is_empty() || p.chars().any(|c| c.is_whitespace() || "\"\\$`".contains(c)) { if p.is_empty()
|| p.chars()
.any(|c| c.is_whitespace() || "'\"\\$`".contains(c))
{
let escaped = p let escaped = p
.replace('\\', "\\\\") .replace('\\', "\\\\")
.replace('"', "\\\"") .replace('"', "\\\"")
@@ -396,7 +446,7 @@ fn quote_environment(env: &str) -> String {
if env.is_empty() if env.is_empty()
|| env || env
.chars() .chars()
.any(|c| c.is_whitespace() || "\"\\$`".contains(c)) .any(|c| c.is_whitespace() || "'\"\\$`".contains(c))
{ {
let escaped = env let escaped = env
.replace('\\', "\\\\") .replace('\\', "\\\\")
@@ -430,6 +480,12 @@ impl QuadletUnit {
other if !other.is_empty() && other != "isolated" => NetworkMode::Bridge(other.into()), other if !other.is_empty() && other != "isolated" => NetworkMode::Bridge(other.into()),
_ => match app.container.network.as_deref() { _ => match app.container.network.as_deref() {
Some("slirp4netns") => NetworkMode::Slirp4netns, Some("slirp4netns") => NetworkMode::Slirp4netns,
Some("slirp4netns:allow_host_loopback=true") => {
NetworkMode::Slirp4netnsHostLoopback
}
Some("slirp4netns:allow_host_loopback=true,cidr=169.254.1.0/24") => {
NetworkMode::Slirp4netnsLegacyGateway
}
Some("pasta") => NetworkMode::Pasta, Some("pasta") => NetworkMode::Pasta,
Some(n) if !n.is_empty() && n != "host" => NetworkMode::Bridge(n.into()), Some(n) if !n.is_empty() && n != "host" => NetworkMode::Bridge(n.into()),
_ => NetworkMode::Default, _ => NetworkMode::Default,
@@ -525,6 +581,9 @@ impl QuadletUnit {
// Always, not OnFailure: with quadlet's `--rm`, OnFailure left a // Always, not OnFailure: with quadlet's `--rm`, OnFailure left a
// cleanly-exited app deleted and unrestarted. See RestartPolicy. // cleanly-exited app deleted and unrestarted. See RestartPolicy.
restart_policy: RestartPolicy::Always, restart_policy: RestartPolicy::Always,
stop_grace_secs: Some(super::prod_orchestrator::resolve_stop_grace_secs(
manifest, name,
)),
} }
} }
} }
@@ -676,6 +735,13 @@ pub async fn unit_exists(name: &str) -> bool {
/// Resolve the per-user quadlet dir under $HOME. Created if missing. /// Resolve the per-user quadlet dir under $HOME. Created if missing.
pub async fn unit_dir() -> Result<PathBuf> { pub async fn unit_dir() -> Result<PathBuf> {
#[cfg(test)]
{
static TEST_UNITS: std::sync::OnceLock<PathBuf> = std::sync::OnceLock::new();
return Ok(TEST_UNITS
.get_or_init(|| tempfile::tempdir().unwrap().keep())
.clone());
}
let home = std::env::var_os("HOME") let home = std::env::var_os("HOME")
.map(PathBuf::from) .map(PathBuf::from)
.ok_or_else(|| anyhow!("HOME not set; cannot locate quadlet unit dir"))?; .ok_or_else(|| anyhow!("HOME not set; cannot locate quadlet unit dir"))?;
@@ -686,29 +752,76 @@ pub async fn unit_dir() -> Result<PathBuf> {
Ok(dir) Ok(dir)
} }
/// Atomically write `unit` into `dir/<name>.container` if the bytes /// The early same-node Portainer repair used a managed Quadlet drop-in. Once
/// differ from what's already there. Returns true if the file changed. /// the manifest supplies slirp, the two Network= entries are additive and
/// Podman rejects startup. Retire only that exact redundant managed override;
/// arbitrary operator settings must survive reconciliation.
pub async fn redundant_managed_network_override(
unit: &QuadletUnit,
dir: &Path,
) -> Result<Option<PathBuf>> {
if unit.name != "portainer" || !matches!(unit.network, NetworkMode::Slirp4netns) {
return Ok(None);
}
let path = dir.join("portainer.container.d/archy-same-node-network.conf");
let body = match fs::read_to_string(&path).await {
Ok(body) => body,
Err(error) if error.kind() == std::io::ErrorKind::NotFound => return Ok(None),
Err(error) => return Err(error).context("read managed Portainer network override"),
};
let lines: Vec<&str> = body
.lines()
.map(str::trim)
.filter(|line| !line.is_empty() && !line.starts_with(['#', ';']))
.collect();
Ok((lines == ["[Container]", "Network=slirp4netns"]).then_some(path))
}
async fn retire_managed_network_override(path: &Path) -> Result<()> {
let backup = path.with_extension("conf.retired");
match fs::hard_link(path, &backup).await {
Ok(()) => {}
Err(error) if error.kind() == std::io::ErrorKind::AlreadyExists => {
anyhow::ensure!(
fs::read(path).await? == fs::read(&backup).await?,
"Existing Portainer override backup differs; preserve both for operator review"
);
}
Err(error) => return Err(error).context("back up managed Portainer network override"),
}
fs::remove_file(path)
.await
.context("retire redundant Portainer network override")?;
tracing::info!("Retired redundant managed Portainer network override; backup retained");
Ok(())
}
/// Atomically write the manifest unit and retire known redundant managed
/// overrides. Returns true whenever systemd needs a daemon-reload.
pub async fn write_if_changed(unit: &QuadletUnit, dir: &Path) -> Result<bool> { pub async fn write_if_changed(unit: &QuadletUnit, dir: &Path) -> Result<bool> {
let path = dir.join(unit.unit_filename()); let path = dir.join(unit.unit_filename());
let new_bytes = unit.render(); let new_bytes = unit.render();
let redundant = redundant_managed_network_override(unit, dir).await?;
if let Ok(old) = fs::read_to_string(&path).await { let changed = fs::read_to_string(&path)
if old == new_bytes { .await
return Ok(false); .map(|old| old != new_bytes)
} .unwrap_or(true);
if changed {
fs::create_dir_all(dir)
.await
.with_context(|| format!("create_dir_all {}", dir.display()))?;
let tmp = path.with_extension("container.tmp");
fs::write(&tmp, new_bytes.as_bytes())
.await
.with_context(|| format!("write tmp {}", tmp.display()))?;
fs::rename(&tmp, &path)
.await
.with_context(|| format!("rename {} -> {}", tmp.display(), path.display()))?;
} }
if let Some(override_path) = &redundant {
fs::create_dir_all(dir) retire_managed_network_override(override_path).await?;
.await }
.with_context(|| format!("create_dir_all {}", dir.display()))?; Ok(changed || redundant.is_some())
let tmp = path.with_extension("container.tmp");
fs::write(&tmp, new_bytes.as_bytes())
.await
.with_context(|| format!("write tmp {}", tmp.display()))?;
fs::rename(&tmp, &path)
.await
.with_context(|| format!("rename {} -> {}", tmp.display(), path.display()))?;
Ok(true)
} }
/// Reload the user systemd manager. Required after any quadlet write /// Reload the user systemd manager. Required after any quadlet write
@@ -785,7 +898,11 @@ pub async fn stop_service(service: &str) -> Result<()> {
/// corruption — so the orchestrator passes the per-app grace here. Never waits /// corruption — so the orchestrator passes the per-app grace here. Never waits
/// less than `QUADLET_STOP_TIMEOUT`. /// less than `QUADLET_STOP_TIMEOUT`.
pub async fn stop_service_with_timeout(service: &str, timeout: Duration) -> Result<()> { pub async fn stop_service_with_timeout(service: &str, timeout: Duration) -> Result<()> {
let timeout = timeout.max(QUADLET_STOP_TIMEOUT); let name = service.strip_suffix(".service").unwrap_or(service);
let body = fs::read_to_string(unit_dir().await?.join(format!("{name}.container")))
.await
.unwrap_or_default();
let timeout = timeout.max(stop_wait_timeout(name, &body));
match systemctl_user_status(&["stop", service], timeout).await { match systemctl_user_status(&["stop", service], timeout).await {
Ok(status) if status.success() => Ok(()), Ok(status) if status.success() => Ok(()),
Ok(status) => Err(anyhow!("systemctl --user stop {service} exited {status}")), Ok(status) => Err(anyhow!("systemctl --user stop {service} exited {status}")),
@@ -806,10 +923,29 @@ pub async fn stop_service_with_timeout(service: &str, timeout: Duration) -> Resu
} }
} }
/// The command waiter must outlive both the container grace and systemd's
/// stop deadline. Restart/repair callers must not kill Bitcoin at 45 seconds.
fn stop_wait_timeout(name: &str, unit_body: &str) -> Duration {
Duration::from_secs(stop_grace_from_unit(name, unit_body).saturating_add(30))
.max(QUADLET_STOP_TIMEOUT)
}
fn stop_grace_from_unit(name: &str, unit_body: &str) -> u64 {
directive_values(unit_body, "StopTimeout=")
.last()
.and_then(|value| value.parse::<u64>().ok())
.unwrap_or_else(|| archipelago_container::runtime::stop_grace_secs_for(name))
}
async fn systemctl_user_status( async fn systemctl_user_status(
args: &[&str], args: &[&str],
timeout: Duration, timeout: Duration,
) -> Result<std::process::ExitStatus> { ) -> Result<std::process::ExitStatus> {
#[cfg(test)]
{
use std::os::unix::process::ExitStatusExt;
return Ok(std::process::ExitStatus::from_raw(0));
}
let mut cmd = Command::new("systemctl"); let mut cmd = Command::new("systemctl");
cmd.arg("--user").args(args); cmd.arg("--user").args(args);
cmd.kill_on_drop(true); cmd.kill_on_drop(true);
@@ -856,6 +992,10 @@ async fn wait_not_deactivating(service: &str, timeout: Duration) -> bool {
} }
async fn systemctl_user_output(args: &[&str], timeout: Duration) -> Result<std::process::Output> { async fn systemctl_user_output(args: &[&str], timeout: Duration) -> Result<std::process::Output> {
#[cfg(test)]
{
anyhow::bail!("Unit tests have no real user service manager");
}
let mut cmd = Command::new("systemctl"); let mut cmd = Command::new("systemctl");
cmd.arg("--user").args(args); cmd.arg("--user").args(args);
cmd.kill_on_drop(true); cmd.kill_on_drop(true);
@@ -887,12 +1027,77 @@ pub fn health_cmd_changed(old_body: &str, new_body: &str) -> bool {
!= directive_values(new_body, "HealthRetries=") != directive_values(new_body, "HealthRetries=")
} }
/// A unit rewrite and a successful systemd restart are separate operations.
/// Keep the restart obligation across errors or a management-daemon restart.
pub struct RestartObligation {
marker: PathBuf,
pending: bool,
}
impl RestartObligation {
pub async fn prepare(unit_path: &Path, newly_required: bool) -> Result<Self> {
let marker = unit_path.with_extension("restart-pending");
if newly_required {
// Contents contain no manifest environment or credentials. sync_all
// makes the obligation durable before the subsequent unit rename.
let file = tokio::fs::OpenOptions::new()
.write(true)
.create(true)
.truncate(false)
.open(&marker)
.await
.context("record pending Quadlet restart")?;
file.sync_all().await?;
if let Some(parent) = marker.parent() {
tokio::fs::File::open(parent).await?.sync_all().await?;
}
}
let pending = tokio::fs::try_exists(&marker).await?;
Ok(Self { marker, pending })
}
pub fn is_pending(&self) -> bool {
self.pending
}
/// Call only after systemd accepted the replacement service successfully.
pub async fn complete(self) -> Result<()> {
if self.pending {
tokio::fs::remove_file(&self.marker)
.await
.context("clear completed Quadlet restart")?;
if let Some(parent) = self.marker.parent() {
tokio::fs::File::open(parent).await?.sync_all().await?;
}
}
Ok(())
}
}
pub fn publish_ports_changed(old_body: &str, new_body: &str) -> bool { pub fn publish_ports_changed(old_body: &str, new_body: &str) -> bool {
let old_ports = directive_values(old_body, "PublishPort="); let old_ports = directive_values(old_body, "PublishPort=");
let new_ports = directive_values(new_body, "PublishPort="); let new_ports = directive_values(new_body, "PublishPort=");
old_ports != new_ports old_ports != new_ports
} }
pub fn security_changed(old_body: &str, new_body: &str) -> bool {
[
"AddCapability=",
"DropCapability=",
"NoNewPrivileges=",
"ReadOnly=",
"User=",
]
.iter()
.any(|directive| {
let mut old = directive_values(old_body, directive);
let mut new = directive_values(new_body, directive);
old.sort();
new.sort();
old != new
})
}
pub fn network_aliases_changed(old_body: &str, new_body: &str) -> bool { pub fn network_aliases_changed(old_body: &str, new_body: &str) -> bool {
let old_network = directive_values(old_body, "Network="); let old_network = directive_values(old_body, "Network=");
let new_network = directive_values(new_body, "Network="); let new_network = directive_values(new_body, "Network=");
@@ -905,7 +1110,8 @@ pub fn exec_changed(old_body: &str, new_body: &str) -> bool {
// Entrypoint= and Exec= together define what the container runs, so a drift // Entrypoint= and Exec= together define what the container runs, so a drift
// in either must recreate the container (e.g. when this renderer first // in either must recreate the container (e.g. when this renderer first
// splits a folded `Exec=sh -lc ...` into `Entrypoint=sh` + `Exec=-lc ...`). // splits a folded `Exec=sh -lc ...` into `Entrypoint=sh` + `Exec=-lc ...`).
directive_values(old_body, "Exec=") != directive_values(new_body, "Exec=") directive_values(old_body, "ExecStartPre=") != directive_values(new_body, "ExecStartPre=")
|| directive_values(old_body, "Exec=") != directive_values(new_body, "Exec=")
|| directive_values(old_body, "Entrypoint=") != directive_values(new_body, "Entrypoint=") || directive_values(old_body, "Entrypoint=") != directive_values(new_body, "Entrypoint=")
} }
@@ -923,6 +1129,10 @@ fn directive_values(unit_body: &str, prefix: &str) -> Vec<String> {
/// that systemd no longer knows about. /// that systemd no longer knows about.
pub async fn disable_remove(unit_name: &str, dir: &Path) -> Result<()> { pub async fn disable_remove(unit_name: &str, dir: &Path) -> Result<()> {
let svc = format!("{unit_name}.service"); let svc = format!("{unit_name}.service");
let path = dir.join(format!("{unit_name}.container"));
let body = fs::read_to_string(&path).await.unwrap_or_default();
let timeout = stop_wait_timeout(unit_name, &body);
let grace = stop_grace_from_unit(unit_name, &body).to_string();
// Stop first; ignore failure (unit may already be down). BOUNDED — on // Stop first; ignore failure (unit may already be down). BOUNDED — on
// rootless podman a generated unit can wedge in "deactivating" while // rootless podman a generated unit can wedge in "deactivating" while
// `podman rm -f` hangs underneath it, and an unbounded `systemctl stop` // `podman rm -f` hangs underneath it, and an unbounded `systemctl stop`
@@ -930,13 +1140,12 @@ pub async fn disable_remove(unit_name: &str, dir: &Path) -> Result<()> {
// the package entry is stranded in `Removing` (a ghost in My Apps that also // the package entry is stranded in `Removing` (a ghost in My Apps that also
// blocks reinstall). If the graceful stop times out, escalate to // blocks reinstall). If the graceful stop times out, escalate to
// SIGKILL + reset-failed so teardown always proceeds. // SIGKILL + reset-failed so teardown always proceeds.
if systemctl_user_status(&["stop", &svc], QUADLET_STOP_TIMEOUT) if systemctl_user_status(&["stop", &svc], timeout)
.await .await
.is_err() .is_err()
{ {
let _ = kill_and_reset_service(&svc).await; let _ = kill_and_reset_service(&svc).await;
} }
let path = dir.join(format!("{unit_name}.container"));
if fs::try_exists(&path).await.unwrap_or(false) { if fs::try_exists(&path).await.unwrap_or(false) {
match fs::remove_file(&path).await { match fs::remove_file(&path).await {
Ok(()) => {} Ok(()) => {}
@@ -949,9 +1158,9 @@ pub async fn disable_remove(unit_name: &str, dir: &Path) -> Result<()> {
// Bounded so a hung podman store can't re-introduce the stall this function // Bounded so a hung podman store can't re-introduce the stall this function
// exists to avoid. // exists to avoid.
let _ = tokio::time::timeout( let _ = tokio::time::timeout(
QUADLET_STOP_TIMEOUT, timeout,
Command::new("podman") Command::new("podman")
.args(["rm", "-f", unit_name]) .args(["rm", "-f", "--ignore", "--time", &grace, unit_name])
.status(), .status(),
) )
.await; .await;
@@ -960,6 +1169,9 @@ pub async fn disable_remove(unit_name: &str, dir: &Path) -> Result<()> {
/// Is the quadlet-generated service currently active? /// Is the quadlet-generated service currently active?
pub async fn is_active(service: &str) -> bool { pub async fn is_active(service: &str) -> bool {
if cfg!(test) {
return false;
}
Command::new("systemctl") Command::new("systemctl")
.args(["--user", "is-active", "--quiet", service]) .args(["--user", "is-active", "--quiet", service])
.status() .status()
@@ -970,9 +1182,143 @@ pub async fn is_active(service: &str) -> bool {
#[cfg(test)] #[cfg(test)]
mod tests { mod tests {
#[test]
fn filebrowser_prestart_credentials_are_a_required_runtime_change() {
let unit = super::QuadletUnit {
name: "filebrowser".into(),
image: "registry.example/filebrowser:v2.63.23".into(),
..Default::default()
};
let rendered = unit.render();
assert!(rendered.contains("ExecStartPre=/usr/bin/python3 /opt/archipelago/scripts/filebrowser-credentials.py --image registry.example/filebrowser:v2.63.23"));
let old = rendered
.lines()
.filter(|line| !line.starts_with("ExecStartPre="))
.collect::<Vec<_>>()
.join("\n");
assert!(super::exec_changed(&old, &rendered));
assert!(!super::exec_changed(&rendered, &rendered));
let other = super::QuadletUnit {
name: "other-app".into(),
..unit
};
assert!(!other.render().contains("filebrowser-credentials.py"));
}
use super::*; use super::*;
use tempfile::tempdir; use tempfile::tempdir;
#[test]
fn shutdown_grace_covers_container_systemd_and_caller() {
for (name, grace) in [
("bitcoin-core", 600),
("bitcoin-knots", 600),
("lnd", 330),
("electrumx", 300),
("other", 30),
] {
let unit = QuadletUnit {
name: name.into(),
..Default::default()
};
let body = unit.render();
assert!(body.contains(&format!("StopTimeout={grace}\n")));
assert!(body.contains(&format!("TimeoutStopSec={}\n", grace + 15)));
assert!(body.contains(&format!("podman stop --ignore --time={grace} --cidfile=")));
assert_eq!(
stop_wait_timeout(name, &body),
Duration::from_secs(grace + 30)
);
// Legacy units have no StopTimeout directive yet.
assert_eq!(stop_wait_timeout(name, ""), Duration::from_secs(grace + 30));
}
}
#[test]
fn custom_stop_grace_survives_render_and_restart_budget() {
let manifest: AppManifest = serde_yaml::from_str(
r#"
app:
id: custom-db
name: Custom database
version: 1.0.0
stop_grace_secs: 900
container:
image: example/db:1
"#,
)
.unwrap();
let unit = QuadletUnit::from_manifest(&manifest, "custom-db");
assert_eq!(unit.stop_grace_secs, Some(900));
assert_eq!(
stop_wait_timeout("custom-db", &unit.render()),
Duration::from_secs(930)
);
assert_eq!(
stop_wait_timeout("lnd", "StopTimeout=invalid"),
Duration::from_secs(360)
);
}
#[test]
fn stop_grace_migration_does_not_request_an_execution_restart() {
let unit = sample_unit();
let new = unit.render();
let old = new
.lines()
.filter(|line| {
!line.starts_with("StopTimeout=")
&& !line.starts_with("TimeoutStopSec=")
&& !line.starts_with("ExecStop=")
})
.collect::<Vec<_>>()
.join("\n");
assert!(!exec_changed(&old, &new));
assert!(!publish_ports_changed(&old, &new));
assert!(!network_aliases_changed(&old, &new));
assert!(!health_cmd_changed(&old, &new));
}
#[test]
fn actual_quadlet_generator_stops_before_forced_removal() {
let generator = Path::new("/usr/lib/systemd/system-generators/podman-system-generator");
if !generator.exists() {
eprintln!(
"Quadlet generator unavailable; run this regression on the Linux release host"
);
return;
}
let dir = tempdir().unwrap();
let unit = QuadletUnit {
name: "grace-test".into(),
image: "localhost/test:latest".into(),
stop_grace_secs: Some(600),
..Default::default()
};
std::fs::write(dir.path().join("grace-test.container"), unit.render()).unwrap();
let output = std::process::Command::new(generator)
.args(["--user", "--dryrun"])
.env("QUADLET_UNIT_DIRS", dir.path())
.output()
.unwrap();
assert!(
output.status.success(),
"{}",
String::from_utf8_lossy(&output.stderr)
);
let generated = String::from_utf8_lossy(&output.stdout).to_string()
+ &String::from_utf8_lossy(&output.stderr);
let stop = generated
.find("ExecStop=/usr/bin/podman stop --ignore --time=600")
.unwrap();
let remove = generated.find("ExecStop=/usr/bin/podman rm ").unwrap();
assert!(
stop < remove,
"Legacy container must stop gracefully before removal"
);
assert!(generated.contains("--stop-timeout 600"));
assert!(generated.contains("TimeoutStopSec=615"));
}
#[test] #[test]
fn render_emits_secret_env_by_reference_never_value() { fn render_emits_secret_env_by_reference_never_value() {
let u = QuadletUnit { let u = QuadletUnit {
@@ -1160,6 +1506,18 @@ mod tests {
); );
} }
#[test]
fn apostrophes_survive_quadlet_argument_and_environment_parsing() {
// A whitespace-free Node script reproduced this in a real Quadlet:
// unquoted apostrophes were consumed by the parser, changing JS strings
// into identifiers and preventing the app from starting.
assert_eq!(
shell_join(&["require('http')".into()]),
"\"require('http')\""
);
assert_eq!(quote_environment("NAME=O'Brien"), "\"NAME=O'Brien\"");
}
#[test] #[test]
fn quote_environment_quotes_values_with_spaces() { fn quote_environment_quotes_values_with_spaces() {
assert_eq!( assert_eq!(
@@ -1372,6 +1730,26 @@ app:
assert!(!s.contains("Network=host")); assert!(!s.contains("Network=host"));
} }
#[test]
fn portainer_catalog_network_repairs_same_node_routing_without_exposing_backend() {
let manifest = AppManifest::parse(include_str!("../../../../apps/portainer/manifest.yml"))
.expect("shipped Portainer manifest must parse");
let new = QuadletUnit::from_manifest(&manifest, "portainer").render();
assert!(new.contains("Network=slirp4netns\n"));
assert!(!new.contains("NetworkAlias="));
assert!(new.contains("PublishPort=127.0.0.1:9000:9000/tcp"));
assert!(!new.contains("PublishPort=0.0.0.0"));
// The upgrade changes networking only: retain both state mounts and the
// existing rootless socket, without an app.ini or repository rewrite.
assert!(new.contains("Volume=/var/lib/archipelago/portainer:/data"));
assert!(new.contains("Volume=/var/lib/archipelago/portainer/compose:/data/compose"));
assert!(new.contains("Volume=/run/user/1000/podman/podman.sock:/var/run/docker.sock"));
let old = new.replace("Network=slirp4netns\n", "");
assert!(network_aliases_changed(&old, &new));
assert!(!network_aliases_changed(&new, &new));
assert!(!publish_ports_changed(&old, &new));
}
#[test] #[test]
fn from_manifest_slirp4netns_omits_network_alias() { fn from_manifest_slirp4netns_omits_network_alias() {
let yaml = r#" let yaml = r#"
@@ -1393,6 +1771,24 @@ app:
assert!(!s.contains("--network-alias")); assert!(!s.contains("--network-alias"));
} }
#[test]
fn npm_network_preserves_same_node_upstreams_without_aliases() {
let m = AppManifest::parse(include_str!(
"../../../../apps/nginx-proxy-manager/manifest.yml"
))
.unwrap();
let rendered = QuadletUnit::from_manifest(&m, "nginx-proxy-manager").render();
assert_eq!(
rendered
.lines()
.filter(|line| line.starts_with("Network="))
.collect::<Vec<_>>(),
vec!["Network=slirp4netns:allow_host_loopback=true,cidr=169.254.1.0/24"]
);
assert!(!rendered.contains("NetworkAlias="));
assert!(!rendered.contains("--network-alias"));
}
#[test] #[test]
fn from_manifest_pasta_omits_network_alias() { fn from_manifest_pasta_omits_network_alias() {
let yaml = r#" let yaml = r#"
@@ -1722,6 +2118,133 @@ app:
assert!(!network_aliases_changed(new, new)); assert!(!network_aliases_changed(new, new));
} }
#[tokio::test]
async fn redundant_portainer_override_is_backed_up_and_retired_even_when_base_matches() {
let dir = tempfile::tempdir().unwrap();
let manifest =
AppManifest::parse(include_str!("../../../../apps/portainer/manifest.yml")).unwrap();
let unit = QuadletUnit::from_manifest(&manifest, "portainer");
assert!(write_if_changed(&unit, dir.path()).await.unwrap());
let path = dir
.path()
.join("portainer.container.d/archy-same-node-network.conf");
fs::create_dir_all(path.parent().unwrap()).await.unwrap();
let old = "[Container]\nNetwork=slirp4netns\n";
fs::write(&path, old).await.unwrap();
assert!(redundant_managed_network_override(&unit, dir.path())
.await
.unwrap()
.is_some());
assert!(write_if_changed(&unit, dir.path()).await.unwrap());
assert!(!path.exists());
assert_eq!(
fs::read_to_string(path.with_extension("conf.retired"))
.await
.unwrap(),
old
);
assert!(!write_if_changed(&unit, dir.path()).await.unwrap());
assert_eq!(
fs::read_to_string(dir.path().join("portainer.container"))
.await
.unwrap()
.matches("Network=slirp4netns")
.count(),
1
);
}
#[tokio::test]
async fn network_override_migration_preserves_operator_customizations_and_failed_backups() {
let dir = tempfile::tempdir().unwrap();
let manifest =
AppManifest::parse(include_str!("../../../../apps/portainer/manifest.yml")).unwrap();
let mut unit = QuadletUnit::from_manifest(&manifest, "portainer");
let path = dir
.path()
.join("portainer.container.d/archy-same-node-network.conf");
fs::create_dir_all(path.parent().unwrap()).await.unwrap();
for custom in [
"[Container]\nNetwork=custom-net\n",
"[Container]\nNetwork=slirp4netns\nEnvironment=OPERATOR_SETTING=1\n",
] {
fs::write(&path, custom).await.unwrap();
assert!(redundant_managed_network_override(&unit, dir.path())
.await
.unwrap()
.is_none());
write_if_changed(&unit, dir.path()).await.unwrap();
assert_eq!(fs::read_to_string(&path).await.unwrap(), custom);
}
fs::write(&path, "[Container]\nNetwork=slirp4netns\n")
.await
.unwrap();
unit.network = NetworkMode::Pasta;
assert!(redundant_managed_network_override(&unit, dir.path())
.await
.unwrap()
.is_none());
unit.network = NetworkMode::Slirp4netns;
fs::write(path.with_extension("conf.retired"), "different backup")
.await
.unwrap();
assert!(write_if_changed(&unit, dir.path()).await.is_err());
assert!(path.exists(), "failure must preserve the active override");
}
#[tokio::test]
async fn failed_runtime_change_remains_pending_when_unit_already_matches() {
let dir = tempfile::tempdir().unwrap();
let unit = dir.path().join("portainer.container");
tokio::fs::write(&unit, "[Container]\n").await.unwrap();
let pending = RestartObligation::prepare(&unit, true).await.unwrap();
assert!(pending.is_pending());
tokio::fs::write(&unit, "[Container]\nNetwork=slirp4netns\n")
.await
.unwrap();
// Simulate systemctl failure or daemon interruption after unit rewrite.
drop(pending);
let retry = RestartObligation::prepare(&unit, false).await.unwrap();
assert!(
retry.is_pending(),
"matching unit must not discard failed restart"
);
retry.complete().await.unwrap();
assert!(!RestartObligation::prepare(&unit, false)
.await
.unwrap()
.is_pending());
}
#[tokio::test]
async fn pending_runtime_change_errors_are_not_reported_as_success() {
let dir = tempfile::tempdir().unwrap();
let missing = dir.path().join("missing/app.container");
assert!(RestartObligation::prepare(&missing, true).await.is_err());
let unit = dir.path().join("app.container");
let pending = RestartObligation::prepare(&unit, true).await.unwrap();
tokio::fs::remove_file(unit.with_extension("restart-pending"))
.await
.unwrap();
assert!(pending.complete().await.is_err());
}
#[test]
fn gitea_ssh_sandbox_capability_is_applied_as_a_runtime_change() {
let manifest =
AppManifest::parse(include_str!("../../../../apps/gitea/manifest.yml")).unwrap();
manifest.validate().unwrap();
let new = QuadletUnit::from_manifest(&manifest, "gitea").render();
assert!(new.contains("AddCapability=SYS_CHROOT\n"));
let old = new.replace("AddCapability=SYS_CHROOT\n", "");
assert!(security_changed(&old, &new));
assert!(!security_changed(&new, &new));
assert!(!security_changed(
"AddCapability=CHOWN\nAddCapability=SETUID\n",
"AddCapability=SETUID\nAddCapability=CHOWN\n"
));
}
#[test] #[test]
fn network_aliases_changed_detects_network_mode_drift() { fn network_aliases_changed_detects_network_mode_drift() {
let old = "[Container]\nNetwork=slirp4netns\n"; let old = "[Container]\nNetwork=slirp4netns\n";
+136 -61
View File
@@ -1,80 +1,155 @@
//! Seller-side pending entitlements for Lightning-invoice peer-file sales (#46). //! Durable seller-side entitlements for peer-file invoices and on-chain sales.
//! //! Payment records must outlive browser polling, process restarts and invoice
//! When a buyer asks to pay for a paid catalog item with an external wallet (as //! expiry: an invoice can settle while the buyer is disconnected.
//! opposed to the local-ecash fast path), the *selling* node mints a Lightning
//! invoice on its own LND and records a pending entitlement here, keyed by the
//! invoice's payment hash. The buyer pays the invoice from any wallet and polls
//! for settlement; once the seller's LND confirms the invoice is settled we mark
//! the entitlement paid, and the content gate (`content_server::serve_content`)
//! then releases the file to anyone presenting that payment hash.
//!
//! State is in-memory and bounded by a TTL. If the seller restarts before the
//! buyer pays, the buyer simply requests a fresh invoice — no value is lost
//! because an unpaid invoice represents no money.
use std::collections::HashMap; use anyhow::{Context, Result};
use std::sync::LazyLock; use serde::{Deserialize, Serialize};
use std::time::{Duration, Instant}; use sha2::{Digest, Sha256};
use tokio::sync::Mutex; use std::path::{Path, PathBuf};
use tokio::{fs, io::AsyncWriteExt, sync::Mutex};
/// How long a pending/paid entitlement is retained. Generous enough for a human static WRITES: Mutex<()> = Mutex::const_new(());
/// to pay an invoice and download, short enough to keep the map small.
const ENTITLEMENT_TTL: Duration = Duration::from_secs(3600); // 1 hour
#[derive(Clone)] #[derive(Clone, Serialize, Deserialize)]
struct Entitlement { struct Entitlement {
content_id: String, content_id: String,
price_sats: u64, price_sats: u64,
paid: bool, paid: bool,
created_at: Instant,
} }
static ENTITLEMENTS: LazyLock<Mutex<HashMap<String, Entitlement>>> = fn path(data_dir: &Path, token: &str) -> PathBuf {
LazyLock::new(|| Mutex::new(HashMap::new())); data_dir.join("content-entitlements").join(format!(
"{}.json",
/// Drop expired entries. Caller must hold the lock. hex::encode(Sha256::digest(token.as_bytes()))
fn prune(map: &mut HashMap<String, Entitlement>) { ))
map.retain(|_, e| e.created_at.elapsed() < ENTITLEMENT_TTL);
} }
/// Record a freshly-minted invoice as a pending (unpaid) entitlement. async fn read(data_dir: &Path, token: &str) -> Result<Option<Entitlement>> {
pub async fn record_pending(payment_hash: &str, content_id: &str, price_sats: u64) { match fs::read(path(data_dir, token)).await {
let mut map = ENTITLEMENTS.lock().await; Ok(bytes) => Ok(Some(
prune(&mut map); serde_json::from_slice(&bytes).context("Invalid payment entitlement")?,
map.insert( )),
payment_hash.to_string(), Err(e) if e.kind() == std::io::ErrorKind::NotFound => Ok(None),
Entitlement { Err(e) => Err(e).context("Reading payment entitlement"),
content_id: content_id.to_string(),
price_sats,
paid: false,
created_at: Instant::now(),
},
);
}
/// Mark the entitlement for `payment_hash` paid. No-op if unknown/expired.
pub async fn mark_paid(payment_hash: &str) {
let mut map = ENTITLEMENTS.lock().await;
prune(&mut map);
if let Some(e) = map.get_mut(payment_hash) {
e.paid = true;
} }
} }
/// The content_id + price an entitlement was issued for, if still live. async fn write(data_dir: &Path, token: &str, entry: &Entitlement) -> Result<()> {
pub async fn lookup(payment_hash: &str) -> Option<(String, u64)> { let target = path(data_dir, token);
let mut map = ENTITLEMENTS.lock().await; let dir = target.parent().unwrap();
prune(&mut map); fs::create_dir_all(dir).await?;
map.get(payment_hash) let tmp = target.with_extension("tmp");
.map(|e| (e.content_id.clone(), e.price_sats)) let mut file = fs::OpenOptions::new()
.write(true)
.create(true)
.truncate(true)
.mode(0o600)
.open(&tmp)
.await?;
file.write_all(&serde_json::to_vec(entry)?).await?;
file.sync_all().await?;
drop(file);
fs::rename(&tmp, &target).await?;
fs::File::open(dir).await?.sync_all().await?;
Ok(())
} }
/// True if `payment_hash` is a paid entitlement for exactly `content_id`. /// Save before exposing an invoice/address to the buyer. Never overwrite an
/// This is the gate the content server consults to release a file. /// existing payment or silently rebind its token to another item or price.
pub async fn is_paid_for(payment_hash: &str, content_id: &str) -> bool { pub async fn record_pending(
let mut map = ENTITLEMENTS.lock().await; data_dir: &Path,
prune(&mut map); token: &str,
map.get(payment_hash) content_id: &str,
price_sats: u64,
) -> Result<()> {
let _lock = WRITES.lock().await;
if let Some(existing) = read(data_dir, token).await? {
anyhow::ensure!(
existing.content_id == content_id && existing.price_sats == price_sats,
"Payment entitlement mismatch"
);
return Ok(());
}
write(
data_dir,
token,
&Entitlement {
content_id: content_id.into(),
price_sats,
paid: false,
},
)
.await
}
pub async fn mark_paid(data_dir: &Path, token: &str) -> Result<()> {
let _lock = WRITES.lock().await;
let mut entry = read(data_dir, token)
.await?
.context("Unknown payment entitlement")?;
entry.paid = true;
write(data_dir, token, &entry).await
}
pub async fn lookup(data_dir: &Path, token: &str) -> Result<Option<(String, u64)>> {
Ok(read(data_dir, token)
.await?
.map(|e| (e.content_id, e.price_sats)))
}
pub async fn is_paid_for(data_dir: &Path, token: &str, content_id: &str) -> bool {
read(data_dir, token)
.await
.ok()
.flatten()
.map(|e| e.paid && e.content_id == content_id) .map(|e| e.paid && e.content_id == content_id)
.unwrap_or(false) .unwrap_or(false)
} }
#[cfg(test)]
mod tests {
use super::*;
#[tokio::test]
async fn paid_entitlement_survives_reload_and_cannot_be_rebound() {
let dir = tempfile::tempdir().unwrap();
record_pending(dir.path(), "hash", "file", 12)
.await
.unwrap();
assert!(!is_paid_for(dir.path(), "hash", "file").await);
mark_paid(dir.path(), "hash").await.unwrap();
// All reads reopen disk; no process-local entitlement map exists.
assert!(is_paid_for(dir.path(), "hash", "file").await);
assert!(!is_paid_for(dir.path(), "hash", "other").await);
record_pending(dir.path(), "hash", "file", 12)
.await
.unwrap();
assert!(is_paid_for(dir.path(), "hash", "file").await);
assert!(record_pending(dir.path(), "hash", "other", 12)
.await
.is_err());
assert!(record_pending(dir.path(), "hash", "file", 13)
.await
.is_err());
let other = tempfile::tempdir().unwrap();
assert!(!is_paid_for(other.path(), "hash", "file").await);
assert!(mark_paid(dir.path(), "unknown").await.is_err());
}
#[tokio::test]
async fn corrupt_or_unwritable_records_fail_closed() {
let dir = tempfile::tempdir().unwrap();
record_pending(dir.path(), "../../token", "file", 1)
.await
.unwrap();
fs::write(path(dir.path(), "../../token"), b"broken")
.await
.unwrap();
assert!(lookup(dir.path(), "../../token").await.is_err());
assert!(!is_paid_for(dir.path(), "../../token", "file").await);
assert!(record_pending(dir.path(), "../../token", "file", 1)
.await
.is_err());
let file = dir.path().join("not-directory");
fs::write(&file, b"x").await.unwrap();
assert!(record_pending(&file, "hash", "file", 1).await.is_err());
}
}
+135 -12
View File
@@ -12,7 +12,9 @@
use anyhow::{Context, Result}; use anyhow::{Context, Result};
use serde::{Deserialize, Serialize}; use serde::{Deserialize, Serialize};
use std::path::{Path, PathBuf}; use std::path::{Path, PathBuf};
use tokio::fs; use tokio::{fs, io::AsyncWriteExt, sync::Mutex};
static PURCHASE_WRITES: Mutex<()> = Mutex::const_new(());
const OWNED_DIR: &str = "purchased-content"; const OWNED_DIR: &str = "purchased-content";
const OWNED_INDEX: &str = "owned.json"; const OWNED_INDEX: &str = "owned.json";
@@ -66,20 +68,51 @@ fn bytes_path(data_dir: &Path, onion: &str, content_id: &str) -> PathBuf {
.join(sanitize(content_id)) .join(sanitize(content_id))
} }
async fn load_index(data_dir: &Path) -> OwnedIndex { async fn load_index_checked(data_dir: &Path) -> Result<OwnedIndex> {
match fs::read_to_string(index_path(data_dir)).await { match fs::read_to_string(index_path(data_dir)).await {
Ok(s) => serde_json::from_str(&s).unwrap_or_default(), Ok(s) => serde_json::from_str(&s)
Err(_) => OwnedIndex::default(), .context("Invalid purchase index; existing records were preserved"),
Err(error) if error.kind() == std::io::ErrorKind::NotFound => Ok(OwnedIndex::default()),
Err(error) => Err(error).context("Reading purchase index"),
} }
} }
async fn load_index(data_dir: &Path) -> OwnedIndex {
load_index_checked(data_dir).await.unwrap_or_default()
}
async fn atomic_write(path: &Path, bytes: &[u8]) -> Result<()> {
let parent = path.parent().context("Purchase path has no parent")?;
fs::create_dir_all(parent).await?;
let temp = parent.join(format!(".purchase-{}.tmp", uuid::Uuid::new_v4()));
let result = async {
let mut file = fs::OpenOptions::new()
.write(true)
.create_new(true)
.mode(0o600)
.open(&temp)
.await?;
file.write_all(bytes).await?;
file.sync_all().await?;
drop(file);
fs::rename(&temp, path).await?;
fs::File::open(parent).await?.sync_all().await?;
Ok::<_, anyhow::Error>(())
}
.await;
if result.is_err() {
let _ = fs::remove_file(&temp).await;
}
result
}
async fn save_index(data_dir: &Path, index: &OwnedIndex) -> Result<()> { async fn save_index(data_dir: &Path, index: &OwnedIndex) -> Result<()> {
let root = owned_root(data_dir); let root = owned_root(data_dir);
fs::create_dir_all(&root) fs::create_dir_all(&root)
.await .await
.with_context(|| format!("creating {}", root.display()))?; .with_context(|| format!("creating {}", root.display()))?;
let content = serde_json::to_string_pretty(index).context("serializing owned index")?; let content = serde_json::to_string_pretty(index).context("serializing owned index")?;
fs::write(index_path(data_dir), content) atomic_write(&index_path(data_dir), content.as_bytes())
.await .await
.context("writing owned index") .context("writing owned index")
} }
@@ -98,17 +131,15 @@ pub async fn record_purchase(
ecash_backend: &str, ecash_backend: &str,
purchased_at: &str, purchased_at: &str,
) -> Result<()> { ) -> Result<()> {
// Read-modify-write must be one serialized transaction. Never replace a
// damaged index with an empty one, and never expose partially written bytes.
let _lock = PURCHASE_WRITES.lock().await;
let mut index = load_index_checked(data_dir).await?;
let path = bytes_path(data_dir, onion, content_id); let path = bytes_path(data_dir, onion, content_id);
if let Some(parent) = path.parent() { atomic_write(&path, bytes)
fs::create_dir_all(parent)
.await
.with_context(|| format!("creating {}", parent.display()))?;
}
fs::write(&path, bytes)
.await .await
.with_context(|| format!("writing purchased bytes to {}", path.display()))?; .with_context(|| format!("writing purchased bytes to {}", path.display()))?;
let mut index = load_index(data_dir).await;
let entry = OwnedItem { let entry = OwnedItem {
onion: onion.to_string(), onion: onion.to_string(),
content_id: content_id.to_string(), content_id: content_id.to_string(),
@@ -131,6 +162,11 @@ pub async fn record_purchase(
save_index(data_dir, &index).await save_index(data_dir, &index).await
} }
/// Payment decisions must not interpret an unreadable index as no purchases.
pub async fn list_owned_checked(data_dir: &Path) -> Result<Vec<OwnedItem>> {
Ok(load_index_checked(data_dir).await?.items)
}
/// Every item this node owns. /// Every item this node owns.
pub async fn list_owned(data_dir: &Path) -> Vec<OwnedItem> { pub async fn list_owned(data_dir: &Path) -> Vec<OwnedItem> {
load_index(data_dir).await.items load_index(data_dir).await.items
@@ -165,3 +201,90 @@ pub async fn read_owned(
.unwrap_or_else(|| "application/octet-stream".to_string()); .unwrap_or_else(|| "application/octet-stream".to_string());
Some((mime, bytes)) Some((mime, bytes))
} }
#[cfg(test)]
mod tests {
use super::*;
#[tokio::test]
async fn concurrent_purchases_preserve_every_item_and_exact_bytes() {
let dir = tempfile::tempdir().unwrap();
let mut jobs = tokio::task::JoinSet::new();
for n in 0..24 {
let root = dir.path().to_path_buf();
jobs.spawn(async move {
let id = format!("file-{n}");
record_purchase(
&root,
"seller.onion",
&id,
&id,
"text/plain",
id.as_bytes(),
5,
"lightning",
"now",
)
.await
.unwrap();
});
}
while let Some(result) = jobs.join_next().await {
result.unwrap();
}
assert_eq!(list_owned(dir.path()).await.len(), 24);
for n in 0..24 {
let id = format!("file-{n}");
assert!(is_owned(dir.path(), "seller.onion", &id).await);
let (mime, bytes) = read_owned(dir.path(), "seller.onion", &id).await.unwrap();
assert_eq!(mime, "text/plain");
assert_eq!(bytes, id.as_bytes());
}
record_purchase(
dir.path(),
"seller.onion",
"file-0",
"file-0",
"text/plain",
b"updated",
5,
"lightning",
"later",
)
.await
.unwrap();
assert_eq!(list_owned(dir.path()).await.len(), 24);
assert_eq!(
read_owned(dir.path(), "seller.onion", "file-0")
.await
.unwrap()
.1,
b"updated"
);
}
#[tokio::test]
async fn damaged_index_is_preserved_instead_of_erasing_prior_ownership() {
let dir = tempfile::tempdir().unwrap();
fs::create_dir_all(owned_root(dir.path())).await.unwrap();
fs::write(index_path(dir.path()), b"damaged but preserve me")
.await
.unwrap();
assert!(record_purchase(
dir.path(),
"seller.onion",
"new",
"new",
"text/plain",
b"bytes",
5,
"lightning",
"now"
)
.await
.is_err());
assert_eq!(
fs::read(index_path(dir.path())).await.unwrap(),
b"damaged but preserve me"
);
assert!(!bytes_path(dir.path(), "seller.onion", "new").exists());
}
}
+521 -86
View File
@@ -238,6 +238,11 @@ pub enum ServeResult {
Forbidden, Forbidden,
/// Content not found. /// Content not found.
NotFound, NotFound,
/// The catalog entry and file exist but this node can't read the file.
/// Returned before any payment is taken.
Unavailable,
/// Requested byte range cannot be served; no payment was taken.
RangeNotSatisfiable(u64),
} }
/// Serve a content item by ID with access control and optional range request. /// Serve a content item by ID with access control and optional range request.
@@ -252,6 +257,39 @@ pub async fn serve_content(
range: Option<ByteRange>, range: Option<ByteRange>,
owner_session: bool, owner_session: bool,
) -> Result<ServeResult> { ) -> Result<ServeResult> {
serve_content_with(
data_dir,
id,
payment_token,
invoice_hash,
peer_did,
range,
owner_session,
|path, range, mime| prepare_content(data_dir, path, range, mime),
|token, amount| async move { verify_payment_token(data_dir, &token, amount).await },
)
.await
}
// Inject only the read and payment boundaries, so tests can prove ordering
// without mint access, file-permission assumptions or privileged commands.
async fn serve_content_with<R, RF, V, VF>(
data_dir: &Path,
id: &str,
payment_token: Option<&str>,
invoice_hash: Option<&str>,
peer_did: Option<&str>,
range: Option<ByteRange>,
owner_session: bool,
read: R,
verify: V,
) -> Result<ServeResult>
where
R: FnOnce(PathBuf, Option<ByteRange>, String) -> RF,
RF: std::future::Future<Output = Result<ServeResult>>,
V: FnOnce(String, u64) -> VF,
VF: std::future::Future<Output = bool>,
{
let catalog = load_catalog(data_dir).await?; let catalog = load_catalog(data_dir).await?;
let item = match catalog.items.iter().find(|i| i.id == id) { let item = match catalog.items.iter().find(|i| i.id == id) {
Some(i) => i, Some(i) => i,
@@ -296,46 +334,6 @@ pub async fn serve_content(
} }
} }
// Check access control
if !owner_session {
match &item.access {
AccessControl::Paid { price_sats, .. } => {
// Two ways to satisfy payment:
// (a) a valid ecash token (the local-wallet fast path), or
// (b) a Lightning-invoice payment hash this node issued and has
// since confirmed settled (the "pay from any wallet" path, #46).
// Each path only counts when the sharer accepts that method.
let mut authorized = false;
if let Some(token) = payment_token {
if (method_accepted(&item.access, "ecash")
|| method_accepted(&item.access, "fedimint"))
&& verify_payment_token(data_dir, token, *price_sats).await
{
authorized = true;
}
}
if !authorized {
if let Some(hash) = invoice_hash {
if method_accepted(&item.access, "lightning")
&& crate::content_invoice::is_paid_for(hash, id).await
{
authorized = true;
}
}
}
if !authorized {
return Ok(ServeResult::PaymentRequired(*price_sats));
}
}
AccessControl::PeersOnly => {
if !is_known_peer {
return Ok(ServeResult::Forbidden);
}
}
AccessControl::Free => {}
}
}
let file_path = content_file_path(data_dir, item); let file_path = content_file_path(data_dir, item);
if !file_path.exists() { if !file_path.exists() {
// The catalog entry survived (it's a separate JSON file) but its // The catalog entry survived (it's a separate JSON file) but its
@@ -354,55 +352,194 @@ pub async fn serve_content(
return Ok(ServeResult::NotFound); return Ok(ServeResult::NotFound);
} }
let metadata = fs::metadata(&file_path) // Refuse unauthorized viewers before opening or reading any bytes.
.await if !owner_session && matches!(item.access, AccessControl::PeersOnly) && !is_known_peer {
.context("Failed to read file metadata")?; return Ok(ServeResult::Forbidden);
let total_size = metadata.len(); }
if !owner_session {
// Handle range request for streaming if let AccessControl::Paid { price_sats, .. } = &item.access {
if let Some(range) = range { if payment_token.is_none() && invoice_hash.is_none() {
let start = range.start.min(total_size.saturating_sub(1)); return Ok(ServeResult::PaymentRequired(*price_sats));
let end = range }
.end
.map(|e| e.min(total_size - 1))
.unwrap_or(total_size - 1);
if start > end || start >= total_size {
return Ok(ServeResult::NotFound);
} }
let len = (end - start + 1) as usize;
use tokio::io::{AsyncReadExt, AsyncSeekExt};
let mut file = tokio::fs::File::open(&file_path)
.await
.context("Failed to open content file")?;
file.seek(std::io::SeekFrom::Start(start))
.await
.context("Failed to seek")?;
let mut buf = vec![0u8; len];
file.read_exact(&mut buf)
.await
.context("Failed to read range")?;
debug!(
"Serving content '{}' range {}-{}/{} ({} bytes)",
id, start, end, total_size, len
);
return Ok(ServeResult::Partial {
bytes: buf,
mime_type: item.mime_type.clone(),
start,
end,
total: total_size,
});
} }
let bytes = fs::read(&file_path) // Finish all file I/O before consuming bearer payment. Merely opening then
.await // reopening after charging still lost payments on read errors or deletion.
.context("Failed to read content file")?; let prepared = match read(file_path, range, item.mime_type.clone()).await {
Ok(result @ (ServeResult::Ok(..) | ServeResult::Partial { .. })) => result,
Ok(other) => return Ok(other),
Err(error) => {
warn!(content_id = %id, "Cannot prepare shared content: {error:#}");
return Ok(ServeResult::Unavailable);
}
};
debug!("Serving content '{}' ({} bytes)", id, bytes.len()); // Check access control
Ok(ServeResult::Ok(bytes, item.mime_type.clone())) if !owner_session {
match &item.access {
AccessControl::Paid { price_sats, .. } => {
// Two ways to satisfy payment:
// (a) a valid ecash token (the local-wallet fast path), or
// (b) a Lightning-invoice payment hash this node issued and has
// since confirmed settled (the "pay from any wallet" path, #46).
// Each path only counts when the sharer accepts that method.
let mut authorized = false;
if let Some(token) = payment_token {
let method = if token.trim().starts_with("cashu") {
"ecash"
} else {
"fedimint"
};
if method_accepted(&item.access, method)
&& verify(token.to_owned(), *price_sats).await
{
authorized = true;
}
}
if !authorized {
if let Some(hash) = invoice_hash {
if method_accepted(&item.access, "lightning")
&& crate::content_invoice::is_paid_for(data_dir, hash, id).await
{
authorized = true;
}
}
}
if !authorized {
return Ok(ServeResult::PaymentRequired(*price_sats));
}
}
AccessControl::PeersOnly => {
if !is_known_peer {
return Ok(ServeResult::Forbidden);
}
}
AccessControl::Free => {}
}
}
Ok(prepared)
}
async fn prepare_content(
data_dir: &Path,
path: PathBuf,
range: Option<ByteRange>,
mime: String,
) -> Result<ServeResult> {
use tokio::io::{AsyncReadExt, AsyncSeekExt};
let mut file = match fs::OpenOptions::new()
.read(true)
.custom_flags(libc::O_NONBLOCK)
.open(&path)
.await
{
Ok(file) => file,
Err(error) if error.kind() == std::io::ErrorKind::PermissionDenied => {
let bytes = read_filebrowser_via_userns(data_dir, &path).await?;
return slice_prepared_content(bytes, range, mime);
}
Err(error) => return Err(error).context("Opening shared content"),
};
let metadata = file.metadata().await?;
anyhow::ensure!(metadata.is_file(), "Shared content is not a regular file");
let total = metadata.len();
if let Some(range) = range {
let Some((start, end)) = checked_range(&range, total) else {
return Ok(ServeResult::RangeNotSatisfiable(total));
};
file.seek(std::io::SeekFrom::Start(start)).await?;
let len = usize::try_from(end - start + 1).context("Content range is too large")?;
let mut bytes = vec![0; len];
file.read_exact(&mut bytes)
.await
.context("Reading shared content range")?;
return Ok(ServeResult::Partial {
bytes,
mime_type: mime,
start,
end,
total,
});
}
let mut bytes = Vec::new();
file.read_to_end(&mut bytes)
.await
.context("Reading shared content")?;
Ok(ServeResult::Ok(bytes, mime))
}
fn checked_range(range: &ByteRange, total: u64) -> Option<(u64, u64)> {
let last = total.checked_sub(1)?;
let end = range.end.unwrap_or(last).min(last);
(range.start <= end && range.start < total).then_some((range.start, end))
}
fn slice_prepared_content(
bytes: Vec<u8>,
range: Option<ByteRange>,
mime: String,
) -> Result<ServeResult> {
let total = bytes.len() as u64;
match range {
None => Ok(ServeResult::Ok(bytes, mime)),
Some(range) => match checked_range(&range, total) {
Some((start, end)) => Ok(ServeResult::Partial {
bytes: bytes[start as usize..=end as usize].to_vec(),
mime_type: mime,
start,
end,
total,
}),
None => Ok(ServeResult::RangeNotSatisfiable(total)),
},
}
}
/// Read only an explicitly shared, regular file within FileBrowser storage.
/// Do not change its mode or grant world-readable access to paid/private data.
async fn filebrowser_read_path(data_dir: &Path, path: &Path) -> Result<PathBuf> {
let root = fs::canonicalize(data_dir.join("filebrowser")).await?;
let target = fs::canonicalize(path).await?;
anyhow::ensure!(
target.starts_with(&root) && target != root,
"Shared file is outside Files storage"
);
anyhow::ensure!(
fs::metadata(&target).await?.is_file(),
"Shared content is not a regular file"
);
Ok(target)
}
async fn read_filebrowser_via_userns(data_dir: &Path, path: &Path) -> Result<Vec<u8>> {
let path = filebrowser_read_path(data_dir, path).await?;
// Tests exercise the boundary explicitly; they never launch the host Podman.
#[cfg(test)]
{
let _ = path;
anyhow::bail!("Files namespace read disabled in unit tests")
}
#[cfg(not(test))]
{
let output = tokio::time::timeout(
std::time::Duration::from_secs(900),
tokio::process::Command::new("podman")
.args(["unshare", "cat", "--"])
.arg(path)
.kill_on_drop(true)
.output(),
)
.await
.context("Files namespace read timed out")??;
anyhow::ensure!(
output.status.success(),
"Files namespace read failed: {}",
output.status
);
Ok(output.stdout)
}
} }
/// Result of attempting to serve a preview. /// Result of attempting to serve a preview.
@@ -573,7 +710,7 @@ pub async fn serve_content_preview(data_dir: &Path, id: &str) -> Result<PreviewR
} }
/// Verify a payment token covers the required amount. /// Verify a payment token covers the required amount.
/// Accepts both cashuA tokens (real Cashu) and legacy cashuSend_ format. /// Accepts real Cashu tokens and Fedimint notes.
/// Swaps proofs at the mint to verify they're unspent before accepting. /// Swaps proofs at the mint to verify they're unspent before accepting.
async fn verify_payment_token(data_dir: &Path, token: &str, required_sats: u64) -> bool { async fn verify_payment_token(data_dir: &Path, token: &str, required_sats: u64) -> bool {
match crate::wallet::ecash::verify_and_receive_payment(data_dir, token, required_sats).await { match crate::wallet::ecash::verify_and_receive_payment(data_dir, token, required_sats).await {
@@ -725,3 +862,301 @@ mod prune_missing_content_tests {
assert_eq!(reloaded.items[0].id, "present-item"); assert_eq!(reloaded.items[0].id, "present-item");
} }
} }
#[cfg(test)]
mod paid_read_order_tests {
use super::*;
use std::sync::atomic::{AtomicUsize, Ordering};
async fn fixture(bytes: &[u8]) -> tempfile::TempDir {
let dir = tempfile::tempdir().unwrap();
fs::create_dir_all(dir.path().join("content/files"))
.await
.unwrap();
fs::write(dir.path().join("content/files/test.bin"), bytes)
.await
.unwrap();
save_catalog(
dir.path(),
&ContentCatalog {
items: vec![ContentItem {
id: "paid".into(),
filename: "test.bin".into(),
mime_type: "application/octet-stream".into(),
size_bytes: bytes.len() as u64,
description: String::new(),
access: AccessControl::Paid {
price_sats: 10,
accepted: vec!["ecash".into()],
},
availability: Availability::AllPeers,
added_at: "2026-09-30".into(),
}],
},
)
.await
.unwrap();
dir
}
#[tokio::test]
async fn all_read_failures_precede_redemption_even_as_root() {
for kind in [
std::io::ErrorKind::PermissionDenied,
std::io::ErrorKind::UnexpectedEof,
std::io::ErrorKind::NotFound,
std::io::ErrorKind::Other,
] {
let dir = fixture(b"abc").await;
let charged = AtomicUsize::new(0);
let result = serve_content_with(
dir.path(),
"paid",
Some("cashuBtest"),
None,
None,
None,
false,
|_, _, _| async move { Err(std::io::Error::from(kind).into()) },
|_, _| async {
charged.fetch_add(1, Ordering::SeqCst);
true
},
)
.await
.unwrap();
assert!(matches!(result, ServeResult::Unavailable));
assert_eq!(charged.load(Ordering::SeqCst), 0);
assert_eq!(load_catalog(dir.path()).await.unwrap().items.len(), 1);
}
}
#[tokio::test]
async fn deletion_during_payment_cannot_lose_prepared_bytes() {
let dir = fixture(b"original").await;
let result = serve_content_with(
dir.path(),
"paid",
Some("cashuBtest"),
None,
None,
None,
false,
|path, range, mime| prepare_content(dir.path(), path, range, mime),
|_, amount| {
assert_eq!(amount, 10);
async {
fs::remove_file(dir.path().join("content/files/test.bin"))
.await
.unwrap();
true
}
},
)
.await
.unwrap();
assert!(matches!(result, ServeResult::Ok(bytes, _) if bytes == b"original"));
}
#[tokio::test]
async fn empty_out_of_bounds_and_reversed_ranges_never_charge() {
for (bytes, start, end) in [
(b"".as_slice(), 0, None),
(b"abc".as_slice(), 3, None),
(b"abc".as_slice(), 2, Some(1)),
] {
let dir = fixture(bytes).await;
let result = serve_content_with(
dir.path(),
"paid",
Some("cashuBtest"),
None,
None,
Some(ByteRange { start, end }),
false,
|path, range, mime| prepare_content(dir.path(), path, range, mime),
|_, _| async { panic!("invalid range reached payment") },
)
.await
.unwrap();
assert!(
matches!(result, ServeResult::RangeNotSatisfiable(n) if n == bytes.len() as u64)
);
}
}
#[tokio::test]
async fn prepared_range_survives_file_change_while_payment_is_verified() {
let dir = fixture(b"abcdef").await;
let result = serve_content_with(
dir.path(),
"paid",
Some("cashuBtest"),
None,
None,
Some(ByteRange {
start: 2,
end: Some(999),
}),
false,
|path, range, mime| prepare_content(dir.path(), path, range, mime),
|_, _| async {
fs::write(dir.path().join("content/files/test.bin"), b"x")
.await
.unwrap();
true
},
)
.await
.unwrap();
assert!(
matches!(result, ServeResult::Partial { bytes, start: 2, end: 5, total: 6, .. } if bytes == b"cdef")
);
}
#[tokio::test]
async fn payment_denial_never_returns_prepared_content() {
let dir = fixture(b"secret").await;
let charged = AtomicUsize::new(0);
let result = serve_content_with(
dir.path(),
"paid",
Some("cashuBtest"),
None,
None,
None,
false,
|path, range, mime| prepare_content(dir.path(), path, range, mime),
|_, _| async {
charged.fetch_add(1, Ordering::SeqCst);
false
},
)
.await
.unwrap();
assert!(matches!(result, ServeResult::PaymentRequired(10)));
assert_eq!(charged.load(Ordering::SeqCst), 1);
}
#[tokio::test]
async fn missing_payment_and_peer_restrictions_precede_file_reads() {
let dir = fixture(b"secret").await;
let result = serve_content_with(
dir.path(),
"paid",
None,
None,
None,
None,
false,
|_, _, _| async { panic!("unauthorized file read") },
|_, _| async { panic!("unexpected payment") },
)
.await
.unwrap();
assert!(matches!(result, ServeResult::PaymentRequired(10)));
let mut catalog = load_catalog(dir.path()).await.unwrap();
catalog.items[0].access = AccessControl::PeersOnly;
save_catalog(dir.path(), &catalog).await.unwrap();
let result = serve_content_with(
dir.path(),
"paid",
None,
None,
None,
None,
false,
|_, _, _| async { panic!("unauthorized file read") },
|_, _| async { panic!("unexpected payment") },
)
.await
.unwrap();
assert!(matches!(result, ServeResult::Forbidden));
}
#[tokio::test]
async fn owner_reads_paid_content_without_redemption() {
let dir = fixture(b"own file").await;
let result = serve_content_with(
dir.path(),
"paid",
None,
None,
None,
None,
true,
|path, range, mime| prepare_content(dir.path(), path, range, mime),
|_, _| async { panic!("owner charged") },
)
.await
.unwrap();
assert!(matches!(result, ServeResult::Ok(bytes, _) if bytes == b"own file"));
}
#[tokio::test]
async fn directory_in_place_of_file_does_not_charge() {
let dir = fixture(b"abc").await;
let path = dir.path().join("content/files/test.bin");
fs::remove_file(&path).await.unwrap();
fs::create_dir(&path).await.unwrap();
let result = serve_content_with(
dir.path(),
"paid",
Some("cashuBtest"),
None,
None,
None,
false,
|path, range, mime| prepare_content(dir.path(), path, range, mime),
|_, _| async { panic!("directory charged") },
)
.await
.unwrap();
assert!(matches!(result, ServeResult::Unavailable));
}
#[tokio::test]
async fn files_namespace_read_is_scoped_to_regular_files_and_keeps_mode() {
use std::os::unix::fs::{symlink, PermissionsExt};
let dir = fixture(b"outside").await;
let root = dir.path().join("filebrowser");
fs::create_dir(&root).await.unwrap();
let inside = root.join("song");
fs::write(&inside, b"song").await.unwrap();
fs::set_permissions(&inside, std::fs::Permissions::from_mode(0o640))
.await
.unwrap();
assert_eq!(
filebrowser_read_path(dir.path(), &inside).await.unwrap(),
inside
);
assert_eq!(
fs::metadata(&inside).await.unwrap().permissions().mode() & 0o777,
0o640
);
let outside = dir.path().join("content/files/test.bin");
symlink(&outside, root.join("escape")).unwrap();
for path in [outside, root.join("escape"), root.clone()] {
assert!(filebrowser_read_path(dir.path(), &path).await.is_err());
}
}
#[test]
fn user_namespace_bytes_use_the_same_range_rules() {
assert!(matches!(
slice_prepared_content(
vec![],
Some(ByteRange {
start: 0,
end: None
}),
"x".into()
)
.unwrap(),
ServeResult::RangeNotSatisfiable(0)
));
assert!(
matches!(slice_prepared_content(b"abc".to_vec(), Some(ByteRange { start: 1, end: None }), "x".into()).unwrap(), ServeResult::Partial { bytes, start: 1, end: 2, total: 3, .. } if bytes == b"bc")
);
}
}
+83 -18
View File
@@ -194,6 +194,7 @@ pub async fn clear_user_stopped(data_dir: &Path, name: &str) {
// Installation is a decision, not a runtime observation, so it gets a record // Installation is a decision, not a runtime observation, so it gets a record
// of its own that no amount of downtime erodes. // of its own that no amount of downtime erodes.
const INSTALLED_APPS_FILE: &str = "installed-apps.json"; const INSTALLED_APPS_FILE: &str = "installed-apps.json";
static INSTALLED_APPS_LOCK: tokio::sync::Mutex<()> = tokio::sync::Mutex::const_new(());
/// Load the durable set of installed app ids / container names. /// Load the durable set of installed app ids / container names.
pub async fn load_installed_apps(data_dir: &Path) -> std::collections::HashSet<String> { pub async fn load_installed_apps(data_dir: &Path) -> std::collections::HashSet<String> {
@@ -220,12 +221,23 @@ pub async fn load_installed_apps_if_recorded(
async fn save_installed_apps(data_dir: &Path, installed: &std::collections::HashSet<String>) { async fn save_installed_apps(data_dir: &Path, installed: &std::collections::HashSet<String>) {
let path = data_dir.join(INSTALLED_APPS_FILE); let path = data_dir.join(INSTALLED_APPS_FILE);
if let Ok(json) = serde_json::to_string_pretty(installed) { if let Ok(json) = serde_json::to_string_pretty(installed) {
let _ = fs::write(&path, json).await; let tmp = path.with_extension("json.tmp");
let result = async {
fs::write(&tmp, json).await?;
fs::File::open(&tmp).await?.sync_all().await?;
fs::rename(&tmp, &path).await?;
fs::File::open(data_dir).await?.sync_all().await
}
.await;
if let Err(error) = result {
warn!(%error, "could not persist installed apps");
}
} }
} }
/// Record that an app is installed. Called when an install succeeds. /// Record that an app is installed. Called when an install succeeds.
pub async fn mark_installed(data_dir: &Path, name: &str) { pub async fn mark_installed(data_dir: &Path, name: &str) {
let _guard = INSTALLED_APPS_LOCK.lock().await;
let mut installed = load_installed_apps(data_dir).await; let mut installed = load_installed_apps(data_dir).await;
if installed.insert(name.to_string()) { if installed.insert(name.to_string()) {
save_installed_apps(data_dir, &installed).await; save_installed_apps(data_dir, &installed).await;
@@ -235,6 +247,7 @@ pub async fn mark_installed(data_dir: &Path, name: &str) {
/// Forget an app. Called on uninstall, beside `mark_user_uninstalled` — the /// Forget an app. Called on uninstall, beside `mark_user_uninstalled` — the
/// two must move together or a reinstall-after-uninstall leaves a stale claim. /// two must move together or a reinstall-after-uninstall leaves a stale claim.
pub async fn clear_installed(data_dir: &Path, name: &str) { pub async fn clear_installed(data_dir: &Path, name: &str) {
let _guard = INSTALLED_APPS_LOCK.lock().await;
let mut installed = load_installed_apps(data_dir).await; let mut installed = load_installed_apps(data_dir).await;
if installed.remove(name) { if installed.remove(name) {
save_installed_apps(data_dir, &installed).await; save_installed_apps(data_dir, &installed).await;
@@ -252,6 +265,7 @@ pub async fn clear_installed(data_dir: &Path, name: &str) {
/// need it. Runs on every boot, so an app installed before the upgrade is /// need it. Runs on every boot, so an app installed before the upgrade is
/// still picked up whenever it is next seen alive. /// still picked up whenever it is next seen alive.
pub async fn backfill_installed_apps(data_dir: &Path, present_container_names: &[String]) { pub async fn backfill_installed_apps(data_dir: &Path, present_container_names: &[String]) {
let _guard = INSTALLED_APPS_LOCK.lock().await;
if present_container_names.is_empty() { if present_container_names.is_empty() {
return; return;
} }
@@ -650,6 +664,10 @@ pub async fn start_stopped_stack_containers(data_dir: &Path) -> RecoveryReport {
start_stopped_app_stacks(data_dir).await start_stopped_app_stacks(data_dir).await
} }
fn stack_member_needs_recovery(state: Option<&str>, user_stopped: bool) -> bool {
!user_stopped && matches!(state, Some("exited" | "stopped" | "created" | "configured"))
}
async fn start_stopped_app_stacks(data_dir: &Path) -> RecoveryReport { async fn start_stopped_app_stacks(data_dir: &Path) -> RecoveryReport {
let user_stopped = load_user_stopped(data_dir).await; let user_stopped = load_user_stopped(data_dir).await;
let mut report = RecoveryReport { let mut report = RecoveryReport {
@@ -663,24 +681,27 @@ async fn start_stopped_app_stacks(data_dir: &Path) -> RecoveryReport {
continue; continue;
} }
info!( // Healthy members must never acquire a restarting overlay merely
"Recovering stopped {} stack containers after boot", // because the periodic recovery scan ran. Queue existing stopped
stack.name // members only; recheck each immediately before starting below.
); let mut pending = Vec::new();
for container in stack.containers {
let state = container_state(container).await;
if stack_member_needs_recovery(state.as_deref(), user_stopped.contains(*container)) {
pending.push((*container).to_string());
}
}
if pending.is_empty() {
continue;
}
info!("Recovering stopped {} stack containers", stack.name);
repair_stack_network_aliases(stack).await; repair_stack_network_aliases(stack).await;
pending_boot_starts_add(pending.iter().cloned());
// Register the whole stack up front: the per-member dependency waits
// below can take minutes, and the UI should say "Restarting", not
// "Stopped", for members still queued behind them.
pending_boot_starts_add(
stack
.containers
.iter()
.filter(|c| !user_stopped.contains(**c))
.map(|c| (*c).to_string()),
);
for container in stack.containers { for container in stack.containers {
if !pending.iter().any(|name| name.as_str() == *container) {
continue;
}
if user_stopped.contains(*container) { if user_stopped.contains(*container) {
info!("Skipping user-stopped container: {}", container); info!("Skipping user-stopped container: {}", container);
continue; continue;
@@ -692,8 +713,8 @@ async fn start_stopped_app_stacks(data_dir: &Path) -> RecoveryReport {
pending_boot_start_done(container); pending_boot_start_done(container);
continue; continue;
} }
Some(_) => {} Some(state) if stack_member_needs_recovery(Some(&state), false) => {}
None => { _ => {
pending_boot_start_done(container); pending_boot_start_done(container);
continue; continue;
} }
@@ -1497,3 +1518,47 @@ mod tests {
); );
} }
} }
#[cfg(test)]
mod installed_concurrency_tests {
use super::*;
#[tokio::test]
async fn concurrent_install_records_are_not_lost() {
let dir = tempfile::tempdir().unwrap();
let mut tasks = Vec::new();
for i in 0..24 {
let path = dir.path().to_owned();
tasks.push(tokio::spawn(async move {
mark_installed(&path, &format!("app-{i}")).await;
}));
}
for task in tasks {
task.await.unwrap();
}
assert_eq!(load_installed_apps(dir.path()).await.len(), 24);
clear_installed(dir.path(), "app-3").await;
assert_eq!(load_installed_apps(dir.path()).await.len(), 23);
assert!(!dir.path().join("installed-apps.json.tmp").exists());
}
}
#[cfg(test)]
mod stack_recovery_overlay_tests {
use super::stack_member_needs_recovery;
#[test]
fn only_existing_stopped_members_receive_recovery_overlay() {
for state in [
None,
Some("running"),
Some("paused"),
Some("restarting"),
Some("removing"),
] {
assert!(!stack_member_needs_recovery(state, false));
}
for state in ["exited", "stopped", "created", "configured"] {
assert!(stack_member_needs_recovery(Some(state), false));
assert!(!stack_member_needs_recovery(Some(state), true));
}
}
}
+6
View File
@@ -146,6 +146,10 @@ pub enum PackageState {
#[derive(Debug, Clone, Serialize, Deserialize, PartialEq)] #[derive(Debug, Clone, Serialize, Deserialize, PartialEq)]
pub struct PackageDataEntry { pub struct PackageDataEntry {
/// Whether the app's HTTP upstream answered this scan (independent of
/// container health and blockchain sync). Missing on older nodes.
#[serde(rename = "ui-ready", default, skip_serializing_if = "Option::is_none")]
pub ui_ready: Option<bool>,
pub state: PackageState, pub state: PackageState,
/// Container health: "healthy", "unhealthy", "starting", or null /// Container health: "healthy", "unhealthy", "starting", or null
#[serde(skip_serializing_if = "Option::is_none")] #[serde(skip_serializing_if = "Option::is_none")]
@@ -297,6 +301,8 @@ pub enum InstallPhase {
/// `podman pull` in progress (the longest phase — up to several /// `podman pull` in progress (the longest phase — up to several
/// minutes for large images on slow networks). /// minutes for large images on slow networks).
PullingImage, PullingImage,
/// Orchestrator owns download/build and startup as one operation.
PreparingApp,
/// Creating data directories, writing app-specific configs /// Creating data directories, writing app-specific configs
/// (bitcoin.conf, lnd.conf, searxng settings.yml, chown). /// (bitcoin.conf, lnd.conf, searxng settings.yml, chown).
CreatingContainer, CreatingContainer,
+40 -3
View File
@@ -5,8 +5,45 @@
//! are reachable over the mesh; ports of apps that aren't installed have //! are reachable over the mesh; ports of apps that aren't installed have
//! no listener, so allowing them is inert. //! no listener, so allowing them is inert.
#[rustfmt::skip]
pub const APP_LAUNCH_PORTS: &[u16] = &[ pub const APP_LAUNCH_PORTS: &[u16] = &[
2283, 2342, 3000, 3001, 3002, 4080, 5180, 7778, 8080, 8081, 8082, 8083, 8084, 8085, 8087, 8090, 2283,
8096, 8123, 8175, 8176, 8187, 8240, 8334, 8336, 8337, 8888, 8999, 9000, 9100, 10380, 11434, 2342,
18081, 18083, 18091, 23000, 32838, 50002, 3000,
3001,
3002,
4080,
5180,
7778,
8080,
8081,
8082,
8083,
8084,
8085,
8087,
8090,
8091,
8096,
8123,
8175,
8176,
8187,
8240,
8334,
8336,
8337,
8888,
8998,
8999,
9000,
9100,
10380,
11434,
18081,
18083,
18091,
23000,
32838,
50002,
]; ];
+268 -7
View File
@@ -46,6 +46,25 @@ fn fips_should_fall_back(status: reqwest::StatusCode) -> bool {
status == reqwest::StatusCode::NOT_FOUND || status.is_server_error() status == reqwest::StatusCode::NOT_FOUND || status.is_server_error()
} }
/// Is this FIPS answer the final one, or should the request go again over
/// Tor? A single-delivery request already reached the peer, so any answer
/// is final: a Tor replay would carry the same (possibly spent) payload.
fn fips_answer_is_final(
pref: crate::settings::transport::TransportPref,
single_delivery: bool,
status: reqwest::StatusCode,
) -> bool {
pref == crate::settings::transport::TransportPref::Fips
|| single_delivery
|| !fips_should_fall_back(status)
}
/// May a failed FIPS attempt be sent again? Only a failed connect proves the
/// peer never saw it; a timeout can land after the request was delivered.
fn fips_retryable(single_delivery: bool, e: &reqwest::Error) -> bool {
e.is_connect() || (!single_delivery && e.is_timeout())
}
/// DNS suffix appended to a peer's bech32 npub. /// DNS suffix appended to a peer's bech32 npub.
pub const FIPS_DNS_SUFFIX: &str = "fips"; pub const FIPS_DNS_SUFFIX: &str = "fips";
@@ -113,7 +132,21 @@ pub fn client() -> reqwest::Client {
/// before the Tor fallback ever gets a chance. The generous `connect_timeout` /// before the Tor fallback ever gets a chance. The generous `connect_timeout`
/// is preserved so a cold hole-punched path still gets time to establish. /// is preserved so a cold hole-punched path still gets time to establish.
pub fn client_with_timeout(timeout: Duration) -> reqwest::Client { pub fn client_with_timeout(timeout: Duration) -> reqwest::Client {
client_with_delivery_policy(timeout, false)
}
fn delivery_redirect_policy(single: bool) -> reqwest::redirect::Policy {
if single {
reqwest::redirect::Policy::none()
} else {
reqwest::redirect::Policy::default()
}
}
fn client_with_delivery_policy(timeout: Duration, single: bool) -> reqwest::Client {
reqwest::Client::builder() reqwest::Client::builder()
.no_proxy()
.redirect(delivery_redirect_policy(single))
.timeout(timeout) .timeout(timeout)
.connect_timeout(Duration::from_secs(8)) .connect_timeout(Duration::from_secs(8))
.user_agent("archipelago-fips/1") .user_agent("archipelago-fips/1")
@@ -130,10 +163,18 @@ pub fn client_with_timeout(timeout: Duration) -> reqwest::Client {
/// robust". Only connect/timeout errors are retried (a real HTTP response, /// robust". Only connect/timeout errors are retried (a real HTTP response,
/// including 4xx/5xx, is returned as-is for the caller to interpret). /// including 4xx/5xx, is returned as-is for the caller to interpret).
async fn send_with_retry(rb: reqwest::RequestBuilder) -> Result<reqwest::Response, reqwest::Error> { async fn send_with_retry(rb: reqwest::RequestBuilder) -> Result<reqwest::Response, reqwest::Error> {
send_with_retry_if(rb, |e| e.is_connect() || e.is_timeout()).await
}
/// [`send_with_retry`], retrying only on errors `retryable` accepts.
async fn send_with_retry_if(
rb: reqwest::RequestBuilder,
retryable: impl Fn(&reqwest::Error) -> bool,
) -> Result<reqwest::Response, reqwest::Error> {
let retry = rb.try_clone(); let retry = rb.try_clone();
match rb.send().await { match rb.send().await {
Ok(resp) => Ok(resp), Ok(resp) => Ok(resp),
Err(e) if (e.is_connect() || e.is_timeout()) && retry.is_some() => { Err(e) if retryable(&e) && retry.is_some() => {
// Brief pause so the hole-punch packets from the first attempt can // Brief pause so the hole-punch packets from the first attempt can
// traverse before we re-dial onto the warmed path. // traverse before we re-dial onto the warmed path.
tokio::time::sleep(Duration::from_millis(600)).await; tokio::time::sleep(Duration::from_millis(600)).await;
@@ -350,6 +391,9 @@ pub struct PeerRequest<'a> {
/// the per-peer FIPS/Tor badge reflects reality. Opt-in because not /// the per-peer FIPS/Tor badge reflects reality. Opt-in because not
/// every caller has a data dir in scope. /// every caller has a data dir in scope.
pub record_data_dir: Option<std::path::PathBuf>, pub record_data_dir: Option<std::path::PathBuf>,
/// The request carries something that must reach the peer at most once
/// (a bearer ecash token). See [`PeerRequest::single_delivery`].
pub single_delivery: bool,
} }
impl<'a> PeerRequest<'a> { impl<'a> PeerRequest<'a> {
@@ -363,9 +407,25 @@ impl<'a> PeerRequest<'a> {
fips_timeout: None, fips_timeout: None,
service: None, service: None,
record_data_dir: None, record_data_dir: None,
single_delivery: false,
} }
} }
/// Never send this request twice. A paid download carries a bearer ecash
/// token that the seller redeems on first sight; replaying it over Tor
/// after FIPS already delivered it hands the seller a spent token, so the
/// buyer is charged and gets a 402 instead of the file (2026-09-29: FIPS
/// answered 404 after the seller redeemed, the Tor retry got 402).
///
/// With this set, whatever FIPS answers is final, the FIPS retry fires
/// only when the first attempt never connected, and Tor is used only when
/// FIPS could not have delivered the request. An attempt that may have
/// been delivered but timed out is an error, not a fallback.
pub fn single_delivery(mut self) -> Self {
self.single_delivery = true;
self
}
/// Record the transport that serves this request into federation storage /// Record the transport that serves this request into federation storage
/// (matched by this request's onion host). Best-effort, off the hot path. /// (matched by this request's onion host). Best-effort, off the hot path.
pub fn record_transport(mut self, data_dir: impl Into<std::path::PathBuf>) -> Self { pub fn record_transport(mut self, data_dir: impl Into<std::path::PathBuf>) -> Self {
@@ -442,7 +502,7 @@ impl<'a> PeerRequest<'a> {
// Use the FIPS reply unless it's one a Tor retry could // Use the FIPS reply unless it's one a Tor retry could
// fix (404 path-not-served / 5xx) and we're allowed to // fix (404 path-not-served / 5xx) and we're allowed to
// fall back. FIPS-only never falls back. // fall back. FIPS-only never falls back.
if pref == TransportPref::Fips || !fips_should_fall_back(resp.status()) { if fips_answer_is_final(pref, self.single_delivery, resp.status()) {
telemetry::record_fips_ok(); telemetry::record_fips_ok();
self.spawn_record(crate::transport::TransportKind::Fips); self.spawn_record(crate::transport::TransportKind::Fips);
return Ok((resp, crate::transport::TransportKind::Fips)); return Ok((resp, crate::transport::TransportKind::Fips));
@@ -481,7 +541,7 @@ impl<'a> PeerRequest<'a> {
if matches!(pref, TransportPref::Auto | TransportPref::Fips) { if matches!(pref, TransportPref::Auto | TransportPref::Fips) {
match self.try_fips_get().await? { match self.try_fips_get().await? {
Some(resp) => { Some(resp) => {
if pref == TransportPref::Fips || !fips_should_fall_back(resp.status()) { if fips_answer_is_final(pref, self.single_delivery, resp.status()) {
telemetry::record_fips_ok(); telemetry::record_fips_ok();
self.spawn_record(crate::transport::TransportKind::Fips); self.spawn_record(crate::transport::TransportKind::Fips);
return Ok((resp, crate::transport::TransportKind::Fips)); return Ok((resp, crate::transport::TransportKind::Fips));
@@ -551,13 +611,21 @@ impl<'a> PeerRequest<'a> {
} else { } else {
budget budget
}; };
let c = client_with_timeout(per_attempt); let c = client_with_delivery_policy(per_attempt, self.single_delivery);
let mut rb = c.post(&url).json(body); let mut rb = c.post(&url).json(body);
for (k, v) in &self.headers { for (k, v) in &self.headers {
rb = rb.header(*k, v); rb = rb.header(*k, v);
} }
match tokio::time::timeout(budget, send_with_retry(rb)).await { let single = self.single_delivery;
let attempt = send_with_retry_if(rb, |e| fips_retryable(single, e));
match tokio::time::timeout(budget, attempt).await {
Ok(Ok(r)) => Ok(Some(r)), Ok(Ok(r)) => Ok(Some(r)),
Ok(Err(e)) if single && !e.is_connect() => Err(anyhow::anyhow!(
"FIPS POST failed after possible delivery; not replaying: {e}"
)),
Err(_) if single => Err(anyhow::anyhow!(
"FIPS POST exceeded its budget after possible delivery; not replaying"
)),
Ok(Err(e)) => { Ok(Err(e)) => {
telemetry::record_fallback(FallbackReason::ConnectFail); telemetry::record_fallback(FallbackReason::ConnectFail);
tracing::info!( tracing::info!(
@@ -612,13 +680,28 @@ impl<'a> PeerRequest<'a> {
} else { } else {
budget budget
}; };
let c = client_with_timeout(per_attempt); let c = client_with_delivery_policy(per_attempt, self.single_delivery);
let mut rb = c.get(&url); let mut rb = c.get(&url);
for (k, v) in &self.headers { for (k, v) in &self.headers {
rb = rb.header(*k, v); rb = rb.header(*k, v);
} }
match tokio::time::timeout(budget, send_with_retry(rb)).await { let single = self.single_delivery;
let attempt = send_with_retry_if(rb, |e| fips_retryable(single, e));
match tokio::time::timeout(budget, attempt).await {
Ok(Ok(r)) => Ok(Some(r)), Ok(Ok(r)) => Ok(Some(r)),
// Anything but a failed connect may have reached the peer.
Ok(Err(e)) if single && !e.is_connect() => Err(anyhow::anyhow!(
"FIPS GET {} failed after the request may have been delivered \
(not retrying over Tor): {}",
self.path,
e
)),
Err(_) if single => Err(anyhow::anyhow!(
"FIPS GET {} exceeded its {:?} budget after the request may have \
been delivered (not retrying over Tor)",
self.path,
budget
)),
Ok(Err(e)) => { Ok(Err(e)) => {
telemetry::record_fallback(FallbackReason::ConnectFail); telemetry::record_fallback(FallbackReason::ConnectFail);
tracing::info!( tracing::info!(
@@ -676,6 +759,7 @@ impl<'a> PeerRequest<'a> {
.context("Invalid Tor SOCKS proxy URL")?; .context("Invalid Tor SOCKS proxy URL")?;
reqwest::Client::builder() reqwest::Client::builder()
.proxy(proxy) .proxy(proxy)
.redirect(delivery_redirect_policy(self.single_delivery))
.timeout(self.timeout) .timeout(self.timeout)
.build() .build()
.context("Build Tor HTTP client") .context("Build Tor HTTP client")
@@ -759,4 +843,181 @@ mod tests {
let err = decode_response(0xAABB, &r, "x").unwrap_err(); let err = decode_response(0xAABB, &r, "x").unwrap_err();
assert!(err.to_string().contains("no AAAA")); assert!(err.to_string().contains("no AAAA"));
} }
#[test]
fn a_single_delivery_answer_is_final_whatever_its_status() {
use crate::settings::transport::TransportPref;
use reqwest::StatusCode;
// Regression (2026-09-29): the seller redeemed a paid download's
// token, answered 404, and the Tor fallback replayed the spent token.
for status in [
StatusCode::NOT_FOUND,
StatusCode::INTERNAL_SERVER_ERROR,
StatusCode::SERVICE_UNAVAILABLE,
StatusCode::OK,
] {
assert!(fips_answer_is_final(TransportPref::Auto, true, status));
}
// Everything else keeps the existing fallback rules.
assert!(!fips_answer_is_final(
TransportPref::Auto,
false,
StatusCode::NOT_FOUND
));
assert!(!fips_answer_is_final(
TransportPref::Auto,
false,
StatusCode::BAD_GATEWAY
));
assert!(fips_answer_is_final(
TransportPref::Auto,
false,
StatusCode::PAYMENT_REQUIRED
));
assert!(fips_answer_is_final(
TransportPref::Fips,
false,
StatusCode::NOT_FOUND
));
}
/// A listener that accepts connections and never answers, counting them.
async fn silent_peer() -> (String, std::sync::Arc<std::sync::atomic::AtomicUsize>) {
let listener = tokio::net::TcpListener::bind("127.0.0.1:0").await.unwrap();
let addr = listener.local_addr().unwrap();
let seen = std::sync::Arc::new(std::sync::atomic::AtomicUsize::new(0));
let counter = seen.clone();
tokio::spawn(async move {
let mut held = Vec::new();
while let Ok((stream, _)) = listener.accept().await {
counter.fetch_add(1, std::sync::atomic::Ordering::SeqCst);
held.push(stream); // keep it open, never reply
}
});
(format!("http://{addr}/content/x"), seen)
}
#[tokio::test]
async fn a_single_delivery_request_is_not_resent_after_a_timeout() {
let (url, seen) = silent_peer().await;
let c = client_with_timeout(Duration::from_millis(300));
let err = send_with_retry_if(c.get(&url), |e| fips_retryable(true, e))
.await
.expect_err("peer never answers");
assert!(err.is_timeout());
assert_eq!(seen.load(std::sync::atomic::Ordering::SeqCst), 1);
}
#[tokio::test]
async fn an_ordinary_request_is_still_retried_once_after_a_timeout() {
let (url, seen) = silent_peer().await;
let c = client_with_timeout(Duration::from_millis(300));
let _ = send_with_retry_if(c.get(&url), |e| fips_retryable(false, e)).await;
assert_eq!(seen.load(std::sync::atomic::Ordering::SeqCst), 2);
}
#[tokio::test]
async fn a_single_delivery_request_still_retries_a_refused_connect() {
// Nothing listening: the peer provably never saw the request.
let listener = std::net::TcpListener::bind("127.0.0.1:0").unwrap();
let addr = listener.local_addr().unwrap();
drop(listener);
let c = client_with_timeout(Duration::from_millis(500));
let err = send_with_retry_if(c.get(format!("http://{addr}/")), |e| {
fips_retryable(true, e)
})
.await
.expect_err("nothing listening");
assert!(err.is_connect());
assert!(fips_retryable(true, &err));
}
}
#[cfg(test)]
mod delivery_redirect_tests {
use super::*;
use hyper::{
service::{make_service_fn, service_fn},
Body, Response, Server,
};
use std::{
convert::Infallible,
sync::{
atomic::{AtomicUsize, Ordering},
Arc,
},
};
#[tokio::test]
async fn paid_bearer_request_does_not_follow_redirects_but_normal_get_does() {
let seen = Arc::new(AtomicUsize::new(0));
let counter = seen.clone();
let server = Server::bind(&([127, 0, 0, 1], 0).into());
let address = server.local_addr();
let service = make_service_fn(move |_| {
let counter = counter.clone();
async move {
Ok::<_, Infallible>(service_fn(move |request: hyper::Request<Body>| {
let counter = counter.clone();
async move {
counter.fetch_add(1, Ordering::SeqCst);
let response = if request.uri().path() == "/first" {
Response::builder()
.status(302)
.header("Location", "/replay")
.body(Body::empty())
.unwrap()
} else {
Response::new(Body::from("replayed"))
};
Ok::<_, Infallible>(response)
}
}))
}
});
let task = tokio::spawn(server.serve(service));
let url = format!("http://{address}/first");
let response = client_with_delivery_policy(Duration::from_secs(2), true)
.get(&url)
.header("X-Payment-Token", "dummy-test-token")
.send()
.await
.unwrap();
assert_eq!(response.status(), reqwest::StatusCode::FOUND);
assert_eq!(seen.load(Ordering::SeqCst), 1);
let response = client_with_delivery_policy(Duration::from_secs(2), false)
.get(url)
.send()
.await
.unwrap();
assert_eq!(response.status(), reqwest::StatusCode::OK);
assert_eq!(seen.load(Ordering::SeqCst), 3);
task.abort();
}
#[tokio::test]
async fn paid_request_is_not_resent_when_peer_disconnects_after_reading_it() {
use tokio::io::AsyncReadExt;
let listener = tokio::net::TcpListener::bind("127.0.0.1:0").await.unwrap();
let address = listener.local_addr().unwrap();
let seen = Arc::new(AtomicUsize::new(0));
let counter = seen.clone();
let task = tokio::spawn(async move {
while let Ok((mut stream, _)) = listener.accept().await {
let mut buf = [0; 4096];
let _ = stream.read(&mut buf).await;
counter.fetch_add(1, Ordering::SeqCst);
drop(stream);
}
});
let c = client_with_delivery_policy(Duration::from_secs(2), true);
let error = send_with_retry_if(c.get(format!("http://{address}/")), |e| {
fips_retryable(true, e)
})
.await
.unwrap_err();
assert!(!error.is_connect());
assert_eq!(seen.load(Ordering::SeqCst), 1);
task.abort();
}
} }
+64 -7
View File
@@ -501,12 +501,12 @@ async fn check_containers() -> Vec<ContainerHealth> {
out out
} }
fn host_tcp_ports_from_container(c: &serde_json::Value) -> Vec<u16> { fn host_tcp_ports_from_container(c: &serde_json::Value) -> Vec<std::net::SocketAddr> {
let Some(ports) = c.get("Ports").and_then(|v| v.as_array()) else { let Some(ports) = c.get("Ports").and_then(|v| v.as_array()) else {
return Vec::new(); return Vec::new();
}; };
let mut out: Vec<u16> = ports let mut out: Vec<std::net::SocketAddr> = ports
.iter() .iter()
.filter(|p| { .filter(|p| {
p.get("protocol") p.get("protocol")
@@ -515,9 +515,19 @@ fn host_tcp_ports_from_container(c: &serde_json::Value) -> Vec<u16> {
.eq_ignore_ascii_case("tcp") .eq_ignore_ascii_case("tcp")
}) })
.filter_map(|p| { .filter_map(|p| {
p.get("host_port") let port = p.get("host_port")?.as_u64()?;
.and_then(|v| v.as_u64()) let port = u16::try_from(port).ok().filter(|port| *port != 0)?;
.and_then(|port| u16::try_from(port).ok()) let bind = p.get("host_ip").and_then(|v| v.as_str()).unwrap_or("");
// Wildcard listeners are reachable through the corresponding
// loopback family. Explicit binds must be probed at that address:
// probing a WireGuard-only port on 127.0.0.1 creates false failures
// and endlessly restarts an otherwise healthy app.
let address: std::net::IpAddr = match bind {
"" | "0.0.0.0" => "127.0.0.1".parse().ok()?,
"::" => "::1".parse().ok()?,
explicit => explicit.parse().ok()?,
};
Some(std::net::SocketAddr::new(address, port))
}) })
.collect(); .collect();
out.sort_unstable(); out.sort_unstable();
@@ -525,11 +535,11 @@ fn host_tcp_ports_from_container(c: &serde_json::Value) -> Vec<u16> {
out out
} }
async fn host_ports_ready(ports: &[u16]) -> bool { async fn host_ports_ready(ports: &[std::net::SocketAddr]) -> bool {
for port in ports { for port in ports {
let ready = tokio::time::timeout( let ready = tokio::time::timeout(
std::time::Duration::from_secs(2), std::time::Duration::from_secs(2),
tokio::net::TcpStream::connect(("127.0.0.1", *port)), tokio::net::TcpStream::connect(*port),
) )
.await .await
.is_ok_and(|r| r.is_ok()); .is_ok_and(|r| r.is_ok());
@@ -1662,4 +1672,51 @@ mod tests {
"Prefetcher:catching up to daemon height 953,480" "Prefetcher:catching up to daemon height 953,480"
)); ));
} }
#[test]
fn published_port_probes_preserve_explicit_bind_addresses() {
let c = serde_json::json!({"Ports": [
{"host_ip":"127.0.0.1","host_port":8081,"protocol":"tcp"},
{"host_ip":"10.77.0.2","host_port":18081,"protocol":"tcp"},
{"host_ip":"10.77.0.2","host_port":18443,"protocol":"tcp"},
{"host_ip":"::1","host_port":8082,"protocol":"tcp"}
]});
let targets = host_tcp_ports_from_container(&c);
for target in [
"127.0.0.1:8081",
"10.77.0.2:18081",
"10.77.0.2:18443",
"[::1]:8082",
] {
assert!(targets.contains(&target.parse().unwrap()));
}
assert!(!targets.contains(&"127.0.0.1:18081".parse().unwrap()));
}
#[test]
fn published_port_probes_normalize_wildcards_and_ignore_invalid_entries() {
let c = serde_json::json!({"Ports": [
{"host_ip":"0.0.0.0","host_port":8080},
{"host_ip":"","host_port":8080},
{"host_ip":"::","host_port":8080},
{"host_ip":"10.0.0.1","host_port":53,"protocol":"udp"},
{"host_ip":"bad","host_port":8080},
{"host_port":0}, {"host_port":65536}, {"container_port":80}
]});
let targets = host_tcp_ports_from_container(&c);
assert_eq!(targets.len(), 2);
assert!(targets.contains(&"127.0.0.1:8080".parse().unwrap()));
assert!(targets.contains(&"[::1]:8080".parse().unwrap()));
}
#[tokio::test]
async fn health_probe_reaches_non_default_loopback_and_detects_closed_port() {
let listener = tokio::net::TcpListener::bind("127.0.0.2:0").await.unwrap();
let address = listener.local_addr().unwrap();
assert!(host_ports_ready(&[address]).await);
// Same port, wrong local address reproduces the former false failure.
let wrong = std::net::SocketAddr::new("127.0.0.1".parse().unwrap(), address.port());
assert!(!host_ports_ready(&[wrong]).await);
drop(listener);
assert!(!host_ports_ready(&[address]).await);
}
} }
+209
View File
@@ -115,6 +115,30 @@ fn relay_url_matches(a: &str, b: &str) -> bool {
norm(a) == norm(b) norm(a) == norm(b)
} }
/// True when `record` is the node's own identity: the one whose ed25519 key
/// is the node key (`server_info.pubkey`). `identity.list` reports this as
/// `is_node`, and clients must never offer it as an app signer.
pub fn is_node_identity(record: &IdentityRecord, node_pubkey_hex: &str) -> bool {
!node_pubkey_hex.is_empty() && record.pubkey_hex == node_pubkey_hex
}
/// True when the app identity picker hides `record`, mirroring
/// `NostrIdentityPicker.vue`'s filter exactly: the node identity
/// (`is_node`), any `node-*` id and any identity named "Node".
pub(crate) fn is_hidden_from_app_signer(record: &IdentityRecord, node_pubkey_hex: &str) -> bool {
// Match ECMAScript trim exactly: Rust includes U+0085 and excludes U+FEFF.
let trim_js = |value: &str| value.trim_matches(is_js_whitespace).to_lowercase();
is_node_identity(record, node_pubkey_hex)
|| trim_js(&record.id).starts_with("node-")
|| trim_js(&record.name) == "node"
}
fn is_js_whitespace(c: char) -> bool {
matches!(c, '\u{0009}'..='\u{000D}' | '\u{0020}' | '\u{00A0}' | '\u{1680}'
| '\u{2000}'..='\u{200A}' | '\u{2028}' | '\u{2029}' | '\u{202F}'
| '\u{205F}' | '\u{3000}' | '\u{FEFF}')
}
impl IdentityManager { impl IdentityManager {
pub async fn new(data_dir: &Path) -> Result<Self> { pub async fn new(data_dir: &Path) -> Result<Self> {
let identities_dir = data_dir.join(IDENTITIES_DIR); let identities_dir = data_dir.join(IDENTITIES_DIR);
@@ -150,6 +174,30 @@ impl IdentityManager {
Ok((identities, default_id)) Ok((identities, default_id))
} }
/// Nostr public keys of the identities an app may sign with through the
/// NIP-07 bridge, as sorted, de-duplicated, comma-joined lowercase hex.
///
/// Leaves out what the identity picker hides (`is_hidden_from_app_signer`)
/// and identities without a Nostr key (they cannot sign). Empty when no
/// identity qualifies.
pub async fn app_signable_nostr_pubkeys(&self, node_pubkey_hex: &str) -> Result<String> {
let (identities, _) = self.list().await?;
let mut pubkeys: Vec<String> = identities
.iter()
.filter(|r| !is_hidden_from_app_signer(r, node_pubkey_hex))
.filter_map(|r| r.nostr_pubkey.as_deref())
.map(|key| {
anyhow::ensure!(key.len() == 64, "invalid app owner Nostr public key");
nostr_sdk::PublicKey::from_hex(key)
.map(|key| key.to_hex())
.context("invalid app owner Nostr public key")
})
.collect::<Result<Vec<_>>>()?;
pubkeys.sort();
pubkeys.dedup();
Ok(pubkeys.join(","))
}
/// Create a new identity. /// Create a new identity.
pub async fn create(&self, name: String, purpose: IdentityPurpose) -> Result<IdentityRecord> { pub async fn create(&self, name: String, purpose: IdentityPurpose) -> Result<IdentityRecord> {
let signing_key = SigningKey::generate(&mut OsRng); let signing_key = SigningKey::generate(&mut OsRng);
@@ -966,6 +1014,167 @@ mod tests {
assert_ne!(default_id, Some(r1.id)); assert_ne!(default_id, Some(r1.id));
} }
fn record(id: &str, name: &str, pubkey_hex: &str) -> IdentityRecord {
IdentityRecord {
id: id.to_string(),
name: name.to_string(),
purpose: IdentityPurpose::Personal,
pubkey_hex: pubkey_hex.to_string(),
did: String::new(),
dht_did: None,
created_at: String::new(),
nostr_pubkey: None,
nostr_npub: None,
profile: None,
}
}
#[test]
fn is_node_identity_matches_only_the_node_pubkey() {
let node = "ab".repeat(32);
let other = "cd".repeat(32);
assert!(is_node_identity(&record("uuid-1", "Laptop", &node), &node));
assert!(!is_node_identity(
&record("uuid-1", "Laptop", &other),
&node
));
// An unknown node key matches nothing, not the records without a key.
assert!(!is_node_identity(&record("uuid-1", "Laptop", ""), ""));
// The id and name rules belong to the picker filter, not to `is_node`.
assert!(!is_node_identity(
&record("node-abc", "Node", &other),
&node
));
}
#[test]
fn is_hidden_from_app_signer_mirrors_the_picker_rules() {
let node = "ab".repeat(32);
let other = "cd".repeat(32);
let hidden = |id: &str, name: &str, pk: &str| {
is_hidden_from_app_signer(&record(id, name, pk), &node)
};
// is_node: matched by key alone, whatever the id and name.
assert!(hidden("uuid-1", "Laptop", &node));
// node-* id, any case, surrounding whitespace ignored.
assert!(hidden("node-0123456789abcdef", "Laptop", &other));
assert!(hidden(" NODE-x ", "Laptop", &other));
assert!(hidden("Node-x", "Laptop", &other));
// The name "Node", any case, surrounding whitespace ignored.
assert!(hidden("uuid-1", "Node", &other));
assert!(hidden("uuid-1", " nODe\t", &other));
assert!(hidden("\u{FEFF}NODE-x\u{FEFF}", "Laptop", &other));
assert!(hidden("uuid-1", "\u{FEFF}Node\u{FEFF}", &other));
// ECMAScript keeps U+0085; do not add owners that the picker hides,
// or hide identities that it offers.
assert!(!hidden("uuid-1", "\u{0085}Node\u{0085}", &other));
// Near misses stay visible.
assert!(!hidden("uuid-1", "Laptop", &other));
assert!(!hidden("my-node-1", "Node 2", &other));
assert!(!hidden("nodes", "Nodes", &other));
}
#[tokio::test]
async fn app_signable_nostr_pubkeys_mirror_the_identity_picker() {
let dir = tempdir().unwrap();
let mgr = IdentityManager::new(dir.path()).await.unwrap();
let personal = mgr
.create("Personal".to_string(), IdentityPurpose::Personal)
.await
.unwrap();
let business = mgr
.create("Business".to_string(), IdentityPurpose::Business)
.await
.unwrap();
// The node identity as mirrored at startup: a `node-` id named
// "Node", given a Nostr key so only the id and name rules hide it.
let mirrored_key = SigningKey::generate(&mut OsRng);
let mirrored = mgr
.create_from_signing_key("Node".to_string(), IdentityPurpose::Personal, mirrored_key)
.await
.unwrap();
assert!(mirrored.id.starts_with("node-"));
mgr.create_nostr_key(&mirrored.id).await.unwrap();
// A user-created identity named "Node" is hidden by the picker too.
let named_node = mgr
.create(" node ".to_string(), IdentityPurpose::Anonymous)
.await
.unwrap();
// The node key belongs to an identity with a uuid id and an ordinary
// name, so only the `is_node` match can hide it.
let laptop = mgr
.create("Laptop".to_string(), IdentityPurpose::Personal)
.await
.unwrap();
assert!(!laptop.id.starts_with("node-"));
let (all, _) = mgr.list().await.unwrap();
assert!(all
.iter()
.any(|r| r.id == mirrored.id && r.nostr_pubkey.is_some()));
assert!(all.iter().any(|r| r.id == named_node.id));
assert!(all
.iter()
.any(|r| r.id == laptop.id && r.nostr_pubkey.is_some()));
let mut expected = vec![
personal.nostr_pubkey.unwrap().to_ascii_lowercase(),
business.nostr_pubkey.unwrap().to_ascii_lowercase(),
];
expected.sort();
assert_eq!(
mgr.app_signable_nostr_pubkeys(&laptop.pubkey_hex)
.await
.unwrap(),
expected.join(",")
);
// Without the node key, the same identity is offered like any other.
let mut with_laptop = expected.clone();
with_laptop.push(laptop.nostr_pubkey.unwrap().to_ascii_lowercase());
with_laptop.sort();
assert_eq!(
mgr.app_signable_nostr_pubkeys("").await.unwrap(),
with_laptop.join(",")
);
}
#[tokio::test]
async fn app_owner_pubkeys_reject_malformed_key_material() {
let dir = tempdir().unwrap();
let mgr = IdentityManager::new(dir.path()).await.unwrap();
let identity = mgr
.create("Owner".into(), IdentityPurpose::Personal)
.await
.unwrap();
let path = mgr.identities_dir.join(format!("{}.json", identity.id));
let original = fs::read(&path).await.unwrap();
for invalid in ["", "not-a-key", "owner,another-owner", "\nINJECTED=true"] {
let mut data: serde_json::Value = serde_json::from_slice(&original).unwrap();
data["nostr_pubkey_hex"] = serde_json::json!(invalid);
fs::write(&path, serde_json::to_vec(&data).unwrap())
.await
.unwrap();
assert!(mgr.app_signable_nostr_pubkeys("").await.is_err());
}
}
#[tokio::test]
async fn app_signable_nostr_pubkeys_is_empty_with_only_the_node_identity() {
let dir = tempdir().unwrap();
let mgr = IdentityManager::new(dir.path()).await.unwrap();
let node_key = SigningKey::generate(&mut OsRng);
let node_pubkey_hex = hex::encode(node_key.verifying_key().as_bytes());
mgr.create_from_signing_key("Node".to_string(), IdentityPurpose::Personal, node_key)
.await
.unwrap();
assert_eq!(
mgr.app_signable_nostr_pubkeys(&node_pubkey_hex)
.await
.unwrap(),
""
);
}
#[tokio::test] #[tokio::test]
async fn test_delete_default_shifts() { async fn test_delete_default_shifts() {
let dir = tempdir().unwrap(); let dir = tempdir().unwrap();
+6
View File
@@ -70,6 +70,8 @@ mod node_message;
mod nostr_discovery; mod nostr_discovery;
mod nostr_handshake; mod nostr_handshake;
mod nostr_relays; mod nostr_relays;
#[cfg(test)]
mod nostr_security_tests;
mod peers; mod peers;
mod port_allocator; mod port_allocator;
mod rate_limit; mod rate_limit;
@@ -256,6 +258,10 @@ async fn main() -> Result<()> {
boot_report.recovered, boot_report.total, boot_report.failed boot_report.recovered, boot_report.total, boot_report.failed
); );
} }
// Disk manifests must be stable before the initial load and all later
// catalog reloads. Do not move this into the background doctor bootstrap.
bootstrap::ensure_runtime_assets_ready().await;
// Construct the container orchestrator once. In prod mode we load the // Construct the container orchestrator once. In prod mode we load the
// on-disk app manifests, do an initial adoption pass, and spawn the // on-disk app manifests, do an initial adoption pass, and spawn the
// BootReconciler loop (Step 5/6 of the rust-orchestrator migration). // BootReconciler loop (Step 5/6 of the rust-orchestrator migration).
+395 -69
View File
@@ -7,7 +7,8 @@
//! to a full chip erase before write. //! to a full chip erase before write.
//! //!
//! MeshCore and Meshtastic are flashed the same way: download a released //! MeshCore and Meshtastic are flashed the same way: download a released
//! image, `esptool erase_flash`, then `esptool write_flash 0x0 <image>`. //! image, verify it, then run `write_flash --erase-all 0x0 <image>` with
//! the packaged esptool executable.
//! Reticulum/RNode is different: `archy-rnodeconf --autoinstall` owns the //! Reticulum/RNode is different: `archy-rnodeconf --autoinstall` owns the
//! whole fetch+erase+flash+EEPROM-bootstrap sequence itself (confirmed live //! whole fetch+erase+flash+EEPROM-bootstrap sequence itself (confirmed live
//! via `archy-rnodeconf --help` — there is no raw esptool path exposed for //! via `archy-rnodeconf --help` — there is no raw esptool path exposed for
@@ -49,32 +50,18 @@ impl FlashBoard {
} }
} }
/// Map a detected USB vid:pid to a known flashable board, using the same /// Generic CP2102 and native ESP32-S3 USB IDs identify adapters/chips, not
/// table as `image-recipe/configs/99-mesh-radio.rules`. CP2102 (10c4:ea60) /// board wiring. Require an explicit board until a board-specific identity is
/// is confirmed there as Heltec V3's USB-UART bridge chip, and is safe to /// available; guessing a Heltec model can write incompatible firmware.
/// auto-match since that vid:pid is bridge-chip-specific. pub fn resolve_flash_board(_info: &DetectedDeviceInfo) -> Option<FlashBoard> {
/// None
/// Heltec V4 is NOT auto-matchable and deliberately has no entry here: it
/// was confirmed live (real hardware, 2026-07-23) to use the ESP32-S3's
/// built-in native-USB JTAG/serial peripheral, reporting vid:pid 303a:1001
/// with product string "USB JTAG/serial debug unit" — that descriptor is
/// baked into the chip's ROM and is IDENTICAL across every ESP32-S3 board
/// with native USB enabled, not just Heltec V4. Adding `303a:1001 =>
/// HeltecV4` here would silently misidentify any other native-USB ESP32-S3
/// board (a T3-S3, a bare devkit, etc.) as a V4 and risk writing the wrong
/// board's image. Callers (the RPC layer / frontend) must let the user pick
/// the board manually whenever this returns `None`.
pub fn resolve_flash_board(info: &DetectedDeviceInfo) -> Option<FlashBoard> {
match (info.vid.as_deref(), info.pid.as_deref()) {
(Some("10c4"), Some("ea60")) => Some(FlashBoard::HeltecV3),
_ => None,
}
} }
#[derive(Debug, Clone, Copy, PartialEq, Eq, Serialize)] #[derive(Debug, Clone, Copy, PartialEq, Eq, Serialize)]
#[serde(rename_all = "lowercase")] #[serde(rename_all = "lowercase")]
pub enum FlashStage { pub enum FlashStage {
Downloading, Downloading,
Preparing,
Erasing, Erasing,
Writing, Writing,
Autoinstalling, Autoinstalling,
@@ -101,11 +88,10 @@ const LOG_TAIL_MAX: usize = 200;
/// start opening the port (which itself toggles DTR/RTS) again. /// start opening the port (which itself toggles DTR/RTS) again.
const POST_FLASH_SETTLE_DELAY: std::time::Duration = std::time::Duration::from_secs(5); const POST_FLASH_SETTLE_DELAY: std::time::Duration = std::time::Duration::from_secs(5);
/// Absolute ceiling on a whole flash job (download + erase + write, or /// Deadline for preparation and warning threshold for the active flasher.
/// autoinstall), regardless of what it's doing internally. Last-resort /// A download can be cancelled safely. An active write retains ownership until
/// safety net so a hang anywhere can't wedge the single-flash-job guard /// its subprocess exits, even after this threshold: reporting an aborted job
/// forever — generous enough to never trigger on a legitimately slow /// while a detached writer continues would let a retry corrupt the device.
/// multi-hundred-MB transfer.
const MAX_JOB_DURATION: std::time::Duration = std::time::Duration::from_secs(15 * 60); const MAX_JOB_DURATION: std::time::Duration = std::time::Duration::from_secs(15 * 60);
/// How long to wait for MeshService::stop() to release the serial port /// How long to wait for MeshService::stop() to release the serial port
@@ -247,6 +233,16 @@ fn firmware_cache_dir(data_dir: &Path) -> PathBuf {
data_dir.join("mesh").join("firmware-cache") data_dir.join("mesh").join("firmware-cache")
} }
async fn invalidate_radio_settings_marker(data_dir: &Path) -> Result<()> {
// A full-chip flash erased the device's preferences. A previous host-side
// marker is no longer evidence that this radio has the requested settings.
match tokio::fs::remove_file(data_dir.join("meshcore-radio-params.json")).await {
Ok(()) => Ok(()),
Err(error) if error.kind() == std::io::ErrorKind::NotFound => Ok(()),
Err(error) => Err(error).context("Firmware written, but old radio-settings marker could not be cleared; reconnect is paused"),
}
}
/// No blanket `.timeout()` here on purpose: reqwest's request timeout covers /// No blanket `.timeout()` here on purpose: reqwest's request timeout covers
/// the *entire* request including streaming the response body, which would /// the *entire* request including streaming the response body, which would
/// kill a legitimate large download partway through (Meshtastic's esp32s3 /// kill a legitimate large download partway through (Meshtastic's esp32s3
@@ -314,6 +310,10 @@ pub async fn list_firmware(family: DeviceType) -> Result<Vec<String>> {
struct GithubAsset { struct GithubAsset {
name: String, name: String,
browser_download_url: String, browser_download_url: String,
#[serde(default)]
size: Option<u64>,
#[serde(default)]
digest: Option<String>,
} }
#[derive(serde::Deserialize)] #[derive(serde::Deserialize)]
@@ -322,8 +322,24 @@ struct GithubRelease {
assets: Vec<GithubAsset>, assets: Vec<GithubAsset>,
} }
async fn register_flash_job(handle: &FlashJobHandle, job: &Arc<FlashJob>) -> Result<()> {
// Keep checking and registering under one lock: concurrent RPCs must
// never start two writers on the same device.
let mut existing = handle.try_write().map_err(|_| anyhow::anyhow!(
"The radio is being inspected or reconfigured; wait for that operation to finish before flashing"
))?;
if let Some(current) = existing.as_ref() {
if !current.snapshot().await.done {
anyhow::bail!("A firmware flash is already in progress on this node");
}
}
*existing = Some(Arc::clone(job));
Ok(())
}
/// Start a flash job in the background. Returns as soon as the job has been /// Start a flash job in the background. Returns as soon as the job has been
/// registered and the listener released — callers poll `FlashJobHandle` via /// registered — preparation and listener shutdown run in the background.
/// Callers poll `FlashJobHandle` via
/// `mesh.flash-status` for progress. Only one job may be in flight at a time. /// `mesh.flash-status` for progress. Only one job may be in flight at a time.
pub async fn start_flash_job( pub async fn start_flash_job(
handle: &FlashJobHandle, handle: &FlashJobHandle,
@@ -333,21 +349,44 @@ pub async fn start_flash_job(
board: FlashBoard, board: FlashBoard,
family: DeviceType, family: DeviceType,
) -> Result<()> { ) -> Result<()> {
{ // Missing/corrupt tooling must fail before stopping a working radio or
let existing = handle.read().await; // registering a job that can never reach the serial device.
if let Some(job) = existing.as_ref() { if matches!(family, DeviceType::Meshtastic | DeviceType::Meshcore) {
if !job.snapshot().await.done { preflight_esptool().await?;
anyhow::bail!("A firmware flash is already in progress on this node");
}
}
} }
let job = FlashJob::new(board, family, path.clone()); let job = FlashJob::new(board, family, path.clone());
*handle.write().await = Some(Arc::clone(&job)); register_flash_job(handle, &job).await?;
let bg_job = Arc::clone(&job); let bg_job = Arc::clone(&job);
let bg_service = Arc::clone(mesh_service); let bg_service = Arc::clone(mesh_service);
let task = tokio::spawn(async move { let task = tokio::spawn(async move {
// Downloads and checksum checks do not need exclusive serial access.
// Keep a working listener alive if upstream/download verification fails.
let prepared_image = if matches!(family, DeviceType::Meshcore | DeviceType::Meshtastic) {
match tokio::time::timeout(
MAX_JOB_DURATION,
fetch_esptool_image(board, family, &data_dir, &bg_job),
)
.await
{
Ok(Ok(image)) => Some(image),
result => {
let error = match result {
Ok(Err(error)) => error,
_ => anyhow::anyhow!(
"Firmware download exceeded the time limit; radio was not changed"
),
};
bg_job.fail(&error).await;
return;
}
}
} else {
None
};
// Cancellation is safe during download. Once listener shutdown starts,
// allow that bounded operation to finish instead of orphaning its task.
bg_job.set_stage(FlashStage::Preparing).await;
// esptool/archy-rnodeconf need exclusive serial access — release // esptool/archy-rnodeconf need exclusive serial access — release
// the listener's hold on the port before touching it. This USED // the listener's hold on the port before touching it. This USED
// TO run synchronously in start_flash_job before the job was even // TO run synchronously in start_flash_job before the job was even
@@ -404,17 +443,31 @@ pub async fn start_flash_job(
// subsequent mesh.flash-device call failed with "already in // subsequent mesh.flash-device call failed with "already in
// progress" until the service was restarted). Generous enough that // progress" until the service was restarted). Generous enough that
// a legitimately slow multi-hundred-MB transfer still completes. // a legitimately slow multi-hundred-MB transfer still completes.
let result = match tokio::time::timeout( let flash = run_flash(
MAX_JOB_DURATION, board,
run_flash(board, family, &data_dir, &path, &bg_job), family,
) &data_dir,
.await &path,
{ prepared_image.as_deref(),
&bg_job,
);
tokio::pin!(flash);
let result = match tokio::time::timeout(MAX_JOB_DURATION, &mut flash).await {
Ok(inner) => inner, Ok(inner) => inner,
Err(_) => Err(anyhow::anyhow!( Err(_) if bg_job.snapshot().await.stage == FlashStage::Downloading => Err(
"Flash job exceeded the {}-minute ceiling — aborted", anyhow::anyhow!("Firmware download exceeded the time limit; radio was not written"),
MAX_JOB_DURATION.as_secs() / 60 ),
)), Err(_) => {
// Dropping this future does NOT stop its flash subprocess.
// Keep the job busy until it exits, rather than allowing a
// second writer while the first one still owns the device.
bg_job.push_log("Flashing is taking longer than expected; waiting for the active tool to exit before allowing another operation").await;
flash.await
}
};
let result = match result {
Ok(()) => invalidate_radio_settings_marker(&data_dir).await,
error => error,
}; };
let succeeded = result.is_ok(); let succeeded = result.is_ok();
@@ -423,7 +476,6 @@ pub async fn start_flash_job(
bg_job bg_job
.push_log("Flash completed successfully".to_string()) .push_log("Flash completed successfully".to_string())
.await; .await;
bg_job.finish().await;
info!(path = %path, board = ?board, family = %family, "LoRa firmware flash succeeded"); info!(path = %path, board = ?board, family = %family, "LoRa firmware flash succeeded");
} }
Err(e) => { Err(e) => {
@@ -434,7 +486,6 @@ pub async fn start_flash_job(
// erase_flash failed", no actual esptool stderr). // erase_flash failed", no actual esptool stderr).
warn!(path = %path, error = %format!("{e:#}"), "LoRa firmware flash failed"); warn!(path = %path, error = %format!("{e:#}"), "LoRa firmware flash failed");
bg_job.push_log(format!("ERROR: {e:#}")).await; bg_job.push_log(format!("ERROR: {e:#}")).await;
bg_job.fail(e).await;
} }
} }
@@ -450,6 +501,9 @@ pub async fn start_flash_job(
} }
if !succeeded { if !succeeded {
if let Err(error) = &result {
bg_job.fail(error).await;
}
// Deliberately do NOT auto-restart the listener here. A failed // Deliberately do NOT auto-restart the listener here. A failed
// flash means we can't vouch for the board's state — reopening // flash means we can't vouch for the board's state — reopening
// the port immediately (esptool/rnodeconf's own reset sequence // the port immediately (esptool/rnodeconf's own reset sequence
@@ -499,6 +553,7 @@ pub async fn start_flash_job(
Err(e) => warn!(error = %e, "Failed to load mesh config after flash"), Err(e) => warn!(error = %e, "Failed to load mesh config after flash"),
} }
} }
bg_job.finish().await;
}); });
*job.abort_handle.write().await = Some(task.abort_handle()); *job.abort_handle.write().await = Some(task.abort_handle());
@@ -510,12 +565,13 @@ async fn run_flash(
family: DeviceType, family: DeviceType,
data_dir: &Path, data_dir: &Path,
path: &str, path: &str,
prepared_image: Option<&Path>,
job: &Arc<FlashJob>, job: &Arc<FlashJob>,
) -> Result<()> { ) -> Result<()> {
match family { match family {
DeviceType::Meshtastic | DeviceType::Meshcore => { DeviceType::Meshtastic | DeviceType::Meshcore => {
let image = fetch_esptool_image(board, family, data_dir, job).await?; let image = prepared_image.context("Firmware was not prepared before serial access")?;
esptool_erase_and_write(path, &image, job).await esptool_erase_and_write(path, image, job).await
} }
DeviceType::Reticulum => { DeviceType::Reticulum => {
let lora_region = super::load_config(data_dir) let lora_region = super::load_config(data_dir)
@@ -664,15 +720,65 @@ async fn fetch_meshcore_image(
anyhow::anyhow!("No matching MeshCore image in release {}", release.tag_name) anyhow::anyhow!("No matching MeshCore image in release {}", release.tag_name)
})?; })?;
anyhow::ensure!(
Path::new(&asset.name)
.file_name()
.and_then(|name| name.to_str())
== Some(asset.name.as_str()),
"Invalid firmware asset filename"
);
let out_path = cache.join(&asset.name); let out_path = cache.join(&asset.name);
if tokio::fs::metadata(&out_path).await.is_ok() { if tokio::fs::metadata(&out_path).await.is_ok() {
job.push_log(format!("Using cached {}", asset.name)).await; if verify_meshcore_asset(&out_path, asset).await.is_ok() {
return Ok(out_path); job.push_log(format!("Using verified cached {}", asset.name))
.await;
return Ok(out_path);
}
tokio::fs::remove_file(&out_path)
.await
.context("Removing invalid cached firmware")?;
job.push_log("Cached firmware failed verification; downloading a fresh copy")
.await;
} }
download_to_file(client, &asset.browser_download_url, &out_path, job).await?; download_to_file(client, &asset.browser_download_url, &out_path, job).await?;
if let Err(error) = verify_meshcore_asset(&out_path, asset).await {
// A partial/unverified download must not become next attempt's cache.
let _ = tokio::fs::remove_file(&out_path).await;
return Err(error);
}
Ok(out_path) Ok(out_path)
} }
async fn verify_meshcore_asset(path: &Path, asset: &GithubAsset) -> Result<()> {
use sha2::{Digest, Sha256};
let size = asset
.size
.context("MeshCore release did not provide firmware size")?;
anyhow::ensure!(
(1..=16 * 1024 * 1024).contains(&size),
"Invalid MeshCore firmware size"
);
anyhow::ensure!(
tokio::fs::metadata(path).await?.len() == size,
"Firmware size mismatch; radio was not written"
);
let expected = asset
.digest
.as_deref()
.and_then(|value| value.strip_prefix("sha256:"))
.context("MeshCore release did not provide a SHA-256 checksum")?;
anyhow::ensure!(
expected.len() == 64 && expected.bytes().all(|byte| byte.is_ascii_hexdigit()),
"Invalid MeshCore release checksum"
);
let actual = hex::encode(Sha256::digest(tokio::fs::read(path).await?));
anyhow::ensure!(
actual.eq_ignore_ascii_case(expected),
"Firmware checksum mismatch; radio was not written"
);
Ok(())
}
async fn download_to_file( async fn download_to_file(
client: &reqwest::Client, client: &reqwest::Client,
url: &str, url: &str,
@@ -722,7 +828,8 @@ async fn download_to_file(
} }
} }
} }
file.flush().await.ok(); file.flush().await.context("Flushing firmware download")?;
file.sync_all().await.context("Saving firmware download")?;
tokio::fs::rename(&tmp, dest) tokio::fs::rename(&tmp, dest)
.await .await
.context("Finalizing firmware download")?; .context("Finalizing firmware download")?;
@@ -773,19 +880,10 @@ async fn esptool_erase_and_write(path: &str, image: &Path, job: &Arc<FlashJob>)
/// Building global args separately from subcommand args keeps this correct /// Building global args separately from subcommand args keeps this correct
/// by construction instead of relying on call-site ordering. /// by construction instead of relying on call-site ordering.
/// ///
/// Normal stub-loader mode (no --no-stub) needs the esp32s3 stub flasher /// Normal stub-loader mode is required for full-chip erase. The packaged
/// blob at /usr/lib/python3/dist-packages/esptool/targets/stub_flasher/ /// archy-esptool includes and self-tests Espressif's ESP32-S3 stub. Debian's
/// stub_flasher_32s3.json — Debian's `esptool` package (4.7.0+dfsg-0.1) /// stripped esptool package alone did not provide that resource, so changing
/// ships without it (stripped for DFSG compliance: the prebuilt blob has no /// flags to --no-stub cannot repair this operation.
/// buildable-from-source path Debian could verify), so scripts/self-update.sh
/// fetches the exact same file from the matching upstream esptool release
/// tag and installs it alongside the apt package (see the esptool install
/// step there). --no-stub (talk directly to the ROM bootloader, skip the
/// stub) was tried first and works for connecting, but the ROM bootloader
/// doesn't implement a full-chip-erase opcode at all — only the stub does —
/// so --no-stub broke our "always erase before write" default outright
/// rather than just being slower. Restoring the real stub file is the
/// correct fix, not routing around its absence.
fn esptool_global_args<'a>(path: &'a str, baud: Option<&'a str>) -> Vec<&'a str> { fn esptool_global_args<'a>(path: &'a str, baud: Option<&'a str>) -> Vec<&'a str> {
let mut args = vec!["--chip", ESPTOOL_CHIP, "--port", path]; let mut args = vec!["--chip", ESPTOOL_CHIP, "--port", path];
if let Some(b) = baud { if let Some(b) = baud {
@@ -795,24 +893,96 @@ fn esptool_global_args<'a>(path: &'a str, baud: Option<&'a str>) -> Vec<&'a str>
args args
} }
fn esptool_executable() -> Result<PathBuf> {
let mut candidates = vec![PathBuf::from("/usr/local/bin/archy-esptool")];
if let Some(paths) = std::env::var_os("PATH") {
for directory in std::env::split_paths(&paths) {
for name in ["esptool", "esptool.py"] {
candidates.push(directory.join(name));
}
}
}
executable_from_candidates(candidates)
}
fn executable_from_candidates(candidates: impl IntoIterator<Item = PathBuf>) -> Result<PathBuf> {
use std::os::unix::fs::PermissionsExt;
candidates.into_iter().find(|path| {
path.is_file() && path.metadata().is_ok_and(|metadata| metadata.permissions().mode() & 0o111 != 0)
}).ok_or_else(|| anyhow::anyhow!(
"Radio flashing tools are missing. Install the complete Archipelago update and retry; the radio has not been changed."
))
}
async fn preflight_esptool() -> Result<PathBuf> {
let executable = esptool_executable()?;
check_esptool(&executable).await?;
Ok(executable)
}
async fn check_esptool(executable: &Path) -> Result<()> {
let mut command = Command::new(executable);
command
.arg(
if executable
.file_name()
.is_some_and(|name| name == "archy-esptool")
{
"--archy-self-test"
} else {
"version"
},
)
.kill_on_drop(true);
let output = tokio::time::timeout(std::time::Duration::from_secs(20), command.output())
.await
.context("Radio flashing tool did not respond; the radio has not been changed")?
.context("Radio flashing tool could not start; check its installation and permissions")?;
anyhow::ensure!(
output.status.success(),
"Radio flashing tool self-check failed; reinstall the complete update before retrying"
);
Ok(())
}
fn retryable_flash_error(error: &anyhow::Error) -> bool {
if error
.chain()
.any(|cause| cause.downcast_ref::<std::io::Error>().is_some())
{
return false;
}
let detail = format!("{error:#}").to_lowercase();
[
"failed to connect",
"timed out waiting",
"invalid head of packet",
"serial data stream stopped",
]
.iter()
.any(|message| detail.contains(message))
}
async fn esptool_with_retry(path: &str, subcommand: &[&str], job: &Arc<FlashJob>) -> Result<()> { async fn esptool_with_retry(path: &str, subcommand: &[&str], job: &Arc<FlashJob>) -> Result<()> {
let mut cmd = Command::new("esptool"); let executable = preflight_esptool().await?;
let mut cmd = Command::new(&executable);
cmd.args(esptool_global_args(path, None)); cmd.args(esptool_global_args(path, None));
cmd.args(subcommand); cmd.args(subcommand);
match run_streamed(cmd, None, job).await { match run_streamed(cmd, None, job).await {
Ok(()) => Ok(()), Ok(()) => Ok(()),
Err(first_err) => { Err(first_err) if retryable_flash_error(&first_err) => {
job.push_log(format!( job.push_log(format!(
"First attempt failed ({first_err:#}); retrying once at {ESPTOOL_FALLBACK_BAUD} baud" "First attempt failed ({first_err:#}); retrying once at {ESPTOOL_FALLBACK_BAUD} baud"
)) ))
.await; .await;
let mut retry = Command::new("esptool"); let mut retry = Command::new(&executable);
retry.args(esptool_global_args(path, Some(ESPTOOL_FALLBACK_BAUD))); retry.args(esptool_global_args(path, Some(ESPTOOL_FALLBACK_BAUD)));
retry.args(subcommand); retry.args(subcommand);
run_streamed(retry, None, job) run_streamed(retry, None, job)
.await .await
.context(format!("retry also failed (first attempt: {first_err:#})")) .context(format!("retry also failed (first attempt: {first_err:#})"))
} }
Err(error) => Err(error),
} }
} }
@@ -990,3 +1160,159 @@ async fn run_streamed(mut cmd: Command, stdin: Option<Vec<u8>>, job: &Arc<FlashJ
} }
Ok(()) Ok(())
} }
#[cfg(test)]
mod flashing_regression_tests {
use super::*;
#[tokio::test]
async fn full_flash_invalidates_only_radio_settings_marker() {
let dir = tempfile::tempdir().unwrap();
let marker = dir.path().join("meshcore-radio-params.json");
tokio::fs::write(&marker, b"old settings").await.unwrap();
let other = dir.path().join("mesh-config.json");
tokio::fs::write(&other, b"preserved").await.unwrap();
invalidate_radio_settings_marker(dir.path()).await.unwrap();
assert!(!marker.exists());
assert_eq!(tokio::fs::read(&other).await.unwrap(), b"preserved");
invalidate_radio_settings_marker(dir.path()).await.unwrap();
tokio::fs::create_dir(&marker).await.unwrap();
assert!(invalidate_radio_settings_marker(dir.path()).await.is_err());
}
use std::os::unix::fs::PermissionsExt;
#[tokio::test]
async fn meshcore_cache_requires_release_size_and_checksum() {
use sha2::{Digest, Sha256};
let dir = tempfile::tempdir().unwrap();
let file = dir.path().join("fixture.bin");
tokio::fs::write(&file, b"firmware fixture").await.unwrap();
let mut asset = GithubAsset {
name: "fixture.bin".into(),
browser_download_url: "https://example.invalid/fixture.bin".into(),
size: Some(16),
digest: Some(format!(
"sha256:{}",
hex::encode(Sha256::digest(b"firmware fixture"))
)),
};
assert!(verify_meshcore_asset(&file, &asset).await.is_ok());
tokio::fs::write(&file, b"tampered fixture").await.unwrap();
assert!(verify_meshcore_asset(&file, &asset).await.is_err());
tokio::fs::write(&file, b"short").await.unwrap();
assert!(verify_meshcore_asset(&file, &asset).await.is_err());
tokio::fs::write(&file, b"firmware fixture").await.unwrap();
asset.digest = None;
assert!(verify_meshcore_asset(&file, &asset).await.is_err());
}
#[test]
fn generic_usb_ids_do_not_select_firmware() {
for (vid, pid) in [("10c4", "ea60"), ("303a", "1001")] {
let info = DetectedDeviceInfo {
path: "/dev/fixture".into(),
vid: Some(vid.into()),
pid: Some(pid.into()),
product: None,
manufacturer: None,
plugged_at: None,
};
assert_eq!(resolve_flash_board(&info), None);
}
}
#[test]
fn absent_nonexecutable_and_directory_candidates_are_rejected() {
let dir = tempfile::tempdir().unwrap();
let file = dir.path().join("flasher");
std::fs::write(&file, "#!/bin/sh\nexit 0\n").unwrap();
std::fs::set_permissions(&file, std::fs::Permissions::from_mode(0o600)).unwrap();
assert!(executable_from_candidates([
dir.path().join("absent"),
file.clone(),
dir.path().to_path_buf()
])
.is_err());
std::fs::set_permissions(&file, std::fs::Permissions::from_mode(0o700)).unwrap();
assert_eq!(executable_from_candidates([file.clone()]).unwrap(), file);
}
#[tokio::test]
async fn flasher_preflight_reports_missing_interpreter_and_failed_self_check() {
let dir = tempfile::tempdir().unwrap();
let file = dir.path().join("archy-esptool");
for script in ["#!/missing/python\n", "#!/bin/sh\nexit 7\n"] {
std::fs::write(&file, script).unwrap();
std::fs::set_permissions(&file, std::fs::Permissions::from_mode(0o700)).unwrap();
assert!(check_esptool(&file).await.is_err());
}
std::fs::write(&file, "#!/bin/sh\n[ \"$1\" = --archy-self-test ]\n").unwrap();
assert!(check_esptool(&file).await.is_ok());
}
#[tokio::test]
async fn flash_registration_excludes_other_writers_and_active_probes() {
let handle = new_job_handle();
let first = FlashJob::new(
FlashBoard::HeltecV3,
DeviceType::Meshcore,
"/dev/fixture".into(),
);
let second = FlashJob::new(
FlashBoard::HeltecV3,
DeviceType::Meshcore,
"/dev/fixture".into(),
);
let probe = handle.read().await;
assert!(register_flash_job(&handle, &first).await.is_err());
drop(probe);
let (a, b) = tokio::join!(
register_flash_job(&handle, &first),
register_flash_job(&handle, &second)
);
assert_eq!(usize::from(a.is_ok()) + usize::from(b.is_ok()), 1);
handle.read().await.as_ref().unwrap().finish().await;
let next = FlashJob::new(
FlashBoard::HeltecV3,
DeviceType::Meshcore,
"/dev/fixture".into(),
);
assert!(register_flash_job(&handle, &next).await.is_ok());
}
#[test]
fn retries_only_known_serial_transport_failures() {
for kind in [
std::io::ErrorKind::NotFound,
std::io::ErrorKind::PermissionDenied,
] {
assert!(!retryable_flash_error(
&anyhow::Error::from(std::io::Error::from(kind))
.context("Failed to start subprocess")
));
}
for message in [
"Wrong chip",
"Invalid image",
"module missing",
"permission denied",
"port is busy",
] {
assert!(!retryable_flash_error(&anyhow::anyhow!(message)));
}
assert!(retryable_flash_error(&anyhow::anyhow!(
"Failed to connect to ESP32-S3: timed out waiting for packet header"
)));
assert_eq!(
esptool_global_args("/dev/fixture", Some("115200")),
[
"--chip",
"esp32s3",
"--port",
"/dev/fixture",
"--baud",
"115200"
]
);
}
}
+54 -5
View File
@@ -1092,6 +1092,14 @@ impl MeshService {
let (new_tx, new_rx) = tokio::sync::mpsc::channel(32); let (new_tx, new_rx) = tokio::sync::mpsc::channel(32);
*self.state.cmd_tx.write().await = new_tx; *self.state.cmd_tx.write().await = new_tx;
self.cmd_rx = Some(new_rx); self.cmd_rx = Some(new_rx);
{
let mut status = self.state.status.write().await;
status.device_connected = false;
status.device_path = None;
status.firmware_version = None;
status.self_node_id = None;
status.peer_count = 0;
}
info!("Mesh service stopped"); info!("Mesh service stopped");
} }
@@ -2321,7 +2329,10 @@ impl MeshService {
} }
} }
let was_enabled = self.config.enabled; let listener_running = self
.listener_handle
.as_ref()
.is_some_and(|handle| !handle.is_finished());
let needs_session_restart = session_config_changed(&self.config, &config); let needs_session_restart = session_config_changed(&self.config, &config);
self.config = config.clone(); self.config = config.clone();
@@ -2335,10 +2346,13 @@ impl MeshService {
.unwrap_or_else(|| "archipelago".to_string()); .unwrap_or_else(|| "archipelago".to_string());
} }
// If enabled state changed, start/stop the listener // Reconcile desired state with the actual task, not the old enabled
if config.enabled && !was_enabled { // flag. A failed flash can stop the task while keeping enabled=true;
// explicitly reconnecting with the same settings must restart it.
if config.enabled && !listener_running {
self.stop().await;
self.start()?; self.start()?;
} else if !config.enabled && was_enabled { } else if !config.enabled {
self.stop().await; self.stop().await;
// Clear connected state // Clear connected state
let mut status = self.state.status.write().await; let mut status = self.state.status.write().await;
@@ -2347,7 +2361,7 @@ impl MeshService {
status.firmware_version = None; status.firmware_version = None;
status.self_node_id = None; status.self_node_id = None;
status.peer_count = 0; status.peer_count = 0;
} else if config.enabled && was_enabled && needs_session_restart { } else if needs_session_restart {
info!("Mesh session config changed — restarting listener to apply"); info!("Mesh session config changed — restarting listener to apply");
self.stop().await; self.stop().await;
self.start()?; self.start()?;
@@ -2537,6 +2551,41 @@ mod tests {
} }
use super::*; use super::*;
#[tokio::test]
async fn reconnect_with_unchanged_enabled_config_restarts_stopped_listener() {
let dir = tempfile::tempdir().unwrap();
let key = SigningKey::from_bytes(&[7; 32]);
let public = hex::encode(key.verifying_key().to_bytes());
let did = crate::identity::did_key_from_pubkey_hex(&public).unwrap();
let config = MeshConfig {
enabled: true,
..Default::default()
};
save_config(dir.path(), &config).await.unwrap();
let mut service = MeshService::new(dir.path(), &key, &did, &public)
.await
.unwrap();
assert!(service.listener_handle.is_none());
service.configure(config.clone()).await.unwrap();
assert!(service.listener_handle.is_some());
service.stop().await;
assert!(service.config.enabled);
service.configure(config.clone()).await.unwrap();
assert!(service.listener_handle.is_some());
service.stop().await;
service.listener_handle = Some(tokio::spawn(async {}));
tokio::task::yield_now().await;
service.configure(config).await.unwrap();
assert!(!service.listener_handle.as_ref().unwrap().is_finished());
service.stop().await;
let disabled = MeshConfig {
enabled: false,
..Default::default()
};
service.configure(disabled).await.unwrap();
assert!(service.listener_handle.is_none());
}
#[test] #[test]
fn session_config_change_detection() { fn session_config_change_detection() {
let base = MeshConfig::default(); let base = MeshConfig::default();
+42
View File
@@ -378,6 +378,19 @@ fn decode_mesh_name(bytes: &[u8], fallback: &str) -> String {
/// Parse RESP_DEVICE_INFO (0x0D) response. /// Parse RESP_DEVICE_INFO (0x0D) response.
/// Returns firmware version string and device capabilities. /// Returns firmware version string and device capabilities.
pub fn parse_device_info(data: &[u8]) -> Result<(String, u16)> { pub fn parse_device_info(data: &[u8]) -> Result<(String, u16)> {
// Official companion v3+ binary layout: protocol, half-capacity,
// channels, PIN (4), build date (12), model (40), version (20).
// Verified against companion-v1.17.1 MyMesh.cpp and Heltec V3 readback.
if data
.first()
.is_some_and(|version| (3..=31).contains(version))
{
anyhow::ensure!(data.len() >= 79, "Truncated MeshCore device info");
return Ok((
decode_mesh_name(&data[59..79], "unknown"),
u16::from(data[1]) * 2,
));
}
// Device info format varies by firmware version. // Device info format varies by firmware version.
// Minimum: firmware version string (null-terminated) + max_contacts (u16 LE) // Minimum: firmware version string (null-terminated) + max_contacts (u16 LE)
if data.is_empty() { if data.is_empty() {
@@ -404,6 +417,15 @@ pub fn parse_self_info(data: &[u8]) -> Result<(u32, String)> {
anyhow::bail!("Self info response too short: {} bytes", data.len()); anyhow::bail!("Self info response too short: {} bytes", data.len());
} }
// Current companions send type/power/max-power, public key (32),
// position (8), four preference bytes, RF parameters (10), then name.
if data.len() >= 57 {
let node_id = u32::from_le_bytes(data[3..7].try_into().unwrap());
return Ok((
node_id,
decode_mesh_name(&data[57..], &format!("node-{node_id:08x}")),
));
}
let node_id = u32::from_le_bytes([data[0], data[1], data[2], data[3]]); let node_id = u32::from_le_bytes([data[0], data[1], data[2], data[3]]);
// Name follows after fixed fields. A firmware whose fixed-field layout // Name follows after fixed fields. A firmware whose fixed-field layout
@@ -966,4 +988,24 @@ mod tests {
fn test_parse_self_info_too_short() { fn test_parse_self_info_too_short() {
assert!(parse_self_info(&[0x01, 0x02]).is_err()); assert!(parse_self_info(&[0x01, 0x02]).is_err());
} }
#[test]
fn current_companion_binary_metadata_is_not_a_name_or_version() {
let mut info = vec![0u8; 81];
info[0] = 13;
info[1] = 150;
info[19..28].copy_from_slice(b"Heltec V3");
info[59..74].copy_from_slice(b"v1.17.1-d929643");
assert_eq!(
parse_device_info(&info).unwrap(),
("v1.17.1-d929643".into(), 300)
);
assert!(parse_device_info(&info[..78]).is_err());
let mut own = vec![0u8; 57];
own[0..3].copy_from_slice(&[1, 22, 22]);
own[3..7].copy_from_slice(&42u32.to_le_bytes());
own[47..51].copy_from_slice(&869618u32.to_le_bytes());
own.extend_from_slice(b"My radio");
assert_eq!(parse_self_info(&own).unwrap(), (42, "My radio".into()));
}
} }
+48 -11
View File
@@ -277,6 +277,19 @@ fn terminate_group(child: &Child) {
} }
} }
/// Own the daemon during its asynchronous handshake too. Cancellation drops
/// the future without executing an error branch; a bare Child would survive
/// and keep the serial port open even though the listener had stopped.
struct StartingDaemon(Option<Child>);
impl Drop for StartingDaemon {
fn drop(&mut self) {
if let Some(child) = self.0.as_ref() {
terminate_group(child);
}
}
}
/// One peer learned via an RNS announce (LXMF delivery destination). /// One peer learned via an RNS announce (LXMF delivery destination).
#[derive(Clone)] #[derive(Clone)]
struct ReticulumPeer { struct ReticulumPeer {
@@ -517,10 +530,10 @@ impl ReticulumLink {
let child = cmd let child = cmd
.spawn() .spawn()
.context("Failed to spawn reticulum-daemon — is it installed/packaged?")?; .context("Failed to spawn reticulum-daemon — is it installed/packaged?")?;
let mut starting = StartingDaemon(Some(child));
// Wait for the socket to appear, then for the daemon's "ready" event. // Wait for the socket to appear, then for the daemon's "ready" event.
// Runs as a block so every failure path tears the just-spawned daemon // StartingDaemon tears the group down on errors AND cancellation.
// group down via `terminate_group` (the child has no `kill_on_drop`).
let init = async { let init = async {
let deadline = tokio::time::Instant::now() + Duration::from_secs(15); let deadline = tokio::time::Instant::now() + Duration::from_secs(15);
let stream = loop { let stream = loop {
@@ -560,20 +573,17 @@ impl ReticulumLink {
dest_hash = %dest_hash_hex, dest_hash = %dest_hash_hex,
"Reticulum daemon ready" "Reticulum daemon ready"
); );
Ok((write_half, reader, dest_hash, display_name)) Ok::<_, anyhow::Error>((write_half, reader, dest_hash, display_name))
};
let (write_half, reader, dest_hash, display_name) = match init.await {
Ok(parts) => parts,
Err(e) => {
terminate_group(&child);
return Err(e);
}
}; };
let (write_half, reader, dest_hash, display_name) = init.await?;
let mut link = Self { let mut link = Self {
device_path: label, device_path: label,
socket_path, socket_path,
child, child: starting
.0
.take()
.expect("starting daemon is owned until ready"),
writer: write_half, writer: write_half,
reader, reader,
dest_hash, dest_hash,
@@ -1557,6 +1567,33 @@ impl Drop for ReticulumLink {
mod tests { mod tests {
use super::*; use super::*;
#[tokio::test]
async fn cancelled_daemon_handshake_terminates_child() {
let (started, ready) = tokio::sync::oneshot::channel();
let task = tokio::spawn(async move {
let child = Command::new("sleep")
.arg("60")
.process_group(0)
.spawn()
.unwrap();
let pid = child.id().unwrap();
let _starting = StartingDaemon(Some(child));
started.send(pid).unwrap();
std::future::pending::<()>().await;
});
let pid = ready.await.unwrap();
assert_eq!(unsafe { libc::kill(pid as i32, 0) }, 0);
task.abort();
assert!(task.await.unwrap_err().is_cancelled());
tokio::time::timeout(Duration::from_secs(3), async {
while unsafe { libc::kill(pid as i32, 0) } == 0 {
tokio::time::sleep(Duration::from_millis(20)).await;
}
})
.await
.expect("cancelled handshake left its child alive");
}
#[test] #[test]
fn announced_name_precedence() { fn announced_name_precedence() {
// Daemon-decoded LXMF name always wins. // Daemon-decoded LXMF name always wins.
+6 -2
View File
@@ -146,12 +146,16 @@ impl MeshcoreDevice {
} }
} }
let info = DeviceInfo { let mut info = DeviceInfo {
firmware_version: name.clone(), firmware_version: "unknown".to_string(),
node_id, node_id,
max_contacts: 100, max_contacts: 100,
device_type: super::types::DeviceType::Meshcore, device_type: super::types::DeviceType::Meshcore,
}; };
if let Some((version, capacity)) = self.query_device_info().await {
info.firmware_version = version;
info.max_contacts = capacity;
}
self.device_info = Some(info.clone()); self.device_info = Some(info.clone());
info!("Meshcore initialization complete on {}", self.device_path); info!("Meshcore initialization complete on {}", self.device_path);
@@ -0,0 +1,124 @@
//! Compatibility and hostile-relay regression tests for the 0.44 security update.
//! Loopback sockets run only through scripts/test-backend-isolated.sh.
use nostr_sdk::prelude::*;
#[test]
fn native_signer_encryption_remains_compatible_and_rejects_hostile_payloads() {
let alice = Keys::generate();
let bob = Keys::generate();
let message = "Archipelago signing compatibility — \u{1f30a}";
let encrypted = nip44::encrypt(
alice.secret_key(),
&bob.public_key(),
message,
nip44::Version::V2,
)
.unwrap();
assert_eq!(
nip44::decrypt(bob.secret_key(), &alice.public_key(), &encrypted).unwrap(),
message
);
let encrypted = nip04::encrypt(alice.secret_key(), &bob.public_key(), message).unwrap();
assert_eq!(
nip04::decrypt(bob.secret_key(), &alice.public_key(), encrypted).unwrap(),
message
);
// Valid v2 prefix followed by a payload exceeding the codec's maximum.
// This must fail at the size gate, before allocation/decoding/HMAC work.
let oversized = format!("AgAA{}", "A".repeat(100_000));
assert!(matches!(
nip44::decrypt(bob.secret_key(), &alice.public_key(), oversized),
Err(nip44::Error::MessageTooLong)
));
for malformed in ["", "Ag==", "not base64!", "?iv=", "YQ==?iv=YQ=="] {
assert!(nip04::decrypt(bob.secret_key(), &alice.public_key(), malformed).is_err());
assert!(nip44::decrypt(bob.secret_key(), &alice.public_key(), malformed).is_err());
}
}
#[tokio::test]
async fn relay_cannot_substitute_forged_fields_for_a_known_event_id() {
use futures_util::{SinkExt, StreamExt};
use tokio::net::TcpListener;
use tokio_tungstenite::{accept_async, tungstenite::Message};
let test = async {
let author = Keys::generate();
let known = EventBuilder::text_note("already verified")
.sign_with_keys(&author)
.unwrap();
let valid = EventBuilder::text_note("new legitimate message")
.sign_with_keys(&author)
.unwrap();
let mut forged_content = known.clone();
forged_content.content = "forged content".into();
let mut forged_author = known.clone();
forged_author.pubkey = Keys::generate().public_key();
let mut forged_signature = known.clone();
forged_signature.sig = valid.sig;
for forged in [&forged_content, &forged_author, &forged_signature] {
assert!(forged.verify().is_err());
}
let listener = TcpListener::bind("127.0.0.1:0").await.unwrap();
let url = format!("ws://{}", listener.local_addr().unwrap());
let expected_id = valid.id;
let relay = tokio::spawn(async move {
let (socket, _) = listener.accept().await.unwrap();
let mut socket = accept_async(socket).await.unwrap();
while let Some(message) = socket.next().await {
let text = match message.unwrap() {
Message::Text(text) => text,
Message::Ping(payload) => {
socket.send(Message::Pong(payload)).await.unwrap();
continue;
}
Message::Close(_) => break,
_ => continue,
};
let message: serde_json::Value = serde_json::from_str(&text).unwrap();
if message[0] != "REQ" {
continue;
}
let subscription = message[1].as_str().unwrap();
for event in [&forged_content, &forged_author, &forged_signature, &valid] {
socket
.send(Message::Text(
serde_json::json!(["EVENT", subscription, event]).to_string(),
))
.await
.unwrap();
}
socket
.send(Message::Text(
serde_json::json!(["EOSE", subscription]).to_string(),
))
.await
.unwrap();
}
});
let client = Client::new(Keys::generate());
client.database().save_event(&known).await.unwrap();
client.add_relay(&url).await.unwrap();
client
.try_connect_relay(&url, std::time::Duration::from_secs(3))
.await
.unwrap();
let events = client
.fetch_events(
Filter::new().kind(Kind::TextNote),
std::time::Duration::from_secs(5),
)
.await
.unwrap();
assert_eq!(
events.len(),
1,
"forged events must never reach SDK consumers"
);
assert_eq!(events.iter().next().unwrap().id, expected_id);
client.disconnect().await;
relay.abort();
};
tokio::time::timeout(std::time::Duration::from_secs(15), test)
.await
.expect("local relay test timed out");
}
+2
View File
@@ -83,6 +83,8 @@ impl EndpointRateLimiter {
limits.insert("wallet.send".to_string(), (5usize, 300u64)); limits.insert("wallet.send".to_string(), (5usize, 300u64));
limits.insert("wallet.ecash-send".to_string(), (10, 300)); limits.insert("wallet.ecash-send".to_string(), (10, 300));
limits.insert("lnd.sendcoins".to_string(), (5, 300)); limits.insert("lnd.sendcoins".to_string(), (5, 300));
limits.insert("lnd.bump-submit".to_string(), (5, 300));
limits.insert("lnd.bump-quote".to_string(), (30, 60));
limits.insert("lnd.payinvoice".to_string(), (10, 300)); limits.insert("lnd.payinvoice".to_string(), (10, 300));
limits.insert("lnd.openchannel".to_string(), (3, 300)); limits.insert("lnd.openchannel".to_string(), (3, 300));
limits.insert("lnd.closechannel".to_string(), (3, 300)); limits.insert("lnd.closechannel".to_string(), (3, 300));
+223 -19
View File
@@ -1475,6 +1475,48 @@ pub fn is_peer_allowed_path(path: &str) -> bool {
|| path.starts_with("/dwn/") || path.starts_with("/dwn/")
} }
/// The ordinary API listener is a management surface even when contacted
/// directly, without host nginx. Peer traffic has its own path-restricted
/// listener and retains its existing cryptographic authentication.
fn management_peer_is_private(address: std::net::IpAddr) -> bool {
match address {
std::net::IpAddr::V4(ip) => {
ip.is_loopback()
|| ip.is_private()
|| ip.is_link_local()
|| (ip.octets()[0] == 100 && (64..=127).contains(&ip.octets()[1]))
}
std::net::IpAddr::V6(ip) => {
if let Some(mapped) = ip.to_ipv4_mapped() {
management_peer_is_private(std::net::IpAddr::V4(mapped))
} else {
ip.is_loopback() || ip.is_unique_local() || ip.is_unicast_link_local()
}
}
}
}
fn request_surface_allowed(
peer_only: bool,
peer: std::net::IpAddr,
request: &hyper::Request<hyper::Body>,
) -> bool {
if peer_only {
return is_peer_allowed_path(request.uri().path());
}
// Keep purpose-built content/peer HTTP endpoints reachable with their
// existing handler-level checks. The general RPC dispatcher is a management
// surface here; only the dedicated peer listener retains public peer RPC.
if request.uri().path() != "/rpc/v1" && is_peer_allowed_path(request.uri().path()) {
return true;
}
management_peer_is_private(peer)
&& !request
.headers()
.get("x-archipelago-public-ingress")
.is_some_and(|value| !value.as_bytes().is_empty())
}
async fn accept_loop( async fn accept_loop(
handler: Arc<ApiHandler>, handler: Arc<ApiHandler>,
listener: TcpListener, listener: TcpListener,
@@ -1552,7 +1594,7 @@ async fn accept_loop(
// forwarded headers on loopback (nginx) connections. // forwarded headers on loopback (nginx) connections.
req.extensions_mut() req.extensions_mut()
.insert(crate::api::rpc::PeerAddr(peer_addr)); .insert(crate::api::rpc::PeerAddr(peer_addr));
if peer_only && !is_peer_allowed_path(req.uri().path()) { if !request_surface_allowed(peer_only, peer_addr.ip(), &req) {
let resp = hyper::Response::builder() let resp = hyper::Response::builder()
.status(hyper::StatusCode::NOT_FOUND) .status(hyper::StatusCode::NOT_FOUND)
.body(hyper::Body::empty()) .body(hyper::Body::empty())
@@ -1765,12 +1807,17 @@ fn merge_preserving_transitional(
}; };
crate::data_model::PackageDataEntry { crate::data_model::PackageDataEntry {
state, state: state.clone(),
// install_progress and uninstall_stage are also owned by the // install_progress and uninstall_stage are also owned by the
// initiating op (same reason as state) — keep them. // initiating op (same reason as state) — keep them.
install_progress: existing.install_progress.clone(), install_progress: existing.install_progress.clone(),
uninstall_stage: existing.uninstall_stage.clone(), uninstall_stage: existing.uninstall_stage.clone(),
// Everything else comes from the fresh scan. // Everything else comes from the fresh scan.
ui_ready: if state == crate::data_model::PackageState::Running {
fresh.ui_ready
} else {
Some(false)
},
health: fresh.health.clone(), health: fresh.health.clone(),
exit_code: fresh.exit_code, exit_code: fresh.exit_code,
static_files: fresh.static_files.clone(), static_files: fresh.static_files.clone(),
@@ -1809,7 +1856,10 @@ async fn scan_and_update_packages(
absence_tracker: &mut HashMap<String, u32>, absence_tracker: &mut HashMap<String, u32>,
transitional_since: &mut HashMap<String, Instant>, transitional_since: &mut HashMap<String, Instant>,
) -> Result<()> { ) -> Result<()> {
let mut packages = scanner.scan_containers().await?; let (before_scan, _) = state.get_snapshot().await;
let mut packages = scanner
.scan_containers(data_dir, &before_scan.package_data)
.await?;
let user_stopped = crate::crash_recovery::load_user_stopped(data_dir).await; let user_stopped = crate::crash_recovery::load_user_stopped(data_dir).await;
for (id, pkg) in packages.iter_mut() { for (id, pkg) in packages.iter_mut() {
if pkg.state == crate::data_model::PackageState::Exited && user_stopped.contains(id) { if pkg.state == crate::data_model::PackageState::Exited && user_stopped.contains(id) {
@@ -1870,11 +1920,14 @@ async fn scan_and_update_packages(
// once at load ~2). Better to keep saying "scanning…" than to say "empty". // once at load ~2). Better to keep saying "scanning…" than to say "empty".
if packages.is_empty() && (!first_scan || !installed_registry.is_empty()) { if packages.is_empty() && (!first_scan || !installed_registry.is_empty()) {
if tor_changed || update_changed { if tor_changed || update_changed {
let mut data = current_data; state
data.server_info.tor_address = tor_addr.clone(); .mutate_data(|data| {
data.server_info.node_address = tor_addr.as_ref().map(|t| identity.node_address(t)); data.server_info.tor_address = tor_addr.clone();
data.server_info.status_info.updated = update_available; data.server_info.node_address =
state.update_data(data).await; tor_addr.as_ref().map(|t| identity.node_address(t));
data.server_info.status_info.updated = update_available;
})
.await;
} }
return Ok(()); return Ok(());
} }
@@ -1899,6 +1952,13 @@ async fn scan_and_update_packages(
// died without cleanup and let the scan override it. // died without cleanup and let the scan override it.
let now = Instant::now(); let now = Instant::now();
for (id, pkg) in &packages { for (id, pkg) in &packages {
if user_uninstalled.contains(id)
|| user_uninstalled.contains(&format!("archy-{id}"))
|| (before_scan.package_data.contains_key(id)
&& !current_data.package_data.contains_key(id))
{
continue;
}
absence_tracker.remove(id); absence_tracker.remove(id);
let existing = merged.get(id); let existing = merged.get(id);
let overwrite = match existing { let overwrite = match existing {
@@ -2054,22 +2114,40 @@ async fn scan_and_update_packages(
} }
if changed || tor_changed || first_scan || update_changed { if changed || tor_changed || first_scan || update_changed {
let mut data = current_data; state
data.package_data = merged; .mutate_data(|data| {
data.server_info.tor_address = tor_addr.clone(); // A lifecycle operation may have started/finished while this scan
data.server_info.node_address = tor_addr.as_ref().map(|t| identity.node_address(t)); // awaited probes or disk I/O. Never overwrite that newer entry or
data.server_info.status_info.containers_scanned = true; // resurrect one that an uninstall removed in the meantime.
data.server_info.status_info.updated = update_available; apply_scanned_packages(&mut data.package_data, &current_data.package_data, &merged);
state.update_data(data).await; data.server_info.tor_address = tor_addr.clone();
debug!( data.server_info.node_address = tor_addr.as_ref().map(|t| identity.node_address(t));
"📦 State changed (packages={}, tor={}, first_scan={}, update={}), broadcasting update", data.server_info.status_info.containers_scanned = true;
changed, tor_changed, first_scan, update_changed data.server_info.status_info.updated = update_available;
); })
.await;
} }
Ok(()) Ok(())
} }
fn apply_scanned_packages(
latest: &mut HashMap<String, crate::data_model::PackageDataEntry>,
base: &HashMap<String, crate::data_model::PackageDataEntry>,
scanned: &HashMap<String, crate::data_model::PackageDataEntry>,
) {
for (id, fresh) in scanned {
if latest.get(id) == base.get(id) {
latest.insert(id.clone(), fresh.clone());
}
}
for id in base.keys() {
if !scanned.contains_key(id) && latest.get(id) == base.get(id) {
latest.remove(id);
}
}
}
async fn normalize_reachable_package_health( async fn normalize_reachable_package_health(
packages: &mut HashMap<String, crate::data_model::PackageDataEntry>, packages: &mut HashMap<String, crate::data_model::PackageDataEntry>,
) { ) {
@@ -2268,6 +2346,7 @@ mod merge_tests {
fn make_entry(state: PackageState, health: Option<&str>) -> PackageDataEntry { fn make_entry(state: PackageState, health: Option<&str>) -> PackageDataEntry {
PackageDataEntry { PackageDataEntry {
ui_ready: None,
state, state,
health: health.map(|s| s.to_string()), health: health.map(|s| s.to_string()),
exit_code: None, exit_code: None,
@@ -2280,6 +2359,37 @@ mod merge_tests {
} }
} }
#[test]
fn stale_scan_cannot_remove_new_installs_or_overwrite_lifecycle_changes() {
let running = make_entry(PackageState::Running, Some("healthy"));
let restarting = make_entry(PackageState::Restarting, None);
let base = [
("restart".into(), running.clone()),
("uninstalled".into(), running.clone()),
]
.into_iter()
.collect();
let mut latest = [
("restart".into(), restarting.clone()),
("new".into(), running.clone()),
]
.into_iter()
.collect();
let scanned = [
("restart".into(), running.clone()),
("uninstalled".into(), running.clone()),
]
.into_iter()
.collect();
apply_scanned_packages(&mut latest, &base, &scanned);
assert_eq!(latest.get("restart"), Some(&restarting));
assert_eq!(latest.get("new"), Some(&running));
assert!(!latest.contains_key("uninstalled"));
apply_scanned_packages(&mut latest, &base, &HashMap::new());
assert_eq!(latest.get("restart"), Some(&restarting));
assert!(latest.contains_key("new"));
}
#[test] #[test]
fn peer_path_filter_allows_content_catalog_and_items() { fn peer_path_filter_allows_content_catalog_and_items() {
// Regression: the content *catalog* is exactly "/content" (no trailing // Regression: the content *catalog* is exactly "/content" (no trailing
@@ -2452,3 +2562,97 @@ mod merge_tests {
); );
} }
} }
#[cfg(test)]
mod management_surface_tests {
use super::*;
#[test]
fn public_api_listener_rejects_management_despite_forged_headers() {
for peer in [
"198.18.0.2",
"2001:db8::2",
"::ffff:198.18.0.2",
"100.63.255.255",
"100.128.0.1",
] {
for path in ["/", "/login", "/assets/index.js", "/rpc/v1", "/ws"] {
for method in ["GET", "POST"] {
let request = hyper::Request::builder()
.uri(path)
.method(method)
.header("host", "127.0.0.1")
.header("x-forwarded-for", "127.0.0.1")
.header("x-real-ip", "192.168.1.10")
.body(hyper::Body::empty())
.unwrap();
assert!(
!request_surface_allowed(false, peer.parse().unwrap(), &request),
"{peer} {method} {path}"
);
}
}
}
}
#[test]
fn private_management_and_restricted_peer_transport_remain_available() {
let mut request = hyper::Request::builder()
.uri("/rpc/v1")
.body(hyper::Body::empty())
.unwrap();
for peer in [
"127.0.0.1",
"10.0.0.2",
"172.16.0.2",
"192.168.1.2",
"169.254.1.2",
"100.64.0.1",
"100.127.255.254",
"::1",
"fd00::1",
"fe80::1",
"::ffff:192.168.1.2",
] {
assert!(request_surface_allowed(
false,
peer.parse().unwrap(),
&request
));
}
request
.headers_mut()
.insert("x-archipelago-public-ingress", "1".parse().unwrap());
assert!(!request_surface_allowed(
false,
"127.0.0.1".parse().unwrap(),
&request
));
// The dedicated peer listener retains its existing signed RPC contract.
assert!(request_surface_allowed(
true,
"198.18.0.2".parse().unwrap(),
&request
));
for path in [
"/content",
"/content/fixture/invoice",
"/blob/fixture",
"/dwn/health",
"/archipelago/node-message",
] {
*request.uri_mut() = path.parse().unwrap();
assert!(request_surface_allowed(
false,
"198.18.0.2".parse().unwrap(),
&request
));
}
*request.uri_mut() = "/login".parse().unwrap();
assert!(!request_surface_allowed(
true,
"198.18.0.2".parse().unwrap(),
&request
));
}
}
@@ -0,0 +1,51 @@
//! Install-time pruning preference, shared by Bitcoin Core and Knots.
//! Missing preference preserves the existing disk-based automatic selection.
use anyhow::{Context, Result};
use serde::{Deserialize, Serialize};
use std::path::Path;
#[derive(Default, Serialize, Deserialize)]
pub struct BitcoinStorage {
pub prune: bool,
}
pub async fn load(data_dir: &Path) -> Result<BitcoinStorage> {
match tokio::fs::read(data_dir.join("settings/bitcoin-storage.json")).await {
Ok(bytes) => serde_json::from_slice(&bytes).context("Invalid Bitcoin storage settings"),
Err(e) if e.kind() == std::io::ErrorKind::NotFound => Ok(BitcoinStorage::default()),
Err(e) => Err(e.into()),
}
}
pub async fn save(data_dir: &Path, prune: bool) -> Result<()> {
let dir = data_dir.join("settings");
tokio::fs::create_dir_all(&dir).await?;
let path = dir.join("bitcoin-storage.json");
let temporary = dir.join("bitcoin-storage.json.tmp");
tokio::fs::write(&temporary, serde_json::to_vec(&BitcoinStorage { prune })?).await?;
tokio::fs::rename(temporary, path).await?;
Ok(())
}
#[cfg(test)]
mod tests {
use super::*;
#[tokio::test]
async fn missing_setting_keeps_auto_and_explicit_pruning_survives_reload() {
let dir = tempfile::tempdir().unwrap();
assert!(!load(dir.path()).await.unwrap().prune);
save(dir.path(), true).await.unwrap();
assert!(load(dir.path()).await.unwrap().prune);
save(dir.path(), false).await.unwrap();
assert!(!load(dir.path()).await.unwrap().prune);
}
#[tokio::test]
async fn corrupt_setting_is_not_silently_changed_to_archival() {
let dir = tempfile::tempdir().unwrap();
save(dir.path(), true).await.unwrap();
tokio::fs::write(dir.path().join("settings/bitcoin-storage.json"), "broken")
.await
.unwrap();
assert!(load(dir.path()).await.is_err());
}
}
+2
View File
@@ -7,3 +7,5 @@
pub mod ai_permissions; pub mod ai_permissions;
pub mod session_policy; pub mod session_policy;
pub mod transport; pub mod transport;
pub mod bitcoin_storage;
+41
View File
@@ -54,6 +54,21 @@ impl StateManager {
let _ = self.broadcast_tx.send(message); let _ = self.broadcast_tx.send(message);
} }
/// Apply a small state change while holding the write lock. A lifecycle
/// task must not replace the entire model from an earlier snapshot.
pub async fn mutate_data<T>(&self, change: impl FnOnce(&mut DataModel) -> T) -> T {
let mut data = self.data.write().await;
let result = change(&mut data);
let mut rev = self.revision.write().await;
*rev += 1;
let _ = self.broadcast_tx.send(WebSocketMessage {
rev: *rev,
data: Some(data.clone()),
patch: None,
});
result
}
/// Get a WebSocket message with the current state /// Get a WebSocket message with the current state
pub async fn get_initial_message(&self) -> WebSocketMessage { pub async fn get_initial_message(&self) -> WebSocketMessage {
let (data, rev) = self.get_snapshot().await; let (data, rev) = self.get_snapshot().await;
@@ -190,3 +205,29 @@ mod tests {
assert_eq!(rev, 1); assert_eq!(rev, 1);
} }
} }
#[cfg(test)]
mod atomic_mutation_tests {
use super::*;
#[tokio::test]
async fn concurrent_updates_preserve_independent_entries() {
let state = Arc::new(StateManager::new());
let mut tasks = Vec::new();
for i in 0..24 {
let state = state.clone();
tasks.push(tokio::spawn(async move {
state
.mutate_data(|data| {
data.peer_health.insert(format!("peer-{i}"), true);
})
.await;
}));
}
for task in tasks {
task.await.unwrap();
}
let (data, revision) = state.get_snapshot().await;
assert_eq!(data.peer_health.len(), 24);
assert_eq!(revision, 24);
}
}
+51
View File
@@ -1481,6 +1481,21 @@ pub async fn cancel_download(data_dir: &Path) -> Result<()> {
/// service unit that inherits systemd's default protections (i.e. none /// service unit that inherits systemd's default protections (i.e. none
/// of ours), escaping the namespace. /// of ours), escaping the namespace.
pub(crate) async fn host_sudo(args: &[&str]) -> Result<std::process::ExitStatus> { pub(crate) async fn host_sudo(args: &[&str]) -> Result<std::process::ExitStatus> {
#[cfg(test)]
{
anyhow::ensure!(
std::env::var("ARCHY_TEST_ISOLATED").as_deref() == Ok("1"),
"Host-operation tests require scripts/test-backend-isolated.sh"
);
let (program, args) = args.split_first().context("Missing test command")?;
// Run inside the test namespace, never escape through sudo/systemd-run.
return tokio::process::Command::new(program)
.args(args)
.status()
.await
.context("isolated test command failed");
}
let mut full: Vec<&str> = vec![ let mut full: Vec<&str> = vec![
"systemd-run", "systemd-run",
"--wait", "--wait",
@@ -1505,6 +1520,21 @@ pub(crate) async fn host_sudo(args: &[&str]) -> Result<std::process::ExitStatus>
/// Same mechanism as `host_sudo` but captures stdout — for read-only probes /// Same mechanism as `host_sudo` but captures stdout — for read-only probes
/// (e.g. `stat`) where the answer is in the output, not the exit status. /// (e.g. `stat`) where the answer is in the output, not the exit status.
pub(crate) async fn host_sudo_output(args: &[&str]) -> Result<std::process::Output> { pub(crate) async fn host_sudo_output(args: &[&str]) -> Result<std::process::Output> {
#[cfg(test)]
{
anyhow::ensure!(
std::env::var("ARCHY_TEST_ISOLATED").as_deref() == Ok("1"),
"Host-operation tests require scripts/test-backend-isolated.sh"
);
let (program, args) = args.split_first().context("Missing test command")?;
// Run inside the test namespace, never escape through sudo/systemd-run.
return tokio::process::Command::new(program)
.args(args)
.output()
.await
.context("isolated test command failed");
}
let mut full: Vec<&str> = vec![ let mut full: Vec<&str> = vec![
"systemd-run", "systemd-run",
"--wait", "--wait",
@@ -2029,6 +2059,25 @@ pub async fn rollback_update(data_dir: &Path) -> Result<()> {
let backup_binary = backup_dir.join("archipelago"); let backup_binary = backup_dir.join("archipelago");
if backup_binary.exists() { if backup_binary.exists() {
// The restored frontend can contain a pre-guard runtime template. An
// older binary copies that template verbatim on startup, undoing live
// containment. Protect it before permitting the binary downgrade.
let template = "/opt/archipelago/web-ui/archipelago-runtime/image-recipe/configs/nginx-archipelago.conf";
if Path::new(template).exists() {
let protected = host_sudo(&[
"python3",
"-c",
include_str!("../../../scripts/dashboard-public-guard.py"),
"--protect-template",
template,
])
.await
.context("protect nginx runtime template before rollback")?;
anyhow::ensure!(
protected.success(),
"unsafe nginx rollback template; previous binary not restored"
);
}
// Same two namespace gotchas as apply_update()'s binary swap: // Same two namespace gotchas as apply_update()'s binary swap:
// `cp` straight onto the running binary is O_TRUNC and fails // `cp` straight onto the running binary is O_TRUNC and fails
// ETXTBSY (exit 1 — exactly what broke the .116 rollback), and // ETXTBSY (exit 1 — exactly what broke the .116 rollback), and
@@ -2624,6 +2673,8 @@ mod tests {
#[test] #[test]
fn test_is_newer() { fn test_is_newer() {
assert!(is_newer("1.9.0-alpha", "1.8.22-alpha"));
assert!(!is_newer("1.9.0-alpha", "1.9.0-alpha"));
assert!(is_newer("1.7.19-alpha", "1.7.18-alpha")); assert!(is_newer("1.7.19-alpha", "1.7.18-alpha"));
assert!(is_newer("1.8.0-alpha", "1.7.99-alpha")); assert!(is_newer("1.8.0-alpha", "1.7.99-alpha"));
assert!(is_newer("1.7.10-alpha", "1.7.9-alpha")); // numeric, not lexical assert!(is_newer("1.7.10-alpha", "1.7.9-alpha")); // numeric, not lexical
+49 -58
View File
@@ -775,7 +775,9 @@ pub async fn send_token_at(data_dir: &Path, mint_url: &str, amount_sats: u64) ->
let mut all_target: Vec<u64> = send_denoms.clone(); let mut all_target: Vec<u64> = send_denoms.clone();
all_target.extend(&change_denoms); all_target.extend(&change_denoms);
let swap_result = client.swap(&selected_proofs, &all_target).await?; let swap_result = client
.swap_at_least(&selected_proofs, &all_target, amount_sats)
.await?;
// Mark original proofs as spent // Mark original proofs as spent
wallet.mark_spent(&indices); wallet.mark_spent(&indices);
@@ -1192,7 +1194,11 @@ pub async fn receive_token(data_dir: &Path, token_str: &str) -> Result<u64> {
// Verify all mints in the token are accepted // Verify all mints in the token are accepted
let accepted = load_accepted_mints(data_dir).await?; let accepted = load_accepted_mints(data_dir).await?;
for mint_url in token.mint_urls() { for mint_url in token.mint_urls() {
if !accepted.mints.iter().any(|m| m == mint_url) { if !accepted
.mints
.iter()
.any(|m| m.trim_end_matches('/') == mint_url.trim_end_matches('/'))
{
anyhow::bail!("Mint '{}' is not in accepted mints list", mint_url); anyhow::bail!("Mint '{}' is not in accepted mints list", mint_url);
} }
} }
@@ -1217,7 +1223,7 @@ pub async fn receive_token(data_dir: &Path, token_str: &str) -> Result<u64> {
received_total += amount; received_total += amount;
} }
Err(e) => { Err(e) => {
warn!("Failed to swap proofs from mint {}: {:#}", entry.mint, e); warn!("Failed to swap proofs from mint {}: {}", entry.mint, e);
all_already_redeemed &= e.is::<super::mint_client::AlreadyRedeemed>(); all_already_redeemed &= e.is::<super::mint_client::AlreadyRedeemed>();
last_reason = Some(e.to_string()); last_reason = Some(e.to_string());
// Continue with other mints if any // Continue with other mints if any
@@ -1298,22 +1304,10 @@ pub async fn verify_and_receive_payment(
token_str: &str, token_str: &str,
required_sats: u64, required_sats: u64,
) -> Result<u64> { ) -> Result<u64> {
// Handle legacy tokens let token_str = token_str.trim();
// Synthetic legacy balances are not cryptographic proof of payment.
if token_str.starts_with("cashuSend_") { if token_str.starts_with("cashuSend_") {
let amount = token_str anyhow::bail!("Legacy ecash cannot authorize a paid download");
.split('_')
.nth(1)
.and_then(|s| s.parse::<u64>().ok())
.unwrap_or(0);
if amount < required_sats {
anyhow::bail!(
"Insufficient payment: {} sats, need {} sats",
amount,
required_sats
);
}
let received = receive_legacy_token(data_dir, token_str).await?;
return Ok(received);
} }
// Fedimint notes (#3): a buyer whose balance is in Fedimint pays with notes // Fedimint notes (#3): a buyer whose balance is in Fedimint pays with notes
@@ -1336,52 +1330,45 @@ pub async fn verify_and_receive_payment(
// Parse and validate the token (cashuA or cashuB) // Parse and validate the token (cashuA or cashuB)
let token = CashuToken::deserialize(token_str)?; let token = CashuToken::deserialize(token_str)?;
let total = token.total_amount(); if token.unit.as_deref().unwrap_or("sat") != "sat" {
anyhow::bail!("Payment must be denominated in sats");
}
// A sale must redeem atomically at one mint. Otherwise a later mint
// failure can consume earlier inputs without delivering the purchase.
let entry = match token.token.as_slice() {
[entry] => entry,
_ => anyhow::bail!("Use a single-mint token for this payment"),
};
let total = entry
.proofs
.iter()
.try_fold(0u64, |sum, p| sum.checked_add(p.amount))
.ok_or_else(|| anyhow::anyhow!("Payment amount overflow"))?;
if total < required_sats { if total < required_sats {
anyhow::bail!( anyhow::bail!("Insufficient payment: {total} sats, need {required_sats} sats");
"Insufficient payment: {} sats, need {} sats",
total,
required_sats
);
} }
// Verify mints are accepted
let accepted = load_accepted_mints(data_dir).await?; let accepted = load_accepted_mints(data_dir).await?;
for mint_url in token.mint_urls() { if !accepted
if !accepted.mints.iter().any(|m| m == mint_url) { .mints
anyhow::bail!("Mint '{}' not accepted", mint_url); .iter()
} .any(|m| m.trim_end_matches('/') == entry.mint.trim_end_matches('/'))
{
anyhow::bail!("Mint is not in the seller's accepted mints list");
} }
// Swap proofs at mint (this verifies they're unspent and gives us fresh proofs) let client = mint_client(data_dir, &entry.mint).await?;
let result = client
.swap_at_least(
&entry.proofs,
&amount_to_denominations(total),
required_sats,
)
.await?;
let received_total = result.new_proofs.iter().map(|p| p.amount).sum();
// Load after the network call, so an unrelated wallet update during the
// swap is not overwritten with a pre-swap snapshot.
let mut wallet = load_wallet(data_dir).await?; let mut wallet = load_wallet(data_dir).await?;
let mut received_total = 0u64; wallet.add_proofs(entry.mint.trim_end_matches('/'), result.new_proofs);
for entry in &token.token {
let client = mint_client(data_dir, &entry.mint).await?;
let entry_total: u64 = entry.proofs.iter().map(|p| p.amount).sum();
let target_amounts = amount_to_denominations(entry_total);
match client.swap(&entry.proofs, &target_amounts).await {
Ok(result) => {
let amount: u64 = result.new_proofs.iter().map(|p| p.amount).sum();
wallet.add_proofs(&entry.mint, result.new_proofs);
received_total += amount;
}
Err(e) => {
warn!("Payment verification failed at mint {}: {}", entry.mint, e);
}
}
}
if received_total < required_sats {
anyhow::bail!(
"Payment verification failed: only {} of {} sats verified",
received_total,
required_sats
);
}
wallet.record_tx( wallet.record_tx(
TransactionType::Receive, TransactionType::Receive,
@@ -2465,3 +2452,7 @@ mod tests {
assert_eq!(w.mint_url, "https://mint.minibits.cash/Bitcoin"); assert_eq!(w.mint_url, "https://mint.minibits.cash/Bitcoin");
} }
} }
#[cfg(test)]
#[path = "payment_tests.rs"]
mod payment_tests;
+77 -32
View File
@@ -153,6 +153,20 @@ fn mint_error(op: &str, status: reqwest::StatusCode, body: &str) -> anyhow::Erro
cause.context(describe_mint_error_body(status, body)) cause.context(describe_mint_error_body(status, body))
} }
fn fee_adjusted_targets(requested: &[u64], mut available: u64) -> Vec<u64> {
let mut outputs = Vec::new();
for &amount in requested {
if available >= amount {
outputs.push(amount);
available -= amount;
} else {
outputs.extend(amount_to_denominations(available));
break;
}
}
outputs
}
/// HTTP client for a single Cashu mint. /// HTTP client for a single Cashu mint.
pub struct MintClient { pub struct MintClient {
url: String, url: String,
@@ -512,6 +526,21 @@ impl MintClient {
/// Swap proofs for new proofs of different denominations. /// Swap proofs for new proofs of different denominations.
/// This is how we "receive" a token — swap it for fresh proofs that only we know. /// This is how we "receive" a token — swap it for fresh proofs that only we know.
pub async fn swap(&self, inputs: &[Proof], target_amounts: &[u64]) -> Result<SwapResult> { pub async fn swap(&self, inputs: &[Proof], target_amounts: &[u64]) -> Result<SwapResult> {
self.swap_at_least(inputs, target_amounts, 0).await
}
/// Refuse a payment whose mint fees would leave the seller underpaid,
/// before consuming any input proofs.
pub async fn swap_at_least(
&self,
inputs: &[Proof],
target_amounts: &[u64],
minimum: u64,
) -> Result<SwapResult> {
// V4 tokens carry short keyset IDs. Every swap path (including paid
// files and streams) must expand these, not only wallet imports.
let resolved = self.resolve_truncated_keyset_ids(inputs).await?;
let inputs = resolved.as_slice();
let keyset = self.get_active_sat_keyset().await?; let keyset = self.get_active_sat_keyset().await?;
// NUT-02: a mint may charge a per-input fee, and it rejects the swap // NUT-02: a mint may charge a per-input fee, and it rejects the swap
@@ -519,16 +548,35 @@ impl MintClient {
// should equal outputs less fee`). Applied here rather than at each // should equal outputs less fee`). Applied here rather than at each
// call site so send, receive and cross-mint swaps are all covered. // call site so send, receive and cross-mint swaps are all covered.
// Fee-free mints (Minibits) compute 0 and are unaffected. // Fee-free mints (Minibits) compute 0 and are unaffected.
let inputs_total: u64 = inputs.iter().map(|p| p.amount).sum(); anyhow::ensure!(!inputs.is_empty(), "No input proofs to swap");
let fee = match self.get_keysets().await { let inputs_total = inputs
Ok(ks) => super::cashu::swap_fee_for(inputs, &ks), .iter()
Err(e) => { .try_fold(0u64, |sum, p| sum.checked_add(p.amount))
debug!("Could not read keyset fees ({e:#}) — assuming fee-free mint"); .context("Input amount overflow")?;
0 let keysets = self.get_keysets().await?;
} let mut fee_ppk = 0u64;
}; for proof in inputs {
let input_keyset = keysets
.iter()
.find(|k| k.id == proof.id)
.context("The mint does not recognize an input keyset")?;
anyhow::ensure!(
input_keyset.unit == "sat",
"Input keyset is not denominated in sats"
);
fee_ppk = fee_ppk
.checked_add(input_keyset.input_fee_ppk)
.context("Mint fee overflow")?;
}
let fee = fee_ppk.div_ceil(1000);
let spendable = inputs_total.saturating_sub(fee); let spendable = inputs_total.saturating_sub(fee);
let requested: u64 = target_amounts.iter().sum(); if spendable < minimum {
anyhow::bail!("Payment would leave {spendable} sats after mint fees; need {minimum} sats. No proofs were redeemed.");
}
let requested = target_amounts
.iter()
.try_fold(0u64, |sum, amount| sum.checked_add(*amount))
.context("Output amount overflow")?;
let owned_targets: Vec<u64>; let owned_targets: Vec<u64>;
let target_amounts: &[u64] = if requested > spendable { let target_amounts: &[u64] = if requested > spendable {
if spendable == 0 { if spendable == 0 {
@@ -539,7 +587,10 @@ impl MintClient {
debug!( debug!(
"Reducing swap outputs {requested} -> {spendable} to cover a {fee} sat mint fee" "Reducing swap outputs {requested} -> {spendable} to cover a {fee} sat mint fee"
); );
owned_targets = amount_to_denominations(spendable); // Callers put payment outputs before change. Keep that prefix
// intact while fees reduce change; re-splitting the entire sum
// can omit a payment denomination after consuming the inputs.
owned_targets = fee_adjusted_targets(target_amounts, spendable);
&owned_targets &owned_targets
} else { } else {
target_amounts target_amounts
@@ -584,6 +635,9 @@ impl MintClient {
let mut new_proofs = Vec::new(); let mut new_proofs = Vec::new();
for (sig, (secret, r, amount)) in signatures.iter().zip(blinding_data.iter()) { for (sig, (secret, r, amount)) in signatures.iter().zip(blinding_data.iter()) {
if sig.amount != *amount || sig.id != keyset.id {
anyhow::bail!("Mint returned a swap signature for an unexpected amount or keyset");
}
let c_prime = sig.c_prime_as_pubkey()?; let c_prime = sig.c_prime_as_pubkey()?;
let mint_key = keyset.key_for_amount(*amount)?; let mint_key = keyset.key_for_amount(*amount)?;
let c = bdhke::unblind_signature(&c_prime, r, &mint_key)?; let c = bdhke::unblind_signature(&c_prime, r, &mint_key)?;
@@ -730,43 +784,35 @@ impl MintClient {
/// Repair proofs whose keyset id is a truncated NUT-02 **v2** id. /// Repair proofs whose keyset id is a truncated NUT-02 **v2** id.
/// ///
/// A v2 keyset id is 33 bytes (version byte `0x01` + 32-byte hash), but /// A v2 keyset id is 33 bytes (version byte `0x01` + 32-byte hash), but
/// wallets written against the original 8-byte format truncate it when /// compact V4 tokens carry an 8-byte short ID. The swap endpoint needs
/// they build a token. The mint then reads the `0x01` version, expects 33 /// the full ID restored from the mint's keyset list. The mint then reads the `0x01` version, expects 33
/// bytes, and rejects the swap — reported as /// bytes, and rejects the swap — reported as
/// `inputs[0].id: NUT02: ID length invalid` behind a bare 422 (seen with /// `inputs[0].id: NUT02: ID length invalid` behind a bare 422 (seen with
/// a Minibits-issued token, 2026-08-17). /// a Minibits-issued token, 2026-08-17).
/// ///
/// The id only names which keyset signed the proof, so restoring the full /// The id only names which keyset signed the proof, so restoring the full
/// id the mint advertises is exactly what the sender meant. It is also /// id the mint advertises is exactly what the sender meant. It is also
/// safe to attempt: an id that names the wrong keyset fails signature /// safe to attempt: the mint still verifies the proof signature. Unknown
/// verification at the mint and no coins move. Anything already valid, or /// or ambiguous short IDs are rejected before redemption.
/// with no unambiguous match, is passed through untouched so the mint's async fn resolve_truncated_keyset_ids(&self, proofs: &[Proof]) -> Result<Vec<Proof>> {
/// own error is what the operator sees.
async fn resolve_truncated_keyset_ids(&self, proofs: &[Proof]) -> Vec<Proof> {
let needs_repair = proofs.iter().any(|p| is_truncated_v2_keyset_id(&p.id)); let needs_repair = proofs.iter().any(|p| is_truncated_v2_keyset_id(&p.id));
if !needs_repair { if !needs_repair {
return proofs.to_vec(); return Ok(proofs.to_vec());
} }
// The mint's own keyset list, in the reference implementation's shape // The mint's own keyset list, in the reference implementation's shape
// so its NUT-02 resolver can consume it directly. // so its NUT-02 resolver can consume it directly.
let known = match self.get_cdk_keysets().await { let known = self.get_cdk_keysets().await?;
Ok(k) => k,
Err(e) => {
debug!("Could not list keysets to repair truncated keyset ids: {e:#}");
return proofs.to_vec();
}
};
proofs proofs
.iter() .iter()
.cloned() .cloned()
.map(|mut p| { .map(|mut p| {
if let Some(full) = super::cashu::resolve_keyset_id(&p.id, &known) { if is_truncated_v2_keyset_id(&p.id) {
debug!("Expanded short keyset id {} to {} for swap", p.id, full); p.id = super::cashu::resolve_keyset_id(&p.id, &known)
p.id = full; .context("The mint cannot resolve this short keyset ID unambiguously")?;
} }
p Ok(p)
}) })
.collect() .collect()
} }
@@ -802,7 +848,7 @@ impl MintClient {
let mut all_new_proofs = Vec::new(); let mut all_new_proofs = Vec::new();
for entry in &token.token { for entry in &token.token {
if entry.mint != self.url { if entry.mint.trim_end_matches('/') != self.url {
debug!( debug!(
"Skipping proofs from different mint {} (ours: {})", "Skipping proofs from different mint {} (ours: {})",
entry.mint, self.url entry.mint, self.url
@@ -813,8 +859,7 @@ impl MintClient {
let total: u64 = entry.proofs.iter().map(|p| p.amount).sum(); let total: u64 = entry.proofs.iter().map(|p| p.amount).sum();
let target_amounts = amount_to_denominations(total); let target_amounts = amount_to_denominations(total);
let proofs = self.resolve_truncated_keyset_ids(&entry.proofs).await; let result = self.swap(&entry.proofs, &target_amounts).await?;
let result = self.swap(&proofs, &target_amounts).await?;
all_new_proofs.extend(result.new_proofs); all_new_proofs.extend(result.new_proofs);
} }
@@ -0,0 +1,428 @@
//! Real HTTP/curve-signature regressions for paid Cashu redemption.
use super::*;
use crate::wallet::{bdhke, cashu::Proof};
use bitcoin::secp256k1::{PublicKey, Scalar, Secp256k1, SecretKey};
use hyper::{
service::{make_service_fn, service_fn},
Body, Request, Response, Server,
};
use serde_json::{json, Value};
use std::{
convert::Infallible,
sync::{Arc, Mutex},
};
const ACTIVE: &str = "0011223344556677";
const V2: &str = "011111111111111111111111111111111111111111111111111111111111111111";
struct Mint {
url: String,
requests: Arc<Mutex<Vec<Value>>>,
task: tokio::task::JoinHandle<()>,
failure: Arc<std::sync::atomic::AtomicU16>,
}
impl Drop for Mint {
fn drop(&mut self) {
self.task.abort();
}
}
fn signing_key() -> SecretKey {
SecretKey::from_slice(&[7; 32]).unwrap()
}
fn signed_point(point: PublicKey) -> String {
point
.mul_tweak(&Secp256k1::new(), &Scalar::from(signing_key()))
.unwrap()
.to_string()
}
fn proof(id: &str, amount: u64) -> Proof {
let secret = format!("test-{id}-{amount}");
Proof {
amount,
id: id.into(),
c: signed_point(bdhke::hash_to_curve(secret.as_bytes()).unwrap()),
secret,
}
}
impl Mint {
async fn start(fee: u64, failure: Option<u16>) -> Self {
let listener = std::net::TcpListener::bind("127.0.0.1:0").unwrap();
listener.set_nonblocking(true).unwrap();
let url = format!("http://{}", listener.local_addr().unwrap());
let requests = Arc::new(Mutex::new(Vec::new()));
let seen = requests.clone();
let failure = Arc::new(std::sync::atomic::AtomicU16::new(failure.unwrap_or(0)));
let rejection = failure.clone();
let spent = Arc::new(Mutex::new(std::collections::HashSet::<String>::new()));
let service = make_service_fn(move |_| {
let seen = seen.clone();
let rejection = rejection.clone();
let spent = spent.clone();
async move {
Ok::<_, Infallible>(service_fn(move |req: Request<Body>| {
let seen = seen.clone();
let rejection = rejection.clone();
let spent = spent.clone();
async move {
let mut status = 200;
let body = match req.uri().path() {
"/v1/keysets" => json!({"keysets":[
{"id": ACTIVE,"unit":"sat","active":true,"input_fee_ppk":fee},
{"id": V2,"unit":"sat","active":false,"input_fee_ppk":fee}
]}),
"/v1/keys" => {
let public =
PublicKey::from_secret_key(&Secp256k1::new(), &signing_key())
.to_string();
let keys: serde_json::Map<String, Value> = (0..16)
.map(|i| ((1u64 << i).to_string(), json!(public)))
.collect();
json!({"keysets":[{"id": ACTIVE,"unit":"sat","keys":keys}]})
}
"/v1/swap" => {
let body: Value = serde_json::from_slice(
&hyper::body::to_bytes(req.into_body()).await.unwrap(),
)
.unwrap();
seen.lock().unwrap().push(body.clone());
let inputs = body["inputs"].as_array().unwrap();
let outputs = body["outputs"].as_array().unwrap();
let code = rejection.load(std::sync::atomic::Ordering::SeqCst);
if code != 0 {
status = code;
json!({"detail":"mock mint rejection"})
} else if inputs.iter().any(|p| p["id"] != V2 && p["id"] != ACTIVE)
{
status = 422;
json!({"detail":[{"msg":"NUT02: ID length invalid"}]})
} else if inputs.iter().any(|p| {
spent
.lock()
.unwrap()
.contains(p["secret"].as_str().unwrap())
}) {
status = 400;
json!({"code":11001,"detail":"Token Already Spent"})
} else {
let total: u64 =
inputs.iter().map(|p| p["amount"].as_u64().unwrap()).sum();
let out: u64 =
outputs.iter().map(|p| p["amount"].as_u64().unwrap()).sum();
assert_eq!(
out,
total - (inputs.len() as u64 * fee).div_ceil(1000)
);
for p in inputs {
spent
.lock()
.unwrap()
.insert(p["secret"].as_str().unwrap().into());
}
json!({"signatures":outputs.iter().map(|o| json!({
"amount":o["amount"],"id":ACTIVE,
"C_":signed_point(o["B_"].as_str().unwrap().parse().unwrap())
})).collect::<Vec<_>>()})
}
}
_ => {
status = 404;
json!({})
}
};
Ok::<_, Infallible>(
Response::builder()
.status(status)
.header("Content-Type", "application/json")
.body(Body::from(body.to_string()))
.unwrap(),
)
}
}))
}
});
let server = Server::from_tcp(listener).unwrap().serve(service);
let task = tokio::spawn(async move {
server.await.unwrap();
});
Self {
url,
requests,
task,
failure,
}
}
async fn wallet(&self) -> tempfile::TempDir {
let dir = tempfile::tempdir().unwrap();
save_accepted_mints(
dir.path(),
&AcceptedMints {
mints: vec![format!("{}/", self.url)],
},
)
.await
.unwrap();
dir
}
}
#[tokio::test]
async fn paid_v4_inactive_v2_keyset_is_expanded_and_cryptographic_proofs_saved() {
let mint = Mint::start(0, None).await;
let dir = mint.wallet().await;
let token = CashuToken::new(&mint.url, vec![proof(V2, 64), proof(V2, 32), proof(V2, 4)])
.serialize_v4()
.unwrap();
let decoded = CashuToken::deserialize(&token).unwrap();
assert_eq!(
decoded.token[0].proofs[0].id.len(),
16,
"reproduce the short V4 ID"
);
assert_eq!(
verify_and_receive_payment(dir.path(), &token, 100)
.await
.unwrap(),
100
);
let wallet = load_wallet(dir.path()).await.unwrap();
assert_eq!(wallet.balance(), 100);
for p in wallet.proofs {
assert_eq!(
p.proof.c,
signed_point(bdhke::hash_to_curve(p.proof.secret.as_bytes()).unwrap())
);
}
assert!(mint.requests.lock().unwrap()[0]["inputs"]
.as_array()
.unwrap()
.iter()
.all(|p| p["id"] == V2));
assert!(verify_and_receive_payment(dir.path(), &token, 100)
.await
.is_err());
assert_eq!(load_wallet(dir.path()).await.unwrap().balance(), 100);
}
#[tokio::test]
async fn paid_v3_full_v2_and_v1_ids_work() {
for id in [V2, ACTIVE] {
let mint = Mint::start(0, None).await;
let dir = mint.wallet().await;
let token = CashuToken::new(&mint.url, vec![proof(id, 128)])
.serialize()
.unwrap();
assert_eq!(
verify_and_receive_payment(dir.path(), &token, 100)
.await
.unwrap(),
128
);
}
}
#[tokio::test]
async fn fees_cannot_consume_underpayment_and_allowed_fees_credit_actual_value() {
let mint = Mint::start(1000, None).await;
let dir = mint.wallet().await;
let token = CashuToken::new(&mint.url, vec![proof(V2, 128)])
.serialize_v4()
.unwrap();
assert!(verify_and_receive_payment(dir.path(), &token, 128)
.await
.unwrap_err()
.to_string()
.contains("after mint fees"));
assert!(mint.requests.lock().unwrap().is_empty());
assert_eq!(
verify_and_receive_payment(dir.path(), &token, 127)
.await
.unwrap(),
127
);
assert_eq!(load_wallet(dir.path()).await.unwrap().balance(), 127);
}
#[tokio::test]
async fn rejected_mint_response_does_not_credit_wallet() {
for status in [200, 400, 422, 500, 503] {
let mint = Mint::start(0, Some(status)).await;
let dir = mint.wallet().await;
let token = CashuToken::new(&mint.url, vec![proof(V2, 128)])
.serialize_v4()
.unwrap();
assert!(verify_and_receive_payment(dir.path(), &token, 100)
.await
.is_err());
assert_eq!(load_wallet(dir.path()).await.unwrap().balance(), 0);
}
}
#[tokio::test]
async fn invalid_untrusted_multimint_and_underpaid_tokens_never_reach_swap() {
let mint = Mint::start(0, None).await;
let dir = mint.wallet().await;
let token = CashuToken::new(&mint.url, vec![proof(V2, 128)]);
let mut invalid = vec![
"cashuSend_500_abc_1700000000".into(),
"cashuBinvalid".into(),
];
let mut wrong_unit = token.clone();
wrong_unit.unit = Some("usd".into());
invalid.push(wrong_unit.serialize().unwrap());
let mut multi = token.clone();
multi.token.push(token.token[0].clone());
invalid.push(multi.serialize().unwrap());
let mut untrusted = token.clone();
untrusted.token[0].mint = "http://127.0.0.1:1".into();
invalid.push(untrusted.serialize().unwrap());
for id in ["00ffffffffffffff", "01ffffffffffffff"] {
invalid.push(
CashuToken::new(&mint.url, vec![proof(id, 128)])
.serialize()
.unwrap(),
);
}
for value in invalid {
assert!(verify_and_receive_payment(dir.path(), &value, 100)
.await
.is_err());
}
assert!(
verify_and_receive_payment(dir.path(), &token.serialize().unwrap(), 129)
.await
.is_err()
);
assert!(mint.requests.lock().unwrap().is_empty());
assert_eq!(load_wallet(dir.path()).await.unwrap().balance(), 0);
}
#[tokio::test]
async fn buyer_token_rejected_by_seller_can_be_refunded_without_balance_loss() {
let mint = Mint::start(0, Some(422)).await;
let buyer = mint.wallet().await;
let seller = mint.wallet().await;
let mut wallet = load_wallet(buyer.path()).await.unwrap();
wallet.mint_url = mint.url.clone();
wallet.add_proofs(&mint.url, vec![proof(V2, 64), proof(V2, 32), proof(V2, 4)]);
save_wallet(buyer.path(), &wallet).await.unwrap();
let token = send_token(buyer.path(), 100).await.unwrap();
assert_eq!(load_wallet(buyer.path()).await.unwrap().balance(), 0);
assert!(verify_and_receive_payment(seller.path(), &token, 100)
.await
.is_err());
mint.failure.store(0, std::sync::atomic::Ordering::SeqCst);
assert_eq!(receive_token(buyer.path(), &token).await.unwrap(), 100);
assert_eq!(load_wallet(buyer.path()).await.unwrap().balance(), 100);
assert_eq!(load_wallet(seller.path()).await.unwrap().balance(), 0);
assert!(receive_token(buyer.path(), &token).await.is_err());
assert_eq!(load_wallet(buyer.path()).await.unwrap().balance(), 100);
}
#[tokio::test]
async fn unreachable_mint_does_not_credit_seller() {
let mint = Mint::start(0, None).await;
let dir = mint.wallet().await;
let token = CashuToken::new(&mint.url, vec![proof(V2, 128)])
.serialize_v4()
.unwrap();
mint.task.abort();
tokio::task::yield_now().await;
assert!(verify_and_receive_payment(dir.path(), &token, 100)
.await
.is_err());
assert_eq!(load_wallet(dir.path()).await.unwrap().balance(), 0);
}
#[tokio::test]
async fn send_with_fees_preserves_payment_denominations_and_saves_change() {
// 128 inputs - 2 fee = 126. Splitting 126 as one sum omits 1,
// which is needed for a 65-sat payment, after consuming the inputs.
let mint = Mint::start(1000, None).await;
let buyer = mint.wallet().await;
let mut wallet = load_wallet(buyer.path()).await.unwrap();
wallet.mint_url = mint.url.clone();
let first = proof(V2, 64);
let mut second = first.clone();
second.secret.push_str("-second");
second.c = signed_point(bdhke::hash_to_curve(second.secret.as_bytes()).unwrap());
wallet.add_proofs(&mint.url, vec![first, second]);
save_wallet(buyer.path(), &wallet).await.unwrap();
let encoded = send_token(buyer.path(), 65).await.unwrap();
assert_eq!(
CashuToken::deserialize(&encoded).unwrap().total_amount(),
65
);
assert_eq!(load_wallet(buyer.path()).await.unwrap().balance(), 61);
}
#[tokio::test]
async fn paid_file_gate_delivers_bytes_only_after_payment_and_does_not_charge_missing_files() {
use crate::content_server::{
self, AccessControl, Availability, ContentCatalog, ContentItem, ServeResult,
};
for (exists, accepts_cashu, price) in [
(true, true, 100),
(true, false, 100),
(false, true, 100),
(true, true, 129),
] {
let mint = Mint::start(0, None).await;
let seller = mint.wallet().await;
let item = ContentItem {
id: "paid-test".into(),
filename: "test.txt".into(),
mime_type: "text/plain".into(),
size_bytes: 5,
description: String::new(),
added_at: String::new(),
availability: Availability::AllPeers,
access: AccessControl::Paid {
price_sats: price,
accepted: vec![if accepts_cashu { "ecash" } else { "fedimint" }.into()],
},
};
content_server::save_catalog(seller.path(), &ContentCatalog { items: vec![item] })
.await
.unwrap();
if exists {
tokio::fs::create_dir_all(seller.path().join("content/files"))
.await
.unwrap();
tokio::fs::write(seller.path().join("content/files/test.txt"), b"hello")
.await
.unwrap();
}
let token = CashuToken::new(&mint.url, vec![proof(V2, 128)])
.serialize_v4()
.unwrap();
let result = content_server::serve_content(
seller.path(),
"paid-test",
Some(&token),
None,
None,
None,
false,
)
.await
.unwrap();
if exists && accepts_cashu && price <= 128 {
match result {
ServeResult::Ok(bytes, mime) => {
assert_eq!(bytes, b"hello");
assert_eq!(mime, "text/plain");
}
_ => panic!("paid content was not delivered"),
}
assert_eq!(load_wallet(seller.path()).await.unwrap().balance(), 128);
} else {
assert!(matches!(
result,
ServeResult::NotFound | ServeResult::PaymentRequired(_)
));
assert_eq!(load_wallet(seller.path()).await.unwrap().balance(), 0);
assert!(mint.requests.lock().unwrap().is_empty());
}
}
}
+143 -32
View File
@@ -263,7 +263,8 @@ pub struct ContainerConfig {
/// Derived-env entry. The template is rendered against `HostFacts` at /// Derived-env entry. The template is rendered against `HostFacts` at
/// apply time; exactly one `{{PLACEHOLDER}}` occurrence per supported /// apply time; exactly one `{{PLACEHOLDER}}` occurrence per supported
/// fact name is allowed (host_ip, host_mdns, disk_gb). /// fact name is allowed (host_ip, host_mdns, disk_gb, bitcoin_host,
/// node_identity_pubkeys).
#[derive(Debug, Clone, Serialize, Deserialize, PartialEq, Eq)] #[derive(Debug, Clone, Serialize, Deserialize, PartialEq, Eq)]
pub struct DerivedEnv { pub struct DerivedEnv {
pub key: String, pub key: String,
@@ -989,6 +990,24 @@ impl AppManifest {
validate_security(&self.app.security)?; validate_security(&self.app.security)?;
validate_ports(&self.app.ports)?; validate_ports(&self.app.ports)?;
validate_interfaces(&self.app.interfaces)?; validate_interfaces(&self.app.interfaces)?;
if let Some(value) = self.app.extensions.get("install_prerequisites") {
let items = value.as_sequence().ok_or_else(|| {
ManifestError::Invalid("install_prerequisites must be a list of app ids".into())
})?;
for item in items {
let id = item.as_str().unwrap_or_default();
if id.is_empty()
|| id == self.app.id
|| !id
.bytes()
.all(|b| b.is_ascii_lowercase() || b.is_ascii_digit() || b == b'-')
{
return Err(ManifestError::Invalid(
"install_prerequisites must contain valid other app ids".into(),
));
}
}
}
validate_environment(&self.app.environment)?; validate_environment(&self.app.environment)?;
validate_devices(&self.app.devices)?; validate_devices(&self.app.devices)?;
@@ -1074,6 +1093,14 @@ impl AppManifest {
// `..` copy sources). See docs/manifest-hooks-design.md. // `..` copy sources). See docs/manifest-hooks-design.md.
self.app.hooks.validate()?; self.app.hooks.validate()?;
if let Some(value) = self.app.extensions.get("backup_before_runtime_change") {
if value.as_bool().is_none() {
return Err(ManifestError::Invalid(
"backup_before_runtime_change must be boolean".into(),
));
}
}
Ok(()) Ok(())
} }
} }
@@ -1111,6 +1138,7 @@ fn validate_security(policy: &SecurityPolicy) -> Result<(), ManifestError> {
"SETGID", "SETGID",
"SETUID", "SETUID",
"SYS_ADMIN", "SYS_ADMIN",
"SYS_CHROOT",
]; ];
let mut seen = HashSet::new(); let mut seen = HashSet::new();
for cap in &policy.capabilities { for cap in &policy.capabilities {
@@ -1401,6 +1429,13 @@ pub struct HostFacts {
/// right host. Both are reachable on archy-net by their container name; /// right host. Both are reachable on archy-net by their container name;
/// only the name differs. Falls back to `bitcoin-knots` when undetected. /// only the name differs. Falls back to `bitcoin-knots` when undetected.
pub bitcoin_host: String, pub bitcoin_host: String,
/// Nostr public keys of the node's identities that the app identity
/// picker offers for signing (the node's own appliance key excluded),
/// as comma-joined, sorted, lowercase 64-char hex. Lets an app grant
/// the node's users owner rights (e.g. a Blossom server's allowed
/// uploaders). Empty unless a manifest templates it; the orchestrator
/// resolves it on demand and refuses to render an empty set.
pub node_identity_pubkeys: String,
} }
impl HostFacts { impl HostFacts {
@@ -1412,13 +1447,20 @@ impl HostFacts {
host_mdns: "test-node.local".to_string(), host_mdns: "test-node.local".to_string(),
disk_gb: 2000, disk_gb: 2000,
bitcoin_host: "bitcoin-knots".to_string(), bitcoin_host: "bitcoin-knots".to_string(),
node_identity_pubkeys: "1111111111111111111111111111111111111111111111111111111111111111,2222222222222222222222222222222222222222222222222222222222222222".to_string(),
} }
} }
} }
/// Supported placeholder names in `DerivedEnv::template`. Keep in sync /// Supported placeholder names in `DerivedEnv::template`. Keep in sync
/// with `HostFacts`. Centralized so validation and rendering agree. /// with `HostFacts`. Centralized so validation and rendering agree.
const DERIVED_PLACEHOLDERS: &[&str] = &["HOST_IP", "HOST_MDNS", "DISK_GB", "BITCOIN_HOST"]; const DERIVED_PLACEHOLDERS: &[&str] = &[
"HOST_IP",
"HOST_MDNS",
"DISK_GB",
"BITCOIN_HOST",
"NODE_IDENTITY_PUBKEYS",
];
fn validate_derived_template(key: &str, template: &str) -> Result<(), ManifestError> { fn validate_derived_template(key: &str, template: &str) -> Result<(), ManifestError> {
// Walk `{{NAME}}` occurrences and ensure each NAME is recognized. // Walk `{{NAME}}` occurrences and ensure each NAME is recognized.
@@ -1502,7 +1544,8 @@ impl ContainerConfig {
.replace("{{HOST_IP}}", &facts.host_ip) .replace("{{HOST_IP}}", &facts.host_ip)
.replace("{{HOST_MDNS}}", &facts.host_mdns) .replace("{{HOST_MDNS}}", &facts.host_mdns)
.replace("{{DISK_GB}}", &facts.disk_gb.to_string()) .replace("{{DISK_GB}}", &facts.disk_gb.to_string())
.replace("{{BITCOIN_HOST}}", &facts.bitcoin_host); .replace("{{BITCOIN_HOST}}", &facts.bitcoin_host)
.replace("{{NODE_IDENTITY_PUBKEYS}}", &facts.node_identity_pubkeys);
format!("{}={}", e.key, value) format!("{}={}", e.key, value)
}) })
.collect() .collect()
@@ -1746,40 +1789,48 @@ app:
} }
} }
exempt.sort(); exempt.sort();
// 28 as of 2026-08-23: the 26 below plus cuprate's two exemptions — // Reviewed 2026-09-30: lightning-stack's three retired endpoints
// 18183 (Monero p2p gossip, same reasoning as bitcoin's 8333) and // disappeared; Cuprate restricted RPC moved from none to gate-open.
// 18090 (host mapping for Monero's canonical 18089 restricted RPC, // Compare exact endpoints, not just a count that can hide substitutions.
// upstream's own safe-for-public let expected = [
// subset that wallets connect to directly as a "remote node" over ("bitcoin-core", 8333),
// plain HTTP JSON-RPC — same reasoning as electrumx's 50001). ("bitcoin-knots", 8333),
// cuprate's unrestricted RPC (full node control) stays loopback-only ("core-lightning", 9736),
// (auth: local), not in this set. ("core-lightning", 9835),
// ("cuprate", 18183),
// 26 as of 2026-08-16: the 25 below plus phoenixd 9740, a ("electrumx", 50001),
// loopback-only JSON API whose own generated http password ("fedimint", 8173),
// authenticates every request (added with the phoenixd onboarding, ("fedimint", 8174),
// which did not update this count — exactly the drift this test ("fedimint-gateway", 8176),
// exists to catch). ("fedimint-gateway", 9737),
// ("gitea", 2222),
// 25 as of the v1.7.123 port-policy round: bitcoin p2p (8333 ×2), ("lnd", 9735),
// core-lightning 9736/9835, electrumx 50001, fedimint 8173/8174, ("lnd", 10009),
// fedimint-gateway 8176/9737, gitea ssh 2222, lightning-stack ("lnd", 18080),
// 8091/9738/10010, lnd 9735/10009/18080, netbird 3478/8086/8087, ("netbird", 8087),
// pine TLS 10381 + the three voice ports (10200/10300/10400 — the ("netbird-server", 3478),
// disclosed known gap), router SSDP/mDNS 1900/5353. Every one is a ("netbird-server", 8086),
// deliberate, rationale-carrying exemption; the release-gate test ("phoenixd", 9740),
// stage timed out that cycle, so the count here lagged at 17. ("pine", 10381),
("pine-openwakeword", 10400),
("pine-piper", 10200),
("pine-whisper", 10300),
("router", 1900),
("router", 5353),
]
.into_iter()
.map(|(id, port)| (id.to_owned(), port))
.collect::<Vec<_>>();
assert_eq!( assert_eq!(
exempt.len(), exempt, expected,
28, "unauthenticated endpoint set changed; review each exemption"
"unauthenticated port set changed — review before updating this count: {exempt:?}"
); );
} }
/// `auth: open` ports are served by the gate WITHOUT its login challenge, /// `auth: open` ports are served by the gate WITHOUT its login challenge,
/// so they are the second unauthenticated-by-the-gate surface and get the /// so they are the second unauthenticated-by-the-gate surface and get the
/// same review guard as `auth: none`. Each one must be an app that /// same review guard as `auth: none`. Each must enforce its own login or
/// enforces a real login of its own. /// have an explicitly reviewed public protocol purpose.
#[test] #[test]
fn gate_open_ports_are_all_accounted_for() { fn gate_open_ports_are_all_accounted_for() {
let apps = std::path::Path::new(env!("CARGO_MANIFEST_DIR")).join("../../apps"); let apps = std::path::Path::new(env!("CARGO_MANIFEST_DIR")).join("../../apps");
@@ -1801,6 +1852,8 @@ app:
} }
} }
open.sort(); open.sort();
// Cuprate 18090 is its deliberately public restricted RPC subset;
// unrestricted node-control RPC remains container-loopback-only.
// Gitea 3001 (git clients speak basic-auth, not browser cookies), // Gitea 3001 (git clients speak basic-auth, not browser cookies),
// BTCPay 23000 (checkout/invoice/webhook endpoints must be reachable // BTCPay 23000 (checkout/invoice/webhook endpoints must be reachable
// by anonymous payers), and — since the v1.8.7 platform round — the // by anonymous payers), and — since the v1.8.7 platform round — the
@@ -1808,18 +1861,35 @@ app:
// nginx-proxy-manager 8081 (NPM admin accounts), tailscale 8240 // nginx-proxy-manager 8081 (NPM admin accounts), tailscale 8240
// (tailnet login on the web console). Both enforce their own login, // (tailnet login on the web console). Both enforce their own login,
// and an operator can re-gate either from Settings → Access control. // and an operator can re-gate either from Settings → Access control.
// Angor's indexer exposes public chain data/transaction broadcast;
// its optional standalone relay accepts signed public Nostr events.
// Neither mounts credentials or the node's internal relay database.
assert_eq!( assert_eq!(
open, open,
vec![ vec![
("angor-indexer".to_string(), 8998u16),
("angor-relay".to_string(), 8091u16),
("btcpay-server".to_string(), 23000u16), ("btcpay-server".to_string(), 23000u16),
("cuprate".to_string(), 18090u16),
("gitea".to_string(), 3001u16), ("gitea".to_string(), 3001u16),
("nginx-proxy-manager".to_string(), 8081u16), ("nginx-proxy-manager".to_string(), 8081u16),
("tailscale".to_string(), 8240u16), ("tailscale".to_string(), 8240u16),
], ],
"gate-open port set changed — every entry must be an app with its own login" "gate-open port set changed — review login or intentional public protocol purpose"
); );
} }
#[test]
fn invalid_install_prerequisites_are_rejected() {
for value in ["not-a-list", "[demo]", "['../other']", "[false]", "['']"] {
let yaml = format!("app:\n id: demo\n name: Demo\n version: 1.0.0\n container:\n image: docker.io/library/alpine:3.20\n install_prerequisites: {value}\n");
assert!(AppManifest::parse(&yaml)
.unwrap_err()
.to_string()
.contains("install_prerequisites"));
}
}
#[test] #[test]
fn an_undeclared_port_classifies_as_session_but_is_not_declared() { fn an_undeclared_port_classifies_as_session_but_is_not_declared() {
// Two different questions, and conflating them caused both gate // Two different questions, and conflating them caused both gate
@@ -2342,6 +2412,46 @@ app:
); );
} }
#[test]
fn node_identity_pubkeys_placeholder_is_accepted() {
let yaml = r#"
app:
id: wildbloom-node
name: Wildbloom Node
version: 0.2.2
container:
image: ghcr.io/forgesworn/wildbloom-node:0.2.2
derived_env:
- key: WILDBLOOM_ALLOW_PUBKEYS
template: "{{NODE_IDENTITY_PUBKEYS}}"
"#;
AppManifest::parse(yaml).expect("NODE_IDENTITY_PUBKEYS is a supported placeholder");
}
#[test]
fn resolve_derived_env_renders_node_identity_pubkeys() {
let yaml = r#"
app:
id: wildbloom-node
name: Wildbloom Node
version: 0.2.2
container:
image: ghcr.io/forgesworn/wildbloom-node:0.2.2
derived_env:
- key: WILDBLOOM_ALLOW_PUBKEYS
template: "{{NODE_IDENTITY_PUBKEYS}}"
"#;
let manifest = AppManifest::parse(yaml).unwrap();
let facts = HostFacts::sample();
assert_eq!(
manifest.app.container.resolve_derived_env(&facts),
vec![format!(
"WILDBLOOM_ALLOW_PUBKEYS={}",
facts.node_identity_pubkeys
)]
);
}
#[test] #[test]
fn path_traversal_secret_file_is_rejected() { fn path_traversal_secret_file_is_rejected() {
let yaml = r#" let yaml = r#"
@@ -2397,6 +2507,7 @@ app:
host_mdns: "test-node.local".to_string(), host_mdns: "test-node.local".to_string(),
disk_gb: 2000, disk_gb: 2000,
bitcoin_host: "bitcoin-core".to_string(), bitcoin_host: "bitcoin-core".to_string(),
node_identity_pubkeys: String::new(),
}; };
let out = c.resolve_derived_env(&facts); let out = c.resolve_derived_env(&facts);
+62 -54
View File
@@ -310,59 +310,7 @@ impl PodmanClient {
); );
continue; continue;
} }
// Honour the manifest's protocol (default tcp). netbird's STUN port port_mappings.push(podman_publish_mapping(port));
// is 3478/udp; forcing tcp here would publish the wrong protocol and
// silently break relay discovery.
let protocol = match port.protocol.to_ascii_lowercase().as_str() {
"udp" => "udp",
"sctp" => "sctp",
_ => "tcp",
};
// Effective bind. A gated port with no declared bind would
// publish 0.0.0.0 — the app would own every host address, which
// is both the exposure itself and the reason the daemon's app
// gate cannot bind those addresses to authenticate them. Pin it
// to loopback so the gate can take the external addresses.
//
// Doing it HERE, at container creation, is the point: the pin and
// the gate's takeover then both come from the daemon and cannot
// disagree. The earlier attempt put this decision in manifest
// data instead, and a node whose manifests lagged the binary
// published Bitcoin's loopback-only RPC across the LAN
// (test node, 2026-08-03).
//
// A port that already declares a bind is never overridden — that
// is exactly what keeps `bind: 127.0.0.1` ports host-local and
// leaves `auth: none` protocol ports (LND gRPC/REST, electrum)
// published as they are, so remote wallets keep working.
// NOTE: the daemon deliberately does NOT rewrite this. Pinning a
// published port to loopback is how an app hands its external
// addresses to the gate, but it belongs in the manifest, not in
// daemon-side inference:
//
// * `bind` is already honoured by every publish path (here and
// in package::install), so a manifest edit needs no code.
// * inference here would cover only THIS path — proven on
// a test node, where a recreate went through another one and
// the pin never applied.
// * and inferring from an ABSENT field is what republished
// Bitcoin's loopback RPC across the LAN, and came within one
// container-recreate of pinning LND's gRPC/REST and breaking
// every remote wallet.
//
// So the migration ships as `bind: 127.0.0.1` in the signed
// catalog. Verified 2026-08-03 that a disk-only manifest edit is
// overridden by the catalog, which is precisely why the catalog is
// the right and only place to carry it.
let mut mapping = serde_json::json!({
"container_port": port.container,
"host_port": port.host,
"protocol": protocol,
});
if !port.bind.is_empty() {
mapping["host_ip"] = serde_json::json!(port.bind);
}
port_mappings.push(mapping);
} }
let mut mounts = Vec::new(); let mut mounts = Vec::new();
@@ -502,6 +450,17 @@ impl PodmanClient {
"nsmode": net_mode "nsmode": net_mode
}, },
}); });
if matches!(
manifest.app.container.network.as_deref(),
Some(
"slirp4netns:allow_host_loopback=true"
| "slirp4netns:allow_host_loopback=true,cidr=169.254.1.0/24"
)
) {
body["network_options"] = serde_json::json!({
"slirp4netns": manifest.app.container.network.as_deref().unwrap().split_once(':').unwrap().1.split(',').collect::<Vec<_>>()
});
}
if let Some(network) = custom_network { if let Some(network) = custom_network {
// The container always answers to its own name; manifest // The container always answers to its own name; manifest
// network_aliases add extra short hostnames peers may bake in // network_aliases add extra short hostnames peers may bake in
@@ -751,6 +710,25 @@ pub fn image_uses_insecure_registry(image: &str) -> bool {
.is_some_and(|host| INSECURE_REGISTRY_HOSTS.contains(&host)) .is_some_and(|host| INSECURE_REGISTRY_HOSTS.contains(&host))
} }
// Keep the explicitly declared bind and transport identical to Quadlet. The
// app gate owns external listeners; container publication must not bypass it.
fn podman_publish_mapping(port: &crate::manifest::PortMapping) -> serde_json::Value {
let protocol = match port.protocol.to_ascii_lowercase().as_str() {
"udp" => "udp",
"sctp" => "sctp",
_ => "tcp",
};
let mut mapping = serde_json::json!({
"container_port": port.container,
"host_port": port.host,
"protocol": protocol,
});
if !port.bind.is_empty() {
mapping["host_ip"] = serde_json::json!(port.bind);
}
mapping
}
fn podman_network_settings( fn podman_network_settings(
network: Option<&str>, network: Option<&str>,
network_policy: &str, network_policy: &str,
@@ -760,7 +738,11 @@ fn podman_network_settings(
Some("host") => ("host", None), Some("host") => ("host", None),
Some("bridge") => ("bridge", None), Some("bridge") => ("bridge", None),
Some("none") => ("none", None), Some("none") => ("none", None),
Some("slirp4netns") => ("slirp4netns", None), Some(
"slirp4netns"
| "slirp4netns:allow_host_loopback=true"
| "slirp4netns:allow_host_loopback=true,cidr=169.254.1.0/24",
) => ("slirp4netns", None),
Some("pasta") => ("pasta", None), Some("pasta") => ("pasta", None),
Some("private") => ("private", None), Some("private") => ("private", None),
Some(custom) => ("bridge", Some(custom.to_string())), Some(custom) => ("bridge", Some(custom.to_string())),
@@ -1110,6 +1092,32 @@ mod tests {
)); ));
} }
#[test]
fn npm_rootless_options_are_not_a_named_bridge() {
assert_eq!(
podman_network_settings(Some("slirp4netns:allow_host_loopback=true"), "isolated"),
("slirp4netns", None)
);
}
#[test]
fn portainer_manifest_keeps_private_network_and_loopback_api_publication() {
let m = AppManifest::parse(include_str!("../../../apps/portainer/manifest.yml")).unwrap();
assert_eq!(
podman_network_settings(
m.app.container.network.as_deref(),
&m.app.security.network_policy
),
("slirp4netns", None)
);
assert_eq!(
podman_publish_mapping(&m.app.ports[0]),
serde_json::json!({
"container_port": 9000, "host_port": 9000, "protocol": "tcp", "host_ip": "127.0.0.1"
})
);
}
#[test] #[test]
fn podman_network_settings_uses_networks_map_for_custom_networks() { fn podman_network_settings_uses_networks_map_for_custom_networks() {
assert_eq!( assert_eq!(
+75 -19
View File
@@ -40,6 +40,11 @@ pub fn stop_grace_secs_for(container_name: &str) -> u64 {
#[async_trait] #[async_trait]
pub trait ContainerRuntime: Send + Sync { pub trait ContainerRuntime: Send + Sync {
/// CLI used for offline app provisioning in this runtime's storage scope.
fn cli_name(&self) -> &'static str {
"podman"
}
async fn pull_image(&self, image: &str, signature: Option<&str>) -> Result<()>; async fn pull_image(&self, image: &str, signature: Option<&str>) -> Result<()>;
async fn create_container( async fn create_container(
&self, &self,
@@ -618,8 +623,58 @@ impl DockerRuntime {
} }
} }
// Docker is a development fallback. Refuse Podman-only network modes instead
// of silently installing a different topology; still honor binds for other apps.
fn docker_network_and_ports(manifest: &AppManifest, offset: u16) -> Result<Vec<String>> {
let network = manifest
.app
.container
.network
.as_deref()
.filter(|v| !v.is_empty())
.unwrap_or(&manifest.app.security.network_policy);
if matches!(
network,
"slirp4netns"
| "slirp4netns:allow_host_loopback=true"
| "slirp4netns:allow_host_loopback=true,cidr=169.254.1.0/24"
| "pasta"
) {
anyhow::bail!("this app requires rootless Podman networking ({network})");
}
let mut args = Vec::new();
if !network.is_empty() && network != "isolated" {
args.extend(["--network".to_owned(), network.to_owned()]);
}
for port in &manifest.app.ports {
let host = port
.host
.checked_add(offset)
.context("published port offset overflow")?;
let bind = if port.bind.is_empty() {
String::new()
} else {
format!("{}:", port.bind)
};
let protocol = if port.protocol.is_empty() {
"tcp"
} else {
&port.protocol
};
args.extend([
"-p".to_owned(),
format!("{bind}{host}:{}/{protocol}", port.container),
]);
}
Ok(args)
}
#[async_trait] #[async_trait]
impl ContainerRuntime for DockerRuntime { impl ContainerRuntime for DockerRuntime {
fn cli_name(&self) -> &'static str {
"docker"
}
async fn pull_image(&self, image: &str, signature: Option<&str>) -> Result<()> { async fn pull_image(&self, image: &str, signature: Option<&str>) -> Result<()> {
// Same signature gate as the podman path — the docker fallback is // Same signature gate as the podman path — the docker fallback is
// dev-only, but a declared signature must never be skippable by // dev-only, but a declared signature must never be skippable by
@@ -657,25 +712,7 @@ impl ContainerRuntime for DockerRuntime {
cmd.arg("--read-only"); cmd.arg("--read-only");
} }
match manifest.app.security.network_policy.as_str() { cmd.args(docker_network_and_ports(manifest, port_offset)?);
"host" => {
cmd.arg("--network").arg("host");
}
"isolated" => {
// Docker uses bridge network by default
}
_ => {
cmd.arg("--network")
.arg(&manifest.app.security.network_policy);
}
}
// Port mappings with offset
for port in &manifest.app.ports {
let host_port = port.host + port_offset;
cmd.arg("-p")
.arg(format!("{}:{}", host_port, port.container));
}
// Volumes // Volumes
for volume in &manifest.app.volumes { for volume in &manifest.app.volumes {
@@ -969,6 +1006,10 @@ impl AutoRuntime {
#[async_trait] #[async_trait]
impl ContainerRuntime for AutoRuntime { impl ContainerRuntime for AutoRuntime {
fn cli_name(&self) -> &'static str {
self.runtime.cli_name()
}
async fn pull_image(&self, image: &str, signature: Option<&str>) -> Result<()> { async fn pull_image(&self, image: &str, signature: Option<&str>) -> Result<()> {
self.runtime.pull_image(image, signature).await self.runtime.pull_image(image, signature).await
} }
@@ -1035,6 +1076,21 @@ mod tests {
use super::*; use super::*;
use std::collections::HashMap; use std::collections::HashMap;
#[test]
fn docker_fallback_rejects_rootless_only_topology_and_preserves_bind_protocol() {
let mut m =
AppManifest::parse(include_str!("../../../apps/portainer/manifest.yml")).unwrap();
assert!(docker_network_and_ports(&m, 0).is_err());
m.app.container.network = Some("bridge".into());
m.app.ports[0].protocol = "udp".into();
let args = docker_network_and_ports(&m, 1).unwrap();
assert_eq!(
args,
vec!["--network", "bridge", "-p", "127.0.0.1:9001:9000/udp"]
);
assert!(docker_network_and_ports(&m, u16::MAX).is_err());
}
#[test] #[test]
fn missing_container_classifier_covers_podman5_phrasings() { fn missing_container_classifier_covers_podman5_phrasings() {
// podman 5.x `inspect` phrasing for a missing container. // podman 5.x `inspect` phrasing for a missing container.

Some files were not shown because too many files have changed in this diff Show More