Compare commits

..
Author SHA1 Message Date
archipelago cedfbb2b07 docs: record public release verification and storage follow-up 2026-10-05 19:52:07 -04:00
archipelago 7ae812e3f6 chore: publish verified 1.9.0-alpha OTA and app catalog 2026-10-05 19:13:08 -04:00
archipelago bc386965da docs: require FIPS for distributed media streaming 2026-10-05 18:59:49 -04:00
archipelago 7abd04a7f6 docs: record corrected signed OTA reboot acceptance
Demo images / Build & push demo images (push) Failing after 39s
2026-10-05 18:56:17 -04:00
archipelago 441733646f chore: stage signed 1.9.0-alpha release metadata 2026-10-05 18:50:26 -04:00
archipelago ccf823590a fix: reject stale first-boot scripts in OTA release payloads 2026-10-05 18:44:46 -04:00
archipelago d9775ac144 docs: require headless node qualification and IndeeHub app delivery 2026-10-05 18:25:18 -04:00
archipelago 0925c58821 docs: track Fleet metrics and secure FIPS performance work 2026-10-05 18:22:51 -04:00
archipelago 2d27f9c475 docs: track node availability and navigation latency follow-ups 2026-10-05 18:21:02 -04:00
archipelago 868e46fac9 docs: queue companion Fleet and AIUI setup follow-ups 2026-10-05 18:18:58 -04:00
archipelago 2aa77d1b7b docs: record installer acceptance and post-release work 2026-10-05 18:18:11 -04:00
archipelago a6b9e7ab49 fix: preserve apps and diagnostics when first-boot setup retries 2026-10-05 17:21:08 -04:00
archipelago a902cc84fe test: qualify built demo images through mobile and upload flows 2026-10-05 16:48:52 -04:00
archipelago 157c9ec055 fix: keep public demo online during optional upstream DNS outages 2026-10-05 16:15:41 -04:00
archipelago 415826f0a6 fix: package the qualified UI in release ISOs 2026-10-05 16:02:48 -04:00
archipelago 69857c4ace fix: require explicit dispatch for public demo redeployment 2026-10-05 15:42:46 -04:00
archipelago e6e46a1427 fix: build demo AIUI from the reviewed source revision 2026-10-05 15:34:41 -04:00
archipelago e0b2181ae9 fix: isolate NPM upstream TLS sessions across public domains 2026-10-05 15:26:28 -04:00
archipelago 446fa7b7fd fix: retain management guard through legacy runtime install and rollback 2026-10-05 15:08:42 -04:00
archipelago ba8b1f29b2 docs: record accepted companion and Yaya deployment 2026-10-05 14:44:03 -04:00
archipelago b8266c2872 fix: support resumable Cloud uploads in the public demo 2026-10-05 14:41:28 -04:00
archipelago 5aa74d0513 fix: qualify mobile Cloud viewer and companion downloads 2026-10-05 14:41:08 -04:00
archipelago daac47cac4 fix: harden node upgrades and prepare 1.9.0-alpha 2026-10-05 12:43:49 -04:00
archipelago 138a541d01 docs: make ngit canonical and gate mirror publication 2026-10-05 11:47:12 -04:00
archipelago 833c939220 docs: make alpha status and funds risk prominent 2026-10-05 10:26:10 -04:00
archipelago 2c1bcacf0a Test Nostr encryption compatibility and forged relay messages
Demo images / Build & push demo images (push) Failing after 52s
2026-10-05 10:00:41 -04:00
archipelago f1d0092e57 Validate app owner keys and match identity picker whitespace rules 2026-10-05 09:31:46 -04:00
archipelago 7dfb0e0013 Merge opt-in app owner identity placeholder
nevent1qqs9d76qm6f5xj2vrtjfnkqz5exrc8r0s9zev4f672kqyd0wjh7wwvqpz3mhxue69uhhyetvv9ujumn8d96zuer9wcx2tvaw
2026-10-05 09:29:18 -04:00
archipelago 775d7b9877 Merge Nostr 0.44 security fixes
nevent1qqsgj7l3gewl20m6xxxeu89zsmjy9prvc9g8ggkr6cysa5p9tc3hc2gpz3mhxue69uhhyetvv9ujumn8d96zuer9wcyr3wqc
2026-10-05 09:29:08 -04:00
TheCryptoDonkey c18ebd7f5b fix: update nostr to 0.44.7 and nostr-relay-pool to 0.44.3
Patch releases within 0.44 that clear eleven RustSec advisories against
the versions in the lock: RUSTSEC-2026-0216, -0219, -0224 to -0232.
They cover NIP-04 and NIP-44 decryption panics and resource exhaustion,
Debug output exposing NIP-46 and NIP-60 credentials, and the relay pool's
handling of unverified events. No other package changes.
2026-10-03 11:11:06 +02:00
TheCryptoDonkey 494d248356 feat: add NODE_IDENTITY_PUBKEYS derived-env placeholder
Lets an app grant the node's users owner rights, e.g. a Blossom server's
allowed uploaders. The value is the Nostr keys of the identities the app
identity picker offers for NIP-07 signing, chosen by the same rule as
NostrIdentityPicker.vue, so the node's own appliance identity is never
included. It is resolved only for manifests that template it, and an
empty set is an error rather than an empty owner list.

identity.list now shares its is_node test with the new helper.
2026-10-03 11:10:29 +02:00
archipelago 3acefecc24 Serve the shared Mempool explorer through the Angor indexer 2026-10-01 16:03:19 -04:00
archipelago 19c49c6605 Remove remaining blur from transaction filter container 2026-10-01 14:48:51 -04:00
archipelago d6e0c142c6 Return retryable payment status errors and record NPM release gate 2026-10-01 14:24:24 -04:00
archipelago 57729f8e18 Record candidate deployment and corrected payment evidence 2026-10-01 12:15:41 -04:00
archipelago 4fdadad89d Record final build status and live resource checks 2026-10-01 10:46:44 -04:00
archipelago f4d3455496 Fix paid-file recovery, app lifecycle regressions and wallet controls
Demo images / Build & push demo images (push) Failing after 1m10s
2026-10-01 10:31:55 -04:00
archipelago 227174e541 docs: close 1.8.22 publication with verified Git and ngit assets 2026-10-01 06:19:12 -04:00
archipelago 2e72b38778 release: publish verified signed 1.8.22 OTA and app catalog 2026-10-01 06:15:32 -04:00
archipelago 0be7aee49d docs: record final 1.8.22 OTA and ISO acceptance 2026-09-30 19:50:01 -04:00
archipelago 6d5f3ffb85 fix: select NPM admin port regardless of binding order
Demo images / Build & push demo images (push) Failing after 34s
2026-09-30 18:24:35 -04:00
archipelago d1bc1273d4 fix: replace cached container doctor before ISO first boot 2026-09-30 17:52:30 -04:00
archipelago 96fb5a4f19 fix: prevent stale snapshots resurrecting orphaned dashboards
Demo images / Build & push demo images (push) Failing after 36s
2026-09-30 17:45:18 -04:00
archipelago f91c1f33db fix: keep apps running when network diagnostics fail 2026-09-30 17:34:11 -04:00
archipelago 02b840f2d1 chore: prepare 1.8.22-alpha release candidate
Demo images / Build & push demo images (push) Failing after 36s
2026-09-30 16:45:43 -04:00
archipelago f992780957 fix: probe Angor IPv4 health endpoint inside the actual image 2026-09-30 16:40:16 -04:00
archipelago c82c1eee98 fix: prevent NPM tunnel collisions and false app health restarts 2026-09-30 16:30:40 -04:00
archipelago 2992443d5d docs: record verified NPM tunnel port conflict and node repair 2026-09-30 16:12:04 -04:00
archipelago 259c353147 Clear completed candidate deployment instructions 2026-09-30 13:46:50 -04:00
archipelago 1724ea05d1 Show readiness reason first and record live dashboard acceptance
Demo images / Build & push demo images (push) Failing after 45s
2026-09-30 13:46:16 -04:00
archipelago c1e20a71ae Check companion dashboards and omit headless UI waiting messages
Demo images / Build & push demo images (push) Failing after 37s
2026-09-30 13:28:37 -04:00
archipelago bf56956790 Record UI acceptance and remaining normal-startup release gate 2026-09-30 12:57:35 -04:00
archipelago 2f1a3ade07 Promote runtime manifests before starting app reconciliation 2026-09-30 12:50:46 -04:00
archipelago ef8254272c Name waiting apps, align card actions, and update Angor icon
Demo images / Build & push demo images (push) Failing after 39s
2026-09-30 12:43:53 -04:00
archipelago d50be13232 Normalize Mempool frontend aliases in restored app inventory 2026-09-30 12:41:15 -04:00
archipelago 439b55a236 Use manifest names for new services and document release acceptance
Demo images / Build & push demo images (push) Failing after 38s
2026-09-30 12:20:29 -04:00
archipelago 5ab65f7581 Preserve apostrophes in Quadlet commands and record funded acceptance 2026-09-30 12:08:35 -04:00
archipelago 169bf77de6 Add headless Angor services and shared-index install guard
Demo images / Build & push demo images (push) Failing after 43s
2026-09-30 11:52:19 -04:00
archipelago 7c4169867c docs: consolidate release scope and record migration recovery checks 2026-09-30 10:52:41 -04:00
archipelago acf544500f fix(apps): preserve state across runtime repairs and restore Gitea SSH 2026-09-30 10:46:38 -04:00
archipelago 7d767c8cb0 fix(catalog): gate network migration manifests on backup support 2026-09-30 10:08:56 -04:00
archipelago eb3ccfa00b Merge branch 'fix/gitea-portainer-20260930' 2026-09-30 09:57:49 -04:00
archipelago d69e845216 Merge remote-tracking branch 'origin/main'
Demo images / Build & push demo images (push) Failing after 1m10s
2026-09-30 09:32:20 -04:00
archipelago dc962c53b0 docs: record live lifecycle acceptance and next release blockers 2026-09-30 09:31:18 -04:00
chaum b02ba4100d Merge pull request 'fix(files): save purchased files atomically with rootless ownership' (#162) from fix/filebrowser-purchase-filing into main 2026-09-30 12:58:36 +00:00
chaum 3daea6623b Merge pull request 'fix(ecash): prevent paid-download replay and read failures after charging' (#161) from fix/ecash-paid-download-v2-keyset into main 2026-09-30 12:58:33 +00:00
archipelago 0677924a64 Merge current main and make purchase filing atomic under concurrent writes 2026-09-30 07:26:51 -04:00
archipelago 971d477795 Merge current main and harden paid-download delivery 2026-09-30 07:25:47 -04:00
ssmithxandClaude Opus 5.5 33477f284b fix(files): file purchased content into FileBrowser folders again
Every paid download logged "filing into filebrowser/Music/... failed
(non-fatal): Permission denied". The purchase played in-app but never
appeared in Files. FileBrowser's folders belong to its rootless container
range (host uid 100000, mode 755). This service is host uid 1000, outside
that range, so it can read them but not create files in them.

New container::filebrowser::save_new_file:
- Writes directly when the folder allows it.
- Otherwise writes through `podman unshare`, where that uid range is
  ours: to a temp file, then chowned to the folder's owner, set to 0644,
  and hard-linked into place. FileBrowser never sees a partial file and an
  existing file is never replaced. A missing folder is created and given
  its parent's owner. No sudo.
- Keeps the "name (2).ext" de-duplication the RPC did inline.

Checked the unshare script on amishparadise in a scratch folder owned
like FileBrowser's: new folder + file OK, owner/mode right, no clobber,
no temp file left, and the service can read the result.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-09-29 22:36:31 +00:00
ssmithxandClaude Opus 5.5 03e38d1ca3 test: regression tests for the paid-download fixes
- mint_client: a stub mint shows swap() sends the full v2 keyset id when
  given a cashuB short id, and leaves complete v1/v2 ids unchanged.
- fips::dial: the single-delivery decisions are now small functions
  (fips_answer_is_final, fips_retryable). Tests cover them and, against a
  silent local peer, check that a single-delivery request isn't resent
  after a timeout while an ordinary one still is.
- content_server: an unreadable paid file returns Unavailable before the
  payment gate runs, and a readable one still returns 402. Also covers
  ensure_readable's grant/reopen behaviour. The podman grant is replaced
  by a refusal under cfg(test) so results don't depend on the host.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-09-29 20:12:16 +00:00
ssmithxandClaude Opus 5.5 e5fc99d66c fix(content): never charge for a file the seller can't serve or replay a spent token
After the keyset-id fix, a Minibits paid download still failed and the
buyer lost the sats. What happened, 2026-09-29, amishparadise:

1. The seller redeemed the token, then failed to read the file. It was a
   FileBrowser upload owned by the container subuid (100999) with mode
   0640. The handler mapped that Err to 404.
2. The buyer's FIPS dial treats 404 as "fall back to Tor" and resent the
   request with the same, now spent, token. The seller answered 402, and
   the buyer showed "seller doesn't accept your Cashu mint".

Fixes:
- serve_content checks the file is readable before the paid gate. If it
  isn't, it grants read with `podman unshare chmod a+r`, which matches
  the other shared files. If that also fails it returns Unavailable (503)
  without taking payment.
- The content handler returns 500 on internal errors and logs them,
  instead of a silent 404.
- New PeerRequest::single_delivery(), used for the paid download: the
  FIPS answer is final, FIPS retries only when it never connected, and
  there's no Tor replay once the request may have been delivered.
- The buyer shows the seller's error text for non-402 failures.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-09-29 18:42:20 +00:00
ssmithxandClaude Opus 5.5 8b74803290 fix(ecash): repair short v2 keyset ids on every swap, not just receive
Paid cloud downloads paid with Minibits ecash were always rejected. The
buyer sends a cashuB token, which carries NUT-02 v2 keyset ids in their
8-byte short form. Minibits rotated its active keyset to a v2 id, and the
seller's verify_and_receive_payment called MintClient::swap directly,
skipping the short->full id repair that only receive_token applied. The
mint answered 422 ("ID length invalid"). The buyer then showed the
misleading "seller doesn't accept your Cashu mint" hint.

Move the repair into swap() so every caller is covered: payment verify,
streaming gate, send change, and cross-mint swaps.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-09-29 15:47:23 +00:00
239 changed files with 21359 additions and 1879 deletions
+7 -1
View File
@@ -4,7 +4,13 @@
# Allow neode-ui (frontend + mock backend + docker configs)
!neode-ui/
# Allow demo assets (AIUI pre-built dist)
!aiui/
aiui/**/node_modules
aiui/**/dist
aiui/**/.turbo
aiui/**/.build-aiui-last-*
# Allow curated demo assets
!demo/
# Allow the Bitcoin UI + ElectrumX UI mock shells (served from /docker/*)
+6 -1
View File
@@ -17,6 +17,9 @@ on:
branches: [main]
paths:
- 'neode-ui/**'
- 'aiui/**'
- 'scripts/build-aiui.sh'
- '.dockerignore'
- 'docker-compose.demo.yml'
- '.gitea/workflows/demo-images.yml'
workflow_dispatch:
@@ -65,10 +68,12 @@ jobs:
push: true
build-args: |
VITE_DEMO=1
SOURCE_REVISION=${{ github.sha }}
tags: |
${{ vars.DEMO_REGISTRY }}/archy-demo-web:demo
${{ vars.DEMO_REGISTRY }}/archy-demo-web:${{ github.sha }}
- name: Trigger Portainer redeploy
if: ${{ success() && secrets.PORTAINER_WEBHOOK != '' }}
# Source pushes prepare images; public deployment is an explicit post-release action.
if: ${{ success() && github.event_name == 'workflow_dispatch' && secrets.PORTAINER_WEBHOOK != '' }}
run: curl -fsS -X POST "${{ secrets.PORTAINER_WEBHOOK }}"
+6 -1
View File
@@ -17,6 +17,9 @@ on:
branches: [main]
paths:
- 'neode-ui/**'
- 'aiui/**'
- 'scripts/build-aiui.sh'
- '.dockerignore'
- 'docker-compose.demo.yml'
- '.github/workflows/demo-images.yml'
workflow_dispatch:
@@ -65,10 +68,12 @@ jobs:
push: true
build-args: |
VITE_DEMO=1
SOURCE_REVISION=${{ github.sha }}
tags: |
${{ vars.DEMO_REGISTRY }}/archy-demo-web:demo
${{ vars.DEMO_REGISTRY }}/archy-demo-web:${{ github.sha }}
- name: Trigger Portainer redeploy
if: ${{ success() && secrets.PORTAINER_WEBHOOK != '' }}
# Source pushes prepare images; public deployment is an explicit post-release action.
if: ${{ success() && github.event_name == 'workflow_dispatch' && secrets.PORTAINER_WEBHOOK != '' }}
run: curl -fsS -X POST "${{ secrets.PORTAINER_WEBHOOK }}"
+42
View File
@@ -29,3 +29,45 @@ unrestricted `cargo test` on a node with installed apps: older mocked-runtime
tests still reached real service commands. The runner isolates wallet data,
service buses, container storage, networking, and process IDs. Compilation with
`cargo test --no-run` is safe. Keep separately authorized live checks explicit.
## Active release regression checklist
Before resuming release work, read
`docs/post-1.8.22-regressions-20261001.md` and retain its unfinished tasks.
The operator requested that every reported issue be tracked, fixed and tested
before another OTA/ISO. Keep source/unit-test results separate from actual-node
acceptance. In particular, paid-file recovery must not send another payment,
and app cleanup must preserve wallets, persistent data and uninstall decisions.
Do not mark the new paid-file incident resolved merely because the earlier
Framework LND startup incident was closed.
## Gitea and ngit mirror parity
Nostr Git (`ngit`) is the canonical contribution and review platform. Gitea
(`origin`) mirrors accepted code on `main` and release tags. Both are required
publication mirrors; duplicate PRs and proposal branches on Gitea are not required.
For every change, including fixes and release preparation:
- Review and merge once. Push the exact same resulting commits to both mirrors;
never independently squash, rebase or merge the same change on each platform.
- Open new contributions and PRs on ngit; review and merge there, then mirror the
exact accepted main commits to Gitea. Record the ngit proposal and resulting
merge commit in the release ledger. Existing Gitea PRs must be reviewed and
explicitly linked to their ngit replacement or accepted result before closing;
do not abandon contributions or mark unmerged changes as merged. PR numbers,
reviews and discussions remain platform-specific; matching Git refs does not
prove their synchronization.
- Push main and release tags to both mirrors. Preserve commit history
and annotated tag objects/signatures. Do not resolve drift by force pushing,
deleting remote refs, or rewriting published history without explicit approval.
- After publishing source, run `python3 scripts/check-git-mirrors.py --local`.
Include each additional shared branch or release tag with repeated `--ref`
arguments (full `refs/heads/...` or `refs/tags/...` names).
- Before OTA, catalog or ISO publication, require matching reviewed local and
remote main and release tag refs, and record ngit PR dispositions in the
release acceptance ledger. A failed push, unavailable mirror, missing ref or
mismatch blocks publication; never describe a partial push as synchronized.
Run `--all` for a complete advertised branch/tag audit; a main-only pass must
never be described as full historical mirror parity. Proposal-only branches
may intentionally differ. Existing unrelated drift
must be inventoried explicitly rather than silently overwritten.
+5 -2
View File
@@ -11,8 +11,8 @@ android {
applicationId = "com.archipelago.app"
minSdk = 26
targetSdk = 35
versionCode = 52
versionName = "0.5.32"
versionCode = 54
versionName = "0.5.34"
vectorDrawables {
useSupportLibrary = true
@@ -142,6 +142,9 @@ tasks.matching {
}.configureEach { dependsOn("buildRustArm64") }
dependencies {
testImplementation("junit:junit:4.13.2")
testImplementation("com.squareup.okhttp3:mockwebserver:4.12.0")
testImplementation("org.robolectric:robolectric:4.14.1")
val composeBom = platform("androidx.compose:compose-bom:2024.05.00")
implementation(composeBom)
@@ -0,0 +1,179 @@
package com.archipelago.app.ui.screens
import android.app.Activity
import android.content.Intent
import android.net.Uri
import android.provider.DocumentsContract
import android.webkit.CookieManager
import android.webkit.DownloadListener
import android.webkit.URLUtil
import android.widget.Toast
import androidx.activity.compose.rememberLauncherForActivityResult
import androidx.activity.result.contract.ActivityResultContracts
import androidx.compose.foundation.layout.Column
import androidx.compose.material3.AlertDialog
import androidx.compose.material3.LinearProgressIndicator
import androidx.compose.material3.Text
import androidx.compose.material3.TextButton
import androidx.compose.runtime.*
import androidx.compose.ui.platform.LocalContext
import kotlinx.coroutines.*
import okhttp3.Call
import okhttp3.HttpUrl.Companion.toHttpUrlOrNull
import okhttp3.OkHttpClient
import okhttp3.Request
import java.io.IOException
import java.io.OutputStream
import java.util.concurrent.TimeUnit
internal data class WebDownload(val url: String, val userAgent: String, val cookies: String, val name: String, val mime: String)
/** Only the starting origin receives its WebView cookies, even across redirects. */
internal fun streamWebDownload(
download: WebDownload,
output: OutputStream,
client: OkHttpClient,
onCall: (Call) -> Unit = {},
checkCancelled: () -> Unit = {},
onProgress: (Long, Long) -> Unit = { _, _ -> },
): Long {
val transport = client.newBuilder().followRedirects(false).followSslRedirects(false).build()
val original = download.url.toHttpUrlOrNull() ?: throw IOException("Unsupported download link")
var url = original
var redirects = 0
while (true) {
checkCancelled()
if (url.username.isNotEmpty() || url.password.isNotEmpty()) throw IOException("Unsupported download link")
val request = Request.Builder().url(url).header("User-Agent", download.userAgent)
if (url.scheme == original.scheme && url.host == original.host && url.port == original.port && download.cookies.isNotBlank()) {
request.header("Cookie", download.cookies)
}
val call = transport.newCall(request.build())
onCall(call)
call.execute().use { response ->
if (response.code in listOf(301, 302, 303, 307, 308)) {
if (++redirects > 5) throw IOException("Too many download redirects")
val next = response.header("Location")?.let { url.resolve(it) } ?: throw IOException("Invalid download redirect")
if (url.isHttps && !next.isHttps) throw IOException("Insecure download redirect blocked")
url = next
} else {
if (response.code == 401 || response.code == 403) throw IOException("Sign in to the node again, then retry the download")
if (!response.isSuccessful) throw IOException("Download failed (HTTP ${response.code})")
if (response.header("Content-Type")?.substringBefore(';')?.trim()?.lowercase() == "text/html" &&
download.mime != "text/html" && !download.name.endsWith(".html", true) && !download.name.endsWith(".htm", true)) {
throw IOException("Sign in to the node again, then retry the download")
}
val body = response.body ?: throw IOException("The download was empty")
val total = body.contentLength()
var written = 0L
body.byteStream().use { input ->
val buffer = ByteArray(64 * 1024)
var lastUpdate = 0L
while (true) {
checkCancelled()
val count = input.read(buffer)
if (count == -1) break
output.write(buffer, 0, count)
written += count
val now = System.nanoTime()
if (now - lastUpdate > 100_000_000L) { onProgress(written, total); lastUpdate = now }
}
}
if (total >= 0 && written != total) throw IOException("Download interrupted; please retry")
onProgress(written, total)
return written
}
}
}
}
/** Uses the system Save dialog: no broad storage permission and no external browser login. */
@Composable
internal fun rememberWebViewDownloads(): DownloadListener {
val context = LocalContext.current
val scope = rememberCoroutineScope()
var pending by remember { mutableStateOf<WebDownload?>(null) }
var active by remember { mutableStateOf<WebDownload?>(null) }
var progress by remember { mutableStateOf<Pair<Long, Long>>(0L to -1L) }
var failure by remember { mutableStateOf<String?>(null) }
var job by remember { mutableStateOf<Job?>(null) }
val currentCall = remember { java.util.concurrent.atomic.AtomicReference<Call?>(null) }
val client = remember {
OkHttpClient.Builder().followRedirects(false).followSslRedirects(false)
.connectTimeout(20, TimeUnit.SECONDS).readTimeout(60, TimeUnit.SECONDS).build()
}
fun cancel() { job?.cancel(); currentCall.getAndSet(null)?.cancel() }
DisposableEffect(Unit) { onDispose { currentCall.getAndSet(null)?.cancel() } }
val save = rememberLauncherForActivityResult(ActivityResultContracts.StartActivityForResult()) { result ->
val download = pending
pending = null
val uri = result.data?.data
if (result.resultCode != Activity.RESULT_OK || uri == null || download == null) return@rememberLauncherForActivityResult
job = scope.launch {
active = download
progress = 0L to -1L
var complete = false
try {
withContext(Dispatchers.IO) {
val task = currentCoroutineContext()
context.contentResolver.openOutputStream(uri, "w")?.use { output ->
streamWebDownload(download, output, client,
onCall = { call -> currentCall.set(call); if (!task.isActive) call.cancel() },
checkCancelled = { task.ensureActive() },
onProgress = { done, total -> scope.launch { progress = done to total } })
} ?: throw IOException("Unable to open the selected destination")
}
complete = true
Toast.makeText(context, "Download complete: ${download.name}", Toast.LENGTH_LONG).show()
} catch (error: CancellationException) {
throw error
} catch (error: Exception) {
if (currentCoroutineContext().isActive) {
// Do not expose authenticated URLs or request headers in UI/logs.
failure = when {
error is javax.net.ssl.SSLException -> "The server certificate could not be verified."
error is IOException && error.message?.startsWith("Sign in") == true -> error.message
else -> "Download failed. Check your connection and available storage, then try again."
}
}
} finally {
currentCall.getAndSet(null)?.cancel()
if (!complete) withContext(NonCancellable + Dispatchers.IO) {
// This URI was newly created by ACTION_CREATE_DOCUMENT; never remove an existing user file.
runCatching { DocumentsContract.deleteDocument(context.contentResolver, uri) }
}
active = null
job = null
}
}
}
if (active != null) {
AlertDialog(onDismissRequest = {}, title = { Text("Downloading") }, text = {
Column {
Text(active!!.name)
if (progress.second > 0) LinearProgressIndicator(progress = (progress.first.toFloat() / progress.second).coerceIn(0f, 1f))
else LinearProgressIndicator()
}
}, confirmButton = {}, dismissButton = { TextButton(onClick = { cancel() }) { Text("Cancel") } })
}
failure?.let { message ->
AlertDialog(onDismissRequest = { failure = null }, title = { Text("Download unavailable") },
text = { Text(message) }, confirmButton = { TextButton(onClick = { failure = null }) { Text("OK") } })
}
return DownloadListener { url, userAgent, disposition, mimeType, _ ->
if (active != null || pending != null) {
Toast.makeText(context, "Finish or cancel the current download first", Toast.LENGTH_SHORT).show()
} else if (url.toHttpUrlOrNull() == null) {
failure = "This download link is not supported. Open the file from Cloud and try again."
} else {
val mime = mimeType?.substringBefore(';')?.takeIf { it.contains('/') } ?: "application/octet-stream"
val name = URLUtil.guessFileName(url, disposition, mime).replace(Regex("[\\\\/\\p{Cntrl}]"), "_").take(180).ifBlank { "download" }
pending = WebDownload(url, userAgent ?: "Archipelago Companion", CookieManager.getInstance().getCookie(url).orEmpty(), name, mime)
try {
save.launch(Intent(Intent.ACTION_CREATE_DOCUMENT).apply {
addCategory(Intent.CATEGORY_OPENABLE); type = mime; putExtra(Intent.EXTRA_TITLE, name)
})
} catch (_: Exception) { pending = null; failure = "No file-saving app is available on this device." }
}
}
}
@@ -0,0 +1,99 @@
package com.archipelago.app.ui.screens
import android.content.Context
import android.content.ContextWrapper
import android.graphics.Color
import android.view.View
import android.view.ViewGroup
import android.webkit.WebChromeClient
import android.widget.FrameLayout
import androidx.activity.ComponentActivity
import androidx.activity.OnBackPressedCallback
import androidx.compose.runtime.Composable
import androidx.compose.runtime.DisposableEffect
import androidx.compose.runtime.remember
import androidx.compose.ui.platform.LocalContext
import androidx.core.view.ViewCompat
import androidx.core.view.WindowCompat
import androidx.core.view.WindowInsetsCompat
import androidx.core.view.WindowInsetsControllerCompat
private fun Context.fullscreenActivity(): ComponentActivity? = when (this) {
is ComponentActivity -> this
is ContextWrapper -> baseContext.takeIf { it !== this }?.fullscreenActivity()
else -> null
}
/** Hosts Chromium's custom fullscreen view without replacing or reloading its WebView. */
internal class WebViewFullscreen(private val activity: ComponentActivity?) {
private var overlay: FrameLayout? = null
private var callback: WebChromeClient.CustomViewCallback? = null
private var back: OnBackPressedCallback? = null
private var visibleBars = 0
private var originalBehavior = 0
fun show(view: View?, onHidden: WebChromeClient.CustomViewCallback?) {
val owner = activity
// A second enter must not detach the active video or strand its callback.
if (owner == null || owner.isFinishing || owner.isDestroyed || view == null ||
view.parent != null || overlay != null
) {
onHidden?.onCustomViewHidden()
return
}
val decor = owner.window.decorView as? ViewGroup
if (decor == null) { onHidden?.onCustomViewHidden(); return }
val controller = WindowCompat.getInsetsController(owner.window, decor)
val insets = ViewCompat.getRootWindowInsets(decor)
visibleBars = 0
if (insets?.isVisible(WindowInsetsCompat.Type.statusBars()) != false) {
visibleBars = visibleBars or WindowInsetsCompat.Type.statusBars()
}
if (insets?.isVisible(WindowInsetsCompat.Type.navigationBars()) != false) {
visibleBars = visibleBars or WindowInsetsCompat.Type.navigationBars()
}
originalBehavior = controller.systemBarsBehavior
val host = FrameLayout(owner).apply {
setBackgroundColor(Color.BLACK)
keepScreenOn = true
addView(view, FrameLayout.LayoutParams(-1, -1))
}
overlay = host
callback = onHidden
decor.addView(host, ViewGroup.LayoutParams(-1, -1))
controller.systemBarsBehavior = WindowInsetsControllerCompat.BEHAVIOR_SHOW_TRANSIENT_BARS_BY_SWIPE
controller.hide(WindowInsetsCompat.Type.systemBars())
back = object : OnBackPressedCallback(true) {
override fun handleOnBackPressed() = hide()
}.also { owner.onBackPressedDispatcher.addCallback(it) }
view.requestFocus()
}
fun hide() {
val host = overlay ?: return
// Clear first: Chromium may synchronously call onHideCustomView again.
overlay = null
val notify = callback
callback = null
back?.remove()
back = null
host.keepScreenOn = false
host.removeAllViews()
(host.parent as? ViewGroup)?.removeView(host)
activity?.let { owner ->
val controller = WindowCompat.getInsetsController(owner.window, owner.window.decorView)
controller.systemBarsBehavior = originalBehavior
controller.hide(WindowInsetsCompat.Type.systemBars())
if (visibleBars != 0) controller.show(visibleBars)
}
notify?.onCustomViewHidden()
}
}
@Composable
internal fun rememberWebViewFullscreen(): WebViewFullscreen {
val context = LocalContext.current
val fullscreen = remember(context) { WebViewFullscreen(context.fullscreenActivity()) }
DisposableEffect(fullscreen) { onDispose { fullscreen.hide() } }
return fullscreen
}
@@ -611,6 +611,8 @@ fun WebViewScreen(
// before surfacing the error page: the mesh tunnel works from anywhere.
meshFallbackUrl: String? = null,
) {
val fullscreen = rememberWebViewFullscreen()
val downloads = rememberWebViewDownloads()
var isLoading by remember { mutableStateOf(true) }
// First kiosk load (often over the FIPS mesh) gets the full branded
// loader; later navigations keep just the slim top progress bar.
@@ -913,6 +915,7 @@ fun WebViewScreen(
cookieManager.setAcceptThirdPartyCookies(this, true)
applyArchipelagoSettings()
setDownloadListener(downloads)
settings.apply {
setSupportMultipleWindows(true) // enables onCreateWindow for window.open
// Let JS open windows without a synchronous user-gesture
@@ -1181,6 +1184,12 @@ fun WebViewScreen(
}
webChromeClient = object : WebChromeClient() {
override fun onShowCustomView(view: android.view.View?, callback: CustomViewCallback?) {
fullscreen.show(view, callback)
}
override fun onHideCustomView() = fullscreen.hide()
override fun onProgressChanged(view: WebView?, newProgress: Int) {
loadProgress = newProgress
}
@@ -1546,6 +1555,8 @@ private fun InAppBrowser(
appName: String? = null,
onClose: () -> Unit,
) {
val fullscreen = rememberWebViewFullscreen()
val downloads = rememberWebViewDownloads()
val context = LocalContext.current
// Same-node check across BOTH node addresses (LAN + mesh ULA) — see the
// kiosk's isSameNode; a mismatch here bounced app links to the browser.
@@ -1643,6 +1654,7 @@ private fun InAppBrowser(
CookieManager.getInstance().setAcceptThirdPartyCookies(this, true)
applyArchipelagoSettings()
setDownloadListener(downloads)
// Node apps (BTCPay invoices, LND, Portainer tokens) are
// served over plain HTTP too — same dead-clipboard trap.
addClipboardBridge()
@@ -1662,6 +1674,12 @@ private fun InAppBrowser(
)
webChromeClient = object : WebChromeClient() {
override fun onShowCustomView(view: android.view.View?, callback: CustomViewCallback?) {
fullscreen.show(view, callback)
}
override fun onHideCustomView() = fullscreen.hide()
override fun onProgressChanged(view: WebView?, newProgress: Int) {
progress = newProgress
}
@@ -0,0 +1,88 @@
package com.archipelago.app.ui.screens
import okhttp3.OkHttpClient
import okhttp3.ResponseBody.Companion.toResponseBody
import okhttp3.mockwebserver.MockResponse
import okhttp3.mockwebserver.MockWebServer
import okio.Buffer
import org.junit.Assert.*
import org.junit.Test
import java.io.ByteArrayOutputStream
import java.io.IOException
import java.util.concurrent.CancellationException
class WebViewDownloadsTest {
private fun spec(url: String) = WebDownload(url, "test-agent", "session=test-only", "file.bin", "application/octet-stream")
@Test fun authenticatedDownloadWritesExactBytesAndReportsCompletion() {
MockWebServer().use { server ->
val bytes = ByteArray(256 * 1024 + 13) { (it % 251).toByte() }
server.enqueue(MockResponse().setBody(Buffer().write(bytes)))
val out = ByteArrayOutputStream()
var progress = 0L to 0L
assertEquals(bytes.size.toLong(), streamWebDownload(spec(server.url("/file").toString()), out, OkHttpClient(), onProgress = { done, total -> progress = done to total }))
assertArrayEquals(bytes, out.toByteArray())
assertEquals(bytes.size.toLong() to bytes.size.toLong(), progress)
assertEquals("session=test-only", server.takeRequest().getHeader("Cookie"))
}
}
@Test fun sameOriginRedirectKeepsSessionButCrossOriginNeverReceivesIt() {
MockWebServer().use { first -> MockWebServer().use { second ->
second.enqueue(MockResponse().setBody("final"))
first.enqueue(MockResponse().setResponseCode(302).addHeader("Location", "/relative"))
first.enqueue(MockResponse().setResponseCode(307).addHeader("Location", second.url("/target")))
val out = ByteArrayOutputStream()
streamWebDownload(spec(first.url("/start").toString()), out, OkHttpClient())
assertEquals("final", out.toString())
assertEquals("session=test-only", first.takeRequest().getHeader("Cookie"))
assertEquals("session=test-only", first.takeRequest().getHeader("Cookie"))
assertNull(second.takeRequest().getHeader("Cookie"))
} }
}
@Test fun authenticationFailureDoesNotSaveErrorBody() {
MockWebServer().use { server ->
server.enqueue(MockResponse().setResponseCode(401).setBody("login required"))
val out = ByteArrayOutputStream()
val error = assertThrows(IOException::class.java) { streamWebDownload(spec(server.url("/").toString()), out, OkHttpClient()) }
assertTrue(error.message!!.startsWith("Sign in"))
assertEquals(0, out.size())
}
}
@Test fun redirectsAreBoundedAndUnsafeSchemesAreRejected() {
MockWebServer().use { server ->
repeat(6) { server.enqueue(MockResponse().setResponseCode(302).addHeader("Location", "/loop")) }
assertThrows(IOException::class.java) { streamWebDownload(spec(server.url("/loop").toString()), ByteArrayOutputStream(), OkHttpClient()) }
assertEquals(6, server.requestCount)
}
for (url in listOf("file:///etc/passwd", "data:text/plain,test", "blob:test")) {
assertThrows(IOException::class.java) { streamWebDownload(spec(url), ByteArrayOutputStream(), OkHttpClient()) }
}
}
@Test fun cancellationAndDestinationFailureAreNotReportedAsComplete() {
MockWebServer().use { server ->
server.enqueue(MockResponse().setBody("bytes"))
assertThrows(CancellationException::class.java) { streamWebDownload(spec(server.url("/").toString()), ByteArrayOutputStream(), OkHttpClient(), checkCancelled = { throw CancellationException() }) }
assertEquals(0, server.requestCount)
var progressCalled = false
val out = object : java.io.OutputStream() { override fun write(b: Int) { throw IOException("disk full") } }
assertThrows(IOException::class.java) { streamWebDownload(spec(server.url("/").toString()), out, OkHttpClient(), onProgress = { _, _ -> progressCalled = true }) }
assertFalse(progressCalled)
}
}
@Test fun tlsDowngradeAndLoginHtmlAreRejected() {
var requests = 0
val client = OkHttpClient.Builder().addInterceptor { chain ->
requests++
okhttp3.Response.Builder().request(chain.request()).protocol(okhttp3.Protocol.HTTP_1_1)
.code(302).message("redirect").header("Location", "http://example.test/file").body("".toResponseBody(null)).build()
}.build()
assertThrows(IOException::class.java) { streamWebDownload(spec("https://example.test/file"), ByteArrayOutputStream(), client) }
assertEquals(1, requests)
MockWebServer().use { server ->
server.enqueue(MockResponse().addHeader("Content-Type", "Text/HTML; charset=utf-8").setBody("<html>Sign in</html>"))
val out = ByteArrayOutputStream()
assertThrows(IOException::class.java) { streamWebDownload(spec(server.url("/file").toString()), out, OkHttpClient()) }
assertEquals(0, out.size())
}
}
}
@@ -0,0 +1,71 @@
package com.archipelago.app.ui.screens
import android.view.View
import android.widget.FrameLayout
import androidx.activity.ComponentActivity
import org.junit.Assert.*
import org.junit.Test
import org.junit.runner.RunWith
import org.robolectric.Robolectric
import org.robolectric.RobolectricTestRunner
import org.robolectric.annotation.Config
@RunWith(RobolectricTestRunner::class)
@Config(manifest = Config.NONE, sdk = [28, 35])
class WebViewFullscreenTest {
@Test fun backExitsFullscreenWithoutFinishingActivityAndNotifiesOnce() {
val lifecycle = Robolectric.buildActivity(ComponentActivity::class.java).setup()
try {
val activity = lifecycle.get()
val fullscreen = WebViewFullscreen(activity)
val video = View(activity)
var hidden = 0
fullscreen.show(video) { hidden++ }
assertNotNull(video.parent)
activity.onBackPressedDispatcher.onBackPressed()
assertNull(video.parent)
assertFalse(activity.isFinishing)
assertEquals(1, hidden)
fullscreen.hide()
assertEquals(1, hidden)
} finally { lifecycle.pause().stop().destroy() }
}
@Test fun duplicateRequestPreservesActiveViewAndCanReenterAfterExit() {
val lifecycle = Robolectric.buildActivity(ComponentActivity::class.java).setup()
try {
val activity = lifecycle.get()
val fullscreen = WebViewFullscreen(activity)
val first = View(activity)
val second = View(activity)
var firstHidden = 0
var secondHidden = 0
fullscreen.show(first) { firstHidden++ }
fullscreen.show(second) { secondHidden++ }
assertNotNull(first.parent)
assertNull(second.parent)
assertEquals(0, firstHidden)
assertEquals(1, secondHidden)
fullscreen.hide()
fullscreen.show(second) { secondHidden++ }
assertNotNull(second.parent)
fullscreen.hide()
assertEquals(1, firstHidden)
assertEquals(2, secondHidden)
} finally { lifecycle.pause().stop().destroy() }
}
@Test fun rejectsOwnedViewWithoutReparentingAndHandlesUnavailableActivity() {
val lifecycle = Robolectric.buildActivity(ComponentActivity::class.java).setup()
try {
val activity = lifecycle.get()
val video = View(activity)
val owner = FrameLayout(activity).apply { addView(video) }
var hidden = 0
WebViewFullscreen(activity).show(video) { hidden++ }
assertSame(owner, video.parent)
WebViewFullscreen(null).show(null) { hidden++ }
assertEquals(2, hidden)
} finally { lifecycle.pause().stop().destroy() }
}
}
+54 -1
View File
@@ -1,6 +1,59 @@
# Changelog
## Unreleased
## v1.9.0-alpha (2026-10-05)
Unpublished release candidate; qualification is still in progress.
- Keep Cuprate and NetBird supporting components out of app listings and consolidate BTCPay Server under Commerce.
- Default on-chain sends, channel opens and cooperative closes to a dynamic next-block fee target, preserving explicit slower and custom choices.
- Add reviewed fee-bump quotes, explicit budgets and durable operation tracking for supported wallet transactions.
- Preserve Nginx Proxy Manager storage, same-node upstream connectivity, certificates and access controls through managed migrations.
- Restrict public management access while retaining configured public apps and ACME certificate validation.
- Serve the Mempool explorer on the Angor indexer origin alongside its API.
- Include the self-contained LoRa flashing tool and explicit board selection in update and installer payloads.
- Preserve paid-file Lightning entitlements across restarts and recover settled invoices from LND. Retry delivery without paying again and retain purchased files in the owned cache.
- Return explicit payment-status errors with safe retry guidance when verification is unavailable.
- Keep upload progress on its original screen, show completion there or notify on other screens, and cancel active and queued uploads.
- Resume interrupted uploads while the app remains open, preserve the original destination, and verify saved file contents before reporting completion.
- Provision a unique private File Browser login on each node while keeping Cloud sign-in automatic and preserving existing accounts and files.
- Update Nostr dependencies to reject forged relay events and oversized encrypted messages; preserve native signing and encryption compatibility.
- Allow apps to opt in to a validated public-key list of user identities without granting signing access.
- Make transaction filters transparent and horizontally scrollable on mobile.
- Keep Immich internal services out of My Apps, avoid false recovery states for healthy stacks, and allow removal of retired catalog apps.
- Repair the redundant managed Portainer network override that can prevent startup, preserving custom overrides and persistent state.
- Offer Standard, Medium, Fast and custom fees when cooperatively closing Lightning channels.
- Add a clear-search icon to My Apps, Services and the App Store on desktop and mobile.
- Keep Angor Indexer and the optional Angor Relay in the signed app catalog. Full indexing requires a synced, unpruned Bitcoin node and its indexing dependencies.
## v1.8.22-alpha (2026-09-30)
- Fixed Nginx Proxy Manager launch readiness choosing a proxy listener instead of its admin port after container recreation.
- Network diagnostic failures no longer stop all apps or rebuild shared container networking.
- Prevented orphaned companion dashboards from repeatedly reinstalling themselves after their backend app was removed.
- Fixed companion dashboard builds still referencing a retired image registry.
- Fixed Angor Indexer health checks choosing IPv6 localhost for an IPv4 listener and unnecessarily restarting the working service.
- Prevented false app restarts by probing each published port at its actual bind address; Nginx Proxy Manager now checks its internal admin API.
- Added a backed-up migration for the recognized legacy Nginx Proxy Manager tunnel/LND port conflict in both OTA and ISO startup paths.
- Checked Bitcoin and Electrum companion dashboards instead of backend protocol ports, preserving dashboard access during initial sync.
- Removed web-interface waiting messages from headless services such as Phoenixd and clarified which interface is unavailable for launchable apps.
- Finished runtime app-file promotion before manifest loading, preventing startup catalog refresh from forgetting disk-only apps.
- Named the app in compact readiness messages and kept app-card actions aligned at the bottom.
- Removed duplicate Mempool cards caused by frontend container aliases in restored inventory.
- Kept installed apps visible through restarts and hard refreshes, and delayed app launches until their web interface is ready.
- Made Bitcoin version selection readable and usable in the ThinkPad kiosk, above the pruning settings.
- Restored GitWorkshop build files in installation/update payloads and made slow image-pull progress clearer.
- Fixed same-node Gitea access from Portainer, with persistent runtime migration, state backups and recovery after failed restarts.
- Preserved Gitea configuration and SSH operation during fresh setup and upgrades.
- Improved paid-file delivery, saved-file permissions and repeat-download compatibility; verified Tor-only payment with change, rejection refunds and free repeat downloads.
- Added a headless Angor Indexer service using the existing Mempool/ElectrumX stack, and an optional separate Angor relay.
- Prevented manifest command arguments containing apostrophes from being corrupted in generated services.
## v1.8.21-alpha (2026-09-30)
+38 -1
View File
@@ -64,12 +64,49 @@ App submissions must:
## Pull requests
1. Open one focused PR per behavior or documentation change.
Contributions, PRs and reviews live on **ngit**. Clone the canonical repository:
```text
nostr://npub1w3sqdkrhn0gyuvsex32effzgnfpyde6qrrc4u467flg5e9txh4wsfn5vjg/relay.ngit.dev/archy
```
Gitea is a conventional Git mirror of accepted `main` commits and release tags.
You do not need to open a duplicate Gitea PR. Existing Gitea contributions will
be reviewed and linked to their ngit replacement or accepted result before
closure.
1. Open one focused ngit PR per behavior or documentation change.
2. Explain what changed, why it changed, and how it was verified.
3. Include screenshots for UI changes.
4. Link relevant issues or docs.
5. Keep generated catalog changes in sync with manifest changes.
### Maintainer publication gate
Merge once through the ngit contribution workflow, then push the exact same
accepted commits to Gitea. Do not independently merge or squash on each mirror.
Publish identical release tag objects, including annotations and signatures.
After pushing main, verify:
```bash
python3 scripts/check-git-mirrors.py --local
```
Before publishing release artifacts, also check the actual release tag:
```bash
python3 scripts/check-git-mirrors.py --local --ref refs/tags/v1.9.0-alpha
```
Use the release's actual tag name. Missing refs, inaccessible mirrors or differing
object IDs block publication. Record the ngit PR disposition and resulting merge
commit in the release acceptance ledger. Resolve drift deliberately; do not
force-push or delete published history without explicit approval.
The checker is read-only. `--all` audits every advertised branch and tag; ngit
proposal branches may intentionally differ from Gitea. A main-only pass proves
only main parity, and no Git ref check verifies PR discussions or review state.
Suggested commit format:
```text
+2
View File
@@ -1,5 +1,7 @@
# Archipelago
> **Alpha testing:** Archipelago is experimental software. Any funds you put on it are at your own risk.
> Self-sovereign Bitcoin node OS and manifest-driven app platform.
Archipelago is a bootable personal server OS for Bitcoin infrastructure,
+32 -3
View File
@@ -436,13 +436,13 @@
{
"id": "nginx-proxy-manager",
"title": "Nginx Proxy Manager",
"version": "2.12.1",
"description": "Reverse proxy with SSL. Beautiful web interface for managing proxies. On a node, this manages its admin UI and upstream configuration — the proxy's own :80/:443 listeners are not published (the node's web server owns those ports).",
"version": "2.14.0",
"description": "Reverse proxy with SSL. Beautiful web interface for managing proxies. The node's public web server forwards configured domains through this service, preserving its access lists, certificates and custom routes.",
"icon": "/assets/img/app-icons/nginx.svg",
"author": "Nginx Proxy Manager",
"category": "networking",
"tier": "optional",
"dockerImage": "source.archipelago-foundation.org/lfg2025/nginx-proxy-manager:latest",
"dockerImage": "source.archipelago-foundation.org/lfg2025/nginx-proxy-manager@sha256:8b91afcca90f5f2a7b2b8937999824f623c8a8748ae8013a1c9bf94f62177f08",
"repoUrl": "https://github.com/NginxProxyManager/nginx-proxy-manager"
},
{
@@ -644,6 +644,35 @@
"/var/lib/archipelago/vaultwarden:/data"
]
}
},
{
"id": "angor-indexer",
"title": "Angor Indexer",
"version": "1.0.2",
"description": "Bitcoin indexer endpoint for Angor with the existing Mempool explorer. Reuses this node’s Mempool and Electrum index; requires a synced, unpruned Bitcoin node. Add this service’s address as the custom indexer in Angor settings. A relay is optional and installed separately.",
"dockerImage": "source.archipelago-foundation.org/chaum/angor-indexer:1.0.2",
"author": "Angor / Archipelago",
"requires": [
"Mempool API",
"Unpruned Bitcoin"
],
"category": "money",
"tier": "optional",
"icon": "/assets/img/app-icons/angor-green.png",
"repoUrl": "https://github.com/block-core/angor"
},
{
"id": "angor-relay",
"title": "Angor Relay",
"version": "1.1.2",
"description": "Optional dedicated Nostr relay for Angor project metadata. Separate storage and access settings keep the node’s internal relay private. Add this service’s address to Angor’s relay settings; use WSS for browser clients.",
"dockerImage": "source.archipelago-foundation.org/chaum/angor-relay:1.1.2",
"author": "Angor / Archipelago",
"requires": [],
"category": "nostr",
"tier": "optional",
"icon": "/assets/img/app-icons/angor-green.png",
"repoUrl": "https://github.com/hoytech/strfry"
}
]
}
+109
View File
@@ -0,0 +1,109 @@
# Angor Indexer
Mainnet indexer endpoint for Angor, serving the existing Mempool explorer at
the same origin. The service reuses this node's Mempool frontend/backend and
Electrum index instead of creating another explorer or blockchain database.
An unpruned, fully synced Bitcoin node is required. Installing against a pruned
node must show the existing archival-node requirement; it must never silently
unprune or replace its Bitcoin data.
## Connect Angor
Install **Angor Indexer** in the store. Its API appears under **Services**.
In Angor settings, use `http://<node-address>:8998/` as the custom indexer origin.
The `/health` endpoint reports readiness against Mempool's indexed block height;
it returns 503 while that backend is unavailable. Index building may take time.
Browser clients require a reachable HTTPS origin with a trusted certificate.
Configure your HTTPS reverse proxy to forward to port 8998, then use that HTTPS
origin in Angor. Do not disable browser TLS checks. The API supports both
`/api/v1/` and `/api/` paths, transaction broadcast, and CORS without cookies.
This endpoint intentionally exposes public blockchain queries and transaction
broadcast through the app gate without dashboard-cookie login. It has no Bitcoin
RPC password, wallet keys, or persistent wallet data. The backend stays on the
managed container network; its private port does not become publicly exposed.
You can change network access using the node's normal access controls.
## Relay
A relay is optional. Angor can continue using its configured external relays.
Install **Angor Relay** separately to host project metadata locally, then add
`ws://<node-address>:8091/` in Angor, or a trusted `wss://` proxy origin for browser
clients. Its storage and configuration are separate from the node's internal
relay; installing or uninstalling it does not change the internal relay.
## Verify the complete client flow
The root URL opens the Mempool explorer. `/health` and fee
estimates establish API availability; they do not prove that project discovery,
address history, or browser CORS works. Test a known funded project's address
history, its original Nostr announcement, the Explore page, and project details
in the actual Angor client. A certificate alone does not establish public routing.
Keep existing discovery relays when adding a new relay. A new relay has no
historical project data and does not automatically replicate other relays.
Even with existing relays, an empty Explore page can be a client discovery
failure: Angor Hub v2.0.0 was observed to stop after a batch whose announcements
all failed on-chain validation. The same failure reproduced with our indexer
and Angor's public indexer. Do not bypass the funding transaction's event-ID
commitment or substitute an unsigned announcement to make a project appear.
For opt-in read-only browser acceptance, install the frontend test dependencies
and Playwright Chromium, then run:
```sh
ANGOR_TEST_INDEXER=https://indexer.example.com/ \
ANGOR_TEST_RELAY=wss://relay.example.com/ \
ANGOR_TEST_RELAYS='["wss://relay.angor.io","wss://relay.example.com/"]' \
node tests/lifecycle/angor-public-browser.cjs
```
The relay under test must already contain the known original public project
announcement documented in the test. The test does not import events, send
funds, change your browser profile, or disable TLS verification. It checks the
funding transaction/event commitment and real browser discovery and details.
Relay signed writes, invalid-signature rejection, persistence, full node sync,
and proxy upgrade/renewal tests remain separate acceptance requirements.
## Packaging
Build the pinned image with:
```
podman build -t source.archipelago-foundation.org/chaum/angor-indexer:1.0.2 apps/angor-indexer/container
```
The image runs as UID 101 with a read-only root filesystem and no capabilities.
Only temporary nginx state is writable. Runtime DNS is read from resolv.conf so
Mempool recreation does not require editing IP addresses or restarting this app.
No app-specific Rust installer is required.
Source documentation: [Angor's official deployment guide](https://github.com/block-core/angor/blob/869dd43cf38332dd7128a284a6bf4c1cac44c1a7/docker/DEPLOY-INDEXER-AND-RELAY.md).
The app icon is based on [Angor’s dark-mode app icon](https://angor.io/images/app-icon-dark-mode.png), retrieved 2026-09-30. At the operator’s request, the outer corners use the same green as the background. The built-in imagegen edit preserved the black mark and filled the square green; the project asset is `neode-ui/public/assets/img/app-icons/angor-green.png`.
Tests and release acceptance are recorded in the next-release checklist. The
health probe establishes backend availability, not a guarantee that every
address query is indexed at the latest Bitcoin tip.
Install Mempool Explorer first. The declarative `install_prerequisites` check
refuses a new adapter installation if its Mempool API component is absent, before
creating an installed-app record. It does not install or resync Bitcoin for you.
## Explorer on the public indexer origin
The linked official deployment guide exposes **Mempool frontend and API together**
on the public indexer URL. It uses standard Mempool images and requires no custom
Angor fork or `ANGOR_ENABLED` flag.
The operator now requires that same browser experience: opening the configured
indexer domain must show the existing Mempool explorer, while Angor API requests
continue working on that origin. Reuse the existing Mempool stack, including its
live WebSocket feed; do not install a second explorer or blockchain database.
**Candidate 1.0.2:** `/` and frontend paths proxy to the existing Mempool
frontend; `/api/`, `/api/v1/`, `/health` and the WebSocket feed retain their
indexer routes. Version 1.0.1 served only service JSON at `/`. The candidate
remains pending deployment/release acceptance, which must cover assets and deep links,
desktop/mobile rendering, WebSocket updates, API/CORS/broadcast, trusted HTTPS,
restart/upgrade and management-access isolation before documenting it as shipped.
+6
View File
@@ -0,0 +1,6 @@
FROM docker.io/library/nginx:1.31.3-alpine@sha256:1d40e3eb3bf4f138de1d67193f2aa5309fcaf343eb5ffadbf5e9439de1eb1ebb
COPY nginx.conf /etc/angor-nginx.conf.template
COPY entrypoint.sh /usr/local/bin/angor-indexer
USER 101:101
EXPOSE 8080
ENTRYPOINT ["/usr/local/bin/angor-indexer"]
+12
View File
@@ -0,0 +1,12 @@
#!/bin/sh
set -eu
# Resolve through the container runtime's DNS, including after dependency
# recreation. Never bake a container IP into the indexer endpoint.
DNS_RESOLVER=$(awk '/^nameserver[[:space:]]/ {print $2; exit}' /etc/resolv.conf)
case "$DNS_RESOLVER" in
''|*[!0-9a-fA-F.:]*) echo 'Container DNS resolver is unavailable' >&2; exit 1 ;;
esac
case "$DNS_RESOLVER" in *:*) DNS_RESOLVER="[$DNS_RESOLVER]" ;; esac
export DNS_RESOLVER
envsubst '${DNS_RESOLVER}' < /etc/angor-nginx.conf.template > /tmp/nginx.conf
exec nginx -c /tmp/nginx.conf -g 'daemon off;'
+81
View File
@@ -0,0 +1,81 @@
worker_processes 1;
pid /tmp/nginx.pid;
error_log /dev/stderr warn;
events { worker_connections 512; }
http {
access_log off;
server_tokens off;
client_body_temp_path /tmp/client_temp;
proxy_temp_path /tmp/proxy_temp;
fastcgi_temp_path /tmp/fastcgi_temp;
uwsgi_temp_path /tmp/uwsgi_temp;
scgi_temp_path /tmp/scgi_temp;
resolver ${DNS_RESOLVER} valid=10s ipv6=off;
upstream mempool_backend {
zone mempool_backend 64k;
server mempool-api:8999 resolve;
}
upstream mempool_frontend {
zone mempool_frontend 64k;
server mempool:8080 resolve;
}
map $http_upgrade $angor_connection_upgrade {
default upgrade;
'' close;
}
server {
listen 8080;
client_max_body_size 4m;
proxy_connect_timeout 5s;
proxy_read_timeout 60s;
proxy_send_timeout 30s;
proxy_http_version 1.1;
proxy_set_header Host $host;
proxy_set_header Connection $angor_connection_upgrade;
proxy_set_header Upgrade $http_upgrade;
proxy_set_header Authorization "";
proxy_set_header Cookie "";
proxy_hide_header Access-Control-Allow-Origin;
add_header Access-Control-Allow-Origin '*' always;
add_header Access-Control-Allow-Methods 'GET, HEAD, POST, OPTIONS' always;
add_header Access-Control-Allow-Headers 'Content-Type' always;
add_header Cache-Control 'no-store' always;
if ($request_method = OPTIONS) { return 204; }
# Mempool's backend uses /api/v1. Match its frontend's shorter /api
# surface too, without doubling already-versioned Angor URLs.
rewrite ^/api/(?!v1/)(.*)$ /api/v1/$1 last;
# Readiness checks the indexing backend, not this gateway's process.
location = /health {
limit_except GET { deny all; }
proxy_pass http://mempool_backend/api/v1/blocks/tip/height;
proxy_intercept_errors on;
error_page 500 502 503 504 =503 @waiting;
}
location @waiting {
default_type application/json;
return 503 '{"status":"waiting","message":"Waiting for Bitcoin and Mempool indexing"}\n';
}
location ~ ^/api/(v1/)?tx$ {
limit_except GET POST { deny all; }
proxy_pass http://mempool_backend;
}
location = /api/v1/ws {
limit_except GET { deny all; }
proxy_read_timeout 600s;
proxy_send_timeout 600s;
proxy_pass http://mempool_backend;
}
location /api/ {
limit_except GET { deny all; }
proxy_pass http://mempool_backend;
}
# Share the already-installed explorer; no second frontend or index DB.
# Its SPA handles transaction/block deep links and static assets.
location / {
limit_except GET { deny all; }
proxy_pass http://mempool_frontend;
proxy_intercept_errors on;
error_page 500 502 503 504 =503 @waiting;
}
}
}
+73
View File
@@ -0,0 +1,73 @@
app:
id: angor-indexer
name: Angor Indexer
version: 1.0.2
description: Bitcoin indexer endpoint for Angor with the existing Mempool explorer.
Reuses this node’s Mempool
and Electrum index; requires a synced, unpruned Bitcoin node. Add this service’s
address as the custom indexer in Angor settings. A relay is optional and installed
separately.
category: money
install_prerequisites:
- mempool
- mempool-api
upstream:
kind: github
repo: block-core/angor
container:
image: source.archipelago-foundation.org/chaum/angor-indexer:1.0.2
pull_policy: if-not-present
network: archy-net
dependencies:
- app_id: mempool
version: '>=3.0.0'
- app_id: mempool-api
version: '>=3.0.0'
- bitcoin:archival
resources:
cpu_limit: 1
memory_limit: 128Mi
disk_limit: 128Mi
security:
capabilities: []
readonly_root: true
no_new_privileges: true
user: 101
network_policy: isolated
ports:
- host: 8998
container: 8080
protocol: tcp
bind: 127.0.0.1
auth: open
auth_rationale: Public Bitcoin chain-data API and validated transaction broadcast for Angor clients; no wallet keys or node RPC credentials are exposed. Browser cookie login would break machine clients.
interfaces:
main:
name: Angor Indexer API
description: Use this origin as Angor’s custom mainnet indexer URL, or open it
to view the existing Mempool explorer. HTTPS is required for browser clients.
type: api
port: 8998
protocol: http
path: /
health_check:
type: http
endpoint: http://127.0.0.1:8080
path: /health
interval: 30s
timeout: 8s
retries: 3
bitcoin_integration:
rpc_access: none
sync_required: true
pruning_support: false
metadata:
icon: /assets/img/app-icons/angor-green.png
tier: optional
repo: https://github.com/block-core/angor
features:
- Angor mainnet API
- Mempool explorer on the same origin
- Reuses existing Mempool indexing
- No separate blockchain database
- Optional independent relay
+33
View File
@@ -0,0 +1,33 @@
# Angor Relay
Optional standalone strfry relay for Angor's public project metadata. See
[Angor Indexer setup](../angor-indexer/README.md) for client URLs and HTTPS/WSS.
The gate exposes port 8091 for Nostr clients. strfry validates event signatures;
this is a public relay, not a private messaging archive. It mounts only
`/var/lib/archipelago/angor-relay` and its separate configuration directory.
It never opens, reconfigures or shares the node's internal strfry database.
For a public domain, proxy HTTPS to node port **8091**, enable WebSocket upgrade,
and add `wss://your-relay-domain/` in Angor. Test both NIP-11 metadata (send
`Accept: application/nostr+json`) and a real Nostr subscription over WSS. An
Archipelago login page at this domain is a routing failure, not relay readiness.
New relays start without project history. Keep existing discovery relays alongside
yours until the needed original signed announcements and metadata are available
locally. Relays do not automatically synchronize. Any history import must retain
the original event IDs and signatures; verify funded projects against their
on-chain commitments. A working WebSocket with zero stored events is not proof
that the client's project discovery works. See the indexer README's browser test.
The configuration is seeded only when absent, preserving operator changes.
Stop the service before making a consistent backup of its event database.
Ordinary start/restart/recreation preserves both mounts. Use the standard app
lifecycle; do not manually recreate a systemd-managed container.
## Image provenance
Mirrored from `docker.io/dockurr/strfry:1.1.2`, upstream manifest digest
`sha256:e81d238db13507f6ef24c49d47cd0b0ea58ff207961f10581fa2a7c901054df4`.
The public Angor policy is supplied by this app's own configuration; it does not
reuse the internal relay's event whitelist.
+223
View File
@@ -0,0 +1,223 @@
app:
id: angor-relay
name: Angor Relay
version: 1.1.2
upstream:
kind: github
repo: hoytech/strfry
description: Optional dedicated Nostr relay for Angor project metadata. Separate
storage and access settings keep the node’s internal relay private. Add this service’s
address to Angor’s relay settings; use WSS for browser clients.
container:
image: source.archipelago-foundation.org/chaum/angor-relay:1.1.2
pull_policy: if-not-present
dependencies:
- storage: 5Gi
resources:
cpu_limit: 1
memory_limit: 512Mi
disk_limit: 5Gi
security:
capabilities: []
readonly_root: true
no_new_privileges: true
seccomp_profile: default
network_policy: isolated
apparmor_profile: nostr-relay
ports:
- host: 8091
container: 7777
protocol: tcp
bind: 127.0.0.1
auth: open
auth_rationale: Dedicated public Nostr relay for Angor project metadata; strfry verifies event signatures. It has separate storage from the private node relay and no wallet or node credentials.
volumes:
- type: bind
source: /var/lib/archipelago/angor-relay
target: /app/strfry-db
options:
- rw
- type: bind
source: /var/lib/archipelago/angor-relay-config/angor-relay.conf
target: /etc/strfry.conf
options:
- ro
files:
- path: /var/lib/archipelago/angor-relay-config/angor-relay.conf
overwrite: false
content: |
##
## Default strfry config
##
# Directory that contains the strfry LMDB database (restart required)
db = "./strfry-db/"
dbParams {
# Maximum number of threads/processes that can simultaneously have LMDB transactions open (restart required)
maxreaders = 256
# Size of mmap() to use when loading LMDB (default is 10TB, does *not* correspond to disk-space used) (restart required)
mapsize = 10995116277760
# Disables read-ahead when accessing the LMDB mapping. Reduces IO activity when DB size is larger than RAM. (restart required)
noReadAhead = false
}
events {
# Maximum size of normalised JSON, in bytes
maxEventSize = 65536
# Events newer than this will be rejected
rejectEventsNewerThanSeconds = 900
# Events older than this will be rejected
rejectEventsOlderThanSeconds = 94608000
# Ephemeral events older than this will be rejected
rejectEphemeralEventsOlderThanSeconds = 60
# Ephemeral events will be deleted from the DB when older than this
ephemeralEventsLifetimeSeconds = 300
# Maximum number of tags allowed
maxNumTags = 2000
# Maximum size for tag values, in bytes
maxTagValSize = 1024
}
relay {
# Interface to listen on. Use 0.0.0.0 to listen on all interfaces (restart required)
bind = "0.0.0.0"
# Port to open for the nostr websocket protocol (restart required)
port = 7777
# Set OS-limit on maximum number of open files/sockets (if 0, don't attempt to set) (restart required)
nofiles = 0
# HTTP header that contains the client's real IP, before reverse proxying (ie x-real-ip) (MUST be all lower-case)
realIpHeader = ""
info {
# NIP-11: Name of this server. Short/descriptive (< 30 characters)
name = "Angor Relay"
# NIP-11: Detailed information about relay, free-form
description = "Dedicated public relay for Angor project metadata."
# NIP-11: Administrative nostr pubkey, for contact purposes
pubkey = ""
# NIP-11: Alternative administrative contact (email, website, etc)
contact = ""
# NIP-11: URL pointing to an image to be used as an icon for the relay
icon = ""
# List of supported lists as JSON array, or empty string to use default. Example: "[1,2]"
nips = ""
}
# Maximum accepted incoming websocket frame size (should be larger than max event) (restart required)
maxWebsocketPayloadSize = 131072
# Maximum number of filters allowed in a REQ
maxReqFilterSize = 200
# Websocket-level PING message frequency (should be less than any reverse proxy idle timeouts) (restart required)
autoPingSeconds = 55
# If TCP keep-alive should be enabled (detect dropped connections to upstream reverse proxy)
enableTcpKeepalive = false
# How much uninterrupted CPU time a REQ query should get during its DB scan
queryTimesliceBudgetMicroseconds = 10000
# Maximum records that can be returned per filter
maxFilterLimit = 500
# Maximum number of subscriptions (concurrent REQs) a connection can have open at any time
maxSubsPerConnection = 20
writePolicy {
# If non-empty, path to an executable script that implements the writePolicy plugin logic
plugin = ""
}
compression {
# Use permessage-deflate compression if supported by client. Reduces bandwidth, but slight increase in CPU (restart required)
enabled = true
# Maintain a sliding window buffer for each connection. Improves compression, but uses more memory (restart required)
slidingWindow = true
}
logging {
# Dump all incoming messages
dumpInAll = false
# Dump all incoming EVENT messages
dumpInEvents = false
# Dump all incoming REQ/CLOSE messages
dumpInReqs = false
# Log performance metrics for initial REQ database scans
dbScanPerf = false
# Log reason for invalid event rejection? Can be disabled to silence excessive logging
invalidEvents = true
}
numThreads {
# Ingester threads: route incoming requests, validate events/sigs (restart required)
ingester = 3
# reqWorker threads: Handle initial DB scan for events (restart required)
reqWorker = 3
# reqMonitor threads: Handle filtering of new events (restart required)
reqMonitor = 3
# negentropy threads: Handle negentropy protocol messages (restart required)
negentropy = 2
}
negentropy {
# Support negentropy protocol messages
enabled = true
# Maximum records that sync will process before returning an error
maxSyncEvents = 1000000
}
}
health_check:
type: http
endpoint: http://127.0.0.1:7777
path: /health
interval: 30s
timeout: 5s
retries: 3
nostr_integration:
relay_type: public
monetization_enabled: false
category: nostr
interfaces:
main:
name: Angor Relay
description: Nostr WebSocket endpoint; use ws:// for LAN or wss:// through your
HTTPS domain.
type: api
port: 8091
protocol: http
path: /
metadata:
icon: /assets/img/app-icons/angor-green.png
tier: optional
repo: https://github.com/hoytech/strfry
features:
- Angor project metadata
- Separate from the node relay
- Persistent Nostr event storage
+15 -1
View File
@@ -15,6 +15,9 @@ app:
image: source.archipelago-foundation.org/lfg2025/gitea:1.27.3
pull_policy: if-not-present
# Preserve repositories, database, keys and configuration during runtime repairs.
backup_before_runtime_change: true
dependencies:
# Source history, LFS objects, release artifacts and OCI layers all share
# this persistent store. 500Mi was only suitable for an empty demo node.
@@ -25,7 +28,7 @@ app:
disk_limit: 50Gi
security:
capabilities: [CHOWN, FOWNER, SETUID, SETGID, DAC_OVERRIDE, NET_BIND_SERVICE]
capabilities: [CHOWN, FOWNER, SETUID, SETGID, DAC_OVERRIDE, NET_BIND_SERVICE, SYS_CHROOT]
readonly_root: false
no_new_privileges: false
network_policy: bridge
@@ -62,6 +65,17 @@ app:
target: /etc/gitea
options: [rw]
# Seed a fresh installation with the same origin advertised by the app gate.
# Existing app.ini (including custom HTTPS/domain settings) is never replaced.
files:
- path: /var/lib/archipelago/gitea/data/gitea/conf/app.ini
overwrite: false
content: |
[server]
DOMAIN = {{HOST_IP}}
SSH_DOMAIN = {{HOST_IP}}
ROOT_URL = http://{{HOST_IP}}:3001/
environment:
- GITEA__database__DB_TYPE=sqlite3
- GITEA__server__SSH_PORT=2222
+24 -8
View File
@@ -1,20 +1,23 @@
app:
id: nginx-proxy-manager
name: Nginx Proxy Manager
version: 2.12.1
version: 2.14.0
upstream:
kind: github
repo: NginxProxyManager/nginx-proxy-manager
description: >-
Reverse proxy with SSL. Beautiful web interface for managing proxies.
On a node, this manages its admin UI and upstream configuration — the
proxy's own :80/:443 listeners are not published (the node's web server
owns those ports).
The node's public web server forwards configured domains through this
service, preserving its access lists, certificates and custom routes.
backup_before_runtime_change: true
container:
image: source.archipelago-foundation.org/lfg2025/nginx-proxy-manager:latest
image: source.archipelago-foundation.org/lfg2025/nginx-proxy-manager@sha256:8b91afcca90f5f2a7b2b8937999824f623c8a8748ae8013a1c9bf94f62177f08
pull_policy: if-not-present
network: pasta
# Rootless pasta copies the LAN IP, preventing requests back to this node.
# Retain the old pasta host gateway used by saved NPM upstreams, plus
# host.containers.internal. This subnet stays inside the private rootless namespace.
network: slirp4netns:allow_host_loopback=true,cidr=169.254.1.0/24
dependencies:
- storage: 1Gi
@@ -49,6 +52,17 @@ app:
Nginx Proxy Manager enforces its own admin account on every page;
the initial setup wizard also has to answer before any account exists.
- host: 8088
container: 80
protocol: tcp
bind: 127.0.0.1
auth: local
- host: 8444
container: 443
protocol: tcp
bind: 127.0.0.1
auth: local
volumes:
- type: bind
source: /var/lib/archipelago/nginx-proxy-manager
@@ -64,9 +78,11 @@ app:
environment: []
# Probe the admin API inside the container, independent of optional
# tunnel listeners. This also verifies the Node backend is ready.
health_check:
type: tcp
endpoint: localhost:81
type: http
endpoint: http://127.0.0.1:81/api/
interval: 30s
timeout: 5s
retries: 3
+1 -1
View File
@@ -22,7 +22,7 @@ app:
data_uid: "1000:1000"
# Snapshot state before an upgrade recreates this app with new networking.
backup_on_network_change: true
backup_before_runtime_change: true
dependencies:
- storage: 1Gi
+5 -5
View File
@@ -104,7 +104,7 @@ dependencies = [
[[package]]
name = "archipelago"
version = "1.8.21-alpha"
version = "1.9.0-alpha"
dependencies = [
"anyhow",
"archipelago-container",
@@ -3636,9 +3636,9 @@ dependencies = [
[[package]]
name = "nostr"
version = "0.44.2"
version = "0.44.7"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "3aa5e3b6a278ed061835fe1ee293b71641e6bf8b401cfe4e1834bbf4ef0a34e1"
checksum = "c7d3d987ea7078dc36947cde532637c472a229426702e4331dd7667325378bd9"
dependencies = [
"aes",
"base64 0.22.1",
@@ -3681,9 +3681,9 @@ dependencies = [
[[package]]
name = "nostr-relay-pool"
version = "0.44.0"
version = "0.44.3"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "4b1073ccfbaea5549fb914a9d52c68dab2aecda61535e5143dd73e95445a804b"
checksum = "c85c54d6ca9aae4ae2bf19a7663ba9db5f45f783f1d24aff55f006386b8b99a1"
dependencies = [
"async-utility",
"async-wsocket",
+1 -1
View File
@@ -1,6 +1,6 @@
[package]
name = "archipelago"
version = "1.8.21-alpha"
version = "1.9.0-alpha"
edition = "2021"
license.workspace = true
description = "Archipelago Bitcoin Node OS - Native backend"
+192 -53
View File
@@ -74,7 +74,7 @@ impl ApiHandler {
let invoice_hash = headers
.get("x-invoice-hash")
.and_then(|v| v.to_str().ok())
.map(|s| s.to_string())
.map(|s| s.to_ascii_lowercase())
.or_else(|| {
headers
.get("x-onchain-address")
@@ -98,6 +98,46 @@ impl ApiHandler {
None => false,
};
// Payment settlement is verified on the seller even when no status
// poll preceded this download (e.g. direct payment from another node).
let requires_payment = if !owner_session && headers.contains_key("x-invoice-hash") {
content_server::load_catalog(&config.data_dir)
.await?
.items
.iter()
.any(|item| {
item.id == content_id
&& matches!(item.access, content_server::AccessControl::Paid { .. })
})
} else {
false
};
if requires_payment {
if let Some(hash) = headers.get("x-invoice-hash").and_then(|v| v.to_str().ok()) {
if hash.len() != 64 || !hash.bytes().all(|c| c.is_ascii_hexdigit()) {
return Ok(build_response(
StatusCode::BAD_REQUEST,
"text/plain",
hyper::Body::from("Invalid payment hash"),
));
}
if let Err(error) = self
.rpc_handler
.settle_content_invoice(hash, content_id)
.await
{
tracing::warn!("Cannot verify peer-file invoice settlement: {error:#}");
return Ok(build_response(
StatusCode::SERVICE_UNAVAILABLE,
"application/json",
hyper::Body::from(
r#"{"error":"Payment verification is temporarily unavailable. Retry the download without paying again."}"#,
),
));
}
}
}
// Parse Range header for streaming support
let range = headers
.get("range")
@@ -162,11 +202,33 @@ impl ApiHandler {
r#"{"error":"This file is shared with the host's federation peers only. Federate with that node (exchange invites) so it recognizes you, then try again."}"#,
),
)),
Ok(content_server::ServeResult::NotFound) | Err(_) => Ok(build_response(
Ok(content_server::ServeResult::Unavailable) => Ok(build_response(
StatusCode::SERVICE_UNAVAILABLE,
"application/json",
hyper::Body::from(
r#"{"error":"The seller's node can't read this file right now. This request did not redeem an ecash payment."}"#,
),
)),
Ok(content_server::ServeResult::RangeNotSatisfiable(total)) => Ok(Response::builder()
.status(StatusCode::RANGE_NOT_SATISFIABLE)
.header("Content-Range", format!("bytes */{total}"))
.body(hyper::Body::empty())
.unwrap()),
Ok(content_server::ServeResult::NotFound) => Ok(build_response(
StatusCode::NOT_FOUND,
"text/plain",
hyper::Body::from("Content not found"),
)),
// Not a 404: a paid request may already have been charged by the
// time this fails, and "not found" hid the real error entirely.
Err(e) => {
tracing::error!("Serving content {content_id} failed: {e:#}");
Ok(build_response(
StatusCode::INTERNAL_SERVER_ERROR,
"text/plain",
hyper::Body::from("Failed to serve content"),
))
}
}
}
@@ -227,7 +289,13 @@ impl ApiHandler {
.await
{
Ok((bolt11, payment_hash)) if !payment_hash.is_empty() => {
crate::content_invoice::record_pending(&payment_hash, content_id, price_sats).await;
crate::content_invoice::record_pending(
&self.config.data_dir,
&payment_hash,
content_id,
price_sats,
)
.await?;
let body = serde_json::json!({
"bolt11": bolt11,
"payment_hash": payment_hash,
@@ -268,52 +336,10 @@ impl ApiHandler {
&self,
path: &str,
) -> Result<Response<hyper::Body>> {
let rest = path.strip_prefix("/content/").unwrap_or("");
let (content_id, payment_hash) = match rest.split_once("/invoice-status/") {
Some((id, hash)) => (id, hash),
None => {
return Ok(build_response(
StatusCode::BAD_REQUEST,
"text/plain",
hyper::Body::from("Invalid request"),
))
}
};
if content_id.is_empty() || !is_valid_app_id(content_id) || payment_hash.is_empty() {
return Ok(build_response(
StatusCode::BAD_REQUEST,
"text/plain",
hyper::Body::from("Invalid request"),
));
}
// The hash must be one we issued for exactly this content item.
match crate::content_invoice::lookup(payment_hash).await {
Some((cid, _)) if cid == content_id => {}
_ => {
return Ok(build_response(
StatusCode::NOT_FOUND,
"application/json",
hyper::Body::from(r#"{"error":"Unknown invoice"}"#),
))
}
}
// Already paid? Otherwise ask our LND and persist the result.
let mut paid = crate::content_invoice::is_paid_for(payment_hash, content_id).await;
if !paid {
if let Ok(true) = self.rpc_handler.invoice_is_settled(payment_hash).await {
crate::content_invoice::mark_paid(payment_hash).await;
paid = true;
}
}
let body = serde_json::json!({ "paid": paid });
Ok(build_response(
StatusCode::OK,
"application/json",
hyper::Body::from(serde_json::to_vec(&body).unwrap_or_default()),
))
Ok(invoice_status_response(path, |hash, id| async move {
self.rpc_handler.settle_content_invoice(&hash, &id).await
})
.await)
}
/// Seller side (#46): issue a fresh on-chain address for a paid catalog item
@@ -367,7 +393,13 @@ impl ApiHandler {
match self.rpc_handler.new_onchain_address().await {
Ok(address) if !address.is_empty() => {
crate::content_invoice::record_pending(&address, content_id, price_sats).await;
crate::content_invoice::record_pending(
&self.config.data_dir,
&address,
content_id,
price_sats,
)
.await?;
let body = serde_json::json!({
"address": address,
"amount_sats": price_sats,
@@ -417,7 +449,7 @@ impl ApiHandler {
));
}
// The address must be one we issued for exactly this content item.
let price = match crate::content_invoice::lookup(address).await {
let price = match crate::content_invoice::lookup(&self.config.data_dir, address).await? {
Some((cid, price)) if cid == content_id => price,
_ => {
return Ok(build_response(
@@ -428,10 +460,11 @@ impl ApiHandler {
}
};
let mut paid = crate::content_invoice::is_paid_for(address, content_id).await;
let mut paid =
crate::content_invoice::is_paid_for(&self.config.data_dir, address, content_id).await;
if !paid {
if let Ok(true) = self.rpc_handler.onchain_received(address, price).await {
crate::content_invoice::mark_paid(address).await;
crate::content_invoice::mark_paid(&self.config.data_dir, address).await?;
paid = true;
}
}
@@ -509,3 +542,109 @@ impl ApiHandler {
}
}
}
/// Keep invalid input and an unavailable wallet inside the HTTP protocol so
/// buyers can retry delivery without treating a dropped socket as lost payment.
async fn invoice_status_response<F, Fut>(path: &str, settle: F) -> Response<hyper::Body>
where
F: FnOnce(String, String) -> Fut,
Fut: std::future::Future<Output = Result<bool>>,
{
let parsed = path
.strip_prefix("/content/")
.and_then(|rest| rest.split_once("/invoice-status/"))
.filter(|(id, hash)| {
!id.is_empty()
&& is_valid_app_id(id)
&& hash.len() == 64
&& hash.bytes().all(|c| c.is_ascii_hexdigit())
});
let Some((id, hash)) = parsed else {
return build_response(
StatusCode::BAD_REQUEST,
"application/json",
hyper::Body::from(r#"{"error":"Invalid content ID or payment hash"}"#),
);
};
match settle(hash.to_ascii_lowercase(), id.to_owned()).await {
Ok(paid) => build_response(
StatusCode::OK,
"application/json",
hyper::Body::from(serde_json::json!({"paid": paid}).to_string()),
),
Err(_) => {
tracing::warn!("Peer-file payment status verification is temporarily unavailable");
let mut response = build_response(
StatusCode::SERVICE_UNAVAILABLE,
"application/json",
hyper::Body::from(
r#"{"error":"Payment verification is temporarily unavailable. Retry without paying again."}"#,
),
);
response.headers_mut().insert(
hyper::header::RETRY_AFTER,
hyper::header::HeaderValue::from_static("5"),
);
response
}
}
}
#[cfg(test)]
mod invoice_status_tests {
use super::*;
#[tokio::test]
async fn malformed_requests_do_not_query_the_wallet() {
for path in [
"/bad",
"/content//invoice-status/aa",
"/content/file/invoice-status/aa",
"/content/file/invoice-status/",
"/content/file/invoice-status/not-a-hash",
] {
let response = invoice_status_response(path, |_, _| async {
panic!("Invalid request reached wallet");
#[allow(unreachable_code)]
Ok(false)
})
.await;
assert_eq!(response.status(), StatusCode::BAD_REQUEST);
assert_eq!(response.headers()["content-type"], "application/json");
let body = hyper::body::to_bytes(response.into_body()).await.unwrap();
assert!(
serde_json::from_slice::<serde_json::Value>(&body).unwrap()["error"].is_string()
);
}
}
#[tokio::test]
async fn settlement_results_and_failures_have_explicit_http_responses() {
let hash = "AB".repeat(32);
let path = format!("/content/file/invoice-status/{hash}");
for paid in [false, true] {
let response = invoice_status_response(&path, |hash, id| async move {
assert_eq!(hash, "ab".repeat(32));
assert_eq!(id, "file");
Ok(paid)
})
.await;
assert_eq!(response.status(), StatusCode::OK);
let body = hyper::body::to_bytes(response.into_body()).await.unwrap();
assert_eq!(
serde_json::from_slice::<serde_json::Value>(&body).unwrap()["paid"],
paid
);
}
let response = invoice_status_response(&path, |_, _| async {
anyhow::bail!("private wallet details must not escape")
})
.await;
assert_eq!(response.status(), StatusCode::SERVICE_UNAVAILABLE);
assert_eq!(response.headers()["retry-after"], "5");
let body = hyper::body::to_bytes(response.into_body()).await.unwrap();
let text = String::from_utf8(body.to_vec()).unwrap();
assert!(text.contains("without paying again"));
assert!(!text.contains("private wallet"));
}
}
+1 -1
View File
@@ -136,7 +136,7 @@ impl RpcHandler {
/// ~30% of UI calls error out even though the node is perfectly healthy.
/// With retry + backoff, the UI sees a uniform slow-but-successful
/// response instead of intermittent failures.
async fn bitcoin_rpc_call<T: serde::de::DeserializeOwned>(
pub(in crate::api::rpc) async fn bitcoin_rpc_call<T: serde::de::DeserializeOwned>(
&self,
client: &reqwest::Client,
method: &str,
+171 -121
View File
@@ -43,6 +43,25 @@ async fn reclaim_spent_ecash(data_dir: &std::path::Path, token: &str, backend: &
}
}
/// Only pass through the peer's bounded, printable explanation; refund status
/// is always determined locally and must never come from the peer's wording.
fn seller_error_message(status: reqwest::StatusCode, body: &str) -> String {
let reason = serde_json::from_str::<serde_json::Value>(body)
.ok()
.and_then(|v| v.get("error").and_then(|e| e.as_str()).map(str::to_owned));
match reason {
Some(reason) if !reason.trim().is_empty() => {
let clean: String = reason
.chars()
.filter(|c| !c.is_control())
.take(240)
.collect();
format!("Seller response ({status}): {clean}")
}
_ => format!("Peer returned an error ({status})."),
}
}
/// Keep first purchases and cached repeats compatible with both existing clients.
fn paid_content_response(bytes: &[u8], mime: &str, paid_sats: u64) -> serde_json::Value {
use base64::Engine;
@@ -54,13 +73,48 @@ fn paid_content_response(bytes: &[u8], mime: &str, paid_sats: u64) -> serde_json
})
}
/// FileBrowser owns its files through a rootless UID mapping. Use its authenticated
/// API rather than writing host paths with the backend's unrelated UID. Its
/// override=false upload atomically refuses existing names, including races.
// Resolve known purchases BEFORE any mint/spend. Missing bytes or an unreadable
// index require recovery; neither is authorization to charge the buyer again.
async fn existing_paid_content(
data_dir: &std::path::Path,
onion: &str,
content_id: &str,
filename: Option<&str>,
) -> Result<Option<serde_json::Value>> {
let owned = crate::content_owned::list_owned_checked(data_dir)
.await
.context("Could not verify previous purchases; no new payment was sent")?;
let Some(item) = owned.iter().find(|o| {
o.onion == onion
&& (o.content_id == content_id
|| filename.is_some_and(|f| {
!f.is_empty() && o.filename.trim_start_matches('/') == f.trim_start_matches('/')
}))
}) else {
return Ok(None);
};
let (mime, bytes) = crate::content_owned::read_owned(data_dir, &item.onion, &item.content_id)
.await.context("This purchase is recorded, but its cached file is unavailable. No new payment was sent. Restore the cached file or contact the seller.")?;
let mut response = paid_content_response(&bytes, &mime, 0);
response["already_owned"] = serde_json::json!(true);
response["filename"] = serde_json::json!(item.filename);
Ok(Some(response))
}
// Updated clients open the persisted file through the Range-capable HTTP
// endpoint. Avoid putting two base64 copies of a large video in a JSON reply.
// Keep older clients compatible until both sides have upgraded.
fn invoice_download_response(bytes: &[u8], mime: &str, cache_only: bool) -> serde_json::Value {
if cache_only {
serde_json::json!({ "owned": true, "mime_type": mime, "size_bytes": bytes.len() })
} else {
paid_content_response(bytes, mime, 0)
}
}
/// File purchases through an atomic no-clobber write in Files' own namespace.
async fn file_purchase_in_files(
client: &reqwest::Client,
base_url: &str,
token: &str,
data_dir: &std::path::Path,
filename: &str,
mime: &str,
bytes: &[u8],
@@ -72,59 +126,24 @@ async fn file_purchase_in_files(
} else {
"Documents"
};
let mut folder_url = reqwest::Url::parse(base_url)?;
folder_url
.path_segments_mut()
.map_err(|_| anyhow::anyhow!("Invalid Files URL"))?
.extend(["api", "resources", folder, ""]);
let response = client
.get(folder_url.clone())
.header("X-Auth", token)
.send()
.await?;
if response.status() == reqwest::StatusCode::NOT_FOUND {
let response = client
.post(folder_url.clone())
.header("X-Auth", token)
.send()
.await?;
if response.status() != reqwest::StatusCode::CONFLICT {
response.error_for_status()?;
}
} else {
response.error_for_status()?;
}
let base = std::path::Path::new(filename)
let root = data_dir.join("filebrowser");
anyhow::ensure!(
tokio::fs::metadata(&root).await?.is_dir(),
"Files storage is unavailable"
);
let name = std::path::Path::new(filename)
.file_name()
.and_then(|n| n.to_str())
.filter(|n| !n.is_empty())
.unwrap_or("download");
let (stem, extension) = match base.rsplit_once('.') {
Some((stem, ext)) if !stem.is_empty() => (stem, format!(".{ext}")),
_ => (base, String::new()),
};
for attempt in 1..=100 {
let name = if attempt == 1 {
base.to_string()
} else {
format!("{stem} ({attempt}){extension}")
};
let mut url = folder_url.clone();
url.path_segments_mut().unwrap().pop_if_empty().push(&name);
url.query_pairs_mut().append_pair("override", "false");
let response = client
.post(url)
.header("X-Auth", token)
.body(bytes.to_vec())
.send()
.await?;
if response.status() == reqwest::StatusCode::CONFLICT {
continue;
}
response.error_for_status()?;
return Ok(format!("{folder}/{name}"));
}
anyhow::bail!("Too many existing copies; purchased file remains in the purchase cache")
let path =
crate::container::filebrowser::save_new_file(&root.join(folder), name, bytes).await?;
Ok(format!(
"{folder}/{}",
path.file_name()
.and_then(|n| n.to_str())
.context("Invalid Files name")?
))
}
impl RpcHandler {
@@ -526,36 +545,15 @@ impl RpcHandler {
// by exact (onion, content_id) and by (onion, filename) — the latter
// catches duplicate ids pointing at the same file on the same
// seller. The owned copy is served from the local cache instead.
{
let filename = params.get("filename").and_then(|v| v.as_str());
let owned = crate::content_owned::list_owned(&self.config.data_dir).await;
let already = owned.iter().find(|o| {
o.onion == onion
&& (o.content_id == content_id
|| filename.is_some_and(|f| {
!f.is_empty()
&& o.filename.trim_start_matches('/') == f.trim_start_matches('/')
}))
});
if let Some(o) = already {
tracing::info!(
if let Some(cached) = existing_paid_content(
&self.config.data_dir,
onion,
content_id,
owned_as = %o.content_id,
"paid download: already owned — serving cached copy, NOT paying again"
);
if let Some((mime, bytes)) =
crate::content_owned::read_owned(&self.config.data_dir, &o.onion, &o.content_id)
.await
params.get("filename").and_then(|v| v.as_str()),
)
.await?
{
let mut result = paid_content_response(&bytes, &mime, 0);
result["already_owned"] = serde_json::json!(true);
result["filename"] = serde_json::json!(o.filename);
return Ok(result);
}
// Cache record exists but bytes are gone — fall through and
// repurchase rather than stranding the user.
}
return Ok(cached);
}
// `method` pins the backend the user confirmed in the UI ("cashu" |
@@ -623,13 +621,14 @@ impl RpcHandler {
let path = format!("/content/{}", content_id);
// Surface a real reason instead of the generic sanitized error (#30):
// the dial already tries FIPS/mesh then falls back to Tor, so a failure
// here means the peer is genuinely unreachable on both transports.
// A bearer token must not be replayed after an ambiguous delivery.
// A transport error can mean the seller received it without replying.
let (response, transport) =
match crate::fips::dial::PeerRequest::new(fips_npub.as_deref(), onion, &path)
.service(crate::settings::transport::PeerService::PeerFiles)
.header("X-Federation-DID", local_did)
.header("X-Payment-Token", token_str.clone())
.single_delivery()
.timeout(std::time::Duration::from_secs(900))
.send_get()
.await
@@ -642,7 +641,7 @@ impl RpcHandler {
let refund =
reclaim_spent_ecash(&self.config.data_dir, &token_str, used_backend).await;
return Ok(serde_json::json!({
"error": format!("Could not reach the peer over mesh or Tor. {refund}")
"error": format!("The purchase could not be completed. {refund}")
}));
}
};
@@ -679,7 +678,7 @@ impl RpcHandler {
tracing::warn!("paid download: seller {onion} returned {status}: {body}");
let refund = reclaim_spent_ecash(&self.config.data_dir, &token_str, used_backend).await;
return Ok(serde_json::json!({
"error": format!("Peer returned an error ({status}). {refund}")
"error": format!("{} {refund}", seller_error_message(status, &body))
}));
}
@@ -693,10 +692,17 @@ impl RpcHandler {
.filter(|s| !s.is_empty())
.unwrap_or_else(|| "application/octet-stream".to_string());
let bytes = response
.bytes()
.await
.context("Failed to read response body")?;
let bytes = match response.bytes().await {
Ok(bytes) => bytes,
Err(error) => {
tracing::warn!("paid download: response body failed: {error}");
let refund =
reclaim_spent_ecash(&self.config.data_dir, &token_str, used_backend).await;
return Ok(serde_json::json!({
"error": format!("The file transfer was interrupted after payment was sent. {refund}")
}));
}
};
// Persist the purchase so it "stays unlocked" for this buyer: cache the
// bytes + metadata keyed by (onion, content_id). The gallery then renders
@@ -728,28 +734,8 @@ impl RpcHandler {
// The durable purchased-content cache above is primary. A Files copy
// remains optional: a stopped FileBrowser must not undo a paid download.
let filed = async {
let auth = self.handle_filebrowser_token().await?;
let token = auth
.get("token")
.and_then(|v| v.as_str())
.context("FileBrowser omitted its authentication token")?;
let client = reqwest::Client::builder()
.no_proxy()
.redirect(reqwest::redirect::Policy::none())
.timeout(std::time::Duration::from_secs(30))
.build()?;
file_purchase_in_files(
&client,
"http://127.0.0.1:8083",
token,
&filename,
&mime_type,
&bytes,
)
.await
}
.await;
let filed =
file_purchase_in_files(&self.config.data_dir, &filename, &mime_type, &bytes).await;
match filed {
Ok(path) => tracing::info!("paid download: filed into Files/{path}"),
Err(error) => tracing::warn!(
@@ -902,10 +888,29 @@ impl RpcHandler {
if !is_valid_v3_onion(onion) {
return Err(anyhow::anyhow!("Invalid v3 onion address"));
}
if payment_hash.is_empty() || !payment_hash.chars().all(|c| c.is_ascii_hexdigit()) {
if payment_hash.len() != 64 || !payment_hash.chars().all(|c| c.is_ascii_hexdigit()) {
return Err(anyhow::anyhow!("Invalid payment_hash"));
}
let cache_only = params
.get("cache_only")
.and_then(|v| v.as_bool())
.unwrap_or(false);
if let Some((mime, bytes)) =
crate::content_owned::read_owned(&self.config.data_dir, onion, content_id).await
{
return Ok(invoice_download_response(&bytes, &mime, cache_only));
}
// Older sellers only mark settlement during status polling. Always
// perform that handshake before requesting bytes; retries never pay.
// The download gate remains authoritative: a file may have become
// free, and newer sellers verify directly if status polling fails.
let _ = self
.handle_content_invoice_status(Some(serde_json::json!({
"onion": onion, "content_id": content_id, "payment_hash": payment_hash,
})))
.await;
let (data, _) = self.state_manager.get_snapshot().await;
let local_did = crate::identity::did_key_from_pubkey_hex(&data.server_info.pubkey)?;
let fips_npub = crate::federation::fips_npub_for_onion(&self.config.data_dir, onion).await;
@@ -944,7 +949,7 @@ impl RpcHandler {
if response.status() == reqwest::StatusCode::PAYMENT_REQUIRED {
return Ok(serde_json::json!({
"error": "Seller has not registered this payment yet — wait for settlement and retry."
"error": "The seller has not confirmed access yet. Retry the download without paying again."
}));
}
if !response.status().is_success() {
@@ -953,16 +958,45 @@ impl RpcHandler {
}));
}
let mime = response
.headers()
.get(reqwest::header::CONTENT_TYPE)
.and_then(|v| v.to_str().ok())
.unwrap_or("application/octet-stream")
.split(';')
.next()
.unwrap_or("application/octet-stream")
.to_string();
let bytes = response
.bytes()
.await
.context("Failed to read response body")?;
use base64::Engine;
let encoded = base64::engine::general_purpose::STANDARD.encode(&bytes);
Ok(serde_json::json!({
"data": encoded,
"size": bytes.len(),
}))
.context("Paid file transfer interrupted; retry the download without paying again")?;
let filename = params
.get("filename")
.and_then(|v| v.as_str())
.unwrap_or(content_id);
crate::content_owned::record_purchase(
&self.config.data_dir,
onion,
content_id,
filename,
&mime,
&bytes,
params
.get("price_sats")
.and_then(|v| v.as_u64())
.unwrap_or(0),
"lightning",
&chrono::Utc::now().to_rfc3339(),
)
.await
.context("Paid file could not be saved; retry the download without paying again")?;
if let Err(error) =
file_purchase_in_files(&self.config.data_dir, filename, &mime, &bytes).await
{
tracing::warn!("Lightning purchase cached; optional Files copy failed: {error:#}");
}
Ok(invoice_download_response(&bytes, &mime, cache_only))
}
/// Buyer side (#46): ask the seller for a fresh on-chain address to pay.
@@ -1437,3 +1471,19 @@ impl RpcHandler {
#[cfg(test)]
#[path = "content_tests.rs"]
mod tests;
#[cfg(test)]
mod invoice_delivery_response_tests {
use super::*;
#[test]
fn cached_delivery_avoids_base64_but_keeps_old_clients_compatible() {
let cached = invoice_download_response(b"paid bytes", "video/mp4", true);
assert_eq!(cached["owned"], true);
assert_eq!(cached["size_bytes"], 10);
assert!(cached.get("data").is_none());
assert!(cached.get("data_base64").is_none());
let legacy = invoice_download_response(b"paid bytes", "video/mp4", false);
assert_eq!(legacy["data"], "cGFpZCBieXRlcw==");
assert_eq!(legacy["data"], legacy["data_base64"]);
}
}
+98 -124
View File
@@ -1,69 +1,4 @@
use super::*;
use hyper::{
service::{make_service_fn, service_fn},
Body, Response, Server,
};
use std::{
collections::VecDeque,
convert::Infallible,
sync::{Arc, Mutex},
};
struct FilesApi {
url: String,
seen: Arc<Mutex<Vec<(String, String, Vec<u8>)>>>,
task: tokio::task::JoinHandle<()>,
}
impl Drop for FilesApi {
fn drop(&mut self) {
self.task.abort();
}
}
fn files_api(statuses: Vec<u16>) -> FilesApi {
let statuses = Arc::new(Mutex::new(VecDeque::from(statuses)));
let seen = Arc::new(Mutex::new(Vec::new()));
let history = seen.clone();
let server = Server::bind(&([127, 0, 0, 1], 0).into());
let address = server.local_addr();
let service = make_service_fn(move |_| {
let statuses = statuses.clone();
let seen = history.clone();
async move {
Ok::<_, Infallible>(service_fn(move |request: hyper::Request<Body>| {
let statuses = statuses.clone();
let seen = seen.clone();
async move {
assert_eq!(request.headers().get("X-Auth").unwrap(), "test-session");
let method = request.method().to_string();
let uri = request.uri().to_string();
let body = hyper::body::to_bytes(request.into_body())
.await
.unwrap()
.to_vec();
seen.lock().unwrap().push((method, uri, body));
let status = statuses
.lock()
.unwrap()
.pop_front()
.expect("unexpected extra Files request");
Ok::<_, Infallible>(
Response::builder()
.status(status)
.body(Body::empty())
.unwrap(),
)
}
}))
}
});
FilesApi {
url: format!("http://{address}"),
seen,
task: tokio::spawn(async move {
server.serve(service).await.unwrap();
}),
}
}
#[test]
fn first_and_cached_paid_downloads_have_the_same_client_payload_contract() {
@@ -85,80 +20,119 @@ fn first_and_cached_paid_downloads_have_the_same_client_payload_contract() {
}
#[tokio::test]
async fn files_copy_uses_authenticated_api_and_preserves_existing_names() {
let api = files_api(vec![200, 409, 200]);
let client = reqwest::Client::new();
let path = file_purchase_in_files(
&client,
&api.url,
"test-session",
"../my #file?.txt",
"text/plain",
b"paid bytes",
)
async fn files_copy_routes_media_and_sanitizes_the_filename() {
let dir = tempfile::tempdir().unwrap();
tokio::fs::create_dir(dir.path().join("filebrowser"))
.await
.unwrap();
assert_eq!(path, "Documents/my #file? (2).txt");
let seen = api.seen.lock().unwrap();
assert_eq!(seen[0].0, "GET");
assert_eq!(seen[0].1, "/api/resources/Documents/");
assert_eq!(seen.len(), 3);
for (_, uri, body) in &seen[1..] {
assert!(uri.contains("override=false"));
assert!(uri.contains("%23file%3F"));
assert!(!uri.contains("../"));
assert_eq!(body, b"paid bytes");
}
}
#[tokio::test]
async fn files_copy_creates_missing_media_folder() {
for (mime, folder) in [
("image/png", "Photos"),
("video/mp4", "Photos"),
("audio/ogg", "Music"),
("audio/mpeg", "Music"),
("text/plain", "Documents"),
] {
let api = files_api(vec![404, 200, 200]);
let path = file_purchase_in_files(
&reqwest::Client::new(),
&api.url,
"test-session",
"file",
mime,
b"bytes",
)
let relative = file_purchase_in_files(dir.path(), "../name #?.bin", mime, b"paid")
.await
.unwrap();
assert_eq!(path, format!("{folder}/file"));
let seen = api.seen.lock().unwrap();
assert_eq!(seen[1].0, "POST");
assert!(seen[1].1.ends_with('/'));
assert!(seen[1].2.is_empty());
assert_eq!(seen[2].2, b"bytes");
assert!(relative.starts_with(&format!("{folder}/name #?")));
assert_eq!(
tokio::fs::read(dir.path().join("filebrowser").join(relative))
.await
.unwrap(),
b"paid"
);
}
}
#[tokio::test]
async fn files_copy_fails_without_overwriting_or_claiming_success_on_errors() {
for statuses in [
vec![401],
vec![503],
vec![404, 500],
vec![200, 507],
vec![200, 403],
] {
let expected = statuses.len();
let api = files_api(statuses);
assert!(file_purchase_in_files(
&reqwest::Client::new(),
&api.url,
"test-session",
async fn unavailable_files_storage_is_reported_without_creating_a_fake_installation() {
let dir = tempfile::tempdir().unwrap();
assert!(
file_purchase_in_files(dir.path(), "name", "text/plain", b"bytes")
.await
.is_err()
);
assert!(!dir.path().join("filebrowser").exists());
}
#[test]
fn seller_errors_are_bounded_printable_and_identified_as_peer_text() {
let status = reqwest::StatusCode::SERVICE_UNAVAILABLE;
let message = seller_error_message(status, r#"{"error":"Cannot read file\n\u0000"}"#);
assert!(message.starts_with("Seller response (503"));
assert!(message.ends_with("Cannot read file"));
assert!(!message.contains('\n') && !message.contains('\0'));
let body = serde_json::json!({"error": "é".repeat(1000)}).to_string();
assert!(seller_error_message(status, &body).chars().count() < 300);
for body in ["not JSON", r#"{"error": 7}"#, r#"{"error":" "}"#] {
assert_eq!(
seller_error_message(status, body),
"Peer returned an error (503 Service Unavailable)."
);
}
}
#[tokio::test]
async fn known_purchase_never_becomes_a_new_spend_when_cache_or_index_is_unavailable() {
let dir = tempfile::tempdir().unwrap();
assert!(
existing_paid_content(dir.path(), "seller.onion", "id", None)
.await
.unwrap()
.is_none()
);
crate::content_owned::record_purchase(
dir.path(),
"seller.onion",
"id",
"file.txt",
"text/plain",
b"bytes"
b"paid",
1,
"cashu",
"now",
)
.await
.is_err());
assert_eq!(api.seen.lock().unwrap().len(), expected);
.unwrap();
for (id, filename) in [("id", None), ("duplicate-id", Some("/file.txt"))] {
let cached = existing_paid_content(dir.path(), "seller.onion", id, filename)
.await
.unwrap()
.unwrap();
assert_eq!(cached["paid_sats"], 0);
assert_eq!(cached["already_owned"], true);
assert_eq!(cached["data"], "cGFpZA==");
}
assert!(
existing_paid_content(dir.path(), "different.onion", "id", None)
.await
.unwrap()
.is_none()
);
tokio::fs::remove_file(dir.path().join("purchased-content/seller.onion/id"))
.await
.unwrap();
assert!(
existing_paid_content(dir.path(), "seller.onion", "id", None)
.await
.unwrap_err()
.to_string()
.contains("No new payment")
);
tokio::fs::write(dir.path().join("purchased-content/owned.json"), b"damaged")
.await
.unwrap();
assert!(
existing_paid_content(dir.path(), "seller.onion", "other-id", None)
.await
.unwrap_err()
.to_string()
.contains("no new payment")
);
assert_eq!(
tokio::fs::read(dir.path().join("purchased-content/owned.json"))
.await
.unwrap(),
b"damaged"
);
}
@@ -132,6 +132,9 @@ impl RpcHandler {
"lnd.newaddress" => self.handle_lnd_newaddress().await,
"lnd.sendcoins" => self.handle_lnd_sendcoins(params).await,
"lnd.estimatefee" => self.handle_lnd_estimatefee(params).await,
"lnd.bump-quote" => self.handle_lnd_bump_quote(params).await,
"lnd.bump-submit" => self.handle_lnd_bump_submit(params).await,
"lnd.bump-status" => self.handle_lnd_bump_status(params).await,
"lnd.createinvoice" => self.handle_lnd_createinvoice(params).await,
"lnd.invoicestatus" => self.handle_lnd_invoicestatus(params).await,
"lnd.payinvoice" => self.handle_lnd_payinvoice(params).await,
@@ -1,6 +1,8 @@
use super::*;
use crate::api::rpc::RpcHandler;
use crate::identity_manager::{IdentityManager, IdentityProfile, IdentityPurpose};
use crate::identity_manager::{
is_node_identity, IdentityManager, IdentityProfile, IdentityPurpose,
};
use crate::network::did_dht;
use anyhow::{Context, Result};
use nostr_sdk::ToBech32;
@@ -38,7 +40,7 @@ impl RpcHandler {
.into_iter()
.map(|id| {
let is_default = default_id.as_deref() == Some(&id.id);
let is_node = !node_pubkey_hex.is_empty() && id.pubkey_hex == node_pubkey_hex;
let is_node = is_node_identity(&id, &node_pubkey_hex);
let (nostr_pubkey, nostr_npub) = if is_node {
(
node_nostr_hex.clone().or(id.nostr_pubkey),
+108 -26
View File
@@ -272,28 +272,8 @@ impl RpcHandler {
.and_then(|v| v.as_bool())
.unwrap_or(false);
// Fee control: either a confirmation target or an explicit fee rate
let target_conf = params.get("target_conf").and_then(|v| v.as_i64());
let sat_per_vbyte = params.get("sat_per_vbyte").and_then(|v| v.as_i64());
if target_conf.is_some() && sat_per_vbyte.is_some() {
return Err(anyhow::anyhow!(
"Invalid fee parameters: specify either target_conf or sat_per_vbyte, not both"
));
}
if let Some(tc) = target_conf {
if !(1..=1008).contains(&tc) {
return Err(anyhow::anyhow!(
"Invalid target_conf: must be between 1 and 1008 blocks"
));
}
}
if let Some(rate) = sat_per_vbyte {
if !(1..=5000).contains(&rate) {
return Err(anyhow::anyhow!(
"Invalid sat_per_vbyte: must be between 1 and 5000"
));
}
}
// Omitted fees target the next block; explicit slower/custom choices win.
let (target_conf, sat_per_vbyte) = super::fee_policy::fee_options(&params)?;
info!(
peer = pubkey,
@@ -473,6 +453,7 @@ impl RpcHandler {
));
}
let fee_query = close_channel_fee_query(&params)?;
let force = params
.get("force")
.and_then(|v| v.as_bool())
@@ -498,13 +479,11 @@ impl RpcHandler {
.build()
.context("Failed to create streaming HTTP client")?;
let url = format!(
"{LND_REST_BASE_URL}/v1/channels/{}/{}?force={}",
parts[0], parts[1], force
);
let url = format!("{LND_REST_BASE_URL}/v1/channels/{}/{}", parts[0], parts[1]);
let mut resp = client
.delete(&url)
.query(&fee_query)
.header("Grpc-Metadata-macaroon", &macaroon_hex)
.send()
.await
@@ -572,3 +551,106 @@ impl RpcHandler {
}
}
}
/// LND's CloseChannel REST endpoint takes fee selection as query parameters.
/// With neither parameter LND uses a lax target; keep legacy clients on our
/// explicit next-block target rather than silently accepting that default.
fn close_channel_fee_query(params: &serde_json::Value) -> Result<Vec<(&'static str, String)>> {
let force = match params.get("force") {
None | Some(serde_json::Value::Null) => false,
Some(value) => value
.as_bool()
.ok_or_else(|| anyhow::anyhow!("force must be a boolean"))?,
};
let integer = |key: &str, max: u64| -> Result<Option<u64>> {
match params.get(key) {
None | Some(serde_json::Value::Null) => Ok(None),
Some(value) => {
let n = value
.as_u64()
.ok_or_else(|| anyhow::anyhow!("{key} must be a positive whole number"))?;
anyhow::ensure!((1..=max).contains(&n), "{key} must be between 1 and {max}");
Ok(Some(n))
}
}
};
let target = integer("target_conf", 1008)?;
let rate = integer("sat_per_vbyte", 5000)?;
anyhow::ensure!(
target.is_none() || rate.is_none(),
"Specify either target_conf or sat_per_vbyte, not both"
);
anyhow::ensure!(
!force || (target.is_none() && rate.is_none()),
"Closing fee selection requires a cooperative close"
);
let mut query = vec![("force", force.to_string())];
if !force {
if let Some(rate) = rate {
query.push(("sat_per_vbyte", rate.to_string()));
} else {
query.push((
"target_conf",
target
.unwrap_or(super::fee_policy::DEFAULT_TARGET as u64)
.to_string(),
));
}
}
Ok(query)
}
#[cfg(test)]
mod close_fee_tests {
use super::*;
#[test]
fn close_fee_query_forwards_presets_custom_and_legacy_default() {
for target in [1, 3, 6, 1008] {
assert_eq!(
close_channel_fee_query(&serde_json::json!({"target_conf":target})).unwrap(),
vec![
("force", "false".into()),
("target_conf", target.to_string())
]
);
}
for rate in [1, 25, 5000] {
let query =
close_channel_fee_query(&serde_json::json!({"sat_per_vbyte":rate})).unwrap();
let request = reqwest::Client::new()
.delete("http://localhost/v1/channels/test/0")
.query(&query)
.build()
.unwrap();
assert_eq!(request.method(), reqwest::Method::DELETE);
assert_eq!(
request.url().query(),
Some(format!("force=false&sat_per_vbyte={rate}").as_str())
);
}
assert_eq!(
close_channel_fee_query(&serde_json::json!({})).unwrap(),
vec![("force", "false".into()), ("target_conf", "1".into())]
);
assert_eq!(
close_channel_fee_query(&serde_json::json!({"force":true})).unwrap(),
vec![("force", "true".into())]
);
}
#[test]
fn malformed_or_conflicting_close_fees_fail_before_wallet_access() {
for params in [
serde_json::json!({"target_conf":1,"sat_per_vbyte":2}),
serde_json::json!({"force":true,"target_conf":1}),
serde_json::json!({"force":"false"}),
serde_json::json!({"target_conf":0}),
serde_json::json!({"target_conf":1009}),
serde_json::json!({"sat_per_vbyte":5001}),
serde_json::json!({"sat_per_vbyte":-1}),
serde_json::json!({"sat_per_vbyte":1.5}),
serde_json::json!({"sat_per_vbyte":"25"}),
] {
assert!(close_channel_fee_query(&params).is_err(), "{params}");
}
}
}
@@ -0,0 +1,835 @@
//! WalletKit BumpFee is CPFP for new wallet outputs, RBF only for sweeper inputs.
//! Never feed an ordinary payment input to it and call that a replacement.
use super::LND_REST_BASE_URL;
use crate::api::rpc::RpcHandler;
use anyhow::{bail, ensure, Context, Result};
use serde::{Deserialize, Serialize};
use serde_json::{json, Value};
use std::{collections::HashMap, path::Path, sync::LazyLock};
use tokio::{io::AsyncWriteExt, sync::Mutex};
static QUOTES: LazyLock<Mutex<HashMap<String, Quote>>> = LazyLock::new(Default::default);
// Serialize check/register/persist across dashboard clients. The create_new receipt
// additionally survives process restarts and prevents retries of ambiguous results.
static SUBMIT: Mutex<()> = Mutex::const_new(());
const QUOTE_SECONDS: u64 = 60;
#[derive(Clone, Debug, Serialize, Deserialize, PartialEq)]
struct Plan {
txid: String,
method: String,
input_txid: String,
input_index: u32,
parent_txid: String,
recipient_sats: u64,
rate_sat_vb: u64,
current_fee_sats: u64,
additional_fee_sats: u64,
total_fee_sats: u64,
budget_sats: u64,
input_sats: u64,
parent_vsize: u64,
sweep_vsize_bound: u64,
tip: String,
}
#[derive(Clone, Serialize, Deserialize)]
struct Quote {
quote_id: String,
expires_at: u64,
custom_rate: Option<u64>,
#[serde(flatten)]
plan: Plan,
}
#[derive(Serialize, Deserialize)]
struct Operation {
quote: Quote,
status: String,
message: String,
}
fn now() -> u64 {
std::time::SystemTime::now()
.duration_since(std::time::UNIX_EPOCH)
.unwrap_or_default()
.as_secs()
}
fn number(v: &Value) -> Result<u64> {
v.as_u64()
.or_else(|| v.as_str().and_then(|s| s.parse().ok()))
.context("Missing or invalid wallet amount")
}
fn txid_param(p: &Value) -> Result<String> {
let s = p["txid"].as_str().context("Missing transaction ID")?;
ensure!(
s.len() == 64 && s.bytes().all(|c| c.is_ascii_hexdigit()),
"Invalid transaction ID"
);
Ok(s.to_ascii_lowercase())
}
fn btc_sats(v: &Value) -> Result<u64> {
let n = v.as_f64().context("Missing Bitcoin fee")? * 100_000_000.0;
ensure!(
n.is_finite() && n >= 0.0 && n <= 2_100_000_000_000_000.0,
"Invalid Bitcoin fee"
);
Ok(n.round() as u64)
}
fn outpoint_matches(v: &Value, txid: &str, index: u32) -> bool {
v["txid_str"].as_str() == Some(txid) && v["output_index"].as_u64() == Some(index as u64)
}
fn array<'a>(v: &'a Value, key: &str) -> Result<&'a Vec<Value>> {
v[key]
.as_array()
.with_context(|| format!("Missing wallet field: {key}"))
}
fn sweep_size(output: &Value) -> Result<u64> {
// One native input, one wallet taproot output, including signature rounding.
match output["output_type"].as_str() {
Some("SCRIPT_TYPE_WITNESS_V1_TAPROOT") => Ok(112),
Some("SCRIPT_TYPE_WITNESS_V0_PUBKEY_HASH") => Ok(123),
_ => bail!("This output type is not supported for fee bumping yet"),
}
}
fn fee_budget(
rate: u64,
parent_size: u64,
parent_fee: u64,
size: u64,
old_fee: u64,
relay: u64,
input: u64,
) -> Result<u64> {
ensure!(
(1..=5000).contains(&rate),
"Fee rate must be a whole number from 1 to 5000 sat/vB"
);
ensure!(
parent_size <= 100_000 && size <= 100_000 && relay <= 5000,
"Unsupported package size or relay fee"
);
let required = rate * (parent_size + size);
let mut budget = required.saturating_sub(parent_fee).max(relay * size);
if old_fee > 0 {
budget = budget.max(old_fee + relay * size + 1);
}
ensure!(budget > old_fee, "Choose a higher fee rate");
// Conservative dust buffer; never attach unrelated wallet inputs to fund fees.
ensure!(
budget.checked_add(1000).is_some_and(|v| v <= input),
"Not enough wallet change for this fee; choose a lower rate"
);
Ok(budget)
}
async fn lnd(
client: &reqwest::Client,
macaroon: &str,
path: &str,
body: Option<Value>,
) -> Result<Value> {
let url = format!("{LND_REST_BASE_URL}{path}");
let req = match body {
Some(v) => client.post(url).json(&v),
None => client.get(url),
};
let response = req
.header("Grpc-Metadata-macaroon", macaroon)
.send()
.await?;
let status = response.status();
let value: Value = response.json().await.context("Invalid LND response")?;
ensure!(
status.is_success() && value.get("code").is_none(),
"{}",
value["message"].as_str().unwrap_or("LND request failed")
);
Ok(value)
}
fn validate_quote(quote: &Quote, fresh: &Plan, timestamp: u64) -> Result<()> {
ensure!(
quote.expires_at > timestamp && quote.plan == *fresh,
"Transaction or fees changed; review a fresh quote"
);
Ok(())
}
fn bump_body(plan: &Plan) -> Value {
json!({"outpoint":{"txid_str":plan.input_txid,"output_index":plan.input_index},
"sat_per_vbyte":plan.rate_sat_vb.to_string(), "budget":plan.budget_sats.to_string(),
"deadline_delta":1, "immediate":true})
}
async fn reserve(path: &Path, op: &Operation) -> Result<()> {
let parent = path.parent().context("Invalid operation path")?;
tokio::fs::create_dir_all(parent).await?;
let mut f = tokio::fs::OpenOptions::new()
.write(true)
.create_new(true)
.mode(0o600)
.open(path)
.await
.context("A bump already exists for this transaction; check its status")?;
f.write_all(&serde_json::to_vec(op)?).await?;
f.sync_all().await?;
// Sync directory entry too: a crash must not make a submitted operation vanish.
tokio::fs::File::open(parent).await?.sync_all().await?;
Ok(())
}
// Only a recorded Archy CPFP with one owned input and no external outputs may
// be folded into a payment. Labels and a fee-sized delta alone are not evidence.
fn fee_child_matches(tx: &Value, plan: &Plan) -> bool {
let input = format!("{}:{}", plan.input_txid, plan.input_index);
let amount = tx["amount"]
.as_i64()
.or_else(|| tx["amount"].as_str()?.parse().ok());
let fee = number(&tx["total_fees"])
.ok()
.and_then(|n| i64::try_from(n).ok());
tx["tx_hash"]
.as_str()
.is_some_and(|id| id.len() == 64 && id.bytes().all(|c| c.is_ascii_hexdigit()))
&& amount
.zip(fee)
.is_some_and(|(amount, fee)| fee > 0 && amount == -fee)
&& tx["previous_outpoints"].as_array().is_some_and(|inputs| {
inputs.len() == 1
&& inputs[0]["outpoint"] == input
&& inputs[0]["is_our_output"] == true
})
&& tx["output_details"].as_array().is_some_and(|outputs| {
!outputs.is_empty() && outputs.iter().all(|o| o["is_our_address"] == true)
})
}
impl RpcHandler {
pub(super) async fn group_fee_bump_history(
&self,
raw: &[Value],
normalized: &mut Vec<Value>,
client: &reqwest::Client,
) {
let mut hidden = std::collections::HashSet::new();
for parent in normalized.iter_mut() {
if parent["direction"] != "outgoing" {
continue;
}
let Some(id) = parent["tx_hash"].as_str().map(str::to_owned) else {
continue;
};
if id.len() != 64 || !id.bytes().all(|c| c.is_ascii_hexdigit()) {
continue;
}
let path = self
.config
.data_dir
.join("wallet/fee-bumps")
.join(format!("{id}.json"));
let Ok(bytes) = tokio::fs::read(path).await else {
continue;
};
let Ok(op) = serde_json::from_slice::<Operation>(&bytes) else {
continue;
};
let plan = &op.quote.plan;
if plan.method != "cpfp"
|| plan.txid != id
|| plan.parent_txid != id
|| plan.input_txid != id
{
continue;
}
let candidates: Vec<_> = raw
.iter()
.filter(|tx| fee_child_matches(tx, plan))
.collect();
let mut active = Vec::new();
for child in &candidates {
let child_id = child["tx_hash"].as_str().unwrap();
if child["num_confirmations"].as_i64().unwrap_or(0) > 0
|| self
.bitcoin_rpc_call::<Value>(client, "getmempoolentry", &[json!(child_id)])
.await
.is_ok()
{
active.push(*child);
}
}
// Ambiguous or unavailable chain state must not hide wallet history.
if active.len() != 1 {
continue;
}
let current = active[0];
parent["bump_fee_sats"] = json!(number(&current["total_fees"]).unwrap());
parent["fee_bump_txid"] = current["tx_hash"].clone();
parent["fee_bump_confirmations"] = current["num_confirmations"].clone();
parent["fee_bump_history"] = json!(candidates.iter().map(|child| {
let child_id = child["tx_hash"].as_str().unwrap();
hidden.insert(child_id.to_owned());
json!({"tx_hash":child_id,"fee_sats":number(&child["total_fees"]).unwrap(),
"status":if child["tx_hash"] != current["tx_hash"] { "replaced" }
else if child["num_confirmations"].as_i64().unwrap_or(0) > 0 { "confirmed" } else { "mempool" }})
}).collect::<Vec<_>>());
}
normalized.retain(|tx| !tx["tx_hash"].as_str().is_some_and(|id| hidden.contains(id)));
}
async fn bump_plan(&self, txid: &str, custom_rate: Option<u64>) -> Result<Plan> {
let (client, macaroon) = self.lnd_client().await?;
let info = lnd(&client, &macaroon, "/v1/getinfo", None).await?;
ensure!(
info["synced_to_chain"] == true,
"Wait for the wallet to finish syncing"
);
let version = info["version"]
.as_str()
.context("LND version is unavailable")?;
let mut parts = version.trim_start_matches('v').split('.');
let major: u32 = parts
.next()
.unwrap_or("")
.parse()
.context("Invalid LND version")?;
let minor: u32 = parts
.next()
.unwrap_or("")
.parse()
.context("Invalid LND version")?;
ensure!(
major > 0 || minor >= 21,
"This fee-bump interface requires LND 0.21 or newer"
);
let history = lnd(&client, &macaroon, "/v1/transactions", None).await?;
let txs = array(&history, "transactions")?;
let tx = txs
.iter()
.find(|t| t["tx_hash"] == txid)
.context("Transaction is not in this wallet")?;
ensure!(
tx["num_confirmations"].as_i64() == Some(0),
"This transaction is no longer pending"
);
let entry: Value = self
.bitcoin_rpc_call(&client, "getmempoolentry", &[json!(txid)])
.await
.context("Transaction is not currently in the node's mempool")?;
ensure!(
number(&entry["descendantcount"])? == 1,
"This transaction already has a child; open the child's Bump options instead"
);
let pending = lnd(&client, &macaroon, "/v2/wallet/sweeps/pending", None).await?;
let sweeps = array(&pending, "pending_sweeps")?;
let published = lnd(
&client,
&macaroon,
"/v2/wallet/sweeps?verbose=false&start_height=-1",
None,
)
.await?;
let is_sweep = published["transaction_ids"]["transaction_ids"]
.as_array()
.is_some_and(|ids| ids.iter().any(|id| id == txid));
let outputs = array(tx, "output_details")?;
ensure!(
tx["amount"]
.as_str()
.and_then(|v| v.parse::<i64>().ok())
.or_else(|| tx["amount"].as_i64())
.is_some_and(|v| v < 0),
"Bump is available for outgoing payments and wallet fee sweeps"
);
let (
method,
input_txid,
input_index,
input_sats,
parent_txid,
parent_size,
parent_fee,
old_fee,
size,
recipient_sats,
) = if is_sweep {
// Only a simple wallet CPFP sweep is replaceable here. Anchor/HTLC,
// batched sweeps and arbitrary signed payments need different previews.
let raw: Value = self
.bitcoin_rpc_call(&client, "getrawtransaction", &[json!(txid), json!(true)])
.await?;
let inputs = array(&raw, "vin")?;
ensure!(
inputs.len() == 1 && outputs.len() == 1 && outputs[0]["is_our_address"] == true,
"RBF for batched or channel sweeps is not supported here yet"
);
let input_txid = inputs[0]["txid"]
.as_str()
.context("Missing sweep input")?
.to_string();
let index = u32::try_from(number(&inputs[0]["vout"])?)?;
ensure!(
sweeps.len() == 1 && outpoint_matches(&sweeps[0]["outpoint"], &input_txid, index),
"RBF is unavailable while other wallet sweeps are active"
);
let parent = txs
.iter()
.find(|t| t["tx_hash"] == input_txid)
.context("Sweep parent is unavailable")?;
let parent_output = array(parent, "output_details")?
.iter()
.find(|o| {
number(&o["output_index"]).ok() == Some(index as u64)
&& o["is_our_address"] == true
})
.context("RBF requires a wallet-owned change input")?;
let parent_entry: Value = self
.bitcoin_rpc_call(&client, "getmempoolentry", &[json!(input_txid)])
.await
.context("Only unconfirmed CPFP sweep replacements are supported here")?;
ensure!(
number(&parent_entry["ancestorcount"])? == 1
&& number(&parent_entry["descendantcount"])? == 2,
"Complex sweep package cannot be quoted safely"
);
let recipients = recipient_amount(parent)?;
(
"rbf",
input_txid.clone(),
index,
number(&parent_output["amount"])?,
input_txid,
number(&parent_entry["vsize"])?,
btc_sats(&parent_entry["fees"]["base"])?,
btc_sats(&entry["fees"]["base"])?,
sweep_size(parent_output)?.max(number(&entry["vsize"])?),
recipients,
)
} else {
ensure!(
sweeps.is_empty(),
"Another wallet sweep is active; wait for it before creating a CPFP bump"
);
ensure!(
number(&entry["ancestorcount"])? == 1,
"Fee bumping a chain of unconfirmed payments is not supported yet"
);
let unspent = lnd(
&client,
&macaroon,
"/v2/wallet/utxos",
Some(json!({"unconfirmed_only":true})),
)
.await?;
let utxos = array(&unspent, "utxos")?;
let leases = lnd(
&client,
&macaroon,
"/v2/wallet/utxos/leases",
Some(json!({})),
)
.await?;
let locked = array(&leases, "locked_utxos")?;
let output = outputs
.iter()
.filter(|o| o["is_our_address"] == true && sweep_size(o).is_ok())
.filter(|o| {
number(&o["output_index"]).ok().is_some_and(|i| {
utxos
.iter()
.any(|u| outpoint_matches(&u["outpoint"], txid, i as u32))
&& !locked
.iter()
.any(|u| outpoint_matches(&u["outpoint"], txid, i as u32))
})
})
.max_by_key(|o| number(&o["amount"]).unwrap_or(0))
.context(
"RBF is unavailable for this payment. CPFP needs spendable wallet-owned change",
)?;
let index = u32::try_from(number(&output["output_index"])?)?;
let available: Value = self
.bitcoin_rpc_call(
&client,
"gettxout",
&[json!(txid), json!(index), json!(true)],
)
.await?;
ensure!(
available.is_object()
&& number(&available["confirmations"])? == 0
&& btc_sats(&available["value"])? == number(&output["amount"])?,
"Change is no longer available"
);
(
"cpfp",
txid.to_string(),
index,
number(&output["amount"])?,
txid.to_string(),
number(&entry["vsize"])?,
btc_sats(&entry["fees"]["base"])?,
0,
sweep_size(output)?,
recipient_amount(tx)?,
)
};
let mempool: Value = self
.bitcoin_rpc_call(&client, "getmempoolinfo", &[])
.await?;
let relay = btc_sats(&mempool["incrementalrelayfee"])?
.div_ceil(1000)
.max(1);
let floor = btc_sats(&mempool["mempoolminfee"])?
.max(btc_sats(&mempool["minrelaytxfee"])?)
.div_ceil(1000)
.max(1);
let rate = match custom_rate {
Some(rate) => {
ensure!(
rate >= floor,
"Custom rate is below the current mempool minimum"
);
rate
}
None => {
let estimate = lnd(&client, &macaroon, "/v2/wallet/estimatefee/1", None).await?;
number(&estimate["sat_per_kw"])?.div_ceil(250).max(floor)
}
};
let budget = fee_budget(
rate,
parent_size,
parent_fee,
size,
old_fee,
relay.max(floor),
input_sats,
)?;
let tip: String = self
.bitcoin_rpc_call(&client, "getbestblockhash", &[])
.await?;
Ok(Plan {
txid: txid.to_string(),
method: method.into(),
input_txid,
input_index,
parent_txid,
recipient_sats,
rate_sat_vb: rate,
current_fee_sats: parent_fee + old_fee,
additional_fee_sats: budget - old_fee,
total_fee_sats: parent_fee + budget,
budget_sats: budget,
input_sats,
parent_vsize: parent_size,
sweep_vsize_bound: size,
tip,
})
}
pub(in crate::api::rpc) async fn handle_lnd_bump_quote(
&self,
params: Option<Value>,
) -> Result<Value> {
let p = params.unwrap_or_default();
let txid = txid_param(&p)?;
let path = self
.config
.data_dir
.join("wallet/fee-bumps")
.join(format!("{txid}.json"));
ensure!(
!path.try_exists()?,
"A bump was already submitted for this transaction. Check its status"
);
let custom = p
.get("sat_per_vbyte")
.map(|v| v.as_u64().context("Custom rate must be a whole number"))
.transpose()?;
if let Some(rate) = custom {
ensure!(
(1..=5000).contains(&rate),
"Custom rate must be 1–5000 sat/vB"
);
}
let plan = self.bump_plan(&txid, custom).await?;
let quote = Quote {
quote_id: uuid::Uuid::new_v4().to_string(),
expires_at: now() + QUOTE_SECONDS,
custom_rate: custom,
plan,
};
let mut quotes = QUOTES.lock().await;
quotes.retain(|_, q| q.expires_at > now());
ensure!(quotes.len() < 128, "Too many fee quotes; try again shortly");
quotes.insert(quote.quote_id.clone(), quote.clone());
Ok(serde_json::to_value(quote)?)
}
pub(in crate::api::rpc) async fn handle_lnd_bump_submit(
&self,
params: Option<Value>,
) -> Result<Value> {
let p = params.unwrap_or_default();
let txid = txid_param(&p)?;
let _guard = SUBMIT.lock().await;
let path = self
.config
.data_dir
.join("wallet/fee-bumps")
.join(format!("{txid}.json"));
if path.try_exists()? {
return self
.handle_lnd_bump_status(Some(json!({"txid":txid})))
.await;
}
let id = p["quote_id"]
.as_str()
.context("A reviewed fee quote is required")?;
let quote = QUOTES
.lock()
.await
.get(id)
.cloned()
.context("Quote expired; review the fee again")?;
ensure!(
quote.plan.txid == txid && quote.expires_at > now(),
"Quote expired; review the fee again"
);
let fresh = self.bump_plan(&txid, quote.custom_rate).await?;
validate_quote(&quote, &fresh, now())?;
let op = Operation {
quote: quote.clone(),
status: "unknown".into(),
message: "Submission recorded; checking the wallet. Do not submit another bump.".into(),
};
reserve(&path, &op).await?;
QUOTES.lock().await.remove(id);
let (client, macaroon) = self.lnd_client().await?;
// At a one-block deadline LND may spend ALL this explicitly previewed
// budget. It is always below input value, so no extra funding is requested.
let result = lnd(
&client,
&macaroon,
"/v2/wallet/bumpfee",
Some(bump_body(&fresh)),
)
.await;
// Keep the write-ahead record even for an RPC error: a lost response can
// conceal an accepted bump. Status reconciles from wallet/mempool evidence.
match result {
Ok(_) => Ok(
json!({"status":"registered", "message":"Bump registered with the wallet. Waiting for broadcast.", "quote":quote}),
),
Err(_) => Ok(
json!({"status":"unknown", "message":"The wallet response was not confirmed. Check status; do not submit again.", "quote":quote}),
),
}
}
pub(in crate::api::rpc) async fn handle_lnd_bump_status(
&self,
params: Option<Value>,
) -> Result<Value> {
let txid = txid_param(&params.unwrap_or_default())?;
let path = self
.config
.data_dir
.join("wallet/fee-bumps")
.join(format!("{txid}.json"));
let bytes = match tokio::fs::read(path).await {
Ok(b) => b,
Err(e) if e.kind() == std::io::ErrorKind::NotFound => {
return Ok(json!({"status":"none"}))
}
Err(e) => return Err(e.into()),
};
let op: Operation = serde_json::from_slice(&bytes)
.context("Bump receipt needs recovery; do not resubmit")?;
let (client, macaroon) = self.lnd_client().await?;
let history = lnd(&client, &macaroon, "/v1/transactions", None).await?;
let plan = &op.quote.plan;
let input = format!("{}:{}", plan.input_txid, plan.input_index);
let mut candidates: Vec<&Value> = array(&history, "transactions")?
.iter()
.filter(|t| {
t["tx_hash"] != txid
&& t["output_details"].as_array().is_some_and(|outputs| {
!outputs.is_empty() && outputs.iter().all(|o| o["is_our_address"] == true)
})
&& t["previous_outpoints"]
.as_array()
.is_some_and(|inputs| inputs.iter().any(|i| i["outpoint"] == input))
})
.collect();
candidates.sort_by_key(|t| std::cmp::Reverse(number(&t["time_stamp"]).unwrap_or(0)));
for t in candidates {
let id = t["tx_hash"]
.as_str()
.context("Missing bump transaction ID")?;
let confirmed = t["num_confirmations"].as_i64().unwrap_or(0) > 0;
let accepted = if confirmed {
false
} else {
self.bitcoin_rpc_call::<Value>(&client, "getmempoolentry", &[json!(id)])
.await
.is_ok()
};
if confirmed || accepted {
return Ok(json!({"status":if confirmed {"confirmed"} else {"mempool"},
"message":if confirmed {"Fee bump confirmed."} else {"Fee bump accepted in the node's mempool; awaiting confirmation."},
"bump_txid":id,"confirmations":t["num_confirmations"],"actual_sweep_fee_sats":number(&t["total_fees"])?,"quote":op.quote}));
}
}
let pending = lnd(&client, &macaroon, "/v2/wallet/sweeps/pending", None).await?;
let registered = array(&pending, "pending_sweeps")?.iter().any(|s| {
outpoint_matches(&s["outpoint"], &plan.input_txid, plan.input_index)
&& number(&s["budget"]).ok() == Some(plan.budget_sats)
&& number(&s["requested_sat_per_vbyte"]).ok() == Some(plan.rate_sat_vb)
});
Ok(
json!({"status":if registered {"registered"} else {"unknown"},
"message":if registered {"Bump registered; waiting for a verified broadcast."} else {"Submission outcome is unknown. Do not submit again; check wallet status."}, "quote":op.quote}),
)
}
}
fn recipient_amount(tx: &Value) -> Result<u64> {
array(tx, "output_details")?
.iter()
.filter(|o| o["is_our_address"] == false)
.try_fold(0u64, |sum, o| {
sum.checked_add(number(&o["amount"])?)
.context("Recipient amount overflow")
})
}
#[cfg(test)]
mod tests {
use super::*;
fn sample_plan() -> Plan {
serde_json::from_value(json!({"txid":"a","method":"cpfp","input_txid":"a","input_index":0,"parent_txid":"a","recipient_sats":161650,"rate_sat_vb":3,"current_fee_sats":144,"additional_fee_sats":618,"total_fee_sats":762,"budget_sats":618,"input_sats":21126,"parent_vsize":142,"sweep_vsize_bound":112,"tip":"tip"})).unwrap()
}
#[test]
fn history_requires_owned_simple_fee_only_child() {
let plan = sample_plan();
let tx = json!({"tx_hash":"b".repeat(64),"amount":"-200","total_fees":"200",
"previous_outpoints":[{"outpoint":"a:0","is_our_output":true}],
"output_details":[{"is_our_address":true}]});
assert!(fee_child_matches(&tx, &plan));
for bad in [
json!({"amount":"-201"}),
json!({"amount":"200"}),
json!({"total_fees":"0"}),
json!({"previous_outpoints":[{"outpoint":"a:1","is_our_output":true}]}),
json!({"previous_outpoints":[{"outpoint":"a:0","is_our_output":false}]}),
json!({"previous_outpoints":[{"outpoint":"a:0","is_our_output":true},{"outpoint":"c:0","is_our_output":true}]}),
json!({"output_details":[{"is_our_address":false}]}),
json!({"output_details":[]}),
json!({"tx_hash":"../../invalid"}),
] {
let mut changed = tx.clone();
for (key, value) in bad.as_object().unwrap() {
changed[key] = value.clone();
}
assert!(!fee_child_matches(&changed, &plan), "{bad}");
}
}
#[test]
fn stale_quotes_cannot_silently_change_approved_fee_or_transaction() {
let plan = sample_plan();
let q = Quote {
quote_id: "q".into(),
expires_at: 100,
custom_rate: None,
plan: plan.clone(),
};
assert!(validate_quote(&q, &plan, 99).is_ok());
assert!(validate_quote(&q, &plan, 100).is_err());
let mut changed = plan.clone();
changed.budget_sats += 1;
assert!(validate_quote(&q, &changed, 99).is_err());
changed = plan.clone();
changed.input_index += 1;
assert!(validate_quote(&q, &changed, 99).is_err());
changed = plan.clone();
changed.recipient_sats -= 1;
assert!(validate_quote(&q, &changed, 99).is_err());
changed = plan.clone();
changed.tip = "new block".into();
assert!(validate_quote(&q, &changed, 99).is_err());
}
#[test]
fn mutation_always_has_explicit_budget_and_does_not_send_a_second_payment() {
assert_eq!(
bump_body(&sample_plan()),
json!({"outpoint":{"txid_str":"a","output_index":0},"sat_per_vbyte":"3","budget":"618","deadline_delta":1,"immediate":true})
);
let mut rbf = sample_plan();
rbf.method = "rbf".into();
rbf.txid = "child".into();
// RBF uses the already-registered input, not the child's output.
assert_eq!(bump_body(&rbf)["outpoint"]["txid_str"], "a");
}
#[test]
fn outpoint_ownership_and_recipient_exclude_wallet_change() {
assert!(outpoint_matches(
&json!({"txid_str":"a","output_index":2}),
"a",
2
));
assert!(!outpoint_matches(
&json!({"txid_str":"b","output_index":2}),
"a",
2
));
assert!(!outpoint_matches(
&json!({"txid_str":"a","output_index":3}),
"a",
2
));
assert_eq!(recipient_amount(&json!({"output_details":[{"is_our_address":true,"amount":"21126"},{"is_our_address":false,"amount":"161650"}]})).unwrap(), 161650);
assert!(recipient_amount(
&json!({"output_details":[{"is_our_address":false,"amount":"bad"}]})
)
.is_err());
}
#[test]
fn cpfp_budget_covers_parent_and_preserves_change() {
assert_eq!(fee_budget(3, 142, 144, 112, 0, 1, 21126).unwrap(), 618);
assert!(fee_budget(5000, 142, 144, 112, 0, 1, 21126).is_err());
assert!(fee_budget(0, 142, 144, 112, 0, 1, 21126).is_err());
}
#[test]
fn rbf_pays_incremental_relay_cost_and_counts_only_extra_cost() {
let fee = fee_budget(3, 142, 144, 112, 650, 1, 21126).unwrap();
assert_eq!(fee, 763);
assert_eq!(fee - 650, 113);
}
#[test]
fn unsupported_outputs_and_malformed_ids_fail_closed() {
assert!(sweep_size(&json!({"output_type":"SCRIPT_TYPE_WITNESS_V0_SCRIPT_HASH"})).is_err());
assert!(txid_param(&json!({"txid":"../../file"})).is_err());
assert!(number(&json!(-1)).is_err());
assert!(btc_sats(&json!(-0.1)).is_err());
assert_eq!(btc_sats(&json!(0.00000650)).unwrap(), 650);
}
#[tokio::test]
async fn receipt_prevents_duplicate_submission_after_restart() {
let dir = std::env::temp_dir().join(uuid::Uuid::new_v4().to_string());
let path = dir.join("receipt.json");
let plan: Plan = serde_json::from_value(json!({"txid":"a","method":"cpfp","input_txid":"a","input_index":0,"parent_txid":"a","recipient_sats":1000,"rate_sat_vb":3,"current_fee_sats":144,"additional_fee_sats":618,"total_fee_sats":762,"budget_sats":618,"input_sats":21126,"parent_vsize":142,"sweep_vsize_bound":112,"tip":"tip"})).unwrap();
let op = Operation {
quote: Quote {
quote_id: "q".into(),
expires_at: now() + 60,
custom_rate: None,
plan,
},
status: "unknown".into(),
message: "pending".into(),
};
reserve(&path, &op).await.unwrap();
assert!(reserve(&path, &op).await.is_err());
let restored: Operation =
serde_json::from_slice(&tokio::fs::read(&path).await.unwrap()).unwrap();
assert_eq!(restored.quote.plan.budget_sats, 618);
tokio::fs::remove_dir_all(dir).await.unwrap();
}
}
@@ -0,0 +1,120 @@
//! Explicit on-chain fee choices retain priority; omitted choices target the next block.
use anyhow::{ensure, Context, Result};
use serde_json::Value;
pub(super) const DEFAULT_TARGET: i64 = 1;
pub(super) fn estimated_sat_per_vbyte(value: &Value) -> Result<u64> {
let per_kw = value["sat_per_kw"]
.as_u64()
.or_else(|| value["sat_per_kw"].as_str().and_then(|s| s.parse().ok()))
.context("Next-block fee estimate is unavailable")?;
let rate = per_kw.div_ceil(250);
ensure!(
(1..=5000).contains(&rate),
"Next-block fee estimate is outside supported bounds; choose an explicit fee"
);
Ok(rate)
}
pub(super) fn fee_options(params: &Value) -> Result<(Option<i64>, Option<i64>)> {
let integer = |key: &str, max: i64| -> Result<Option<i64>> {
match params.get(key) {
None | Some(Value::Null) => Ok(None),
Some(value) => {
let n = value
.as_i64()
.with_context(|| format!("{key} must be a positive whole number"))?;
ensure!((1..=max).contains(&n), "{key} must be between 1 and {max}");
Ok(Some(n))
}
}
};
let target = integer("target_conf", 1008)?;
let rate = integer("sat_per_vbyte", 5000)?;
ensure!(
target.is_none() || rate.is_none(),
"Specify either target_conf or sat_per_vbyte, not both"
);
Ok((
if rate.is_none() {
Some(target.unwrap_or(DEFAULT_TARGET))
} else {
None
},
rate,
))
}
#[cfg(test)]
mod tests {
use super::*;
use serde_json::json;
#[test]
fn estimates_round_up_and_missing_or_extreme_estimates_fail_closed() {
assert_eq!(
estimated_sat_per_vbyte(&json!({"sat_per_kw":"501"})).unwrap(),
3
);
assert_eq!(
estimated_sat_per_vbyte(&json!({"sat_per_kw":250})).unwrap(),
1
);
for v in [
json!({}),
json!({"sat_per_kw":0}),
json!({"sat_per_kw":-1}),
json!({"sat_per_kw":1250001}),
] {
assert!(estimated_sat_per_vbyte(&v).is_err());
}
}
#[test]
fn next_block_default_preserves_explicit_slower_and_custom_choices() {
assert_eq!(fee_options(&json!({})).unwrap(), (Some(1), None));
assert_eq!(
fee_options(&json!({"target_conf":null})).unwrap(),
(Some(1), None)
);
for target in [1, 3, 6, 144, 1008] {
assert_eq!(
fee_options(&json!({"target_conf":target})).unwrap(),
(Some(target), None)
);
}
for rate in [1, 17, 5000] {
assert_eq!(
fee_options(&json!({"sat_per_vbyte":rate})).unwrap(),
(None, Some(rate))
);
}
}
#[test]
fn malformed_explicit_fees_never_silently_become_fast() {
for value in [
json!(0),
json!(-1),
json!(1.5),
json!("6"),
json!(true),
json!({}),
json!(1009),
] {
assert!(fee_options(&json!({"target_conf":value})).is_err());
}
for value in [
json!(0),
json!(-1),
json!(1.5),
json!("6"),
json!(true),
json!(5001),
] {
assert!(fee_options(&json!({"sat_per_vbyte":value})).is_err());
}
assert!(fee_options(&json!({"target_conf":1,"sat_per_vbyte":2})).is_err());
}
}
+2
View File
@@ -1,4 +1,6 @@
mod channels;
mod fee_bump;
mod fee_policy;
mod info;
mod macaroons;
mod payments;
@@ -402,6 +402,9 @@ impl RpcHandler {
}));
}
self.group_fee_bump_history(raw_txs, &mut transactions, &client)
.await;
// Sort by timestamp descending (most recent first)
transactions.sort_by(|a, b| {
let ta = a.get("time_stamp").and_then(|v| v.as_i64()).unwrap_or(0);
+154 -37
View File
@@ -124,28 +124,8 @@ impl RpcHandler {
return Err(anyhow::anyhow!("Invalid Bitcoin address format"));
}
// Fee control: either a confirmation target or an explicit fee rate
let target_conf = params.get("target_conf").and_then(|v| v.as_i64());
let sat_per_vbyte = params.get("sat_per_vbyte").and_then(|v| v.as_i64());
if target_conf.is_some() && sat_per_vbyte.is_some() {
return Err(anyhow::anyhow!(
"Invalid fee parameters: specify either target_conf or sat_per_vbyte, not both"
));
}
if let Some(tc) = target_conf {
if !(1..=1008).contains(&tc) {
return Err(anyhow::anyhow!(
"Invalid target_conf: must be between 1 and 1008 blocks"
));
}
}
if let Some(rate) = sat_per_vbyte {
if !(1..=5000).contains(&rate) {
return Err(anyhow::anyhow!(
"Invalid sat_per_vbyte: must be between 1 and 5000"
));
}
}
// Omitted fees target the next block; explicit slower/custom choices win.
let (target_conf, sat_per_vbyte) = super::fee_policy::fee_options(&params)?;
info!(
addr = addr,
@@ -238,15 +218,12 @@ impl RpcHandler {
if !(546..=21_000_000 * 100_000_000).contains(&amount) {
return Err(anyhow::anyhow!("Invalid amount"));
}
let target_conf = params
.get("target_conf")
.and_then(|v| v.as_i64())
.unwrap_or(6);
if !(1..=1008).contains(&target_conf) {
return Err(anyhow::anyhow!(
"Invalid target_conf: must be between 1 and 1008 blocks"
));
}
let (target_conf, custom_rate) = super::fee_policy::fee_options(&params)?;
anyhow::ensure!(
custom_rate.is_none(),
"Fee estimation requires a confirmation target"
);
let target_conf = target_conf.unwrap_or(super::fee_policy::DEFAULT_TARGET);
let (client, macaroon_hex) = self.lnd_client().await?;
@@ -453,6 +430,56 @@ impl RpcHandler {
Ok(settled)
}
/// Verify against LND at download time, rather than relying on a browser
/// having polled first. The memo/amount also recover pre-upgrade in-memory
/// entitlements after restart; unrelated invoices never unlock a file.
pub(crate) async fn settle_content_invoice(
&self,
hash: &str,
content_id: &str,
) -> Result<bool> {
anyhow::ensure!(
hash.len() == 64 && hash.bytes().all(|c| c.is_ascii_hexdigit()),
"Invalid payment hash"
);
let hash = hash.to_ascii_lowercase();
let existing = crate::content_invoice::lookup(&self.config.data_dir, &hash).await?;
if let Some((id, _)) = &existing {
if id != content_id {
return Ok(false);
}
}
if crate::content_invoice::is_paid_for(&self.config.data_dir, &hash, content_id).await {
return Ok(true);
}
let (client, macaroon_hex) = self.lnd_client().await?;
let response = client
.get(format!("{LND_REST_BASE_URL}/v1/invoice/{hash}"))
.header("Grpc-Metadata-macaroon", &macaroon_hex)
.send()
.await?;
if response.status() == reqwest::StatusCode::NOT_FOUND {
return Ok(false);
}
let body: serde_json::Value = response.error_for_status()?.json().await?;
let Some(price) = content_invoice_amount(&body, content_id) else {
return Ok(false);
};
if existing
.as_ref()
.is_some_and(|(_, expected)| *expected != price)
{
return Ok(false);
}
crate::content_invoice::record_pending(&self.config.data_dir, &hash, content_id, price)
.await?;
let settled = content_invoice_fully_settled(&body, price);
if settled {
crate::content_invoice::mark_paid(&self.config.data_dir, &hash).await?;
}
Ok(settled)
}
/// Generate a fresh on-chain receive address (seller side, #46).
pub(crate) async fn new_onchain_address(&self) -> Result<String> {
let (client, macaroon_hex) = self.lnd_client().await?;
@@ -732,10 +759,24 @@ impl RpcHandler {
total_amount += amount;
}
let sat_per_vbyte = params
.get("fee_rate_sat_per_vbyte")
.and_then(|v| v.as_u64())
.unwrap_or(10);
let (_, explicit_rate) = super::fee_policy::fee_options(&serde_json::json!({
"sat_per_vbyte": params.get("fee_rate_sat_per_vbyte")
}))?;
let (client, macaroon_hex) = self.lnd_client().await?;
let sat_per_vbyte = if let Some(rate) = explicit_rate {
rate as u64
} else {
let response = client
.get(format!("{LND_REST_BASE_URL}/v2/wallet/estimatefee/1"))
.header("Grpc-Metadata-macaroon", &macaroon_hex)
.send()
.await
.context("Cannot estimate the next-block fee")?
.error_for_status()
.context("Next-block fee estimate rejected")?;
let estimate: serde_json::Value = response.json().await?;
super::fee_policy::estimated_sat_per_vbyte(&estimate)?
};
info!(
total_amount = total_amount,
@@ -743,8 +784,6 @@ impl RpcHandler {
"Creating PSBT for hardware wallet signing"
);
let (client, macaroon_hex) = self.lnd_client().await?;
let fund_body = serde_json::json!({
"raw": {
"outputs": lnd_outputs,
@@ -1444,3 +1483,81 @@ mod tests {
assert!(s.contains("[LND_REST_UNREACHABLE]"), "got: {s}");
}
}
// LND REST uses decimal strings for int64 fields. Match the complete seller
// memo, not a substring supplied by a buyer or an arbitrary settled invoice.
fn json_u64(value: &serde_json::Value) -> Option<u64> {
value.as_u64().or_else(|| value.as_str()?.parse().ok())
}
fn content_invoice_fully_settled(body: &serde_json::Value, price: u64) -> bool {
let settled = match body.get("state").and_then(|v| v.as_str()) {
Some(state) => state == "SETTLED",
None => body.get("settled").and_then(|v| v.as_bool()) == Some(true),
};
settled
&& price > 0
&& body
.get("amt_paid_sat")
.and_then(json_u64)
.is_some_and(|paid| paid >= price)
}
fn content_invoice_amount(body: &serde_json::Value, content_id: &str) -> Option<u64> {
if body.get("memo")?.as_str()? != format!("Archipelago peer file {content_id}") {
return None;
}
body.get("value").and_then(json_u64).filter(|v| *v > 0)
}
#[cfg(test)]
mod peer_file_invoice_tests {
use super::*;
#[test]
fn settlement_requires_terminal_state_and_full_amount() {
for state in ["OPEN", "ACCEPTED", "CANCELED", "unknown"] {
assert!(!content_invoice_fully_settled(
&serde_json::json!({"state":state,"settled":true,"amt_paid_sat":"100"}),
7
));
}
for amount in [
serde_json::json!(6),
serde_json::json!("-1"),
serde_json::json!(null),
serde_json::json!("bad"),
] {
assert!(!content_invoice_fully_settled(
&serde_json::json!({"state":"SETTLED","amt_paid_sat":amount}),
7
));
}
for amount in [serde_json::json!(7), serde_json::json!("8")] {
assert!(content_invoice_fully_settled(
&serde_json::json!({"state":"SETTLED","amt_paid_sat":amount}),
7
));
}
assert!(content_invoice_fully_settled(
&serde_json::json!({"settled":true,"amt_paid_sat":"7"}),
7
));
assert!(!content_invoice_fully_settled(
&serde_json::json!({"state":"SETTLED","amt_paid_sat":"7"}),
0
));
}
#[test]
fn legacy_recovery_requires_exact_file_memo_and_positive_amount() {
let invoice = serde_json::json!({"memo":"Archipelago peer file file-1", "value":"7"});
assert_eq!(content_invoice_amount(&invoice, "file-1"), Some(7));
assert_eq!(content_invoice_amount(&invoice, "file-2"), None);
for value in [
serde_json::json!("-1"),
serde_json::json!(0),
serde_json::json!("bad"),
] {
let mut invalid = invoice.clone();
invalid["value"] = value;
assert_eq!(content_invoice_amount(&invalid, "file-1"), None);
}
}
}
@@ -221,6 +221,10 @@ impl RpcHandler {
params: Option<serde_json::Value>,
) -> Result<serde_json::Value> {
let params = params.ok_or_else(|| anyhow::anyhow!("Missing params"))?;
if let Some(job) = self.flash_job.read().await.as_ref() {
anyhow::ensure!(job.snapshot().await.done,
"A firmware flash is in progress; wait before reconnecting or changing radio settings");
}
let mut config = mesh::load_config(&self.config.data_dir).await?;
@@ -325,7 +329,16 @@ impl RpcHandler {
{
let service_arc = Arc::clone(&self.mesh_service);
let config_for_apply = config.clone();
let flash_jobs = Arc::clone(&self.flash_job);
tokio::spawn(async move {
// Serialize against flash registration. If a flash started
// after this RPC saved settings, its completion applies them.
let flash_guard = flash_jobs.read().await;
if let Some(job) = flash_guard.as_ref() {
if !job.snapshot().await.done {
return;
}
}
let mut service = service_arc.write().await;
if let Some(svc) = service.as_mut() {
if let Err(e) = svc.configure(config_for_apply).await {
+3 -1
View File
@@ -110,7 +110,8 @@ impl RpcHandler {
// `mesh.probe-device` call (e.g. the hot-swap modal's own re-probe)
// from opening the identical port at the same time and corrupting
// both operations' handshakes.
if let Some(job) = self.flash_job.read().await.as_ref() {
let flash_guard = self.flash_job.read().await;
if let Some(job) = flash_guard.as_ref() {
anyhow::ensure!(
job.snapshot().await.done,
"A firmware flash is in progress — refusing to probe the serial port until it finishes"
@@ -131,6 +132,7 @@ impl RpcHandler {
}
}
let probe = mesh::listener::probe_device(&path).await?;
drop(flash_guard);
Ok(serde_json::to_value(probe)?)
}
@@ -985,28 +985,26 @@ pub(super) async fn get_app_config(
)
}
"nginx-proxy-manager" => {
let storage = crate::container::npm::resolve_storage().await?;
let admin_port = allocator
.allocate_or_get(app_id, 8081, 81)
.await
.unwrap_or(8081);
let http_port = allocator
.allocate_or_get("nginx-proxy-manager-http", 8084, 80)
.allocate_or_get("nginx-proxy-manager-http", 8088, 80)
.await
.unwrap_or(8084);
.unwrap_or(8088);
let https_port = allocator
.allocate_or_get("nginx-proxy-manager-https", 8444, 443)
.await
.unwrap_or(8444);
(
vec![
format!("{}:81", admin_port),
format!("{}:80", http_port),
format!("{}:443", https_port),
],
vec![
"/var/lib/archipelago/nginx-proxy-manager/data:/data".to_string(),
"/var/lib/archipelago/nginx-proxy-manager/letsencrypt:/etc/letsencrypt".to_string(),
format!("127.0.0.1:{}:81", admin_port),
format!("127.0.0.1:{}:80", http_port),
format!("127.0.0.1:{}:443", https_port),
],
storage.bind_mounts(),
vec![],
None,
None,
@@ -22,6 +22,18 @@ const ARCHIVAL_BITCOIN_DEPENDENCY: &str = "bitcoin:archival";
/// hardcoded id list below — a new app just declares the dependency instead
/// of needing a code change here.
fn manifest_declares_archival_bitcoin(package_id: &str) -> bool {
// Registry-only apps need the same guard as OTA-bundled manifests. Honor
// the verified catalog's effective manifest before the disk fallback.
if let Some((_, value)) = crate::container::app_catalog::catalog_manifest_values()
.into_iter()
.find(|(id, _)| id == package_id)
{
if let Some(manifest) =
crate::container::app_catalog::catalog_manifest_overlay(package_id, value)
{
return dependency_list_declares_archival_bitcoin(&manifest.app.dependencies);
}
}
for apps_dir in manifest_apps_dirs() {
let path = apps_dir.join(package_id).join("manifest.yml");
let Ok(contents) = std::fs::read_to_string(&path) else {
@@ -1055,6 +1067,14 @@ mod tests {
// edit to `requires_unpruned_bitcoin`.
assert!(manifest_declares_archival_bitcoin("electrumx"));
assert!(manifest_declares_archival_bitcoin("mempool"));
let angor = archipelago_container::AppManifest::parse(include_str!(concat!(
env!("CARGO_MANIFEST_DIR"),
"/../../apps/angor-indexer/manifest.yml"
)))
.unwrap();
assert!(dependency_list_declares_archival_bitcoin(
&angor.app.dependencies
));
// An app whose manifest exists but never declares the marker.
assert!(!manifest_declares_archival_bitcoin("bitcoin-knots"));
// An id with no manifest on disk at all.
+21 -95
View File
@@ -573,6 +573,9 @@ impl RpcHandler {
"message": format!("Package {} installed and started", package_id)
}));
}
Err(e) if e.downcast_ref::<crate::container::prod_orchestrator::InstallPrerequisiteError>().is_some() => {
return Err(super::dependencies::DependencyGateError(e.to_string()).into());
}
Err(e) if is_unknown_app_id_error(&e) => {
info!(
"Install {}: orchestrator has no manifest mapping yet, falling back to legacy installer",
@@ -690,7 +693,11 @@ impl RpcHandler {
// These standalone web UIs have repeatedly lost host listeners
// under Podman's rootless pasta backend while staying healthy internally.
// Use slirp4netns/rootlessport for this standalone web UI.
run_args.push("--network=slirp4netns:allow_host_loopback=true");
run_args.push(if package_id == "nginx-proxy-manager" {
"--network=slirp4netns:allow_host_loopback=true,cidr=169.254.1.0/24"
} else {
"--network=slirp4netns:allow_host_loopback=true"
});
} else if needs_archy_net(package_id) {
// Create archy-net if it doesn't exist (idempotent — "already exists" is fine)
match tokio::process::Command::new("podman")
@@ -1565,88 +1572,8 @@ autopilot.active=false\n",
super::pine_ha::restart_home_assistant_if_running().await;
}
}
if package_id == "filebrowser" {
// Generate a random password (32 bytes, hex-encoded)
let mut buf = [0u8; 32];
rand::RngCore::fill_bytes(&mut rand::rngs::OsRng, &mut buf);
let password = hex::encode(buf);
let client = match reqwest::Client::builder()
.timeout(std::time::Duration::from_secs(10))
.build()
{
Ok(c) => c,
Err(e) => {
tracing::warn!("Failed to create HTTP client for FileBrowser hook: {}", e);
return;
}
};
// Retry loop: FileBrowser may take time to initialize its SQLite database
let mut password_changed = false;
for attempt in 0..6u32 {
let delay = if attempt == 0 { 5 } else { 10 };
tokio::time::sleep(std::time::Duration::from_secs(delay)).await;
// Try to log in with default credentials
let login_res = client
.post("http://127.0.0.1:8083/api/login")
.json(&serde_json::json!({"username": "admin", "password": "admin"}))
.send()
.await;
let token = match login_res {
Ok(resp) if resp.status().is_success() => match resp.text().await {
Ok(t) => t.trim_matches('"').to_string(),
Err(_) => continue,
},
_ => {
debug!("FileBrowser not ready (attempt {}/6)", attempt + 1);
continue;
}
};
// Change admin password
let change_res = client
.put("http://127.0.0.1:8083/api/users/1")
.header("X-Auth", &token)
.json(&serde_json::json!({"password": password}))
.send()
.await;
match change_res {
Ok(resp) if resp.status().is_success() => {
let secret_dir = "/var/lib/archipelago/secrets/filebrowser";
if let Err(e) = tokio::fs::create_dir_all(secret_dir).await {
tracing::warn!("Failed to create filebrowser secrets dir: {}", e);
}
let pw_path = format!("{}/password", secret_dir);
if let Err(e) = tokio::fs::write(&pw_path, &password).await {
tracing::warn!("Failed to write filebrowser password: {}", e);
}
// Set restrictive permissions on the password file
#[cfg(unix)]
{
use std::os::unix::fs::PermissionsExt;
let _ = std::fs::set_permissions(
&pw_path,
std::fs::Permissions::from_mode(0o600),
);
}
info!("FileBrowser admin password secured (default credentials replaced)");
password_changed = true;
break;
}
_ => continue,
}
}
if !password_changed {
tracing::warn!(
"FileBrowser password could not be changed after 6 attempts — \
default credentials (admin/admin) remain active"
);
}
}
// File Browser credentials are provisioned and verified before the
// server starts. Never attempt a default-password change after launch.
// Auto-configure Tor hidden service for protocol services (LND, ElectrumX, Bitcoin)
{
@@ -1909,10 +1836,10 @@ autopilot.active=false\n",
}
pub(in crate::api::rpc) async fn handle_filebrowser_token(&self) -> Result<serde_json::Value> {
let secret_path = "/var/lib/archipelago/secrets/filebrowser/password";
let password = tokio::fs::read_to_string(secret_path)
.await
.unwrap_or_else(|_| "admin".to_string());
let credentials = crate::container::filebrowser::cloud_credentials(std::path::Path::new(
"/var/lib/archipelago/secrets/filebrowser",
))
.await?;
let client = reqwest::Client::builder()
.timeout(std::time::Duration::from_secs(10))
@@ -1921,7 +1848,7 @@ autopilot.active=false\n",
let resp = client
.post("http://127.0.0.1:8083/api/login")
.json(&serde_json::json!({"username": "admin", "password": password}))
.json(&serde_json::json!({"username": credentials.username, "password": credentials.password}))
.send()
.await
.context("Failed to connect to FileBrowser")?;
@@ -1951,17 +1878,16 @@ autopilot.active=false\n",
super::validation::validate_app_id(app_id)?;
if app_id == "filebrowser" {
let password =
tokio::fs::read_to_string("/var/lib/archipelago/secrets/filebrowser/password")
.await
.map(|p| p.trim().to_string())
.unwrap_or_else(|_| "admin".to_string());
let credentials = crate::container::filebrowser::cloud_credentials(
std::path::Path::new("/var/lib/archipelago/secrets/filebrowser"),
)
.await?;
return Ok(serde_json::json!({
"title": "File Browser credentials",
"description": "Use these credentials when File Browser asks you to sign in.",
"credentials": [
{ "label": "Username", "value": "admin" },
{ "label": "Password", "value": password, "sensitive": true }
{ "label": "Username", "value": credentials.username },
{ "label": "Password", "value": credentials.password, "sensitive": true }
]
}));
}
+208 -59
View File
@@ -1576,41 +1576,110 @@ async fn repair_netbird_network() {
}
async fn repair_nginx_proxy_manager_container() {
repair_nginx_proxy_manager_dirs().await;
// Quadlet owns managed containers; its backed-up reconciliation applies
// port and mount changes. Never remove a systemd-owned container here.
if crate::container::quadlet::unit_exists("nginx-proxy-manager").await {
return;
}
// Serialize repair so a second caller cannot overlap a replacement.
static REPAIR: tokio::sync::Mutex<()> = tokio::sync::Mutex::const_new(());
let _repair = REPAIR.lock().await;
if !nginx_proxy_manager_has_legacy_admin_port().await {
return;
}
if let Err(error) = repair_legacy_nginx_proxy_manager().await {
tracing::warn!(error = %error, "NPM legacy repair failed; persistent state preserved");
}
}
install_log(
"START REPAIR: nginx-proxy-manager - recreating stale container using host port 8081",
const NPM_PREVIOUS_CONTAINER: &str = "archy-npm-upgrade-previous";
async fn restore_failed_npm_repair() -> Result<()> {
let removed = podman_control(&["rm", "-f", "--ignore", "nginx-proxy-manager"]).await?;
anyhow::ensure!(
removed.status.success(),
"cannot remove failed NPM replacement; previous container retained"
);
let renamed =
podman_control(&["rename", NPM_PREVIOUS_CONTAINER, "nginx-proxy-manager"]).await?;
anyhow::ensure!(
renamed.status.success(),
"cannot restore previous NPM container name"
);
let started = podman_control(&["start", "nginx-proxy-manager"]).await?;
anyhow::ensure!(
started.status.success(),
"previous NPM container restored but failed to start"
);
Ok(())
}
async fn repair_legacy_nginx_proxy_manager() -> Result<()> {
let previous = podman_control(&["container", "exists", NPM_PREVIOUS_CONTAINER]).await?;
anyhow::ensure!(previous.status.code() == Some(1),
"NPM previous-container slot is occupied or cannot be inspected; preserve it and review interrupted repair before proceeding");
let inspection = podman_control(&[
"inspect",
"nginx-proxy-manager",
"--format",
"{{json .Config.Env}}",
])
.await?;
anyhow::ensure!(
inspection.status.success(),
"cannot preserve NPM environment before repair"
);
let environment: Vec<String> =
serde_json::from_slice(&inspection.stdout).context("invalid original NPM environment")?;
let environment = npm_repair_environment(&environment)?;
let storage = crate::container::npm::resolve_storage().await?;
let mut manifest: archipelago_container::AppManifest = serde_yaml::from_str(include_str!(
"../../../../../../apps/nginx-proxy-manager/manifest.yml"
))?;
storage.apply(&mut manifest)?;
let stopped = podman_control(&["stop", "--time", "30", "nginx-proxy-manager"]).await?;
anyhow::ensure!(
stopped.status.success(),
"could not stop NPM for a consistent backup"
);
if let Err(error) = crate::container::migration_backup::snapshot(
&manifest,
std::path::Path::new("/var/lib/archipelago"),
None,
)
.await;
let _ = podman_control(&["rm", "-f", "nginx-proxy-manager"]).await;
crate::container::ghost_reaper::reap_for_app("nginx-proxy-manager").await;
if let Err(err) = recreate_nginx_proxy_manager_container().await {
tracing::warn!(error = %err, "failed to recreate stale nginx-proxy-manager container");
.await
{
let _ = podman_control(&["start", "nginx-proxy-manager"]).await;
return Err(error);
}
// Keep the original runtime definition for rollback, including its operator
// options. Never delete it before the replacement has become ready.
let renamed = podman_control(&["rename", "nginx-proxy-manager", NPM_PREVIOUS_CONTAINER]).await;
if !renamed.as_ref().is_ok_and(|out| out.status.success()) {
let _ = podman_control(&["start", "nginx-proxy-manager"]).await;
anyhow::bail!("could not retain legacy NPM runtime; state backup preserved, inspect both container names before retrying");
}
let replacement = async {
recreate_nginx_proxy_manager_container(&storage, &environment).await?;
anyhow::ensure!(
wait_for_runtime_host_port("nginx-proxy-manager", 8081, 180).await,
"replacement NPM admin listener did not become ready"
);
Ok::<_, anyhow::Error>(())
}
async fn repair_nginx_proxy_manager_dirs() {
let _ = tokio::process::Command::new("sudo")
.args([
"mkdir",
"-p",
"/var/lib/archipelago/nginx-proxy-manager/data/letsencrypt-acme-challenge/.well-known/acme-challenge",
"/var/lib/archipelago/nginx-proxy-manager/letsencrypt",
])
.output()
.await;
let _ = tokio::process::Command::new("sudo")
.args([
"chown",
"-R",
"1000:1000",
"/var/lib/archipelago/nginx-proxy-manager",
])
.output()
.await;
if let Err(error) = replacement {
restore_failed_npm_repair()
.await
.context("restoring previous NPM after replacement failure")?;
return Err(error);
}
let removed = podman_control(&["rm", NPM_PREVIOUS_CONTAINER]).await?;
anyhow::ensure!(
removed.status.success(),
"replacement ready but previous NPM cleanup failed; rollback container retained"
);
Ok(())
}
async fn nginx_proxy_manager_has_legacy_admin_port() -> bool {
@@ -1645,36 +1714,75 @@ async fn nginx_proxy_manager_has_legacy_admin_port() -> bool {
ports.contains(":81->81/tcp") || ports.contains(":8443->443/tcp")
}
async fn recreate_nginx_proxy_manager_container() -> Result<()> {
tokio::process::Command::new("sudo")
.args([
"mkdir",
"-p",
"/var/lib/archipelago/nginx-proxy-manager/data/letsencrypt-acme-challenge/.well-known/acme-challenge",
"/var/lib/archipelago/nginx-proxy-manager/letsencrypt",
])
.output()
.await
.context("failed to create nginx-proxy-manager data directories")?;
let _ = tokio::process::Command::new("sudo")
.args([
"chown",
"-R",
"1000:1000",
"/var/lib/archipelago/nginx-proxy-manager",
])
.output()
.await;
fn npm_repair_environment(values: &[String]) -> Result<Vec<(String, String)>> {
values.iter().map(|value| {
let (key, value) = value.split_once('=').context("invalid NPM environment entry")?;
anyhow::ensure!(!key.is_empty() && key.chars().all(|c| c.is_ascii_alphanumeric() || c == '_'),
"unsupported NPM environment name; original container preserved");
anyhow::ensure!(!value.contains(['\n', '\r', '\0']),
"NPM environment requires explicit migration of a multiline value; original container preserved");
Ok((key.to_owned(), value.to_owned()))
}).collect()
}
let image = crate::container::image_versions::pinned_image_for_app("nginx-proxy-manager")
.unwrap_or_else(|| "docker.io/jc21/nginx-proxy-manager:latest".to_string());
struct NpmRepairEnvironmentFile(std::path::PathBuf);
impl NpmRepairEnvironmentFile {
fn create(environment: &[(String, String)]) -> Result<Self> {
use std::io::Write;
use std::os::unix::fs::OpenOptionsExt;
let path = std::env::temp_dir().join(format!(".archy-npm-env-{}", uuid::Uuid::new_v4()));
let mut file = std::fs::OpenOptions::new()
.write(true)
.create_new(true)
.mode(0o600)
.open(&path)?;
let guard = Self(path);
for (key, value) in environment {
writeln!(file, "{key}={value}")?;
}
file.sync_all()?;
Ok(guard)
}
}
impl Drop for NpmRepairEnvironmentFile {
fn drop(&mut self) {
let _ = std::fs::remove_file(&self.0);
}
}
async fn recreate_nginx_proxy_manager_container(
storage: &crate::container::npm::Storage,
environment: &[(String, String)],
) -> Result<()> {
// Existing directories and ownership came from the active runtime. Never
// create a second database tree or recursively rewrite data permissions.
for directory in [&storage.data, &storage.certificates] {
anyhow::ensure!(
std::path::Path::new(directory).is_dir(),
"NPM persistent directory is missing"
);
}
// This repair changes connectivity, not NPM's application version. Keep
// the exact old image so rollback never starts an older binary against a
// database that an incidental mutable-tag update may have migrated.
let image_output =
podman_control(&["inspect", NPM_PREVIOUS_CONTAINER, "--format", "{{.Image}}"]).await?;
anyhow::ensure!(
image_output.status.success(),
"cannot resolve original NPM image for repair"
);
let image = String::from_utf8(image_output.stdout)?.trim().to_string();
anyhow::ensure!(!image.is_empty(), "original NPM image is missing");
let mut args = vec![
"run".to_string(),
"-d".to_string(),
"--name".to_string(),
"nginx-proxy-manager".to_string(),
"--restart=unless-stopped".to_string(),
"--network=slirp4netns:allow_host_loopback=true".to_string(),
"--network=slirp4netns:allow_host_loopback=true,cidr=169.254.1.0/24".to_string(),
"--cap-drop=ALL".to_string(),
"--security-opt=no-new-privileges:true".to_string(),
"--pids-limit=4096".to_string(),
@@ -1682,24 +1790,32 @@ async fn recreate_nginx_proxy_manager_container() -> Result<()> {
args.extend(get_app_capabilities("nginx-proxy-manager"));
args.extend([
"-p".to_string(),
"8081:81".to_string(),
"127.0.0.1:8081:81".to_string(),
"-p".to_string(),
"8084:80".to_string(),
"127.0.0.1:8088:80".to_string(),
"-p".to_string(),
"8444:443".to_string(),
"127.0.0.1:8444:443".to_string(),
"-v".to_string(),
"/var/lib/archipelago/nginx-proxy-manager/data:/data".to_string(),
format!("{}:/data", storage.data),
"-v".to_string(),
"/var/lib/archipelago/nginx-proxy-manager/letsencrypt:/etc/letsencrypt".to_string(),
format!("{}:/etc/letsencrypt", storage.certificates),
"--memory".to_string(),
get_memory_limit("nginx-proxy-manager").to_string(),
"--cpus=2".to_string(),
]);
args.extend(get_health_check_args("nginx-proxy-manager", ""));
// Keep values out of argv/logs AND out of Podman's host environment
// (a container's PATH or LD_PRELOAD must never alter the host command).
let env_file = NpmRepairEnvironmentFile::create(environment)?;
args.extend([
"--env-file".to_string(),
env_file.0.to_string_lossy().into_owned(),
]);
args.push(image);
let refs = args.iter().map(String::as_str).collect::<Vec<_>>();
let output = podman_control(&refs).await?;
let mut command = tokio::process::Command::new("podman");
command.args(&args);
let output = command_with_timeout(command, Duration::from_secs(120), "NPM replacement").await?;
if !output.status.success() {
anyhow::bail!(
"podman run nginx-proxy-manager failed: {}",
@@ -1767,7 +1883,7 @@ fn runtime_host_ports(container_name: &str) -> Vec<u16> {
"vaultwarden" => vec![8082],
"gitea" => vec![3001, 2222, 3000],
"nextcloud" => vec![8085],
"nginx-proxy-manager" => vec![8081, 8084, 8444],
"nginx-proxy-manager" => vec![8081, 8088, 8444],
_ => Vec::new(),
};
ports
@@ -1778,7 +1894,7 @@ fn with_legacy_extra_ports(container_name: &str, mut ports: Vec<u16>) -> Vec<u16
ports.push(3000);
}
if container_name == "nginx-proxy-manager" {
for port in [8084, 8444] {
for port in [8088, 8444] {
if !ports.contains(&port) {
ports.push(port);
}
@@ -1843,6 +1959,7 @@ async fn wait_for_runtime_host_port(container_name: &str, port: u16, timeout_sec
loop {
let ready = match container_name {
"uptime-kuma" => http_host_port_ready(port, "/").await,
"nginx-proxy-manager" => http_host_port_ready(port, "/api/").await,
_ => tokio::net::TcpStream::connect(("127.0.0.1", port))
.await
.is_ok(),
@@ -2151,6 +2268,38 @@ pub(super) fn orchestrator_uninstall_app_ids(package_id: &str) -> Vec<String> {
#[cfg(test)]
mod tests {
#[test]
fn npm_environment_backup_is_private_exact_and_removed_on_drop() {
use std::os::unix::fs::PermissionsExt;
let values = vec![
"DB_PASSWORD=fixture=a b".to_string(),
"PATH=/container/only".to_string(),
];
let parsed = super::npm_repair_environment(&values).unwrap();
let host_path = std::env::var_os("PATH");
let path = {
let file = super::NpmRepairEnvironmentFile::create(&parsed).unwrap();
assert_eq!(
std::fs::metadata(&file.0).unwrap().permissions().mode() & 0o777,
0o600
);
assert_eq!(
std::fs::read_to_string(&file.0).unwrap(),
"DB_PASSWORD=fixture=a b\nPATH=/container/only\n"
);
assert_eq!(std::env::var_os("PATH"), host_path);
file.0.clone()
};
assert!(!path.exists());
for value in [
"INVALID",
"=empty key",
"KEY=value\nINJECTED=true",
"--env=value",
] {
assert!(super::npm_repair_environment(&[value.to_string()]).is_err());
}
}
use super::*;
#[tokio::test]
+132 -18
View File
@@ -138,6 +138,119 @@ const NGINX_FEDIMINT_NEW: &str = " sub_filter_types text/css application/
const NGINX_FEDIMINT_SNIPPET_ANCHOR: &str = "proxy_pass http://127.0.0.1:8175/;";
const NGINX_FEDIMINT_SNIPPET_INSERT: &str = "proxy_pass http://127.0.0.1:8175/;\n proxy_set_header Accept-Encoding \"\";\n sub_filter_types text/css application/javascript application/json;\n sub_filter_once off;\n sub_filter 'href=\"/' 'href=\"/app/fedimint/';\n sub_filter 'src=\"/' 'src=\"/app/fedimint/';\n sub_filter \"href='/\" \"href='/app/fedimint/\";\n sub_filter \"src='/\" \"src='/app/fedimint/\";\n sub_filter 'url(\"/' 'url(\"/app/fedimint/';\n sub_filter \"url('/\" \"url('/app/fedimint/\";\n sub_filter '</head>' '<script src=\"/nostr-provider.js\"></script></head>';";
/// Finish manifest promotion before constructing the orchestrator or starting
/// catalog refresh/reconciliation. Replacing the app tree in the background
/// could let a reload observe its temporary empty state and forget disk-only apps.
pub async fn ensure_runtime_assets_ready() {
// Install the guard before any startup path can reload an older dashboard
// vhost. The canonical OTA/ISO config contains the same guard inline.
if Path::new(NGINX_CONF_PATH).exists() || Path::new(NGINX_ENABLED_CONF_PATH).exists() {
match host_sudo(&[
"python3",
"-c",
include_str!("../../../scripts/dashboard-public-guard.py"),
])
.await
{
Ok(status) if status.success() => debug!("Dashboard public source guard verified"),
Ok(status) => warn!("Dashboard public source guard needs attention: {status}"),
Err(error) => warn!("Dashboard public source guard could not run: {error}"),
}
}
match run_runtime_assets().await {
Ok(changed) if changed => info!("Runtime assets synchronized from OTA payload"),
Ok(_) => debug!("No OTA runtime payload to synchronize"),
Err(e) => warn!("Runtime asset bootstrap failed (non-fatal): {:#}", e),
}
// A binary-only qualification or OTA rollback can precede the matching
// script payload. Install the exact embedded helper before Quadlet
// reconciliation can introduce its required ExecStartPre command.
if let Err(error) = write_root_if_needed(
"/opt/archipelago/scripts/filebrowser-credentials.py",
include_str!("../../../scripts/filebrowser-credentials.py"),
)
.await
{
warn!("File Browser credential helper installation failed: {error:#}");
}
if let Err(error) = run_npm_bridge_bootstrap().await {
warn!("NPM public routing bootstrap needs attention: {error:#}");
}
// Repair the narrowly recognized legacy NPM tunnel override before app
// reconciliation. The embedded script ships in both OTA and ISO binaries.
// It preserves native wallet services and refuses unknown custom routing.
match tokio::process::Command::new("python3")
.arg("-c")
.arg(include_str!("../../../scripts/repair-npm-tunnel.py"))
.output()
.await
{
Ok(output) if output.status.success() => {
if !output.stdout.is_empty() {
info!("{}", String::from_utf8_lossy(&output.stdout).trim());
}
}
Ok(output) => warn!(
"NPM tunnel migration needs attention: {}",
String::from_utf8_lossy(&output.stderr).trim()
),
Err(error) => warn!("NPM tunnel migration could not run: {error}"),
}
match run_apps_dir_repair().await {
Ok(true) => {
info!("Populated /opt/archipelago/apps from installer copy at /etc/archipelago/apps")
}
Ok(false) => debug!("/opt/archipelago/apps already populated (or no installer copy)"),
Err(e) => warn!("Apps dir repair failed (non-fatal): {:#}", e),
}
}
async fn run_npm_bridge_bootstrap() -> Result<()> {
if !Path::new("/opt/archipelago/scripts").is_dir() {
return Ok(());
}
let mut units_changed = false;
for (path, content) in [
(
"/opt/archipelago/scripts/npm-public-bridge.py",
include_str!("../../../scripts/npm-public-bridge.py"),
),
(
"/opt/archipelago/scripts/dashboard-public-guard.py",
include_str!("../../../scripts/dashboard-public-guard.py"),
),
(
"/etc/systemd/system/archipelago-npm-bridge.service",
include_str!("../../../image-recipe/configs/archipelago-npm-bridge.service"),
),
(
"/etc/systemd/system/archipelago-npm-bridge.timer",
include_str!("../../../image-recipe/configs/archipelago-npm-bridge.timer"),
),
] {
let changed = write_root_if_needed(path, content).await?;
units_changed |= changed && (path.ends_with(".service") || path.ends_with(".timer"));
}
if units_changed {
anyhow::ensure!(
host_sudo(&["systemctl", "daemon-reload"]).await?.success(),
"NPM bridge daemon reload failed"
);
}
anyhow::ensure!(
host_sudo(&[
"systemctl",
"enable",
"--now",
"archipelago-npm-bridge.timer"
])
.await?
.success(),
"NPM bridge timer could not start"
);
Ok(())
}
/// Entry point called from main startup. Never returns an error to the caller —
/// failing to bootstrap host artifacts must not prevent the backend from serving.
pub async fn ensure_doctor_installed() {
@@ -146,11 +259,6 @@ pub async fn ensure_doctor_installed() {
Ok(false) => debug!("No stale Archipelago dev-mode service override found"),
Err(e) => warn!("Service override repair failed (non-fatal): {:#}", e),
}
match run_runtime_assets().await {
Ok(changed) if changed => info!("Runtime assets synchronized from OTA payload"),
Ok(_) => debug!("No OTA runtime payload to synchronize"),
Err(e) => warn!("Runtime asset bootstrap failed (non-fatal): {:#}", e),
}
match run().await {
Ok(changed) if changed => info!("Doctor artifacts synchronized with binary"),
Ok(_) => debug!("Doctor artifacts already in sync"),
@@ -168,13 +276,6 @@ pub async fn ensure_doctor_installed() {
Ok(false) => debug!("No stale bitcoin.conf found"),
Err(e) => warn!("Bitcoin RPC repair failed (non-fatal): {:#}", e),
}
match run_apps_dir_repair().await {
Ok(true) => {
info!("Populated /opt/archipelago/apps from installer copy at /etc/archipelago/apps")
}
Ok(false) => debug!("/opt/archipelago/apps already populated (or no installer copy)"),
Err(e) => warn!("Apps dir repair failed (non-fatal): {:#}", e),
}
match run_tor_helper_sync().await {
Ok(true) => info!("tor-helper.sh synchronized with binary"),
Ok(false) => debug!("tor-helper.sh already current"),
@@ -395,17 +496,28 @@ async fn run_runtime_assets() -> Result<bool> {
if nginx_src.exists() {
let src_s = nginx_src.to_string_lossy().to_string();
let status = host_sudo(&[
"install",
"-m",
"644",
"python3",
"-c",
include_str!("../../../scripts/dashboard-public-guard.py"),
"--install",
&src_s,
"/etc/nginx/sites-available/archipelago",
])
.await
.context("install nginx-archipelago.conf")?;
.context("install guarded nginx-archipelago.conf")?;
if !status.success() {
anyhow::bail!("install nginx-archipelago.conf exited with {}", status);
}
let acme_status = host_sudo(&[
"python3",
"-c",
include_str!("../../../scripts/npm-public-bridge.py"),
"--acme-only",
])
.await?;
anyhow::ensure!(
acme_status.success(),
"active NPM ACME root migration failed"
);
changed = true;
}
@@ -422,6 +534,8 @@ async fn run_runtime_assets() -> Result<bool> {
"archipelago-doctor.service",
"archipelago-doctor.timer",
"archipelago-host-secrets-audit.service",
"archipelago-npm-bridge.service",
"archipelago-npm-bridge.timer",
] {
let src = configs.join(unit);
if src.exists() {
@@ -449,7 +563,7 @@ async fn run_runtime_assets() -> Result<bool> {
// or directory"). Skipped when byte-identical; a running daemon is
// unaffected (install replaces the inode) and picks the new binary up
// on its next spawn.
for tool in ["archy-reticulum-daemon", "archy-rnodeconf"] {
for tool in ["archy-reticulum-daemon", "archy-rnodeconf", "archy-esptool"] {
let src = runtime_dir.join("radio-tools").join(tool);
if !src.exists() {
continue;
+193 -1
View File
@@ -102,6 +102,33 @@ pub struct AppCatalogEntry {
/// `docs/registry-manifest-design.md`.
#[serde(default, skip_serializing_if = "Option::is_none")]
pub manifest: Option<serde_json::Value>,
/// Backward-compatible catalog rollout: old daemons ignore these and keep
/// the base manifest. New daemons choose only variants they can safely apply.
#[serde(default, skip_serializing_if = "Vec::is_empty")]
pub manifest_variants: Vec<CatalogManifestVariant>,
}
#[derive(Debug, Clone, Serialize, Deserialize)]
pub struct CatalogManifestVariant {
pub requires: Vec<String>,
pub manifest: serde_json::Value,
}
fn selected_manifest(entry: AppCatalogEntry) -> Option<serde_json::Value> {
// Never let an unknown future requirement become an unsafe partial match.
for variant in entry.manifest_variants.into_iter().rev() {
if !variant.requires.is_empty()
&& variant.requires.iter().all(|capability| {
matches!(
capability.as_str(),
"runtime-migration-backup-v1" | "npm-legacy-host-gateway-v1"
)
})
{
return Some(variant.manifest);
}
}
entry.manifest
}
/// One selectable version in an app's `versions[]` list. The catalog carries a
@@ -234,7 +261,7 @@ pub fn catalog_manifest_values() -> Vec<(String, serde_json::Value)> {
load_catalog()
.apps
.into_iter()
.filter_map(|(id, e)| e.manifest.map(|m| (id, m)))
.filter_map(|(id, e)| selected_manifest(e).map(|m| (id, m)))
.collect()
}
@@ -443,6 +470,12 @@ pub struct CatalogRefresh {
/// changed. Best-effort: a fetch failure leaves the existing cache untouched
/// (origin-always-wins; updates simply aren't refreshed this cycle).
pub async fn refresh_catalog(data_dir: &Path) -> anyhow::Result<CatalogRefresh> {
// Explicit operator-only qualification of a signed candidate on selected
// nodes. Never change fleet mirrors or fall back to an older public catalog
// while a candidate is selected. Normal signature enforcement still applies.
if let Some(path) = std::env::var_os("ARCHY_APP_CATALOG_CANDIDATE") {
return refresh_candidate_catalog(data_dir, Path::new(&path)).await;
}
let mirrors = crate::update::load_mirrors(data_dir)
.await
.unwrap_or_default();
@@ -489,6 +522,49 @@ pub async fn refresh_catalog(data_dir: &Path) -> anyhow::Result<CatalogRefresh>
Err(last_err.unwrap_or_else(|| anyhow::anyhow!("no catalog mirrors reachable")))
}
async fn refresh_candidate_catalog(data_dir: &Path, path: &Path) -> anyhow::Result<CatalogRefresh> {
anyhow::ensure!(
path.is_absolute(),
"candidate catalog path must be absolute"
);
let metadata = tokio::fs::metadata(path)
.await
.context("inspect candidate catalog")?;
anyhow::ensure!(
metadata.is_file() && metadata.len() <= 4 * 1024 * 1024,
"candidate catalog must be a file no larger than 4 MiB"
);
let body = tokio::fs::read_to_string(path)
.await
.context("read candidate catalog")?;
anyhow::ensure!(
body.len() <= 4 * 1024 * 1024,
"candidate catalog exceeds 4 MiB"
);
let raw: serde_json::Value = serde_json::from_str(&body)?;
anyhow::ensure!(
matches!(
crate::trust::verify_detached(&raw)?,
crate::trust::SignatureStatus::Verified { anchored: true, .. }
),
"candidate catalog requires a signature anchored to the release root"
);
let catalog: AppCatalog = serde_json::from_value(raw)?;
let changed = write_cache(data_dir, &body)?;
if changed {
*CACHE.lock().unwrap() = None;
}
info!(
apps = catalog.apps.len(),
changed,
"app-catalog: using explicitly selected signed candidate; public catalog refresh paused"
);
Ok(CatalogRefresh {
apps: catalog.apps.len(),
changed,
})
}
async fn fetch_one(client: &reqwest::Client, url: &str) -> anyhow::Result<(AppCatalog, String)> {
let resp = client.get(url).send().await?;
if !resp.status().is_success() {
@@ -557,6 +633,122 @@ fn write_cache(data_dir: &Path, body: &str) -> anyhow::Result<bool> {
mod tests {
use super::*;
fn signed_candidate(key_byte: u8) -> serde_json::Value {
// Same test anchor as trust::signed_doc tests; never a production key.
let anchor = ed25519_dalek::SigningKey::from_bytes(&[7u8; 32]);
std::env::set_var(
"ARCHY_RELEASE_ROOT_PUBKEY",
hex::encode(anchor.verifying_key().to_bytes()),
);
let key = ed25519_dalek::SigningKey::from_bytes(&[key_byte; 32]);
let mut value = serde_json::json!({"schema":1,"apps":{"demo":{"version":"2"}},"future_field":{"retain":true}});
let (sig, did) = crate::trust::signed_doc::sign_detached(&key, &value).unwrap();
value["signature"] = sig.into();
value["signed_by"] = did.into();
value
}
#[tokio::test]
async fn candidate_catalog_preserves_signed_bytes_and_is_idempotent() {
let dir = tempfile::tempdir().unwrap();
let path = dir.path().join("candidate.json");
let body = serde_json::to_string_pretty(&signed_candidate(7)).unwrap();
std::fs::write(&path, &body).unwrap();
let first = refresh_candidate_catalog(dir.path(), &path).await.unwrap();
assert!(first.changed);
assert_eq!(first.apps, 1);
assert_eq!(
std::fs::read_to_string(dir.path().join(APP_CATALOG_FILE)).unwrap(),
body
);
assert!(
!refresh_candidate_catalog(dir.path(), &path)
.await
.unwrap()
.changed
);
}
#[tokio::test]
async fn rejected_candidate_never_replaces_previous_catalog() {
let dir = tempfile::tempdir().unwrap();
let path = dir.path().join("candidate.json");
let previous = "previous cached bytes";
write_cache(dir.path(), previous).unwrap();
let mut tampered = signed_candidate(7);
tampered["apps"]["demo"]["version"] = "tampered".into();
for body in [
"malformed".into(),
r#"{"schema":1,"apps":{}}"#.into(),
signed_candidate(11).to_string(),
tampered.to_string(),
" ".repeat(4 * 1024 * 1024 + 1),
] {
std::fs::write(&path, body).unwrap();
assert!(refresh_candidate_catalog(dir.path(), &path).await.is_err());
assert_eq!(
std::fs::read_to_string(dir.path().join(APP_CATALOG_FILE)).unwrap(),
previous
);
}
std::fs::remove_file(&path).unwrap();
assert!(refresh_candidate_catalog(dir.path(), &path).await.is_err());
assert!(
refresh_candidate_catalog(dir.path(), Path::new("relative.json"))
.await
.is_err()
);
assert_eq!(
std::fs::read_to_string(dir.path().join(APP_CATALOG_FILE)).unwrap(),
previous
);
}
#[test]
fn catalog_migration_variant_is_compatible_with_old_and_future_daemons() {
let raw = serde_json::json!({
"version": "2.45.0", "manifest": {"app": {"id": "portainer", "container": {}}},
"manifest_variants": [{"requires": ["runtime-migration-backup-v1"],
"manifest": {"app": {"id": "portainer", "container": {"network": "slirp4netns"}, "backup_before_runtime_change": true}}}]
});
#[derive(Deserialize)]
struct OldEntry {
manifest: serde_json::Value,
}
let old: OldEntry = serde_json::from_value(raw.clone()).unwrap();
assert!(old.manifest["app"]["container"].get("network").is_none());
let current: AppCatalogEntry = serde_json::from_value(raw.clone()).unwrap();
let chosen = selected_manifest(current).unwrap();
assert_eq!(chosen["app"]["container"]["network"], "slirp4netns");
assert_eq!(chosen["app"]["backup_before_runtime_change"], true);
let mut future = raw;
future["manifest_variants"][0]["requires"]
.as_array_mut()
.unwrap()
.push(serde_json::json!("unknown-next-capability"));
let chosen = selected_manifest(serde_json::from_value(future).unwrap()).unwrap();
assert!(chosen["app"]["container"].get("network").is_none());
}
#[test]
fn npm_gateway_variant_requires_explicit_runtime_support() {
let mut raw = serde_json::json!({
"version": "2.12.1", "manifest": {"network":"pasta"},
"manifest_variants": [{"requires":["runtime-migration-backup-v1", "npm-legacy-host-gateway-v1"],
"manifest":{"network":"slirp4netns:allow_host_loopback=true,cidr=169.254.1.0/24"}}]
});
assert_eq!(
selected_manifest(serde_json::from_value(raw.clone()).unwrap()).unwrap()["network"],
"slirp4netns:allow_host_loopback=true,cidr=169.254.1.0/24"
);
// A runtime missing any required capability must retain the base.
raw["manifest_variants"][0]["requires"][1] = serde_json::json!("unknown-gateway-v2");
assert_eq!(
selected_manifest(serde_json::from_value(raw).unwrap()).unwrap()["network"],
"pasta"
);
}
#[test]
fn parses_and_ignores_unknown_fields() {
let json = r#"{
@@ -103,6 +103,15 @@ pub fn companions_for(package_id: &str) -> &'static [CompanionSpec] {
}
}
/// Missing companion UIs are provisioned here, never by snapshot recovery.
/// A stale running-container snapshot must not resurrect an orphaned UI.
pub fn is_companion_app(app_id: &str) -> bool {
ALL_COMPANIONS
.iter()
.flat_map(|specs| specs.iter())
.any(|spec| spec.image_base == app_id)
}
/// Every companion this build knows how to provision. Kept beside
/// `companions_for` — a new companion must be added to both, or the reaper
/// will not recognise it as one of ours and will leave it running forever.
+251 -39
View File
@@ -16,6 +16,19 @@ use crate::data_model::{
PackageDataEntry, PackageState, ServiceStatus, StaticFiles,
};
/// One displayed package for each known container/manifest alias. Keep the
/// stopped-app restoration path in agreement with live-container discovery.
fn canonical_package_id(name: &str) -> &str {
match name.strip_prefix("archy-").unwrap_or(name) {
"immich_server" | "immich-server" => "immich",
"immich-postgres" => "immich_postgres",
"immich-redis" => "immich_redis",
"mempool-web" | "mempool-frontend" => "mempool",
"btcpay" | "btcpayserver" => "btcpay-server",
name => name,
}
}
pub struct DockerPackageScanner {
runtime: Arc<dyn ContainerRuntimeTrait>,
}
@@ -99,24 +112,8 @@ impl DockerPackageScanner {
debug!("Found {} UI containers", ui_containers.len());
for container in containers {
// Extract app ID from container name
// Support both archy-* containers (docker-compose) and plain names (manual)
let app_id = if container.name.starts_with("archy-") {
container
.name
.strip_prefix("archy-")
.unwrap_or(&container.name)
.to_string()
} else {
// Use the container name as-is for manually started containers
container.name.clone()
};
// Normalize multi-container app IDs to their canonical names
let app_id = match app_id.as_str() {
"immich_server" => "immich".to_string(),
_ => app_id,
};
// Use the same alias mapping as stopped-app restoration.
let app_id = canonical_package_id(&container.name).to_owned();
// Skip backend services (databases, APIs, etc.)
if excluded_services.contains(&app_id.as_str()) {
@@ -178,13 +175,11 @@ impl DockerPackageScanner {
} else {
// Prefer the known web UI port over arbitrary first binding
// (for example Gitea exposes SSH on 2222 before web on 3001).
let candidate = if uses_allocated_launch_port(&app_id) {
extract_lan_address(&container.ports)
.or_else(|| PodmanClient::lan_address_for(&app_id))
} else {
PodmanClient::lan_address_for(&app_id)
.or_else(|| extract_lan_address(&container.ports))
};
let candidate = package_launch_candidate(
&app_id,
&container.ports,
PodmanClient::lan_address_for(&app_id),
);
reachable_lan_address(&app_id, candidate).await
};
@@ -388,21 +383,16 @@ fn restore_absent_installed(
installed: &std::collections::HashSet<String>,
uninstalled: &std::collections::HashSet<String>,
) {
fn canonical(name: &str) -> &str {
let name = name.strip_prefix("archy-").unwrap_or(name);
match name {
"immich_server" => "immich",
_ => name,
}
}
let mut present: std::collections::HashSet<String> = containers
.iter()
.map(|c| canonical(&c.name).to_owned())
.map(|c| canonical_package_id(&c.name).to_owned())
.collect();
let removed: std::collections::HashSet<_> = uninstalled
.iter()
.map(|id| canonical_package_id(id))
.collect();
let removed: std::collections::HashSet<_> =
uninstalled.iter().map(|id| canonical(id)).collect();
for name in installed {
let id = canonical(name);
let id = canonical_package_id(name);
if removed.contains(id) || !present.insert(id.to_owned()) {
continue;
}
@@ -456,6 +446,28 @@ mod lifecycle_regression_tests {
use super::*;
use tokio::io::{AsyncReadExt, AsyncWriteExt};
#[test]
fn btcpay_aliases_share_one_package_without_promoting_dependencies() {
for name in ["btcpay", "btcpayserver", "btcpay-server", "archy-btcpay"] {
assert_eq!(canonical_package_id(name), "btcpay-server");
}
assert_eq!(canonical_package_id("archy-btcpay-db"), "btcpay-db");
assert_eq!(canonical_package_id("archy-nbxplorer"), "nbxplorer");
}
#[test]
fn immich_dependency_aliases_share_the_hidden_component_ids() {
for id in [
"immich-postgres",
"immich_postgres",
"archy-immich-postgres",
] {
assert_eq!(canonical_package_id(id), "immich_postgres");
}
assert_eq!(canonical_package_id("immich-redis"), "immich_redis");
assert_eq!(canonical_package_id("immich-server"), "immich");
}
#[test]
fn registry_survives_empty_runtime_and_deduplicates_aliases() {
let installed = ["archy-gitea", "gitea", "immich_server", "archy-removed"]
@@ -475,6 +487,33 @@ mod lifecycle_regression_tests {
assert_eq!(containers[0].state, ContainerState::Running);
}
#[test]
fn mempool_frontend_inventory_alias_does_not_create_a_second_package() {
let installed = ["mempool", "archy-mempool-web", "mempool-web"]
.into_iter()
.map(str::to_owned)
.collect();
let mut containers = Vec::new();
restore_absent_installed(&mut containers, &installed, &Default::default());
assert_eq!(containers.len(), 1);
assert_eq!(containers[0].name, "mempool");
containers[0].id = "live-frontend".into();
containers[0].state = ContainerState::Running;
restore_absent_installed(&mut containers, &installed, &Default::default());
assert_eq!(containers.len(), 1);
assert_eq!(containers[0].id, "live-frontend");
assert_eq!(containers[0].state, ContainerState::Running);
assert_eq!(canonical_package_id("archy-mempool-web"), "mempool");
assert_eq!(canonical_package_id("mempool-api"), "mempool-api");
containers.clear();
restore_absent_installed(
&mut containers,
&installed,
&["mempool".into()].into_iter().collect(),
);
assert!(containers.is_empty());
}
#[tokio::test]
async fn readiness_rejects_startup_errors_and_accepts_auth_and_redirects() {
for (status, expected) in [
@@ -964,14 +1003,20 @@ fn extract_lan_address(ports: &[String]) -> Option<String> {
let mut first_candidate = None;
for port_str in ports {
// Parse port strings like "0.0.0.0:18443->18443/tcp" or "0.0.0.0:18443-18444->18443-18444/tcp"
let Some(public_part) = port_str.split("->").next() else {
let Some((public_part, _)) = port_str.split_once("->") else {
continue;
};
let Some(port_part) = public_part.split(':').nth(1) else {
let Some((_, port_part)) = public_part.rsplit_once(':') else {
continue;
};
// Extract just the first port if it's a range (e.g., "18443-18444" -> "18443")
let host_port = port_part.split('-').next().unwrap_or(port_part);
let Ok(host_port) = host_port.parse::<u16>() else {
continue;
};
if host_port == 0 {
continue;
}
let candidate = format!("http://localhost:{}", host_port);
if first_candidate.is_none() {
first_candidate = Some(candidate.clone());
@@ -1089,6 +1134,46 @@ fn companion_lan_address(app_id: &str) -> Option<String> {
}
}
/// Companion dashboards remain usable while their backend is syncing. Never
/// probe a Bitcoin RPC or Electrum protocol socket as dashboard readiness.
fn package_launch_candidate(
app_id: &str,
ports: &[String],
known: Option<String>,
) -> Option<String> {
if let Some(companion) = companion_lan_address(app_id) {
return Some(companion);
}
if app_id == "nginx-proxy-manager" {
// 80/443 serve users' proxy hosts; only container port 81 serves the
// admin UI. Podman's binding order is unstable across recreation.
// Resolve its actual host allocation rather than guessing the first
// HTTP port or hardcoding the default host port 8081.
let admin_ports: Vec<String> = ports
.iter()
.filter(|port| {
port.split_once("->")
.is_some_and(|(_, target)| target == "81/tcp")
})
.cloned()
.collect();
// With published bindings, a missing admin mapping is not evidence
// that some unrelated service on the default host port is this UI.
return extract_lan_address(&admin_ports).or_else(|| {
if ports.is_empty() {
known
} else {
None
}
});
}
if uses_allocated_launch_port(app_id) {
extract_lan_address(ports).or(known)
} else {
known.or_else(|| extract_lan_address(ports))
}
}
fn uses_allocated_launch_port(app_id: &str) -> bool {
matches!(
app_id,
@@ -1173,7 +1258,134 @@ mod tor_service_name_tests {
#[cfg(test)]
mod extract_lan_address_tests {
use super::extract_lan_address;
use super::{extract_lan_address, package_launch_candidate};
#[test]
fn companion_dashboard_wins_over_backend_protocol_ports() {
for id in ["bitcoin", "bitcoin-core", "bitcoin-knots"] {
assert_eq!(
package_launch_candidate(
id,
&["127.0.0.1:8332->8332/tcp".into()],
Some("http://localhost:8332".into())
)
.as_deref(),
Some("http://localhost:8334")
);
}
for id in ["electrumx", "electrs", "mempool-electrs"] {
assert_eq!(
package_launch_candidate(
id,
&["127.0.0.1:50001->50001/tcp".into()],
Some("http://localhost:50001".into())
)
.as_deref(),
Some("http://localhost:50002")
);
}
assert_eq!(
package_launch_candidate(
"filebrowser",
&["127.0.0.1:19080->80/tcp".into()],
Some("http://localhost:8080".into())
)
.as_deref(),
Some("http://localhost:19080")
);
}
#[test]
fn npm_admin_launch_is_independent_of_proxy_binding_order() {
let mappings = [
"10.77.0.2:18081->80/tcp",
"10.77.0.2:18443->443/tcp",
"127.0.0.1:8081->81/tcp",
];
for order in [
[0, 1, 2],
[0, 2, 1],
[1, 0, 2],
[1, 2, 0],
[2, 0, 1],
[2, 1, 0],
] {
let ports: Vec<String> = order.iter().map(|&i| mappings[i].into()).collect();
assert_eq!(
package_launch_candidate(
"nginx-proxy-manager",
&ports,
Some("http://localhost:8081/".into())
)
.as_deref(),
Some("http://localhost:8081")
);
}
}
#[test]
fn npm_admin_launch_respects_host_allocation_and_ipv6_bindings() {
for binding in ["127.0.0.1", "0.0.0.0", "[::1]", "[::]"] {
let ports = vec![
"10.77.0.2:18081->80/tcp".into(),
format!("{binding}:28081->81/tcp"),
];
assert_eq!(
package_launch_candidate(
"nginx-proxy-manager",
&ports,
Some("http://localhost:8081/".into())
)
.as_deref(),
Some("http://localhost:28081")
);
}
let proxies = vec![
"10.77.0.2:18081->80/tcp".into(),
"10.77.0.2:18443->443/tcp".into(),
];
assert_eq!(
package_launch_candidate("nginx-proxy-manager", &proxies, None),
None
);
assert_eq!(
package_launch_candidate(
"nginx-proxy-manager",
&proxies,
Some("http://localhost:8081/".into())
),
None
);
}
#[test]
fn npm_without_port_information_uses_declared_admin_url() {
assert_eq!(
package_launch_candidate(
"nginx-proxy-manager",
&[],
Some("http://localhost:8081/".into())
)
.as_deref(),
Some("http://localhost:8081/")
);
}
#[test]
fn malformed_published_ports_do_not_become_launch_urls() {
for port in [
"81/tcp",
"127.0.0.1:bad->81/tcp",
"[::1]:0->81/tcp",
"[::]:65536->81/tcp",
"127.0.0.1:8081->81/udp",
] {
assert_eq!(
package_launch_candidate("nginx-proxy-manager", &[port.into()], None),
None
);
}
}
#[test]
fn skips_ssh_port_when_web_port_is_published() {
+567 -3
View File
@@ -5,7 +5,7 @@
//! starting the container with `--config /data/.filebrowser.json`.
use anyhow::{Context, Result};
use std::path::PathBuf;
use std::path::{Path, PathBuf};
use tokio::fs;
use crate::update::host_sudo;
@@ -17,6 +17,84 @@ pub const DEFAULT_CONFIG_PATH: &str = "/var/lib/archipelago/filebrowser-data/.fi
const DEFAULT_CONFIG_JSON: &str =
"{\"port\":80,\"baseURL\":\"\",\"address\":\"0.0.0.0\",\"database\":\"/data/filebrowser.db\",\"root\":\"/srv\",\"log\":\"stdout\"}\n";
/// One atomically published record shared by setup, Cloud and credentials UI.
/// Deliberately has no Debug implementation: the password must never be logged.
#[derive(serde::Deserialize)]
pub struct CloudCredentials {
pub schema: u32,
pub username: String,
pub password: String,
}
pub async fn cloud_credentials(directory: &Path) -> Result<CloudCredentials> {
let path = directory.join("credentials.json");
match fs::read(&path).await {
Ok(bytes) => {
let value: CloudCredentials = serde_json::from_slice(&bytes)
.context("Invalid private File Browser credential record")?;
let suffix = value.username.strip_prefix("archy-").unwrap_or("");
anyhow::ensure!(
value.schema == 1
&& suffix.len() == 32
&& suffix.bytes().all(|c| c.is_ascii_hexdigit())
&& value.password.len() == 64
&& value.password.bytes().all(|c| c.is_ascii_hexdigit()),
"Invalid managed File Browser credentials"
);
Ok(value)
}
Err(error) if error.kind() == std::io::ErrorKind::NotFound => {
// Compatibility during staged upgrades only. Never invent admin/admin
// when a secret is missing; the pre-start provisioner repairs legacy DBs.
let password = fs::read_to_string(directory.join("password"))
.await
.context("File Browser secure Cloud login has not been provisioned")?;
let password = password.trim().to_owned();
anyhow::ensure!(
!password.is_empty() && password != "admin",
"File Browser default credentials must be migrated before Cloud login"
);
Ok(CloudCredentials {
schema: 0,
username: "admin".into(),
password,
})
}
Err(error) => Err(error).context("Cannot read private File Browser credentials"),
}
}
/// Prepare a stopped server using the same helper used by Quadlet and the ISO.
pub async fn prepare_credentials(
paths: &EnsurePaths,
secret_dir: &Path,
image: &str,
runtime: &str,
) -> Result<()> {
let output = tokio::process::Command::new("python3")
.args([
"-c",
include_str!("../../../../scripts/filebrowser-credentials.py"),
"--image",
image,
"--runtime",
runtime,
"--data-dir",
&paths.data_dir.to_string_lossy(),
"--srv-root",
&paths.srv_root.to_string_lossy(),
"--secrets-dir",
&secret_dir.to_string_lossy(),
])
.kill_on_drop(true)
.output()
.await
.context("Running File Browser credential setup")?;
anyhow::ensure!(output.status.success(),
"File Browser secure login setup failed; existing state and private rollback backup retained");
Ok(())
}
#[derive(Debug, Clone)]
pub struct EnsurePaths {
pub srv_root: PathBuf,
@@ -82,7 +160,18 @@ async fn create_dir_all_or_sudo(path: &std::path::Path) -> Result<()> {
async fn write_config_atomically(paths: &EnsurePaths) -> Result<()> {
let tmp = paths.config_path.with_extension("tmp");
match fs::write(&tmp, DEFAULT_CONFIG_JSON).await {
let legacy = paths.data_dir.join("database.db").exists();
let canonical = paths.data_dir.join("filebrowser.db").exists();
anyhow::ensure!(
!(legacy && canonical),
"Multiple File Browser databases need explicit config selection"
);
let config = if legacy {
DEFAULT_CONFIG_JSON.replace("/data/filebrowser.db", "/data/database.db")
} else {
DEFAULT_CONFIG_JSON.to_string()
};
match fs::write(&tmp, &config).await {
Ok(()) => {
fs::rename(&tmp, &paths.config_path)
.await
@@ -99,7 +188,7 @@ async fn write_config_atomically(paths: &EnsurePaths) -> Result<()> {
let script = format!(
"set -eu\ncat > '{}' <<'FILEBROWSERCONF'\n{}FILEBROWSERCONF\n",
shell_quote(&paths.config_path.to_string_lossy()),
DEFAULT_CONFIG_JSON
config
);
let status = host_sudo(&["sh", "-lc", &script])
.await
@@ -117,10 +206,257 @@ fn shell_quote(s: &str) -> String {
s.replace('\'', "'\\''")
}
/// Save a complete purchase without overwriting any existing directory entry.
/// Both host and rootless-namespace paths publish with a no-clobber hard link.
pub async fn save_new_file(dir: &Path, name: &str, bytes: &[u8]) -> Result<PathBuf> {
save_new_file_with(dir, name, bytes, write_via_userns).await
}
fn validate_filename(name: &str) -> Result<()> {
anyhow::ensure!(
!name.is_empty()
&& name != "."
&& name != ".."
&& !name.contains(['/', '\\', '\0'])
&& name.len() <= 255,
"Invalid purchased filename"
);
Ok(())
}
async fn save_new_file_with<F, Fut>(
dir: &Path,
name: &str,
bytes: &[u8],
fallback: F,
) -> Result<PathBuf>
where
F: FnOnce(PathBuf, String, Vec<u8>) -> Fut,
Fut: std::future::Future<Output = Result<PathBuf>>,
{
validate_filename(name)?;
// Never follow a user-created destination directory symlink.
match fs::symlink_metadata(dir).await {
Ok(meta) => anyhow::ensure!(meta.is_dir(), "Files destination is not a directory"),
Err(error) if error.kind() == std::io::ErrorKind::NotFound => {}
Err(error) => return Err(error.into()),
}
save_after_direct_result(
write_direct(dir, name, bytes).await,
dir,
name,
bytes,
fallback,
)
.await
}
async fn save_after_direct_result<F, Fut>(
result: std::io::Result<PathBuf>,
dir: &Path,
name: &str,
bytes: &[u8],
fallback: F,
) -> Result<PathBuf>
where
F: FnOnce(PathBuf, String, Vec<u8>) -> Fut,
Fut: std::future::Future<Output = Result<PathBuf>>,
{
match result {
Ok(path) => Ok(path),
Err(error) if error.kind() == std::io::ErrorKind::PermissionDenied => {
fallback(dir.to_owned(), name.to_owned(), bytes.to_vec())
.await
.context("Saving purchase in Files user namespace")
}
Err(error) => Err(error).context("Saving purchase in Files"),
}
}
fn numbered_name(name: &str, attempt: usize) -> String {
if attempt == 1 {
return name.to_owned();
}
match name.rsplit_once('.') {
Some((stem, extension)) if !stem.is_empty() => format!("{stem} ({attempt}).{extension}"),
_ => format!("{name} ({attempt})"),
}
}
struct PendingFile(PathBuf);
impl Drop for PendingFile {
fn drop(&mut self) {
let _ = std::fs::remove_file(&self.0);
}
}
async fn write_direct(dir: &Path, name: &str, bytes: &[u8]) -> std::io::Result<PathBuf> {
use std::os::unix::fs::PermissionsExt;
use tokio::io::AsyncWriteExt;
fs::create_dir_all(dir).await?;
let temp_path = dir.join(format!(".archy-saving-{}", uuid::Uuid::new_v4()));
let mut file = fs::OpenOptions::new()
.write(true)
.create_new(true)
.mode(0o600)
.open(&temp_path)
.await?;
let temp = PendingFile(temp_path);
file.write_all(bytes).await?;
file.set_permissions(std::fs::Permissions::from_mode(0o644))
.await?;
file.sync_all().await?;
for attempt in 1..=100 {
let target = dir.join(numbered_name(name, attempt));
match fs::hard_link(&temp.0, &target).await {
Ok(()) => return Ok(target),
Err(error) if error.kind() == std::io::ErrorKind::AlreadyExists => continue,
Err(error) => return Err(error),
}
}
Err(std::io::Error::new(
std::io::ErrorKind::AlreadyExists,
"Too many existing copies; purchase cache retained",
))
}
// Positional arguments carry all user-controlled text. mktemp prevents temp-name
// collisions; ln -T refuses files, symlinks and directories, including races.
const WRITE_VIA_USERNS: &str = r#"set -eu
dir=$1
name=$2
expected=$3
[ ! -L "$dir" ] || exit 1
if [ ! -d "$dir" ]; then
mkdir -p -- "$dir"
chown --reference="$(dirname -- "$dir")" -- "$dir"
fi
tmp=$(mktemp "$dir/.archy-saving.XXXXXXXXXX")
trap 'rm -f -- "$tmp"' EXIT HUP INT TERM
cat > "$tmp"
[ "$(wc -c < "$tmp")" -eq "$expected" ] || exit 1
chown --reference="$dir" -- "$tmp"
chmod 0644 -- "$tmp"
sync -f -- "$tmp"
stem=$name
ext=
case "$name" in
*.*) prefix=${name%.*}; if [ -n "$prefix" ]; then stem=$prefix; ext=.${name##*.}; fi ;;
esac
n=1
while [ "$n" -le 100 ]; do
candidate=$name
if [ "$n" -gt 1 ]; then candidate="$stem ($n)$ext"; fi
dst="$dir/$candidate"
if ln -T -- "$tmp" "$dst" 2>/dev/null; then
printf '%s' "$candidate"
exit 0
fi
# A conflict may be a dangling symlink; never follow it or overwrite it.
if [ ! -e "$dst" ] && [ ! -L "$dst" ]; then exit 1; fi
n=$((n + 1))
done
exit 1
"#;
async fn write_via_userns(dir: PathBuf, name: String, bytes: Vec<u8>) -> Result<PathBuf> {
use tokio::io::AsyncWriteExt;
let mut child = tokio::process::Command::new("podman")
.args(["unshare", "sh", "-c", WRITE_VIA_USERNS, "sh"])
.arg(&dir)
.arg(&name)
.arg(bytes.len().to_string())
.kill_on_drop(true)
.stdin(std::process::Stdio::piped())
.stdout(std::process::Stdio::piped())
.stderr(std::process::Stdio::piped())
.spawn()
.context("Starting Files namespace writer")?;
let mut stdin = child.stdin.take().context("Files writer stdin missing")?;
let operation = async {
let fed = stdin.write_all(&bytes).await;
drop(stdin);
let output = child.wait_with_output().await?;
anyhow::ensure!(
output.status.success(),
"Files namespace writer failed: {}",
output.status
);
fed.context("Sending purchase bytes to Files")?;
let chosen =
String::from_utf8(output.stdout).context("Files writer returned an invalid name")?;
validate_filename(&chosen)?;
anyhow::ensure!(
(1..=100).any(|n| numbered_name(&name, n) == chosen),
"Files writer returned an unexpected name"
);
Ok(dir.join(chosen))
};
tokio::time::timeout(std::time::Duration::from_secs(120), operation)
.await
.context("Files namespace writer timed out")?
}
#[cfg(test)]
mod tests {
use super::*;
#[tokio::test]
async fn cloud_credentials_use_unique_record_and_never_default_password() {
let dir = tempfile::tempdir().unwrap();
assert!(cloud_credentials(dir.path()).await.is_err());
fs::write(dir.path().join("password"), "admin")
.await
.unwrap();
assert!(cloud_credentials(dir.path()).await.is_err());
fs::write(dir.path().join("password"), "legacy-unique-password")
.await
.unwrap();
assert_eq!(cloud_credentials(dir.path()).await.unwrap().schema, 0);
let value = serde_json::json!({"schema":1,"username":format!("archy-{}", "a".repeat(32)),"password":"b".repeat(64)});
fs::write(dir.path().join("credentials.json"), value.to_string())
.await
.unwrap();
let loaded = cloud_credentials(dir.path()).await.unwrap();
assert_eq!(loaded.username, value["username"].as_str().unwrap());
assert_eq!(loaded.password, value["password"].as_str().unwrap());
fs::write(dir.path().join("credentials.json"), "{}")
.await
.unwrap();
assert!(
cloud_credentials(dir.path()).await.is_err(),
"damaged managed record must not fall back to old credentials"
);
}
#[tokio::test]
async fn missing_config_preserves_legacy_database_and_refuses_ambiguity() {
let tmp = tempfile::tempdir().unwrap();
let paths = EnsurePaths {
srv_root: tmp.path().join("srv"),
data_dir: tmp.path().join("data"),
config_path: tmp.path().join("data/.filebrowser.json"),
};
fs::create_dir_all(&paths.data_dir).await.unwrap();
fs::write(paths.data_dir.join("database.db"), b"legacy fixture")
.await
.unwrap();
ensure_config(&paths).await.unwrap();
let config: serde_json::Value =
serde_json::from_slice(&fs::read(&paths.config_path).await.unwrap()).unwrap();
assert_eq!(config["database"], "/data/database.db");
fs::remove_file(&paths.config_path).await.unwrap();
fs::write(paths.data_dir.join("filebrowser.db"), b"other fixture")
.await
.unwrap();
assert!(ensure_config(&paths).await.is_err());
assert!(!paths.config_path.exists());
assert_eq!(
fs::read(paths.data_dir.join("database.db")).await.unwrap(),
b"legacy fixture"
);
}
#[tokio::test]
async fn ensure_config_creates_dirs_and_file() {
let tmp = tempfile::TempDir::new().unwrap();
@@ -152,3 +488,231 @@ mod tests {
assert_eq!(second, EnsureOutcome::Unchanged);
}
}
#[cfg(test)]
mod purchase_write_tests {
use super::*;
use std::{
collections::HashSet,
os::unix::fs::{symlink, PermissionsExt},
};
fn no_temps(dir: &Path) {
assert!(std::fs::read_dir(dir).unwrap().all(|e| !e
.unwrap()
.file_name()
.to_string_lossy()
.starts_with(".archy-saving")));
}
#[tokio::test]
async fn direct_write_uses_complete_bytes_and_preserves_originals() {
let dir = tempfile::tempdir().unwrap();
fs::write(dir.path().join("song.mp3"), b"original")
.await
.unwrap();
let target = save_new_file(dir.path(), "song.mp3", b"new").await.unwrap();
assert_eq!(target.file_name().unwrap(), "song (2).mp3");
assert_eq!(fs::read(target).await.unwrap(), b"new");
assert_eq!(
fs::read(dir.path().join("song.mp3")).await.unwrap(),
b"original"
);
no_temps(dir.path());
}
#[tokio::test]
async fn simultaneous_saves_publish_unique_complete_files() {
let dir = tempfile::tempdir().unwrap();
let mut tasks = Vec::new();
for n in 0..24u8 {
let dir = dir.path().to_owned();
tasks.push(tokio::spawn(async move {
let bytes = vec![n; 32768];
let path = save_new_file(&dir, "same.bin", &bytes).await.unwrap();
assert_eq!(fs::read(&path).await.unwrap(), bytes);
path
}));
}
let mut paths = HashSet::new();
for task in tasks {
assert!(paths.insert(task.await.unwrap()));
}
assert_eq!(paths.len(), 24);
no_temps(dir.path());
}
#[tokio::test]
async fn existing_directories_and_dangling_symlinks_are_conflicts() {
let dir = tempfile::tempdir().unwrap();
fs::create_dir(dir.path().join("name")).await.unwrap();
symlink("missing", dir.path().join("name (2)")).unwrap();
let path = save_new_file(dir.path(), "name", b"new").await.unwrap();
assert_eq!(path.file_name().unwrap(), "name (3)");
assert!(dir.path().join("name").is_dir());
assert!(fs::symlink_metadata(dir.path().join("name (2)"))
.await
.unwrap()
.is_symlink());
no_temps(dir.path());
}
#[tokio::test]
async fn invalid_names_and_symlink_destination_are_refused() {
let dir = tempfile::tempdir().unwrap();
for name in [
"",
".",
"..",
"../escape",
"/absolute",
"a/b",
"a\\b",
"a\0b",
] {
assert!(save_new_file(dir.path(), name, b"bytes").await.is_err());
}
let outside = tempfile::tempdir().unwrap();
symlink(outside.path(), dir.path().join("Music")).unwrap();
assert!(save_new_file(&dir.path().join("Music"), "song", b"bytes")
.await
.is_err());
assert_eq!(std::fs::read_dir(outside.path()).unwrap().count(), 0);
}
#[tokio::test]
async fn collision_limit_preserves_all_files_and_cleans_temporary_data() {
let dir = tempfile::tempdir().unwrap();
for n in 1..=100 {
fs::write(dir.path().join(numbered_name("a.txt", n)), b"keep")
.await
.unwrap();
}
assert!(save_new_file(dir.path(), "a.txt", b"new").await.is_err());
for n in 1..=100 {
assert_eq!(
fs::read(dir.path().join(numbered_name("a.txt", n)))
.await
.unwrap(),
b"keep"
);
}
no_temps(dir.path());
}
#[tokio::test]
async fn permission_fallback_is_exercised_without_skipping_as_root() {
let dir = tempfile::tempdir().unwrap();
let result = save_after_direct_result(
Err(std::io::ErrorKind::PermissionDenied.into()),
dir.path(),
"a",
b"abc",
|dir, name, bytes| async move {
assert_eq!(bytes, b"abc");
Ok(dir.join(name))
},
)
.await
.unwrap();
assert_eq!(result, dir.path().join("a"));
assert!(save_after_direct_result(
Err(std::io::ErrorKind::PermissionDenied.into()),
dir.path(),
"a",
b"abc",
|_, _, _| async { anyhow::bail!("namespace unavailable") }
)
.await
.unwrap_err()
.to_string()
.contains("namespace"));
assert!(save_after_direct_result(
Err(std::io::ErrorKind::StorageFull.into()),
dir.path(),
"a",
b"abc",
|_, _, _| async { panic!("disk full must not trigger permission fallback") }
)
.await
.is_err());
}
async fn run_script(
dir: &Path,
name: &str,
bytes: &[u8],
expected: usize,
) -> std::process::Output {
use tokio::io::AsyncWriteExt;
let mut child = tokio::process::Command::new("sh")
.args(["-c", WRITE_VIA_USERNS, "sh"])
.arg(dir)
.arg(name)
.arg(expected.to_string())
.stdin(std::process::Stdio::piped())
.stdout(std::process::Stdio::piped())
.stderr(std::process::Stdio::piped())
.spawn()
.unwrap();
let mut input = child.stdin.take().unwrap();
input.write_all(bytes).await.unwrap();
drop(input);
child.wait_with_output().await.unwrap()
}
#[tokio::test]
async fn namespace_script_preserves_names_bytes_modes_and_existing_entries() {
let dir = tempfile::tempdir().unwrap();
let folder = dir.path().join("Music");
let name = "song ' $() ; #.mp3";
for n in 1..=2 {
let output = run_script(&folder, name, b"abc", 3).await;
assert!(
output.status.success(),
"{}",
String::from_utf8_lossy(&output.stderr)
);
let chosen = String::from_utf8(output.stdout).unwrap();
assert_eq!(chosen, numbered_name(name, n));
let path = folder.join(chosen);
assert_eq!(fs::read(&path).await.unwrap(), b"abc");
assert_eq!(
fs::metadata(path).await.unwrap().permissions().mode() & 0o777,
0o644
);
}
no_temps(&folder);
}
#[tokio::test]
async fn namespace_script_refuses_truncated_input_and_cleans_up() {
let dir = tempfile::tempdir().unwrap();
let output = run_script(dir.path(), "never.bin", b"partial", 100).await;
assert!(!output.status.success());
assert!(!dir.path().join("never.bin").exists());
no_temps(dir.path());
}
#[tokio::test]
async fn namespace_script_does_not_link_inside_existing_directory() {
let dir = tempfile::tempdir().unwrap();
fs::create_dir(dir.path().join("name")).await.unwrap();
symlink("missing", dir.path().join("name (2)")).unwrap();
let output = run_script(dir.path(), "name", b"abc", 3).await;
assert!(output.status.success());
assert_eq!(output.stdout, b"name (3)");
assert_eq!(
std::fs::read_dir(dir.path().join("name")).unwrap().count(),
0
);
no_temps(dir.path());
}
#[test]
fn names_keep_extensions_and_dotfiles() {
assert_eq!(numbered_name("a.tar.gz", 2), "a.tar (2).gz");
assert_eq!(numbered_name(".hidden", 2), ".hidden (2)");
assert_eq!(numbered_name("README", 2), "README (2)");
}
}
@@ -1,15 +1,15 @@
//! Consistent, private snapshots for declaratively opted-in network migrations.
//! Consistent, private snapshots for declaratively opted-in runtime migrations.
use anyhow::{bail, Context, Result};
use archipelago_container::AppManifest;
use std::os::unix::fs::PermissionsExt;
use std::path::{Path, PathBuf};
pub fn enabled(manifest: &AppManifest) -> Result<bool> {
match manifest.app.extensions.get("backup_on_network_change") {
match manifest.app.extensions.get("backup_before_runtime_change") {
None => Ok(false),
Some(value) => value
.as_bool()
.context("backup_on_network_change must be boolean"),
.context("backup_before_runtime_change must be boolean"),
}
}
@@ -24,18 +24,21 @@ fn relative_sources(manifest: &AppManifest, data_dir: &Path) -> Result<Vec<PathB
continue;
}
if volume.volume_type != "bind" {
bail!("network migration backup requires bind-mounted persistent state");
bail!("runtime migration backup requires bind-mounted persistent state");
}
let path = Path::new(&volume.source);
let relative = path
.strip_prefix(data_dir)
.context("network migration state must be inside the node data directory")?;
.context("runtime migration state must be inside the node data directory")?;
if relative.starts_with("migration-backups") {
bail!("migration backup cannot include its own archive directory");
}
if relative.as_os_str().is_empty()
|| relative
.components()
.any(|c| !matches!(c, std::path::Component::Normal(_)))
{
bail!("invalid network migration state path");
bail!("invalid runtime migration state path");
}
sources.push(relative.to_path_buf());
}
@@ -48,7 +51,7 @@ fn relative_sources(manifest: &AppManifest, data_dir: &Path) -> Result<Vec<PathB
}
}
if roots.is_empty() {
bail!("network migration backup has no persistent state mounts");
bail!("runtime migration backup has no persistent state mounts");
}
Ok(roots)
}
@@ -80,11 +83,11 @@ async fn snapshot_with_command(
.file_type()
.is_symlink()
{
bail!("network migration state mount is a symlink; explicit backup required");
bail!("runtime migration state mount is a symlink; explicit backup required");
}
let canonical = tokio::fs::canonicalize(&path).await?;
if !canonical.starts_with(&canonical_root) {
bail!("network migration state path resolves outside node data directory");
bail!("runtime migration state path resolves outside node data directory");
}
}
let root = data_dir.join("migration-backups");
@@ -126,7 +129,7 @@ async fn snapshot_with_command(
tokio::fs::File::open(&partial).await?.sync_all().await?;
tokio::fs::rename(&partial, &archive).await?;
let metadata = serde_json::json!({"app": manifest.app.id, "version": manifest.app.version,
"network": manifest.app.container.network, "sources": sources});
"network": manifest.app.container.network, "capabilities": manifest.app.security.capabilities, "sources": sources});
tokio::fs::write(
dir.join("metadata.json"),
serde_json::to_vec_pretty(&metadata)?,
+1
View File
@@ -13,6 +13,7 @@ pub mod image_policy;
pub mod image_versions;
pub mod lnd;
pub mod migration_backup;
pub mod npm;
pub mod prod_orchestrator;
pub mod quadlet;
pub mod registry;
+115
View File
@@ -0,0 +1,115 @@
//! Preserve NPM's active persistent mounts across installer and runtime paths.
use anyhow::{bail, Context, Result};
use archipelago_container::AppManifest;
use serde::Deserialize;
use std::path::Path;
use std::time::Duration;
#[derive(Debug, Deserialize)]
pub struct Storage {
pub data: String,
pub certificates: String,
}
impl Storage {
pub fn bind_mounts(&self) -> Vec<String> {
vec![
format!("{}:/data", self.data),
format!("{}:/etc/letsencrypt", self.certificates),
]
}
pub fn apply(&self, manifest: &mut AppManifest) -> Result<()> {
for (target, source) in [
("/data", &self.data),
("/etc/letsencrypt", &self.certificates),
] {
let matching: Vec<_> = manifest
.app
.volumes
.iter_mut()
.filter(|volume| volume.target == target)
.collect();
if matching.len() != 1 {
bail!("NPM requires one persistent mount per storage target");
}
let volume = matching.into_iter().next().unwrap();
if volume.volume_type != "bind" {
bail!("NPM persistent state requires bind mounts");
}
volume.source.clone_from(source);
}
Ok(())
}
}
fn parse_storage(bytes: &[u8]) -> Result<Storage> {
let storage: Storage =
serde_json::from_slice(bytes).context("invalid NPM storage resolution")?;
for value in [&storage.data, &storage.certificates] {
if !Path::new(value).is_absolute() || value.chars().any(|c| c.is_control() || c == ':') {
bail!("invalid NPM persistent storage path");
}
}
Ok(storage)
}
pub async fn resolve_storage() -> Result<Storage> {
// Verify live mounts/database before any caller can stop or recreate NPM.
// Remember only validated mount paths so later recreation, after the inspect
// record is removed, still uses the operator's original storage.
let mut command = tokio::process::Command::new("python3");
command
.args([
"-c",
include_str!("../../../../scripts/npm-public-bridge.py"),
"--resolve",
"--remember",
"--prepare-realip",
])
.kill_on_drop(true);
let output = tokio::time::timeout(Duration::from_secs(75), command.output())
.await
.context("NPM storage resolution timed out; existing state preserved")??;
if !output.status.success() {
bail!("NPM storage resolution failed; inspect mount/database ambiguity or permissions before migration");
}
parse_storage(&output.stdout)
}
#[cfg(test)]
mod tests {
use super::*;
#[test]
fn resolved_mounts_preserve_custom_and_legacy_paths() {
for data in [
"/var/lib/archipelago/nginx-proxy-manager/data",
"/srv/operator npm",
] {
let bytes = serde_json::to_vec(
&serde_json::json!({"data": data, "certificates": "/srv/certificates"}),
)
.unwrap();
let storage = parse_storage(&bytes).unwrap();
assert_eq!(
storage.bind_mounts(),
[
format!("{data}:/data"),
"/srv/certificates:/etc/letsencrypt".into(),
]
);
}
}
#[test]
fn invalid_resolution_cannot_become_a_container_mount() {
for data in ["relative", "/srv/data:ro", "/srv/data\nother"] {
let bytes =
serde_json::to_vec(&serde_json::json!({"data": data, "certificates": "/certs"}))
.unwrap();
assert!(parse_storage(&bytes).is_err());
}
assert!(parse_storage(b"{}").is_err());
}
}
@@ -36,6 +36,11 @@ use std::sync::Arc;
use tokio::io::{AsyncReadExt, AsyncWriteExt};
use tokio::sync::{Mutex, RwLock};
/// Refusal before installation has created state or changed any dependency.
#[derive(Debug, thiserror::Error)]
#[error("{0}")]
pub struct InstallPrerequisiteError(pub String);
use crate::config::{Config, ContainerRuntime as ConfigContainerRuntime};
use crate::container::bitcoin_ui;
use crate::container::quadlet;
@@ -87,16 +92,80 @@ fn is_restart_sensitive_app(app_id: &str) -> bool {
fn is_builtin_network_mode(network: &str) -> bool {
matches!(
network,
"host" | "bridge" | "none" | "slirp4netns" | "pasta"
"host"
| "bridge"
| "none"
| "slirp4netns"
| "slirp4netns:allow_host_loopback=true"
| "slirp4netns:allow_host_loopback=true,cidr=169.254.1.0/24"
| "pasta"
)
}
// Only an explicitly selected rootless mode establishes drift. An omitted
// network delegates to Podman and must not recreate unrelated installed apps.
fn rootless_network_mode_drifted(expected: Option<&str>, actual: &str) -> bool {
matches!(expected, Some("slirp4netns" | "pasta"))
&& !actual.trim().is_empty()
&& actual.trim().split(':').next() != expected
let actual = actual.trim();
if actual.is_empty() {
return false;
}
match expected {
Some(
expected @ ("slirp4netns:allow_host_loopback=true"
| "slirp4netns:allow_host_loopback=true,cidr=169.254.1.0/24"),
) => {
let (mode, options) = actual.split_once(':').unwrap_or((actual, ""));
let required = expected.split_once(':').unwrap().1;
mode != "slirp4netns"
|| required.split(',').any(|wanted| {
let key = wanted.split_once('=').unwrap().0;
!options.split(',').any(|option| option == wanted)
|| options.split(',').any(|option| {
option.split_once('=').is_some_and(|(name, _)| name == key)
&& option != wanted
})
})
}
Some(mode @ ("slirp4netns" | "pasta")) => actual.split(':').next() != Some(mode),
_ => false,
}
}
// API-created containers may omit rootless options from HostConfig.NetworkMode.
// generate spec retains them; inspect alone would cause an endless repair loop.
fn rootless_network_from_spec(bytes: &[u8]) -> Option<String> {
let spec: serde_json::Value = serde_json::from_slice(bytes).ok()?;
let mode = spec.get("netns")?.get("nsmode")?.as_str()?;
if !matches!(mode, "slirp4netns" | "pasta") {
return None;
}
let options = spec
.get("network_options")
.and_then(|options| options.get(mode));
match options {
None => Some(mode.to_string()),
Some(options) => {
let options = options
.as_array()?
.iter()
.map(|value| value.as_str())
.collect::<Option<Vec<_>>>()?;
if options.is_empty() {
Some(mode.to_string())
} else {
Some(format!("{mode}:{}", options.join(",")))
}
}
}
}
fn missing_declared_capability(expected: &[String], actual: &[String]) -> bool {
expected.iter().any(|required| {
let required = required.strip_prefix("CAP_").unwrap_or(required);
!actual
.iter()
.any(|cap| cap.strip_prefix("CAP_").unwrap_or(cap) == required)
})
}
fn uses_pasta_network(manifest: &AppManifest) -> bool {
@@ -161,6 +230,11 @@ fn manifest_dependency_app_ids(manifest: &AppManifest) -> Vec<String> {
}
fn host_port_wait_timeout_secs(manifest: &AppManifest) -> u64 {
// First NPM initialization generates keys and migrates its database before
// exposing nginx. Its readiness budget must not depend on the network driver.
if manifest.app.id == "nginx-proxy-manager" {
return 180;
}
if manifest.app.id == "uptime-kuma" {
return 420;
}
@@ -2057,6 +2131,10 @@ impl ProdContainerOrchestrator {
Ok(ReconcileAction::Left(reason))
if mode == ReconcileMode::ExistingOnly
&& reason == "absent"
// companion.rs owns missing UI provisioning/removal.
// Never resurrect an orphan from a stale snapshot.
// Existing UIs still pass through security config repair.
&& !super::companion::is_companion_app(&app_id)
&& (was_running.contains(&compute_container_name(&lm.manifest))
// The durable answer, and the one that does not
// erode. `was_running` only records what was
@@ -2407,6 +2485,49 @@ impl ProdContainerOrchestrator {
}
match status.state {
ContainerState::Running => {
// Legacy runtime path: migrate credentials once without
// recreating accounts or changing operator passwords.
// Quadlet installations receive the same helper through
// the required ExecStartPre drift/restart above.
if app_id == "filebrowser" && !self.use_quadlet_backends && !cfg!(test) {
let secrets = self.secrets_dir.join("filebrowser");
let managed = filebrowser::cloud_credentials(&secrets)
.await
.map(|value| value.schema == 1)
.unwrap_or(false);
if !managed {
if !self.should_attempt_repair(&name).await {
return Ok(ReconcileAction::Left(
"filebrowser-credential-repair-budget-exhausted".into(),
));
}
let unit_managed = self.runtime.cli_name() == "podman"
&& quadlet::unit_exists(&name).await;
let service = format!("{name}.service");
if unit_managed {
quadlet::stop_service(&service).await?;
} else {
self.runtime.stop_container(&name).await?;
}
let prepared = filebrowser::prepare_credentials(
&self.filebrowser_paths,
&secrets,
&status.image,
self.runtime.cli_name(),
)
.await;
// Restore service availability even when setup
// rolled back. Preserve the setup failure itself.
let started = if unit_managed {
quadlet::restart_service(&service).await
} else {
self.runtime.start_container(&name).await
};
prepared?;
started?;
return Ok(ReconcileAction::Started);
}
}
// Zombie guard: podman can report a container "running"
// after its process has died (conmon SIGKILLed in a
// cgroup cascade on archipelago restart, etc.). Such a
@@ -2472,6 +2593,8 @@ impl ProdContainerOrchestrator {
.await
{
tracing::info!(app_id = %app_id, container = %name, "container published-port drift detected — recreating");
self.backup_runtime_change(&name, &resolved_manifest)
.await?;
let _ = self.runtime.stop_container(&name).await;
let _ = self.runtime.remove_container(&name).await;
self.install_fresh(lm).await?;
@@ -2507,7 +2630,8 @@ impl ProdContainerOrchestrator {
return Ok(ReconcileAction::NoOp);
}
tracing::info!(app_id = %app_id, container = %name, "container env drift detected — recreating");
self.backup_network_change(&name, &resolved_manifest).await?;
self.backup_runtime_change(&name, &resolved_manifest)
.await?;
let _ = self.runtime.stop_container(&name).await;
let _ = self.runtime.remove_container(&name).await;
self.install_fresh(lm).await?;
@@ -2564,7 +2688,8 @@ impl ProdContainerOrchestrator {
.await
{
tracing::info!(app_id = %app_id, container = %name, "stopped container env/port drift detected — recreating");
self.backup_network_change(&name, &resolved_manifest).await?;
self.backup_runtime_change(&name, &resolved_manifest)
.await?;
let _ = self.runtime.remove_container(&name).await;
self.install_fresh(lm).await?;
return Ok(ReconcileAction::Installed);
@@ -2621,6 +2746,8 @@ impl ProdContainerOrchestrator {
self.prepare_for_start(&resolved_manifest).await?;
if self.container_env_drifted(&name, &resolved_manifest).await {
tracing::info!(app_id = %app_id, container = %name, "created container env drift detected — recreating");
self.backup_runtime_change(&name, &resolved_manifest)
.await?;
let _ = self.runtime.remove_container(&name).await;
self.install_fresh(lm).await?;
return Ok(ReconcileAction::Installed);
@@ -2947,6 +3074,18 @@ impl ProdContainerOrchestrator {
self.ensure_manifest_files(manifest).await?;
self.apply_data_uid(manifest).await?;
self.run_post_data_uid_hooks(&manifest.app.id).await?;
if manifest.app.id == "filebrowser" && !self.use_quadlet_backends && !cfg!(test) {
let image = manifest.app.container.image.as_deref().ok_or_else(|| {
anyhow::anyhow!("File Browser needs a pinned image for credential setup")
})?;
filebrowser::prepare_credentials(
&self.filebrowser_paths,
&self.secrets_dir.join("filebrowser"),
image,
self.runtime.cli_name(),
)
.await?;
}
Ok(())
}
@@ -3044,6 +3183,11 @@ impl ProdContainerOrchestrator {
container = %name,
"Phase 3.3 migration: replacing pre-Quadlet container with systemd-managed unit"
);
// Resolve active persistent mounts before the old inspect record is
// removed. NPM may use a legacy or operator-selected data directory.
let mut resolved = lm.manifest.clone();
self.resolve_dynamic_env(&mut resolved).await?;
self.backup_runtime_change(name, &resolved).await?;
// Stop+remove the old container record. Volumes survive (host
// bind mounts are not touched by podman rm).
self.runtime
@@ -3053,8 +3197,6 @@ impl ProdContainerOrchestrator {
// Re-render the manifest with dynamic env baked in, then go
// through the same install path a fresh install would.
let mut resolved = lm.manifest.clone();
self.resolve_dynamic_env(&mut resolved).await?;
self.install_via_quadlet(&resolved, name)
.await
.with_context(|| format!("Phase 3.3: re-install {name} via Quadlet"))?;
@@ -3128,11 +3270,18 @@ impl ProdContainerOrchestrator {
quadlet::network_aliases_changed(&old_body, &new_body);
let restart_for_exec_change = quadlet::exec_changed(&old_body, &new_body);
let restart_for_health_change = quadlet::health_cmd_changed(&old_body, &new_body);
let restart_for_security_change = quadlet::security_changed(&old_body, &new_body);
let restart_for_managed_override =
quadlet::redundant_managed_network_override(&unit, &unit_dir)
.await?
.is_some();
let needs_restart = restart_required
|| restart_for_managed_override
|| restart_for_port_change
|| restart_for_network_alias_change
|| restart_for_exec_change
|| restart_for_health_change;
|| restart_for_health_change
|| restart_for_security_change;
// Record the obligation BEFORE replacing the unit. A failed reload or
// restart must not become a no-op on the next tick just because the
// generated file already matches the manifest.
@@ -3140,8 +3289,8 @@ impl ProdContainerOrchestrator {
if pending.is_pending() {
self.ensure_resolved_source_available(lm).await?;
}
if restart_for_network_alias_change {
self.backup_network_change(name, &resolved).await?;
if needs_restart {
self.backup_runtime_change(name, &resolved).await?;
}
let changed = quadlet::write_if_changed(&unit, &unit_dir)
.await
@@ -3503,6 +3652,7 @@ impl ProdContainerOrchestrator {
host_mdns: "test.local".to_string(),
disk_gb: self.test_disk_gb.unwrap_or(1000),
bitcoin_host: "bitcoin-knots".to_string(),
node_identity_pubkeys: String::new(),
};
}
#[allow(unreachable_code)]
@@ -3520,10 +3670,53 @@ impl ProdContainerOrchestrator {
// demand (it costs a podman call) only for manifests that use
// {{BITCOIN_HOST}}, rather than every app on every reconcile.
bitcoin_host: "bitcoin-knots".to_string(),
// Likewise filled on demand, only for manifests that use
// {{NODE_IDENTITY_PUBKEYS}}.
node_identity_pubkeys: String::new(),
}
}
}
/// Nostr public keys of the identities the app identity picker offers, for
/// the `{{NODE_IDENTITY_PUBKEYS}}` derived-env placeholder. The node
/// identity is recognised the way `identity.list` marks `is_node`: by the
/// node's ed25519 public key, read here from `identity/node_key.pub`
/// (the file `server_info.pubkey` is derived from at startup). The record
/// mirrored from the node key has a `node-` id, so the picker's prefix rule
/// hides it either way. The key is only read, never created: a missing or
/// malformed file is an error. An empty set is an error too, so an app is
/// never handed an empty owner list.
async fn node_identity_pubkeys(&self) -> Result<String> {
let node_pubkey_hex = self.node_pubkey_hex().await?;
let pubkeys = crate::identity_manager::IdentityManager::new(&self.data_dir)
.await?
.app_signable_nostr_pubkeys(&node_pubkey_hex)
.await?;
anyhow::ensure!(
!pubkeys.is_empty(),
"no user identity with a Nostr key is available for apps to sign with; \
create one under Web5 \u{2192} Identities"
);
Ok(pubkeys)
}
/// The node's ed25519 public key as lowercase hex, read from
/// `identity/node_key.pub` (raw 32 bytes, as `NodeIdentity` writes it)
/// without the logging or key creation of `NodeIdentity::load_or_create`.
async fn node_pubkey_hex(&self) -> Result<String> {
let path = self.data_dir.join("identity").join("node_key.pub");
let bytes = tokio::fs::read(&path)
.await
.with_context(|| format!("reading the node public key {}", path.display()))?;
anyhow::ensure!(
bytes.len() == 32,
"node public key {} is {} bytes, expected 32",
path.display(),
bytes.len()
);
Ok(hex::encode(bytes))
}
/// Container name of the running Bitcoin node (`bitcoin-knots` or
/// `bitcoin-core`) for the `{{BITCOIN_HOST}}` derived-env placeholder.
/// Defaults to `bitcoin-knots` when none is running (B12).
@@ -3722,6 +3915,11 @@ impl ProdContainerOrchestrator {
}
async fn resolve_dynamic_env(&self, manifest: &mut AppManifest) -> Result<()> {
if manifest.app.id == "nginx-proxy-manager" {
crate::container::npm::resolve_storage()
.await?
.apply(manifest)?;
}
// Idempotency guard: partitioning already ran on this instance.
// Re-running would re-taint against an environment that no longer
// contains the composite entries and silently drop them. Callers
@@ -3791,6 +3989,22 @@ impl ProdContainerOrchestrator {
{
facts.bitcoin_host = self.bitcoin_host().await;
}
// The identities' keys are read only for manifests that template them.
if manifest
.app
.container
.derived_env
.iter()
.any(|e| e.template.contains("{{NODE_IDENTITY_PUBKEYS}}"))
{
facts.node_identity_pubkeys =
self.node_identity_pubkeys().await.with_context(|| {
format!(
"resolving {{{{NODE_IDENTITY_PUBKEYS}}}} for {}",
manifest.app.id
)
})?;
}
let mut env = manifest.app.environment.clone();
env.extend(manifest.app.container.resolve_derived_env(&facts));
if matches!(manifest.app.id.as_str(), "bitcoin-core" | "bitcoin-knots") {
@@ -3870,34 +4084,44 @@ impl ProdContainerOrchestrator {
Ok(())
}
async fn backup_network_change(&self, name: &str, manifest: &AppManifest) -> Result<()> {
async fn backup_runtime_change(&self, name: &str, manifest: &AppManifest) -> Result<()> {
if !crate::container::migration_backup::enabled(manifest)? {
return Ok(());
}
// Only back up an actual network migration, not ordinary env drift.
// A persistent disk/permission failure must not repeatedly stop a
// working old service. Reuse the reconciler's bounded repair budget.
if !self.should_attempt_repair(name).await {
anyhow::bail!("runtime migration retry budget exhausted; original service retained, inspect backup failure before retrying");
}
// Called only before a known runtime change. No app-specific commands;
// opted-in manifests identify their persistent state through bind mounts.
let output = tokio::process::Command::new("podman")
.args(["inspect", name, "--format", "{{.HostConfig.NetworkMode}}"])
.output().await.context("inspect network before migration backup")?;
.output()
.await
.context("inspect network before migration backup")?;
let present = if output.status.success() {
if !rootless_network_mode_drifted(manifest.app.container.network.as_deref(), &String::from_utf8_lossy(&output.stdout)) {
return Ok(());
}
true
} else {
// A crash after gracefully stopping a --rm Quadlet container can
// leave only its data and old unit. Prove absence before snapshotting
// stopped state; an inspect/Podman failure is not proof of absence.
let exists = tokio::process::Command::new("podman")
.args(["container", "exists", name]).status().await?;
.args(["container", "exists", name])
.status()
.await?;
if exists.code() != Some(1) {
anyhow::bail!("cannot verify existing container before network migration backup");
anyhow::bail!("cannot verify existing container before runtime migration backup");
}
false
};
let service = format!("{name}.service");
let managed = quadlet::unit_exists(name).await;
let previous_unit = if managed {
Some(tokio::fs::read(quadlet::unit_dir().await?.join(format!("{name}.container"))).await?)
Some(
tokio::fs::read(quadlet::unit_dir().await?.join(format!("{name}.container")))
.await?,
)
} else {
None
};
@@ -3906,9 +4130,15 @@ impl ProdContainerOrchestrator {
} else if present {
self.runtime.stop_container(name).await?;
}
match crate::container::migration_backup::snapshot(manifest, &self.data_dir, previous_unit.as_deref()).await {
match crate::container::migration_backup::snapshot(
manifest,
&self.data_dir,
previous_unit.as_deref(),
)
.await
{
Ok(archive) => {
tracing::info!(container = %name, backup = %archive.display(), "Persistent state saved before network migration");
tracing::info!(container = %name, backup = %archive.display(), "Persistent state saved before runtime migration");
Ok(())
}
Err(error) => {
@@ -3934,22 +4164,79 @@ impl ProdContainerOrchestrator {
return true;
}
// Generated-unit drift handles managed services; preserve deliberate
// systemd drop-in overrides instead of recreating them every tick.
let unmanaged = !quadlet::unit_exists(name).await;
// Podman's effective bounding set, not Docker-compatible CapAdd (which
// can be empty even when Quadlet supplied capabilities).
if unmanaged && !manifest.app.security.capabilities.is_empty() {
if let Ok(output) = tokio::process::Command::new("podman")
.args(["inspect", name, "--format", "{{json .BoundingCaps}}"])
.output()
.await
{
if output.status.success() {
if let Ok(actual) = serde_json::from_slice::<Vec<String>>(&output.stdout) {
if missing_declared_capability(&manifest.app.security.capabilities, &actual)
{
return true;
}
}
}
}
}
// Quadlet handles declarative Network= drift above. Legacy rootless
// Podman containers need the same convergence when no unit owns them.
if matches!(manifest.app.container.network.as_deref(), Some("slirp4netns" | "pasta")) {
if unmanaged
&& matches!(
manifest.app.container.network.as_deref(),
Some(
"slirp4netns"
| "slirp4netns:allow_host_loopback=true"
| "slirp4netns:allow_host_loopback=true,cidr=169.254.1.0/24"
| "pasta"
)
)
{
if let Ok(output) = tokio::process::Command::new("podman")
.args(["inspect", name, "--format", "{{.HostConfig.NetworkMode}}"])
.output()
.await
{
if output.status.success() && rootless_network_mode_drifted(
if output.status.success() {
let mut actual = String::from_utf8_lossy(&output.stdout).trim().to_string();
if matches!(
manifest.app.container.network.as_deref(),
&String::from_utf8_lossy(&output.stdout),
Some(
"slirp4netns:allow_host_loopback=true"
| "slirp4netns:allow_host_loopback=true,cidr=169.254.1.0/24"
)
) && actual == "slirp4netns"
{
let spec = tokio::process::Command::new("podman")
.args(["generate", "spec", name])
.output()
.await;
actual = match spec {
Ok(spec) if spec.status.success() => {
rootless_network_from_spec(&spec.stdout).unwrap_or_default()
}
_ => String::new(),
};
if actual.is_empty() {
tracing::warn!(app = %name, "Could not verify rootless network options; retaining the existing container");
}
}
if rootless_network_mode_drifted(
manifest.app.container.network.as_deref(),
&actual,
) {
return true;
}
}
}
}
let inspect = tokio::process::Command::new("podman")
.args([
@@ -4517,6 +4804,45 @@ impl ContainerOrchestrator for ProdContainerOrchestrator {
}
async fn install(&self, app_id: &str) -> Result<String> {
let lm = self.loaded(app_id).await?;
// Optional shared-service preconditions are checked before recording
// installation or creating anything. A headless adapter must not claim
// successful installation against a missing indexing stack.
if let Some(required) = lm
.manifest
.app
.extensions
.get("install_prerequisites")
.and_then(|value| value.as_sequence())
{
let present = self
.runtime
.list_containers()
.await
.context("check installed prerequisite services")?;
for id in required.iter().filter_map(|value| value.as_str()) {
let dependency = self.loaded(id).await.map_err(|_| InstallPrerequisiteError(
format!("Required app {id} is unavailable. Refresh the app catalog before installing {}.",
lm.manifest.app.name)))?;
let name = compute_container_name(&dependency.manifest);
if !present
.iter()
.any(|container| container.name.trim_start_matches('/') == name)
{
let owner = crate::app_ops::owning_package(id);
let title = self
.loaded(owner)
.await
.map(|app| app.manifest.app.name)
.unwrap_or(dependency.manifest.app.name);
return Err(InstallPrerequisiteError(format!(
"Install {title} first, then install {}.",
lm.manifest.app.name
))
.into());
}
}
}
{
let mut state = self.state.write().await;
state.disabled.remove(app_id);
@@ -4543,7 +4869,6 @@ impl ContainerOrchestrator for ProdContainerOrchestrator {
// health verification (the .228 "running but unreachable" failure
// mode). Routing every install through here means the orchestrator
// is the one source of truth for what "installed" means.
let lm = self.loaded(app_id).await?;
let name = compute_container_name(&lm.manifest);
// ensure_running takes the per-app lock itself; release the install
// path lock first if we hold one (we don't — install is the entry
@@ -4772,6 +5097,26 @@ impl ContainerOrchestrator for ProdContainerOrchestrator {
/// here (production volumes live under `/var/lib/archipelago` — removal is a
/// separate operation owned by the data layer, not this orchestrator).
async fn remove(&self, app_id: &str, _preserve_data: bool) -> Result<()> {
// A removed catalog entry must remain uninstallable. The RPC caller
// still removes legacy containers and persists uninstall intent after
// confirming they are gone; do not block it on a missing manifest.
if !self.state.read().await.manifests.contains_key(app_id) {
anyhow::ensure!(
!app_id.is_empty()
&& app_id.len() <= 128
&& app_id
.bytes()
.all(|c| c.is_ascii_alphanumeric() || matches!(c, b'-' | b'_')),
"Invalid app id"
);
let lock = self.app_lock(app_id).await;
let _guard = lock.lock().await;
for name in [app_id.to_string(), format!("archy-{app_id}")] {
self.remove_quadlet_unit_if_present(&name).await?;
}
self.state.write().await.disabled.insert(app_id.to_string());
return Ok(());
}
let lm = self.loaded(app_id).await?;
let lock = self.app_lock(app_id).await;
let _guard = lock.lock().await;
@@ -4993,12 +5338,133 @@ mod tests {
/// recovered when its siblings have live containers (the stack is
/// installed), and left alone when the whole stack is gone or the app
/// is not a stack member at all.
#[tokio::test]
async fn gitea_fresh_url_seed_preserves_operator_config_and_reports_write_failure() {
let manifest =
AppManifest::parse(include_str!("../../../../apps/gitea/manifest.yml")).unwrap();
let seed = &manifest.app.files[0];
assert!(!seed.overwrite);
let content = seed.content.replace("{{HOST_IP}}", "192.0.2.1");
assert!(content.contains("ROOT_URL = http://192.0.2.1:3001/"));
let dir = tempfile::tempdir().unwrap();
let path = dir.path().join("fresh/app.ini");
assert_eq!(
ensure_rendered_file(path.to_str().unwrap(), &content, seed.overwrite)
.await
.unwrap(),
HookOutcome::Rewritten
);
assert!(tokio::fs::read_to_string(&path)
.await
.unwrap()
.contains("ROOT_URL"));
let custom =
"[server]\nROOT_URL = https://git.example.test/\n[database]\nDB_TYPE = postgres\n";
tokio::fs::write(&path, custom).await.unwrap();
assert_eq!(
ensure_rendered_file(path.to_str().unwrap(), &content, seed.overwrite)
.await
.unwrap(),
HookOutcome::Unchanged
);
assert_eq!(tokio::fs::read_to_string(&path).await.unwrap(), custom);
let impossible = path.join("app.ini");
assert!(
ensure_rendered_file(impossible.to_str().unwrap(), &content, seed.overwrite)
.await
.is_err()
);
}
#[test]
fn ssh_sandbox_capability_repair_uses_bounding_set_and_preserves_extra_overrides() {
let required = vec!["CHOWN".into(), "SYS_CHROOT".into()];
assert!(missing_declared_capability(
&required,
&["CAP_CHOWN".into()]
));
assert!(!missing_declared_capability(
&required,
&["CAP_CHOWN".into(), "CAP_SYS_CHROOT".into()]
));
assert!(!missing_declared_capability(
&required,
&["CHOWN".into(), "SYS_CHROOT".into(), "CAP_KILL".into()]
));
}
#[test]
fn npm_legacy_gateway_converges_and_rejects_conflicting_network_options() {
let expected = Some("slirp4netns:allow_host_loopback=true,cidr=169.254.1.0/24");
assert!(is_builtin_network_mode(expected.unwrap()));
for actual in [
"pasta",
"slirp4netns:allow_host_loopback=true",
"slirp4netns:allow_host_loopback=true,cidr=10.0.2.0/24",
"slirp4netns:allow_host_loopback=true,cidr=169.254.1.0/24,cidr=10.0.2.0/24",
] {
assert!(rootless_network_mode_drifted(expected, actual), "{actual}");
}
let actual = "slirp4netns:cidr=169.254.1.0/24,allow_host_loopback=true,mtu=65520";
assert!(!rootless_network_mode_drifted(expected, actual));
let spec = br#"{"netns":{"nsmode":"slirp4netns"},"network_options":{"slirp4netns":["cidr=169.254.1.0/24","allow_host_loopback=true"]}}"#;
assert!(!rootless_network_mode_drifted(
expected,
&rootless_network_from_spec(spec).unwrap()
));
}
#[test]
fn npm_initialization_budget_survives_network_migration() {
let mut manifest = AppManifest::parse(include_str!(
"../../../../apps/nginx-proxy-manager/manifest.yml"
))
.unwrap();
for network in ["pasta", "slirp4netns:allow_host_loopback=true"] {
manifest.app.container.network = Some(network.to_string());
assert_eq!(host_port_wait_timeout_secs(&manifest), 180);
}
}
#[test]
fn api_created_rootless_options_do_not_cause_repeated_recreation() {
let expected = Some("slirp4netns:allow_host_loopback=true");
let spec = br#"{"netns":{"nsmode":"slirp4netns"},"network_options":{"slirp4netns":["allow_host_loopback=true"]}}"#;
let actual = rootless_network_from_spec(spec).unwrap();
assert!(!rootless_network_mode_drifted(expected, &actual));
let plain = rootless_network_from_spec(br#"{"netns":{"nsmode":"slirp4netns"}}"#).unwrap();
assert!(rootless_network_mode_drifted(expected, &plain));
assert!(rootless_network_from_spec(b"invalid").is_none());
}
#[test]
fn explicit_rootless_network_change_converges_without_guessing_defaults() {
let npm = Some("slirp4netns:allow_host_loopback=true");
assert!(is_builtin_network_mode(npm.unwrap()));
for actual in [
"pasta",
"bridge",
"slirp4netns",
"slirp4netns:allow_host_loopback=false",
] {
assert!(rootless_network_mode_drifted(npm, actual), "{actual}");
}
for actual in [
"slirp4netns:allow_host_loopback=true",
"slirp4netns:mtu=65520,allow_host_loopback=true",
] {
assert!(!rootless_network_mode_drifted(npm, actual), "{actual}");
}
assert!(rootless_network_mode_drifted(Some("slirp4netns"), "pasta"));
assert!(rootless_network_mode_drifted(Some("slirp4netns"), "bridge"));
assert!(!rootless_network_mode_drifted(Some("slirp4netns"), "slirp4netns"));
assert!(!rootless_network_mode_drifted(Some("slirp4netns"), "slirp4netns:allow_host_loopback=true"));
assert!(!rootless_network_mode_drifted(
Some("slirp4netns"),
"slirp4netns"
));
assert!(!rootless_network_mode_drifted(
Some("slirp4netns"),
"slirp4netns:allow_host_loopback=true"
));
assert!(!rootless_network_mode_drifted(None, "pasta"));
assert!(!rootless_network_mode_drifted(Some("slirp4netns"), ""));
assert!(!rootless_network_mode_drifted(Some("archy-net"), "bridge"));
@@ -5685,6 +6151,38 @@ app:
orch
}
#[tokio::test]
async fn missing_install_prerequisite_refuses_without_inventory_or_container_mutation() {
let rt = Arc::new(MockRuntime::default());
let orch = orch_with(rt.clone()).await;
let mut app = pull_manifest("indexer-adapter", "docker.io/library/alpine:3.20");
app.app.extensions.insert(
"install_prerequisites".into(),
serde_yaml::to_value(vec!["shared-index"]).unwrap(),
);
orch.insert_manifest_for_test(app, PathBuf::from("/tmp"))
.await;
orch.insert_manifest_for_test(
pull_manifest("shared-index", "index:1"),
PathBuf::from("/tmp"),
)
.await;
let error = orch.install("indexer-adapter").await.unwrap_err();
assert!(error.downcast_ref::<InstallPrerequisiteError>().is_some());
assert!(!crate::crash_recovery::load_installed_apps(&orch.data_dir)
.await
.contains("indexer-adapter"));
assert_eq!(rt.calls(), vec!["list_containers"]);
// An installed prerequisite satisfies the guard; it is never recreated
// or reconfigured as part of installing this adapter.
rt.set_state("shared-index", ContainerState::Running);
orch.install("indexer-adapter").await.unwrap();
assert!(!rt
.calls()
.iter()
.any(|c| c.starts_with("create_container:shared-index")));
}
fn pull_manifest_with_dynamic_env(id: &str, image: &str) -> AppManifest {
let yaml = format!(
"app:\n id: {id}\n name: {id}\n version: 1.0.0\n container:\n image: {image}\n derived_env:\n - key: FM_API_URL\n template: \"ws://{{{{HOST_MDNS}}}}:8174\"\n secret_env:\n - key: FM_BITCOIND_PASSWORD\n secret_file: bitcoin-rpc-password\n environment:\n - STATIC=1\n"
@@ -5924,6 +6422,143 @@ app:
}
}
const NODE_IDENTITY_PUBKEYS_YAML: &str = "app:\n id: wildbloom-node\n name: wildbloom-node\n version: 1.0.0\n container:\n image: x:1\n derived_env:\n - key: WILDBLOOM_ALLOW_PUBKEYS\n template: \"{{NODE_IDENTITY_PUBKEYS}}\"\n";
/// Writes `pubkey_hex` as the node public key, in `NodeIdentity`'s format.
async fn write_node_pubkey(orch: &ProdContainerOrchestrator, pubkey_hex: &str) {
let dir = orch.data_dir().join("identity");
tokio::fs::create_dir_all(&dir).await.unwrap();
tokio::fs::write(dir.join("node_key.pub"), hex::decode(pubkey_hex).unwrap())
.await
.unwrap();
}
#[tokio::test]
async fn node_identity_pubkeys_placeholder_renders_the_signable_identities() {
// The owners must be exactly the identities the app signer offers:
// the user identities, never the node's own identity.
let rt = Arc::new(MockRuntime::default());
let orch = orch_with(rt).await;
let mgr = crate::identity_manager::IdentityManager::new(orch.data_dir())
.await
.unwrap();
let mut expected = Vec::new();
for name in ["Personal", "Business"] {
let r = mgr
.create(
name.to_string(),
crate::identity_manager::IdentityPurpose::Personal,
)
.await
.unwrap();
expected.push(r.nostr_pubkey.unwrap().to_ascii_lowercase());
}
expected.sort();
// The node key is held by an identity with a uuid id and an ordinary
// name, so only the `is_node` match, through the key read from
// node_key.pub, can keep it out.
let laptop = mgr
.create(
"Laptop".to_string(),
crate::identity_manager::IdentityPurpose::Personal,
)
.await
.unwrap();
assert!(!laptop.id.starts_with("node-"));
let laptop_nostr = laptop.nostr_pubkey.clone().unwrap();
write_node_pubkey(&orch, &laptop.pubkey_hex).await;
let mut manifest = AppManifest::parse(NODE_IDENTITY_PUBKEYS_YAML).unwrap();
orch.resolve_dynamic_env(&mut manifest).await.unwrap();
let env = &manifest.app.environment;
let want = format!("WILDBLOOM_ALLOW_PUBKEYS={}", expected.join(","));
assert!(env.iter().any(|e| e == &want), "env was {env:?}");
assert!(
!env.iter().any(|e| e.contains(&laptop_nostr)),
"node identity leaked into {env:?}"
);
}
#[tokio::test]
async fn node_identity_pubkeys_placeholder_refuses_an_empty_set() {
let rt = Arc::new(MockRuntime::default());
let orch = orch_with(rt).await;
// A node key with only the node's own identity: nothing is signable.
let node = crate::identity::NodeIdentity::load_or_create(&orch.data_dir().join("identity"))
.await
.unwrap();
crate::identity_manager::IdentityManager::new(orch.data_dir())
.await
.unwrap()
.create_from_signing_key(
"Node".to_string(),
crate::identity_manager::IdentityPurpose::Personal,
node.signing_key().clone(),
)
.await
.unwrap();
let mut manifest = AppManifest::parse(NODE_IDENTITY_PUBKEYS_YAML).unwrap();
let err = orch.resolve_dynamic_env(&mut manifest).await.unwrap_err();
let msg = format!("{err:#}");
assert!(
msg.contains("NODE_IDENTITY_PUBKEYS"),
"unexpected error: {msg}"
);
assert!(msg.contains("no user identity"), "unexpected error: {msg}");
assert!(
!manifest
.app
.environment
.iter()
.any(|e| e.starts_with("WILDBLOOM_ALLOW_PUBKEYS=")),
"an empty owner list must never render"
);
}
#[tokio::test]
async fn node_identity_pubkeys_placeholder_needs_the_node_key_and_never_creates_it() {
let rt = Arc::new(MockRuntime::default());
let orch = orch_with(rt).await;
crate::identity_manager::IdentityManager::new(orch.data_dir())
.await
.unwrap()
.create(
"Personal".to_string(),
crate::identity_manager::IdentityPurpose::Personal,
)
.await
.unwrap();
let identity_dir = orch.data_dir().join("identity");
let mut manifest = AppManifest::parse(NODE_IDENTITY_PUBKEYS_YAML).unwrap();
let err = orch.resolve_dynamic_env(&mut manifest).await.unwrap_err();
assert!(
format!("{err:#}").contains("node public key"),
"unexpected error: {err:#}"
);
assert!(
!identity_dir.join("node_key").exists(),
"a node key was created"
);
assert!(!identity_dir.join("node_key.pub").exists());
// A malformed key file is refused, not reinterpreted.
tokio::fs::create_dir_all(&identity_dir).await.unwrap();
tokio::fs::write(identity_dir.join("node_key.pub"), "ab".repeat(32))
.await
.unwrap();
let err = orch.resolve_dynamic_env(&mut manifest).await.unwrap_err();
assert!(
format!("{err:#}").contains("expected 32"),
"unexpected error: {err:#}"
);
assert!(!manifest
.app
.environment
.iter()
.any(|e| e.starts_with("WILDBLOOM_ALLOW_PUBKEYS=")));
}
/// A fedimint-gateway manifest shaped like the real one: a bcrypt
/// generated secret plus a secret_env that reads it, which is what makes
/// the credential participate in secret_env_hash.
@@ -7027,6 +7662,52 @@ app:
assert!(!calls.iter().any(|c| c.starts_with("start_container:")));
}
#[tokio::test]
async fn reconcile_existing_does_not_resurrect_orphaned_companions() {
let rt = Arc::new(MockRuntime::default());
let mut orch = orch_with(rt.clone()).await;
orch.set_disk_gb_for_test(500);
let companions = [
"bitcoin-ui",
"electrs-ui",
"lnd-ui",
"fedimint-ui",
"cuprate-ui",
];
let mut names = Vec::new();
for id in companions {
let manifest = pull_manifest(id, "localhost/companion:local");
names.push(compute_container_name(&manifest));
orch.insert_manifest_for_test(manifest, PathBuf::from("/tmp/companion"))
.await;
}
let refs: Vec<&str> = names.iter().map(String::as_str).collect();
crate::crash_recovery::save_container_snapshot_for_test(&orch.data_dir, &refs).await;
// Repeated passes must leave lifecycle ownership with companion.rs.
for _ in 0..3 {
let report = orch.reconcile_existing().await;
assert_eq!(report.actions.len(), companions.len());
assert!(report
.actions
.iter()
.all(|(_, action)| *action == ReconcileAction::Left("absent".into())));
assert!(report.failures.is_empty());
}
let calls = rt.calls();
for operation in [
"pull_image:",
"create_container:",
"start_container:",
"stop_container:",
"remove_container:",
] {
assert!(
!calls.iter().any(|call| call.starts_with(operation)),
"{calls:?}"
);
}
}
#[tokio::test]
async fn reconcile_existing_self_heals_missing_optional_installed_app() {
// A non-baseline app (gitea) self-heals ONLY with installation
@@ -7187,6 +7868,19 @@ app:
);
}
#[tokio::test]
async fn removed_catalog_entry_does_not_block_legacy_uninstall() {
let rt = Arc::new(MockRuntime::default());
let orch = orch_with(rt.clone()).await;
orch.remove("cryptpad", true).await.unwrap();
assert!(orch.state.read().await.disabled.contains("cryptpad"));
assert!(
rt.calls().is_empty(),
"legacy RPC teardown owns the actual containers"
);
assert!(orch.remove("../other", true).await.is_err());
}
#[tokio::test]
async fn remove_disables_manifest_so_reconcile_does_not_reinstall() {
let rt = Arc::new(MockRuntime::default());
+275 -19
View File
@@ -68,6 +68,10 @@ pub enum NetworkMode {
/// Rootless slirp4netns networking. Podman rejects network aliases with
/// this mode, so render only Network=slirp4netns.
Slirp4netns,
/// Permit explicit host aliases as well as LAN upstreams for NPM.
Slirp4netnsHostLoopback,
/// Preserve existing NPM upstreams using pasta's former host gateway.
Slirp4netnsLegacyGateway,
/// Rootless pasta networking. This is more reliable than slirp4netns for
/// host port forwarding on long-running web apps.
Pasta,
@@ -229,6 +233,15 @@ impl QuadletUnit {
NetworkMode::Host => {
let _ = writeln!(s, "Network=host");
}
NetworkMode::Slirp4netnsHostLoopback => {
let _ = writeln!(s, "Network=slirp4netns:allow_host_loopback=true");
}
NetworkMode::Slirp4netnsLegacyGateway => {
let _ = writeln!(
s,
"Network=slirp4netns:allow_host_loopback=true,cidr=169.254.1.0/24"
);
}
NetworkMode::Slirp4netns => {
let _ = writeln!(s, "Network=slirp4netns");
}
@@ -348,6 +361,26 @@ impl QuadletUnit {
}
let _ = writeln!(s);
let _ = writeln!(s, "[Service]");
if self.name == "filebrowser" {
// Runs while the managed server is stopped, before it can expose
// a default account. The helper verifies real login and root access.
let mut argv = vec![
"/usr/bin/python3".to_string(),
"/opt/archipelago/scripts/filebrowser-credentials.py".to_string(),
"--image".to_string(),
self.image.clone(),
];
for (target, flag) in [("/data", "--data-dir"), ("/srv", "--srv-root")] {
if let Some(mount) = self
.bind_mounts
.iter()
.find(|m| m.container == Path::new(target))
{
argv.extend([flag.to_string(), mount.host.display().to_string()]);
}
}
let _ = writeln!(s, "ExecStartPre={}", shell_join(&argv));
}
// Dependency-gated apps may legitimately keep their container entrypoint
// in a wait loop before the actual daemon binds ports. Fedimint waits
// for Bitcoin IBD to finish before execing fedimintd; systemd's default
@@ -390,7 +423,10 @@ fn shell_join(parts: &[String]) -> String {
.iter()
.map(|p| {
let p = p.replace(['\r', '\n'], " ").replace('%', "%%");
if p.is_empty() || p.chars().any(|c| c.is_whitespace() || "\"\\$`".contains(c)) {
if p.is_empty()
|| p.chars()
.any(|c| c.is_whitespace() || "'\"\\$`".contains(c))
{
let escaped = p
.replace('\\', "\\\\")
.replace('"', "\\\"")
@@ -410,7 +446,7 @@ fn quote_environment(env: &str) -> String {
if env.is_empty()
|| env
.chars()
.any(|c| c.is_whitespace() || "\"\\$`".contains(c))
.any(|c| c.is_whitespace() || "'\"\\$`".contains(c))
{
let escaped = env
.replace('\\', "\\\\")
@@ -444,6 +480,12 @@ impl QuadletUnit {
other if !other.is_empty() && other != "isolated" => NetworkMode::Bridge(other.into()),
_ => match app.container.network.as_deref() {
Some("slirp4netns") => NetworkMode::Slirp4netns,
Some("slirp4netns:allow_host_loopback=true") => {
NetworkMode::Slirp4netnsHostLoopback
}
Some("slirp4netns:allow_host_loopback=true,cidr=169.254.1.0/24") => {
NetworkMode::Slirp4netnsLegacyGateway
}
Some("pasta") => NetworkMode::Pasta,
Some(n) if !n.is_empty() && n != "host" => NetworkMode::Bridge(n.into()),
_ => NetworkMode::Default,
@@ -710,18 +752,61 @@ pub async fn unit_dir() -> Result<PathBuf> {
Ok(dir)
}
/// Atomically write `unit` into `dir/<name>.container` if the bytes
/// differ from what's already there. Returns true if the file changed.
/// The early same-node Portainer repair used a managed Quadlet drop-in. Once
/// the manifest supplies slirp, the two Network= entries are additive and
/// Podman rejects startup. Retire only that exact redundant managed override;
/// arbitrary operator settings must survive reconciliation.
pub async fn redundant_managed_network_override(
unit: &QuadletUnit,
dir: &Path,
) -> Result<Option<PathBuf>> {
if unit.name != "portainer" || !matches!(unit.network, NetworkMode::Slirp4netns) {
return Ok(None);
}
let path = dir.join("portainer.container.d/archy-same-node-network.conf");
let body = match fs::read_to_string(&path).await {
Ok(body) => body,
Err(error) if error.kind() == std::io::ErrorKind::NotFound => return Ok(None),
Err(error) => return Err(error).context("read managed Portainer network override"),
};
let lines: Vec<&str> = body
.lines()
.map(str::trim)
.filter(|line| !line.is_empty() && !line.starts_with(['#', ';']))
.collect();
Ok((lines == ["[Container]", "Network=slirp4netns"]).then_some(path))
}
async fn retire_managed_network_override(path: &Path) -> Result<()> {
let backup = path.with_extension("conf.retired");
match fs::hard_link(path, &backup).await {
Ok(()) => {}
Err(error) if error.kind() == std::io::ErrorKind::AlreadyExists => {
anyhow::ensure!(
fs::read(path).await? == fs::read(&backup).await?,
"Existing Portainer override backup differs; preserve both for operator review"
);
}
Err(error) => return Err(error).context("back up managed Portainer network override"),
}
fs::remove_file(path)
.await
.context("retire redundant Portainer network override")?;
tracing::info!("Retired redundant managed Portainer network override; backup retained");
Ok(())
}
/// Atomically write the manifest unit and retire known redundant managed
/// overrides. Returns true whenever systemd needs a daemon-reload.
pub async fn write_if_changed(unit: &QuadletUnit, dir: &Path) -> Result<bool> {
let path = dir.join(unit.unit_filename());
let new_bytes = unit.render();
if let Ok(old) = fs::read_to_string(&path).await {
if old == new_bytes {
return Ok(false);
}
}
let redundant = redundant_managed_network_override(unit, dir).await?;
let changed = fs::read_to_string(&path)
.await
.map(|old| old != new_bytes)
.unwrap_or(true);
if changed {
fs::create_dir_all(dir)
.await
.with_context(|| format!("create_dir_all {}", dir.display()))?;
@@ -732,7 +817,11 @@ pub async fn write_if_changed(unit: &QuadletUnit, dir: &Path) -> Result<bool> {
fs::rename(&tmp, &path)
.await
.with_context(|| format!("rename {} -> {}", tmp.display(), path.display()))?;
Ok(true)
}
if let Some(override_path) = &redundant {
retire_managed_network_override(override_path).await?;
}
Ok(changed || redundant.is_some())
}
/// Reload the user systemd manager. Required after any quadlet write
@@ -991,6 +1080,24 @@ pub fn publish_ports_changed(old_body: &str, new_body: &str) -> bool {
old_ports != new_ports
}
pub fn security_changed(old_body: &str, new_body: &str) -> bool {
[
"AddCapability=",
"DropCapability=",
"NoNewPrivileges=",
"ReadOnly=",
"User=",
]
.iter()
.any(|directive| {
let mut old = directive_values(old_body, directive);
let mut new = directive_values(new_body, directive);
old.sort();
new.sort();
old != new
})
}
pub fn network_aliases_changed(old_body: &str, new_body: &str) -> bool {
let old_network = directive_values(old_body, "Network=");
let new_network = directive_values(new_body, "Network=");
@@ -1003,7 +1110,8 @@ pub fn exec_changed(old_body: &str, new_body: &str) -> bool {
// Entrypoint= and Exec= together define what the container runs, so a drift
// in either must recreate the container (e.g. when this renderer first
// splits a folded `Exec=sh -lc ...` into `Entrypoint=sh` + `Exec=-lc ...`).
directive_values(old_body, "Exec=") != directive_values(new_body, "Exec=")
directive_values(old_body, "ExecStartPre=") != directive_values(new_body, "ExecStartPre=")
|| directive_values(old_body, "Exec=") != directive_values(new_body, "Exec=")
|| directive_values(old_body, "Entrypoint=") != directive_values(new_body, "Entrypoint=")
}
@@ -1074,6 +1182,28 @@ pub async fn is_active(service: &str) -> bool {
#[cfg(test)]
mod tests {
#[test]
fn filebrowser_prestart_credentials_are_a_required_runtime_change() {
let unit = super::QuadletUnit {
name: "filebrowser".into(),
image: "registry.example/filebrowser:v2.63.23".into(),
..Default::default()
};
let rendered = unit.render();
assert!(rendered.contains("ExecStartPre=/usr/bin/python3 /opt/archipelago/scripts/filebrowser-credentials.py --image registry.example/filebrowser:v2.63.23"));
let old = rendered
.lines()
.filter(|line| !line.starts_with("ExecStartPre="))
.collect::<Vec<_>>()
.join("\n");
assert!(super::exec_changed(&old, &rendered));
assert!(!super::exec_changed(&rendered, &rendered));
let other = super::QuadletUnit {
name: "other-app".into(),
..unit
};
assert!(!other.render().contains("filebrowser-credentials.py"));
}
use super::*;
use tempfile::tempdir;
@@ -1376,6 +1506,18 @@ app:
);
}
#[test]
fn apostrophes_survive_quadlet_argument_and_environment_parsing() {
// A whitespace-free Node script reproduced this in a real Quadlet:
// unquoted apostrophes were consumed by the parser, changing JS strings
// into identifiers and preventing the app from starting.
assert_eq!(
shell_join(&["require('http')".into()]),
"\"require('http')\""
);
assert_eq!(quote_environment("NAME=O'Brien"), "\"NAME=O'Brien\"");
}
#[test]
fn quote_environment_quotes_values_with_spaces() {
assert_eq!(
@@ -1590,9 +1732,7 @@ app:
#[test]
fn portainer_catalog_network_repairs_same_node_routing_without_exposing_backend() {
let manifest = AppManifest::parse(include_str!(
"../../../../apps/portainer/manifest.yml"
))
let manifest = AppManifest::parse(include_str!("../../../../apps/portainer/manifest.yml"))
.expect("shipped Portainer manifest must parse");
let new = QuadletUnit::from_manifest(&manifest, "portainer").render();
assert!(new.contains("Network=slirp4netns\n"));
@@ -1631,6 +1771,24 @@ app:
assert!(!s.contains("--network-alias"));
}
#[test]
fn npm_network_preserves_same_node_upstreams_without_aliases() {
let m = AppManifest::parse(include_str!(
"../../../../apps/nginx-proxy-manager/manifest.yml"
))
.unwrap();
let rendered = QuadletUnit::from_manifest(&m, "nginx-proxy-manager").render();
assert_eq!(
rendered
.lines()
.filter(|line| line.starts_with("Network="))
.collect::<Vec<_>>(),
vec!["Network=slirp4netns:allow_host_loopback=true,cidr=169.254.1.0/24"]
);
assert!(!rendered.contains("NetworkAlias="));
assert!(!rendered.contains("--network-alias"));
}
#[test]
fn from_manifest_pasta_omits_network_alias() {
let yaml = r#"
@@ -1960,6 +2118,80 @@ app:
assert!(!network_aliases_changed(new, new));
}
#[tokio::test]
async fn redundant_portainer_override_is_backed_up_and_retired_even_when_base_matches() {
let dir = tempfile::tempdir().unwrap();
let manifest =
AppManifest::parse(include_str!("../../../../apps/portainer/manifest.yml")).unwrap();
let unit = QuadletUnit::from_manifest(&manifest, "portainer");
assert!(write_if_changed(&unit, dir.path()).await.unwrap());
let path = dir
.path()
.join("portainer.container.d/archy-same-node-network.conf");
fs::create_dir_all(path.parent().unwrap()).await.unwrap();
let old = "[Container]\nNetwork=slirp4netns\n";
fs::write(&path, old).await.unwrap();
assert!(redundant_managed_network_override(&unit, dir.path())
.await
.unwrap()
.is_some());
assert!(write_if_changed(&unit, dir.path()).await.unwrap());
assert!(!path.exists());
assert_eq!(
fs::read_to_string(path.with_extension("conf.retired"))
.await
.unwrap(),
old
);
assert!(!write_if_changed(&unit, dir.path()).await.unwrap());
assert_eq!(
fs::read_to_string(dir.path().join("portainer.container"))
.await
.unwrap()
.matches("Network=slirp4netns")
.count(),
1
);
}
#[tokio::test]
async fn network_override_migration_preserves_operator_customizations_and_failed_backups() {
let dir = tempfile::tempdir().unwrap();
let manifest =
AppManifest::parse(include_str!("../../../../apps/portainer/manifest.yml")).unwrap();
let mut unit = QuadletUnit::from_manifest(&manifest, "portainer");
let path = dir
.path()
.join("portainer.container.d/archy-same-node-network.conf");
fs::create_dir_all(path.parent().unwrap()).await.unwrap();
for custom in [
"[Container]\nNetwork=custom-net\n",
"[Container]\nNetwork=slirp4netns\nEnvironment=OPERATOR_SETTING=1\n",
] {
fs::write(&path, custom).await.unwrap();
assert!(redundant_managed_network_override(&unit, dir.path())
.await
.unwrap()
.is_none());
write_if_changed(&unit, dir.path()).await.unwrap();
assert_eq!(fs::read_to_string(&path).await.unwrap(), custom);
}
fs::write(&path, "[Container]\nNetwork=slirp4netns\n")
.await
.unwrap();
unit.network = NetworkMode::Pasta;
assert!(redundant_managed_network_override(&unit, dir.path())
.await
.unwrap()
.is_none());
unit.network = NetworkMode::Slirp4netns;
fs::write(path.with_extension("conf.retired"), "different backup")
.await
.unwrap();
assert!(write_if_changed(&unit, dir.path()).await.is_err());
assert!(path.exists(), "failure must preserve the active override");
}
#[tokio::test]
async fn failed_runtime_change_remains_pending_when_unit_already_matches() {
let dir = tempfile::tempdir().unwrap();
@@ -1973,9 +2205,15 @@ app:
// Simulate systemctl failure or daemon interruption after unit rewrite.
drop(pending);
let retry = RestartObligation::prepare(&unit, false).await.unwrap();
assert!(retry.is_pending(), "matching unit must not discard failed restart");
assert!(
retry.is_pending(),
"matching unit must not discard failed restart"
);
retry.complete().await.unwrap();
assert!(!RestartObligation::prepare(&unit, false).await.unwrap().is_pending());
assert!(!RestartObligation::prepare(&unit, false)
.await
.unwrap()
.is_pending());
}
#[tokio::test]
@@ -1985,10 +2223,28 @@ app:
assert!(RestartObligation::prepare(&missing, true).await.is_err());
let unit = dir.path().join("app.container");
let pending = RestartObligation::prepare(&unit, true).await.unwrap();
tokio::fs::remove_file(unit.with_extension("restart-pending")).await.unwrap();
tokio::fs::remove_file(unit.with_extension("restart-pending"))
.await
.unwrap();
assert!(pending.complete().await.is_err());
}
#[test]
fn gitea_ssh_sandbox_capability_is_applied_as_a_runtime_change() {
let manifest =
AppManifest::parse(include_str!("../../../../apps/gitea/manifest.yml")).unwrap();
manifest.validate().unwrap();
let new = QuadletUnit::from_manifest(&manifest, "gitea").render();
assert!(new.contains("AddCapability=SYS_CHROOT\n"));
let old = new.replace("AddCapability=SYS_CHROOT\n", "");
assert!(security_changed(&old, &new));
assert!(!security_changed(&new, &new));
assert!(!security_changed(
"AddCapability=CHOWN\nAddCapability=SETUID\n",
"AddCapability=SETUID\nAddCapability=CHOWN\n"
));
}
#[test]
fn network_aliases_changed_detects_network_mode_drift() {
let old = "[Container]\nNetwork=slirp4netns\n";
+132 -57
View File
@@ -1,80 +1,155 @@
//! Seller-side pending entitlements for Lightning-invoice peer-file sales (#46).
//!
//! When a buyer asks to pay for a paid catalog item with an external wallet (as
//! opposed to the local-ecash fast path), the *selling* node mints a Lightning
//! invoice on its own LND and records a pending entitlement here, keyed by the
//! invoice's payment hash. The buyer pays the invoice from any wallet and polls
//! for settlement; once the seller's LND confirms the invoice is settled we mark
//! the entitlement paid, and the content gate (`content_server::serve_content`)
//! then releases the file to anyone presenting that payment hash.
//!
//! State is in-memory and bounded by a TTL. If the seller restarts before the
//! buyer pays, the buyer simply requests a fresh invoice — no value is lost
//! because an unpaid invoice represents no money.
//! Durable seller-side entitlements for peer-file invoices and on-chain sales.
//! Payment records must outlive browser polling, process restarts and invoice
//! expiry: an invoice can settle while the buyer is disconnected.
use std::collections::HashMap;
use std::sync::LazyLock;
use std::time::{Duration, Instant};
use tokio::sync::Mutex;
use anyhow::{Context, Result};
use serde::{Deserialize, Serialize};
use sha2::{Digest, Sha256};
use std::path::{Path, PathBuf};
use tokio::{fs, io::AsyncWriteExt, sync::Mutex};
/// How long a pending/paid entitlement is retained. Generous enough for a human
/// to pay an invoice and download, short enough to keep the map small.
const ENTITLEMENT_TTL: Duration = Duration::from_secs(3600); // 1 hour
static WRITES: Mutex<()> = Mutex::const_new(());
#[derive(Clone)]
#[derive(Clone, Serialize, Deserialize)]
struct Entitlement {
content_id: String,
price_sats: u64,
paid: bool,
created_at: Instant,
}
static ENTITLEMENTS: LazyLock<Mutex<HashMap<String, Entitlement>>> =
LazyLock::new(|| Mutex::new(HashMap::new()));
/// Drop expired entries. Caller must hold the lock.
fn prune(map: &mut HashMap<String, Entitlement>) {
map.retain(|_, e| e.created_at.elapsed() < ENTITLEMENT_TTL);
fn path(data_dir: &Path, token: &str) -> PathBuf {
data_dir.join("content-entitlements").join(format!(
"{}.json",
hex::encode(Sha256::digest(token.as_bytes()))
))
}
/// Record a freshly-minted invoice as a pending (unpaid) entitlement.
pub async fn record_pending(payment_hash: &str, content_id: &str, price_sats: u64) {
let mut map = ENTITLEMENTS.lock().await;
prune(&mut map);
map.insert(
payment_hash.to_string(),
Entitlement {
content_id: content_id.to_string(),
async fn read(data_dir: &Path, token: &str) -> Result<Option<Entitlement>> {
match fs::read(path(data_dir, token)).await {
Ok(bytes) => Ok(Some(
serde_json::from_slice(&bytes).context("Invalid payment entitlement")?,
)),
Err(e) if e.kind() == std::io::ErrorKind::NotFound => Ok(None),
Err(e) => Err(e).context("Reading payment entitlement"),
}
}
async fn write(data_dir: &Path, token: &str, entry: &Entitlement) -> Result<()> {
let target = path(data_dir, token);
let dir = target.parent().unwrap();
fs::create_dir_all(dir).await?;
let tmp = target.with_extension("tmp");
let mut file = fs::OpenOptions::new()
.write(true)
.create(true)
.truncate(true)
.mode(0o600)
.open(&tmp)
.await?;
file.write_all(&serde_json::to_vec(entry)?).await?;
file.sync_all().await?;
drop(file);
fs::rename(&tmp, &target).await?;
fs::File::open(dir).await?.sync_all().await?;
Ok(())
}
/// Save before exposing an invoice/address to the buyer. Never overwrite an
/// existing payment or silently rebind its token to another item or price.
pub async fn record_pending(
data_dir: &Path,
token: &str,
content_id: &str,
price_sats: u64,
) -> Result<()> {
let _lock = WRITES.lock().await;
if let Some(existing) = read(data_dir, token).await? {
anyhow::ensure!(
existing.content_id == content_id && existing.price_sats == price_sats,
"Payment entitlement mismatch"
);
return Ok(());
}
write(
data_dir,
token,
&Entitlement {
content_id: content_id.into(),
price_sats,
paid: false,
created_at: Instant::now(),
},
);
)
.await
}
/// Mark the entitlement for `payment_hash` paid. No-op if unknown/expired.
pub async fn mark_paid(payment_hash: &str) {
let mut map = ENTITLEMENTS.lock().await;
prune(&mut map);
if let Some(e) = map.get_mut(payment_hash) {
e.paid = true;
}
pub async fn mark_paid(data_dir: &Path, token: &str) -> Result<()> {
let _lock = WRITES.lock().await;
let mut entry = read(data_dir, token)
.await?
.context("Unknown payment entitlement")?;
entry.paid = true;
write(data_dir, token, &entry).await
}
/// The content_id + price an entitlement was issued for, if still live.
pub async fn lookup(payment_hash: &str) -> Option<(String, u64)> {
let mut map = ENTITLEMENTS.lock().await;
prune(&mut map);
map.get(payment_hash)
.map(|e| (e.content_id.clone(), e.price_sats))
pub async fn lookup(data_dir: &Path, token: &str) -> Result<Option<(String, u64)>> {
Ok(read(data_dir, token)
.await?
.map(|e| (e.content_id, e.price_sats)))
}
/// True if `payment_hash` is a paid entitlement for exactly `content_id`.
/// This is the gate the content server consults to release a file.
pub async fn is_paid_for(payment_hash: &str, content_id: &str) -> bool {
let mut map = ENTITLEMENTS.lock().await;
prune(&mut map);
map.get(payment_hash)
pub async fn is_paid_for(data_dir: &Path, token: &str, content_id: &str) -> bool {
read(data_dir, token)
.await
.ok()
.flatten()
.map(|e| e.paid && e.content_id == content_id)
.unwrap_or(false)
}
#[cfg(test)]
mod tests {
use super::*;
#[tokio::test]
async fn paid_entitlement_survives_reload_and_cannot_be_rebound() {
let dir = tempfile::tempdir().unwrap();
record_pending(dir.path(), "hash", "file", 12)
.await
.unwrap();
assert!(!is_paid_for(dir.path(), "hash", "file").await);
mark_paid(dir.path(), "hash").await.unwrap();
// All reads reopen disk; no process-local entitlement map exists.
assert!(is_paid_for(dir.path(), "hash", "file").await);
assert!(!is_paid_for(dir.path(), "hash", "other").await);
record_pending(dir.path(), "hash", "file", 12)
.await
.unwrap();
assert!(is_paid_for(dir.path(), "hash", "file").await);
assert!(record_pending(dir.path(), "hash", "other", 12)
.await
.is_err());
assert!(record_pending(dir.path(), "hash", "file", 13)
.await
.is_err());
let other = tempfile::tempdir().unwrap();
assert!(!is_paid_for(other.path(), "hash", "file").await);
assert!(mark_paid(dir.path(), "unknown").await.is_err());
}
#[tokio::test]
async fn corrupt_or_unwritable_records_fail_closed() {
let dir = tempfile::tempdir().unwrap();
record_pending(dir.path(), "../../token", "file", 1)
.await
.unwrap();
fs::write(path(dir.path(), "../../token"), b"broken")
.await
.unwrap();
assert!(lookup(dir.path(), "../../token").await.is_err());
assert!(!is_paid_for(dir.path(), "../../token", "file").await);
assert!(record_pending(dir.path(), "../../token", "file", 1)
.await
.is_err());
let file = dir.path().join("not-directory");
fs::write(&file, b"x").await.unwrap();
assert!(record_pending(&file, "hash", "file", 1).await.is_err());
}
}
+135 -12
View File
@@ -12,7 +12,9 @@
use anyhow::{Context, Result};
use serde::{Deserialize, Serialize};
use std::path::{Path, PathBuf};
use tokio::fs;
use tokio::{fs, io::AsyncWriteExt, sync::Mutex};
static PURCHASE_WRITES: Mutex<()> = Mutex::const_new(());
const OWNED_DIR: &str = "purchased-content";
const OWNED_INDEX: &str = "owned.json";
@@ -66,20 +68,51 @@ fn bytes_path(data_dir: &Path, onion: &str, content_id: &str) -> PathBuf {
.join(sanitize(content_id))
}
async fn load_index(data_dir: &Path) -> OwnedIndex {
async fn load_index_checked(data_dir: &Path) -> Result<OwnedIndex> {
match fs::read_to_string(index_path(data_dir)).await {
Ok(s) => serde_json::from_str(&s).unwrap_or_default(),
Err(_) => OwnedIndex::default(),
Ok(s) => serde_json::from_str(&s)
.context("Invalid purchase index; existing records were preserved"),
Err(error) if error.kind() == std::io::ErrorKind::NotFound => Ok(OwnedIndex::default()),
Err(error) => Err(error).context("Reading purchase index"),
}
}
async fn load_index(data_dir: &Path) -> OwnedIndex {
load_index_checked(data_dir).await.unwrap_or_default()
}
async fn atomic_write(path: &Path, bytes: &[u8]) -> Result<()> {
let parent = path.parent().context("Purchase path has no parent")?;
fs::create_dir_all(parent).await?;
let temp = parent.join(format!(".purchase-{}.tmp", uuid::Uuid::new_v4()));
let result = async {
let mut file = fs::OpenOptions::new()
.write(true)
.create_new(true)
.mode(0o600)
.open(&temp)
.await?;
file.write_all(bytes).await?;
file.sync_all().await?;
drop(file);
fs::rename(&temp, path).await?;
fs::File::open(parent).await?.sync_all().await?;
Ok::<_, anyhow::Error>(())
}
.await;
if result.is_err() {
let _ = fs::remove_file(&temp).await;
}
result
}
async fn save_index(data_dir: &Path, index: &OwnedIndex) -> Result<()> {
let root = owned_root(data_dir);
fs::create_dir_all(&root)
.await
.with_context(|| format!("creating {}", root.display()))?;
let content = serde_json::to_string_pretty(index).context("serializing owned index")?;
fs::write(index_path(data_dir), content)
atomic_write(&index_path(data_dir), content.as_bytes())
.await
.context("writing owned index")
}
@@ -98,17 +131,15 @@ pub async fn record_purchase(
ecash_backend: &str,
purchased_at: &str,
) -> Result<()> {
// Read-modify-write must be one serialized transaction. Never replace a
// damaged index with an empty one, and never expose partially written bytes.
let _lock = PURCHASE_WRITES.lock().await;
let mut index = load_index_checked(data_dir).await?;
let path = bytes_path(data_dir, onion, content_id);
if let Some(parent) = path.parent() {
fs::create_dir_all(parent)
.await
.with_context(|| format!("creating {}", parent.display()))?;
}
fs::write(&path, bytes)
atomic_write(&path, bytes)
.await
.with_context(|| format!("writing purchased bytes to {}", path.display()))?;
let mut index = load_index(data_dir).await;
let entry = OwnedItem {
onion: onion.to_string(),
content_id: content_id.to_string(),
@@ -131,6 +162,11 @@ pub async fn record_purchase(
save_index(data_dir, &index).await
}
/// Payment decisions must not interpret an unreadable index as no purchases.
pub async fn list_owned_checked(data_dir: &Path) -> Result<Vec<OwnedItem>> {
Ok(load_index_checked(data_dir).await?.items)
}
/// Every item this node owns.
pub async fn list_owned(data_dir: &Path) -> Vec<OwnedItem> {
load_index(data_dir).await.items
@@ -165,3 +201,90 @@ pub async fn read_owned(
.unwrap_or_else(|| "application/octet-stream".to_string());
Some((mime, bytes))
}
#[cfg(test)]
mod tests {
use super::*;
#[tokio::test]
async fn concurrent_purchases_preserve_every_item_and_exact_bytes() {
let dir = tempfile::tempdir().unwrap();
let mut jobs = tokio::task::JoinSet::new();
for n in 0..24 {
let root = dir.path().to_path_buf();
jobs.spawn(async move {
let id = format!("file-{n}");
record_purchase(
&root,
"seller.onion",
&id,
&id,
"text/plain",
id.as_bytes(),
5,
"lightning",
"now",
)
.await
.unwrap();
});
}
while let Some(result) = jobs.join_next().await {
result.unwrap();
}
assert_eq!(list_owned(dir.path()).await.len(), 24);
for n in 0..24 {
let id = format!("file-{n}");
assert!(is_owned(dir.path(), "seller.onion", &id).await);
let (mime, bytes) = read_owned(dir.path(), "seller.onion", &id).await.unwrap();
assert_eq!(mime, "text/plain");
assert_eq!(bytes, id.as_bytes());
}
record_purchase(
dir.path(),
"seller.onion",
"file-0",
"file-0",
"text/plain",
b"updated",
5,
"lightning",
"later",
)
.await
.unwrap();
assert_eq!(list_owned(dir.path()).await.len(), 24);
assert_eq!(
read_owned(dir.path(), "seller.onion", "file-0")
.await
.unwrap()
.1,
b"updated"
);
}
#[tokio::test]
async fn damaged_index_is_preserved_instead_of_erasing_prior_ownership() {
let dir = tempfile::tempdir().unwrap();
fs::create_dir_all(owned_root(dir.path())).await.unwrap();
fs::write(index_path(dir.path()), b"damaged but preserve me")
.await
.unwrap();
assert!(record_purchase(
dir.path(),
"seller.onion",
"new",
"new",
"text/plain",
b"bytes",
5,
"lightning",
"now"
)
.await
.is_err());
assert_eq!(
fs::read(index_path(dir.path())).await.unwrap(),
b"damaged but preserve me"
);
assert!(!bytes_path(dir.path(), "seller.onion", "new").exists());
}
}
+470 -39
View File
@@ -238,6 +238,11 @@ pub enum ServeResult {
Forbidden,
/// Content not found.
NotFound,
/// The catalog entry and file exist but this node can't read the file.
/// Returned before any payment is taken.
Unavailable,
/// Requested byte range cannot be served; no payment was taken.
RangeNotSatisfiable(u64),
}
/// Serve a content item by ID with access control and optional range request.
@@ -252,6 +257,39 @@ pub async fn serve_content(
range: Option<ByteRange>,
owner_session: bool,
) -> Result<ServeResult> {
serve_content_with(
data_dir,
id,
payment_token,
invoice_hash,
peer_did,
range,
owner_session,
|path, range, mime| prepare_content(data_dir, path, range, mime),
|token, amount| async move { verify_payment_token(data_dir, &token, amount).await },
)
.await
}
// Inject only the read and payment boundaries, so tests can prove ordering
// without mint access, file-permission assumptions or privileged commands.
async fn serve_content_with<R, RF, V, VF>(
data_dir: &Path,
id: &str,
payment_token: Option<&str>,
invoice_hash: Option<&str>,
peer_did: Option<&str>,
range: Option<ByteRange>,
owner_session: bool,
read: R,
verify: V,
) -> Result<ServeResult>
where
R: FnOnce(PathBuf, Option<ByteRange>, String) -> RF,
RF: std::future::Future<Output = Result<ServeResult>>,
V: FnOnce(String, u64) -> VF,
VF: std::future::Future<Output = bool>,
{
let catalog = load_catalog(data_dir).await?;
let item = match catalog.items.iter().find(|i| i.id == id) {
Some(i) => i,
@@ -314,6 +352,29 @@ pub async fn serve_content(
return Ok(ServeResult::NotFound);
}
// Refuse unauthorized viewers before opening or reading any bytes.
if !owner_session && matches!(item.access, AccessControl::PeersOnly) && !is_known_peer {
return Ok(ServeResult::Forbidden);
}
if !owner_session {
if let AccessControl::Paid { price_sats, .. } = &item.access {
if payment_token.is_none() && invoice_hash.is_none() {
return Ok(ServeResult::PaymentRequired(*price_sats));
}
}
}
// Finish all file I/O before consuming bearer payment. Merely opening then
// reopening after charging still lost payments on read errors or deletion.
let prepared = match read(file_path, range, item.mime_type.clone()).await {
Ok(result @ (ServeResult::Ok(..) | ServeResult::Partial { .. })) => result,
Ok(other) => return Ok(other),
Err(error) => {
warn!(content_id = %id, "Cannot prepare shared content: {error:#}");
return Ok(ServeResult::Unavailable);
}
};
// Check access control
if !owner_session {
match &item.access {
@@ -331,7 +392,7 @@ pub async fn serve_content(
"fedimint"
};
if method_accepted(&item.access, method)
&& verify_payment_token(data_dir, token, *price_sats).await
&& verify(token.to_owned(), *price_sats).await
{
authorized = true;
}
@@ -339,7 +400,7 @@ pub async fn serve_content(
if !authorized {
if let Some(hash) = invoice_hash {
if method_accepted(&item.access, "lightning")
&& crate::content_invoice::is_paid_for(hash, id).await
&& crate::content_invoice::is_paid_for(data_dir, hash, id).await
{
authorized = true;
}
@@ -358,55 +419,127 @@ pub async fn serve_content(
}
}
let metadata = fs::metadata(&file_path)
.await
.context("Failed to read file metadata")?;
let total_size = metadata.len();
// Handle range request for streaming
if let Some(range) = range {
let start = range.start.min(total_size.saturating_sub(1));
let end = range
.end
.map(|e| e.min(total_size - 1))
.unwrap_or(total_size - 1);
if start > end || start >= total_size {
return Ok(ServeResult::NotFound);
Ok(prepared)
}
let len = (end - start + 1) as usize;
async fn prepare_content(
data_dir: &Path,
path: PathBuf,
range: Option<ByteRange>,
mime: String,
) -> Result<ServeResult> {
use tokio::io::{AsyncReadExt, AsyncSeekExt};
let mut file = tokio::fs::File::open(&file_path)
let mut file = match fs::OpenOptions::new()
.read(true)
.custom_flags(libc::O_NONBLOCK)
.open(&path)
.await
.context("Failed to open content file")?;
file.seek(std::io::SeekFrom::Start(start))
{
Ok(file) => file,
Err(error) if error.kind() == std::io::ErrorKind::PermissionDenied => {
let bytes = read_filebrowser_via_userns(data_dir, &path).await?;
return slice_prepared_content(bytes, range, mime);
}
Err(error) => return Err(error).context("Opening shared content"),
};
let metadata = file.metadata().await?;
anyhow::ensure!(metadata.is_file(), "Shared content is not a regular file");
let total = metadata.len();
if let Some(range) = range {
let Some((start, end)) = checked_range(&range, total) else {
return Ok(ServeResult::RangeNotSatisfiable(total));
};
file.seek(std::io::SeekFrom::Start(start)).await?;
let len = usize::try_from(end - start + 1).context("Content range is too large")?;
let mut bytes = vec![0; len];
file.read_exact(&mut bytes)
.await
.context("Failed to seek")?;
let mut buf = vec![0u8; len];
file.read_exact(&mut buf)
.await
.context("Failed to read range")?;
debug!(
"Serving content '{}' range {}-{}/{} ({} bytes)",
id, start, end, total_size, len
);
.context("Reading shared content range")?;
return Ok(ServeResult::Partial {
bytes: buf,
mime_type: item.mime_type.clone(),
bytes,
mime_type: mime,
start,
end,
total: total_size,
total,
});
}
let bytes = fs::read(&file_path)
let mut bytes = Vec::new();
file.read_to_end(&mut bytes)
.await
.context("Failed to read content file")?;
.context("Reading shared content")?;
Ok(ServeResult::Ok(bytes, mime))
}
debug!("Serving content '{}' ({} bytes)", id, bytes.len());
Ok(ServeResult::Ok(bytes, item.mime_type.clone()))
fn checked_range(range: &ByteRange, total: u64) -> Option<(u64, u64)> {
let last = total.checked_sub(1)?;
let end = range.end.unwrap_or(last).min(last);
(range.start <= end && range.start < total).then_some((range.start, end))
}
fn slice_prepared_content(
bytes: Vec<u8>,
range: Option<ByteRange>,
mime: String,
) -> Result<ServeResult> {
let total = bytes.len() as u64;
match range {
None => Ok(ServeResult::Ok(bytes, mime)),
Some(range) => match checked_range(&range, total) {
Some((start, end)) => Ok(ServeResult::Partial {
bytes: bytes[start as usize..=end as usize].to_vec(),
mime_type: mime,
start,
end,
total,
}),
None => Ok(ServeResult::RangeNotSatisfiable(total)),
},
}
}
/// Read only an explicitly shared, regular file within FileBrowser storage.
/// Do not change its mode or grant world-readable access to paid/private data.
async fn filebrowser_read_path(data_dir: &Path, path: &Path) -> Result<PathBuf> {
let root = fs::canonicalize(data_dir.join("filebrowser")).await?;
let target = fs::canonicalize(path).await?;
anyhow::ensure!(
target.starts_with(&root) && target != root,
"Shared file is outside Files storage"
);
anyhow::ensure!(
fs::metadata(&target).await?.is_file(),
"Shared content is not a regular file"
);
Ok(target)
}
async fn read_filebrowser_via_userns(data_dir: &Path, path: &Path) -> Result<Vec<u8>> {
let path = filebrowser_read_path(data_dir, path).await?;
// Tests exercise the boundary explicitly; they never launch the host Podman.
#[cfg(test)]
{
let _ = path;
anyhow::bail!("Files namespace read disabled in unit tests")
}
#[cfg(not(test))]
{
let output = tokio::time::timeout(
std::time::Duration::from_secs(900),
tokio::process::Command::new("podman")
.args(["unshare", "cat", "--"])
.arg(path)
.kill_on_drop(true)
.output(),
)
.await
.context("Files namespace read timed out")??;
anyhow::ensure!(
output.status.success(),
"Files namespace read failed: {}",
output.status
);
Ok(output.stdout)
}
}
/// Result of attempting to serve a preview.
@@ -729,3 +862,301 @@ mod prune_missing_content_tests {
assert_eq!(reloaded.items[0].id, "present-item");
}
}
#[cfg(test)]
mod paid_read_order_tests {
use super::*;
use std::sync::atomic::{AtomicUsize, Ordering};
async fn fixture(bytes: &[u8]) -> tempfile::TempDir {
let dir = tempfile::tempdir().unwrap();
fs::create_dir_all(dir.path().join("content/files"))
.await
.unwrap();
fs::write(dir.path().join("content/files/test.bin"), bytes)
.await
.unwrap();
save_catalog(
dir.path(),
&ContentCatalog {
items: vec![ContentItem {
id: "paid".into(),
filename: "test.bin".into(),
mime_type: "application/octet-stream".into(),
size_bytes: bytes.len() as u64,
description: String::new(),
access: AccessControl::Paid {
price_sats: 10,
accepted: vec!["ecash".into()],
},
availability: Availability::AllPeers,
added_at: "2026-09-30".into(),
}],
},
)
.await
.unwrap();
dir
}
#[tokio::test]
async fn all_read_failures_precede_redemption_even_as_root() {
for kind in [
std::io::ErrorKind::PermissionDenied,
std::io::ErrorKind::UnexpectedEof,
std::io::ErrorKind::NotFound,
std::io::ErrorKind::Other,
] {
let dir = fixture(b"abc").await;
let charged = AtomicUsize::new(0);
let result = serve_content_with(
dir.path(),
"paid",
Some("cashuBtest"),
None,
None,
None,
false,
|_, _, _| async move { Err(std::io::Error::from(kind).into()) },
|_, _| async {
charged.fetch_add(1, Ordering::SeqCst);
true
},
)
.await
.unwrap();
assert!(matches!(result, ServeResult::Unavailable));
assert_eq!(charged.load(Ordering::SeqCst), 0);
assert_eq!(load_catalog(dir.path()).await.unwrap().items.len(), 1);
}
}
#[tokio::test]
async fn deletion_during_payment_cannot_lose_prepared_bytes() {
let dir = fixture(b"original").await;
let result = serve_content_with(
dir.path(),
"paid",
Some("cashuBtest"),
None,
None,
None,
false,
|path, range, mime| prepare_content(dir.path(), path, range, mime),
|_, amount| {
assert_eq!(amount, 10);
async {
fs::remove_file(dir.path().join("content/files/test.bin"))
.await
.unwrap();
true
}
},
)
.await
.unwrap();
assert!(matches!(result, ServeResult::Ok(bytes, _) if bytes == b"original"));
}
#[tokio::test]
async fn empty_out_of_bounds_and_reversed_ranges_never_charge() {
for (bytes, start, end) in [
(b"".as_slice(), 0, None),
(b"abc".as_slice(), 3, None),
(b"abc".as_slice(), 2, Some(1)),
] {
let dir = fixture(bytes).await;
let result = serve_content_with(
dir.path(),
"paid",
Some("cashuBtest"),
None,
None,
Some(ByteRange { start, end }),
false,
|path, range, mime| prepare_content(dir.path(), path, range, mime),
|_, _| async { panic!("invalid range reached payment") },
)
.await
.unwrap();
assert!(
matches!(result, ServeResult::RangeNotSatisfiable(n) if n == bytes.len() as u64)
);
}
}
#[tokio::test]
async fn prepared_range_survives_file_change_while_payment_is_verified() {
let dir = fixture(b"abcdef").await;
let result = serve_content_with(
dir.path(),
"paid",
Some("cashuBtest"),
None,
None,
Some(ByteRange {
start: 2,
end: Some(999),
}),
false,
|path, range, mime| prepare_content(dir.path(), path, range, mime),
|_, _| async {
fs::write(dir.path().join("content/files/test.bin"), b"x")
.await
.unwrap();
true
},
)
.await
.unwrap();
assert!(
matches!(result, ServeResult::Partial { bytes, start: 2, end: 5, total: 6, .. } if bytes == b"cdef")
);
}
#[tokio::test]
async fn payment_denial_never_returns_prepared_content() {
let dir = fixture(b"secret").await;
let charged = AtomicUsize::new(0);
let result = serve_content_with(
dir.path(),
"paid",
Some("cashuBtest"),
None,
None,
None,
false,
|path, range, mime| prepare_content(dir.path(), path, range, mime),
|_, _| async {
charged.fetch_add(1, Ordering::SeqCst);
false
},
)
.await
.unwrap();
assert!(matches!(result, ServeResult::PaymentRequired(10)));
assert_eq!(charged.load(Ordering::SeqCst), 1);
}
#[tokio::test]
async fn missing_payment_and_peer_restrictions_precede_file_reads() {
let dir = fixture(b"secret").await;
let result = serve_content_with(
dir.path(),
"paid",
None,
None,
None,
None,
false,
|_, _, _| async { panic!("unauthorized file read") },
|_, _| async { panic!("unexpected payment") },
)
.await
.unwrap();
assert!(matches!(result, ServeResult::PaymentRequired(10)));
let mut catalog = load_catalog(dir.path()).await.unwrap();
catalog.items[0].access = AccessControl::PeersOnly;
save_catalog(dir.path(), &catalog).await.unwrap();
let result = serve_content_with(
dir.path(),
"paid",
None,
None,
None,
None,
false,
|_, _, _| async { panic!("unauthorized file read") },
|_, _| async { panic!("unexpected payment") },
)
.await
.unwrap();
assert!(matches!(result, ServeResult::Forbidden));
}
#[tokio::test]
async fn owner_reads_paid_content_without_redemption() {
let dir = fixture(b"own file").await;
let result = serve_content_with(
dir.path(),
"paid",
None,
None,
None,
None,
true,
|path, range, mime| prepare_content(dir.path(), path, range, mime),
|_, _| async { panic!("owner charged") },
)
.await
.unwrap();
assert!(matches!(result, ServeResult::Ok(bytes, _) if bytes == b"own file"));
}
#[tokio::test]
async fn directory_in_place_of_file_does_not_charge() {
let dir = fixture(b"abc").await;
let path = dir.path().join("content/files/test.bin");
fs::remove_file(&path).await.unwrap();
fs::create_dir(&path).await.unwrap();
let result = serve_content_with(
dir.path(),
"paid",
Some("cashuBtest"),
None,
None,
None,
false,
|path, range, mime| prepare_content(dir.path(), path, range, mime),
|_, _| async { panic!("directory charged") },
)
.await
.unwrap();
assert!(matches!(result, ServeResult::Unavailable));
}
#[tokio::test]
async fn files_namespace_read_is_scoped_to_regular_files_and_keeps_mode() {
use std::os::unix::fs::{symlink, PermissionsExt};
let dir = fixture(b"outside").await;
let root = dir.path().join("filebrowser");
fs::create_dir(&root).await.unwrap();
let inside = root.join("song");
fs::write(&inside, b"song").await.unwrap();
fs::set_permissions(&inside, std::fs::Permissions::from_mode(0o640))
.await
.unwrap();
assert_eq!(
filebrowser_read_path(dir.path(), &inside).await.unwrap(),
inside
);
assert_eq!(
fs::metadata(&inside).await.unwrap().permissions().mode() & 0o777,
0o640
);
let outside = dir.path().join("content/files/test.bin");
symlink(&outside, root.join("escape")).unwrap();
for path in [outside, root.join("escape"), root.clone()] {
assert!(filebrowser_read_path(dir.path(), &path).await.is_err());
}
}
#[test]
fn user_namespace_bytes_use_the_same_range_rules() {
assert!(matches!(
slice_prepared_content(
vec![],
Some(ByteRange {
start: 0,
end: None
}),
"x".into()
)
.unwrap(),
ServeResult::RangeNotSatisfiable(0)
));
assert!(
matches!(slice_prepared_content(b"abc".to_vec(), Some(ByteRange { start: 1, end: None }), "x".into()).unwrap(), ServeResult::Partial { bytes, start: 1, end: 2, total: 3, .. } if bytes == b"bc")
);
}
}
+45 -17
View File
@@ -664,6 +664,10 @@ pub async fn start_stopped_stack_containers(data_dir: &Path) -> RecoveryReport {
start_stopped_app_stacks(data_dir).await
}
fn stack_member_needs_recovery(state: Option<&str>, user_stopped: bool) -> bool {
!user_stopped && matches!(state, Some("exited" | "stopped" | "created" | "configured"))
}
async fn start_stopped_app_stacks(data_dir: &Path) -> RecoveryReport {
let user_stopped = load_user_stopped(data_dir).await;
let mut report = RecoveryReport {
@@ -677,24 +681,27 @@ async fn start_stopped_app_stacks(data_dir: &Path) -> RecoveryReport {
continue;
}
info!(
"Recovering stopped {} stack containers after boot",
stack.name
);
// Healthy members must never acquire a restarting overlay merely
// because the periodic recovery scan ran. Queue existing stopped
// members only; recheck each immediately before starting below.
let mut pending = Vec::new();
for container in stack.containers {
let state = container_state(container).await;
if stack_member_needs_recovery(state.as_deref(), user_stopped.contains(*container)) {
pending.push((*container).to_string());
}
}
if pending.is_empty() {
continue;
}
info!("Recovering stopped {} stack containers", stack.name);
repair_stack_network_aliases(stack).await;
// Register the whole stack up front: the per-member dependency waits
// below can take minutes, and the UI should say "Restarting", not
// "Stopped", for members still queued behind them.
pending_boot_starts_add(
stack
.containers
.iter()
.filter(|c| !user_stopped.contains(**c))
.map(|c| (*c).to_string()),
);
pending_boot_starts_add(pending.iter().cloned());
for container in stack.containers {
if !pending.iter().any(|name| name.as_str() == *container) {
continue;
}
if user_stopped.contains(*container) {
info!("Skipping user-stopped container: {}", container);
continue;
@@ -706,8 +713,8 @@ async fn start_stopped_app_stacks(data_dir: &Path) -> RecoveryReport {
pending_boot_start_done(container);
continue;
}
Some(_) => {}
None => {
Some(state) if stack_member_needs_recovery(Some(&state), false) => {}
_ => {
pending_boot_start_done(container);
continue;
}
@@ -1534,3 +1541,24 @@ mod installed_concurrency_tests {
assert!(!dir.path().join("installed-apps.json.tmp").exists());
}
}
#[cfg(test)]
mod stack_recovery_overlay_tests {
use super::stack_member_needs_recovery;
#[test]
fn only_existing_stopped_members_receive_recovery_overlay() {
for state in [
None,
Some("running"),
Some("paused"),
Some("restarting"),
Some("removing"),
] {
assert!(!stack_member_needs_recovery(state, false));
}
for state in ["exited", "stopped", "created", "configured"] {
assert!(stack_member_needs_recovery(Some(state), false));
assert!(!stack_member_needs_recovery(Some(state), true));
}
}
}
+40 -3
View File
@@ -5,8 +5,45 @@
//! are reachable over the mesh; ports of apps that aren't installed have
//! no listener, so allowing them is inert.
#[rustfmt::skip]
pub const APP_LAUNCH_PORTS: &[u16] = &[
2283, 2342, 3000, 3001, 3002, 4080, 5180, 7778, 8080, 8081, 8082, 8083, 8084, 8085, 8087, 8090,
8096, 8123, 8175, 8176, 8187, 8240, 8334, 8336, 8337, 8888, 8999, 9000, 9100, 10380, 11434,
18081, 18083, 18091, 23000, 32838, 50002,
2283,
2342,
3000,
3001,
3002,
4080,
5180,
7778,
8080,
8081,
8082,
8083,
8084,
8085,
8087,
8090,
8091,
8096,
8123,
8175,
8176,
8187,
8240,
8334,
8336,
8337,
8888,
8998,
8999,
9000,
9100,
10380,
11434,
18081,
18083,
18091,
23000,
32838,
50002,
];
+268 -7
View File
@@ -46,6 +46,25 @@ fn fips_should_fall_back(status: reqwest::StatusCode) -> bool {
status == reqwest::StatusCode::NOT_FOUND || status.is_server_error()
}
/// Is this FIPS answer the final one, or should the request go again over
/// Tor? A single-delivery request already reached the peer, so any answer
/// is final: a Tor replay would carry the same (possibly spent) payload.
fn fips_answer_is_final(
pref: crate::settings::transport::TransportPref,
single_delivery: bool,
status: reqwest::StatusCode,
) -> bool {
pref == crate::settings::transport::TransportPref::Fips
|| single_delivery
|| !fips_should_fall_back(status)
}
/// May a failed FIPS attempt be sent again? Only a failed connect proves the
/// peer never saw it; a timeout can land after the request was delivered.
fn fips_retryable(single_delivery: bool, e: &reqwest::Error) -> bool {
e.is_connect() || (!single_delivery && e.is_timeout())
}
/// DNS suffix appended to a peer's bech32 npub.
pub const FIPS_DNS_SUFFIX: &str = "fips";
@@ -113,7 +132,21 @@ pub fn client() -> reqwest::Client {
/// before the Tor fallback ever gets a chance. The generous `connect_timeout`
/// is preserved so a cold hole-punched path still gets time to establish.
pub fn client_with_timeout(timeout: Duration) -> reqwest::Client {
client_with_delivery_policy(timeout, false)
}
fn delivery_redirect_policy(single: bool) -> reqwest::redirect::Policy {
if single {
reqwest::redirect::Policy::none()
} else {
reqwest::redirect::Policy::default()
}
}
fn client_with_delivery_policy(timeout: Duration, single: bool) -> reqwest::Client {
reqwest::Client::builder()
.no_proxy()
.redirect(delivery_redirect_policy(single))
.timeout(timeout)
.connect_timeout(Duration::from_secs(8))
.user_agent("archipelago-fips/1")
@@ -130,10 +163,18 @@ pub fn client_with_timeout(timeout: Duration) -> reqwest::Client {
/// robust". Only connect/timeout errors are retried (a real HTTP response,
/// including 4xx/5xx, is returned as-is for the caller to interpret).
async fn send_with_retry(rb: reqwest::RequestBuilder) -> Result<reqwest::Response, reqwest::Error> {
send_with_retry_if(rb, |e| e.is_connect() || e.is_timeout()).await
}
/// [`send_with_retry`], retrying only on errors `retryable` accepts.
async fn send_with_retry_if(
rb: reqwest::RequestBuilder,
retryable: impl Fn(&reqwest::Error) -> bool,
) -> Result<reqwest::Response, reqwest::Error> {
let retry = rb.try_clone();
match rb.send().await {
Ok(resp) => Ok(resp),
Err(e) if (e.is_connect() || e.is_timeout()) && retry.is_some() => {
Err(e) if retryable(&e) && retry.is_some() => {
// Brief pause so the hole-punch packets from the first attempt can
// traverse before we re-dial onto the warmed path.
tokio::time::sleep(Duration::from_millis(600)).await;
@@ -350,6 +391,9 @@ pub struct PeerRequest<'a> {
/// the per-peer FIPS/Tor badge reflects reality. Opt-in because not
/// every caller has a data dir in scope.
pub record_data_dir: Option<std::path::PathBuf>,
/// The request carries something that must reach the peer at most once
/// (a bearer ecash token). See [`PeerRequest::single_delivery`].
pub single_delivery: bool,
}
impl<'a> PeerRequest<'a> {
@@ -363,9 +407,25 @@ impl<'a> PeerRequest<'a> {
fips_timeout: None,
service: None,
record_data_dir: None,
single_delivery: false,
}
}
/// Never send this request twice. A paid download carries a bearer ecash
/// token that the seller redeems on first sight; replaying it over Tor
/// after FIPS already delivered it hands the seller a spent token, so the
/// buyer is charged and gets a 402 instead of the file (2026-09-29: FIPS
/// answered 404 after the seller redeemed, the Tor retry got 402).
///
/// With this set, whatever FIPS answers is final, the FIPS retry fires
/// only when the first attempt never connected, and Tor is used only when
/// FIPS could not have delivered the request. An attempt that may have
/// been delivered but timed out is an error, not a fallback.
pub fn single_delivery(mut self) -> Self {
self.single_delivery = true;
self
}
/// Record the transport that serves this request into federation storage
/// (matched by this request's onion host). Best-effort, off the hot path.
pub fn record_transport(mut self, data_dir: impl Into<std::path::PathBuf>) -> Self {
@@ -442,7 +502,7 @@ impl<'a> PeerRequest<'a> {
// Use the FIPS reply unless it's one a Tor retry could
// fix (404 path-not-served / 5xx) and we're allowed to
// fall back. FIPS-only never falls back.
if pref == TransportPref::Fips || !fips_should_fall_back(resp.status()) {
if fips_answer_is_final(pref, self.single_delivery, resp.status()) {
telemetry::record_fips_ok();
self.spawn_record(crate::transport::TransportKind::Fips);
return Ok((resp, crate::transport::TransportKind::Fips));
@@ -481,7 +541,7 @@ impl<'a> PeerRequest<'a> {
if matches!(pref, TransportPref::Auto | TransportPref::Fips) {
match self.try_fips_get().await? {
Some(resp) => {
if pref == TransportPref::Fips || !fips_should_fall_back(resp.status()) {
if fips_answer_is_final(pref, self.single_delivery, resp.status()) {
telemetry::record_fips_ok();
self.spawn_record(crate::transport::TransportKind::Fips);
return Ok((resp, crate::transport::TransportKind::Fips));
@@ -551,13 +611,21 @@ impl<'a> PeerRequest<'a> {
} else {
budget
};
let c = client_with_timeout(per_attempt);
let c = client_with_delivery_policy(per_attempt, self.single_delivery);
let mut rb = c.post(&url).json(body);
for (k, v) in &self.headers {
rb = rb.header(*k, v);
}
match tokio::time::timeout(budget, send_with_retry(rb)).await {
let single = self.single_delivery;
let attempt = send_with_retry_if(rb, |e| fips_retryable(single, e));
match tokio::time::timeout(budget, attempt).await {
Ok(Ok(r)) => Ok(Some(r)),
Ok(Err(e)) if single && !e.is_connect() => Err(anyhow::anyhow!(
"FIPS POST failed after possible delivery; not replaying: {e}"
)),
Err(_) if single => Err(anyhow::anyhow!(
"FIPS POST exceeded its budget after possible delivery; not replaying"
)),
Ok(Err(e)) => {
telemetry::record_fallback(FallbackReason::ConnectFail);
tracing::info!(
@@ -612,13 +680,28 @@ impl<'a> PeerRequest<'a> {
} else {
budget
};
let c = client_with_timeout(per_attempt);
let c = client_with_delivery_policy(per_attempt, self.single_delivery);
let mut rb = c.get(&url);
for (k, v) in &self.headers {
rb = rb.header(*k, v);
}
match tokio::time::timeout(budget, send_with_retry(rb)).await {
let single = self.single_delivery;
let attempt = send_with_retry_if(rb, |e| fips_retryable(single, e));
match tokio::time::timeout(budget, attempt).await {
Ok(Ok(r)) => Ok(Some(r)),
// Anything but a failed connect may have reached the peer.
Ok(Err(e)) if single && !e.is_connect() => Err(anyhow::anyhow!(
"FIPS GET {} failed after the request may have been delivered \
(not retrying over Tor): {}",
self.path,
e
)),
Err(_) if single => Err(anyhow::anyhow!(
"FIPS GET {} exceeded its {:?} budget after the request may have \
been delivered (not retrying over Tor)",
self.path,
budget
)),
Ok(Err(e)) => {
telemetry::record_fallback(FallbackReason::ConnectFail);
tracing::info!(
@@ -676,6 +759,7 @@ impl<'a> PeerRequest<'a> {
.context("Invalid Tor SOCKS proxy URL")?;
reqwest::Client::builder()
.proxy(proxy)
.redirect(delivery_redirect_policy(self.single_delivery))
.timeout(self.timeout)
.build()
.context("Build Tor HTTP client")
@@ -759,4 +843,181 @@ mod tests {
let err = decode_response(0xAABB, &r, "x").unwrap_err();
assert!(err.to_string().contains("no AAAA"));
}
#[test]
fn a_single_delivery_answer_is_final_whatever_its_status() {
use crate::settings::transport::TransportPref;
use reqwest::StatusCode;
// Regression (2026-09-29): the seller redeemed a paid download's
// token, answered 404, and the Tor fallback replayed the spent token.
for status in [
StatusCode::NOT_FOUND,
StatusCode::INTERNAL_SERVER_ERROR,
StatusCode::SERVICE_UNAVAILABLE,
StatusCode::OK,
] {
assert!(fips_answer_is_final(TransportPref::Auto, true, status));
}
// Everything else keeps the existing fallback rules.
assert!(!fips_answer_is_final(
TransportPref::Auto,
false,
StatusCode::NOT_FOUND
));
assert!(!fips_answer_is_final(
TransportPref::Auto,
false,
StatusCode::BAD_GATEWAY
));
assert!(fips_answer_is_final(
TransportPref::Auto,
false,
StatusCode::PAYMENT_REQUIRED
));
assert!(fips_answer_is_final(
TransportPref::Fips,
false,
StatusCode::NOT_FOUND
));
}
/// A listener that accepts connections and never answers, counting them.
async fn silent_peer() -> (String, std::sync::Arc<std::sync::atomic::AtomicUsize>) {
let listener = tokio::net::TcpListener::bind("127.0.0.1:0").await.unwrap();
let addr = listener.local_addr().unwrap();
let seen = std::sync::Arc::new(std::sync::atomic::AtomicUsize::new(0));
let counter = seen.clone();
tokio::spawn(async move {
let mut held = Vec::new();
while let Ok((stream, _)) = listener.accept().await {
counter.fetch_add(1, std::sync::atomic::Ordering::SeqCst);
held.push(stream); // keep it open, never reply
}
});
(format!("http://{addr}/content/x"), seen)
}
#[tokio::test]
async fn a_single_delivery_request_is_not_resent_after_a_timeout() {
let (url, seen) = silent_peer().await;
let c = client_with_timeout(Duration::from_millis(300));
let err = send_with_retry_if(c.get(&url), |e| fips_retryable(true, e))
.await
.expect_err("peer never answers");
assert!(err.is_timeout());
assert_eq!(seen.load(std::sync::atomic::Ordering::SeqCst), 1);
}
#[tokio::test]
async fn an_ordinary_request_is_still_retried_once_after_a_timeout() {
let (url, seen) = silent_peer().await;
let c = client_with_timeout(Duration::from_millis(300));
let _ = send_with_retry_if(c.get(&url), |e| fips_retryable(false, e)).await;
assert_eq!(seen.load(std::sync::atomic::Ordering::SeqCst), 2);
}
#[tokio::test]
async fn a_single_delivery_request_still_retries_a_refused_connect() {
// Nothing listening: the peer provably never saw the request.
let listener = std::net::TcpListener::bind("127.0.0.1:0").unwrap();
let addr = listener.local_addr().unwrap();
drop(listener);
let c = client_with_timeout(Duration::from_millis(500));
let err = send_with_retry_if(c.get(format!("http://{addr}/")), |e| {
fips_retryable(true, e)
})
.await
.expect_err("nothing listening");
assert!(err.is_connect());
assert!(fips_retryable(true, &err));
}
}
#[cfg(test)]
mod delivery_redirect_tests {
use super::*;
use hyper::{
service::{make_service_fn, service_fn},
Body, Response, Server,
};
use std::{
convert::Infallible,
sync::{
atomic::{AtomicUsize, Ordering},
Arc,
},
};
#[tokio::test]
async fn paid_bearer_request_does_not_follow_redirects_but_normal_get_does() {
let seen = Arc::new(AtomicUsize::new(0));
let counter = seen.clone();
let server = Server::bind(&([127, 0, 0, 1], 0).into());
let address = server.local_addr();
let service = make_service_fn(move |_| {
let counter = counter.clone();
async move {
Ok::<_, Infallible>(service_fn(move |request: hyper::Request<Body>| {
let counter = counter.clone();
async move {
counter.fetch_add(1, Ordering::SeqCst);
let response = if request.uri().path() == "/first" {
Response::builder()
.status(302)
.header("Location", "/replay")
.body(Body::empty())
.unwrap()
} else {
Response::new(Body::from("replayed"))
};
Ok::<_, Infallible>(response)
}
}))
}
});
let task = tokio::spawn(server.serve(service));
let url = format!("http://{address}/first");
let response = client_with_delivery_policy(Duration::from_secs(2), true)
.get(&url)
.header("X-Payment-Token", "dummy-test-token")
.send()
.await
.unwrap();
assert_eq!(response.status(), reqwest::StatusCode::FOUND);
assert_eq!(seen.load(Ordering::SeqCst), 1);
let response = client_with_delivery_policy(Duration::from_secs(2), false)
.get(url)
.send()
.await
.unwrap();
assert_eq!(response.status(), reqwest::StatusCode::OK);
assert_eq!(seen.load(Ordering::SeqCst), 3);
task.abort();
}
#[tokio::test]
async fn paid_request_is_not_resent_when_peer_disconnects_after_reading_it() {
use tokio::io::AsyncReadExt;
let listener = tokio::net::TcpListener::bind("127.0.0.1:0").await.unwrap();
let address = listener.local_addr().unwrap();
let seen = Arc::new(AtomicUsize::new(0));
let counter = seen.clone();
let task = tokio::spawn(async move {
while let Ok((mut stream, _)) = listener.accept().await {
let mut buf = [0; 4096];
let _ = stream.read(&mut buf).await;
counter.fetch_add(1, Ordering::SeqCst);
drop(stream);
}
});
let c = client_with_delivery_policy(Duration::from_secs(2), true);
let error = send_with_retry_if(c.get(format!("http://{address}/")), |e| {
fips_retryable(true, e)
})
.await
.unwrap_err();
assert!(!error.is_connect());
assert_eq!(seen.load(Ordering::SeqCst), 1);
task.abort();
}
}
+64 -7
View File
@@ -501,12 +501,12 @@ async fn check_containers() -> Vec<ContainerHealth> {
out
}
fn host_tcp_ports_from_container(c: &serde_json::Value) -> Vec<u16> {
fn host_tcp_ports_from_container(c: &serde_json::Value) -> Vec<std::net::SocketAddr> {
let Some(ports) = c.get("Ports").and_then(|v| v.as_array()) else {
return Vec::new();
};
let mut out: Vec<u16> = ports
let mut out: Vec<std::net::SocketAddr> = ports
.iter()
.filter(|p| {
p.get("protocol")
@@ -515,9 +515,19 @@ fn host_tcp_ports_from_container(c: &serde_json::Value) -> Vec<u16> {
.eq_ignore_ascii_case("tcp")
})
.filter_map(|p| {
p.get("host_port")
.and_then(|v| v.as_u64())
.and_then(|port| u16::try_from(port).ok())
let port = p.get("host_port")?.as_u64()?;
let port = u16::try_from(port).ok().filter(|port| *port != 0)?;
let bind = p.get("host_ip").and_then(|v| v.as_str()).unwrap_or("");
// Wildcard listeners are reachable through the corresponding
// loopback family. Explicit binds must be probed at that address:
// probing a WireGuard-only port on 127.0.0.1 creates false failures
// and endlessly restarts an otherwise healthy app.
let address: std::net::IpAddr = match bind {
"" | "0.0.0.0" => "127.0.0.1".parse().ok()?,
"::" => "::1".parse().ok()?,
explicit => explicit.parse().ok()?,
};
Some(std::net::SocketAddr::new(address, port))
})
.collect();
out.sort_unstable();
@@ -525,11 +535,11 @@ fn host_tcp_ports_from_container(c: &serde_json::Value) -> Vec<u16> {
out
}
async fn host_ports_ready(ports: &[u16]) -> bool {
async fn host_ports_ready(ports: &[std::net::SocketAddr]) -> bool {
for port in ports {
let ready = tokio::time::timeout(
std::time::Duration::from_secs(2),
tokio::net::TcpStream::connect(("127.0.0.1", *port)),
tokio::net::TcpStream::connect(*port),
)
.await
.is_ok_and(|r| r.is_ok());
@@ -1662,4 +1672,51 @@ mod tests {
"Prefetcher:catching up to daemon height 953,480"
));
}
#[test]
fn published_port_probes_preserve_explicit_bind_addresses() {
let c = serde_json::json!({"Ports": [
{"host_ip":"127.0.0.1","host_port":8081,"protocol":"tcp"},
{"host_ip":"10.77.0.2","host_port":18081,"protocol":"tcp"},
{"host_ip":"10.77.0.2","host_port":18443,"protocol":"tcp"},
{"host_ip":"::1","host_port":8082,"protocol":"tcp"}
]});
let targets = host_tcp_ports_from_container(&c);
for target in [
"127.0.0.1:8081",
"10.77.0.2:18081",
"10.77.0.2:18443",
"[::1]:8082",
] {
assert!(targets.contains(&target.parse().unwrap()));
}
assert!(!targets.contains(&"127.0.0.1:18081".parse().unwrap()));
}
#[test]
fn published_port_probes_normalize_wildcards_and_ignore_invalid_entries() {
let c = serde_json::json!({"Ports": [
{"host_ip":"0.0.0.0","host_port":8080},
{"host_ip":"","host_port":8080},
{"host_ip":"::","host_port":8080},
{"host_ip":"10.0.0.1","host_port":53,"protocol":"udp"},
{"host_ip":"bad","host_port":8080},
{"host_port":0}, {"host_port":65536}, {"container_port":80}
]});
let targets = host_tcp_ports_from_container(&c);
assert_eq!(targets.len(), 2);
assert!(targets.contains(&"127.0.0.1:8080".parse().unwrap()));
assert!(targets.contains(&"[::1]:8080".parse().unwrap()));
}
#[tokio::test]
async fn health_probe_reaches_non_default_loopback_and_detects_closed_port() {
let listener = tokio::net::TcpListener::bind("127.0.0.2:0").await.unwrap();
let address = listener.local_addr().unwrap();
assert!(host_ports_ready(&[address]).await);
// Same port, wrong local address reproduces the former false failure.
let wrong = std::net::SocketAddr::new("127.0.0.1".parse().unwrap(), address.port());
assert!(!host_ports_ready(&[wrong]).await);
drop(listener);
assert!(!host_ports_ready(&[address]).await);
}
}
+209
View File
@@ -115,6 +115,30 @@ fn relay_url_matches(a: &str, b: &str) -> bool {
norm(a) == norm(b)
}
/// True when `record` is the node's own identity: the one whose ed25519 key
/// is the node key (`server_info.pubkey`). `identity.list` reports this as
/// `is_node`, and clients must never offer it as an app signer.
pub fn is_node_identity(record: &IdentityRecord, node_pubkey_hex: &str) -> bool {
!node_pubkey_hex.is_empty() && record.pubkey_hex == node_pubkey_hex
}
/// True when the app identity picker hides `record`, mirroring
/// `NostrIdentityPicker.vue`'s filter exactly: the node identity
/// (`is_node`), any `node-*` id and any identity named "Node".
pub(crate) fn is_hidden_from_app_signer(record: &IdentityRecord, node_pubkey_hex: &str) -> bool {
// Match ECMAScript trim exactly: Rust includes U+0085 and excludes U+FEFF.
let trim_js = |value: &str| value.trim_matches(is_js_whitespace).to_lowercase();
is_node_identity(record, node_pubkey_hex)
|| trim_js(&record.id).starts_with("node-")
|| trim_js(&record.name) == "node"
}
fn is_js_whitespace(c: char) -> bool {
matches!(c, '\u{0009}'..='\u{000D}' | '\u{0020}' | '\u{00A0}' | '\u{1680}'
| '\u{2000}'..='\u{200A}' | '\u{2028}' | '\u{2029}' | '\u{202F}'
| '\u{205F}' | '\u{3000}' | '\u{FEFF}')
}
impl IdentityManager {
pub async fn new(data_dir: &Path) -> Result<Self> {
let identities_dir = data_dir.join(IDENTITIES_DIR);
@@ -150,6 +174,30 @@ impl IdentityManager {
Ok((identities, default_id))
}
/// Nostr public keys of the identities an app may sign with through the
/// NIP-07 bridge, as sorted, de-duplicated, comma-joined lowercase hex.
///
/// Leaves out what the identity picker hides (`is_hidden_from_app_signer`)
/// and identities without a Nostr key (they cannot sign). Empty when no
/// identity qualifies.
pub async fn app_signable_nostr_pubkeys(&self, node_pubkey_hex: &str) -> Result<String> {
let (identities, _) = self.list().await?;
let mut pubkeys: Vec<String> = identities
.iter()
.filter(|r| !is_hidden_from_app_signer(r, node_pubkey_hex))
.filter_map(|r| r.nostr_pubkey.as_deref())
.map(|key| {
anyhow::ensure!(key.len() == 64, "invalid app owner Nostr public key");
nostr_sdk::PublicKey::from_hex(key)
.map(|key| key.to_hex())
.context("invalid app owner Nostr public key")
})
.collect::<Result<Vec<_>>>()?;
pubkeys.sort();
pubkeys.dedup();
Ok(pubkeys.join(","))
}
/// Create a new identity.
pub async fn create(&self, name: String, purpose: IdentityPurpose) -> Result<IdentityRecord> {
let signing_key = SigningKey::generate(&mut OsRng);
@@ -966,6 +1014,167 @@ mod tests {
assert_ne!(default_id, Some(r1.id));
}
fn record(id: &str, name: &str, pubkey_hex: &str) -> IdentityRecord {
IdentityRecord {
id: id.to_string(),
name: name.to_string(),
purpose: IdentityPurpose::Personal,
pubkey_hex: pubkey_hex.to_string(),
did: String::new(),
dht_did: None,
created_at: String::new(),
nostr_pubkey: None,
nostr_npub: None,
profile: None,
}
}
#[test]
fn is_node_identity_matches_only_the_node_pubkey() {
let node = "ab".repeat(32);
let other = "cd".repeat(32);
assert!(is_node_identity(&record("uuid-1", "Laptop", &node), &node));
assert!(!is_node_identity(
&record("uuid-1", "Laptop", &other),
&node
));
// An unknown node key matches nothing, not the records without a key.
assert!(!is_node_identity(&record("uuid-1", "Laptop", ""), ""));
// The id and name rules belong to the picker filter, not to `is_node`.
assert!(!is_node_identity(
&record("node-abc", "Node", &other),
&node
));
}
#[test]
fn is_hidden_from_app_signer_mirrors_the_picker_rules() {
let node = "ab".repeat(32);
let other = "cd".repeat(32);
let hidden = |id: &str, name: &str, pk: &str| {
is_hidden_from_app_signer(&record(id, name, pk), &node)
};
// is_node: matched by key alone, whatever the id and name.
assert!(hidden("uuid-1", "Laptop", &node));
// node-* id, any case, surrounding whitespace ignored.
assert!(hidden("node-0123456789abcdef", "Laptop", &other));
assert!(hidden(" NODE-x ", "Laptop", &other));
assert!(hidden("Node-x", "Laptop", &other));
// The name "Node", any case, surrounding whitespace ignored.
assert!(hidden("uuid-1", "Node", &other));
assert!(hidden("uuid-1", " nODe\t", &other));
assert!(hidden("\u{FEFF}NODE-x\u{FEFF}", "Laptop", &other));
assert!(hidden("uuid-1", "\u{FEFF}Node\u{FEFF}", &other));
// ECMAScript keeps U+0085; do not add owners that the picker hides,
// or hide identities that it offers.
assert!(!hidden("uuid-1", "\u{0085}Node\u{0085}", &other));
// Near misses stay visible.
assert!(!hidden("uuid-1", "Laptop", &other));
assert!(!hidden("my-node-1", "Node 2", &other));
assert!(!hidden("nodes", "Nodes", &other));
}
#[tokio::test]
async fn app_signable_nostr_pubkeys_mirror_the_identity_picker() {
let dir = tempdir().unwrap();
let mgr = IdentityManager::new(dir.path()).await.unwrap();
let personal = mgr
.create("Personal".to_string(), IdentityPurpose::Personal)
.await
.unwrap();
let business = mgr
.create("Business".to_string(), IdentityPurpose::Business)
.await
.unwrap();
// The node identity as mirrored at startup: a `node-` id named
// "Node", given a Nostr key so only the id and name rules hide it.
let mirrored_key = SigningKey::generate(&mut OsRng);
let mirrored = mgr
.create_from_signing_key("Node".to_string(), IdentityPurpose::Personal, mirrored_key)
.await
.unwrap();
assert!(mirrored.id.starts_with("node-"));
mgr.create_nostr_key(&mirrored.id).await.unwrap();
// A user-created identity named "Node" is hidden by the picker too.
let named_node = mgr
.create(" node ".to_string(), IdentityPurpose::Anonymous)
.await
.unwrap();
// The node key belongs to an identity with a uuid id and an ordinary
// name, so only the `is_node` match can hide it.
let laptop = mgr
.create("Laptop".to_string(), IdentityPurpose::Personal)
.await
.unwrap();
assert!(!laptop.id.starts_with("node-"));
let (all, _) = mgr.list().await.unwrap();
assert!(all
.iter()
.any(|r| r.id == mirrored.id && r.nostr_pubkey.is_some()));
assert!(all.iter().any(|r| r.id == named_node.id));
assert!(all
.iter()
.any(|r| r.id == laptop.id && r.nostr_pubkey.is_some()));
let mut expected = vec![
personal.nostr_pubkey.unwrap().to_ascii_lowercase(),
business.nostr_pubkey.unwrap().to_ascii_lowercase(),
];
expected.sort();
assert_eq!(
mgr.app_signable_nostr_pubkeys(&laptop.pubkey_hex)
.await
.unwrap(),
expected.join(",")
);
// Without the node key, the same identity is offered like any other.
let mut with_laptop = expected.clone();
with_laptop.push(laptop.nostr_pubkey.unwrap().to_ascii_lowercase());
with_laptop.sort();
assert_eq!(
mgr.app_signable_nostr_pubkeys("").await.unwrap(),
with_laptop.join(",")
);
}
#[tokio::test]
async fn app_owner_pubkeys_reject_malformed_key_material() {
let dir = tempdir().unwrap();
let mgr = IdentityManager::new(dir.path()).await.unwrap();
let identity = mgr
.create("Owner".into(), IdentityPurpose::Personal)
.await
.unwrap();
let path = mgr.identities_dir.join(format!("{}.json", identity.id));
let original = fs::read(&path).await.unwrap();
for invalid in ["", "not-a-key", "owner,another-owner", "\nINJECTED=true"] {
let mut data: serde_json::Value = serde_json::from_slice(&original).unwrap();
data["nostr_pubkey_hex"] = serde_json::json!(invalid);
fs::write(&path, serde_json::to_vec(&data).unwrap())
.await
.unwrap();
assert!(mgr.app_signable_nostr_pubkeys("").await.is_err());
}
}
#[tokio::test]
async fn app_signable_nostr_pubkeys_is_empty_with_only_the_node_identity() {
let dir = tempdir().unwrap();
let mgr = IdentityManager::new(dir.path()).await.unwrap();
let node_key = SigningKey::generate(&mut OsRng);
let node_pubkey_hex = hex::encode(node_key.verifying_key().as_bytes());
mgr.create_from_signing_key("Node".to_string(), IdentityPurpose::Personal, node_key)
.await
.unwrap();
assert_eq!(
mgr.app_signable_nostr_pubkeys(&node_pubkey_hex)
.await
.unwrap(),
""
);
}
#[tokio::test]
async fn test_delete_default_shifts() {
let dir = tempdir().unwrap();
+6
View File
@@ -70,6 +70,8 @@ mod node_message;
mod nostr_discovery;
mod nostr_handshake;
mod nostr_relays;
#[cfg(test)]
mod nostr_security_tests;
mod peers;
mod port_allocator;
mod rate_limit;
@@ -256,6 +258,10 @@ async fn main() -> Result<()> {
boot_report.recovered, boot_report.total, boot_report.failed
);
}
// Disk manifests must be stable before the initial load and all later
// catalog reloads. Do not move this into the background doctor bootstrap.
bootstrap::ensure_runtime_assets_ready().await;
// Construct the container orchestrator once. In prod mode we load the
// on-disk app manifests, do an initial adoption pass, and spawn the
// BootReconciler loop (Step 5/6 of the rust-orchestrator migration).
+394 -68
View File
@@ -7,7 +7,8 @@
//! to a full chip erase before write.
//!
//! MeshCore and Meshtastic are flashed the same way: download a released
//! image, `esptool erase_flash`, then `esptool write_flash 0x0 <image>`.
//! image, verify it, then run `write_flash --erase-all 0x0 <image>` with
//! the packaged esptool executable.
//! Reticulum/RNode is different: `archy-rnodeconf --autoinstall` owns the
//! whole fetch+erase+flash+EEPROM-bootstrap sequence itself (confirmed live
//! via `archy-rnodeconf --help` — there is no raw esptool path exposed for
@@ -49,32 +50,18 @@ impl FlashBoard {
}
}
/// Map a detected USB vid:pid to a known flashable board, using the same
/// table as `image-recipe/configs/99-mesh-radio.rules`. CP2102 (10c4:ea60)
/// is confirmed there as Heltec V3's USB-UART bridge chip, and is safe to
/// auto-match since that vid:pid is bridge-chip-specific.
///
/// Heltec V4 is NOT auto-matchable and deliberately has no entry here: it
/// was confirmed live (real hardware, 2026-07-23) to use the ESP32-S3's
/// built-in native-USB JTAG/serial peripheral, reporting vid:pid 303a:1001
/// with product string "USB JTAG/serial debug unit" — that descriptor is
/// baked into the chip's ROM and is IDENTICAL across every ESP32-S3 board
/// with native USB enabled, not just Heltec V4. Adding `303a:1001 =>
/// HeltecV4` here would silently misidentify any other native-USB ESP32-S3
/// board (a T3-S3, a bare devkit, etc.) as a V4 and risk writing the wrong
/// board's image. Callers (the RPC layer / frontend) must let the user pick
/// the board manually whenever this returns `None`.
pub fn resolve_flash_board(info: &DetectedDeviceInfo) -> Option<FlashBoard> {
match (info.vid.as_deref(), info.pid.as_deref()) {
(Some("10c4"), Some("ea60")) => Some(FlashBoard::HeltecV3),
_ => None,
}
/// Generic CP2102 and native ESP32-S3 USB IDs identify adapters/chips, not
/// board wiring. Require an explicit board until a board-specific identity is
/// available; guessing a Heltec model can write incompatible firmware.
pub fn resolve_flash_board(_info: &DetectedDeviceInfo) -> Option<FlashBoard> {
None
}
#[derive(Debug, Clone, Copy, PartialEq, Eq, Serialize)]
#[serde(rename_all = "lowercase")]
pub enum FlashStage {
Downloading,
Preparing,
Erasing,
Writing,
Autoinstalling,
@@ -101,11 +88,10 @@ const LOG_TAIL_MAX: usize = 200;
/// start opening the port (which itself toggles DTR/RTS) again.
const POST_FLASH_SETTLE_DELAY: std::time::Duration = std::time::Duration::from_secs(5);
/// Absolute ceiling on a whole flash job (download + erase + write, or
/// autoinstall), regardless of what it's doing internally. Last-resort
/// safety net so a hang anywhere can't wedge the single-flash-job guard
/// forever — generous enough to never trigger on a legitimately slow
/// multi-hundred-MB transfer.
/// Deadline for preparation and warning threshold for the active flasher.
/// A download can be cancelled safely. An active write retains ownership until
/// its subprocess exits, even after this threshold: reporting an aborted job
/// while a detached writer continues would let a retry corrupt the device.
const MAX_JOB_DURATION: std::time::Duration = std::time::Duration::from_secs(15 * 60);
/// How long to wait for MeshService::stop() to release the serial port
@@ -247,6 +233,16 @@ fn firmware_cache_dir(data_dir: &Path) -> PathBuf {
data_dir.join("mesh").join("firmware-cache")
}
async fn invalidate_radio_settings_marker(data_dir: &Path) -> Result<()> {
// A full-chip flash erased the device's preferences. A previous host-side
// marker is no longer evidence that this radio has the requested settings.
match tokio::fs::remove_file(data_dir.join("meshcore-radio-params.json")).await {
Ok(()) => Ok(()),
Err(error) if error.kind() == std::io::ErrorKind::NotFound => Ok(()),
Err(error) => Err(error).context("Firmware written, but old radio-settings marker could not be cleared; reconnect is paused"),
}
}
/// No blanket `.timeout()` here on purpose: reqwest's request timeout covers
/// the *entire* request including streaming the response body, which would
/// kill a legitimate large download partway through (Meshtastic's esp32s3
@@ -314,6 +310,10 @@ pub async fn list_firmware(family: DeviceType) -> Result<Vec<String>> {
struct GithubAsset {
name: String,
browser_download_url: String,
#[serde(default)]
size: Option<u64>,
#[serde(default)]
digest: Option<String>,
}
#[derive(serde::Deserialize)]
@@ -322,8 +322,24 @@ struct GithubRelease {
assets: Vec<GithubAsset>,
}
async fn register_flash_job(handle: &FlashJobHandle, job: &Arc<FlashJob>) -> Result<()> {
// Keep checking and registering under one lock: concurrent RPCs must
// never start two writers on the same device.
let mut existing = handle.try_write().map_err(|_| anyhow::anyhow!(
"The radio is being inspected or reconfigured; wait for that operation to finish before flashing"
))?;
if let Some(current) = existing.as_ref() {
if !current.snapshot().await.done {
anyhow::bail!("A firmware flash is already in progress on this node");
}
}
*existing = Some(Arc::clone(job));
Ok(())
}
/// Start a flash job in the background. Returns as soon as the job has been
/// registered and the listener released — callers poll `FlashJobHandle` via
/// registered — preparation and listener shutdown run in the background.
/// Callers poll `FlashJobHandle` via
/// `mesh.flash-status` for progress. Only one job may be in flight at a time.
pub async fn start_flash_job(
handle: &FlashJobHandle,
@@ -333,21 +349,44 @@ pub async fn start_flash_job(
board: FlashBoard,
family: DeviceType,
) -> Result<()> {
{
let existing = handle.read().await;
if let Some(job) = existing.as_ref() {
if !job.snapshot().await.done {
anyhow::bail!("A firmware flash is already in progress on this node");
// Missing/corrupt tooling must fail before stopping a working radio or
// registering a job that can never reach the serial device.
if matches!(family, DeviceType::Meshtastic | DeviceType::Meshcore) {
preflight_esptool().await?;
}
}
}
let job = FlashJob::new(board, family, path.clone());
*handle.write().await = Some(Arc::clone(&job));
register_flash_job(handle, &job).await?;
let bg_job = Arc::clone(&job);
let bg_service = Arc::clone(mesh_service);
let task = tokio::spawn(async move {
// Downloads and checksum checks do not need exclusive serial access.
// Keep a working listener alive if upstream/download verification fails.
let prepared_image = if matches!(family, DeviceType::Meshcore | DeviceType::Meshtastic) {
match tokio::time::timeout(
MAX_JOB_DURATION,
fetch_esptool_image(board, family, &data_dir, &bg_job),
)
.await
{
Ok(Ok(image)) => Some(image),
result => {
let error = match result {
Ok(Err(error)) => error,
_ => anyhow::anyhow!(
"Firmware download exceeded the time limit; radio was not changed"
),
};
bg_job.fail(&error).await;
return;
}
}
} else {
None
};
// Cancellation is safe during download. Once listener shutdown starts,
// allow that bounded operation to finish instead of orphaning its task.
bg_job.set_stage(FlashStage::Preparing).await;
// esptool/archy-rnodeconf need exclusive serial access — release
// the listener's hold on the port before touching it. This USED
// TO run synchronously in start_flash_job before the job was even
@@ -404,17 +443,31 @@ pub async fn start_flash_job(
// subsequent mesh.flash-device call failed with "already in
// progress" until the service was restarted). Generous enough that
// a legitimately slow multi-hundred-MB transfer still completes.
let result = match tokio::time::timeout(
MAX_JOB_DURATION,
run_flash(board, family, &data_dir, &path, &bg_job),
)
.await
{
let flash = run_flash(
board,
family,
&data_dir,
&path,
prepared_image.as_deref(),
&bg_job,
);
tokio::pin!(flash);
let result = match tokio::time::timeout(MAX_JOB_DURATION, &mut flash).await {
Ok(inner) => inner,
Err(_) => Err(anyhow::anyhow!(
"Flash job exceeded the {}-minute ceiling — aborted",
MAX_JOB_DURATION.as_secs() / 60
)),
Err(_) if bg_job.snapshot().await.stage == FlashStage::Downloading => Err(
anyhow::anyhow!("Firmware download exceeded the time limit; radio was not written"),
),
Err(_) => {
// Dropping this future does NOT stop its flash subprocess.
// Keep the job busy until it exits, rather than allowing a
// second writer while the first one still owns the device.
bg_job.push_log("Flashing is taking longer than expected; waiting for the active tool to exit before allowing another operation").await;
flash.await
}
};
let result = match result {
Ok(()) => invalidate_radio_settings_marker(&data_dir).await,
error => error,
};
let succeeded = result.is_ok();
@@ -423,7 +476,6 @@ pub async fn start_flash_job(
bg_job
.push_log("Flash completed successfully".to_string())
.await;
bg_job.finish().await;
info!(path = %path, board = ?board, family = %family, "LoRa firmware flash succeeded");
}
Err(e) => {
@@ -434,7 +486,6 @@ pub async fn start_flash_job(
// erase_flash failed", no actual esptool stderr).
warn!(path = %path, error = %format!("{e:#}"), "LoRa firmware flash failed");
bg_job.push_log(format!("ERROR: {e:#}")).await;
bg_job.fail(e).await;
}
}
@@ -450,6 +501,9 @@ pub async fn start_flash_job(
}
if !succeeded {
if let Err(error) = &result {
bg_job.fail(error).await;
}
// Deliberately do NOT auto-restart the listener here. A failed
// flash means we can't vouch for the board's state — reopening
// the port immediately (esptool/rnodeconf's own reset sequence
@@ -499,6 +553,7 @@ pub async fn start_flash_job(
Err(e) => warn!(error = %e, "Failed to load mesh config after flash"),
}
}
bg_job.finish().await;
});
*job.abort_handle.write().await = Some(task.abort_handle());
@@ -510,12 +565,13 @@ async fn run_flash(
family: DeviceType,
data_dir: &Path,
path: &str,
prepared_image: Option<&Path>,
job: &Arc<FlashJob>,
) -> Result<()> {
match family {
DeviceType::Meshtastic | DeviceType::Meshcore => {
let image = fetch_esptool_image(board, family, data_dir, job).await?;
esptool_erase_and_write(path, &image, job).await
let image = prepared_image.context("Firmware was not prepared before serial access")?;
esptool_erase_and_write(path, image, job).await
}
DeviceType::Reticulum => {
let lora_region = super::load_config(data_dir)
@@ -664,15 +720,65 @@ async fn fetch_meshcore_image(
anyhow::anyhow!("No matching MeshCore image in release {}", release.tag_name)
})?;
anyhow::ensure!(
Path::new(&asset.name)
.file_name()
.and_then(|name| name.to_str())
== Some(asset.name.as_str()),
"Invalid firmware asset filename"
);
let out_path = cache.join(&asset.name);
if tokio::fs::metadata(&out_path).await.is_ok() {
job.push_log(format!("Using cached {}", asset.name)).await;
if verify_meshcore_asset(&out_path, asset).await.is_ok() {
job.push_log(format!("Using verified cached {}", asset.name))
.await;
return Ok(out_path);
}
tokio::fs::remove_file(&out_path)
.await
.context("Removing invalid cached firmware")?;
job.push_log("Cached firmware failed verification; downloading a fresh copy")
.await;
}
download_to_file(client, &asset.browser_download_url, &out_path, job).await?;
if let Err(error) = verify_meshcore_asset(&out_path, asset).await {
// A partial/unverified download must not become next attempt's cache.
let _ = tokio::fs::remove_file(&out_path).await;
return Err(error);
}
Ok(out_path)
}
async fn verify_meshcore_asset(path: &Path, asset: &GithubAsset) -> Result<()> {
use sha2::{Digest, Sha256};
let size = asset
.size
.context("MeshCore release did not provide firmware size")?;
anyhow::ensure!(
(1..=16 * 1024 * 1024).contains(&size),
"Invalid MeshCore firmware size"
);
anyhow::ensure!(
tokio::fs::metadata(path).await?.len() == size,
"Firmware size mismatch; radio was not written"
);
let expected = asset
.digest
.as_deref()
.and_then(|value| value.strip_prefix("sha256:"))
.context("MeshCore release did not provide a SHA-256 checksum")?;
anyhow::ensure!(
expected.len() == 64 && expected.bytes().all(|byte| byte.is_ascii_hexdigit()),
"Invalid MeshCore release checksum"
);
let actual = hex::encode(Sha256::digest(tokio::fs::read(path).await?));
anyhow::ensure!(
actual.eq_ignore_ascii_case(expected),
"Firmware checksum mismatch; radio was not written"
);
Ok(())
}
async fn download_to_file(
client: &reqwest::Client,
url: &str,
@@ -722,7 +828,8 @@ async fn download_to_file(
}
}
}
file.flush().await.ok();
file.flush().await.context("Flushing firmware download")?;
file.sync_all().await.context("Saving firmware download")?;
tokio::fs::rename(&tmp, dest)
.await
.context("Finalizing firmware download")?;
@@ -773,19 +880,10 @@ async fn esptool_erase_and_write(path: &str, image: &Path, job: &Arc<FlashJob>)
/// Building global args separately from subcommand args keeps this correct
/// by construction instead of relying on call-site ordering.
///
/// Normal stub-loader mode (no --no-stub) needs the esp32s3 stub flasher
/// blob at /usr/lib/python3/dist-packages/esptool/targets/stub_flasher/
/// stub_flasher_32s3.json — Debian's `esptool` package (4.7.0+dfsg-0.1)
/// ships without it (stripped for DFSG compliance: the prebuilt blob has no
/// buildable-from-source path Debian could verify), so scripts/self-update.sh
/// fetches the exact same file from the matching upstream esptool release
/// tag and installs it alongside the apt package (see the esptool install
/// step there). --no-stub (talk directly to the ROM bootloader, skip the
/// stub) was tried first and works for connecting, but the ROM bootloader
/// doesn't implement a full-chip-erase opcode at all — only the stub does —
/// so --no-stub broke our "always erase before write" default outright
/// rather than just being slower. Restoring the real stub file is the
/// correct fix, not routing around its absence.
/// Normal stub-loader mode is required for full-chip erase. The packaged
/// archy-esptool includes and self-tests Espressif's ESP32-S3 stub. Debian's
/// stripped esptool package alone did not provide that resource, so changing
/// flags to --no-stub cannot repair this operation.
fn esptool_global_args<'a>(path: &'a str, baud: Option<&'a str>) -> Vec<&'a str> {
let mut args = vec!["--chip", ESPTOOL_CHIP, "--port", path];
if let Some(b) = baud {
@@ -795,24 +893,96 @@ fn esptool_global_args<'a>(path: &'a str, baud: Option<&'a str>) -> Vec<&'a str>
args
}
fn esptool_executable() -> Result<PathBuf> {
let mut candidates = vec![PathBuf::from("/usr/local/bin/archy-esptool")];
if let Some(paths) = std::env::var_os("PATH") {
for directory in std::env::split_paths(&paths) {
for name in ["esptool", "esptool.py"] {
candidates.push(directory.join(name));
}
}
}
executable_from_candidates(candidates)
}
fn executable_from_candidates(candidates: impl IntoIterator<Item = PathBuf>) -> Result<PathBuf> {
use std::os::unix::fs::PermissionsExt;
candidates.into_iter().find(|path| {
path.is_file() && path.metadata().is_ok_and(|metadata| metadata.permissions().mode() & 0o111 != 0)
}).ok_or_else(|| anyhow::anyhow!(
"Radio flashing tools are missing. Install the complete Archipelago update and retry; the radio has not been changed."
))
}
async fn preflight_esptool() -> Result<PathBuf> {
let executable = esptool_executable()?;
check_esptool(&executable).await?;
Ok(executable)
}
async fn check_esptool(executable: &Path) -> Result<()> {
let mut command = Command::new(executable);
command
.arg(
if executable
.file_name()
.is_some_and(|name| name == "archy-esptool")
{
"--archy-self-test"
} else {
"version"
},
)
.kill_on_drop(true);
let output = tokio::time::timeout(std::time::Duration::from_secs(20), command.output())
.await
.context("Radio flashing tool did not respond; the radio has not been changed")?
.context("Radio flashing tool could not start; check its installation and permissions")?;
anyhow::ensure!(
output.status.success(),
"Radio flashing tool self-check failed; reinstall the complete update before retrying"
);
Ok(())
}
fn retryable_flash_error(error: &anyhow::Error) -> bool {
if error
.chain()
.any(|cause| cause.downcast_ref::<std::io::Error>().is_some())
{
return false;
}
let detail = format!("{error:#}").to_lowercase();
[
"failed to connect",
"timed out waiting",
"invalid head of packet",
"serial data stream stopped",
]
.iter()
.any(|message| detail.contains(message))
}
async fn esptool_with_retry(path: &str, subcommand: &[&str], job: &Arc<FlashJob>) -> Result<()> {
let mut cmd = Command::new("esptool");
let executable = preflight_esptool().await?;
let mut cmd = Command::new(&executable);
cmd.args(esptool_global_args(path, None));
cmd.args(subcommand);
match run_streamed(cmd, None, job).await {
Ok(()) => Ok(()),
Err(first_err) => {
Err(first_err) if retryable_flash_error(&first_err) => {
job.push_log(format!(
"First attempt failed ({first_err:#}); retrying once at {ESPTOOL_FALLBACK_BAUD} baud"
))
.await;
let mut retry = Command::new("esptool");
let mut retry = Command::new(&executable);
retry.args(esptool_global_args(path, Some(ESPTOOL_FALLBACK_BAUD)));
retry.args(subcommand);
run_streamed(retry, None, job)
.await
.context(format!("retry also failed (first attempt: {first_err:#})"))
}
Err(error) => Err(error),
}
}
@@ -990,3 +1160,159 @@ async fn run_streamed(mut cmd: Command, stdin: Option<Vec<u8>>, job: &Arc<FlashJ
}
Ok(())
}
#[cfg(test)]
mod flashing_regression_tests {
use super::*;
#[tokio::test]
async fn full_flash_invalidates_only_radio_settings_marker() {
let dir = tempfile::tempdir().unwrap();
let marker = dir.path().join("meshcore-radio-params.json");
tokio::fs::write(&marker, b"old settings").await.unwrap();
let other = dir.path().join("mesh-config.json");
tokio::fs::write(&other, b"preserved").await.unwrap();
invalidate_radio_settings_marker(dir.path()).await.unwrap();
assert!(!marker.exists());
assert_eq!(tokio::fs::read(&other).await.unwrap(), b"preserved");
invalidate_radio_settings_marker(dir.path()).await.unwrap();
tokio::fs::create_dir(&marker).await.unwrap();
assert!(invalidate_radio_settings_marker(dir.path()).await.is_err());
}
use std::os::unix::fs::PermissionsExt;
#[tokio::test]
async fn meshcore_cache_requires_release_size_and_checksum() {
use sha2::{Digest, Sha256};
let dir = tempfile::tempdir().unwrap();
let file = dir.path().join("fixture.bin");
tokio::fs::write(&file, b"firmware fixture").await.unwrap();
let mut asset = GithubAsset {
name: "fixture.bin".into(),
browser_download_url: "https://example.invalid/fixture.bin".into(),
size: Some(16),
digest: Some(format!(
"sha256:{}",
hex::encode(Sha256::digest(b"firmware fixture"))
)),
};
assert!(verify_meshcore_asset(&file, &asset).await.is_ok());
tokio::fs::write(&file, b"tampered fixture").await.unwrap();
assert!(verify_meshcore_asset(&file, &asset).await.is_err());
tokio::fs::write(&file, b"short").await.unwrap();
assert!(verify_meshcore_asset(&file, &asset).await.is_err());
tokio::fs::write(&file, b"firmware fixture").await.unwrap();
asset.digest = None;
assert!(verify_meshcore_asset(&file, &asset).await.is_err());
}
#[test]
fn generic_usb_ids_do_not_select_firmware() {
for (vid, pid) in [("10c4", "ea60"), ("303a", "1001")] {
let info = DetectedDeviceInfo {
path: "/dev/fixture".into(),
vid: Some(vid.into()),
pid: Some(pid.into()),
product: None,
manufacturer: None,
plugged_at: None,
};
assert_eq!(resolve_flash_board(&info), None);
}
}
#[test]
fn absent_nonexecutable_and_directory_candidates_are_rejected() {
let dir = tempfile::tempdir().unwrap();
let file = dir.path().join("flasher");
std::fs::write(&file, "#!/bin/sh\nexit 0\n").unwrap();
std::fs::set_permissions(&file, std::fs::Permissions::from_mode(0o600)).unwrap();
assert!(executable_from_candidates([
dir.path().join("absent"),
file.clone(),
dir.path().to_path_buf()
])
.is_err());
std::fs::set_permissions(&file, std::fs::Permissions::from_mode(0o700)).unwrap();
assert_eq!(executable_from_candidates([file.clone()]).unwrap(), file);
}
#[tokio::test]
async fn flasher_preflight_reports_missing_interpreter_and_failed_self_check() {
let dir = tempfile::tempdir().unwrap();
let file = dir.path().join("archy-esptool");
for script in ["#!/missing/python\n", "#!/bin/sh\nexit 7\n"] {
std::fs::write(&file, script).unwrap();
std::fs::set_permissions(&file, std::fs::Permissions::from_mode(0o700)).unwrap();
assert!(check_esptool(&file).await.is_err());
}
std::fs::write(&file, "#!/bin/sh\n[ \"$1\" = --archy-self-test ]\n").unwrap();
assert!(check_esptool(&file).await.is_ok());
}
#[tokio::test]
async fn flash_registration_excludes_other_writers_and_active_probes() {
let handle = new_job_handle();
let first = FlashJob::new(
FlashBoard::HeltecV3,
DeviceType::Meshcore,
"/dev/fixture".into(),
);
let second = FlashJob::new(
FlashBoard::HeltecV3,
DeviceType::Meshcore,
"/dev/fixture".into(),
);
let probe = handle.read().await;
assert!(register_flash_job(&handle, &first).await.is_err());
drop(probe);
let (a, b) = tokio::join!(
register_flash_job(&handle, &first),
register_flash_job(&handle, &second)
);
assert_eq!(usize::from(a.is_ok()) + usize::from(b.is_ok()), 1);
handle.read().await.as_ref().unwrap().finish().await;
let next = FlashJob::new(
FlashBoard::HeltecV3,
DeviceType::Meshcore,
"/dev/fixture".into(),
);
assert!(register_flash_job(&handle, &next).await.is_ok());
}
#[test]
fn retries_only_known_serial_transport_failures() {
for kind in [
std::io::ErrorKind::NotFound,
std::io::ErrorKind::PermissionDenied,
] {
assert!(!retryable_flash_error(
&anyhow::Error::from(std::io::Error::from(kind))
.context("Failed to start subprocess")
));
}
for message in [
"Wrong chip",
"Invalid image",
"module missing",
"permission denied",
"port is busy",
] {
assert!(!retryable_flash_error(&anyhow::anyhow!(message)));
}
assert!(retryable_flash_error(&anyhow::anyhow!(
"Failed to connect to ESP32-S3: timed out waiting for packet header"
)));
assert_eq!(
esptool_global_args("/dev/fixture", Some("115200")),
[
"--chip",
"esp32s3",
"--port",
"/dev/fixture",
"--baud",
"115200"
]
);
}
}
+54 -5
View File
@@ -1092,6 +1092,14 @@ impl MeshService {
let (new_tx, new_rx) = tokio::sync::mpsc::channel(32);
*self.state.cmd_tx.write().await = new_tx;
self.cmd_rx = Some(new_rx);
{
let mut status = self.state.status.write().await;
status.device_connected = false;
status.device_path = None;
status.firmware_version = None;
status.self_node_id = None;
status.peer_count = 0;
}
info!("Mesh service stopped");
}
@@ -2321,7 +2329,10 @@ impl MeshService {
}
}
let was_enabled = self.config.enabled;
let listener_running = self
.listener_handle
.as_ref()
.is_some_and(|handle| !handle.is_finished());
let needs_session_restart = session_config_changed(&self.config, &config);
self.config = config.clone();
@@ -2335,10 +2346,13 @@ impl MeshService {
.unwrap_or_else(|| "archipelago".to_string());
}
// If enabled state changed, start/stop the listener
if config.enabled && !was_enabled {
// Reconcile desired state with the actual task, not the old enabled
// flag. A failed flash can stop the task while keeping enabled=true;
// explicitly reconnecting with the same settings must restart it.
if config.enabled && !listener_running {
self.stop().await;
self.start()?;
} else if !config.enabled && was_enabled {
} else if !config.enabled {
self.stop().await;
// Clear connected state
let mut status = self.state.status.write().await;
@@ -2347,7 +2361,7 @@ impl MeshService {
status.firmware_version = None;
status.self_node_id = None;
status.peer_count = 0;
} else if config.enabled && was_enabled && needs_session_restart {
} else if needs_session_restart {
info!("Mesh session config changed — restarting listener to apply");
self.stop().await;
self.start()?;
@@ -2537,6 +2551,41 @@ mod tests {
}
use super::*;
#[tokio::test]
async fn reconnect_with_unchanged_enabled_config_restarts_stopped_listener() {
let dir = tempfile::tempdir().unwrap();
let key = SigningKey::from_bytes(&[7; 32]);
let public = hex::encode(key.verifying_key().to_bytes());
let did = crate::identity::did_key_from_pubkey_hex(&public).unwrap();
let config = MeshConfig {
enabled: true,
..Default::default()
};
save_config(dir.path(), &config).await.unwrap();
let mut service = MeshService::new(dir.path(), &key, &did, &public)
.await
.unwrap();
assert!(service.listener_handle.is_none());
service.configure(config.clone()).await.unwrap();
assert!(service.listener_handle.is_some());
service.stop().await;
assert!(service.config.enabled);
service.configure(config.clone()).await.unwrap();
assert!(service.listener_handle.is_some());
service.stop().await;
service.listener_handle = Some(tokio::spawn(async {}));
tokio::task::yield_now().await;
service.configure(config).await.unwrap();
assert!(!service.listener_handle.as_ref().unwrap().is_finished());
service.stop().await;
let disabled = MeshConfig {
enabled: false,
..Default::default()
};
service.configure(disabled).await.unwrap();
assert!(service.listener_handle.is_none());
}
#[test]
fn session_config_change_detection() {
let base = MeshConfig::default();
+42
View File
@@ -378,6 +378,19 @@ fn decode_mesh_name(bytes: &[u8], fallback: &str) -> String {
/// Parse RESP_DEVICE_INFO (0x0D) response.
/// Returns firmware version string and device capabilities.
pub fn parse_device_info(data: &[u8]) -> Result<(String, u16)> {
// Official companion v3+ binary layout: protocol, half-capacity,
// channels, PIN (4), build date (12), model (40), version (20).
// Verified against companion-v1.17.1 MyMesh.cpp and Heltec V3 readback.
if data
.first()
.is_some_and(|version| (3..=31).contains(version))
{
anyhow::ensure!(data.len() >= 79, "Truncated MeshCore device info");
return Ok((
decode_mesh_name(&data[59..79], "unknown"),
u16::from(data[1]) * 2,
));
}
// Device info format varies by firmware version.
// Minimum: firmware version string (null-terminated) + max_contacts (u16 LE)
if data.is_empty() {
@@ -404,6 +417,15 @@ pub fn parse_self_info(data: &[u8]) -> Result<(u32, String)> {
anyhow::bail!("Self info response too short: {} bytes", data.len());
}
// Current companions send type/power/max-power, public key (32),
// position (8), four preference bytes, RF parameters (10), then name.
if data.len() >= 57 {
let node_id = u32::from_le_bytes(data[3..7].try_into().unwrap());
return Ok((
node_id,
decode_mesh_name(&data[57..], &format!("node-{node_id:08x}")),
));
}
let node_id = u32::from_le_bytes([data[0], data[1], data[2], data[3]]);
// Name follows after fixed fields. A firmware whose fixed-field layout
@@ -966,4 +988,24 @@ mod tests {
fn test_parse_self_info_too_short() {
assert!(parse_self_info(&[0x01, 0x02]).is_err());
}
#[test]
fn current_companion_binary_metadata_is_not_a_name_or_version() {
let mut info = vec![0u8; 81];
info[0] = 13;
info[1] = 150;
info[19..28].copy_from_slice(b"Heltec V3");
info[59..74].copy_from_slice(b"v1.17.1-d929643");
assert_eq!(
parse_device_info(&info).unwrap(),
("v1.17.1-d929643".into(), 300)
);
assert!(parse_device_info(&info[..78]).is_err());
let mut own = vec![0u8; 57];
own[0..3].copy_from_slice(&[1, 22, 22]);
own[3..7].copy_from_slice(&42u32.to_le_bytes());
own[47..51].copy_from_slice(&869618u32.to_le_bytes());
own.extend_from_slice(b"My radio");
assert_eq!(parse_self_info(&own).unwrap(), (42, "My radio".into()));
}
}
+48 -11
View File
@@ -277,6 +277,19 @@ fn terminate_group(child: &Child) {
}
}
/// Own the daemon during its asynchronous handshake too. Cancellation drops
/// the future without executing an error branch; a bare Child would survive
/// and keep the serial port open even though the listener had stopped.
struct StartingDaemon(Option<Child>);
impl Drop for StartingDaemon {
fn drop(&mut self) {
if let Some(child) = self.0.as_ref() {
terminate_group(child);
}
}
}
/// One peer learned via an RNS announce (LXMF delivery destination).
#[derive(Clone)]
struct ReticulumPeer {
@@ -517,10 +530,10 @@ impl ReticulumLink {
let child = cmd
.spawn()
.context("Failed to spawn reticulum-daemon — is it installed/packaged?")?;
let mut starting = StartingDaemon(Some(child));
// Wait for the socket to appear, then for the daemon's "ready" event.
// Runs as a block so every failure path tears the just-spawned daemon
// group down via `terminate_group` (the child has no `kill_on_drop`).
// StartingDaemon tears the group down on errors AND cancellation.
let init = async {
let deadline = tokio::time::Instant::now() + Duration::from_secs(15);
let stream = loop {
@@ -560,20 +573,17 @@ impl ReticulumLink {
dest_hash = %dest_hash_hex,
"Reticulum daemon ready"
);
Ok((write_half, reader, dest_hash, display_name))
};
let (write_half, reader, dest_hash, display_name) = match init.await {
Ok(parts) => parts,
Err(e) => {
terminate_group(&child);
return Err(e);
}
Ok::<_, anyhow::Error>((write_half, reader, dest_hash, display_name))
};
let (write_half, reader, dest_hash, display_name) = init.await?;
let mut link = Self {
device_path: label,
socket_path,
child,
child: starting
.0
.take()
.expect("starting daemon is owned until ready"),
writer: write_half,
reader,
dest_hash,
@@ -1557,6 +1567,33 @@ impl Drop for ReticulumLink {
mod tests {
use super::*;
#[tokio::test]
async fn cancelled_daemon_handshake_terminates_child() {
let (started, ready) = tokio::sync::oneshot::channel();
let task = tokio::spawn(async move {
let child = Command::new("sleep")
.arg("60")
.process_group(0)
.spawn()
.unwrap();
let pid = child.id().unwrap();
let _starting = StartingDaemon(Some(child));
started.send(pid).unwrap();
std::future::pending::<()>().await;
});
let pid = ready.await.unwrap();
assert_eq!(unsafe { libc::kill(pid as i32, 0) }, 0);
task.abort();
assert!(task.await.unwrap_err().is_cancelled());
tokio::time::timeout(Duration::from_secs(3), async {
while unsafe { libc::kill(pid as i32, 0) } == 0 {
tokio::time::sleep(Duration::from_millis(20)).await;
}
})
.await
.expect("cancelled handshake left its child alive");
}
#[test]
fn announced_name_precedence() {
// Daemon-decoded LXMF name always wins.
+6 -2
View File
@@ -146,12 +146,16 @@ impl MeshcoreDevice {
}
}
let info = DeviceInfo {
firmware_version: name.clone(),
let mut info = DeviceInfo {
firmware_version: "unknown".to_string(),
node_id,
max_contacts: 100,
device_type: super::types::DeviceType::Meshcore,
};
if let Some((version, capacity)) = self.query_device_info().await {
info.firmware_version = version;
info.max_contacts = capacity;
}
self.device_info = Some(info.clone());
info!("Meshcore initialization complete on {}", self.device_path);
@@ -0,0 +1,124 @@
//! Compatibility and hostile-relay regression tests for the 0.44 security update.
//! Loopback sockets run only through scripts/test-backend-isolated.sh.
use nostr_sdk::prelude::*;
#[test]
fn native_signer_encryption_remains_compatible_and_rejects_hostile_payloads() {
let alice = Keys::generate();
let bob = Keys::generate();
let message = "Archipelago signing compatibility — \u{1f30a}";
let encrypted = nip44::encrypt(
alice.secret_key(),
&bob.public_key(),
message,
nip44::Version::V2,
)
.unwrap();
assert_eq!(
nip44::decrypt(bob.secret_key(), &alice.public_key(), &encrypted).unwrap(),
message
);
let encrypted = nip04::encrypt(alice.secret_key(), &bob.public_key(), message).unwrap();
assert_eq!(
nip04::decrypt(bob.secret_key(), &alice.public_key(), encrypted).unwrap(),
message
);
// Valid v2 prefix followed by a payload exceeding the codec's maximum.
// This must fail at the size gate, before allocation/decoding/HMAC work.
let oversized = format!("AgAA{}", "A".repeat(100_000));
assert!(matches!(
nip44::decrypt(bob.secret_key(), &alice.public_key(), oversized),
Err(nip44::Error::MessageTooLong)
));
for malformed in ["", "Ag==", "not base64!", "?iv=", "YQ==?iv=YQ=="] {
assert!(nip04::decrypt(bob.secret_key(), &alice.public_key(), malformed).is_err());
assert!(nip44::decrypt(bob.secret_key(), &alice.public_key(), malformed).is_err());
}
}
#[tokio::test]
async fn relay_cannot_substitute_forged_fields_for_a_known_event_id() {
use futures_util::{SinkExt, StreamExt};
use tokio::net::TcpListener;
use tokio_tungstenite::{accept_async, tungstenite::Message};
let test = async {
let author = Keys::generate();
let known = EventBuilder::text_note("already verified")
.sign_with_keys(&author)
.unwrap();
let valid = EventBuilder::text_note("new legitimate message")
.sign_with_keys(&author)
.unwrap();
let mut forged_content = known.clone();
forged_content.content = "forged content".into();
let mut forged_author = known.clone();
forged_author.pubkey = Keys::generate().public_key();
let mut forged_signature = known.clone();
forged_signature.sig = valid.sig;
for forged in [&forged_content, &forged_author, &forged_signature] {
assert!(forged.verify().is_err());
}
let listener = TcpListener::bind("127.0.0.1:0").await.unwrap();
let url = format!("ws://{}", listener.local_addr().unwrap());
let expected_id = valid.id;
let relay = tokio::spawn(async move {
let (socket, _) = listener.accept().await.unwrap();
let mut socket = accept_async(socket).await.unwrap();
while let Some(message) = socket.next().await {
let text = match message.unwrap() {
Message::Text(text) => text,
Message::Ping(payload) => {
socket.send(Message::Pong(payload)).await.unwrap();
continue;
}
Message::Close(_) => break,
_ => continue,
};
let message: serde_json::Value = serde_json::from_str(&text).unwrap();
if message[0] != "REQ" {
continue;
}
let subscription = message[1].as_str().unwrap();
for event in [&forged_content, &forged_author, &forged_signature, &valid] {
socket
.send(Message::Text(
serde_json::json!(["EVENT", subscription, event]).to_string(),
))
.await
.unwrap();
}
socket
.send(Message::Text(
serde_json::json!(["EOSE", subscription]).to_string(),
))
.await
.unwrap();
}
});
let client = Client::new(Keys::generate());
client.database().save_event(&known).await.unwrap();
client.add_relay(&url).await.unwrap();
client
.try_connect_relay(&url, std::time::Duration::from_secs(3))
.await
.unwrap();
let events = client
.fetch_events(
Filter::new().kind(Kind::TextNote),
std::time::Duration::from_secs(5),
)
.await
.unwrap();
assert_eq!(
events.len(),
1,
"forged events must never reach SDK consumers"
);
assert_eq!(events.iter().next().unwrap().id, expected_id);
client.disconnect().await;
relay.abort();
};
tokio::time::timeout(std::time::Duration::from_secs(15), test)
.await
.expect("local relay test timed out");
}
+2
View File
@@ -83,6 +83,8 @@ impl EndpointRateLimiter {
limits.insert("wallet.send".to_string(), (5usize, 300u64));
limits.insert("wallet.ecash-send".to_string(), (10, 300));
limits.insert("lnd.sendcoins".to_string(), (5, 300));
limits.insert("lnd.bump-submit".to_string(), (5, 300));
limits.insert("lnd.bump-quote".to_string(), (30, 60));
limits.insert("lnd.payinvoice".to_string(), (10, 300));
limits.insert("lnd.openchannel".to_string(), (3, 300));
limits.insert("lnd.closechannel".to_string(), (3, 300));
+137 -1
View File
@@ -1475,6 +1475,48 @@ pub fn is_peer_allowed_path(path: &str) -> bool {
|| path.starts_with("/dwn/")
}
/// The ordinary API listener is a management surface even when contacted
/// directly, without host nginx. Peer traffic has its own path-restricted
/// listener and retains its existing cryptographic authentication.
fn management_peer_is_private(address: std::net::IpAddr) -> bool {
match address {
std::net::IpAddr::V4(ip) => {
ip.is_loopback()
|| ip.is_private()
|| ip.is_link_local()
|| (ip.octets()[0] == 100 && (64..=127).contains(&ip.octets()[1]))
}
std::net::IpAddr::V6(ip) => {
if let Some(mapped) = ip.to_ipv4_mapped() {
management_peer_is_private(std::net::IpAddr::V4(mapped))
} else {
ip.is_loopback() || ip.is_unique_local() || ip.is_unicast_link_local()
}
}
}
}
fn request_surface_allowed(
peer_only: bool,
peer: std::net::IpAddr,
request: &hyper::Request<hyper::Body>,
) -> bool {
if peer_only {
return is_peer_allowed_path(request.uri().path());
}
// Keep purpose-built content/peer HTTP endpoints reachable with their
// existing handler-level checks. The general RPC dispatcher is a management
// surface here; only the dedicated peer listener retains public peer RPC.
if request.uri().path() != "/rpc/v1" && is_peer_allowed_path(request.uri().path()) {
return true;
}
management_peer_is_private(peer)
&& !request
.headers()
.get("x-archipelago-public-ingress")
.is_some_and(|value| !value.as_bytes().is_empty())
}
async fn accept_loop(
handler: Arc<ApiHandler>,
listener: TcpListener,
@@ -1552,7 +1594,7 @@ async fn accept_loop(
// forwarded headers on loopback (nginx) connections.
req.extensions_mut()
.insert(crate::api::rpc::PeerAddr(peer_addr));
if peer_only && !is_peer_allowed_path(req.uri().path()) {
if !request_surface_allowed(peer_only, peer_addr.ip(), &req) {
let resp = hyper::Response::builder()
.status(hyper::StatusCode::NOT_FOUND)
.body(hyper::Body::empty())
@@ -2520,3 +2562,97 @@ mod merge_tests {
);
}
}
#[cfg(test)]
mod management_surface_tests {
use super::*;
#[test]
fn public_api_listener_rejects_management_despite_forged_headers() {
for peer in [
"198.18.0.2",
"2001:db8::2",
"::ffff:198.18.0.2",
"100.63.255.255",
"100.128.0.1",
] {
for path in ["/", "/login", "/assets/index.js", "/rpc/v1", "/ws"] {
for method in ["GET", "POST"] {
let request = hyper::Request::builder()
.uri(path)
.method(method)
.header("host", "127.0.0.1")
.header("x-forwarded-for", "127.0.0.1")
.header("x-real-ip", "192.168.1.10")
.body(hyper::Body::empty())
.unwrap();
assert!(
!request_surface_allowed(false, peer.parse().unwrap(), &request),
"{peer} {method} {path}"
);
}
}
}
}
#[test]
fn private_management_and_restricted_peer_transport_remain_available() {
let mut request = hyper::Request::builder()
.uri("/rpc/v1")
.body(hyper::Body::empty())
.unwrap();
for peer in [
"127.0.0.1",
"10.0.0.2",
"172.16.0.2",
"192.168.1.2",
"169.254.1.2",
"100.64.0.1",
"100.127.255.254",
"::1",
"fd00::1",
"fe80::1",
"::ffff:192.168.1.2",
] {
assert!(request_surface_allowed(
false,
peer.parse().unwrap(),
&request
));
}
request
.headers_mut()
.insert("x-archipelago-public-ingress", "1".parse().unwrap());
assert!(!request_surface_allowed(
false,
"127.0.0.1".parse().unwrap(),
&request
));
// The dedicated peer listener retains its existing signed RPC contract.
assert!(request_surface_allowed(
true,
"198.18.0.2".parse().unwrap(),
&request
));
for path in [
"/content",
"/content/fixture/invoice",
"/blob/fixture",
"/dwn/health",
"/archipelago/node-message",
] {
*request.uri_mut() = path.parse().unwrap();
assert!(request_surface_allowed(
false,
"198.18.0.2".parse().unwrap(),
&request
));
}
*request.uri_mut() = "/login".parse().unwrap();
assert!(!request_surface_allowed(
true,
"198.18.0.2".parse().unwrap(),
&request
));
}
}
+21
View File
@@ -2059,6 +2059,25 @@ pub async fn rollback_update(data_dir: &Path) -> Result<()> {
let backup_binary = backup_dir.join("archipelago");
if backup_binary.exists() {
// The restored frontend can contain a pre-guard runtime template. An
// older binary copies that template verbatim on startup, undoing live
// containment. Protect it before permitting the binary downgrade.
let template = "/opt/archipelago/web-ui/archipelago-runtime/image-recipe/configs/nginx-archipelago.conf";
if Path::new(template).exists() {
let protected = host_sudo(&[
"python3",
"-c",
include_str!("../../../scripts/dashboard-public-guard.py"),
"--protect-template",
template,
])
.await
.context("protect nginx runtime template before rollback")?;
anyhow::ensure!(
protected.success(),
"unsafe nginx rollback template; previous binary not restored"
);
}
// Same two namespace gotchas as apply_update()'s binary swap:
// `cp` straight onto the running binary is O_TRUNC and fails
// ETXTBSY (exit 1 — exactly what broke the .116 rollback), and
@@ -2654,6 +2673,8 @@ mod tests {
#[test]
fn test_is_newer() {
assert!(is_newer("1.9.0-alpha", "1.8.22-alpha"));
assert!(!is_newer("1.9.0-alpha", "1.9.0-alpha"));
assert!(is_newer("1.7.19-alpha", "1.7.18-alpha"));
assert!(is_newer("1.8.0-alpha", "1.7.99-alpha"));
assert!(is_newer("1.7.10-alpha", "1.7.9-alpha")); // numeric, not lexical
+135 -15
View File
@@ -263,7 +263,8 @@ pub struct ContainerConfig {
/// Derived-env entry. The template is rendered against `HostFacts` at
/// apply time; exactly one `{{PLACEHOLDER}}` occurrence per supported
/// fact name is allowed (host_ip, host_mdns, disk_gb).
/// fact name is allowed (host_ip, host_mdns, disk_gb, bitcoin_host,
/// node_identity_pubkeys).
#[derive(Debug, Clone, Serialize, Deserialize, PartialEq, Eq)]
pub struct DerivedEnv {
pub key: String,
@@ -989,6 +990,24 @@ impl AppManifest {
validate_security(&self.app.security)?;
validate_ports(&self.app.ports)?;
validate_interfaces(&self.app.interfaces)?;
if let Some(value) = self.app.extensions.get("install_prerequisites") {
let items = value.as_sequence().ok_or_else(|| {
ManifestError::Invalid("install_prerequisites must be a list of app ids".into())
})?;
for item in items {
let id = item.as_str().unwrap_or_default();
if id.is_empty()
|| id == self.app.id
|| !id
.bytes()
.all(|b| b.is_ascii_lowercase() || b.is_ascii_digit() || b == b'-')
{
return Err(ManifestError::Invalid(
"install_prerequisites must contain valid other app ids".into(),
));
}
}
}
validate_environment(&self.app.environment)?;
validate_devices(&self.app.devices)?;
@@ -1074,6 +1093,14 @@ impl AppManifest {
// `..` copy sources). See docs/manifest-hooks-design.md.
self.app.hooks.validate()?;
if let Some(value) = self.app.extensions.get("backup_before_runtime_change") {
if value.as_bool().is_none() {
return Err(ManifestError::Invalid(
"backup_before_runtime_change must be boolean".into(),
));
}
}
Ok(())
}
}
@@ -1111,6 +1138,7 @@ fn validate_security(policy: &SecurityPolicy) -> Result<(), ManifestError> {
"SETGID",
"SETUID",
"SYS_ADMIN",
"SYS_CHROOT",
];
let mut seen = HashSet::new();
for cap in &policy.capabilities {
@@ -1401,6 +1429,13 @@ pub struct HostFacts {
/// right host. Both are reachable on archy-net by their container name;
/// only the name differs. Falls back to `bitcoin-knots` when undetected.
pub bitcoin_host: String,
/// Nostr public keys of the node's identities that the app identity
/// picker offers for signing (the node's own appliance key excluded),
/// as comma-joined, sorted, lowercase 64-char hex. Lets an app grant
/// the node's users owner rights (e.g. a Blossom server's allowed
/// uploaders). Empty unless a manifest templates it; the orchestrator
/// resolves it on demand and refuses to render an empty set.
pub node_identity_pubkeys: String,
}
impl HostFacts {
@@ -1412,13 +1447,20 @@ impl HostFacts {
host_mdns: "test-node.local".to_string(),
disk_gb: 2000,
bitcoin_host: "bitcoin-knots".to_string(),
node_identity_pubkeys: "1111111111111111111111111111111111111111111111111111111111111111,2222222222222222222222222222222222222222222222222222222222222222".to_string(),
}
}
}
/// Supported placeholder names in `DerivedEnv::template`. Keep in sync
/// with `HostFacts`. Centralized so validation and rendering agree.
const DERIVED_PLACEHOLDERS: &[&str] = &["HOST_IP", "HOST_MDNS", "DISK_GB", "BITCOIN_HOST"];
const DERIVED_PLACEHOLDERS: &[&str] = &[
"HOST_IP",
"HOST_MDNS",
"DISK_GB",
"BITCOIN_HOST",
"NODE_IDENTITY_PUBKEYS",
];
fn validate_derived_template(key: &str, template: &str) -> Result<(), ManifestError> {
// Walk `{{NAME}}` occurrences and ensure each NAME is recognized.
@@ -1502,7 +1544,8 @@ impl ContainerConfig {
.replace("{{HOST_IP}}", &facts.host_ip)
.replace("{{HOST_MDNS}}", &facts.host_mdns)
.replace("{{DISK_GB}}", &facts.disk_gb.to_string())
.replace("{{BITCOIN_HOST}}", &facts.bitcoin_host);
.replace("{{BITCOIN_HOST}}", &facts.bitcoin_host)
.replace("{{NODE_IDENTITY_PUBKEYS}}", &facts.node_identity_pubkeys);
format!("{}={}", e.key, value)
})
.collect()
@@ -1750,18 +1793,38 @@ app:
// disappeared; Cuprate restricted RPC moved from none to gate-open.
// Compare exact endpoints, not just a count that can hide substitutions.
let expected = [
("bitcoin-core", 8333), ("bitcoin-knots", 8333),
("core-lightning", 9736), ("core-lightning", 9835),
("cuprate", 18183), ("electrumx", 50001),
("fedimint", 8173), ("fedimint", 8174),
("fedimint-gateway", 8176), ("fedimint-gateway", 9737),
("gitea", 2222), ("lnd", 9735), ("lnd", 10009), ("lnd", 18080),
("netbird", 8087), ("netbird-server", 3478), ("netbird-server", 8086),
("phoenixd", 9740), ("pine", 10381), ("pine-openwakeword", 10400),
("pine-piper", 10200), ("pine-whisper", 10300),
("router", 1900), ("router", 5353),
].into_iter().map(|(id, port)| (id.to_owned(), port)).collect::<Vec<_>>();
assert_eq!(exempt, expected, "unauthenticated endpoint set changed; review each exemption");
("bitcoin-core", 8333),
("bitcoin-knots", 8333),
("core-lightning", 9736),
("core-lightning", 9835),
("cuprate", 18183),
("electrumx", 50001),
("fedimint", 8173),
("fedimint", 8174),
("fedimint-gateway", 8176),
("fedimint-gateway", 9737),
("gitea", 2222),
("lnd", 9735),
("lnd", 10009),
("lnd", 18080),
("netbird", 8087),
("netbird-server", 3478),
("netbird-server", 8086),
("phoenixd", 9740),
("pine", 10381),
("pine-openwakeword", 10400),
("pine-piper", 10200),
("pine-whisper", 10300),
("router", 1900),
("router", 5353),
]
.into_iter()
.map(|(id, port)| (id.to_owned(), port))
.collect::<Vec<_>>();
assert_eq!(
exempt, expected,
"unauthenticated endpoint set changed; review each exemption"
);
}
/// `auth: open` ports are served by the gate WITHOUT its login challenge,
@@ -1798,9 +1861,14 @@ app:
// nginx-proxy-manager 8081 (NPM admin accounts), tailscale 8240
// (tailnet login on the web console). Both enforce their own login,
// and an operator can re-gate either from Settings → Access control.
// Angor's indexer exposes public chain data/transaction broadcast;
// its optional standalone relay accepts signed public Nostr events.
// Neither mounts credentials or the node's internal relay database.
assert_eq!(
open,
vec![
("angor-indexer".to_string(), 8998u16),
("angor-relay".to_string(), 8091u16),
("btcpay-server".to_string(), 23000u16),
("cuprate".to_string(), 18090u16),
("gitea".to_string(), 3001u16),
@@ -1811,6 +1879,17 @@ app:
);
}
#[test]
fn invalid_install_prerequisites_are_rejected() {
for value in ["not-a-list", "[demo]", "['../other']", "[false]", "['']"] {
let yaml = format!("app:\n id: demo\n name: Demo\n version: 1.0.0\n container:\n image: docker.io/library/alpine:3.20\n install_prerequisites: {value}\n");
assert!(AppManifest::parse(&yaml)
.unwrap_err()
.to_string()
.contains("install_prerequisites"));
}
}
#[test]
fn an_undeclared_port_classifies_as_session_but_is_not_declared() {
// Two different questions, and conflating them caused both gate
@@ -2333,6 +2412,46 @@ app:
);
}
#[test]
fn node_identity_pubkeys_placeholder_is_accepted() {
let yaml = r#"
app:
id: wildbloom-node
name: Wildbloom Node
version: 0.2.2
container:
image: ghcr.io/forgesworn/wildbloom-node:0.2.2
derived_env:
- key: WILDBLOOM_ALLOW_PUBKEYS
template: "{{NODE_IDENTITY_PUBKEYS}}"
"#;
AppManifest::parse(yaml).expect("NODE_IDENTITY_PUBKEYS is a supported placeholder");
}
#[test]
fn resolve_derived_env_renders_node_identity_pubkeys() {
let yaml = r#"
app:
id: wildbloom-node
name: Wildbloom Node
version: 0.2.2
container:
image: ghcr.io/forgesworn/wildbloom-node:0.2.2
derived_env:
- key: WILDBLOOM_ALLOW_PUBKEYS
template: "{{NODE_IDENTITY_PUBKEYS}}"
"#;
let manifest = AppManifest::parse(yaml).unwrap();
let facts = HostFacts::sample();
assert_eq!(
manifest.app.container.resolve_derived_env(&facts),
vec![format!(
"WILDBLOOM_ALLOW_PUBKEYS={}",
facts.node_identity_pubkeys
)]
);
}
#[test]
fn path_traversal_secret_file_is_rejected() {
let yaml = r#"
@@ -2388,6 +2507,7 @@ app:
host_mdns: "test-node.local".to_string(),
disk_gb: 2000,
bitcoin_host: "bitcoin-core".to_string(),
node_identity_pubkeys: String::new(),
};
let out = c.resolve_derived_env(&facts);
+36 -4
View File
@@ -450,6 +450,17 @@ impl PodmanClient {
"nsmode": net_mode
},
});
if matches!(
manifest.app.container.network.as_deref(),
Some(
"slirp4netns:allow_host_loopback=true"
| "slirp4netns:allow_host_loopback=true,cidr=169.254.1.0/24"
)
) {
body["network_options"] = serde_json::json!({
"slirp4netns": manifest.app.container.network.as_deref().unwrap().split_once(':').unwrap().1.split(',').collect::<Vec<_>>()
});
}
if let Some(network) = custom_network {
// The container always answers to its own name; manifest
// network_aliases add extra short hostnames peers may bake in
@@ -727,7 +738,11 @@ fn podman_network_settings(
Some("host") => ("host", None),
Some("bridge") => ("bridge", None),
Some("none") => ("none", None),
Some("slirp4netns") => ("slirp4netns", None),
Some(
"slirp4netns"
| "slirp4netns:allow_host_loopback=true"
| "slirp4netns:allow_host_loopback=true,cidr=169.254.1.0/24",
) => ("slirp4netns", None),
Some("pasta") => ("pasta", None),
Some("private") => ("private", None),
Some(custom) => ("bridge", Some(custom.to_string())),
@@ -1077,13 +1092,30 @@ mod tests {
));
}
#[test]
fn npm_rootless_options_are_not_a_named_bridge() {
assert_eq!(
podman_network_settings(Some("slirp4netns:allow_host_loopback=true"), "isolated"),
("slirp4netns", None)
);
}
#[test]
fn portainer_manifest_keeps_private_network_and_loopback_api_publication() {
let m = AppManifest::parse(include_str!("../../../apps/portainer/manifest.yml")).unwrap();
assert_eq!(podman_network_settings(m.app.container.network.as_deref(), &m.app.security.network_policy), ("slirp4netns", None));
assert_eq!(podman_publish_mapping(&m.app.ports[0]), serde_json::json!({
assert_eq!(
podman_network_settings(
m.app.container.network.as_deref(),
&m.app.security.network_policy
),
("slirp4netns", None)
);
assert_eq!(
podman_publish_mapping(&m.app.ports[0]),
serde_json::json!({
"container_port": 9000, "host_port": 9000, "protocol": "tcp", "host_ip": "127.0.0.1"
}));
})
);
}
#[test]
+49 -8
View File
@@ -40,6 +40,11 @@ pub fn stop_grace_secs_for(container_name: &str) -> u64 {
#[async_trait]
pub trait ContainerRuntime: Send + Sync {
/// CLI used for offline app provisioning in this runtime's storage scope.
fn cli_name(&self) -> &'static str {
"podman"
}
async fn pull_image(&self, image: &str, signature: Option<&str>) -> Result<()>;
async fn create_container(
&self,
@@ -621,10 +626,20 @@ impl DockerRuntime {
// Docker is a development fallback. Refuse Podman-only network modes instead
// of silently installing a different topology; still honor binds for other apps.
fn docker_network_and_ports(manifest: &AppManifest, offset: u16) -> Result<Vec<String>> {
let network = manifest.app.container.network.as_deref()
let network = manifest
.app
.container
.network
.as_deref()
.filter(|v| !v.is_empty())
.unwrap_or(&manifest.app.security.network_policy);
if matches!(network, "slirp4netns" | "pasta") {
if matches!(
network,
"slirp4netns"
| "slirp4netns:allow_host_loopback=true"
| "slirp4netns:allow_host_loopback=true,cidr=169.254.1.0/24"
| "pasta"
) {
anyhow::bail!("this app requires rootless Podman networking ({network})");
}
let mut args = Vec::new();
@@ -632,16 +647,34 @@ fn docker_network_and_ports(manifest: &AppManifest, offset: u16) -> Result<Vec<S
args.extend(["--network".to_owned(), network.to_owned()]);
}
for port in &manifest.app.ports {
let host = port.host.checked_add(offset).context("published port offset overflow")?;
let bind = if port.bind.is_empty() { String::new() } else { format!("{}:", port.bind) };
let protocol = if port.protocol.is_empty() { "tcp" } else { &port.protocol };
args.extend(["-p".to_owned(), format!("{bind}{host}:{}/{protocol}", port.container)]);
let host = port
.host
.checked_add(offset)
.context("published port offset overflow")?;
let bind = if port.bind.is_empty() {
String::new()
} else {
format!("{}:", port.bind)
};
let protocol = if port.protocol.is_empty() {
"tcp"
} else {
&port.protocol
};
args.extend([
"-p".to_owned(),
format!("{bind}{host}:{}/{protocol}", port.container),
]);
}
Ok(args)
}
#[async_trait]
impl ContainerRuntime for DockerRuntime {
fn cli_name(&self) -> &'static str {
"docker"
}
async fn pull_image(&self, image: &str, signature: Option<&str>) -> Result<()> {
// Same signature gate as the podman path — the docker fallback is
// dev-only, but a declared signature must never be skippable by
@@ -973,6 +1006,10 @@ impl AutoRuntime {
#[async_trait]
impl ContainerRuntime for AutoRuntime {
fn cli_name(&self) -> &'static str {
self.runtime.cli_name()
}
async fn pull_image(&self, image: &str, signature: Option<&str>) -> Result<()> {
self.runtime.pull_image(image, signature).await
}
@@ -1041,12 +1078,16 @@ mod tests {
#[test]
fn docker_fallback_rejects_rootless_only_topology_and_preserves_bind_protocol() {
let mut m = AppManifest::parse(include_str!("../../../apps/portainer/manifest.yml")).unwrap();
let mut m =
AppManifest::parse(include_str!("../../../apps/portainer/manifest.yml")).unwrap();
assert!(docker_network_and_ports(&m, 0).is_err());
m.app.container.network = Some("bridge".into());
m.app.ports[0].protocol = "udp".into();
let args = docker_network_and_ports(&m, 1).unwrap();
assert_eq!(args, vec!["--network", "bridge", "-p", "127.0.0.1:9001:9000/udp"]);
assert_eq!(
args,
vec!["--network", "bridge", "-p", "127.0.0.1:9001:9000/udp"]
);
assert!(docker_network_and_ports(&m, u16::MAX).is_err());
}
+18 -1
View File
@@ -22,7 +22,9 @@ To pick up a new build, redeploy the stack (or wire the CI Portainer webhook).
The images are built from the Archipelago monorepo by
`.github/workflows/demo-images.yml` on every change to `neode-ui/`, tagged `:demo`
and `:<git-sha>`, and pushed to `REGISTRY`. Editing the real UI → CI rebuilds →
redeploy here. No source lives in this repo.
redeploy here. The optional Portainer webhook runs only on an explicit workflow
dispatch after release qualification; a source push alone does not invoke it.
No source lives in this repo.
## What's mocked
@@ -31,3 +33,18 @@ redeploy here. No source lives in this repo.
- **Wallet/Bitcoin** — signet-flavored; use the in-UI faucet for test sats.
- **Files** — real per-session upload/rename/delete, 50 MB quota, wiped on reap.
- **Intro** — replays once per calendar day per browser.
## Building a reviewed demo revision
The web image builds both the dashboard and AIUI from the checked-out source;
it does not use the historical `demo/aiui` bundle. CI supplies the full commit
as `SOURCE_REVISION`. For a local source build, run:
```sh
SOURCE_REVISION=$(git rev-parse HEAD) docker compose -f docker-compose.demo.yml build
```
AIUI uses its frozen dependency lockfile, type checking and the canonical
`/aiui/` build verifier. Its `BUILD-INFO` identifies that exact source revision.
Prepare and test the images before changing the public demo stack; retain the
previous image IDs for rollback.
+1
View File
@@ -44,6 +44,7 @@ services:
dockerfile: neode-ui/Dockerfile.web
args:
VITE_DEMO: "1"
SOURCE_REVISION: ${SOURCE_REVISION:?Set SOURCE_REVISION to git rev-parse HEAD}
container_name: archy-demo-web
ports:
- "2100:80"
+1 -1
View File
@@ -1,4 +1,4 @@
FROM git.tx1138.com/lfg2025/nginx:1.27.4-alpine
FROM source.archipelago-foundation.org/lfg2025/nginx:1.27.4-alpine
# Static site content.
COPY index.html /usr/share/nginx/html/
COPY tailwind.css /usr/share/nginx/html/
+1 -1
View File
@@ -1,4 +1,4 @@
FROM git.tx1138.com/lfg2025/nginx:1.27.4-alpine
FROM source.archipelago-foundation.org/lfg2025/nginx:1.27.4-alpine
# Static site content.
COPY index.html /usr/share/nginx/html/
COPY 50x.html /usr/share/nginx/html/
+1 -1
View File
@@ -1,4 +1,4 @@
FROM git.tx1138.com/lfg2025/nginx:1.27.4-alpine
FROM source.archipelago-foundation.org/lfg2025/nginx:1.27.4-alpine
COPY index.html /usr/share/nginx/html/
COPY 50x.html /usr/share/nginx/html/
COPY qrcode.js /usr/share/nginx/html/
+1 -1
View File
@@ -1,4 +1,4 @@
FROM git.tx1138.com/lfg2025/nginx:1.27.4-alpine
FROM source.archipelago-foundation.org/lfg2025/nginx:1.27.4-alpine
COPY index.html /usr/share/nginx/html/index.html
COPY nginx.conf /etc/nginx/conf.d/default.conf
+1 -1
View File
@@ -1,4 +1,4 @@
FROM git.tx1138.com/lfg2025/nginx:1.27.4-alpine
FROM source.archipelago-foundation.org/lfg2025/nginx:1.27.4-alpine
# Static site content.
COPY index.html /usr/share/nginx/html/
#
+1 -1
View File
@@ -1,4 +1,4 @@
FROM git.tx1138.com/lfg2025/nginx:1.27.4-alpine
FROM source.archipelago-foundation.org/lfg2025/nginx:1.27.4-alpine
# Copy the HTML file
COPY index.html /usr/share/nginx/html/
+61 -15
View File
@@ -3,6 +3,17 @@
Working backlog of forward-looking items not yet scoped into a dedicated plan
doc. See [`ROADMAP.md`](ROADMAP.md) for the curated, public-facing direction.
## NPM certificate release blocker — reported 2026-10-01
- [ ] **NPM first certificate and renewal must work without manual repair.**
Shorty's HTTP-01 404 exposed inconsistent active data/ACME paths between the
manifest, legacy creation paths and host nginx bridge. Independent reviewer
acknowledged the handoff; separate release owner must acknowledge receipt,
implement data-safe migration and complete fresh-install, legacy-upgrade,
issuance/renewal, restart/reboot and OTA/ISO acceptance. See
[repair evidence and required release tests](npm-certificate-handoff-20261001.md).
Do not equate the live challenge-route repair with a tested release fix.
## Framework incident — closed with operator acceptance
- **CLOSED WITH OPERATOR ACCEPTANCE (2026-09-30): Framework LND startup /
@@ -14,30 +25,65 @@ doc. See [`ROADMAP.md`](ROADMAP.md) for the curated, public-facing direction.
## Next release after 1.8.21 — reported 2026-09-30
Release status and acceptance gates: [execution checklist](next-release-20260930.md).
- [ ] **Release blocker: Gitea → Portainer repository integration.** Diagnose
smart-HTTP reachability from Portainer's actual request namespace, then provide
one declarative topology and idempotent migration for fresh installs and
existing nodes. Preserve gate/auth boundaries, operator configuration,
repository/key/database mounts and Portainer stacks. Cover install order,
lifecycle/reboot/update convergence, clone/push and source-branch/Compose-file
acceptance with a disposable integration setup. Ship in both OTA and ISO;
a healthy Gitea root page is insufficient. Operator supplied a private handover;
deployment addresses and credentials must not be committed.
- [ ] **New X250: GitWorkshop failed at 70%; slow Nginx installation.** Missing
ISO build contexts restored on-node; package staging/smoke checks added.
GitWorkshop dependency audit refreshed and build/HTTP recovery verified;
Nginx was a slow successful image pull. Aggregate progress label corrected.
Include the validated repair in the next OTA/ISO. See lifecycle evidence.
- [ ] **Angor indexer service in the app store**, requested after the other
current repair/review work (2026-09-30). Follow the repository's app-development
and packaging documentation; treat it as a headless service unless upstream
documentation establishes a UI. Verify Bitcoin/Mempool requirements, decide
whether an existing first-class relay meets Angor's requirements or a relay
must be packaged with the indexer, and use the Angor logo from angor.io for its
service icon. Official current deployment documentation located and reviewed: stock Mempool
plus an optional strfry relay. Both headless services and the dependency guard
are implemented; API outage/recovery and five relay lifecycle cycles passed.
Final candidate install/lifecycle checks and signed delivery remain pending.
Install on the development box; Bitcoin must finish syncing for indexed queries.
- [ ] **App lifecycle: keep installed apps visible through restart and hard
refresh; gate embedded/browser launches on actual web and listener readiness.**
Source repair and scoped live acceptance passed; full release gate pending.
Includes durable inventory reconstruction,
concurrent inventory writes, stale scan/lifecycle updates, delayed HTTP startup,
and the app gate's post-install listener delay. See
[app lifecycle repair evidence](app-lifecycle-repair-20260930.md).
- [x] Review and repair open paid-download PRs #161 and #162, refresh both
branches from main, run independent and combined isolated suites, and verify
rootless file permissions in disposable scratch storage. Combined result:
1,585 passed, zero failed, four existing tests ignored. See the
[review evidence and remaining acceptance work](pr-review-20260930.md).
- [ ] Integrate the reviewed PR branches into the next release and run funded
- [x] Integrate the reviewed PR branches into the next release and run funded
candidate acceptance, including Tor-only transport and payments with change.
PRs remain open; the reviewed code has not been deployed to live wallets.
Operator authorized completing the normal merge/closure workflow on
2026-09-30. Both PRs are now merged and closed through Gitea; integrate
local repair commits and sync git/ngit before release. The combined candidate
is deployed on both test endpoints. Funded Tor-only purchase with change,
confirmed refund, exact Files bytes and zero-cost repeat delivery passed.
The updated source still needs inclusion in signed OTA/ISO artifacts.
- [ ] Design durable recovery for an accepted payment whose response is lost.
Preserve the truthful unconfirmed-refund warning and prevent automatic
duplicate payment while that recovery work is outstanding.
- [ ] **ThinkPad X250 kiosk: Bitcoin installation version selector is unreadable
and appears underneath the pruning information.** Operator reports white
styling with invisible text on the actual kiosk; the same flow works in remote
Brave. Reproduce on the X250's kiosk engine and record its version, display
scale and resolution. Inspect the native `<select>` in
`neode-ui/src/components/InstallVersionModal.vue`, its option colors, and the
scroll/stacking behavior in `BaseModal.vue`; these are investigation leads,
not a confirmed cause. Fix contrast and popup visibility without changing
version selection or pruning behavior. Validate Core and Knots, open/closed
and scrolled dropdowns, keyboard/touch selection, and pruning on/off on the
actual kiosk, with remote Brave and mobile regression checks. Browser mocks
alone do not establish that the kiosk rendering is fixed. Track for the next
release; the signed 1.8.21 artifacts remain unchanged.
- [x] **ThinkPad X250 kiosk: Bitcoin version choices readable above pruning.**
Replaced the native popup with inline radio choices. Actual Chromium 152 kiosk
assertions and screenshot verify white-on-dark choices, selection changes and
layout above pruning controls. Focused component tests pass. Included in the
next-release source; published 1.8.21 artifacts remain unchanged.
## 1.8.21 repair and release tasks — completed 2026-09-30
+112
View File
@@ -0,0 +1,112 @@
# Angor client acceptance and remaining project recovery
Status: **OPEN — live services and one complete project flow pass; full project recovery is incomplete.**
The operator requires actual Angor-client verification and any necessary application
fixes to reach the next OTA, app catalog and ISO. Main release owner acknowledged
ownership of NPM/public management isolation; this investigation owns Angor evidence,
app setup documentation and the scoped browser test. Do not treat this report as
permission to waive the outstanding discovery/recovery requirements.
## Verified live behavior
- Trusted HTTPS indexer health, fees, address transactions and cursor pagination work.
- Real browser requests from `https://angor.io` reach the selected custom indexer;
checked transaction IDs, confirmation data and output scripts match the reference.
- The indexer serves all 35 funding transactions listed in the public reference
snapshot, confirmed, with matching original announcement commitments. This checks
that snapshot, not every possible Bitcoin address or the entire project universe.
- Relay NIP-11 and encrypted WebSocket/Nostr read work. An invalid signature was
rejected before importing the valid original announcement; the original signed
announcement was accepted unchanged and read back exactly.
- Imported ONLY the original public kind3030 event
`adbf94d6152097f3d503cd68cc00dee34c1e1007914c00cfc3d6698d1ee01834`, after checking its
Schnorr signature with nostr-tools and matching funding transaction
`72d14227b78a260c9410a65585cb49e81e35eaa95b1b23ec702eb0512ca016a7` on our indexer.
No fabricated, re-signed or modified announcement, wallet operation or payment.
- Restarted only the managed Angor relay. Its original announcement persisted;
node-internal strfry container ID and start time remained unchanged.
- Clean Chromium, configured with our indexer plus original relays and our relay,
now displays **Casa Bitcoin sv** in Explore and loads its full project page,
metadata, funding transaction and Project Statistics.
- `tests/lifecycle/angor-public-browser.cjs` passes all three acceptance groups:
trusted public API/WSS and chain commitment; actual Explore discovery;
actual project details/statistics. It is opt-in and read-only, uses a disposable
profile and known immutable public fixture, and does not disable TLS checks.
Live test command (run from repository root):
```sh
ANGOR_TEST_INDEXER=https://angor-indexer.tx1138.com/ \
ANGOR_TEST_RELAY=wss://angor-relay.tx1138.com/ \
ANGOR_TEST_RELAYS='["wss://relay.angor.io","wss://relay2.angor.io","wss://angor-relay.tx1138.com"]' \
node tests/lifecycle/angor-public-browser.cjs
```
## Empty project list: reproduced upstream behavior
The user retained the original relays. Our earlier suggestion that an empty new
relay explained the entire failure was incorrect.
Angor Hub v2.0.0 (`main-575CWKJX.js`) was tested in separate clean Chromium profiles
using our indexer and `https://indexer.angor.io/`. Both received candidate Nostr
announcements, but the current batch failed validation: `event-mismatch` for
updated announcements of one funded project, and `not-found-in-mempool` for another
address that both indexers correctly return empty. Both displayed zero projects.
The default primary `fulcrum.angor.online` separately failed browser CORS in a
fresh default-config test. These upstream failures must not be attributed to
missing data in our indexer without evidence.
Current `block-core/angor-hub` `src/app/services/indexer.service.ts` discovers
kind3030 events, filters by network, and calls `validateAndAddProjects` in the
background. When a batch has candidates but none validate, it does not continue
discovery to older valid original announcements. The Load More UI is unavailable
when the project list is empty. An empty relay can coexist with this bug but is
not its sole cause. After the authentic announcement was copied to our relay,
one original passed the unchanged validation and appeared. This is a bounded
history-availability recovery, NOT an upstream UI fix or global relay sync.
The current browser uses ordinary `/api/v1/address/.../txs`, not the deprecated
`/api/v1/query/Angor/projects` listing. Our adapter's 404 for the latter is NOT the
observed browser failure. Do not add an opaque upstream proxy to hide that 404.
## Other projects remain unresolved
The public reference query returned 35 project records (limit50). All35 on-chain
funding commitments were verified through our indexer. Exact original-event-ID
queries were sent to the configured relays `relay.angor.io`, `relay2.angor.io`,
`nos.lol`, `relay.primal.net`, and `relay.damus.io` (Damus had an initial connection
failure, then answered EOSE on retry). Only one original announcement was found.
Additional queries to `relay.snort.social`, `nostr.mom`, and `no.str.cr` returned
no matches; `relay.nostr.band` and `relay.f7z.io` were unavailable. This does not
prove the other34 events are globally lost or that all storage sources were checked.
Exact project IDs, transaction IDs, event IDs and confirmed commitment results:
[recovery inventory](angor-project-recovery-20261001.json). Recover originals from
founders/known archives or authoritative relay backups and retain their signatures.
On-chain commitments cannot reconstruct missing off-chain project content.
Do not fabricate replacements or accept mismatched events to make cards appear.
## Release handoff and open gates
- [x] Release owner acknowledges these final findings and the 34-project gap.
Confirmed 2026-10-01 18:46:50 UTC in `/tmp/angor-final-handoff-ack.txt`: full
recovery remains open; publication held for required gates; preserve relay
data and rerun browser acceptance after bridge migration and OTA.
- [ ] Resolve or explicitly carry the upstream discovery bug with accurate user
guidance; do not claim full recovery because the fixture passes.
- [ ] Locate and verify remaining original project metadata/history, or obtain an
explicit operator scope decision; one project is not complete acceptance.
- [ ] Preserve the verified public relay event/data through the candidate upgrade.
- [ ] Repeat scoped browser acceptance after NPM bridge migration and OTA install.
- [ ] Verify fresh ISO setup, NPM host propagation, public IP/default-host isolation,
certificate issuance/renewal and relay WebSocket routing; see NPM handoff.
- [ ] Include updated app setup documentation and the read-only acceptance test.
No Angor image/config change was justified by the verified transaction data;
no image or catalog version should be bumped solely to disguise upstream failure.
- [ ] Existing signed transaction-broadcast integration tests remain required;
this live investigation did not send real Bitcoin or test a funded investment.
Raw local diagnostic logs are in `/tmp/angor-*-browser*.log` and
`/tmp/angor-public-browser-acceptance.log`. The durable facts and recovery inventory
above must remain available after temporary logs expire.
+286
View File
@@ -0,0 +1,286 @@
{
"status": "INCOMPLETE: 34 original announcements not recovered from queried relays",
"source_inventory": "https://indexer.angor.io/api/v1/query/Angor/projects?limit=50",
"projects": [
{
"project": "angor1qryhse38vcyqnp0j6976q9f00a9jpj2ary03nlc",
"txid": "72d14227b78a260c9410a65585cb49e81e35eaa95b1b23ec702eb0512ca016a7",
"event_id": "adbf94d6152097f3d503cd68cc00dee34c1e1007914c00cfc3d6698d1ee01834",
"confirmed": true,
"commitment_matches": true,
"original_announcement_recovered": true
},
{
"project": "angor1q3eh4xg7t2hge7ctqk4yhmj7q23t6mdqa0ahg28",
"txid": "f5ff2e1a6b7340ddb9944c2ae6477c6b0b12993c9919f6e9b7b9e7a5e9a68f6d",
"event_id": "f4417e39c9b47afafd84cdda2017207e0929df2852badbf8d909bf6f647f4f2d",
"confirmed": true,
"commitment_matches": true,
"original_announcement_recovered": false
},
{
"project": "angor1qde7y830vtajref5vwag5x459m9w5y75gd49v4t",
"txid": "205ce39688572ef0168c1c1231c25ec632abb2c2b64c52fccb4f53eb0a49e300",
"event_id": "ca76084c2bc3301f8fe00dd8a93c6d6c0ad015e50676cdd3e44c8bd765f157d5",
"confirmed": true,
"commitment_matches": true,
"original_announcement_recovered": false
},
{
"project": "angor1q9k9976ytwkkmswlq24e89l2fxuxl57gfp8yxgr",
"txid": "00a0120818698ef3d72d360e29e9e4e7d0a35c180967186d1e715aef797c26a8",
"event_id": "a04d1eed8c1261cb5fca6a6faf16a5f87a9672cf40d1117b25f2da60e6e8f84c",
"confirmed": true,
"commitment_matches": true,
"original_announcement_recovered": false
},
{
"project": "angor1q67h8cktwy8yv32t0uk0c8zrtpmtwgtd56sylzx",
"txid": "1bfa726353a40d5fce1a76ad86dc7915bee214573d30d5751cd6312c94519089",
"event_id": "b3189f84b490da422e17b6b857d393b13781bb82e1c8813328769e3d2840098a",
"confirmed": true,
"commitment_matches": true,
"original_announcement_recovered": false
},
{
"project": "angor1qm999ekmrdjl4sq3qywl889w74qetxe3ghdxthl",
"txid": "ae9d471dd6e58b318120fbdf72929c621cf23d9c6f047df3c57923a8a915d01c",
"event_id": "89ae5e612e857f89dbbdc662198b894f1f7b70564769bedf08ec37f5e7b5efe0",
"confirmed": true,
"commitment_matches": true,
"original_announcement_recovered": false
},
{
"project": "angor1qtunna00uqyx52rr0nvqa059z2gknhrmn7urd6z",
"txid": "c2cb77ebf6b9ded802b677c600c3869b81fc76f31a3e4742b6a9cb16e0ac3dee",
"event_id": "f5e66707f8fa0fc601bef403020e64d9a164d6cf201599ec0c3ddb0acf58491e",
"confirmed": true,
"commitment_matches": true,
"original_announcement_recovered": false
},
{
"project": "angor1qu8h0ezv596z35uggg0r0ppkma93tnreyjg5du7",
"txid": "fb6bc0b721810957ae8910d5dd6e7ebc45d804a3e1f636d153df50f036e90645",
"event_id": "ce61a11b79aa258238db63f67472341169944bce02600045bf7278992aeb796c",
"confirmed": true,
"commitment_matches": true,
"original_announcement_recovered": false
},
{
"project": "angor1q8u9c2h2l0eqjn3qeyvdps89dkkvwteg5q3m708",
"txid": "04ed05297e146206c104c7b5d8a51fc3453f1829950675b4694f91a7ea609be1",
"event_id": "f31c6ef3a66e74ebfc26f5b2905e6d4641f73bdb407aa92022a2202b9be54716",
"confirmed": true,
"commitment_matches": true,
"original_announcement_recovered": false
},
{
"project": "angor1qqnxefpr4f59r5f4u34c7crzst2ya83wavh8407",
"txid": "bae879110e78ad44624980ea4a47de21b03d3994139714bb09e910187027583d",
"event_id": "cbd23077098059c5092bb7ea8df14dd73f21d47d1b690ff1e6fa789ad118864b",
"confirmed": true,
"commitment_matches": true,
"original_announcement_recovered": false
},
{
"project": "angor1qfydf802v2efvxufevrfg0mvtlrxln63rkzmdxw",
"txid": "4bbb04fed973c968e76faaca880197092be288d9897072ab5e6dfb719c19eb69",
"event_id": "e9761e1303de7eca0ade33936489d2b4b7d51f4b52e8ecbc1acfc394df48b95d",
"confirmed": true,
"commitment_matches": true,
"original_announcement_recovered": false
},
{
"project": "angor1qpt96uj5jg77q5566pmcdggyuztt26xn3xymwhn",
"txid": "bc46fea91acf4714f5b7949e9bb937f39e425468242c2459b581a14914e641b4",
"event_id": "f683c12dbc3d574797bd2251d7e9d96e1d33d6263f75a1538d2ce5fd0c86e064",
"confirmed": true,
"commitment_matches": true,
"original_announcement_recovered": false
},
{
"project": "angor1q3nleprspa3h6thy9c25wzu4q7ywajhcdg3j9au",
"txid": "7b76a8297f16ad5ff3d69fc85e37405ad4f77a71dfc8f70206a5a9c1827698ba",
"event_id": "893e5b7f09a12368f208120deb9c02b3692aee15240a914fb428e7e4f86b1123",
"confirmed": true,
"commitment_matches": true,
"original_announcement_recovered": false
},
{
"project": "angor1qc8jlugwgp90vzkhf336d8exldhwd8z5u4ssaen",
"txid": "5e06eb3684bf0d3402d20d643d6fe1b5a295680a29db440e070a89f80e52a241",
"event_id": "375eafd0e03c50d683de4f465501e919a8816ae618ca2b5c14e8d3f56daa90c0",
"confirmed": true,
"commitment_matches": true,
"original_announcement_recovered": false
},
{
"project": "angor1qdhwn7s0p8wcr9kw2932m448y6hd8ywl005fdwu",
"txid": "7cc5bff45f0b5e9ae81cadaf787dd4d4680387ab9fddedf8438eae8f87ef34ec",
"event_id": "6c7767eaf6ee0ae4ffe4b23c8477f2293216279e5908dd979abd4abdf1557578",
"confirmed": true,
"commitment_matches": true,
"original_announcement_recovered": false
},
{
"project": "angor1q8qzylut0gv7urycxukdfumzw6znghrk4g928k2",
"txid": "2c7eed8d9b8eef530a4a8f39339f7dfaa82d5ec09261305203d1a367a624be1b",
"event_id": "24323aae4bca910ddb48544b788860119c3cc13ac19b24694221f1ba2521cd6b",
"confirmed": true,
"commitment_matches": true,
"original_announcement_recovered": false
},
{
"project": "angor1qzwrm6hf5l0tgen99anr9hj7ylk38v6zhmnq7w0",
"txid": "934c45244c283ff24834bab7d01a0964192433cd2bc11143fc5e590b2b954deb",
"event_id": "6cfae243c276a602ce2da33842ad930019d75e80ffe81b4cd84b1b187830bba7",
"confirmed": true,
"commitment_matches": true,
"original_announcement_recovered": false
},
{
"project": "angor1qpdp5d2ahknhtr7kawsl62m7y4hktsfn4uwg6r3",
"txid": "dc85d36b324ff829a4d49a606573e136fbc29498bc707d95f5bd0d9ef49956d0",
"event_id": "c6d22deef6d1babc99716746f50509e5413b97445d991a0869f0567b7301fc97",
"confirmed": true,
"commitment_matches": true,
"original_announcement_recovered": false
},
{
"project": "angor1q9v88fe2q2efr4z57wed4gklmkh7trmx6usac0d",
"txid": "b98b23e49630f92ac0dedb7e0a8ac5ad2c51f813039f2d9dd708c9a0861b2ca2",
"event_id": "e8f518eaed658e9331b3a3feba49f36eda5eb7fe68c81c7b3f71057844729bcb",
"confirmed": true,
"commitment_matches": true,
"original_announcement_recovered": false
},
{
"project": "angor1q0gtztyk24ea2028ewfn026838k3ll322qrvrhu",
"txid": "31b38cf48cf18936b7c370ee72e8bda6e9ee40f24ec676a85b9b8b1abefebdd4",
"event_id": "fe34db328e51cc10f9c4d035b0f0aab7f9137a4cdee3a2afc4c6610404e79fe4",
"confirmed": true,
"commitment_matches": true,
"original_announcement_recovered": false
},
{
"project": "angor1q65fuwpyvek3fxk5p757zyknjd9k9sava3fd98j",
"txid": "0b7bf7b9119157edc778bcdc7088b70a439fa5dc8c46e839b95fd2f0a8fbf046",
"event_id": "991d0f7d1c261e4d79507238e3d5e4b3d3adce267cf40107ee2ade32485f5cf6",
"confirmed": true,
"commitment_matches": true,
"original_announcement_recovered": false
},
{
"project": "angor1qjqfp23ac4s4llgcgs3qnenjhpasgrmvt373usp",
"txid": "8b4887aba04b12729e609658b0fdbb1e6f1b172ba10360c7a5261c3e9bd590f9",
"event_id": "254910567be531d7862afcbcc80b490ee92a5f9ed801424f3d56108d9a114b80",
"confirmed": true,
"commitment_matches": true,
"original_announcement_recovered": false
},
{
"project": "angor1qfyzk83tqznc3aqg5ly0cewn7ytsrwl474v4dyn",
"txid": "3e2ea870b17eab2023a04dca267c45d2c53a41abb3f20b206e095fcbba6c5f02",
"event_id": "1e27aa63f276048ba78ef73f69dc5045441feafaa090aff9b995341883f22fb4",
"confirmed": true,
"commitment_matches": true,
"original_announcement_recovered": false
},
{
"project": "angor1qw7m67a5r6vpmtw2jqpsp4xtxvtu6mmnkgj6gxw",
"txid": "f3d3549b2a30c78e7c3b51bc9122e0ae8a7ecb378f9b3564ebf931769637df49",
"event_id": "fc4289a4888bfe157588e507204d93723a8c07074cde37bb98cf866793f81c98",
"confirmed": true,
"commitment_matches": true,
"original_announcement_recovered": false
},
{
"project": "angor1qgdc07hkk0l4kntg2k5fczk7tcx0qeqqx2saru0",
"txid": "c1332c51da2c5706f6fc74e3275856438304a7761ced4dcd2e738a7e3c62bd2f",
"event_id": "ae1d1f7f883907dea36902fc2dc78c8ad81d22bd59bd7293ba1725c6cc959846",
"confirmed": true,
"commitment_matches": true,
"original_announcement_recovered": false
},
{
"project": "angor1qhklyl9mw5zwzwynxv6ekaz5f9h5zv3wnd8dn2c",
"txid": "be637cd8a23c80f49195345eca5d4c29bf4a9fa0600e0625af702444e300d218",
"event_id": "78a1443b14cb252b2510925263889e8c8f40b12fd911d137d63b1dbc2b841a3a",
"confirmed": true,
"commitment_matches": true,
"original_announcement_recovered": false
},
{
"project": "angor1q4heck5tf6svq7x8jp5twak329xtv9805xppvq5",
"txid": "509db524eba0b90e8607396e9eb42ed379a270f3aa602a0bc16eec050c959823",
"event_id": "dec392d7d962e7dfd2c9eaa2b05dfc03c909d87982c4d73b9f321dc13487622f",
"confirmed": true,
"commitment_matches": true,
"original_announcement_recovered": false
},
{
"project": "angor1qc6gykcw82hu3pa4pn94gfxwgpp8dewlsh45rwc",
"txid": "087390128a78270cd1465656e3ab63b9b47982dca1a910b7a5664f67f8ab9970",
"event_id": "cf634987daa4ee17bb2d160ddb04ce56a0d73e664b9822e5e9a8edf870630a9c",
"confirmed": true,
"commitment_matches": true,
"original_announcement_recovered": false
},
{
"project": "angor1qfzxd8n94r592gt3mrmgwe7knk6dhajmutpuhkx",
"txid": "29cb361fe78b6f199f169b7b4a7d9663b7e7f46607f382ddcf8cb2224a6023f1",
"event_id": "ca7a6e0bb27beb46fb8e709378908feb5357e7edd43a9262318cadf72da5f141",
"confirmed": true,
"commitment_matches": true,
"original_announcement_recovered": false
},
{
"project": "angor1q0whhl3qgq28g443h9mj6dl8n2ssshm6hkd3xse",
"txid": "4d70fba03ec51d87df7df16498a95a907b9cff00035455e3ab8242935f260030",
"event_id": "3a19b34d76ec7b99eb98ed299737c06cd3710268febf7d2acf6ce1fece9ba459",
"confirmed": true,
"commitment_matches": true,
"original_announcement_recovered": false
},
{
"project": "angor1q2a5m2zcwpmkh49z05pg6gd9cxm4dhx3ywfclem",
"txid": "5da06a119db273bbb7c64e2cc103a8edb48a9934ab9b5b972ae55f697023ce07",
"event_id": "8c3a8dcaf9c55e7797cf4ff9a25b0e7cd7dcbe78c8ed248d53a49c70f9ea4a6b",
"confirmed": true,
"commitment_matches": true,
"original_announcement_recovered": false
},
{
"project": "angor1qq9ngpm2w8xss5sf0amt63z076y2x885wh0jfv7",
"txid": "7f42da75e4b3dd92f9870aefa177c2fb3783f189996abf193b3b353ee21e805e",
"event_id": "2f0c6e3b29a74be45033062742b3fbac4a4c7b7be4eeecc7021df4a8b1b195cf",
"confirmed": true,
"commitment_matches": true,
"original_announcement_recovered": false
},
{
"project": "angor1qwdgxjuzhjykgpn5q8p3l2q9vyrgqdlrkfp5sjr",
"txid": "c15d07fd14d58e7204889ba24fe47a9b86aa7bea9992d30f4be36864e7634725",
"event_id": "733b28b35f771839bc719125af94916a5400675185d4a75edb09645c8eb4cc24",
"confirmed": true,
"commitment_matches": true,
"original_announcement_recovered": false
},
{
"project": "angor1qyuj8z532tnhy7srutwecu3j789z22peu2t8c7v",
"txid": "821193743f7ca7b0b178a78379d79f5783d874a182a8bf36b70a0a2a6cb12d24",
"event_id": "56fce837c628728953138ca57895c7ef9533640a989934c432d1040808781b9f",
"confirmed": true,
"commitment_matches": true,
"original_announcement_recovered": false
},
{
"project": "angor1qznva9wmw3anr7qdhhs4v0xptd0pz07gdr2fuwz",
"txid": "187e39a0ba9714ae3543bbe775dd9fff9c7a4ac946ce0a850480c3871de287fe",
"event_id": "baff907b6f1fb97893e63e1bef6919f819c1b6e37560ed2d4255b77cc26d08cd",
"confirmed": true,
"commitment_matches": true,
"original_announcement_recovered": false
}
]
}
+11 -1
View File
@@ -108,7 +108,7 @@ app:
| `app.container.pull_policy` | Pull behavior, usually `if-not-present` |
| `app.container.network` | Podman network setting such as `archy-net` or `pasta`; dangerous namespace-sharing modes are rejected |
| `app.container.entrypoint` / `custom_args` | Entrypoint and command override |
| `app.container.derived_env` | Environment values rendered from host facts. The complete placeholder set is `{{HOST_IP}}`, `{{HOST_MDNS}}`, `{{DISK_GB}}`, `{{BITCOIN_HOST}}`; an unknown name or an unbalanced `{{` is a parse error, so typos fail loudly |
| `app.container.derived_env` | Environment values rendered from host facts. The complete placeholder set is `{{HOST_IP}}`, `{{HOST_MDNS}}`, `{{DISK_GB}}`, `{{BITCOIN_HOST}}`, `{{NODE_IDENTITY_PUBKEYS}}`; an unknown name or an unbalanced `{{` is a parse error, so typos fail loudly. `{{NODE_IDENTITY_PUBKEYS}}` is the Nostr public keys of the identities the app signer (the NIP-07 bridge's identity picker) offers, the node's own appliance identity excluded, as sorted, comma-joined 64-char hex. It is resolved when the app is installed or started, so changes to your identities, including removals, take effect on the app's next restart: until then a removed identity keeps its access; with no such identity the app refuses to start rather than render an empty value |
| `app.container.secret_env` | Environment values read from `/var/lib/archipelago/secrets/<secret_file>`, injected as podman secrets (never visible in `podman inspect` or unit files) |
| `app.container.generated_secrets` | Secrets the orchestrator creates on first use (`hex16`/`hex32`/`base64`/`bcrypt`) — self-healing, 0600, no host provisioning |
| `app.container.generated_certs` | Self-signed TLS certs materialised before create; CN/SANs rendered from host facts |
@@ -765,3 +765,13 @@ Every supported app must satisfy the lifecycle contract:
For apps with special dependencies, launch must explain dependency wait states instead of showing a dead iframe. Examples include Bitcoin sync/IBD, Lightning wallet readiness, Nostr signer bridge injection, Tailscale login/auth, and app-specific setup screens.
Runtime changes should be validated with focused tests first, then the release lifecycle harness on the validation host when host access is intentionally resumed.
### Adapters for shared services
A service that reuses an installed stack can declare `install_prerequisites`
with the required component app ids and keep the runtime relationship in
`dependencies`. This refuses an incomplete installation before creating the
adapter instead of reporting a successful installation with no usable backend.
For example, Angor Indexer requires `mempool-api` (shown to users as its owning
Mempool app), shares that index and declares only an `api` interface. API-only
interfaces belong in Services and do not generate browser launch buttons.
+111
View File
@@ -0,0 +1,111 @@
# App lifecycle repair — 2026-09-30
Status: source repairs, optimized build, new-node recovery and scoped live
lifecycle acceptance verified. Full release gate remains pending.
These are next-release changes. Published 1.8.21 artifacts remain unchanged.
## Report
The operator reports that restarting an app can make it disappear, and a hard
refresh offers installation again. Newly installed apps sometimes fail to
connect in both embedded views and browser tabs. The new X250 additionally reproduced GitWorkshop disappearing during install
and Nginx Proxy Manager spending approximately 14 minutes at 70%. A disposable
app on the dev box exposed a separate restart failure.
## Findings and repairs
- Quadlet removes containers during stop/restart. The scanner protected existing
in-memory entries but did not reconstruct an absent app on a fresh daemon.
It now synthesizes stopped entries from the durable installed set, respecting
uninstall records, normalizing container prefixes, and preserving cached
metadata. Absence does not establish an image version or available update.
- Concurrent read/modify/write operations could lose installed-app records;
in-place writes could expose truncated JSON to readers. Serialize writers,
publish by atomic rename, and sync the file and parent directory. Legacy
package install/uninstall success paths update the durable record too.
- Scans and lifecycle/progress operations could replace a newer model from an
older snapshot. Use locked mutations for lifecycle/progress, and merge scan
results only into entries unchanged since the scan's merge snapshot.
- Container running state and TCP accept alone did not establish HTTP readiness.
Add explicit `ui-ready` based on bounded HTTP probes of the loopback upstream;
reject connection failures and server errors, accept normal redirects and
authentication challenges, and do not follow redirects or send credentials.
Self-signed HTTPS apps are probed locally without certificate validation.
- The app gate swept new listeners only every 60 seconds. Wake that sweep
immediately for a ready upstream whose declared gate port is not yet claimed,
and withhold readiness until external and Tor listener claims exist.
- Fixed launch URLs could bypass suppressed runtime URLs. Enforce readiness in
app cards, details, centralized embedded/browser launchers, and session frames.
Starting/restarting clears readiness immediately. A waiting frame does not
load an iframe and resumes when the backend reports readiness.
### New X250 findings
- The published ISO copied only `bitcoin-ui`, `lnd-ui` and `electrs-ui` build
directories. GitWorkshop failed because `/opt/archipelago/docker/archipelago-source`
was missing. Copy the complete docker source tree for bundled and unbundled
ISOs, matching OTA packaging. Validate every manifest build context and
Dockerfile in OTA staging, ISO staging and the mounted ISO smoke test.
- After restoring the omitted contexts, GitWorkshop's retained npm audit rejected
newly reported brace-expansion, fast-uri and ip-address vulnerabilities.
Refresh the existing pinned dependency patch, keeping the audit enabled.
Clean install/audit (zero advisories), type-check, 152 upstream tests and
subpath production build pass. The image builds on the X250 and `/healthz`
returns 200. No wallet or Bitcoin container restart was needed.
- Nginx was receiving data, not frozen: over 1 GB read during the pull. It
completed at 12:40:46 UTC after starting at 12:26:27; its web endpoint returns
200. The orchestrated path previously labelled the entire download/build/start
operation "Creating container" at 70%. Give that aggregate operation its own
truthful label and earlier phase; no byte-level download estimate is claimed.
- Restore install progress immediately from an already-loaded server snapshot,
so a new store created after hard refresh does not wait for another mutation.
- Replace the install modal's native version popup with inline radio choices.
On this actual X250's Chromium 152 kiosk renderer, selection changes work,
options have white text on dark backgrounds, and remain above pruning controls.
Screenshot and browser assertions captured; no install confirmation was clicked.
### Restart safety
The disposable fixture restart at 12:38:05 UTC stopped its container, then
`ss | kill` in runtime port cleanup sent SIGTERM to the management daemon at
12:38:35. The daemon owned the gate listener on the same port at other addresses.
Systemd restarted management; Bitcoin and LND container IDs/start times were
unchanged. Remove port-owner kills and broad `pkill` patterns from restart,
install recovery and Grafana preparation. Recovery now uses the existing
container-ID-aware ghost reaper: absent container ownership must be established
before a process is terminated. A real listening-socket regression checks that
conflict cleanup preserves the host listener. App-gate manifest lookup now honors
`ARCHIPELAGO_APPS_DIR`, matching the orchestrator's configured manifest root.
## Validation
- Full frontend suite: 139 files, 1,126 tests passed; final focused kiosk/store
checks: nine passed. Production frontend build passed.
- Final isolated backend suite: 1,567 passed, zero failed, four existing ignored
tests. Optimized backend build passed and was deployed to the development node.
- Tests cover empty runtime inventory, alias deduplication, uninstall exclusion,
concurrent durable writes, concurrent state changes, stale scan publication,
TCP-without-HTTP, HTTP statuses including 502/503, and gate listener claims.
- Live disposable Node fixture delayed HTTP startup by 25 seconds. Desktop and
mobile retained the waiting screen through hard refresh without mounting an
iframe, then opened the exact fixture page automatically when ready.
- Restart retained the app in both state APIs throughout and returned to ready;
the management PID did not change. Stopping removed the Quadlet container;
restarting management reconstructed its installed/stopped entry without a
false update offer. Starting it again succeeded. Desktop and mobile continued
to show the installed app after hard refresh.
- LAN access required node authentication and returned exact fixture bytes after
authentication. The fixture was uninstalled through the package lifecycle API;
its temporary manifest root and service override were removed.
- Bitcoin and LND container IDs and start times stayed unchanged through all
scoped checks and management restarts. No wallet data was used by the fixture.
- X250 kiosk checks also opened the repaired GitWorkshop and Nginx Proxy Manager
pages successfully, with no failed local resource loads.
## Limits
This prevents the identified lifecycle/readiness failures; it cannot guarantee
that an app or network never fails after a successful readiness check. Actual
application failures must remain visible rather than being labelled successful.
The full lifecycle/reboot release gate and funded acceptance of the reviewed
paid-download PRs remain pending. The X250 kiosk fix has live rendering evidence.
+36 -5
View File
@@ -93,7 +93,7 @@ Exactly **one** of `image` or `build` must be present (image XOR build).
| `network_aliases` | list of string | Extra DNS names on `network` (podman `--network-alias`) — lets stack members answer to short baked-in hostnames (`api`, `minio`, `relay`). |
| `entrypoint` | list of string | Entrypoint override. |
| `custom_args` | list of string | Extra positional args appended after the image. |
| `derived_env` | list | `- { key, template }` — template rendered against host facts at apply time. The allow-list is exactly `{{HOST_IP}}`, `{{HOST_MDNS}}`, `{{DISK_GB}}`, `{{BITCOIN_HOST}}` (`DERIVED_PLACEHOLDERS`); an unknown name or unbalanced `{{` fails validation. `{{BITCOIN_HOST}}` resolves to whichever Bitcoin app is running (`bitcoin-knots` or `bitcoin-core`, defaulting to knots). Never hard-code host specifics. |
| `derived_env` | list | `- { key, template }` — template rendered against host facts at apply time. The allow-list is exactly `{{HOST_IP}}`, `{{HOST_MDNS}}`, `{{DISK_GB}}`, `{{BITCOIN_HOST}}`, `{{NODE_IDENTITY_PUBKEYS}}` (`DERIVED_PLACEHOLDERS`); an unknown name or unbalanced `{{` fails validation. `{{BITCOIN_HOST}}` resolves to whichever Bitcoin app is running (`bitcoin-knots` or `bitcoin-core`, defaulting to knots). `{{NODE_IDENTITY_PUBKEYS}}` resolves to the Nostr public keys of the identities the app signer (the NIP-07 bridge's identity picker) offers, the node's own appliance identity excluded, as sorted, comma-joined 64-char hex. It is resolved at install and on every start, so changes to your identities, including removals, take effect on the app's next restart: until then a removed identity keeps its access; with no such identity the app refuses to start rather than render an empty value. Never hard-code host specifics. |
| `secret_env` | list | `- { key, secret_file }` — value read from `/var/lib/archipelago/secrets/<secret_file>` and injected as a **podman secret**, so it never appears in `podman inspect` or unit files. `secret_file` must be a bare filename (no `/`, no `..`). |
| `generated_secrets` | list | `- { name, kind }` — orchestrator materialises the secret on first use (0600, rootless service user, idempotent + self-healing). `kind ∈ hex16 | hex32 | base64 | bcrypt` (bcrypt writes `<name>` = hash and `<name>.pw` = plaintext). |
| `generated_certs` | list | `- { crt, key, common_name?, sans? }` — self-signed TLS materialised before create; CN/SANs rendered against host facts. |
@@ -291,14 +291,45 @@ Validate with `scripts/validate-app-manifest.sh` and regenerate the catalog
with `scripts/generate-app-catalog.py` (drift-checked in CI by
`scripts/check-app-catalog-drift.py`).
### Persistent-state backup for network migrations
### Persistent-state backup for runtime repairs
`app.backup_on_network_change: true` opts an app into a stopped-state snapshot
before an explicitly selected rootless network mode is migrated. The orchestrator
`app.backup_before_runtime_change: true` opts an app into a stopped-state snapshot
before reconciliation changes a service’s network, ports, security settings,
command or health configuration. Image-upgrade backup policy remains separate. The orchestrator
archives writable persistent bind mounts under the node data directory, collapses
nested mounts, excludes the runtime Podman socket, and preserves the previous
Quadlet definition for rollback. Named volumes, outside-data-root state and
symlinked mount roots fail closed rather than silently producing an incomplete
backup. A failed snapshot resumes the original service and leaves migration
pending. Private archives are retained under `migration-backups/`; fresh installs
and unchanged network configurations do not create migration snapshots.
and unchanged runtime configurations do not create migration snapshots.
Catalog generation preserves the previously published base manifest for older
daemons and puts opted-in network changes in a signed `manifest_variants` entry
requiring `runtime-migration-backup-v1`. New runtimes select only variants whose
complete requirement list they support. Supply `BASE_CATALOG` when generating
against a different reviewed pre-migration catalog. This keeps catalog refresh
from applying a migration before the matching OTA code is installed.
### Existing shared-service prerequisites
`app.install_prerequisites` is an optional list of existing app ids, for example
`[mempool-api]` for a headless indexer adapter. The runtime checks their manifest
container names before recording installation or changing any dependency. If one
is missing, installation refuses with its owning app's title and removes the
optimistic install tile. Runtime observation errors fail closed. This does not
automatically install dependencies, alter Bitcoin pruning, or require a synced
backend merely to recognize an already-installed service. Declare ongoing
relationships separately in `dependencies`; use the app health check for actual
API readiness. Self-dependencies and malformed ids are invalid.
### App owner identity placeholder
`{{NODE_IDENTITY_PUBKEYS}}` is opt-in per manifest. It gives the application the
comma-separated public keys of all user identities offered by the app signer,
excluding the appliance identity. It grants no signing capability or private
keys. Describe any resulting owner/upload privileges in the app documentation: a
shared list also links those identities to the same installation. An absent node
key, malformed owner key or empty owner set fails setup rather than rendering an
empty allow-list. Do not use this as an anonymous or per-profile authorization
mechanism. Existing manifests that omit it keep their existing configuration.
+42
View File
@@ -0,0 +1,42 @@
# Private signed-catalog qualification
Use this only on explicitly selected development/acceptance nodes. It permits
testing a release-root-signed catalog before fleet publication. It does not
publish an app image, change the update mirrors, replace the trust anchor, or
authorize an unsigned catalog.
Set `ARCHY_APP_CATALOG_CANDIDATE` in a management-service systemd drop-in to an
absolute local catalog path. Keep that file readable by the service and outside
temporary storage if testing reboot persistence. The file must be at most 4 MiB
and carry a signature verified against the configured release-root anchor.
Malformed, missing, unsigned, tampered and wrong-key candidates fail before
replacing the previous cached bytes. An invalid explicitly selected candidate
does not fall back to the public catalog. The previous cache remains available;
inspect the refresh error rather than assuming the candidate was accepted.
Before activation, record the exact candidate hash, service binary hash, native
Bitcoin/LND identities and start times, app configuration, and existing catalog
cache/drop-ins. Back up persistent state before any app runtime migration.
Verify the candidate signature with `archipelago ceremony verify PATH` and
retain the original signed bytes. A private signing ceremony is not publication
approval.
After management restart, verify:
- Cached bytes exactly equal the signed candidate and still verify.
- Desired app manifests select the expected capability-compatible variant.
- Changed apps migrate through their supported lifecycle, with state backups.
- Native wallets, intentionally stopped/uninstalled apps and unrelated services
retain their previous state.
- App requests succeed from the actual caller/container namespace; container
health alone is insufficient.
- Repeated reconciliation, app restart, and separately arranged node reboot
preserve routing, state, certificates and management isolation.
The setting intentionally pins catalog selection. Track its removal as part of
release completion: after the tested catalog is published and verified, remove
only the qualification drop-in, reload systemd, restart management, and confirm
the normal public refresh returns the expected signed catalog. Do not leave the
override behind to silently prevent future app updates. For an aborted test,
restore the reviewed previous catalog/runtime/configuration together; removing
the override alone can reintroduce older manifest settings.
+72 -9
View File
@@ -1,7 +1,7 @@
# Same-node Gitea sources in Portainer
Status: root cause reproduced and network repair verified in disposable Portainer
instances; final migration integration and release acceptance remain in progress.
Status: root cause reproduced and network repair verified in disposable and actual
production Portainer instances; final migration integration and release acceptance remain in progress.
This change belongs to the next signed catalog, OTA and ISO. It does not modify
published 1.8.21 artifacts.
@@ -37,12 +37,16 @@ this public record.
authentication. Remove obsolete port-3000 nginx metadata/template and the old
best-effort installer commands which silently rewrote app.ini and falsely
claimed success. Gitea owns first-run setup and operator configuration.
- Gitea SSH also failed before authentication: OpenSSH logged a denied
`chroot("/var/empty")` because the manifest dropped `SYS_CHROOT`. Add that
specific sandbox capability and reconcile security-directive changes. A
disposable fixture then passed SSH clone/push with host-key checking enabled.
- Existing Quadlet reconciliation applies Network= drift. Record a durable
pending restart before updating the unit and clear it only after a successful
restart, so failed reloads/restarts and management interruptions retry.
- Detect explicit rootless network-mode drift in the older Podman runtime too.
Unspecified networks do not trigger inferred changes to unrelated apps.
- Portainer opts into `backup_on_network_change`. Before recreation, gracefully
- Portainer and Gitea opt into `backup_before_runtime_change`. Before recreation, gracefully
stop the app and archive its writable persistent bind mounts, including nested
Compose state, once each. Runtime sockets are excluded. Save the previous
Quadlet definition, where present. Archives live under the node data directory's
@@ -73,12 +77,18 @@ verification stays enabled and API redirects are refused.
## Upgrade and rollback
The signed catalog embeds manifests and overrides installed disk copies. A disk
The signed catalog embeds manifests and overrides installed disk copies.
Capability-gated manifest variants keep the previous Portainer manifest as the
base for older daemons; only daemons supporting `runtime-migration-backup-v1`
select the network repair. This prevents catalog refresh from triggering an
unbacked recreation before the OTA is installed. A disk
edit alone cannot deliver this fix. Publish the matching catalog with the tested
runtime, then verify the generated unit, actual network mode and Source API.
Expect a Portainer interruption while the snapshot and recreation run; duration
depends on its saved state size.
Gitea does not need recreation or an app.ini rewrite for this repair.
The Portainer routing repair does not require a Gitea configuration change.
The separate SSH capability repair does recreate Gitea, preserving and snapshotting
both data/config mounts first. Supported systemd drop-in overrides remain intact.
Keep the previous trusted catalog/runtime for rollback. Restore that catalog
before restoring the saved `previous.container`, reloading user systemd and
@@ -94,11 +104,64 @@ repositories or the production Portainer database with disposable test data.
saved account/Source survive recreation; restart succeeds.
- Invalid Git credentials produce a repository-authentication error, distinct
from TCP refusal. Requested branch and Compose file read from Portainer context.
- Final expanded backend suite: 1,575 passed, zero failed, four existing ignored
- Combined backend suite including the reviewed paid-download PRs and catalog
rollout guard: 1,605 passed, zero failed, four existing ignored
tests, including stopped-state archive round trips and failure preservation. Container runtime suite: 78 passed.
Five diagnostic regression tests passed. Combined tests with the merged
paid-download PRs remain pending.
Five diagnostic regression tests passed; catalog regeneration is idempotent
and the generated catalog has zero manifest metadata drift.
- Fresh managed Gitea and Portainer fixtures: authenticated private Source
creation, invalid-token rejection, workstation clone/push and exact branch
lookup from Portainer namespace passed. LFS batch/upload/download and OCI
registry authentication/blob/manifest round trips passed. Desktop and mobile
login/private-repository/assets/hard-refresh checks passed.
- Still required before release: live automatic migration with the new runtime,
snapshot/rollback verification, private-repository and install-order acceptance,
snapshot/rollback verification and reversed install-order acceptance,
lifecycle/reboot convergence, and signed-catalog delivery to the existing app.
Record LFS/registry/SSH/browser checks and actual hardware/runtime coverage.
### Affected X250: production routing repair verified
Applied the tested rootless network setting to the actual installed Portainer
through a persistent Quadlet drop-in, after gracefully stopping it and creating a
private archive of its database and Compose directory. Compared the archive
against the stopped original before changing configuration; retained the original
unit and a rollback path. A verification helper initially compared mount list
order rather than mount identity and safely rolled back; the corrected check
compares sorted source/destination/write-mode tuples and passed.
The actual production Portainer namespace reproduced connection refusal before
repair. After repair it received a Git smart-HTTP advertisement, fetched the
requested branch at its current tip and read its Compose file. Repeating these
checks after restarting the managed Portainer service passed. All original data
and socket mounts and the loopback-only HTTP binding are retained. Gitea,
Bitcoin and the wallet container IDs and start times were unchanged. No stack
was deployed and no repository credential was changed.
This establishes the routing repair on the affected hardware. A logged-in
production Portainer Source UI/API acceptance has not yet been recorded; the
corresponding API checks passed on disposable instances as documented above.
The installed-node drop-in persists through service restart/reboot but is not the
fleet delivery mechanism. Automatic migration and signed catalog/OTA/ISO release
validation remain pending; the source manifest declares the same network mode.
Private deployment addresses, branch details and state archives are not committed.
### Managed automatic migration and archive restore
The new runtime candidate migrated an existing managed fixture from pasta to
slirp without a manual unit edit. It preserved the account, saved Source and
mount set, saved a private stopped-state archive plus the previous unit, restored
Source API access, and cleared the pending restart marker. A management-service
restart preserved the new container identity/start time and did not create
another archive. The archive extracted into an isolated scratch directory and
compared cleanly, including the database and Compose directory. Rootless archive
ownership required scratch cleanup inside `podman unshare`; no production data
was overwritten. Native Bitcoin and LND IDs/start times remained unchanged.
This optimized candidate predates the final bounded backup-retry guard; that
latest source passed the isolated 1,605-test suite and must also be exercised in
the final release build. A fixture-only systemd start failure was then injected during a security
directive migration. The failure retained the durable restart marker. After
removing the injected failure, the reconciler restarted the service without a
manual container start, restored Source API access and cleared the marker.
Reverse install order, final-build retry-budget coverage, full reboot and
signed delivery remain open.
+459
View File
@@ -0,0 +1,459 @@
# Next OTA and raw ISO after 1.8.21
**Status: COMPLETE — 1.8.22-alpha OTA, compatible signed app catalog and raw ISO published on Git and ngit on 2026-10-01; artifact signatures, public downloads and fleet feed verified. Angor full-chain indexing still awaits dev Bitcoin synchronization.**
Current acceptance evidence: [1.8.22 release acceptance](release-1.8.22-acceptance.md).
The chronological notes below retain earlier failures and superseded candidates;
the final tested source is `6d5f3ffb`.
This is the consolidated execution checklist for the operator's chat requests.
Release acceptance and publication are complete, with live wallet and app data
preservation checks documented below. No universal absence of future failures
is claimed.
## Changes already shipped in 1.8.21 or earlier
Keep these fixes in the next build and include relevant regressions:
- Mempool image/catalog version agreement and update-button behavior.
- Minibits integration; Framework automatic LND startup and safe unavailable
balances. Framework incident closed with operator acceptance.
- Shorter, single-column ecash backup messaging.
- AIUI transparent background on desktop/mobile.
- Cashu paid-file keyset/mint/error/refund corrections, with live purchases.
- mempool.space explorer fallback, preserving local/custom explorer settings.
- Bitcoin install pruning choice and matching automatic-pruning behavior.
- Friendly Bitcoin warmup and LND install/start/sync waiting states.
- Raw ISO publishing and upload support.
The Primal automatic LNURL comment problem was traced to sender behavior and
Minibits metadata. The user accepted clearing the sender's automatic comment;
no unsupported local metadata rewrite or wallet-identity replacement is planned.
See the Framework incident and 1.8.21 execution records for evidence/limits.
## New release scope and gates
| Task | Implemented/verified | Remaining before release |
| --- | --- | --- |
| X250 Bitcoin picker | Inline choices; actual Chromium kiosk selection, readability and pruning layout passed | Final UI/build checks passed |
| App disappearance/readiness | Durable inventory and safe lifecycle repair; delayed HTTP and desktop/mobile hard-refresh checks passed | Final candidate lifecycle, hard-refresh and stability checks passed |
| X250 GitWorkshop/Nginx | Missing build contexts restored, dependency/build checks and live UI passed; Nginx slow pull diagnosed; truthful progress label | OTA and ISO build-context/content checks passed |
| PRs 161/162 | Reviewed, repaired, merged/closed normally; combined regression suite passed | Funded Tor-only candidate purchase, retained change, refund, Files bytes and cached repeat passed; included in signed artifacts |
| Gitea/Portainer | Root cause confirmed; source network/backup/retry/catalog changes; real X250 routing repair and restart verified; private Git, SSH, LFS, registry and browser fixture checks passed | Automatic migration, scratch restore, failed-start recovery and reverse installation order passed. Operator confirms production site works through Portainer; production host reboot also preserved network/Git/Compose access; final candidate delivery and integration checks passed |
| Angor headless store service | Implemented standard Mempool adapter and separate optional relay, official logo, headless store entries and declarative dependency guard. API security/outage/DNS tests and five relay lifecycle cycles passed | Final candidate prerequisite/API, lifecycle and catalog checks passed; real indexing on dev waits for Bitcoin sync |
Durable payment receipts after a lost seller response remain a separately
recorded design follow-up. Preserve the truthful unconfirmed-refund warning and
prevent duplicate automatic payment; do not describe an unconfirmed refund as
completed. See PR review for the accepted scope and coverage limits.
## Final release checklist
- [x] Finish new-scope implementation and release acceptance; full-chain Angor
indexing still depends on the dev node finishing initial sync.
- [x] Remove disposable fixtures and temporary test overrides; verify native
Bitcoin/LND identity and start-state baselines remain protected.
- [x] Commit and push completed source changes to git and ngit.
- [x] Run final backend/UI/regression/release gates on the final source; inspect
skipped tests and report actual hardware/runtime coverage.
- [x] Prepare compatible signed app catalog; old runtimes must not apply a
migration before they have backup/recovery support.
- [x] Version/changelog and OTA payload prepared, validated and signed by user.
- [x] Raw ISO built; payload hashes/content verified; full installation and
installed-system boot tested in QEMU/KVM without network.
- [x] User signs ISO checksums; publish OTA and ISO plus verification files on
git and ngit; independently read back hashes and update discovery.
- [x] Provide LAN scp command for the new raw ISO.
Latest backend source verification: 1,617 passed, zero failed, four existing
ignored tests. This is one layer of evidence, not a substitute for live gates.
## Angor verification — 2026-09-30
- Isolated backend suite: 1,606 passed, four existing ignored; container suite:
79 passed. Frontend: 140 files / 1,130 tests passed; production build passed.
- Disposable rootless API gateway: versioned and legacy API paths, query/body
forwarding, transaction-only POST, method/body limits, CORS, removal of
dashboard credentials, read-only non-root operation, truthful backend outage
and DNS recovery after backend recreation passed. No real transaction broadcast.
- Dedicated relay: NIP-11, signed event publish/read, invalid signature rejection
and event/config persistence across five managed stop/start/restart cycles
passed. Internal relay identity and start time stayed unchanged. Follow-up
acknowledgement samples were 2–9 ms through both backend and app gate.
- Published adapter 1.0.1 and relay 1.1.2 to the authenticated maintainer namespace.
Anonymous registry readback succeeded. Adapter digest:
`sha256:997be611700b55c521ad801fa92daaca2ae6951ac71407434c85eb9603f77c38`;
relay mirror digest:
`sha256:80444ad1304a0e504948b48ea1550c091b18b9f10757f07ce9a68fc261b8f6c1`.
- Delivery target is the development box, as clarified by the operator. Do not
install Angor on the separate Portainer node. Full indexer availability still
requires the dev box's Bitcoin sync and Mempool/Electrum indexing to finish.
- Funded PR acceptance passed after the operator funded the dev Cashu wallet
with 16 sats. Exact net payment was 1 sat; underpayment refunded in full;
repeat delivery cost zero. Both endpoints ran the combined candidate.
No spent proofs were reactivated and no native Bitcoin/LND funds were moved.
## Development candidate and cleanup
The combined optimized backend and production UI are deployed on the development
box with a private rollback copy. Native Bitcoin/LND containers were unchanged
during deployment. The operator separately uninstalled/reinstalled Bitcoin Core
to select an unpruned node; RPC confirmed `pruned=false`, and a separate baseline
was recorded after that operator action. Do not compare subsequent checks with
the pre-reinstall container start times.
Completed Gitea setup/private-repository and Portainer integration fixtures were
uninstalled through the supported lifecycle and removed from installed inventory.
Their private evidence/data were retained outside the active manifests. The old
Cuprate UI review container was also removed. Active Angor acceptance fixtures
must be removed on completion; the requested Angor services remain installed.
Funded acceptance used Tor-only peer-file transport, verified exact delivery
bytes and compatibility response fields, and read the result back through
FileBrowser. The original transport preference was restored, and temporary
seller catalog entries/files and the exact buyer test document were removed.
Financial receipt history was retained.
The final managed-install fixture exposed a separate Quadlet quoting defect:
whitespace-free command arguments containing apostrophes lost those characters
in the generated service. The renderer now quotes these arguments and
environment values; the updated isolated backend suite passed (1,607 passed, four opt-in tests
ignored), and the final candidate rebuild is in progress. Do not tag a release before this live regression is verified.
The production Portainer host subsequently rebooted after the routing repair.
A post-boot probe from the actual Portainer namespace again verified the Git
smart-HTTP response type, current branch ref and Compose contents. Its
slirp4netns route and all production app containers survived. The temporary
Portainer fixture was absent. This verifies the repaired production route
across reboot; it does not substitute for final new-runtime delivery checks.
## Follow-up acceptance: app cards and Angor icon
- Mempool duplicate traced to `archy-mempool-web` durable inventory alias being
restored beside the real `mempool` frontend. Shared scanner canonicalization
fixes live and absent-container paths without deleting installed markers.
Frontend suppresses aliases only while a canonical tile exists.
- Readiness text names the app and condition: “Web UI not ready: Gitea”. It shares the status row,
with full text available through its title; card actions use bottom alignment.
- Angor uses the operator-supplied dark-mode icon with green outer corners.
Built-in imagegen prompt: fill transparent/white corners with the existing
flat green, preserve the black symbol, square opaque PNG, no added details.
- Backend alias suite: 1608 passed, 4 ignored. Focused readiness/frame UI tests:
30 passed. Production UI build passed and is live on dev. Browser checks at
1440 and 1024 pixels verified named waiting text, bottom-aligned actions,
equal row heights, no overflow and one Mempool card after hard refresh.
The 390-pixel mobile icon layout also passed hard refresh. Final-source
isolated Mempool alias regression passed after the scanner simplification.
- Managed Angor adapter acceptance: five stop/start/restart cycles, missing
prerequisite refusal, management restart and cleanup all passed. Both temporary
fixtures and their network were removed. Actual dev API verification remains
pending after removing an incomplete legacy-created adapter.
## Startup manifest reload race
Live Angor acceptance exposed a separate startup race: runtime asset bootstrap
cleared and copied `/opt/archipelago/apps` in the background while the startup
catalog refresh reloaded it. The daemon logged 62 loaded manifests followed by
54 and then rejected the new disk-only app as unknown. A stable manifest snapshot
confirmed the diagnosis: supported uninstall/reinstall produced the correct
rootless Quadlet service with its declared port and network.
Runtime promotion and the legacy installer-directory repair now finish before
orchestrator construction. The background doctor no longer changes that tree.
The final source backend suite passed 1,608 tests (four existing opt-in tests
ignored). Optimized build and normal-path live startup/restart verification have now passed (see final follow-up below).
Actual dev Angor acceptance passed managed service identity, no capabilities,
UID 101:101, archy-net, public block height, CORS and both fee URL forms. During
Bitcoin initial sync, the real Mempool fee API returns 503; the adapter faithfully
returns the same status and body. Full-sync fee availability remains unverified;
ready-backend API and failure/recovery behavior passed the isolated live fixture.
The temporary `/run/archy-candidate-manifests` snapshot override and snapshot
are now removed; normal startup/reload verification passed.
The latest complete UI suite passed 140 files / 1,132 tests. Release preflight
passed all static, manifest, catalog, type and UI gates. The requested named
waiting message, compact card layout and green Angor icon are deployed to dev;
desktop 1440/1024 and mobile 390 browser checks passed after hard refresh.
The startup-order optimized build and normal-path startup checks are complete.
The later dashboard-address candidate is now deployed with rollback; see the
final live follow-up below. Do not rerun the earlier deployment helper: its
temporary override has already been removed. No new release version/tag, OTA
or ISO has been created.
## Mempool and dashboard follow-up
- Deployed the Mempool alias and runtime-promotion-order backend to dev. Real
server state contains one healthy `mempool`; the stale `mempool-web` record
is gone. Real-data browser checks at 1440/390 pixels found exactly one tile
before and after hard refresh. Bitcoin/LND identities/start times unchanged.
- Removed the candidate manifest override. Normal management startup passed
two full cycles with 62 manifests retained through both initial catalog
refreshes. The next cycle hit a single readiness assertion; a subsequent
read-only check found Angor healthy and the manifest count intact. Remaining
repeat coverage should use bounded polling to distinguish transient request
failures from loss of app definitions; do not report five cycles passed yet.
- Bitcoin Core's dashboard was serving HTTP 200 on 8334 while readiness checked
RPC 8332. Companion URL selection now takes priority over protocol sockets
for Core/Knots and Electrum aliases, with a regression preserving allocated
UI ports for other apps. Backend suite: 1,609 passed, four opt-in ignored.
Optimized build is `/tmp/archy-dashboard-address-build.log`; deployment and
live IBD verification helper: `/tmp/archy-dashboard-address-deploy.py`.
- Phoenixd has no browser UI. Headless services now omit web-readiness messages;
actual browser apps name their web interface rather than waiting for
themselves. Focused 23 UI tests and production build passed; deployed to dev.
## Final live follow-up: all three reported readiness/display defects fixed
- Final optimized backend is deployed on dev. Bitcoin Core's launch address is
`http://localhost:8334` and `ui-ready` is true during initial block download.
Live verification recorded height 293,855 with `initialblockdownload=true`.
Chromium at 1440 and 390 pixels opened the embedded dashboard, read a numeric
current block height, and repeated that check after hard refresh.
- One healthy Mempool remains in server state and in desktop/mobile My Apps
after hard refresh. Its durable install markers were preserved.
- Phoenixd remains a running headless service without a web launcher or false
web-readiness message. Desktop/mobile browser checks passed. For actual web
apps, the compact copy is “Web UI not ready: [app]”; the reason comes first so
narrower cards do not truncate it into a misleading self-dependency.
- Five normal management startup and managed Angor restart cycles passed
across the two acceptance logs. The retry harness uses bounded readiness
polling; it does not accept a running container alone as API readiness.
Disk + catalog manifest count remained 62 across startup refreshes, replacing
the previous 62-to-54 failure. Temporary override and snapshot are removed.
- Final backend tests: 1,609 passed, zero failed, four existing opt-in ignored.
Final UI tests: 140 files / 1,133 passed. Production UI build passed and is live.
Bitcoin/LND container identities and start timestamps stayed unchanged.
- Evidence: `/tmp/archy-dashboard-address-deploy.log`,
`/tmp/archy-bitcoin-ibd-browser.log`, `/tmp/archy-mempool-live-browser.log`,
`/tmp/archy-service-readiness-browser.log`,
`/tmp/archy-runtime-order-remaining-cycles.log`, and
`/tmp/archy-readiness-final-ui-tests.log`.
- These are live development fixes. The new signed catalog, versioned OTA and
raw ISO still need preparation, artifact verification, signing and publication.
### X250 Nginx Proxy Manager tunnel repair (2026-09-30)
A further live report was a real startup failure, separate from the earlier slow
image pull. An operator-specific Quadlet `web-tunnel.conf` published NPM's HTTP
listener on tunnel port 18080. LND subsequently occupied 18080 on all addresses;
pasta failed before NPM could start, with more than 1,400 systemd retries. The
standard NPM manifest only publishes admin port 8081 and did not introduce this
extra mapping. Changing the standard manifest would not repair this override.
The node's override now uses free tunnel-local port 18081. Its persistent nftables
configuration redirects only HTTP arriving from the configured WireGuard peer on
the original tunnel destination to that port. The peer/public routing is unchanged;
the input rule accepts the translated port and retains the existing interface,
peer and forwarding restrictions. LND's REST port and native processes were not
changed. This deployment-specific topology must not be copied into global app
manifests or applied indiscriminately to other nodes.
An abandoned certificate request also left an unreferenced database record and
a temporary nginx challenge server for the same hostname. After backing up the
entire NPM data directory and both configuration files, the unused failed record
was soft-deleted and the stale challenge file archived. The referenced, valid
certificate, proxy host, keys and user accounts were preserved.
Live checks: NPM admin and API HTTP 200; nginx configuration validation with no
duplicate-host warning; public HTTP redirects to HTTPS; valid public TLS reaches
the site's existing authentication response, matching its direct upstream. NPM
starts with zero automatic restarts and no missing-certificate renewal error.
Bitcoin, LND and the production site container identities/start times were
unchanged by the port repair. Rollback copies and the data archive are retained
in the node's private support directory. No global OTA or ISO was published by
this repair; the remaining release gates above still apply.
#### Follow-up: fleet delivery and false health failures
A longer observation exposed a second, generic defect after the port conflict
was repaired: the health monitor probed all published ports at `127.0.0.1`,
including NPM's tunnel-only listeners. Every monitor interval could therefore
restart a healthy app. The short initial restart check did not catch this.
The next backend now probes the actual `host_ip` from Podman; only wildcard
addresses map to the corresponding loopback family. Regression tests cover
explicit IPv4/IPv6 binds, wildcards, UDP/unpublished/invalid entries, and a real
listener on a different loopback address. NPM's manifest now checks its internal
admin HTTP API. The same check is deployed as a persistent Quadlet drop-in on
the affected node so its older backend stops making false recovery attempts.
The backend embeds `scripts/repair-npm-tunnel.py` and runs it before app
reconciliation, after runtime asset promotion. This makes the targeted legacy
port migration available to both OTA and ISO installations without relying on
an independently installed script. Standard fresh installs are a no-op. Only
the recognized legacy tunnel/firewall profile is migrated; unknown operator
routing, occupied replacement ports and live-only firewall changes fail closed
with a startup warning. Configuration backups, an interrupted-migration journal,
atomic nft transactions and rollback protect the existing routing. Native wallet
services and certificate databases are never modified by this fleet migration.
The Python migration tests run in the release gate. The unsigned next catalog
was regenerated successfully with the new NPM HTTP health check. These changes
are prepared for the next release; existing published OTA/ISO artifacts remain
unchanged and the new signed artifacts still require the release gates above.
Verification for this follow-up: 18 migration tests passed; 43 health-monitor
backend tests passed through the isolated runner. A disposable network-namespace
regression exercised actual peer traffic through the nft redirect while a
separate simulated LND listener retained port 18080. The generated rules also
passed nft validation and atomic replacement. Run that regression with
`sudo unshare --net python3 tests/regression/npm-tunnel-network.py`; it refuses
to run in the host network namespace. The migration is a verified no-op on the
already repaired node and on a standard development install without the override.
After deploying the API health check, a 270-second live observation crossed
multiple health-monitor intervals: NPM stayed healthy with the same container
ID/start time, every API probe returned success, and Bitcoin/LND/production-site
container IDs/start times were unchanged. This supersedes the initial short
restart-only acceptance recorded above. The generic backend fix is committed
for release, while the live node uses the equivalent internal NPM health check.
### Final-gate Angor health-check correction
Final release observation found the adapter healthy over IPv4 but marked
unhealthy by its in-container BusyBox wget: `localhost` resolved to `::1`, where
nginx does not listen. Its manifest now explicitly probes `127.0.0.1`. The live
managed service was refreshed and its real Podman health check passed. The
rootless gateway integration now runs the manifest's health check inside the
actual image, in addition to endpoint/security/outage/DNS recovery assertions;
all passed. A metadata regression covers the address-family requirement. Test
containers and their network were removed by the fixture cleanup.
## 1.8.22-alpha release preparation
Final implementation gate passed: 1,612 isolated backend tests, zero failures,
four existing opt-in tests ignored; 140 frontend files / 1,133 tests; frontend
type check and production build; static/catalog/trust/build-context checks.
The four exclusions require external AI backends, Reticulum subprocess/live
transport, physical RNode hardware, or creation of a live Minibits profile.
They are not claimed as executed by the isolated suite. Existing funded
Cashu/Minibits and Framework acceptance remains recorded above.
The real dev Angor stack now returns HTTP 200 fee estimates through both API
forms; Bitcoin is still in initial sync, so full-chain completion remains an
operational prerequisite rather than a completed test. The image-level health
probe, outage/recovery and live native-state preservation checks passed after
reloading the corrected manifest. Production Portainer again fetched the exact
repository branch and Compose content from its own network namespace.
Version preparation is 1.8.22-alpha. No new release tag or fleet-visible update
manifest is published by the version commit. Optimized candidate deployment,
artifact inspection, ISO smoke/boot checks and offline signatures follow.
### Release blocker discovered during candidate observation: scheduled doctor
The initial `02b840f2` 1.8.22 candidate is rejected for release. On the X250,
2026-09-30 21:10–21:11 UTC, the scheduled `archipelago-doctor.service` explicitly
ran `podman stop --all --time 30`, killed rootless network helpers and ran
`podman system migrate` after a two-attempt external network probe failed.
The journal attributes the stop to that unit, not the app health monitor or a
host reboot. All apps restarted, including Bitcoin, LND and the production site.
The earlier unchanged-container acceptance applies only to immediate deployment;
the later observation failed and must not be represented as a stability pass.
No persistent-data loss has been established. Keep this distinct from the closed
Framework incident; do not wipe or recreate any wallet as a recovery action.
Containment: stopped doctor timers on both test boxes, installed a safe diagnostic
script into both the executable and runtime payload, and rejected/stopped the
old ISO build. Network failure now produces a warning without stopping apps,
killing network processes, migrating Podman or deleting network state. Repeated
failures remain warnings, never a successful repair/check. Regression cases cover
healthy, absent network, non-root invocation, host failure, transient recovery,
repeated endpoint failure and namespace access failure, with mutation tripwires.
Live scheduled-cycle observation and final rebuilt-artifact acceptance are pending.
Recovery also exposed retired `git.tx1138.com` nginx base references in six
companion UI Dockerfiles. They now use the existing primary registry at the same
pinned version. All six images built successfully against that registry; payload
validation rejects the retired host before OTA/ISO packaging.
The post-recovery X250 check passes: Bitcoin authenticated RPC responds and IBD
advances; NPM/Gitea/Portainer APIs respond; Portainer's real namespace fetches
`demo-portainer` at `3ae171d6b0c728665a860520fe393c0abb772798` and its Compose
file; Portainer's original persistent mounts match the earlier backup evidence;
LND wallet/channel databases remain present on their persistent mount. No new
pre-incident cryptographic wallet-identity baseline was available, so these checks
must not be described as an exact identity/balance comparison.
The dev all-container observation also caught a separate Cuprate UI orphan loop:
`companion.rs` removed it because Cuprate was not installed, while generic desired-
state recovery resurrected it from an old running snapshot, using a unit without
nginx's required capabilities. Generic desired-state recovery now excludes missing companions
owned by `companion.rs`; existing companion provisioning/reaping remains the
single owner. Running UIs still receive the existing security configuration repairs. Regression runs repeated reconciliation against stale companion
snapshots and checks that no image/container lifecycle operations occur.
Safe-doctor live acceptance: the X250 completed a 12-minute observation with all
running container IDs, start times and data mounts unchanged. Its journal records
successful doctor runs at 21:22:06, 21:27:51 and 21:33:10 UTC. Both doctor timers
are restored with the safe script. Dev's native Bitcoin/LND stayed running;
all-container dev acceptance remains pending the companion-loop backend fix.
Final-source UI suite: 1,133 passed. Heavy backend compilation is serialized with
remaining build steps to reduce memory/IO pressure on the syncing dev node.
Final source release gates at `96fb5a4f`: 1,613 backend tests passed, zero failed,
four explicitly ignored; 1,133 UI tests passed; type-check, production UI build,
catalog/trust, shell, pruning, LND readiness, NPM migration and doctor regressions
passed. The isolated companion-loop regression passed independently as well.
ISO cache hardening: the installer now carries the current doctor script and
service/timer separately from rootfs.tar and overwrites both historical and active
script locations before first boot. This prevents a cached base image restoring
the old recovery code. A regression executes the actual installer block against
stale disposable files twice and confirms a missing safety payload fails closed.
The mounted-ISO smoke test also compares all three overlay files to source.
The final ISO build captures the exact newly deployed OTA UI/runtime payload.
### Final kiosk acceptance found nondeterministic NPM launch selection
Do not publish the staged `d1bc1273` candidate. NPM itself remains healthy and its
API, Portainer integration, site, and native services passed stability checks.
However, final kiosk acceptance found its card stuck at "Web UI not ready".
The runtime reported bindings in proxy-HTTP, proxy-HTTPS, admin order. The scanner
chose the first non-database/SSH binding, then rejected its tunnel-only host port
as unreachable on loopback, leaving the launch address empty. Earlier tests had
passed with admin first. This is a confirmed order-dependent scanner defect.
The candidate fix explicitly resolves NPM container port 81 to its actual host
allocation. Proxy ports never become the admin URL. Missing/malformed admin
bindings do not fall back to another service when published bindings are present.
Port parsing handles IPv6 authorities and rejects invalid ports. Regressions
cover all six three-port permutations, allocated admin ports, IPv4/IPv6 binding
strings, missing admin mappings, missing runtime port information, and malformed
or UDP bindings. Full backend regression execution is pending for this change.
The ISO build is frozen at installer-environment creation; no release was signed
or published. Rebuild/revalidate the OTA and ISO with this correction.
The companion orphan fix worked live: Cuprate UI was automatically removed and
all installed app container IDs remained unchanged. One observation helper raced
that expected removal between `podman ps` and `inspect`; it now excludes that
known orphan before inspection and repeats the stability check. This was a test
snapshot race, not another installed-app restart.
NPM selector final backend gate passed: 1,617 tests, zero failures, four explicitly
ignored, through the isolated runner. This includes all new port-selection cases
and the existing companion security/configuration and lifecycle regressions.
Rebuild the release binary and UI metadata, deploy those exact OTA bytes to both
boxes, and require actual kiosk hard-refresh/Launch acceptance before ISO assembly.
## Final accepted artifacts — 1.8.22-alpha
Source `6d5f3ffb` passed 1,617 backend tests (four explicit opt-in exclusions),
1,133 frontend tests and final release gates. Exact OTA bytes were deployed to
both boxes. Actual X250 kiosk NPM Launch, version/pruning, desktop/mobile
readiness/AIUI, production Portainer Git/Compose and 12-minute stability checks
on both boxes passed. No installed app was restarted by the safe diagnostics,
and the Cuprate orphan stayed absent. Native Bitcoin/LND and the production site
were preserved during final management deployment.
The raw ISO passed mounted payload checks and matches all 653 OTA frontend/runtime
files plus the backend. Full offline installation and installed UEFI boot to the
visible setup screen passed in a disposable QEMU/KVM VM. Both installed doctor
paths and the installed backend have the expected hashes. No VM wallet was set up.
See `release-1.8.22-acceptance.md` for exact artifact hashes, hardware/runtime
coverage and limits. Draft upload verification, offline signatures, publication
and public readback remain; the fleet still advertises 1.8.21 until those gates
finish. Do not confuse a draft asset or source push with completed publication.
+294
View File
@@ -0,0 +1,294 @@
# NPM certificate failure: repair and next-release gate
Status: OPEN for release — affected Angor endpoint repaired and publicly verified;
durable source correction and release regression acceptance remain pending.
The operator requested immediate repair on Shorty's node and a complete, tested
fix for subsequent releases. The independent review agent acknowledged receipt
of the repair/handoff on 2026-10-01. This does **not** establish receipt by the
owner of another release session. That owner must acknowledge this document and
record implementation and acceptance before shipping the next OTA or ISO.
## Evidence and immediate repair
The live investigator reported NPM certificate failures at 18:06, 18:07 and
18:10 UTC on 2026-10-01: the CA received HTTP 404 for its HTTP-01 challenge.
The running container mounts `/var/lib/archipelago/nginx-proxy-manager` at
`/data`, but host nginx served the challenge from the obsolete nested
`/var/lib/archipelago/nginx-proxy-manager/data/letsencrypt-acme-challenge`.
NPM writes to `/data/letsencrypt-acme-challenge` inside its container. These are
different host directories. Host nginx owns public ports 80 and 443.
The investigator backed up `/etc/nginx/sites-available/archipelago` as
`/etc/nginx/sites-available/archipelago.before-angor-acme-1790878342`, corrected
the default HTTP challenge root to the actual mount's challenge directory,
passed `nginx -t`, and reloaded nginx. A temporary challenge file written inside
NPM returned its exact expected body over the public domain's port 80.
This initial probe proves the repaired challenge route; it alone does not prove issuance,
certificate attachment, public HTTPS routing, renewal, or release persistence.
No wallet or channel data is involved in this repair.
## Independently confirmed source inconsistencies
- `apps/nginx-proxy-manager/manifest.yml`: base app directory mounts at `/data`;
separate `letsencrypt` directory mounts at `/etc/letsencrypt`; only the admin
port is published. NPM's own public listeners are not published by this path.
- `image-recipe/configs/nginx-archipelago.conf`: default challenge location uses
the obsolete nested `data/letsencrypt-acme-challenge` root.
- `scripts/sync-npm-public-hosts.sh`: both SQLite DB and challenge root use the
nested directory. Missing DB exits successfully without syncing any hosts.
- `scripts/container-doctor.sh::fix_npm_public_hosts`: its independent nested
DB existence guard prevents the synchronizer from running on manifest installs.
- `core/archipelago/src/api/rpc/package/config.rs` and `runtime.rs`: legacy
creation/repair still mount the nested `data` directory at `/data`.
- `scripts/first-boot-containers.sh`: legacy first boot creates and mounts nested
data, and publishes different public-listener ports from the manifest path.
- The synchronizer exports selected NPM fields to host nginx. It checks enabled
hosts with a certificate, but does not filter deleted rows, validate inserted
configuration values, or preserve all NPM access/custom-location behavior.
It restores the old generated file on syntax failure, but not reload failure.
## Required implementation
1. Define one authoritative method for finding the active NPM data mount and
certificate store across fresh installs and supported legacy layouts. Do not
blindly change mounts and strand the operator's existing DB, hosts or account.
Detect ambiguous dual databases explicitly. Back up before any migration;
preserve existing certificates, private keys, renewal files, account records,
custom settings, permissions and uninstall decisions. Repeated migration must
be harmless. Failed migration must leave the original usable state intact.
2. Make default and named-host HTTP challenge routes use that active directory.
Include pre-certificate issuance and renewal under forced HTTPS. Do not expose
account/private-key/database directories through nginx.
3. Correct the synchronizer and doctor gate together, with reliable lifecycle
invocation after host/certificate edits and service startup. A certificate
created in NPM must actually become the certificate served by public nginx.
Avoid requiring users to run a repair command for each host or renewal.
4. Define how the host bridge preserves NPM routing and security settings.
Handle disabled/deleted hosts, host edits, certificate replacement/deletion,
multiple domains, custom locations and access restrictions correctly. Do not
silently publish a restricted NPM host as an unrestricted host-nginx proxy.
Validate DB-derived configuration, serialize concurrent writers, avoid
unnecessary reloads, and retain a working config on generation/test/reload
failure. Report actionable failure causes instead of apparent success.
5. Carry the correction through actual OTA migration and fresh ISO paths.
Include source, runtime scripts, nginx configuration, and app catalog as
applicable. Verify candidate package contents and installed behavior rather
than assuming a source edit is shipped by every packaging path.
## Acceptance matrix — all relevant gates require recorded results
Use disposable fixtures/test domains for destructive/error cases and ACME
staging for repeated issuance/renewal. Avoid production CA retry loops. Backend
unit tests on installed nodes must use `scripts/test-backend-isolated.sh` per
`AGENTS.md`. Do not reboot an operator node without the necessary recovery/access
arrangements and authorization; use a disposable VM for release lifecycle tests.
- [ ] Fresh manifest installation: actual mount, DB path, admin API, public
challenge file and first certificate request work without manual repair.
- [ ] Legacy nested-data upgrade: hosts, accounts, certs, renewal data and
custom settings remain intact; the active DB is still the original DB.
- [ ] Current flat-data upgrade: same preservation assertions; no empty database
is initialized and no old nested DB is silently chosen instead.
- [ ] Ambiguous dual DBs, missing/corrupt DB, backup failure and permission errors
fail safely with useful diagnostics; no deletion or identity replacement.
- [ ] Challenge file created in container is fetched byte-for-byte from public
port 80 for an unconfigured domain, named host and forced-HTTPS host.
- [ ] Staging first issuance completes through the normal NPM UI/API. One live
production issuance on the affected node is independently verified with
hostname, trust chain, validity and actual served certificate.
- [ ] Certificate binding and public HTTPS route reach the intended Angor
backend; normal API health and a representative read-only indexed request
are checked separately from TLS. Backend readiness failures stay explicit.
- [ ] Staging renewal succeeds through the normal scheduling/renewal path and
public nginx reloads the renewed certificate without manual intervention.
- [ ] Host create/edit/disable/delete, certificate replacement, multi-domain
routing, restrictions and custom locations match supported NPM behavior.
- [ ] Injection/invalid data, concurrent sync, nginx syntax failure and reload
failure preserve the previous working service; retries converge safely.
- [ ] NPM restart, manager restart, host nginx restart, controlled VM reboot and
repeated reconciliation preserve public routing and certificates.
- [ ] Migration is idempotent; upgrade rollback retains original data and
usable routing. Existing unrelated public hosts continue working.
- [ ] Signed OTA and RAW ISO candidate contents contain the same correction;
installed OTA legacy/flat layouts and fresh ISO pass the relevant checks.
- [ ] Release owner acknowledges receipt and records exact commit/artifact IDs,
test commands/results, live evidence, remaining limits and release decision.
## Handoff acknowledgements
- 2026-10-01: independent review agent received the parent investigator's live
fix details and explicitly acknowledged responsibility for source review and
this test/handoff checklist. No production code or node changes by reviewer.
- 2026-10-01: next-release owner explicitly acknowledged this handoff in
`/tmp/npm-release-handoff-ack.txt` and adopted the matrix as a required
1.8.23-alpha OTA/raw ISO gate. Received the operator report that Shorty
certificate npm-8 is issued and public HTTPS health returns 200. Independent
release validation and the durable fleet correction remain pending.
## Final live repair evidence — investigator report, 2026-10-01
The live investigator issued the certificate through NPM's own API using an
ephemeral in-memory local admin token, without changing passwords or disclosing
the token. Certificate ID 8 expires at `2026-12-30 17:16:35`; existing proxy host
ID 2 now uses that certificate with forced HTTPS enabled.
The running container publishes only its admin listener (container 81 to host
`127.0.0.1:8081`). The investigator therefore added a domain-scoped host-nginx
configuration at `/etc/nginx/conf.d/angor-indexer-npm.conf`. It serves HTTP 80 and
HTTPS 443 on IPv4 and IPv6, uses the corrected ACME root and NPM certificate 8,
and forwards to `http://127.0.0.1:8998`. `nginx -t` and reload passed.
External requests with normal TLS verification confirmed:
- `/health`: HTTP 200, indexed height 969474.
- `/`: valid mainnet JSON response.
- `/api/v1/fees/recommended`: valid JSON response.
- HTTP requests redirect to HTTPS with HTTP 301.
- CORS preflight `OPTIONS /api/tx` with origin `https://angor.io`: HTTP 204,
allowed origin `*`, method POST and header Content-Type. This was a preflight
check, not a transaction submission.
- An existing shop endpoint continued to return HTTPS 302.
The temporary challenge probe was removed. These are investigator-reported live
checks, not independently repeated node checks by the review agent. They establish
that the affected endpoint now serves trusted HTTPS and responds as an indexer.
They do not establish renewal, automatic host bridge updates, restart/reboot or
packaged-release correctness; the source repair remains the release owner's work.
A separate read-only public probe of the existing Shorty's website failed TLS
hostname verification. Its configuration was unchanged by this repair, and no
pre-repair baseline establishes when that mismatch began. The release owner must
investigate it separately and verify existing-host compatibility; do not attribute
it to this repair without evidence or silently mark that gate passed.
The investigator located the actual main release session, delivered the handoff,
and observed its explicit acknowledgement. The owner then recorded receipt in
`/tmp/npm-release-handoff-ack.txt` and in the acknowledgement section above.
Publication remains held for the NPM release gate. Unavailable external acceptance
must be stated explicitly and cannot be silently treated as passed.
## Urgent public dashboard exposure gate — 2026-10-01
Investigator reports the Angor relay hostname reached the default Archipelago
login because its certificate existed without a corresponding host-nginx route.
The investigator owns the immediate Shorty nginx repair; the release session
will not modify that configuration concurrently. Exact final evidence is pending.
- [ ] Unknown public HTTP Host / TLS SNI and direct public-IP requests cannot
expose the dashboard, login assets or RPC, including IPv6 and any trusted
reverse-proxy/tunnel path. Test spoofed forwarding headers explicitly.
- [ ] LAN/private/tailnet dashboard access remains available as intended.
- [ ] Public HTTP ACME challenge access survives those restrictions.
- [ ] NPM host creation/edits automatically propagate HTTP/TLS routing.
- [ ] Relay hostname serves the intended relay and WebSocket upgrade using its
correct certificate; certificate existence is not route acceptance.
- [ ] These protections survive manager/nginx restart, renewal and OTA/ISO.
## Live security containment and project discovery follow-up
2026-10-01: relay certificate existed but named public route was absent; default
HTTP/HTTPS vhosts exposed the dashboard to public clients, including direct WAN
IP access. Investigator added live `angor-relay-npm.conf` forwarding TLS cert11
to loopback8091 with WebSocket upgrade; added `00-dashboard-source-guard.conf`
private-source geo/map guard to both management default vhosts, preserving public
ACME challenge paths. Backup: `/etc/nginx/sites-available/archipelago.before-public-guard-1790879144`.
Nginx syntax validation/reload passed. External tests: public IP root and RPC
404 over HTTP and HTTPS (IP HTTPS certificate validation bypassed only for this
negative routing probe); spoofed private Host, X-Forwarded-For and X-Real-IP and
unknown Host POST RPC all404. Tailnet dashboard200; indexer health200 height969475;
relay trusted TLS NIP11 metadata200, WebSocket101, read-only Nostr REQ returned EOSE.
These are live containment results, not fleet/IPv6/reboot/security-audit completion.
Release owner acknowledged security scope in `/tmp/npm-release-handoff-ack.txt`.
User then reported no Angor projects after changing BOTH indexer and relays.
Read-only kind3030 subscription limit5: new relay returned zero events + EOSE;
`wss://relay.angor.io/` returned five events + EOSE. Advised retaining original
Angor relays alongside own relay; a newly hosted relay does not automatically
contain global project metadata. Full app project discovery acceptance remains
required. Also observed own `/api/v1/query/Angor/projects?limit=10` returns404;
reference MempoolIndexerAngorApi.GetProjectsAsync uses this older specialized
route, whereas current deployment docs recommend stock Mempool. Verify actual
client version/discovery path rather than claiming fees/health prove compatibility.
## Shorty live qualification: cached-runtime guard regression — 2026-10-05
The operator signed the final NPM candidate. Release-root verification and exact
reviewed payload comparison pass; signed SHA256
`479f6193835a16dd4ab167e5c22306a878ac39b77c2e2793971e807874fbc0cb`.
This signature authorizes private qualification; it is not release publication.
Shorty baseline public shop/www/indexer/relay trusted HTTPS passes. Its prior
NPM image bytes match the pinned2.14.0 image. Consistent stopped-NPM state,
backend, unit, nginx, helpers and app metadata were backed up under
`/var/lib/archipelago/support/190-npm-20261005`. Migration reached the new private
network/listeners but failed the guard acceptance check and was rolled back.
The test initially expected the emergency guard's legacy variable; further
inspection found a real source defect, not merely that assertion mismatch.
Confirmed cause: `ensure_runtime_assets_ready` applies the management guard,
then `run_runtime_assets` installs the cached OTA's nginx template verbatim.
Shorty's cached template predates the guard. It overwrote protection before a
subsequent nginx reload; restoring the older backend repeated that path. A live
public IPv4 root probe returned200. Immediate containment applied the tested
source guard; HTTP/HTTPS root and HTTP RPC again return404. The cached legacy
runtime template is now also guarded, with its original saved privately, so
that old startup installer cannot remove protection on restart. Both emergency
and current guards are present in the active configuration. Do not claim this
attempt passed or that the broader migration is complete.
Source fix: runtime installation now renders/validates the guarded candidate
before atomic replacement under the nginx transaction lock; syntax/reload
failure restores the previous protected bytes. Rollback protects the restored
runtime template before permitting an older binary to start.11 focused tests
pass; real isolated nginx verifies the actual legacy install, old-binary copy,
invalid-template rollback, public IPv4/IPv6 denial, ACME/private access and the
existing120-case Host/SNI/forwarded-header/UI/assets/RPC/WS matrix.
The first backend suite passed1,681/0failed/4ignored before the final rollback
addition; final rerun and optimized build are required. Logs:
`/tmp/archy-190-guard-runtime-final-unit.log`,
`/tmp/archy-190-guard-runtime-final-network.log`,
`/tmp/archy-190-shorty-activation.log` (failed attempt),
`/tmp/archy-190-shorty-prepare.log`.
Only NPM's container restarted; Bitcoin, LND, ElectrumX, Angor indexer and relay
IDs/start times are unchanged. Restored shop/www/indexer/relay HTTPS returns200.
Shorty's old backend remains active under containment. Rebuild and requalify the
migration, external security and restart persistence before closing this gate.
The signed catalog contents are unchanged and need no further operator signature.
### NPM multi-domain TLS regression found by public acceptance
After the private-listener migration, local first requests passed, but public
Angor health intermittently returned502. Host nginx recorded upstream certificate
hostname mismatches when different public domains used the same NPM TLS listener.
The generated bridge inherited upstream TLS session reuse. This matches nginx's
[documented cross-SNI session-cache behaviour](https://trac.nginx.org/nginx/ticket/1340).
The bridge now explicitly sets `proxy_ssl_session_reuse off` while retaining
SNI, hostname/chain verification and the existing trusted certificates. A real
NPM fixture with two distinct certificates reproduces failure with the old
configuration on the second hostname; the fixed fixture passes40 alternating
trusted TLS requests plus ACLs, WSS, certificate replacement, restart, failed-bind
rollback and disable/delete propagation.24 Python NPM regressions pass.
`/tmp/archy-190-npm-multicert-before.log` is the expected failing reproduction;
`/tmp/archy-190-npm-multicert-integration.log` is the fixed flat-layout pass.
Nested-layout issuance/renewal qualification is running separately.
The exact helper correction is temporarily installed on Shorty and transactional
sync succeeds.40 mixed local TLS requests across four hostnames pass. Public
read-only Angor browser acceptance now passes TLS, WSS, funding/event commitment,
Explore discovery of the known fixture and full project details/statistics:
`/tmp/archy-190-shorty-migrated-angor-browser-2.log`. This remains one known fixture,
not all35-project recovery.32 external IPv4 management-denial checks and tailnet
access pass;10 HTTP/HTTPS ACME routes return the exact probe written in NPM's data
mount. Six NPM database tables and42 certificate/renewal files exactly match the
pre-migration backup (`/tmp/archy-190-shorty-state-preservation.log`).
The previously running optimized build was stopped because it predates this
embedded-helper correction. A complete new build/deployment remains mandatory.
Shorty currently has the prior A5 candidate backend plus protected runtime nginx
and this qualified helper; an A5 restart can reinstall its older helper. Do not
claim final persistence until the new binary is deployed and restart is retested.
No certificate verification was disabled for a public application or upstream.
Raw-IP/unknown-SNI negative routing probes alone bypass hostname matching.
File diff suppressed because it is too large Load Diff
+304
View File
@@ -0,0 +1,304 @@
# Work requested after 1.9.0-alpha
Status: queued by the operator on 2026-10-05. Complete the current release first;
these requests do not silently expand its artifact scope. No implementation or
acceptance is claimed by this backlog.
## 1. Distributed IndeeHub publishing and paid viewing
- Recover and reconcile the earlier design in
[the streaming plan](phase4-streaming-ecash-plan.md) and
[the distribution design](dht-distribution-design.md) against current source.
Their implementation/status statements are historical, not fresh evidence.
- Review current primary Nostr, Cashu and Bitcoin/Lightning specifications before
selecting the protocol. Distinguish interoperable standards from custom events.
- Publishing through one instance's Backstage must make the content discoverable
through other instances' **Archipelago** content source. This source is intended
to become the default eventually; do not change the default without qualification.
- Reuse supported node-to-node discovery/transports and the file-payment method
negotiation: show methods the recipient actually accepts. Include Cashu and
Lightning to the automatically provisioned ecash Lightning address from first
use; do not require a producer to operate LND to receive initial payments.
- Pay the producer's wallet, verify settlement before granting access, and make
payment retries/delivery recovery idempotent. Keep producer revenue separate
from any optional hosting/relay bandwidth charges in the older plan.
- Define timed viewing entitlements, start/expiry semantics, reconnect, resume,
seek, device/session scope and clock/error handling. Evaluate encrypted media
and authorized key delivery without claiming that delivered video or keys can
be made impossible to copy or revoked retroactively.
- Use the operator's video uploaded to **Yaya Cloud**, publishing through Backstage
for the demo. Identify the exact file and preserve the source; do not select an
unrelated personal video or publish other Cloud contents. Payment-test amounts
need explicit bounded authorization before real funds are spent.
- Deliver a coherent demo: publish on one node, discover on another, select a
supported payment method, pay producer, watch, resume without repayment, and
enforce expiry. Cover publisher outage, duplicate events, wrong mint, failed/
delayed payment, restart and lost responses. Preserve privacy and access rules.
- Treat this as the foundation for future fully featured node-sharing apps,
introduced and qualified individually.
## 2. IndeeHub native Nostr signer and companion reliability
- Reproduce intermittent native-signer login failure and companion grey screen
requiring refresh/re-login. Inspect both browser and actual Android WebView.
- Cover iframe origins, signing permissions, redirects, callback/session state,
background/resume, expired sessions, refresh and cancellation without weakening
authentication or exporting signing keys. Preserve useful errors and recovery.
- Consult existing signer bridge documentation and prior origin/reload fixes;
do not assume those earlier fixes address this fresh report.
## 3. Node peering and discovery
- Diagnose Yaya ↔ Archy dev: requests appear approved/pending but neither node
appears in the other's peers; the flow is also slow.
- Trace request, delivery, approval, identity, persistence and both-node peer-list
reconciliation. Test restart, retry/duplicates, offline recovery and reciprocal
visibility; distinguish requested, approved, connecting and connected states.
- Show Nostr requests in the appropriate discovery/request UI.
- Rename **Find Nodes** to **Connect with Nodes**, consistently with accessibility,
navigation and translation conventions.
## 4. Framework Monitoring
- Investigate the reported non-working Monitoring screen on Framework with
read-only diagnostics first. Verify actual metrics, loading/error states,
permissions and refresh/reconnect on that node. Preserve wallet/radio state.
## 5. Immich / Nextcloud files in Cloud
- Assess integration so installed Immich/Nextcloud files appear under the correct
Cloud Files categories, without duplicating storage or exposing another user's
private data. Determine supported APIs, user identity/permissions, thumbnails,
originals, virtual paths and large-library pagination/indexing.
- Define view/download/edit/delete semantics per source. Preserve application
ownership, databases, metadata and trash/versioning; do not directly mutate
application-managed storage to bypass its API.
- Test installation/removal, permissions, unavailable apps, overlapping filenames,
duplicate detection and category accuracy before enabling an integration.
## 6. Web5 header and node connection flow planning
- Inspect the top-bar **Wallet** label in Web5. The operator requests removing
this cosmetic label; preserve any actual wallet navigation or accessibility
function until its role is established, and report if it is more than a label.
- Plan a less hidden, clearer discovery and peering journey on mobile and desktop
using the existing design system, visual styles and components. This is a flow
and information-placement change, not a visual redesign.
- Include visible entry to **Connect with Nodes**, incoming/outgoing Nostr
requests, approval, pending/connecting/connected status, reciprocal peer
confirmation, offline/retry recovery, and clear return paths. Show how this
relates to existing peers and node details rather than adding duplicate flows.
- Produce reviewable mobile/desktop flow plans before broad navigation changes;
cover first connection and repeat use, touch/keyboard access, empty/error
states, and the current Yaya/dev approval bug. Keep diagnostic implementation
details out of normal user-facing steps.
## 7. Companion app launch latency
- Reproduce intermittent long app-opening delays on the actual companion and
compare desktop/mobile browser timing for the same node/app. Measure discovery,
readiness polling, authentication/signing, route/proxy connection, WebView
creation and first useful rendered content separately.
- Remove avoidable waits, duplicated checks and retry loops without launching
before an app can accept connections. Cover cold/warm launches, switching apps,
background/resume, flaky connectivity, expired authentication and app restarts.
- Keep feedback clear and immediate, with bounded cancellation/retry and preserved
navigation. Record before/after measurements and test actual Android devices.
## 8. Fleet comprehensive acceptance
- Inventory every current Fleet capability and turn it into a test matrix rather
than assuming a working overview proves all functions work.
- Cover discovery, identity/deduplication, authorization, adding/removing nodes,
status/metrics freshness, selections/filters, remote actions, update discovery
and progress/results, reconnect/offline/restart recovery and error handling.
- Include mixed software versions, slow/unreachable nodes, partial success,
duplicate/late replies, permission rejection and desktop/mobile behavior.
- Use disposable fixtures for destructive/restart/update scenarios; preserve real
node wallets, application data and user choices. No new spending is authorized.
## 9. AIUI first-use/provider/funding experience
- When AIUI opens without a usable AI connection, guide the user to setup rather
than presenting only a failed model response. Distinguish missing configuration,
invalid credentials, insufficient Routstr funds and temporary provider outage.
- Offer concise choices to add a Claude credential, configure the supported
OpenAI/Codex connection, or fund Routstr. Verify the actual provider/runtime
authentication methods before labeling a credential field or promising support.
- Chat may present action buttons; use a private credential form/modal for keys,
not an ordinary chat message. Keep credentials out of model prompts, history,
logs and screenshots; use existing secure storage and permission boundaries.
- Open Routstr funding as a coherent modal where supported, show balance/payment
state and update availability after funding. Preserve any unsent prompt and
let the user continue when setup succeeds. No surprise automatic charges.
- Match the existing design system and support small screens, keyboard access,
cancellation, invalid/expired keys, funding delay, reload/background/resume and
successful first response. Test actual companion and desktop flows.
## 10. Node availability, offline duration and network map
- Make availability easy to scan in Fleet and Connected Nodes using explicit
status text as well as existing visual indicators. Show last successful contact
and an elapsed duration, such as "Last seen 2 hours ago"; only say "Offline for"
when an observed offline transition supports that claim. Never equate an old
relay advertisement, failed monitoring query or unknown status with proof that
a node has been continuously offline.
- Place confirmed offline nodes after online nodes by default, with stable order
within each group. Specify how connecting and unknown/stale nodes are ordered;
preserve user-selected sorting, filters, selection and scroll position.
- Show offline and unknown/stale nodes distinctly on the network map, with text
or accessible detail including last contact. Do not present retained historical
links as live connections or silently drop long-offline nodes from the map.
- Retain useful last-contact history across refresh/restart; reconcile timestamps
across discovery, peering and monitoring. Handle clock skew, never-seen nodes,
stale cached data, long outages and reconnects without false precision.
- Test short/long outages, stale relay events, monitor-only failures, network
partitions, app background/resume and status recovery on desktop and companion.
Use the existing design system and avoid color-only status communication.
## 11. Web5 connection/sync speed and responsive navigation
- Profile and improve connection establishment, discovery, peer synchronization
and Web5 data loading. Measure each stage and reduce avoidable serial waits,
duplicate requests, excessive timeouts and retry storms. Preserve identity,
trust verification and accurate readiness/status reporting.
- Reproduce delayed taps/navigation for Web5 and Cloud, especially in the Android
companion, and the Find Nodes / nodes entry into **Federation & Peers**. Record
tap-to-feedback, route-render and useful-data timings separately on cold/warm
loads and with realistic slow/offline nodes and larger peer/file lists.
- Every navigation tap must give immediate feedback using existing components;
render the destination shell promptly and load independent data incrementally.
One slow node/request must not block the whole screen or tab navigation.
- Keep cached content visibly identified when stale; use bounded loading states
and useful retry/error feedback. Preserve back navigation, scroll, selection
and in-progress work; cancel or ignore superseded requests safely.
- Cover all tabs, not just these reported routes. Test rapid tab switching,
repeated taps, background/resume, expired sessions, unavailable peers, partial
responses and recovery on actual companion hardware and desktop/mobile browsers.
- Record before/after latency distributions and agree measurable budgets after
establishing a baseline. Faster feedback alone is not evidence that the
underlying connection/synchronization delay has been fixed.
## 12. Missing Fleet card metrics and secure FIPS transport
- Reproduce missing metrics on multiple Fleet **Nodes** cards; trace collection,
authorization, transport delivery, identity matching, subscriptions/cache and
rendering separately. Verify each metric is real, current and associated with
the correct node; show unavailable/stale explicitly rather than invented zeros.
- Evaluate and use existing FIPS transport wherever supported and measurably
beneficial across Fleet, discovery, peering and synchronization. Preserve peer
authentication, access controls, confidentiality and existing trust boundaries;
transport reachability must never grant permission to read metrics or act.
- Prefer fresh authenticated updates without duplicate polling or excessive
subscriptions. Measure propagation latency and resource use, including large
fleets, while retaining safe fallback for unavailable/incompatible FIPS peers.
- Test spoofed/unauthorized peers, expired trust, disconnect/reconnect, partial
metrics, stale/out-of-order/duplicate messages, mixed transport capability and
fallback recovery. Do not claim FIPS is faster until timings demonstrate it.
## Required qualification before user acceptance testing
Operator explicitly requires extensive headless testing using the authorized
nodes before deploying the next features for UAT. For each task, qualify source
regressions, actual browser behavior, integration between appropriate nodes,
permissions and failure/recovery paths first. Use existing access and useful
read-only checks on real nodes; use isolated fixtures for destructive scenarios.
Headless browser success does not substitute for actual companion/WebView checks
where lifecycle, native signing, media or navigation behavior is involved.
Record exact revisions/artifacts, nodes, environments, before/after measurements,
executed scenarios, failures and untested boundaries. Retest fixes and relevant
regressions; do not mark gaps passed or claim perfection. UAT handoff must contain
short node-specific steps with expected outcomes, deployment scope and rollback.
Preserve wallets, files, identities and operator app choices throughout.
If IndeeHub changes are needed, include its app update explicitly: build and test
the versioned image, preserve app data and configuration, qualify fresh install,
upgrade/restart and rollback, publish through the signed app catalog, and verify
existing nodes discover and apply the intended version. Distinguish an app-only
release from any backend/OTA dependency; use the documented decoupled app-update
path where supported. Do not silently require a full OTA for an app-only change.
Sequencing clarification: finish1.9.0-alpha first. Publish any required IndeeHub
app update at the end of the follow-up implementation and qualification, not
ahead of that work or as an untested addition to the current release.
## 13. V4V Portainer demo as a Yaya node app
- After the current release, deploy/package the existing V4V Portainer deployment
as a demo app on Yaya, showcasing how an ordinary third-party app works on a
node **without native Nostr signer integration**, as explicitly requested.
- Inspect the actual existing Portainer source, branch/image revision, Compose
stack, access/authentication and data before changes; retain the working V4V
site, stack and persistent state. Do not confuse this with the public Archipelago
software demo at demo.archipelago-foundation.org.
- Follow the current app-development guide and supported app packaging/gate/
lifecycle conventions. Define the app card/icon/category, launch URL/readiness,
network/auth boundaries, health, configuration and persistent mounts properly.
Do not expose the native signer or implicitly grant signing permissions.
- Qualify fresh installation in isolation, then Yaya deployment, desktop/mobile/
companion launch, normal app functionality, restart, update/rollback and safe
removal behavior. Verify the deployed app actually runs the intended V4V source
revision, not a stale build or unrelated image.
- Record a short reproducible demo flow and operator UAT checklist. Preserve
existing Gitea/Portainer connectivity and unrelated apps; no new payment or
public sharing of private test content is implied by the demo packaging.
### V4V node-only catalog and Sovereign Music promotion
- Source is on the existing Gitea on the146 server; locate the actual V4V repo,
branch and deployment revision there rather than guessing a replacement source.
- Add a **Sovereign Music** banner for V4V on Yaya, following the existing
Sovereign Streaming banner treatment and using the app's own login background.
Retrieve and inspect the real asset; do not invent a replacement illustration.
- Both demo app availability and its promotion must be confined to Yaya. Evaluate
a signed per-node/DID-scoped demo catalog or existing supported node-specific
candidate mechanism. Do not publish the demo to the global catalog or let
unrelated nodes install it implicitly through a shared catalog cache.
- Test matching/nonmatching identities, copying URLs/catalogs between nodes,
missing identity, refresh/restart/update and promotion visibility. Catalog
selection or visibility must not bypass normal artifact-signature enforcement.
- This may become a real app later; preserve an explicit tested promotion path
from node-only demo to proper public app release without duplicate app IDs,
conflicting state, lost configuration or automatic exposure before approval.
### V4V persistent playback and existing demo music catalog
- Use the demo song catalog from the existing V4V Portainer demo in the Gitea
repository identified above. Inspect its actual catalog configuration and media
sources; preserve artist attribution and payment destinations. Do not substitute
an invented catalog or copy credentials into public app configuration.
- Integrate V4V with the local player bar: leaving or closing the app view must
retain playback, expose track information and playback controls, and offer a
button to reopen V4V at the same track and position without restarting playback.
Explicit stop remains available. This concerns closing the app view; operating
system termination/background restrictions require separate qualification.
- Inspect supported app/player integration before choosing an implementation.
Maintain one playback session, prevent duplicate audio on reopen, authenticate
app-to-shell messages, and preserve the requirement of no native Nostr signer.
- Test navigation, repeated close/reopen, pause/resume/seek, track changes,
disconnect/recovery and unavailable media on desktop and the actual companion.
Check track, queue and position continuity, accessible compact controls and
mobile background behavior. Record platform limitations rather than promising
playback after an operating system terminates the app.
- Keep catalog and player integration within the Yaya-only demo scope until
separately approved for general release.
## 14. FIPS media transport requirement
- Operator explicitly requires FIPS for streaming peer files and distributed
IndeeHub media. Verify the actual node-to-node media-byte path uses FIPS, not
merely discovery, metrics, payment negotiation or a connection-status label.
- Reconcile the earlier multi-transport/swarm design with this requirement.
Preserve normal authenticated browser/companion playback interfaces while
carrying the inter-node stream over authenticated, authorized FIPS connections.
- Cover progressive/range requests, HLS segments as applicable, seek, pause/resume,
reconnect, producer/cache-peer outage, backpressure, bandwidth and resource use.
Preserve encrypted-content and timed-entitlement/payment enforcement end-to-end.
- Define and expose behavior when no usable FIPS route exists. Do not silently
call another media transport FIPS or mark this requirement complete while the
actual stream continues over an unrelated fallback. Any fallback policy must
be explicit and reconciled with the operator's all-streaming requirement.
- Instrument transport selection and verify it in headless multi-node integration
tests and actual mobile/desktop playback. Keep technical diagnostics available
without burdening normal playback with implementation details.
+19 -6
View File
@@ -6,8 +6,8 @@ Reviewed both open PRs from the repository pull-request list: [#161](https://sou
and [#162](https://source.archipelago-foundation.org/lfg2025/archy/pulls/162).
Both branches were updated from main, repaired and tested independently and
together. Their existing remote branches were advanced without rewriting the
contributors' history. They remain open for integration into the release after
1.8.21; no reviewed code was merged into main or deployed to a live wallet.
contributors' history. Both were subsequently merged and closed and are now
integrated on main. Candidate live-wallet acceptance remains pending.
The signed 1.8.21 artifacts are unchanged.
| Candidate | Tested commit | Isolated backend result |
@@ -95,8 +95,8 @@ Logs on the development box:
## Next-release acceptance and limits
- Integrate the reviewed branches and repeat the release gates against the
final release commit if additional code changes land.
- Both reviewed branches are integrated on main alongside the lifecycle fixes.
Repeat release gates against the final release commit after remaining changes.
- Perform funded peer-to-peer acceptance on the candidate build, including a
Tor-only purchase and a purchase requiring change, before the next release.
The new review branches were not deployed to funded live wallets here.
@@ -109,5 +109,18 @@ Logs on the development box:
ordinary write failures and truncated input are tested to clean up. The final
filename is published only after complete input, and existing files remain
protected.
- The separately reported X250 kiosk version-selector rendering issue remains
open in `TODO.md` and requires validation on the actual kiosk.
- The separately reported X250 kiosk selector is fixed and verified on the
actual kiosk; see the lifecycle evidence and consolidated release checklist.
## Authorized merge — 2026-09-30
The operator explicitly requested normal merged/closed PR status after review.
Re-read both PRs and verified their heads still exactly matched the reviewed
commits. Changes from the integration-test base to main were documentation only.
Gitea normal merges completed and read-back confirmed `merged=true`, `state=closed`:
- #161: `3daea6623be3e2c7222101b8e6ac411423c7e16c`.
- #162: `b02ba4100d922dd1b75c6a78121ef446c2159a54`.
Local next-release lifecycle work was integrated with main at `d69e8452`. Funded release acceptance and the documented delivery-receipt
limitation remain as recorded above; merging does not claim a new release.
+101
View File
@@ -0,0 +1,101 @@
# Archipelago 1.8.22-alpha acceptance
Source: `6d5f3ffb850bfd3dcd396bac986ba770935d1daa`.
## Verified application and runtime changes
- Full isolated backend suite: 1,617 passed, zero failed, four explicit opt-in exclusions.
- Frontend suite: 1,133 passed. Final frontend and AIUI production builds succeeded.
- Container suite: 79 passed. Catalog compatibility/trust, release manifest, build contexts, pruning, Lightning readiness, NPM migration, safe doctor, companion recovery and ISO doctor-overlay regressions passed.
- Six companion dashboard images built using the current registry.
- Final OTA backend SHA-256: `e108b78bbbd21cb7d5d47c8d0b7b9b19b63fb0c44678773603202440ec7d6f5b`.
- Final OTA frontend SHA-256: `2da485a2da75ff2fbe4aba52d6f217150e303be43a031723480c9c4ff9d43f41`.
## Live acceptance
The exact OTA bytes were deployed to the development box and ThinkPad X250.
Native Bitcoin/LND and the X250 production site retained their container identity
and start time during these final management deployments. Both boxes completed
12-minute observations including scheduled diagnostics with running containers
and persistent mounts unchanged. The orphaned Cuprate dashboard stayed absent.
Actual X250 Chromium kiosk: hard refresh, NPM Launch to the correct admin URL,
visible login/admin page, readable inline Bitcoin version choices and pruning
checkbox passed. No Bitcoin installation was triggered by this test.
Final desktop/mobile checks passed for Bitcoin's IBD dashboard, one Mempool card,
headless Phoenixd, LND waiting/unknown-balance behavior and all five transparent
AIUI embedding layers. Standalone AIUI retains its wallpaper.
Portainer's actual production network namespace fetched Git refs and the Compose
file after final deployment. Original mounts were preserved. Earlier disposable
fresh/reverse-install and migration/rollback tests, and the production host's
operator-initiated reboot check, passed.
Live Tor-only Cashu paid-file acceptance verified a one-satoshi net purchase,
change, rejected-payment refund, exact file bytes, Files access and free repeat
delivery. No native Bitcoin/LND funds were moved. PRs 161/162 are merged and
closed; the open pull-request list is empty.
## Boundaries
- Angor's real dev API, fees, block tip, CORS and rootless/headless configuration
passed. Full-chain indexing remains dependent on initial Bitcoin sync finishing.
- Optional live AI providers, physical RNode hardware, the opt-in Reticulum TCP
subprocess test and creation of a production Minibits profile were not run.
- The previously recorded unsafe-doctor incident changed X250 container start
times before the final fix. Persistent databases were present after recovery,
but no pre-incident cryptographic wallet-identity baseline was available.
Do not describe recovery evidence as an exact pre-incident balance comparison.
- No claim of perfect behavior on every device, network or future failure is made.
## Raw ISO acceptance
The raw ISO is 2,755,072,000 bytes. SHA-256:
`cf7be6378dcd52f6f62774523341fa75dd453fa73a9cadff5390846f483e0140`.
Mounted-artifact smoke checks passed, including BIOS/UEFI boot files, live-boot
hooks, build contexts, current doctor overlay, crash-capture configuration,
version and frontend payload. The ISO backend and all 653 OTA frontend/runtime
files match exactly. AIUI metadata names the tested source commit.
A disposable QEMU/KVM x86_64 VM with UEFI firmware, 3 GiB RAM, two vCPUs, a fresh
64 GiB NVMe virtual disk and no network completed the full installation. This
covered partitioning, LUKS2 data encryption, swap, system configuration, UEFI
bootloader and initramfs generation. Cold boot with the ISO detached reached the
visible Welcome to Archipelago setup screen. The installed backend and both
historical/current doctor paths matched source hashes. Backend/nginx were active;
health reported RPC/sessions ready, crash recovery complete and version 1.8.22.
No wallet was initialized in this disposable VM.
The first automatic VM reboot selected the still-attached installer ISO. That
was corrected in the test configuration by detaching the ISO and explicitly
booting NVMe. It was not accepted as an installed-system boot. The subsequent
cold boot above is the successful acceptance run.
The dev native Bitcoin/LND identity/start-time baseline also remained unchanged
after the ISO build and VM acceptance.
## Publication verification
All three operator signatures verify against the pinned release root. The five
Gitea assets match independent server-side SHA-256 checks. Both OTA components
also passed complete public HTTPS downloads with exact hashes and sizes; the
raw ISO passed public size/range checks and both checksum sidecars read back
exactly. Only after these checks were the signed OTA manifest and compatible
app catalog promoted. The release tag identifies the tested source above.
Git and ngit publication completed on 2026-10-01. Both repository relays
acknowledged the ngit release; independent `release view` resolved all five
assets with exact hashes and sizes and no unresolved asset IDs. Main and the
release tag were pushed to both remotes.
Public main-branch OTA manifests and the app catalog read back byte-for-byte
and verified cryptographically. Live `update.check` on the accepted dev node
reported 1.8.22-alpha with no further update, as expected for the installed
release. Discovery on an older production node was not repeated during this
publication step.
- [Release and verification files](https://source.archipelago-foundation.org/lfg2025/archy/releases/tag/v1.8.22-alpha)
- [Raw ISO](https://source.archipelago-foundation.org/lfg2025/archy/releases/download/v1.8.22-alpha/archipelago-installer-1.8.22-alpha-unbundled-x86_64_RC1.iso)
+588
View File
@@ -0,0 +1,588 @@
# Archipelago 1.9.0 acceptance
The operator changed the release target from 1.8.23-alpha to **1.9.0**. This file retains its historical path so handoff links remain valid.
Status: PREPARING. Do not publish until artifact checks and offline signatures pass.
## Scope
Durable Lightning paid-file entitlements and delivery retries, atomic buyer
ownership, compact persistent upload progress/cancellation, mobile transaction
filters, Immich inventory and retired-app uninstall, Portainer duplicate-network
migration, channel-close fee selection, and shared app-search clearing.
Angor Indexer and the optional Angor Relay remain in the signed catalog. They
were already included in 1.8.22; full-chain acceptance still awaits dev Bitcoin
initial sync. Do not advertise full-chain verification as complete.
## Validation carried into preparation
- Candidate deployed on dev and Framework with matching backend/dashboard bytes.
- Native Bitcoin/LND and Framework Immich container IDs/start times preserved.
- Six live desktop/mobile app-search cases passed.
- Actual uploads verified exact bytes, original destination after navigation,
compact progress geometry and cancellation. These are browser checks, not
physical companion acceptance.
- Framework seller's settled invoice recovered to a mode-0600 entitlement and
remained paid across another manager restart; mismatched item rejected.
- Framework CryptPad stale inventory removed with data preserved; removal
persisted after management restart. Dev normal package.uninstall RPC also
completed with preserve_data=true and zero cleanup errors for the retired ID.
- Cooperative-close fee selector tested with intercepted requests; no real
channel was closed. Real payment recovery never sent another payment.
- Shorty's Mempool inspected read-only; frontend/API healthy for over two weeks,
systemd zero restarts. Operator reported normal status after their update.
## Follow-ups accepted for release preparation
The operator authorized release preparation after available tests pass.
- Actual failed-purchase delivery still requires buyer/file confirmation. The
located invoice's source file is missing from Framework's recorded paths.
Seller settlement recovery does not prove buyer delivery.
- Physical companion upload diagnosis needs the affected device/route.
- Framework rendered inventory/uninstall checks need dashboard second-factor
authentication; SSH/runtime and dev API acceptance are recorded separately.
- Angor full-chain indexing awaits Bitcoin IBD.
- Prior Primal automatic-comment and lost-response ecash receipt follow-ups
retain their documented boundaries; this release does not claim to fix them.
## Final release checks
Pending full release gates, versioned builds, exact artifact deployment, ISO
payload/boot acceptance, pinned-root signatures and publication verification.
No universal or future-failure guarantee is implied by these tests.
## Required NPM certificate gate
The release owner acknowledged `docs/npm-certificate-handoff-20261001.md` in
`/tmp/npm-release-handoff-ack.txt`. Shorty's npm-8 issuance and HTTPS health 200
were reported by the operator; durable data-path resolution, safe host routing,
automatic certificate renewal/reload, and the handoff acceptance matrix remain
required before publication. Earlier authorization does not waive this new gate.
## Urgent public dashboard exposure gate — 2026-10-01
Investigator reports the Angor relay hostname reached the default Archipelago
login because its certificate existed without a corresponding host-nginx route.
The investigator owns the immediate Shorty nginx repair; the release session
will not modify that configuration concurrently. Exact final evidence is pending.
- [ ] Unknown public HTTP Host / TLS SNI and direct public-IP requests cannot
expose the dashboard, login assets or RPC, including IPv6 and any trusted
reverse-proxy/tunnel path. Test spoofed forwarding headers explicitly.
- [ ] LAN/private/tailnet dashboard access remains available as intended.
- [ ] Public HTTP ACME challenge access survives those restrictions.
- [ ] NPM host creation/edits automatically propagate HTTP/TLS routing.
- [ ] Relay hostname serves the intended relay and WebSocket upgrade using its
correct certificate; certificate existence is not route acceptance.
- [ ] These protections survive manager/nginx restart, renewal and OTA/ISO.
## Additional isolated security checks — not deployed
The management source-guard prototype passed five unit checks including legacy
address-specific HTTPS, idempotence, backup permissions and syntax/reload rollback.
An actual nginx instance in a private network namespace passed 120 negative
HTTP/TLS cases across IPv4/IPv6, raw/unknown/spoofed Host/SNI and forwarded headers,
including POST RPC and WebSocket upgrade requests. Exact ACME token reads,
private LAN/tailnet/ULA access, named public HTTP app routing and reload passed.
These are scoped checks, not complete fleet, trusted-tunnel, reboot or artifact
acceptance. Shorty's containment was not modified.
The NPM storage/routing prototype passed eight focused tests: fresh/flat/nested/
custom mount selection without mutation, ambiguity/wrong-mount refusal, corrupt
or uninitialized database preservation, duplicate/missing mounts, deleted/disabled
host exclusion, domain injection rejection, and rejection of mixed public and
loopback listener bindings. Automatic application/migration and end-to-end NPM
security/routing remain unfinished and block release.
Final Angor handoff was read and acknowledged in
`/tmp/angor-final-handoff-ack.txt`; see `angor-client-acceptance-20261001.md`.
One complete project flow is investigator-verified; 34 original announcements
remain unrecovered from queried sources. Full recovery acceptance remains open.
## Additional Angor public explorer gate
- [ ] Public indexer hostname serves Mempool UI and its assets/deep links/live
WebSocket updates while preserving Angor API/CORS/broadcast/readiness.
- [ ] Existing stack reused; no duplicate Mempool app/database and no management
or Bitcoin RPC exposure.
- [ ] Documentation/catalog/runtime metadata and exact OTA/ISO reflect the tested
implementation; repeat official-client browser acceptance afterward.
Confirmed official deployment guide describes a shared frontend/API origin.
Current adapter 1.0.1 is API-only; this requirement is not yet implemented.
## NPM real-image integration progress
Disposable real NPM API tests passed for both flat and legacy nested `/data`
layouts: initial account/host creation, multiple domains, custom location,
forwarded-client spoof rejection, exact challenge file access under forced HTTPS,
trusted TLS and WSS upgrade/frame, certificate replacement/reload, password and
network access lists, disable/enable/delete propagation, and restart with the
original database and account authentication preserved. Local fixture certificates
are not public Let's Encrypt staging issuance/renewal evidence; that gate is open.
Certificate replacement initially failed because the updated bridge could not
complete the upstream TLS request after replacing the fixture certificate.
Reloading and validating NPM's own TLS listener on certificate fingerprint changes,
as well as host nginx, resolved the test. Rollback/retry unit coverage was added.
The initial dev guard deployment changed only the inactive sites-available copy;
private HTTP 200 and unchanged entry bytes were insufficient acceptance evidence.
A later public-ingress-marker probe caught this: it incorrectly returned 200.
The resolver now chooses the active sites-enabled copy or resolves its symlink
without replacing the link. Guard backups live outside nginx include directories.
After the correction, live private requests return200 and marked requests 404.
No app was restarted. Direct-backend protection still awaits its candidate build.
Angor candidate UI was exercised against the actual dev Mempool stack in a
throwaway gateway: desktop/mobile rendered, zero failed JS/CSS assets, one
WebSocket connection each. The gateway was removed afterward; this is candidate
integration evidence, not a published or permanently installed app update.
### Same-node NPM networking correction
Read-only inspection of the production NPM namespace reproduced HTTP 502 for its
own configured indexer route. Host requests to the LAN upstream returned 200;
requests from the existing pasta namespace to that same LAN address were refused.
A disposable container on the proposed `slirp4netns:allow_host_loopback=true`
network returned 200 for both the LAN upstream and `host.containers.internal`.
No production NPM/nginx configuration or container was changed in this check.
The candidate now declares this network in the manifest and first-boot path,
with explicit Quadlet/API support and legacy drift detection. The Podman API
`network_options` shape was checked against `podman generate spec` locally.
The real NPM integration fixture now uses the proposed network and a LAN-bound
same-node upstream, rather than placing both fixtures on one custom bridge.
Flat-layout integration passed namespace reachability, host routing, verified
TLS/WSS, ACME file access, certificate replacement/reload, custom routes, password
and IP ACLs, spoof rejection, host lifecycle and NPM restart with preserved DB.
The earlier loopback-only fixture failed because this machine resolves the host
alias to its LAN address; its bind was corrected before repeating the test.
The latest isolated nginx guard test passed 120 public IPv4/IPv6 negative cases
plus private access, ACME, proxy-marker rejection and reload. Python guard/bridge
regressions passed 23 tests, including exact emergency-route retirement, failed
migration rollback and preserving operator-modified routes. Backend compilation
and new direct-listener tests are still pending. Required public staging renewal,
actual upgrade/reboot, yaya and exact OTA/ISO acceptance remain open.
### Active nginx site layout regression
The dev node has a regular `sites-enabled/archipelago` file, not a symlink to
`sites-available`. Both the guard and ACME resolver now select the active file.
Unit coverage verifies copied sites and symlink targets, preserving inactive
operator copies and the links themselves. Nginx configuration backups must not
be created inside `sites-enabled`, whose wildcard include would load them.
The active guard was applied on dev, with private HTTP 200 and public-ingress
marker 404 verified after reload. Shorty remains untouched. Do not count the
initial inactive-file edit as a security deployment pass.
### LoRa flasher added to release gates
Both dev and Framework lack the esptool executable and Python module. The
backend invokes a bare `esptool` and retries even process-spawn failures. The
shell updater installs it opportunistically, but the OTA runtime tool list
omits it. Candidate packaging adds a pinned self-contained `archy-esptool`
with its ESP32-S3 stub to mandatory OTA/ISO payloads. Candidate preflight runs
before stopping the radio; retries are limited to recognized serial transport
failures. Concurrent flash registration now uses one exclusive lock.
CP2102 USB identity does not uniquely identify a Heltec V3. The candidate removes
that unsafe inference from backend and UI and requires explicit board selection.
Actual board models were requested before firmware writes. Dev exposes one
CP2102 serial radio. Framework SSH works but currently exposes no mesh-radio,
ttyUSB or ttyACM port. No radio has been erased or flashed in this investigation.
Physical MeshCore UK acceptance on both nodes remains required and pending.
Dev connection diagnosis: the failed flash stopped its listener before spawn
failed and left the enabled setting true. A read-only protocol probe identifies
the existing radio as Reticulum/RNode. An explicit listener disable/enable restored
`device_connected: true` on `/dev/mesh-radio`, without flashing or native-service
restarts. Candidate configure logic now compares desired enabled state with the
actual listener task, including stopped/finished handles; same-settings reconnect
is covered by a new isolated regression. Probe/configure and flash registration
are coordinated to prevent concurrent serial owners.
The packaged `archy-esptool` build passes in a clean environment, including loading
the actual ESP32-S3 stub through esptool's own loader. It is installed and self-tested
on dev and Framework; a compatibility command supports their current backends.
This verifies tooling availability, not physical flashing. Framework's USB sysfs
inventory shows its hub/storage/network/keyboard/display devices but no serial
radio. Board confirmation and Framework radio detection remain pending.
Additional Podman API acceptance: a disposable API service created a container
with the candidate `netns`/`network_options` payload. Its effective generated
specification preserves `allow_host_loopback=true`. The normal inspect network
mode omits options for API-created containers, unlike the CLI-created case;
candidate drift detection now consults the effective specification before
recreating such a container. Added a regression against repeated recreation.
The probe container and temporary API service were removed. The real isolated
nftables tunnel regression also passed (NPM peer port and LND remain separate).
### Latest acceptance checkpoint: radio connection and release status
The complete frontend suite passed: 143 files, 1,159 tests. Type checking and
both new radio setup tests also passed. The final isolated backend build/test
run is still pending; the previous run exposed an NPM/Router port collision,
which was corrected by assigning NPM's local HTTP listener port 8088. Do not
report the previous run as fully passing or the final run as completed.
Yaya's identity has been confirmed. Its existing managed web-tunnel drop-in
clears manifest port publications and supplies private tunnel HTTP/HTTPS ports
plus the local admin port. This would suppress the candidate bridge's new
loopback HTTP/TLS listeners. Migration must preserve the working tunnel/site,
add the required local listeners, validate the managed firewall/lifecycle,
retain custom overrides, and cover repeat upgrade and rollback. The current
bridge rejects non-loopback listeners, so the supported tunnel topology also
needs explicit qualification. No tunnel or NPM runtime change was applied to
yaya during this diagnosis. Its management source guard was applied and tested:
private dashboard HTTP 200, public-ingress-marked request 404.
Dev radio connection has been restored on its existing Reticulum firmware.
Framework still does not enumerate a USB serial radio. Both nodes now have the
self-tested packaged flasher, but physical MeshCore UK flashing, post-flash
handshake and reconnect acceptance remain open pending board identification and
Framework USB detection. No device firmware has been written.
OTA, app catalog and raw ISO publication remain held. Outstanding acceptance
includes NPM migration/renewal/reboot and exact artifacts, end-to-end delivery
of the reported paid file, physical companion uploads, full Angor discovery
(the 34 missing original announcements), radio hardware tests, and the final
dev/yaya candidate deployment checks. Retained tasks above remain in scope.
### Dev Heltec V3 physical flashing result
Full 8 MiB pre-flash backup saved privately with mode 0600 and checksum. After
removing the confirmed stale radio sidecar, the exclusive read completed.
The normal mesh.flash-device RPC then flashed the official Heltec V3 Companion
USB v1.17.1-d929643 merged image successfully (100%, no error). The image's
size and SHA-256 were checked against the official GitHub release metadata.
Independent serial protocol queries confirmed model Heltec V3, firmware
v1.17.1-d929643 and actual RF readback: 869618 kHz, 62500 Hz bandwidth, SF8,
CR8 (EU/UK Narrow). This is device readback, not merely saved host settings.
The listener was then re-enabled and reported connected as meshcore. No wallet
or native Bitcoin/LND service restart was used. MeshCore remote reboot is not
supported by the current API; that attempted check returned an explicit error.
Physical unplug/replug and communication to Framework remain pending.
Live qualification additionally found incorrect binary DEVICE_INFO/SELF_INFO
parsing and a stale host-side RF-applied marker after full-chip flashing.
Candidate changes decode the current official binary layout, query the actual
firmware version during initialization, and invalidate the RF marker after a
successful flash. The dev marker was backed up and cleared before provisioning;
the independent readback above confirms settings applied. New parser, startup
cancellation and marker lifecycle regressions are queued/running; the prior
1,647 passing tests do not cover these later changes.
Framework's V4 official USB image has been downloaded and verified. Despite the
operator confirming it is plugged in, repeated sysfs/device checks show no
ESP32 USB or serial port. USER/BOOT plus RST bootloader entry was requested;
Framework has NOT been flashed and the two-device acceptance remains open.
### Operator deferral and latest qualification
Operator explicitly deferred Framework radio/hardware work and instructed us to
continue all other release tasks. Framework V4 flashing, USB reconnect and
radio-to-radio acceptance remain UNVERIFIED / OPERATOR-DEFERRED; this is not a
pass. Do not request further Framework radio operations unless needed and the
operator resumes that work. Dev V3 acceptance and durable fleet fixes remain.
The final radio backend suite passed 1,650 tests, zero failed, four ignored,
including sidecar-startup cancellation, current MeshCore metadata parsing, and
post-flash RF-marker invalidation. Frontend baseline remains 1,159 passed.
Correction to the earlier yaya tunnel concern: direct inspection of the active
Quadlet and all drop-ins confirms web-tunnel.conf ADDS private tunnel ports; it
does not contain an empty PublishPort reset. The earlier statement that it
cleared manifest listeners was incorrect. The new loopback publications therefore
coexist declaratively without editing that working tunnel drop-in. The bridge
validator now permits only the known HTTP/TLS tunnel ports bound to a currently
assigned RFC1918 address on an actual WireGuard interface named wg-web; it still
requires a separate loopback upstream, and rejects wildcard/public/unassigned
bindings and any admin-port exception. Python bridge/guard tests: 27 passed.
Actual yaya candidate upgrade and public route acceptance remain pending.
### NPM public certificate and restart qualification checkpoint
- Main backend: 1,651 passed, zero failed, four explicitly ignored hardware /
external integration tests. Container runtime library: 80 passed, zero failed.
- Bridge/management guard Python suite: 27 passed. Shell syntax and actual ISO
overlay-content test passed.
- Candidate validator inspected yaya's real runtime/WireGuard interface and
accepted its existing web tunnel publications while selecting proposed
loopback HTTP/TLS upstreams. This was read-only, not a runtime upgrade.
- Existing yaya public HTTP ACME route returned the exact random token body
written inside NPM. Lets Encrypt STAGING initial issuance and renewal dry run
succeeded using separate temporary account/config/work/log storage. Current
production certificate and host records were not replaced. Public site HTTP
and trusted HTTPS retain their authentication requirement (401).
- First renewal harness timed out while Certbot used a 292.7-second randomized
delay; the remote log confirmed successful simulated renewal. A deterministic
rerun with --no-random-sleep-on-renew returned exit 0 and success confirmation.
Only the temporary staging directory was removed afterward.
- Real disposable NPM nested-layout test completed: namespace LAN upstream,
API host creation, custom locations, client-IP spoof rejection, exact ACME
token, trusted TLS/WSS, certificate replacement, password/network ACLs,
enable/disable/delete, and restart with retained DB. Latest restart took 7.6s.
Earlier rerun exceeded the fixture's 90-second restart window; the test now
uses the manifest's 180-second budget and records both route/admin statuses
and container state on failure. Do not erase that earlier observed failure.
- Cleanup was hardened to continue cleaning other fixtures after a timeout.
Two early test runs passed functional assertions but failed cleanup; their
leftover disposable containers/networks were explicitly removed. The latest
full run exited successfully.
Legacy non-Quadlet repair now retains the old container for rollback, restores
it after replacement failure, preserves its exact image and environment values,
and waits for HTTP API readiness. Environment values use an exclusive mode0600
file cleaned on drop, not argv or the host process environment. Invalid/multiline
entries fail before stopping the original. An occupied rollback slot is preserved
for review rather than deleting an unknown container. Live interrupted-migration,
additional operator-override and rollback acceptance remain OPEN; unit success
is not proof of those deployment paths.
The deployment backend and frontend build are in progress. Full candidate dev/
yaya upgrade acceptance, reboot, exact signed OTA/catalog and booted raw ISO
remain open. Framework radio is operator-deferred, not passed. The original paid
file bytes, physical companion upload and 34 missing Angor announcements remain
separately tracked.
### Further completed acceptance
The complete disposable NPM test now includes a deliberately failed replacement
with an occupied host port. Restoring the retained container preserved its exact
container ID, database, configured hosts and authenticated API access; the test
exited successfully and cleaned its fixtures. This verifies the Podman rollback
mechanism, not yet the full installed backend's legacy-repair entry point.
Dev frontend build completed and was deployed with a separate rollback backup.
Served production Transactions layout passed at widths 390 and 1440: transparent
background, no image/blur/shadow/border, nowrap and exactly one row. Mobile rail
is 324px wide with 419px scroll content. Backend candidate compilation is still
in progress, so the latest backend changes are not yet deployed.
Dev Bitcoin remains unpruned and in IBD (observed block543676/header969495,
verification progress0.2284). This is not full-chain Angor acceptance. The operator
identified the seller of the failed Lightning purchase as Amish Paradise.
On 2026-10-02 the operator confirmed the other tester received the file and
accepted closure of this individual recovery. Seller access is no longer needed
for that recovery. This does not establish that the candidate fix delivered it;
durable settlement/delivery regression acceptance remains required before
release. No additional payment was made. Earlier references in this document
to missing original purchase bytes are superseded by this operator acceptance.
### Read-only Shorty migration preflight: remaining ownership conflict
Preflight found both flat and nested NPM databases. The live container's explicit
/data mount identifies the active one, so the candidate resolver now uses that
verified mount (or its saved validated receipt after a managed stop), preserves
both databases, and still refuses multiple databases without an authoritative
selection. Python coverage verifies both explicit choices and unchanged bytes.
This helper update occurred after the deployment binary build started; a final
release rebuild must include it. Do not claim the in-progress binary contains it.
After resolving storage, the two Angor emergency routes match the exact known
handoff templates. Another existing file, shop-btcpay.conf, conflicts with an
enabled NPM record: the manual route supplies HTTPS using certificate10, while
the NPM host currently has certificate_id0 and SSL forcing disabled. The manual
route and NPM record cover the same two public names and backend web port. Blindly
retiring the route would break its HTTPS. No database, host, certificate, route
or runtime was changed on Shorty. The bridge correctly refuses this ownership
conflict and now names its configuration file in the diagnostic. Align TLS/route
ownership and verify the public shop before retiring that manual configuration;
Shorty's full migration acceptance remains OPEN. Preserve live containment.
### Candidate deployment and additional real-upgrade ACME regression
The operator explicitly deferred Framework's physical radio investigation and
requested continuation of all other work. Framework radio remains unverified.
Dev and yaya now run the backed-up unpublished backend candidate SHA256
4c47269b3480ca0362df18dae160c073a19ea33507e04cacdad5b96837990ca6 and production
frontend index 3099c4ba44528a4a4c524f9a26159414558efa16abdd12cc7bfd64376cbb6089.
Both management health checks passed; native Bitcoin/LND container identities
and start times were unchanged. Yaya public site retains trusted TLS and its
401 authentication requirement; NPM admin API200, private dashboard200 and
public-ingress-marked dashboard404. The first yaya staging attempt stopped
before binary replacement because rsync was absent; deployment now uses Python
copying without that dependency and completed successfully.
Actual startup exposed an additional regression: the canonical nginx template
had only HTTP ACME, while the bridge demanded two locations. Startup rewrote the
previously repaired config and the bridge rejected it before fixing the nested
root. Source now adds HTTPS ACME to the shipped template and migrates the exact
recognized legacy default-server layout; custom/ambiguous layouts still fail
closed. The missing-token route must return404 rather than the dashboard SPA.
28 Python checks passed. The real isolated nginx suite now starts from the
legacy missing-HTTPS layout, applies the migration, and passes all120 public
negative cases plus HTTP/TLS exact-token, private-access and reload checks.
Applied the latest helper and its atomic ACME-only repair to yaya: actual token
written inside NPM returned exact200 over host HTTP, host HTTPS and public HTTP;
missing tokens returned404 for allthree. Public site trustedTLS/auth preserved.
Local self-signed host HTTPS was tested with certificate verification disabled;
public site HTTPS used normal certificate verification. Shorty was not modified.
The running backend still embeds the older helper/template; final rebuild is
REQUIRED before restart/reboot/persistent upgrade acceptance can pass.
The prepared unsigned catalog validates with zero metadata drift and trusted
registry hosts. Its only changed entries are NPM and Angor indexer1.0.2. It has
not been signed, installed or published. Yaya's current signed catalog retains
NPM's old pasta network/tunnel-only HTTP+TLS listeners; full NPM runtime/bridge
migration acceptance awaits the reviewed signed catalog. Do not mistake the
backend/UI deployment or ACME-only fix for completed catalog migration.
### 2026-10-02: rebuilt candidate deployed; private catalog qualification prepared
Release-profile candidate build completed successfully. SHA256:
af0648ad4ef8b6183d3c1ff485721fa40b12bb730c6e0322bc5f209ed06fce39.
Focused bootstrap tests:10 passed. Full isolated backend rerun:1651 passed,
zero failed, four explicit hardware/external ignores. Python guard/bridge28
passed again. No new source changes occurred between these checks and deployment.
Deployed this rebuilt backend on dev and yaya, with private previous-binary,
nginx and native-container baselines. Both manager health checks passed. A later
post-startup comparison confirmed Bitcoin/LND identities/start times unchanged.
The installed bridge helper now matches latest source bytes after restart.
Private UI200, marked-public HTTP/HTTPS404 and missing HTTPS challenge404 passed.
Dev HTTPS intentionally binds its LAN/WireGuard addresses, not127.0.0.1; an
initial loopback probe got connection refused, corrected to the actual listener.
This was a test-address error, not a product outage. Local self-signed HTTPS
checks skip certificate verification; public-site TLS checks use normal trust.
Full-machine reboot qualification is still pending.
Prepared a fresh candidate catalog with only NPM and Angor indexer entries changed.
Metadata drift0; registry trust check passed. Unsigned SHA256:
5801309bf21d3f6fd03ce5702d68b383a518f734092bf265f5e0ad243095a25e.
NPM's new manifest is capability-gated by runtime-migration-backup-v1; older
nodes retain the original manifest. This candidate is for private qualification,
not fleet publication. An operator-only hidden-input signer validates the exact
catalog and binary hashes, checks the pinned release root and restores the
unsigned original on failure. Its noninteractive refusal was tested. Signature
is required before testing through the nodes' normal trusted-catalog path.
No catalog, app image, OTA or ISO was published. Framework remains deferred;
all other open requirements retain their previous status.
### Signed catalog and private qualification selector
The operator signed the qualification catalog. Cryptographic release-root
verification passed locally and on yaya; after removing signature envelope fields,
its contents exactly match the reviewed unsigned candidate. No publication.
The catalog is staged under /var/lib/archipelago/qualification on both test nodes,
with previous catalog/app metadata and container identities privately backed up.
Normal mirror loading deliberately forces the public origin first, so simply
prepending a private mirror cannot reliably test an unpublished candidate.
Implemented ARCHY_APP_CATALOG_CANDIDATE as an explicit absolute-file selection:
requires an anchored release-root signature, validates before cache replacement,
retains exact signed bytes, does not alter mirrors/trust, and fails without
public fallback when the selected file is invalid. Added unsigned, tampered,
wrong-key, malformed, missing, oversized, relative-path, valid and idempotent
coverage. Full isolated backend suite:1653 passed,0 failed,4 explicit ignores.
The optimized selector build is still in progress; selection is not enabled yet.
See docs/candidate-catalog-qualification.md for activation and mandatory removal
once the tested public catalog is available. Do not leave test nodes pinned.
Yaya NPM preflight:8088/8444 are free, active public host has no conflicting
host-nginx ownership, database tables and certificate-file hashes saved privately.
No Shorty mutation. Dev public Angor reference acceptance passed TLS/WSS, exact
funding commitment, official Explore and detail/statistics again; this still
checks only the known original project, not all35. Dev Bitcoin continues syncing
(reported sync_progress approximately0.307); full-chain acceptance remains open.
Current tested hardware product codes:dev20CLS7S900 and yaya20CLS6BH00, both Podman
5.4.2; kernels6.12.74+deb13+1-amd64 and6.12.107+deb13-amd64 respectively. Framework
remains deferred. No Docker or new ISO-boot acceptance is implied.
### Signed qualification deployment and legacy upstream compatibility
The optimized candidate selector build completed, SHA256
`9cc9271ee1b4f8f13d798193cef97c1e4304a89a240f11f851eb71568bd105e1`.
Both development acceptance nodes now run it with the exact root-verified private
catalog. The isolated backend suite passed 1,653 tests, zero failures, four
explicit ignores. This is unpublished qualification, not a release.
Actual NPM migration exposed an additional regression that the LAN-upstream
fixture missed: a saved site uses pasta's former host gateway `169.254.1.2`.
Default slirp's gateway differs, so the request timed out from inside NPM even
though admin readiness passed. A disposable container verified the same saved
upstream with `slirp4netns:allow_host_loopback=true,cidr=169.254.1.0/24`.
A temporary qualification Quadlet drop-in now selects that network, using an
empty `Network=` reset before the replacement to avoid multiple network modes.
The existing site's trusted public HTTPS authentication response is restored.
Post-repair checks passed: request from NPM's actual namespace; exact saved user,
host, certificate, ACL and settings rows; unchanged certificate bytes; private
migration backup and prior unit; unchanged unrelated container IDs/start times;
retained WireGuard tunnel ports; host bridge completion; private dashboard200,
marked public HTTP/HTTPS404; missing challenge404; exact challenge body from
inside NPM over local HTTP/HTTPS and public HTTP. Native Bitcoin/LND identities
and start times remain unchanged.
**Release blocker:** integrate and test legacy gateway compatibility in all
supported runtime paths and signed manifest, including fresh/upgrade/restart
cases and LAN/host.containers.internal upstreams. The current signed candidate
alone is insufficient. Temporary user-unit drop-in
`nginx-proxy-manager.container.d/90-qualification-host-gateway.conf` must be
removed after the corrected managed configuration is verified. Do not remove
it before then or claim the migration passed without it. Candidate catalog
service selectors also require the cleanup described in
`docs/candidate-catalog-qualification.md` after final publication.
### Development Angor candidate update
The supported `package.update` RPC selected the private signed catalog and
upgraded only `angor-indexer` to the locally built 1.0.2 image. The actual dev
endpoint rendered the Mempool explorer at widths 390 and 1440 with zero failed
JavaScript/CSS requests and one WebSocket connection each. All unrelated dev
containers retained their exact IDs and start times. This verifies the local
explorer presentation, not complete blockchain indexing or recovery of the 34
missing original Angor project announcements. Bitcoin remains in IBD.
The repaired NPM namespace also reached the saved gateway,
`host.containers.internal`, and the node LAN address with the expected site
authentication response. The durable compatibility blocker remains open.
## Live Lightning purchase: Framework to Shorty — 2026-10-02
Operator explicitly authorized a very small new test purchase, then performed
it from Framework. This is separate from recovering the Amish Paradise sale.
Fixture: `archy-lightning-delivery-test-20261002.txt`, price 1 sat, Lightning only.
Read-only checks confirmed one matching seller invoice, SETTLED for exactly
1 sat, and Framework payment SUCCEEDED for 1 sat with 1 sat routing fee
(1,000 msat). Total spent was 2 sats. The assistant sent no payment.
Framework has exactly one durable purchased-content ownership entry for the
fixture, with backend `lightning`, paid_sats=1 and size_bytes=121. Its cached
file SHA256 equals the original seller fixture:
`d55f7a6acd77bdc3c35c65ecac6d1492096e99252d07d433e6286544540cde7e`.
**PASS: actual node-wallet payment, seller settlement, delivered bytes and
persisted buyer ownership/cache.** Framework runs the candidate backend
`8fb6249d1869bb8c9aea26d0f846de5b3eec328113a5c7f573628306e26e652e`;
Shorty runs `e108b78bbbd21cb7d5d47c8d0b7b9b19b63fb0c44678773603202440ec7d6f5b`.
This also exercises the candidate buyer against the existing seller version.
Live reopen without payment and restart acceptance are not established by
this check. Shorty had no matching durable entitlement JSON in the inspected
location; do not attribute the candidate seller persistence implementation to
this older seller binary. No node or wallet was restarted. The tiny fixture
remains available for a free cached reopen check; remove only its catalog
entry/source file afterwards, preserving buyer ownership and payment records.
### Operator-confirmed free reopen — 2026-10-02
After the verified one-sat purchase, the operator reopened the file on Framework
and confirmed it worked without another payment. **PASS: live paid delivery,
durable buyer ownership/cache, and free repeat access**, with independent
settlement/byte checks above and operator confirmation of the reopen UI.
This closes that specific live acceptance check; it does not establish an
untested restart, outage, or seller-upgrade scenario.
The temporary seller catalog entry and source fixture were removed after
acceptance. Buyer purchased bytes/ownership and all payment records were preserved.
File diff suppressed because it is too large Load Diff
+37
View File
@@ -0,0 +1,37 @@
# 1.9.0-alpha: known limitations
## Angor historical discovery
Historical project discovery is incomplete. Funding commitments for all35
reference projects were verified, but34 original signed announcements were not
recovered from the relays and archives checked. They are not proven globally
lost. One recovered project passes Explore, details, statistics and public
TLS/WebSocket acceptance using the self-hosted services.
The operator explicitly accepted shipping with this limitation on2026-10-05.
Recovery remains follow-up work; this acceptance does not mean all-project
discovery passed. Retain the original relays alongside the self-hosted relay.
Full indexing requires a synced, unpruned Bitcoin node and indexing dependencies.
The dev node is still syncing; full-chain evidence comes from Shorty.
Evidence and inventory: [client acceptance](angor-client-acceptance-20261001.md),
[project recovery inventory](angor-project-recovery-20261001.json), and
[release acceptance](release-1.9.0-acceptance.md).
## Chat/contact storage migration — discovered during follow-up testing
A subsequent full backend test run exposed an existing random-prefix collision
in `storage_crypto::is_plaintext_json`: an encrypted store whose nonce begins
with `{` or `[` can be mistaken for legacy plaintext. This can prevent chat or
contact state loading correctly, and the message migration path can overwrite
that state. This helper is used for chat messages and mesh contact customizations,
not wallet databases.
The follow-up branch replaces the prefix-only heuristic with complete JSON
validation and adds deterministic encrypted-prefix regression cases. Qualification
is in progress; the published 1.9.0-alpha artifacts do not include that fix.
Existing release signatures and tags must not be silently replaced. Track a
corrective update and preserve affected state before further node restarts.
The earlier green release run did not detect this probabilistic failure; do not
interpret it as proof that this newly discovered issue is absent.

Some files were not shown because too many files have changed in this diff Show More