fix: reject stale first-boot scripts in OTA release payloads

docs/post-1.9.0-work-backlog.md

@@ -218,3 +218,46 @@ upgrade/restart and rollback, publish through the signed app catalog, and verify
 existing nodes discover and apply the intended version. Distinguish an app-only
 release from any backend/OTA dependency; use the documented decoupled app-update
 path where supported. Do not silently require a full OTA for an app-only change.
+
+Sequencing clarification: finish1.9.0-alpha first. Publish any required IndeeHub
+app update at the end of the follow-up implementation and qualification, not
+ahead of that work or as an untested addition to the current release.
+
+## 13. V4V Portainer demo as a Yaya node app
+
+- After the current release, deploy/package the existing V4V Portainer deployment
+  as a demo app on Yaya, showcasing how an ordinary third-party app works on a
+  node **without native Nostr signer integration**, as explicitly requested.
+- Inspect the actual existing Portainer source, branch/image revision, Compose
+  stack, access/authentication and data before changes; retain the working V4V
+  site, stack and persistent state. Do not confuse this with the public Archipelago
+  software demo at demo.archipelago-foundation.org.
+- Follow the current app-development guide and supported app packaging/gate/
+  lifecycle conventions. Define the app card/icon/category, launch URL/readiness,
+  network/auth boundaries, health, configuration and persistent mounts properly.
+  Do not expose the native signer or implicitly grant signing permissions.
+- Qualify fresh installation in isolation, then Yaya deployment, desktop/mobile/
+  companion launch, normal app functionality, restart, update/rollback and safe
+  removal behavior. Verify the deployed app actually runs the intended V4V source
+  revision, not a stale build or unrelated image.
+- Record a short reproducible demo flow and operator UAT checklist. Preserve
+  existing Gitea/Portainer connectivity and unrelated apps; no new payment or
+  public sharing of private test content is implied by the demo packaging.
+
+### V4V node-only catalog and Sovereign Music promotion
+
+- Source is on the existing Gitea on the146 server; locate the actual V4V repo,
+  branch and deployment revision there rather than guessing a replacement source.
+- Add a **Sovereign Music** banner for V4V on Yaya, following the existing
+  Sovereign Streaming banner treatment and using the app's own login background.
+  Retrieve and inspect the real asset; do not invent a replacement illustration.
+- Both demo app availability and its promotion must be confined to Yaya. Evaluate
+  a signed per-node/DID-scoped demo catalog or existing supported node-specific
+  candidate mechanism. Do not publish the demo to the global catalog or let
+  unrelated nodes install it implicitly through a shared catalog cache.
+- Test matching/nonmatching identities, copying URLs/catalogs between nodes,
+  missing identity, refresh/restart/update and promotion visibility. Catalog
+  selection or visibility must not bypass normal artifact-signature enforcement.
+- This may become a real app later; preserve an explicit tested promotion path
+  from node-only demo to proper public app release without duplicate app IDs,
+  conflicting state, lost configuration or automatic exposure before approval.

docs/release-1.9.0-acceptance.md

@@ -868,3 +868,71 @@ OTA apply/rollback test necessarily follows signing. Mirror/tag parity, public
 artifact downloads, catalog promotion, fleet discovery and demo deployment
 remain required. The Angor historical limitation is explicitly accepted and
 documented in [known limitations](release-1.9.0-known-limitations.md).
+
+### Signed OTA qualification — 2026-10-05
+
+Operator signed final OTA manifest and raw-ISO checksum document; both verify
+against the pinned release root. Existing signed catalog also verifies. On the
+disposable installed VM, the actual published1.8.22 backend and frontend were
+verified against their published hashes, installed as the baseline, and used to
+discover/download/apply the final signed1.9.0 artifacts through normal authenticated
+RPC. Component verification, automatic manager restart, post-OTA verification,
+exact backend/UI hashes, Cloud access and persistent credential/file checks pass.
+
+A deliberately missing new frontend plus the real pending-verification marker
+triggered automatic rollback: exact1.8.22 binary/UI restored, file and credentials
+preserved.32 public-source IPv4/IPv6 management-denial requests still pass with
+the restored old binary/template. Reapplying the signed1.9.0 OTA succeeds with the
+same post-update assertions. Final whole-VM reboot qualification is running.
+Logs: `/tmp/archy-190-vm-ota-cycle-2.log`,
+`/tmp/archy-190-vm-ota-rollback.log`,
+`/tmp/archy-190-vm-ota-rollback-security.log`,
+`/tmp/archy-190-vm-ota-reapply.log`. Fixture setup corrections (missing systemd
+drop-in directory and underscore in update_state.json) preceded the passing run;
+no failed fixture run is counted as acceptance.
+
+Publication storage required temporary upload headroom beyond the previous
+cleanup. Under the operator's existing old-ISO retention authorization, removed
+only1.8.18 ISO attachment235 after verifying its retained local copy against the
+public signed checksum. All other assets unchanged;1.8.19/21/22 server ISOs remain.
+Server free space is7.2GB before upload. Use an SSH-tunneled direct Gitea upload
+so the public reverse proxy does not buffer an additional multi-GB copy.
+
+Historical mirror audit identified three missing ngit tags1.8.16/17/18; their
+local annotated tag objects exactly matched Gitea and were copied to ngit without
+rewriting history. Unrelated proposal-only branch differences are inventoried
+in `/tmp/archy-190-historical-mirror-audit.log`; full branch parity is not claimed.
+Final main/tag parity remains a separate publication gate.
+
+### Final reboot caught a stale OTA runtime script — corrected package
+
+The whole-VM reboot itself reached healthy1.9.0, but the first-boot retry check
+failed: the OTA runtime overlay still shipped the old first-boot script and
+bootstrap installed it over the ISO's corrected version. Retry logged missing
+`log` and changed running container IDs/start times. This is a real package
+regression, not a passed check. The original signed frontend archive3047a19f is
+obsolete and MUST NOT be published.
+
+Repacked only `archipelago-runtime/scripts/first-boot-containers.sh` with the
+already qualified source repair. Full archive comparison proves every other
+entry, content and mode unchanged. Corrected frontend SHA256 is
+`6d5135fa8e79b1bc84c8ed972770ebf99fe6611d819e5c1453f38f1593c26e66`.
+ISO/backend/dashboard/AIUI/APK/catalog bytes are unchanged; ISO and catalog
+signatures remain valid. Only the corrected OTA manifest needs renewed signing.
+
+Added a release-manifest payload gate that rejects stale, absent or duplicate
+first-boot scripts. Four archive fixture cases and existing executable first-boot
+retry regression pass; the stale real archive fails, corrected real archive
+passes. Actual backend bootstrap successfully promotes the corrected member on
+the disposable node. Post-promotion retry/Cloud checks are in progress.
+Logs: `/tmp/archy-190-stale-ota-reproduced.log`,
+`/tmp/archy-190-repackage-runtime-2.log`,
+`/tmp/archy-190-corrected-manifest-integrity.log`,
+`/tmp/archy-190-vm-corrected-runtime.log`,
+`/tmp/archy-190-vm-corrected-retry.log`. Renewed signature and exact signed apply
+remain required. Keep earlier rollback evidence for the unchanged backend, but
+do not claim the obsolete frontend is the final accepted artifact.
+
+Corrected runtime post-promotion retry now PASS: container IDs/start times and
+credentials preserved, Cloud token/listing work, default/anonymous access denied.
+Corrected OTA pre-sign receipt is ready; existing ISO/catalog signatures retained.

scripts/check-ota-runtime.py

@@ -0,0 +1,34 @@
+#!/usr/bin/env python3
+"""Reject OTA archives that would overwrite the qualified first-boot repair."""
+import argparse
+import pathlib
+import tarfile
+
+
+def check(archive, source):
+    member_name = 'archipelago-runtime/scripts/first-boot-containers.sh'
+    with tarfile.open(archive, 'r:gz') as package:
+        matches = [m for m in package.getmembers()
+                   if m.name.removeprefix('./') == member_name]
+        if len(matches) != 1 or not matches[0].isfile():
+            raise ValueError('OTA must contain exactly one regular first-boot script')
+        with package.extractfile(matches[0]) as stream:
+            actual = stream.read()
+    if actual != source.read_bytes():
+        raise ValueError('OTA first-boot script differs from qualified source; repackage before signing')
+
+
+def main():
+    parser = argparse.ArgumentParser(description=__doc__)
+    parser.add_argument('archive', type=pathlib.Path)
+    args = parser.parse_args()
+    try:
+        check(args.archive, pathlib.Path(__file__).resolve().parents[1] /
+              'scripts/first-boot-containers.sh')
+    except (OSError, ValueError, tarfile.TarError) as error:
+        parser.exit(1, f'FAIL: {error}\n')
+    print('PASS: OTA first-boot script matches qualified source')
+
+
+if __name__ == '__main__':
+    main()

scripts/check-release-manifest.sh

@@ -114,6 +114,10 @@ for i in $(seq 0 $((COMPONENT_COUNT - 1))); do
     if [ "$ACTUAL_SIZE" != "$DECLARED_SIZE" ]; then
         fail "component '$NAME' size mismatch (declared=$DECLARED_SIZE actual=$ACTUAL_SIZE)"
     fi
+    if [[ "$NAME" == *frontend*.tar.gz ]]; then
+        python3 "$REPO_ROOT/scripts/check-ota-runtime.py" "$FILE" \
+            || fail "frontend runtime payload is stale or incomplete"
+    fi
     ok "component '$NAME': sha256 + size match on-disk artifact"
 done
 

tests/regression/ota-runtime-firstboot.py

@@ -0,0 +1,43 @@
+#!/usr/bin/env python3
+"""Exercise the release payload gate with real archives, including stale bytes."""
+import importlib.util
+import io
+import pathlib
+import tarfile
+import tempfile
+import unittest
+
+ROOT = pathlib.Path(__file__).resolve().parents[2]
+spec = importlib.util.spec_from_file_location('check_ota_runtime', ROOT / 'scripts/check-ota-runtime.py')
+validator = importlib.util.module_from_spec(spec)
+spec.loader.exec_module(validator)
+
+
+class RuntimePayloadTests(unittest.TestCase):
+    def test_actual_archives(self):
+        with tempfile.TemporaryDirectory() as directory:
+            base = pathlib.Path(directory)
+            source = base / 'first-boot-containers.sh'
+            source.write_bytes(b'#!/bin/sh\necho qualified\n')
+            for case, contents, valid in [
+                ('qualified', [source.read_bytes()], True),
+                ('stale', [b'#!/bin/sh\necho obsolete\n'], False),
+                ('missing', [], False),
+                ('duplicate', [source.read_bytes(), source.read_bytes()], False),
+            ]:
+                with self.subTest(case=case):
+                    archive = base / (case + '.tar.gz')
+                    with tarfile.open(archive, 'w:gz') as output:
+                        for content in contents:
+                            member = tarfile.TarInfo('./archipelago-runtime/scripts/first-boot-containers.sh')
+                            member.size = len(content)
+                            output.addfile(member, io.BytesIO(content))
+                    if valid:
+                        validator.check(archive, source)
+                    else:
+                        with self.assertRaises(ValueError):
+                            validator.check(archive, source)
+
+
+if __name__ == '__main__':
+    unittest.main()

tests/release/run.sh

@@ -79,6 +79,7 @@ stage "demo-resumable-uploads" timeout 120 node --test neode-ui/scripts/demo-upl
 stage "companion-signature-regression" python3 scripts/tests/test_companion_apk_verification.py
 stage "cargo-fmt"        timeout 240 cargo fmt --manifest-path core/Cargo.toml --all --check
 stage "app-build-contexts" python3 tests/regression/app-build-contexts.py
+stage "ota-runtime-firstboot" python3 tests/regression/ota-runtime-firstboot.py
 stage "manifest-shell"  python3 scripts/check-manifest-shell.py
 stage "npm-tunnel-migration" python3 -m unittest discover -s scripts/tests -p test_repair_npm_tunnel.py
 stage "npm-public-bridge" python3 -m unittest discover -s scripts/tests -p test_npm_public_bridge.py