fix(lifecycle): let user app actions wait out background lock holds #165

2026-10-24T11:58:06Z

ssmithx commented

2026-10-24 11:58:06 +00:00

Problem

User Start / Stop / Restart / Install / Uninstall actions were refused with "An app lifecycle operation is active; retry when it completes".

The background reconcile pass holds the app-lifecycle flock for its whole walk over every loaded manifest. On a 2-CPU test node that was 20–21s per pass, every ~50s, even when idle. Health recovery also holds it during auto-restarts. User-facing handlers tried the lock once with LOCK_NB, so any action that landed during a hold failed immediately.

Fix (4 commits)

Wait instead of refusing. Guard::acquire_waiting / acquire_within poll every 200ms. Background acquirers (reconcile, health monitor, crash recovery) keep try-once semantics and retry next cycle. flock errors other than EWOULDBLOCK are now reported as themselves.
Background ops wait longer. install / update / uninstall take the lock inside their spawned task, after the RPC has answered, so they use BACKGROUND_LOCK_WAIT (5 min). An uninstall issued 13s after an Archipelago restart had previously given up while the first reconcile pass held the lock.
Reconcile yields between apps. Waiting commands register per data dir. Between apps, the pass checks for a waiter, records where it stopped and releases the lock. The next pass resumes from that app, so the end of the priority order isn't starved. The pass still never acts without the lock.
Start/stop/restart hand a long wait to the background. A single app's reconcile step (drift checks, hooks, sudo ownership probes) can exceed 10s, so yielding alone still let a Stop time out. These commands try for USER_LOCK_WAIT (10s, inside the UI's 15s RPC timeout), which keeps errors synchronous in the common case. If the lock still isn't free, the command is accepted: state flips immediately, and the background task waits (5 min, as a registered waiter), runs the same require_clear / require_unheld checks, then does the work. A failure there reverts the state and is logged, matching install/uninstall.

Tests

New: user_commands_wait_out_a_short_background_hold, user_commands_give_up_when_the_hold_outlasts_the_wait, waiting_command_is_visible_only_while_it_waits, reconcile_pass_yields_to_a_waiting_command_and_resumes_there, busy_lifecycle_lock_defers_the_command_to_its_background_task.
124 lifecycle/package/update-transaction tests pass. The prod_orchestrator failures in this environment (6, sometimes 7) also fail on unmodified main; they need scripts/test-backend-isolated.sh.

Verified on a real node

Lifecycle gate (run-gate.sh, 5 iterations, on-node), with suites for apps not installed on that node excluded: 5/5 green, 155 passed, 0 failed, 0 refused commands. Before commits 3–4 the same run failed 2 of 5 iterations on filebrowser:stop-rpc.
Stops issued mid-reconcile-pass were answered in 0.2–0.4s.
A 4-container stack uninstall → reinstall right after an Archipelago restart completed.

🤖 Generated with Claude Code

## Problem User Start / Stop / Restart / Install / Uninstall actions were refused with *"An app lifecycle operation is active; retry when it completes"*. The background reconcile pass holds the app-lifecycle `flock` for its whole walk over every loaded manifest. On a 2-CPU test node that was **20–21s per pass, every ~50s, even when idle**. Health recovery also holds it during auto-restarts. User-facing handlers tried the lock once with `LOCK_NB`, so any action that landed during a hold failed immediately. ## Fix (4 commits) 1. **Wait instead of refusing.** `Guard::acquire_waiting` / `acquire_within` poll every 200ms. Background acquirers (reconcile, health monitor, crash recovery) keep try-once semantics and retry next cycle. `flock` errors other than `EWOULDBLOCK` are now reported as themselves. 2. **Background ops wait longer.** `install` / `update` / `uninstall` take the lock inside their spawned task, after the RPC has answered, so they use `BACKGROUND_LOCK_WAIT` (5 min). An uninstall issued 13s after an Archipelago restart had previously given up while the first reconcile pass held the lock. 3. **Reconcile yields between apps.** Waiting commands register per data dir. Between apps, the pass checks for a waiter, records where it stopped and releases the lock. The next pass resumes from that app, so the end of the priority order isn't starved. The pass still never acts without the lock. 4. **Start/stop/restart hand a long wait to the background.** A single app's reconcile step (drift checks, hooks, sudo ownership probes) can exceed 10s, so yielding alone still let a Stop time out. These commands try for `USER_LOCK_WAIT` (10s, inside the UI's 15s RPC timeout), which keeps errors synchronous in the common case. If the lock still isn't free, the command is accepted: state flips immediately, and the background task waits (5 min, as a registered waiter), runs the same `require_clear` / `require_unheld` checks, then does the work. A failure there reverts the state and is logged, matching install/uninstall. ## Tests - New: `user_commands_wait_out_a_short_background_hold`, `user_commands_give_up_when_the_hold_outlasts_the_wait`, `waiting_command_is_visible_only_while_it_waits`, `reconcile_pass_yields_to_a_waiting_command_and_resumes_there`, `busy_lifecycle_lock_defers_the_command_to_its_background_task`. - 124 lifecycle/package/update-transaction tests pass. The prod_orchestrator failures in this environment (6, sometimes 7) also fail on unmodified `main`; they need `scripts/test-backend-isolated.sh`. ## Verified on a real node - Lifecycle gate (`run-gate.sh`, 5 iterations, on-node), with suites for apps not installed on that node excluded: **5/5 green, 155 passed, 0 failed, 0 refused commands**. Before commits 3–4 the same run failed 2 of 5 iterations on `filebrowser:stop-rpc`. - Stops issued mid-reconcile-pass were answered in 0.2–0.4s. - A 4-container stack uninstall → reinstall right after an Archipelago restart completed. 🤖 Generated with [Claude Code](https://claude.com/claude-code)

ssmithx added 1 commit 2026-10-24 11:58:07 +00:00

fix(lifecycle): let user app actions wait out background lock holds 974279d3a5

The reconcile pass takes the app-lifecycle lock for several seconds on
every ~50s cycle, and health recovery holds it during auto-restarts.
User-initiated start/stop/restart/install/update tried the lock once
with LOCK_NB, so a click that landed during a pass was refused with
"An app lifecycle operation is active".

User commands now poll for the lock for up to 10s (well inside the UI's
15s RPC timeout) before refusing. Background passes keep try-once
semantics and simply retry next cycle. Errors other than contention
from flock are now reported instead of being shown as "busy".

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

ssmithx referenced this pull request

2026-10-24 12:21:29 +00:00

fix(btcpay): declare btcpay-db-password; stop reconcile failing on absent apps #166

ssmithx added 1 commit 2026-10-24 14:04:53 +00:00

fix(lifecycle): give background install/update/uninstall a longer lock wait b224676da8

These three take the lifecycle lock inside their spawned task, after the
RPC has already answered, so the 10s limit meant for start/stop/restart
(which hold the RPC open) was needlessly short. On a test node an
uninstall issued 13s after an Archipelago restart gave up while the first
reconcile pass still held the lock, and the app stayed installed.

They now wait up to 5 minutes, which covers the first pass after a
restart and a health-monitor restart (up to 120s).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

ssmithx added 1 commit 2026-10-24 15:44:03 +00:00

fix(reconcile): yield the lifecycle lock to waiting user commands 8b3dc0c496

The reconcile pass holds the app-lifecycle lock for its whole walk over
every loaded manifest. On a small 2-CPU test node that was 20-21s per
pass, every ~50s, even when idle, so a Stop click that landed in a pass
still outlasted the 10s USER_LOCK_WAIT. The lifecycle gate hit it twice
in 5 iterations (filebrowser:stop-rpc).

Commands waiting in acquire_waiting now register per data dir. Between
apps, the pass checks for a waiter, records where it stopped and
releases the lock. The next pass starts from that app, so a busy UI
can't starve the end of the priority order. The pass still never acts
without the lock, and held names are still read under it.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

ssmithx added 1 commit 2026-10-24 16:28:23 +00:00

fix(lifecycle): hand a long lock wait for start/stop/restart to the background ee40ce8f9c

Yielding between apps was not enough: a single app's reconcile step
(drift checks, hooks, sudo ownership probes) took over 10s on a 2-CPU
test node, so a Stop issued at the start of that step still timed out.
The lifecycle gate hit this in 2 of 5 iterations.

start/stop/restart still try the lock for USER_LOCK_WAIT so errors reach
the UI directly in the common case. If it isn't free by then, the
command is accepted anyway: state flips to Stopping/Starting/Restarting
and the background task waits for the lock (BACKGROUND_LOCK_WAIT, as a
registered waiter so the reconcile pass yields), runs the same
require_clear/require_unheld checks, then does the work. A failure there
reverts the state and is logged like any other background failure.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

chaum commented

2026-11-09 12:31:31 +00:00

Reconciled against canonical ngit review: fix(lifecycle): let user app actions wait out background lock holds is applied.

This PR's exact head ee40ce8f9c329a155f1919cf20891944e39fb758 is already an ancestor of accepted main 1a4ea15df9c8800b6d288277d2839c7b93662ff7, which matches on ngit and Gitea. The existing review fixes and commit history are preserved.

Closing the redundant Gitea PR record because its contribution has already been integrated through ngit. No second merge or history rewrite is being performed. Artifact publication and installed-device acceptance remain separate from this source-review disposition.

Reconciled against canonical ngit review: [fix(lifecycle): let user app actions wait out background lock holds](https://gitworkshop.dev/nevent1qqsze9uz5lxr77552acm624pje5nfg62lmf9hg6rt3tj6gr3672sqscpz3mhxue69uhhyetvv9ujumn8d96zuer9wctg437t) is **applied**. This PR's exact head `ee40ce8f9c329a155f1919cf20891944e39fb758` is already an ancestor of accepted main `1a4ea15df9c8800b6d288277d2839c7b93662ff7`, which matches on ngit and Gitea. The existing review fixes and commit history are preserved. Closing the redundant Gitea PR record because its contribution has already been integrated through ngit. No second merge or history rewrite is being performed. Artifact publication and installed-device acceptance remain separate from this source-review disposition.

chaum closed this pull request

2026-11-09 12:31:32 +00:00

Pull request closed

Please reopen this pull request to perform a merge.

Sign in to join this conversation.

2 Participants

Notifications

Due Date

No due date set.

Dependencies

No dependencies set.

Reference: lfg2025/archy#165