From 8a2707ee37e2d3ac10f1734b6a7b456a83fcd3ff Mon Sep 17 00:00:00 2001 From: Gmer4Lfe Date: Sat, 1 Aug 2026 22:59:07 -0400 Subject: [PATCH] Correct and extend folder docs for Docker Essentials, Watchdogs, Media and Arrs Stack MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The docs had drifted from the scripts — a script that no longer exists, three wrong variable names, a reversed run order, and seven scheduled scripts that were never documented at all. --- Arrs_Stack/README-Arrs_Stack.md | 69 ++++++++++++++ Docker_Essentials/Manual-Docker_Essentials.md | 89 ++++++++++++++++--- Docker_Essentials/README-Docker_Essentials.md | 73 +++++++++++++-- Media/Manual-Media.md | 40 +++++++-- Media/README-Media.md | 40 +++++++++ Watchdogs/README-Watchdogs.md | 50 +++++++++++ 6 files changed, 331 insertions(+), 30 deletions(-) diff --git a/Arrs_Stack/README-Arrs_Stack.md b/Arrs_Stack/README-Arrs_Stack.md index 7fe062f..5fe1baf 100644 --- a/Arrs_Stack/README-Arrs_Stack.md +++ b/Arrs_Stack/README-Arrs_Stack.md @@ -200,6 +200,75 @@ Lidarr configured — no HOST1_LIDARR_URL means nothing runs. | `playback_aware_lidarr_discovery.sh` | Behavior-driven music discovery — Emby plays → Last.fm similar → Lidarr | Weekly | | `playback_aware_radarr_discovery.sh` | Behavior-driven movie discovery — Emby watches → TMDB recommendations → Radarr | Weekly | | `playback_aware_sonarr_discovery.sh` | Behavior-driven TV discovery — Emby episodes → TMDB TV recommendations → Sonarr | Weekly | +| `arr_download_orphan_cleaner.sh` | Clear orphaned completed downloads out of the SABnzbd Completed folders | Daily | +| `sonarr_classification_scan.sh` | Detect series sitting in the wrong root (anime / kids / general); `--move` acts | Daily | +| `radarr_classification_scan.sh` | Same for movies, plus junk-metadata detection via `--remove-junk` | Daily | +| `lidarr_duplicate_artist_cleanup.sh` | Remove phantom zero-file duplicate artists; flag real ones for review | Daily | +| `arr_cache_prefill.sh` | Warm the shared tracked-data cache so consumers never read cold | Array start + every 4hr | +| `arr_corruption_scan.sh` | ffprobe every tracked video for corrupt headers; `--remediate` deletes + re-searches | Weekly | +| `arr_full_rescan.sh` | Force a real disk↔database reconciliation on all three arrs | Weekly | + +--- + +## ━━━ THE NEWER LAYERS ━━━ + +The original folder was "delete what the arrs no longer track". These were added as distinct +failure modes surfaced — each exists because something went wrong that the cleanups could not +have caught. + +### 🗑️ Download-Side Orphans — `arr_download_orphan_cleaner.sh` + +Every cleanup script here walks the **library** side. Nothing walked the **download** side — +so completed downloads the arrs had stopped tracking accumulated in SABnzbd's Completed +folders indefinitely. Discovered as **755 GB of orphaned TV downloads, oldest from 2022**, +filling the cache pool to 89%. + +Classifies every entry as TRACKED / RECENT / JUNK / REDUNDANT / IMPORTABLE / UNMATCHED and +acts only on the ones it can justify. The queue is a hard gate: if it cannot be read, the arr +is skipped entirely, because without it there is no way to tell an active import from an +orphan. A run total over `DOWNLOAD_ORPHAN_MAX_DELETE_GB` aborts — an abnormally large delete +is the visible symptom of a partial queue fetch. + +### 🎭 Wrong-Root Detection — `sonarr_classification_scan.sh` + `radarr_classification_scan.sh` + +Overseerr lets any user request content into the wrong root folder — kids shows into general +TV, anime into Kids_Tv_Shows. These classify every item from metadata alone (genre, +certification, network/studio, original language) and report where the computed classification +disagrees with the folder the item actually sits in. + +Report-only by default. `--move` acts on forward misplacements and adult-content-in-kids-root +leaks. It deliberately does **not** move non-anime content out of the anime root — deliberate +style placements (Western animation grouped with anime by choice) are genuine judgment calls. + +Both poll the arr's async move command to completion before verifying, because `moveFiles=true` +flips the database instantly while the physical move is still queued behind others. + +### 🎨 Phantom Artists — `lidarr_duplicate_artist_cleanup.sh` + +MusicBrainz duplicates leave two Lidarr entries for one artist, one holding the files and one +holding nothing. Removes only the zero-file side, with `deleteFiles=false` so nothing on disk +is touched. Pairs where both sides hold files are flagged for review, never auto-resolved. + +Gated on the tracked-count floor shared with `lidarr_cleanup.sh` — during a library-wide desync +both sides of a real duplicate can read as zero-file phantoms. + +### 🩺 Corruption + Reconciliation — `arr_corruption_scan.sh` + `arr_full_rescan.sh` + +`arr_corruption_scan.sh` ffprobes tracked video files for corrupt headers. Report-only unless +`--remediate`, which deletes the file record and triggers an explicit re-search. Requires +repeat detections across separate runs before acting, so a transient probe failure cannot +delete a healthy file. + +`arr_full_rescan.sh` forces a genuine disk↔database reconciliation. Organic scans only touch +files involved in an import, so an untouched library silently drifts — confirmed when Lidarr +reported **~23% of its true track count** for 1,004 of 1,357 artists with no scan running and +every file present on disk. + +### ⚡ Cache Warmth — `arr_cache_prefill.sh` + +Populates the shared tracked-data cache at array start and every 4 hours, so consumers never +pay a cold fetch. Pure enhancement: nothing depends on it having run, and every consumer still +writes through on a cold cache. --- diff --git a/Docker_Essentials/Manual-Docker_Essentials.md b/Docker_Essentials/Manual-Docker_Essentials.md index 3e130ae..dadd7ef 100644 --- a/Docker_Essentials/Manual-Docker_Essentials.md +++ b/Docker_Essentials/Manual-Docker_Essentials.md @@ -94,20 +94,54 @@ HOST1_NETWORK_CONNECT_CONTAINERS=( ```bash # master.conf -DAILY_CONTAINER_UPDATES=true # enable/disable daily image pull +DAILY_CONTAINER_UPDATES=true # enable/disable the daily image pull # docker_daily_restart.sh still runs regardless # update and restart are independent -WEEKLY_REMAINING_UPDATES=true # enable/disable weekly remainder pull + prune - # to disable: set false or remove from WEEKLY_MAINTENANCE_SCRIPTS +WEEKLY_CONTAINER_UPDATES=true # enable/disable the weekly image pull + # docker_weekly_restart.sh still runs regardless + +MONTHLY_REMAINING_UPDATES=true # enable/disable the monthly remainder pull + # to disable: set false, or remove + # "docker_update.sh --remainder" from + # MONTHLY_MAINTENANCE_SCRIPTS ``` -`docker_update.sh` in normal mode targets `DAILY_RESTART_CONTAINERS` — the same list -used by `docker_daily_restart.sh`. No second list to maintain. +There is one update script, `docker_update.sh`, with three modes. Each reuses the restart +list it pairs with, so there is no second list to maintain: -`docker_update_remaining.sh` derives its target list automatically: -all running containers minus `DAILY_RESTART_CONTAINERS` minus `WEEKLY_RESTART_CONTAINERS`. -Everything gets updated at least once per week with no explicit configuration. +| Mode | Targets | Runs | +|------|---------|------| +| *(default)* | `DAILY_RESTART_CONTAINERS` | Daily, before `docker_daily_restart.sh` | +| `--weekly` | `WEEKLY_RESTART_CONTAINERS` | Weekly, before `docker_weekly_restart.sh` | +| `--remainder` | derived, see below | Monthly, via `MONTHLY_MAINTENANCE_SCRIPTS` | + +Remainder mode needs no configuration at all. It takes every **running** container and +subtracts: + +``` + DAILY_RESTART_CONTAINERS already updated daily + WEEKLY_RESTART_CONTAINERS already updated weekly + PROFILE_CRITICAL_CONTAINER_NAMES[emby] updated inline by the weekly sync window + PROFILE_CRITICAL_CONTAINER_NAMES[critical-data] updated inline by the weekly sync window + FALLBACK__TIER1..4 owned by the remote's update cycle +``` + +Stopped containers are never targeted in any mode — pulling for a stopped container adds +nothing, and it was most likely stopped deliberately. + +The fallback exclusion is a correctness rule, not an optimisation. This server only runs +those containers during a fallback; the remote owns their version. If remainder updated them +independently and a handback then occurred, the remote's older image could meet data written +by the newer one. + +**Ordering matters.** The update always runs *before* its matching restart so the restart +lands on the freshly pulled image. If a container's image actually changed, `docker_update.sh` +rebuilds it from its template (a plain `docker restart` reuses the image ID baked in at +creation time and would never pick up the new digest) and records it in +`DOCKER_UPDATE_REBUILT_*_FILE`. The restart script reads that file and skips those containers +rather than restarting them a second time — and discards the file as stale if it is older +than `DOCKER_UPDATE_REBUILT_STALE_HOURS`. --- @@ -142,12 +176,22 @@ HOST1_NETWORK_CONNECT_CONTAINERS=() # Shared — applies to both servers # ── Container updates ────────────────────────────────────────────────── -DAILY_CONTAINER_UPDATES=true -WEEKLY_REMAINING_UPDATES=true +DAILY_CONTAINER_UPDATES=true # daily pull (DAILY_RESTART_CONTAINERS) +WEEKLY_CONTAINER_UPDATES=true # weekly pull (WEEKLY_RESTART_CONTAINERS) +MONTHLY_REMAINING_UPDATES=true # monthly pull (everything else) + +# Handoff between update and restart — written by docker_update.sh, +# read by the matching restart script so it skips containers already +# rebuilt onto a new image this run. +DOCKER_UPDATE_REBUILT_DAILY_FILE +DOCKER_UPDATE_REBUILT_WEEKLY_FILE +DOCKER_UPDATE_REBUILT_STALE_HOURS=12 # older than this = discarded, restart all # ── Retry behaviour (shared by restart scripts) ──────────────────────── RETRY_COUNT=3 # retry attempts before marking failed SLEEP=5 # seconds between retry attempts +CONTAINER_DELAY # seconds between a dependency and its dependents +RESTART_VERIFY_WAIT=3 # settle time before verifying a restart stuck # Watchdog thresholds (CPU, memory, HTTP, restart loop): # → see Watchdogs/Manual-Watchdogs.md @@ -211,7 +255,24 @@ All scripts support these standard flags: summary block: identity, duration, counts, and a status line. Per-container detail only appears with `--log`. Warnings and errors are always visible regardless of `--log`. -### `docker_update.sh --remainder` -Switches to remainder mode — updates all running containers not in the managed daily/weekly -lists. Called by `weekly_sync_maintenance.sh`. Can be run manually to sweep containers -that haven't been updated recently. +### Script-specific flags + +| Script | Flag | What it does | +|--------|------|-------------| +| `docker_update.sh` | `--weekly` | Target `WEEKLY_RESTART_CONTAINERS`. Called by `weekly_sync_maintenance.sh` before the weekly restart. | +| `docker_update.sh` | `--remainder` | Target every running container not in the daily list, weekly list, emby/critical-data profiles, or fallback tiers. Called by `monthly_maintenance.sh`. Safe to run manually to sweep anything missed. | +| `media_cleaner.sh` *(Media/)* | `` | Required positional profile — there is no default. | + +`docker_update.sh` with no mode flag is normal mode: `DAILY_RESTART_CONTAINERS`, called by +`daily_sync_maintenance.sh` before the daily restart. + +### Exit codes + +| Code | Meaning | +|------|---------| +| `0` | Success, or nothing to do (empty list, disabled toggle, no running containers) | +| `1` | One or more containers failed — the summary names them | + +`docker_update.sh` deliberately exits `0` even when pulls fail: a failed pull is not a reason +to abort the restart that follows, which simply proceeds on the existing image. The failure +is reported in the summary. diff --git a/Docker_Essentials/README-Docker_Essentials.md b/Docker_Essentials/README-Docker_Essentials.md index 6e2f208..024eed9 100644 --- a/Docker_Essentials/README-Docker_Essentials.md +++ b/Docker_Essentials/README-Docker_Essentials.md @@ -178,12 +178,28 @@ windows so any downtime from restarts is absorbed by the window that's already h --- -### 🔄 Image Currency — `docker_update.sh` + `docker_update_remaining.sh` +### 🔄 Image Currency — `docker_update.sh` -Keeps all container images current without manual intervention. Daily updates for the -auth/proxy stack (the containers that restart daily anyway — no extra downtime). Weekly -remainder pass for everything else — derives the target list automatically from `docker ps` -minus what was already updated, so there is no second list to maintain. +One script, three modes — there is no separate remainder script. + +| Mode | Targets | Runs | +|------|---------|------| +| *(default)* | `DAILY_RESTART_CONTAINERS` | Daily, **before** `docker_daily_restart.sh` | +| `--weekly` | `WEEKLY_RESTART_CONTAINERS` | Weekly, **before** `docker_weekly_restart.sh` | +| `--remainder` | Everything running that is in neither list | Monthly, via `monthly_maintenance.sh` | + +The update always runs *before* its matching restart, so the restart lands on the freshly +pulled image. Reversing that order would restart onto the old image and leave the new one +sitting unused until the next window. + +Each mode reuses the restart list it pairs with rather than maintaining its own — add a +container to `DAILY_RESTART_CONTAINERS` once and it gets both the restart and the image pull. +Remainder mode needs no list at all: it derives its targets from `docker ps` minus the daily +list, the weekly list, the emby/critical-data sync-window profiles, and the fallback tiers. + +Fallback containers are deliberately excluded from remainder mode. This server only runs them +during a fallback; the remote owns their version. Updating them here would risk the remote's +older image meeting data written by a newer one after a handback. --- @@ -229,7 +245,7 @@ full coordination model between all four watchdogs. |--------|------|-------------| | `docker_daily_restart.sh` | Nightly proactive restart of degradation-prone containers | 1am via `daily_sync_maintenance.sh` | | `docker_weekly_restart.sh` | Weekly restart of less-critical services | 2:30am Sunday via `weekly_sync_maintenance.sh` | -| `docker_update.sh` | Container image updates — daily list + weekly remainder mode | Daily before restart; weekly remainder pass | +| `docker_update.sh` | Container image updates — three modes (default / `--weekly` / `--remainder`) | Daily and weekly before each restart; monthly for the remainder | | `docker_network_connect.sh` | Network existence + container connection enforcement | Every array start | | `docker_container_stop.sh` | Ordered container shutdown — graceful then forced | Called by `array_stopping.sh` | | `downloaders_reset.sh` | Download client hygiene — slskd / SABnzbd / qBittorrent | Every 30min via `critical_sync_maintenance.sh` | @@ -257,8 +273,13 @@ Daily maintenance window (1am): Weekly maintenance window (2:30am Sunday): weekly_sync_maintenance.sh - ├── docker_weekly_restart.sh ──── restart less-critical services - └── docker_update.sh --remainder ─ update containers not in managed lists + ├── docker_update.sh --weekly ─── pull latest (WEEKLY_RESTART_CONTAINERS) + └── docker_weekly_restart.sh ──── restart onto the fresh image + +Monthly maintenance window: + monthly_maintenance.sh + ├── docker_update.sh --remainder update everything not in the managed lists + └── Tools/docker_prune_images.sh --all Critical maintenance window (every 30min): critical_sync_maintenance.sh @@ -266,5 +287,39 @@ Critical maintenance window (every 30min): Array stopping: array_stopping.sh - └── docker_container_stop.sh ──── ordered graceful shutdown + └── docker_container_stop.sh ──── ordered graceful shutdown, verified per container ``` + +--- + +## ━━━ SAFEGUARDS COMMON TO THIS FOLDER ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ + +Every script here talks to the Docker daemon, so they share the same protections. Each +script's own header documents its full set; these are the ones worth knowing folder-wide. + +**Daemon health is checked, not assumed.** A hung daemon returns an empty container list, +which is indistinguishable from "no containers running". Without the check, +`docker_container_stop.sh` would report a clean shutdown that never happened, and +`docker_update.sh --remainder` would report "nothing to update" while doing nothing. + +**Every docker call is timeout-wrapped.** A wedged daemon cannot stall a maintenance window +or hold a lock open. The one deliberate exception is `docker pull` — a large image +legitimately outlasts any sane timeout, and killing it mid-layer wastes the transfer. + +**State is respected.** Running containers get restarted; stopped ones stay stopped. A +stopped container was almost certainly stopped on purpose, and none of these scripts has the +authority to overrule that. + +**Restarts are verified, not assumed.** After each restart the container is re-checked once +it has had time to settle. A container that starts and immediately crashes is recorded as a +failure and notified — a restart that did not stick is never reported as success. + +**Dependency ordering is shared with the watchdog.** Restarts follow +`HOST*_WATCHDOG_DEPENDENCIES`, with `CONTAINER_DELAY` between a dependency and its dependents, +so a dependent is never brought up while what it needs is still initialising. + +**Locks prevent overlap.** Long windows can outlast their interval; +`downloaders_reset.sh` uses wait-mode because it runs every 30 minutes and the previous pass +may still be finishing. + +--- diff --git a/Media/Manual-Media.md b/Media/Manual-Media.md index 8693cbc..704ed9a 100644 --- a/Media/Manual-Media.md +++ b/Media/Manual-Media.md @@ -86,10 +86,25 @@ MEDIA_FILE_PATTERNS=( ### Play State Sync ```bash -PLAY_STATE_SYNC_ENABLED=true # toggle entire sync -PLAY_STATE_SYNC_LOOKBACK_DAYS=30 # history window for played items +PLAY_SYNC_ENABLED=true # toggle entire sync +PLAY_SYNC_REMOTE=true # sync across hosts via Tailscale + # false = local servers only (this host's Emby + Jellyfin) +PLAY_SYNC_TYPES="Movie,Episode" # item types to sync — Audio excluded, music library too large +PLAY_SYNC_FAV_TYPES="MusicArtist,MusicAlbum,Movie,Series" # favourites, union sync, never unmarks + +PLAY_SYNC_PROBE=true # skip per-item work when nothing changed since last run +PLAY_SYNC_PROBE_MAX_AGE_HOURS=24 # force a full comparison when the fingerprint is older + +PLAY_SYNC_HANDBACK_RETRIES=5 # fallback handback: attempts before DNS cutover proceeds +PLAY_SYNC_HANDBACK_RETRY_DELAY=60 # seconds between those attempts ``` +> **There is no date/lookback filter, and one must not be re-added.** An earlier version +> gated on played-date; it was removed once the real cost was measured — the 30-minute +> runtime was fork overhead per item, not API volume or item count. The fix was jq epoch +> parsing plus the response-hash probe below. Re-introducing a date window would reduce +> correctness (older items silently stop syncing) without meaningfully reducing runtime. + Emby and Jellyfin servers configured per-host: ```bash @@ -194,13 +209,24 @@ Show every file examined, not just those removed. ### play_state_sync.sh `play_state_sync.sh` -Sync played/unplayed state and resume positions from local Emby to the remote Emby. -Only items played within PLAY_STATE_SYNC_LOOKBACK_DAYS are synced. +Sync played/unplayed state, resume positions and favourites across **every** configured Emby +and Jellyfin server — not just local→remote. Servers are discovered from every +`HOST*_TRANSCODE_SERVERS` entry, with remote hosts' localhost URLs rewritten to their +Tailscale IP. Newest `LastPlayedDate` wins; state only ever moves forward, never clears. + +No age filter — every matched item is considered on every run. The **change probe** is what +keeps that cheap: the raw API responses are hashed and compared against the fingerprint from +the last successful run, and per-item processing is skipped entirely when nothing moved. +Fetches still happen every run, so nothing can be missed by the probe. `play_state_sync.sh --full` -Ignore PLAY_STATE_SYNC_LOOKBACK_DAYS — sync all played items regardless of age. May be -slow on large libraries. Use after a new Emby install or database restore to rebuild -full play history. +Bypass the change probe and force the full per-item comparison even when the fingerprint +matches. Use after a new Emby install or database restore, or when debugging a sync that +appears to be skipping work it should be doing. + +`play_state_sync.sh --wait` +Wait for an in-progress run instead of exiting. For manual runs that would otherwise be +skipped by the scheduled every-30-minute pass. Used by `fallback.sh` during handback. `play_state_sync.sh --dry-run` Show what would be synced without writing any state. diff --git a/Media/README-Media.md b/Media/README-Media.md index 57ad3a1..6e84011 100644 --- a/Media/README-Media.md +++ b/Media/README-Media.md @@ -100,3 +100,43 @@ syncs between them. | `media_shares_permissions.sh` | Apply `nobody:users` ownership + correct permissions to all media shares | Daily — runs first | | `media_cleaner.sh` | Remove junk files (two profiles: `anime` + `media`) | Daily — runs before arr cleanup | | `play_state_sync.sh` | Sync watched/played state + resume positions across Emby + Jellyfin | Every 30 min | + +--- + +## ━━━ THE ctime INVARIANT — READ BEFORE CHANGING PERMISSIONS ━━━ + +`media_shares_permissions.sh` applies every pass **conditionally** — it touches only entries +whose owner or mode is actually wrong. That is not an optimisation, and it must stay that way. + +`chown` and `chmod` rewrite an inode's ctime **even when the value does not change**. A +blanket pass would therefore restamp every file in the library every night. + +The arr cleanup scripts (`sonarr_cleanup.sh`, `radarr_cleanup.sh`, `lidarr_cleanup.sh`) gate +orphan deletion on ctime. mtime cannot substitute: an import preserves the release's original +timestamp, so mtime says nothing about when a file arrived here. Measured 2026-07-27 — of 400 +files imported that week, **all 400 had mtimes over 7 days old, one of them 9613 days.** + +So: + +``` + blanket chown/chmod → every ctime resets to today + → no file ever appears older than *_ORPHAN_AGE + → orphan collection silently stops + → nothing errors, nothing warns, disk just fills +``` + +**The failure is invisible.** No script fails, no notification fires. The only symptom is +orphans quietly accumulating until a pool fills — which is exactly how the 755 GB / 89%-full +cache pool incident happened. + +Two rules follow, and both are load-bearing: + +1. **`media_shares_permissions.sh` passes stay conditional.** Making any of them unconditional + breaks orphan collection ecosystem-wide. +2. **`Tools/bulk_permissions_repair.sh` is unconditional by design** — it exists to repair + known-wrong paths where correctness beats preserving a clock. That is precisely why it is a + manual, targeted tool and not scheduled. Pointing it at a whole media root pauses orphan + collection there for `*_ORPHAN_AGE` days. + +Both scripts' headers carry this warning too. If you are reading this because you are about to +"simplify" the permissions job, this is the thing that breaks. diff --git a/Watchdogs/README-Watchdogs.md b/Watchdogs/README-Watchdogs.md index 6e38873..4649de3 100644 --- a/Watchdogs/README-Watchdogs.md +++ b/Watchdogs/README-Watchdogs.md @@ -234,3 +234,53 @@ Every 15 minutes — watchdog_orchestrator.sh fires: | `NETWORK_WATCHDOG_NPM_STATE_FILE` | `System/network_watchdog.sh` | itself | NPM proxy strike count | All state files are in `$STATE_DIR` (adapts to storage mode). See `master.conf` for actual variable values. + +--- + +## ━━━ SAFEGUARDS COMMON TO THIS FOLDER ━━━ + +These run unattended every 15 minutes and hold the only irreversible remedy in the +ecosystem, so the guards matter as much as the checks. Each script's header documents its +full set; these are the folder-wide ones. + +**Nothing acts on a single reading.** Every check that can trigger an action is strike-gated. +One bad sample is a reading; N consecutive samples is a condition. Strikes clear themselves +when the condition resolves, so a transient blip never accumulates toward an alarm across +unrelated weeks. + +**Escalation is asymmetric on purpose.** Protective actions happen fast; undoing them happens +slowly. `resource_watchdog.sh` throttles before it pauses and pauses before it stops, then +requires `RW_RECOVER_CYCLES` clear cycles and de-escalates one level at a time. Recovering on +a single good reading would flap under sustained load. + +**Ownership boundaries are enforced, not conventional.** Container health belongs to +`docker_watchdog.sh`; `stability_watchdog.sh` deliberately does not check it, and its Docker +daemon check writes a flag for `docker_watchdog.sh` rather than acting. Two watchdogs +remediating the same subsystem would race. + +**Watchdogs coordinate through state files.** `resource_watchdog.sh` publishes +`mem_shutdown_active` so `docker_watchdog.sh` defers restarts — otherwise one would stop +containers to free RAM while the other restarted them to restore health. The reader applies a +staleness guard so a crashed writer cannot suppress restarts forever. + +**Data safety outranks uptime.** `stability_watchdog.sh` aborts a reboot while a ZFS pool is +unhealthy, parity is running, or the mover is active. Only Tier 1 — conditions the system +cannot recover from and which worsen every cycle — bypasses that. + +**Reboot loops end in shutdown, not more reboots.** No more than +`SYS_WATCHDOG_REBOOT_LIMIT` reboots within `SYS_WATCHDOG_REBOOT_WINDOW_HRS`. On hitting the +limit the host powers off: a fault that survives repeated reboots will not be fixed by more +of them, and a cleanly-down box is better than one cycling endlessly. + +**An aborted reboot restores what it stopped.** An EXIT trap is armed the moment containers +start being stopped for a reboot and disarmed only once the reboot commits. If the script +dies in between, everything it stopped comes back — the failure mode is a running system, +never a host left stripped with no reboot. + +**An unconfigured job list fails loudly.** An empty `WATCHDOG_ORCHESTRATOR_SCRIPTS` or +`SYSTEM_WATCHDOG_SCRIPTS` would otherwise report "0/0 passed" and exit 0 every cycle — +indistinguishable from healthy while nothing is monitored at all. + +**Verify the path, not the process.** `network_watchdog.sh` checks NPM by fetching an external +URL rather than asking whether the container is running. A running container behind broken DNS +or a broken upstream still serves nothing.