Compare commits

..
3 Commits
Author SHA1 Message Date
Gmer4Lfe 7a8fbbe06a Correct stale references across the remaining folder docs
Three renamed or dissolved names were still cited, and three scheduled scripts had no
entry in their folder's table.
2026-08-01 23:07:50 -04:00
Gmer4Lfe 53aa72d38c Correct Fallback tier variable naming and document rsync merge-run
The fallback docs used an older COVERS naming in the wrong conf file, and rsync's merge-run
mode — the only path that carries --delete — was undocumented entirely.
2026-08-01 23:05:30 -04:00
Gmer4Lfe 8a2707ee37 Correct and extend folder docs for Docker Essentials, Watchdogs, Media and Arrs Stack
The docs had drifted from the scripts — a script that no longer exists, three wrong variable
names, a reversed run order, and seven scheduled scripts that were never documented at all.
2026-08-01 22:59:07 -04:00
16 changed files with 458 additions and 67 deletions
+69
View File
@@ -200,6 +200,75 @@ Lidarr configured — no HOST1_LIDARR_URL means nothing runs.
| `playback_aware_lidarr_discovery.sh` | Behavior-driven music discovery — Emby plays → Last.fm similar → Lidarr | Weekly | | `playback_aware_lidarr_discovery.sh` | Behavior-driven music discovery — Emby plays → Last.fm similar → Lidarr | Weekly |
| `playback_aware_radarr_discovery.sh` | Behavior-driven movie discovery — Emby watches → TMDB recommendations → Radarr | Weekly | | `playback_aware_radarr_discovery.sh` | Behavior-driven movie discovery — Emby watches → TMDB recommendations → Radarr | Weekly |
| `playback_aware_sonarr_discovery.sh` | Behavior-driven TV discovery — Emby episodes → TMDB TV recommendations → Sonarr | Weekly | | `playback_aware_sonarr_discovery.sh` | Behavior-driven TV discovery — Emby episodes → TMDB TV recommendations → Sonarr | Weekly |
| `arr_download_orphan_cleaner.sh` | Clear orphaned completed downloads out of the SABnzbd Completed folders | Daily |
| `sonarr_classification_scan.sh` | Detect series sitting in the wrong root (anime / kids / general); `--move` acts | Daily |
| `radarr_classification_scan.sh` | Same for movies, plus junk-metadata detection via `--remove-junk` | Daily |
| `lidarr_duplicate_artist_cleanup.sh` | Remove phantom zero-file duplicate artists; flag real ones for review | Daily |
| `arr_cache_prefill.sh` | Warm the shared tracked-data cache so consumers never read cold | Array start + every 4hr |
| `arr_corruption_scan.sh` | ffprobe every tracked video for corrupt headers; `--remediate` deletes + re-searches | Weekly |
| `arr_full_rescan.sh` | Force a real disk↔database reconciliation on all three arrs | Weekly |
---
## ━━━ THE NEWER LAYERS ━━━
The original folder was "delete what the arrs no longer track". These were added as distinct
failure modes surfaced — each exists because something went wrong that the cleanups could not
have caught.
### 🗑️ Download-Side Orphans — `arr_download_orphan_cleaner.sh`
Every cleanup script here walks the **library** side. Nothing walked the **download** side —
so completed downloads the arrs had stopped tracking accumulated in SABnzbd's Completed
folders indefinitely. Discovered as **755 GB of orphaned TV downloads, oldest from 2022**,
filling the cache pool to 89%.
Classifies every entry as TRACKED / RECENT / JUNK / REDUNDANT / IMPORTABLE / UNMATCHED and
acts only on the ones it can justify. The queue is a hard gate: if it cannot be read, the arr
is skipped entirely, because without it there is no way to tell an active import from an
orphan. A run total over `DOWNLOAD_ORPHAN_MAX_DELETE_GB` aborts — an abnormally large delete
is the visible symptom of a partial queue fetch.
### 🎭 Wrong-Root Detection — `sonarr_classification_scan.sh` + `radarr_classification_scan.sh`
Overseerr lets any user request content into the wrong root folder — kids shows into general
TV, anime into Kids_Tv_Shows. These classify every item from metadata alone (genre,
certification, network/studio, original language) and report where the computed classification
disagrees with the folder the item actually sits in.
Report-only by default. `--move` acts on forward misplacements and adult-content-in-kids-root
leaks. It deliberately does **not** move non-anime content out of the anime root — deliberate
style placements (Western animation grouped with anime by choice) are genuine judgment calls.
Both poll the arr's async move command to completion before verifying, because `moveFiles=true`
flips the database instantly while the physical move is still queued behind others.
### 🎨 Phantom Artists — `lidarr_duplicate_artist_cleanup.sh`
MusicBrainz duplicates leave two Lidarr entries for one artist, one holding the files and one
holding nothing. Removes only the zero-file side, with `deleteFiles=false` so nothing on disk
is touched. Pairs where both sides hold files are flagged for review, never auto-resolved.
Gated on the tracked-count floor shared with `lidarr_cleanup.sh` — during a library-wide desync
both sides of a real duplicate can read as zero-file phantoms.
### 🩺 Corruption + Reconciliation — `arr_corruption_scan.sh` + `arr_full_rescan.sh`
`arr_corruption_scan.sh` ffprobes tracked video files for corrupt headers. Report-only unless
`--remediate`, which deletes the file record and triggers an explicit re-search. Requires
repeat detections across separate runs before acting, so a transient probe failure cannot
delete a healthy file.
`arr_full_rescan.sh` forces a genuine disk↔database reconciliation. Organic scans only touch
files involved in an import, so an untouched library silently drifts — confirmed when Lidarr
reported **~23% of its true track count** for 1,004 of 1,357 artists with no scan running and
every file present on disk.
### ⚡ Cache Warmth — `arr_cache_prefill.sh`
Populates the shared tracked-data cache at array start and every 4 hours, so consumers never
pay a cold fetch. Pure enhancement: nothing depends on it having run, and every consumer still
writes through on a cold cache.
--- ---
+75 -14
View File
@@ -94,20 +94,54 @@ HOST1_NETWORK_CONNECT_CONTAINERS=(
```bash ```bash
# master.conf # master.conf
DAILY_CONTAINER_UPDATES=true # enable/disable daily image pull DAILY_CONTAINER_UPDATES=true # enable/disable the daily image pull
# docker_daily_restart.sh still runs regardless # docker_daily_restart.sh still runs regardless
# update and restart are independent # update and restart are independent
WEEKLY_REMAINING_UPDATES=true # enable/disable weekly remainder pull + prune WEEKLY_CONTAINER_UPDATES=true # enable/disable the weekly image pull
# to disable: set false or remove from WEEKLY_MAINTENANCE_SCRIPTS # docker_weekly_restart.sh still runs regardless
MONTHLY_REMAINING_UPDATES=true # enable/disable the monthly remainder pull
# to disable: set false, or remove
# "docker_update.sh --remainder" from
# MONTHLY_MAINTENANCE_SCRIPTS
``` ```
`docker_update.sh` in normal mode targets `DAILY_RESTART_CONTAINERS` — the same list There is one update script, `docker_update.sh`, with three modes. Each reuses the restart
used by `docker_daily_restart.sh`. No second list to maintain. list it pairs with, so there is no second list to maintain:
`docker_update_remaining.sh` derives its target list automatically: | Mode | Targets | Runs |
all running containers minus `DAILY_RESTART_CONTAINERS` minus `WEEKLY_RESTART_CONTAINERS`. |------|---------|------|
Everything gets updated at least once per week with no explicit configuration. | *(default)* | `DAILY_RESTART_CONTAINERS` | Daily, before `docker_daily_restart.sh` |
| `--weekly` | `WEEKLY_RESTART_CONTAINERS` | Weekly, before `docker_weekly_restart.sh` |
| `--remainder` | derived, see below | Monthly, via `MONTHLY_MAINTENANCE_SCRIPTS` |
Remainder mode needs no configuration at all. It takes every **running** container and
subtracts:
```
DAILY_RESTART_CONTAINERS already updated daily
WEEKLY_RESTART_CONTAINERS already updated weekly
PROFILE_CRITICAL_CONTAINER_NAMES[emby] updated inline by the weekly sync window
PROFILE_CRITICAL_CONTAINER_NAMES[critical-data] updated inline by the weekly sync window
FALLBACK_<REMOTE>_TIER1..4 owned by the remote's update cycle
```
Stopped containers are never targeted in any mode — pulling for a stopped container adds
nothing, and it was most likely stopped deliberately.
The fallback exclusion is a correctness rule, not an optimisation. This server only runs
those containers during a fallback; the remote owns their version. If remainder updated them
independently and a handback then occurred, the remote's older image could meet data written
by the newer one.
**Ordering matters.** The update always runs *before* its matching restart so the restart
lands on the freshly pulled image. If a container's image actually changed, `docker_update.sh`
rebuilds it from its template (a plain `docker restart` reuses the image ID baked in at
creation time and would never pick up the new digest) and records it in
`DOCKER_UPDATE_REBUILT_*_FILE`. The restart script reads that file and skips those containers
rather than restarting them a second time — and discards the file as stale if it is older
than `DOCKER_UPDATE_REBUILT_STALE_HOURS`.
--- ---
@@ -142,12 +176,22 @@ HOST1_NETWORK_CONNECT_CONTAINERS=()
# Shared — applies to both servers # Shared — applies to both servers
# ── Container updates ────────────────────────────────────────────────── # ── Container updates ──────────────────────────────────────────────────
DAILY_CONTAINER_UPDATES=true DAILY_CONTAINER_UPDATES=true # daily pull (DAILY_RESTART_CONTAINERS)
WEEKLY_REMAINING_UPDATES=true WEEKLY_CONTAINER_UPDATES=true # weekly pull (WEEKLY_RESTART_CONTAINERS)
MONTHLY_REMAINING_UPDATES=true # monthly pull (everything else)
# Handoff between update and restart — written by docker_update.sh,
# read by the matching restart script so it skips containers already
# rebuilt onto a new image this run.
DOCKER_UPDATE_REBUILT_DAILY_FILE
DOCKER_UPDATE_REBUILT_WEEKLY_FILE
DOCKER_UPDATE_REBUILT_STALE_HOURS=12 # older than this = discarded, restart all
# ── Retry behaviour (shared by restart scripts) ──────────────────────── # ── Retry behaviour (shared by restart scripts) ────────────────────────
RETRY_COUNT=3 # retry attempts before marking failed RETRY_COUNT=3 # retry attempts before marking failed
SLEEP=5 # seconds between retry attempts SLEEP=5 # seconds between retry attempts
CONTAINER_DELAY # seconds between a dependency and its dependents
RESTART_VERIFY_WAIT=3 # settle time before verifying a restart stuck
# Watchdog thresholds (CPU, memory, HTTP, restart loop): # Watchdog thresholds (CPU, memory, HTTP, restart loop):
# → see Watchdogs/Manual-Watchdogs.md # → see Watchdogs/Manual-Watchdogs.md
@@ -211,7 +255,24 @@ All scripts support these standard flags:
summary block: identity, duration, counts, and a status line. Per-container detail only summary block: identity, duration, counts, and a status line. Per-container detail only
appears with `--log`. Warnings and errors are always visible regardless of `--log`. appears with `--log`. Warnings and errors are always visible regardless of `--log`.
### `docker_update.sh --remainder` ### Script-specific flags
Switches to remainder mode — updates all running containers not in the managed daily/weekly
lists. Called by `weekly_sync_maintenance.sh`. Can be run manually to sweep containers | Script | Flag | What it does |
that haven't been updated recently. |--------|------|-------------|
| `docker_update.sh` | `--weekly` | Target `WEEKLY_RESTART_CONTAINERS`. Called by `weekly_sync_maintenance.sh` before the weekly restart. |
| `docker_update.sh` | `--remainder` | Target every running container not in the daily list, weekly list, emby/critical-data profiles, or fallback tiers. Called by `monthly_maintenance.sh`. Safe to run manually to sweep anything missed. |
| `media_cleaner.sh` *(Media/)* | `<anime\|media>` | Required positional profile — there is no default. |
`docker_update.sh` with no mode flag is normal mode: `DAILY_RESTART_CONTAINERS`, called by
`daily_sync_maintenance.sh` before the daily restart.
### Exit codes
| Code | Meaning |
|------|---------|
| `0` | Success, or nothing to do (empty list, disabled toggle, no running containers) |
| `1` | One or more containers failed — the summary names them |
`docker_update.sh` deliberately exits `0` even when pulls fail: a failed pull is not a reason
to abort the restart that follows, which simply proceeds on the existing image. The failure
is reported in the summary.
+64 -9
View File
@@ -178,12 +178,28 @@ windows so any downtime from restarts is absorbed by the window that's already h
--- ---
### 🔄 Image Currency — `docker_update.sh` + `docker_update_remaining.sh` ### 🔄 Image Currency — `docker_update.sh`
Keeps all container images current without manual intervention. Daily updates for the One script, three modes — there is no separate remainder script.
auth/proxy stack (the containers that restart daily anyway — no extra downtime). Weekly
remainder pass for everything else — derives the target list automatically from `docker ps` | Mode | Targets | Runs |
minus what was already updated, so there is no second list to maintain. |------|---------|------|
| *(default)* | `DAILY_RESTART_CONTAINERS` | Daily, **before** `docker_daily_restart.sh` |
| `--weekly` | `WEEKLY_RESTART_CONTAINERS` | Weekly, **before** `docker_weekly_restart.sh` |
| `--remainder` | Everything running that is in neither list | Monthly, via `monthly_maintenance.sh` |
The update always runs *before* its matching restart, so the restart lands on the freshly
pulled image. Reversing that order would restart onto the old image and leave the new one
sitting unused until the next window.
Each mode reuses the restart list it pairs with rather than maintaining its own — add a
container to `DAILY_RESTART_CONTAINERS` once and it gets both the restart and the image pull.
Remainder mode needs no list at all: it derives its targets from `docker ps` minus the daily
list, the weekly list, the emby/critical-data sync-window profiles, and the fallback tiers.
Fallback containers are deliberately excluded from remainder mode. This server only runs them
during a fallback; the remote owns their version. Updating them here would risk the remote's
older image meeting data written by a newer one after a handback.
--- ---
@@ -229,7 +245,7 @@ full coordination model between all four watchdogs.
|--------|------|-------------| |--------|------|-------------|
| `docker_daily_restart.sh` | Nightly proactive restart of degradation-prone containers | 1am via `daily_sync_maintenance.sh` | | `docker_daily_restart.sh` | Nightly proactive restart of degradation-prone containers | 1am via `daily_sync_maintenance.sh` |
| `docker_weekly_restart.sh` | Weekly restart of less-critical services | 2:30am Sunday via `weekly_sync_maintenance.sh` | | `docker_weekly_restart.sh` | Weekly restart of less-critical services | 2:30am Sunday via `weekly_sync_maintenance.sh` |
| `docker_update.sh` | Container image updates — daily list + weekly remainder mode | Daily before restart; weekly remainder pass | | `docker_update.sh` | Container image updates — three modes (default / `--weekly` / `--remainder`) | Daily and weekly before each restart; monthly for the remainder |
| `docker_network_connect.sh` | Network existence + container connection enforcement | Every array start | | `docker_network_connect.sh` | Network existence + container connection enforcement | Every array start |
| `docker_container_stop.sh` | Ordered container shutdown — graceful then forced | Called by `array_stopping.sh` | | `docker_container_stop.sh` | Ordered container shutdown — graceful then forced | Called by `array_stopping.sh` |
| `downloaders_reset.sh` | Download client hygiene — slskd / SABnzbd / qBittorrent | Every 30min via `critical_sync_maintenance.sh` | | `downloaders_reset.sh` | Download client hygiene — slskd / SABnzbd / qBittorrent | Every 30min via `critical_sync_maintenance.sh` |
@@ -257,8 +273,13 @@ Daily maintenance window (1am):
Weekly maintenance window (2:30am Sunday): Weekly maintenance window (2:30am Sunday):
weekly_sync_maintenance.sh weekly_sync_maintenance.sh
├── docker_weekly_restart.sh ──── restart less-critical services ├── docker_update.sh --weekly ─── pull latest (WEEKLY_RESTART_CONTAINERS)
└── docker_update.sh --remainder ─ update containers not in managed lists └── docker_weekly_restart.sh ──── restart onto the fresh image
Monthly maintenance window:
monthly_maintenance.sh
├── docker_update.sh --remainder update everything not in the managed lists
└── Tools/docker_prune_images.sh --all
Critical maintenance window (every 30min): Critical maintenance window (every 30min):
critical_sync_maintenance.sh critical_sync_maintenance.sh
@@ -266,5 +287,39 @@ Critical maintenance window (every 30min):
Array stopping: Array stopping:
array_stopping.sh array_stopping.sh
└── docker_container_stop.sh ──── ordered graceful shutdown └── docker_container_stop.sh ──── ordered graceful shutdown, verified per container
``` ```
---
## ━━━ SAFEGUARDS COMMON TO THIS FOLDER ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
Every script here talks to the Docker daemon, so they share the same protections. Each
script's own header documents its full set; these are the ones worth knowing folder-wide.
**Daemon health is checked, not assumed.** A hung daemon returns an empty container list,
which is indistinguishable from "no containers running". Without the check,
`docker_container_stop.sh` would report a clean shutdown that never happened, and
`docker_update.sh --remainder` would report "nothing to update" while doing nothing.
**Every docker call is timeout-wrapped.** A wedged daemon cannot stall a maintenance window
or hold a lock open. The one deliberate exception is `docker pull` — a large image
legitimately outlasts any sane timeout, and killing it mid-layer wastes the transfer.
**State is respected.** Running containers get restarted; stopped ones stay stopped. A
stopped container was almost certainly stopped on purpose, and none of these scripts has the
authority to overrule that.
**Restarts are verified, not assumed.** After each restart the container is re-checked once
it has had time to settle. A container that starts and immediately crashes is recorded as a
failure and notified — a restart that did not stick is never reported as success.
**Dependency ordering is shared with the watchdog.** Restarts follow
`HOST*_WATCHDOG_DEPENDENCIES`, with `CONTAINER_DELAY` between a dependency and its dependents,
so a dependent is never brought up while what it needs is still initialising.
**Locks prevent overlap.** Long windows can outlast their interval;
`downloaders_reset.sh` uses wait-mode because it runs every 30 minutes and the previous pass
may still be finishing.
---
+42 -24
View File
@@ -105,18 +105,36 @@ internet connectivity. Most hosts leave this empty.
--- ---
``` ```
FALLBACK_HOST*_COVERS_HOST*_TIER1=(...) FALLBACK_HOST*_TIER1=(...)
FALLBACK_HOST*_COVERS_HOST*_TIER2=(...) FALLBACK_HOST*_TIER2=(...)
FALLBACK_HOST*_COVERS_HOST*_TIER3=(...) FALLBACK_HOST*_TIER3=(...)
FALLBACK_HOST*_COVERS_HOST*_TIER4=(...) FALLBACK_HOST*_TIER4=(...)
``` ```
Containers this host starts for the remote host when the remote is down. TIER1 starts
immediately. TIER24 activate after the corresponding delay thresholds.
Variable pattern: `FALLBACK_${MY_ID}_COVERS_${REMOTE_ID}_TIER${N}` **Each host declares its OWN services, in its OWN conf.** `FALLBACK_HOST2_TIER1` lives in
`host2.conf` and lists HOST2's vital containers — it is not a list HOST1 maintains.
The DDNS container for the remote's domain must be the first entry in TIER1 — DNS At runtime `fallback.sh` reads its **partner's** list:
coverage before anything else.
```bash
get_tier_containers() {
local var_name="FALLBACK_${REMOTE_ID}_TIER${tier}" # note: REMOTE_ID, not MY_ID
```
So HOST1, covering HOST2, reads `FALLBACK_HOST2_TIER1` — a variable defined in `host2.conf`
and delivered to HOST1 through the partner conf cache (`conf_sync.sh`), because sparse
checkout means HOST1 never pulls `host2.conf` from git.
This is why the naming is what it is. The alternative — each host keeping a copy of its
partner's container list — would need editing on both machines every time either one changed
a service, and the two copies would silently diverge. Declaring once, on the host that owns
the services, means a host is always the authority on what covering it requires.
TIER1 starts immediately. TIER24 activate after their delay thresholds
(`HOST*_TIER2_DELAY` and friends, also in that host's own conf).
The DDNS container for that host's domain must be the **first entry in TIER1** — DNS coverage
before anything else.
--- ---
@@ -181,15 +199,15 @@ FALLBACK_TEST_HANDBACK_WAIT=300
--- ---
### host2.conf — HOST2 covering HOST1 ### host1.conf — HOST1's own services (started by HOST2 when HOST1 is down)
```bash ```bash
HOST2_DDNS_CONTAINERS=("Gmer4Lfe.us-DDNS") HOST1_DDNS_CONTAINERS=("Gmer4Lfe.com-DDNS")
FALLBACK_HOST2_STOP_ON_NO_NET=() FALLBACK_HOST1_STOP_ON_NO_NET=()
# Tier 1 — immediate (vital services + Live TV) # Tier 1 — immediate (vital services + Live TV)
FALLBACK_HOST2_COVERS_HOST1_TIER1=( FALLBACK_HOST1_TIER1=(
"Gmer4Lfe.com-DDNS" # ALWAYS FIRST — DNS coverage before anything else "Gmer4Lfe.com-DDNS" # ALWAYS FIRST — DNS coverage before anything else
"Emby" # media server — people are watching "Emby" # media server — people are watching
"NginxProxyManager" # reverse proxy — all external access routes through this "NginxProxyManager" # reverse proxy — all external access routes through this
@@ -205,7 +223,7 @@ FALLBACK_HOST2_COVERS_HOST1_TIER1=(
) )
# Tier 2 — after 4 hours (shared productivity services) # Tier 2 — after 4 hours (shared productivity services)
FALLBACK_HOST2_COVERS_HOST1_TIER2=( FALLBACK_HOST1_TIER2=(
"Postgres-NextCloud" # must start before NextCloud "Postgres-NextCloud" # must start before NextCloud
"NextCloud" "NextCloud"
"PostgreSQL-Immich" # must start before Immich "PostgreSQL-Immich" # must start before Immich
@@ -214,7 +232,7 @@ FALLBACK_HOST2_COVERS_HOST1_TIER2=(
) )
# Tier 3 — after 12 hours (secondary services) # Tier 3 — after 12 hours (secondary services)
FALLBACK_HOST2_COVERS_HOST1_TIER3=( FALLBACK_HOST1_TIER3=(
"Organizrv2-Gmer4Lfe" "Organizrv2-Gmer4Lfe"
"AdGuard-Home" "AdGuard-Home"
"UptimeKuma" "UptimeKuma"
@@ -223,7 +241,7 @@ FALLBACK_HOST2_COVERS_HOST1_TIER3=(
) )
# Tier 4 — after 24 hours (arrs + downloaders) # Tier 4 — after 24 hours (arrs + downloaders)
FALLBACK_HOST2_COVERS_HOST1_TIER4=( FALLBACK_HOST1_TIER4=(
"Sonarr-Gmer4Lfe" "Sonarr-Gmer4Lfe"
"Radarr-Gmer4Lfe" "Radarr-Gmer4Lfe"
"Lidarr-Gmer4Lfe" "Lidarr-Gmer4Lfe"
@@ -259,22 +277,22 @@ FALLBACK_HOST1_WRITEBACK_TIER3=(
--- ---
### host1.conf — HOST1 covering HOST2 ### host2.conf — HOST2's own services (started by HOST1 when HOST2 is down)
```bash ```bash
HOST1_DDNS_CONTAINERS=("Gmer4Lfe.com-DDNS") HOST2_DDNS_CONTAINERS=("Gmer4Lfe.us-DDNS")
FALLBACK_HOST1_STOP_ON_NO_NET=() FALLBACK_HOST2_STOP_ON_NO_NET=()
# Tier 1 — immediate (HOST2's vital services) # Tier 1 — immediate (HOST2's vital services)
FALLBACK_HOST1_COVERS_HOST2_TIER1=( FALLBACK_HOST2_TIER1=(
"Gmer4Lfe.us-DDNS" # ALWAYS FIRST "Gmer4Lfe.us-DDNS" # ALWAYS FIRST
# HOST2's Tier 1 services — fill per HOST2's stack # HOST2's Tier 1 services — fill per HOST2's stack
) )
FALLBACK_HOST1_COVERS_HOST2_TIER2=(...) FALLBACK_HOST2_TIER2=(...)
FALLBACK_HOST1_COVERS_HOST2_TIER3=(...) FALLBACK_HOST2_TIER3=(...)
FALLBACK_HOST1_COVERS_HOST2_TIER4=(...) FALLBACK_HOST2_TIER4=(...)
# Tier delays for HOST2 outage # Tier delays for HOST2 outage
HOST2_TIER2_DELAY=240 HOST2_TIER2_DELAY=240
@@ -492,7 +510,7 @@ Restart fallback.sh via User Scripts plugin. It will resume from NORMAL on its n
1. Create the container on the covering server (stopped), with volume mounts pointing at 1. Create the container on the covering server (stopped), with volume mounts pointing at
the mirrored share path (e.g. `/mnt/user/Movies` must exist on the covering server) the mirrored share path (e.g. `/mnt/user/Movies` must exist on the covering server)
2. Add the container name to `FALLBACK_HOST*_COVERS_HOST*_TIER*` in host*.conf 2. Add the container name to `FALLBACK_<THAT-HOST>_TIER*` in **that host's own** conf
in the appropriate tier position (dependency ordering — databases before apps) in the appropriate tier position (dependency ordering — databases before apps)
3. Verify: `fallback.sh --status` shows the container in the expected tier list 3. Verify: `fallback.sh --status` shows the container in the expected tier list
4. Run `fallback_test.sh --dry-run` to confirm the full configuration is valid 4. Run `fallback_test.sh --dry-run` to confirm the full configuration is valid
+3 -3
View File
@@ -115,7 +115,7 @@ Both servers run `fallback.sh` independently as a continuous background process.
makes all decisions from two pings every `FALLBACK_CHECK_INTERVAL` seconds: makes all decisions from two pings every `FALLBACK_CHECK_INTERVAL` seconds:
```bash ```bash
ping REMOTE_TAILSCALE_IP # is the other server reachable? ping "$(resolve_tailscale_ip "$REMOTE_SERVER_NAME")" # is the other server reachable?
ping EXTERNAL_IP # do I have internet? (default: 8.8.8.8) ping EXTERNAL_IP # do I have internet? (default: 8.8.8.8)
``` ```
@@ -220,8 +220,8 @@ determines which server is local and which is remote at runtime, then selects th
container arrays and tier delays from config via MY_ID. container arrays and tier delays from config via MY_ID.
``` ```
HOST2 covers HOST1: FALLBACK_HOST2_COVERS_HOST1_TIER* (in host2.conf) HOST2 covers HOST1: FALLBACK_HOST1_TIER* (in host2.conf)
HOST1 covers HOST2: FALLBACK_HOST1_COVERS_HOST2_TIER* (in host1.conf) HOST1 covers HOST2: FALLBACK_HOST2_TIER* (in host1.conf)
``` ```
Both servers run identical scripts. MY_ID selects the correct arrays. No hostname Both servers run identical scripts. MY_ID selects the correct arrays. No hostname
+33 -7
View File
@@ -86,10 +86,25 @@ MEDIA_FILE_PATTERNS=(
### Play State Sync ### Play State Sync
```bash ```bash
PLAY_STATE_SYNC_ENABLED=true # toggle entire sync PLAY_SYNC_ENABLED=true # toggle entire sync
PLAY_STATE_SYNC_LOOKBACK_DAYS=30 # history window for played items PLAY_SYNC_REMOTE=true # sync across hosts via Tailscale
# false = local servers only (this host's Emby + Jellyfin)
PLAY_SYNC_TYPES="Movie,Episode" # item types to sync — Audio excluded, music library too large
PLAY_SYNC_FAV_TYPES="MusicArtist,MusicAlbum,Movie,Series" # favourites, union sync, never unmarks
PLAY_SYNC_PROBE=true # skip per-item work when nothing changed since last run
PLAY_SYNC_PROBE_MAX_AGE_HOURS=24 # force a full comparison when the fingerprint is older
PLAY_SYNC_HANDBACK_RETRIES=5 # fallback handback: attempts before DNS cutover proceeds
PLAY_SYNC_HANDBACK_RETRY_DELAY=60 # seconds between those attempts
``` ```
> **There is no date/lookback filter, and one must not be re-added.** An earlier version
> gated on played-date; it was removed once the real cost was measured — the 30-minute
> runtime was fork overhead per item, not API volume or item count. The fix was jq epoch
> parsing plus the response-hash probe below. Re-introducing a date window would reduce
> correctness (older items silently stop syncing) without meaningfully reducing runtime.
Emby and Jellyfin servers configured per-host: Emby and Jellyfin servers configured per-host:
```bash ```bash
@@ -194,13 +209,24 @@ Show every file examined, not just those removed.
### play_state_sync.sh ### play_state_sync.sh
`play_state_sync.sh` `play_state_sync.sh`
Sync played/unplayed state and resume positions from local Emby to the remote Emby. Sync played/unplayed state, resume positions and favourites across **every** configured Emby
Only items played within PLAY_STATE_SYNC_LOOKBACK_DAYS are synced. and Jellyfin server — not just local→remote. Servers are discovered from every
`HOST*_TRANSCODE_SERVERS` entry, with remote hosts' localhost URLs rewritten to their
Tailscale IP. Newest `LastPlayedDate` wins; state only ever moves forward, never clears.
No age filter — every matched item is considered on every run. The **change probe** is what
keeps that cheap: the raw API responses are hashed and compared against the fingerprint from
the last successful run, and per-item processing is skipped entirely when nothing moved.
Fetches still happen every run, so nothing can be missed by the probe.
`play_state_sync.sh --full` `play_state_sync.sh --full`
Ignore PLAY_STATE_SYNC_LOOKBACK_DAYS — sync all played items regardless of age. May be Bypass the change probe and force the full per-item comparison even when the fingerprint
slow on large libraries. Use after a new Emby install or database restore to rebuild matches. Use after a new Emby install or database restore, or when debugging a sync that
full play history. appears to be skipping work it should be doing.
`play_state_sync.sh --wait`
Wait for an in-progress run instead of exiting. For manual runs that would otherwise be
skipped by the scheduled every-30-minute pass. Used by `fallback.sh` during handback.
`play_state_sync.sh --dry-run` `play_state_sync.sh --dry-run`
Show what would be synced without writing any state. Show what would be synced without writing any state.
+40
View File
@@ -100,3 +100,43 @@ syncs between them.
| `media_shares_permissions.sh` | Apply `nobody:users` ownership + correct permissions to all media shares | Daily — runs first | | `media_shares_permissions.sh` | Apply `nobody:users` ownership + correct permissions to all media shares | Daily — runs first |
| `media_cleaner.sh` | Remove junk files (two profiles: `anime` + `media`) | Daily — runs before arr cleanup | | `media_cleaner.sh` | Remove junk files (two profiles: `anime` + `media`) | Daily — runs before arr cleanup |
| `play_state_sync.sh` | Sync watched/played state + resume positions across Emby + Jellyfin | Every 30 min | | `play_state_sync.sh` | Sync watched/played state + resume positions across Emby + Jellyfin | Every 30 min |
---
## ━━━ THE ctime INVARIANT — READ BEFORE CHANGING PERMISSIONS ━━━
`media_shares_permissions.sh` applies every pass **conditionally** — it touches only entries
whose owner or mode is actually wrong. That is not an optimisation, and it must stay that way.
`chown` and `chmod` rewrite an inode's ctime **even when the value does not change**. A
blanket pass would therefore restamp every file in the library every night.
The arr cleanup scripts (`sonarr_cleanup.sh`, `radarr_cleanup.sh`, `lidarr_cleanup.sh`) gate
orphan deletion on ctime. mtime cannot substitute: an import preserves the release's original
timestamp, so mtime says nothing about when a file arrived here. Measured 2026-07-27 — of 400
files imported that week, **all 400 had mtimes over 7 days old, one of them 9613 days.**
So:
```
blanket chown/chmod → every ctime resets to today
→ no file ever appears older than *_ORPHAN_AGE
→ orphan collection silently stops
→ nothing errors, nothing warns, disk just fills
```
**The failure is invisible.** No script fails, no notification fires. The only symptom is
orphans quietly accumulating until a pool fills — which is exactly how the 755 GB / 89%-full
cache pool incident happened.
Two rules follow, and both are load-bearing:
1. **`media_shares_permissions.sh` passes stay conditional.** Making any of them unconditional
breaks orphan collection ecosystem-wide.
2. **`Tools/bulk_permissions_repair.sh` is unconditional by design** — it exists to repair
known-wrong paths where correctness beats preserving a clock. That is precisely why it is a
manual, targeted tool and not scheduled. Pointing it at a whole media root pauses orphan
collection there for `*_ORPHAN_AGE` days.
Both scripts' headers carry this warning too. If you are reading this because you are about to
"simplify" the permissions job, this is the thing that breaks.
+1 -1
View File
@@ -230,7 +230,7 @@ data — it just doesn't trigger a notification for that condition.
| State File | Source | What It Shows | | State File | Source | What It Shows |
|-----------|--------|---------------| |-----------|--------|---------------|
| `FALLBACK_STATE_FILE` | `Fallback/fallback.sh` | Current fallback state (NORMAL/FALLBACK/etc.) | | `FALLBACK_STATE_FILE` | `Fallback/fallback.sh` | Current fallback state (NORMAL/FALLBACK/etc.) |
| `SYS_WATCHDOG_FAILED_FILE` | `Watchdogs/docker_watchdog.sh` | Container skip list — needs human attention | | `DOCKER_WATCHDOG_FAILED_FILE` | `Watchdogs/docker_watchdog.sh` | Container skip list — needs human attention |
| `WATCHDOG_STATE_FILE` | `Watchdogs/docker_watchdog.sh` | Active container strike counts | | `WATCHDOG_STATE_FILE` | `Watchdogs/docker_watchdog.sh` | Active container strike counts |
| `SYS_WATCHDOG_STATE_FILE` | `Watchdogs/stability_watchdog.sh` | Active system watchdog strikes | | `SYS_WATCHDOG_STATE_FILE` | `Watchdogs/stability_watchdog.sh` | Active system watchdog strikes |
| `BANDWIDTH_LOG` | `bandwidth_monitor.sh` | Yesterday's transfer history | | `BANDWIDTH_LOG` | `bandwidth_monitor.sh` | Yesterday's transfer history |
+1 -1
View File
@@ -153,7 +153,7 @@ Daily 8am:
weekly_health_digest.sh ── reads ──────────► FALLBACK_STATE_FILE weekly_health_digest.sh ── reads ──────────► FALLBACK_STATE_FILE
── reads ──────────► WATCHDOG_STATE_FILE ── reads ──────────► WATCHDOG_STATE_FILE
── reads ──────────► SYS_WATCHDOG_STATE_FILE ── reads ──────────► SYS_WATCHDOG_STATE_FILE
── reads ──────────► SYS_WATCHDOG_FAILED_FILE ── reads ──────────► DOCKER_WATCHDOG_FAILED_FILE
── reads ──────────► BANDWIDTH_LOG ── reads ──────────► BANDWIDTH_LOG
── reads ──────────► TRANSCODE_DAILY_LOG ── reads ──────────► TRANSCODE_DAILY_LOG
── reads ──────────► TUNING_MONITOR_LOG ── reads ──────────► TUNING_MONITOR_LOG
+3 -3
View File
@@ -538,7 +538,7 @@ Runs on both servers; `detect_hosts()` determines which direction each sync goes
# Drive temperature exit codes respected — skip share or abort all on CRIT. # Drive temperature exit codes respected — skip share or abort all on CRIT.
# #
# 3. Post-sync: DAILY_MAINTENANCE_SCRIPTS (everything except git pull) # 3. Post-sync: DAILY_MAINTENANCE_SCRIPTS (everything except git pull)
# media_management.sh → permissions + cleaners + arr cleanup # Media/ + Arrs_Stack/ scripts → permissions, cleaners, arr cleanup, classification
# docker_daily_restart.sh → nightly container restarts # docker_daily_restart.sh → nightly container restarts
# ───────────────────────────────────────────────────────────────────────────── # ─────────────────────────────────────────────────────────────────────────────
``` ```
@@ -557,12 +557,12 @@ Runs on both servers; `detect_hosts()` determines which direction each sync goes
# #
# HOST1 runs this script at 1am: # HOST1 runs this script at 1am:
# → pushes HOST1_DAILY_SYNC_SHARES (Movies, Tv_Shows, Music) → HOST2 # → pushes HOST1_DAILY_SYNC_SHARES (Movies, Tv_Shows, Music) → HOST2
# → media_management.sh on HOST1's shares # → Media/ + Arrs_Stack/ scripts on HOST1's shares
# → docker_daily_restart.sh on HOST1's containers # → docker_daily_restart.sh on HOST1's containers
# #
# HOST2 runs this script at 1am: # HOST2 runs this script at 1am:
# → pushes HOST2_DAILY_SYNC_SHARES (Anime_Shows, Anime_Movies) → HOST1 # → pushes HOST2_DAILY_SYNC_SHARES (Anime_Shows, Anime_Movies) → HOST1
# → media_management.sh on HOST2's shares # → Media/ + Arrs_Stack/ scripts on HOST2's shares
# → docker_daily_restart.sh on HOST2's containers # → docker_daily_restart.sh on HOST2's containers
# ───────────────────────────────────────────────────────────────────────────── # ─────────────────────────────────────────────────────────────────────────────
``` ```
+2
View File
@@ -144,6 +144,8 @@ independently at 4-hour cadence.
| `partnership_transfer.sh` | Transfer ownership from current owner to current mirror | Via `partnership_manager.sh --transfer`; owner only | | `partnership_transfer.sh` | Transfer ownership from current owner to current mirror | Via `partnership_manager.sh --transfer`; owner only |
| `ssh_setup.sh` | SSH key generation, remote install, auth validation | Called by onboard; manually for re-keying or validation | | `ssh_setup.sh` | SSH key generation, remote install, auth validation | Called by onboard; manually for re-keying or validation |
| `onboard_cancel.sh` | Remove SSH keys in one or both directions, clear setup flags | During cancelled or failed onboard; manual cleanup | | `onboard_cancel.sh` | Remove SSH keys in one or both directions, clear setup flags | During cancelled or failed onboard; manual cleanup |
| `gitea_ssh_setup.sh` | Generate a Gitea keypair and register it via the Gitea API | Called during onboard; manually when re-keying a server |
| `share_setup.sh` | Create missing Unraid shares on the mirror from the owner's sync lists | Called during onboard; safe to re-run — existing shares are never modified |
--- ---
+70 -1
View File
@@ -607,7 +607,7 @@ DEFAULT_RSYNC_OPTS="-az --no-perms --no-owner --no-group --inplace"
BW_LIMIT=0 # KB/s, 0 = unlimited BW_LIMIT=0 # KB/s, 0 = unlimited
RETRY_COUNT=3 RETRY_COUNT=3
SLEEP=60 # seconds between retries SLEEP=60 # seconds between retries
ROOTFS_WARN_PCT=75 # abort if remote rootfs above this % ROOTFS_WARN=75 # abort if remote rootfs above this %
# Shared with Monitors/ # Shared with Monitors/
BANDWIDTH_LOG="$DATA_DIR/bandwidth_history.db" BANDWIDTH_LOG="$DATA_DIR/bandwidth_history.db"
@@ -696,3 +696,72 @@ name doesn't match any profile key, or when testing a specific profile.
rsync.sh /mnt/user/appdata-Fallback/Critical-Data --profile=critical-data rsync.sh /mnt/user/appdata-Fallback/Critical-Data --profile=critical-data
rsync.sh /mnt/user/Media_Server/Emby --profile=emby rsync.sh /mnt/user/Media_Server/Emby --profile=emby
``` ```
### --seed
Skip the empty-remote-share guard. Normally `check_remote_share()` aborts when the target
directory on the remote is missing or empty, because that usually means the array is not
mounted and syncing into it would write onto the underlying root filesystem. `--seed` is the
deliberate exception, for the first push of a genuinely new share.
```bash
rsync.sh /mnt/user/New_Share --seed
```
### --merge-run
Bidirectional merge. Two passes:
```
Pass 1 pull remote → local, with --ignore-existing
Pass 2 push local → remote, with --delete
```
Local ends up authoritative: the remote may contribute files the local lacks, but never a
competing version of a file the local already has. Also triggered automatically when the
pre-scan detects ≥75% directory overlap with the remote.
**`--delete` is applied only if pass 1 completed.** Its entire justification is "local is now
the authoritative superset", and a pull that was capped by `RSYNC_MAX_RUNTIME_HOURS` or failed
outright means it is not — the remote still holds content the local never received. On an
incomplete pull the push runs **without** `--delete`, warns, and notifies. Local content still
propagates; nothing remote-unique is destroyed; the delete happens on a later run whose pull
actually succeeded.
> This was a real defect, fixed 2026-08-01. `--delete` used to be unconditional, so a capped
> or failed pull would permanently delete remote content the pull had not yet retrieved — and
> the timeout branch's own comment promised it would "resume the pull next run", which was
> impossible once the data was gone. If you are changing this function, that interlock is the
> part that matters.
---
## ━━━ SOURCE PATH GUARDS ━━━
`rsync.sh` pushes to `root@remote` at the **same absolute path**, and `--merge-run` adds
`--delete`. The source argument is therefore a remote-side hazard, not just a local one, so
two guards run before anything else:
| Guard | Rejects |
|-------|---------|
| Absolute path, ≥3 components | `/`, `/mnt`, `/mnt/user`, empty |
| Directory must exist locally | typo'd paths |
Every real job path clears both — see `HOST*_DAILY_SYNC_SHARES` and friends, all of which are
`/mnt/user/<Share>` or deeper.
Without the existence check, a mistyped path under `--merge-run` would be *created* locally by
the pull pass, filled from the remote, then pushed back with `--delete`.
---
## ━━━ RUNTIME CEILING ━━━
No single transfer attempt may exceed `RSYNC_MAX_RUNTIME_HOURS` (default 23). On timeout the
transfer is terminated and resumes on the next scheduled run.
This is only safe because `--partial` is in `DEFAULT_RSYNC_OPTS` — a terminated transfer picks
up where it left off rather than restarting. The cap exists so one huge or stuck transfer
cannot hold its per-profile lock indefinitely and starve every other profile of a turn.
Both `--merge-run` passes are capped independently.
@@ -77,7 +77,7 @@ Orchestrators/
────────────────────────► inotify_tuning.sh (before docker_network_connect) ────────────────────────► inotify_tuning.sh (before docker_network_connect)
array_stopping.sh ───────────────────────► conf_cache_save.sh (first — while cache is fresh) array_stopping.sh ───────────────────────► conf_cache_save.sh (first — while cache is fresh)
───────────────────────► rsync_stop.sh --rsync-only (stop transfers) ───────────────────────► rsync_stop.sh --rsync-only (stop transfers)
weekly_maintenance.sh ──────────────────► clear_logs.sh weekly_sync_maintenance.sh ──────────────────► clear_logs.sh
Watchdogs/System/ Watchdogs/System/
conf_cache_watchdog.sh ─────────────────► maintains $PERSISTENT_CONF_CACHE while partner offline conf_cache_watchdog.sh ─────────────────► maintains $PERSISTENT_CONF_CACHE while partner offline
@@ -109,7 +109,7 @@ Plugin/unraid/Watchdogs/System/
| `conf_sync.sh` | Sync partner confs: pull → RAM cache, push own conf to partners | At array start + every 4 h (--pull-only) | | `conf_sync.sh` | Sync partner confs: pull → RAM cache, push own conf to partners | At array start + every 4 h (--pull-only) |
| `conf_cache_save.sh` | Snapshot partner RAM cache → `$PERSISTENT_CONF_CACHE` | At array stop — first step | | `conf_cache_save.sh` | Snapshot partner RAM cache → `$PERSISTENT_CONF_CACHE` | At array stop — first step |
| `conf_cache_restore.sh` | Load missing partner confs from persistent backup into RAM | At array start — after conf_sync | | `conf_cache_restore.sh` | Load missing partner confs from persistent backup into RAM | At array start — after conf_sync |
| `clear_logs.sh` | Size-threshold log cleanup | Weekly via weekly_maintenance.sh | | `clear_logs.sh` | Size-threshold log cleanup | Weekly via weekly_sync_maintenance.sh |
| `rsync_stop.sh` | Orchestrator-aware rsync stop | Manual / called by array_stopping.sh | | `rsync_stop.sh` | Orchestrator-aware rsync stop | Manual / called by array_stopping.sh |
| `server_reboot.sh` | Graceful reboot with pre-flight warnings | Manual | | `server_reboot.sh` | Graceful reboot with pre-flight warnings | Manual |
@@ -141,7 +141,7 @@ Every 15 minutes (watchdog_orchestrator.sh):
Every 4 hours (intermediate_sync_maintenance.sh): Every 4 hours (intermediate_sync_maintenance.sh):
└─ conf_sync.sh --pull-only ← refresh partner conf in RAM without pushing own conf └─ conf_sync.sh --pull-only ← refresh partner conf in RAM without pushing own conf
Weekly (weekly_maintenance.sh): Weekly (weekly_sync_maintenance.sh):
└─ clear_logs.sh └─ clear_logs.sh
System logs: clear if > LOG_MIN_SIZE_MB System logs: clear if > LOG_MIN_SIZE_MB
Docker logs: clear per-container if > LOG_DOCKER_MAX_MB Docker logs: clear per-container if > LOG_DOCKER_MAX_MB
+1 -1
View File
@@ -250,7 +250,7 @@ docker start ContainerName
# Both live on /boot/config — survive reboots intentionally. # Both live on /boot/config — survive reboots intentionally.
# A container that was skip-listed before a reboot is still broken after it. # A container that was skip-listed before a reboot is still broken after it.
$SYS_WATCHDOG_FAILED_FILE # persistent skip list $DOCKER_WATCHDOG_FAILED_FILE # persistent skip list
$WATCHDOG_CONTAINER_RESTART_LOG # restart loop tracking $WATCHDOG_CONTAINER_RESTART_LOG # restart loop tracking
``` ```
+1
View File
@@ -130,6 +130,7 @@ The relationship is one-way: Tools act on state that other scripts have written.
| `emby_to_lidarr_sync.sh` | Add all Emby album artists not yet tracked in Lidarr | After Lidarr setup, database wipe, or when you suspect gaps | | `emby_to_lidarr_sync.sh` | Add all Emby album artists not yet tracked in Lidarr | After Lidarr setup, database wipe, or when you suspect gaps |
| `emby_to_sonarr_sync.sh` | Add all Emby TV series not yet tracked in Sonarr | After Sonarr setup, database wipe, or when you suspect gaps | | `emby_to_sonarr_sync.sh` | Add all Emby TV series not yet tracked in Sonarr | After Sonarr setup, database wipe, or when you suspect gaps |
| `emby_to_radarr_sync.sh` | Add all Emby movies not yet tracked in Radarr | After Radarr setup, database wipe, or when you suspect gaps | | `emby_to_radarr_sync.sh` | Add all Emby movies not yet tracked in Radarr | After Radarr setup, database wipe, or when you suspect gaps |
| `trailer_folder_migration.sh` | Relocate TV trailers Emby was ingesting as fake episodes into `<show>/trailers/` | Once, after Trailarr's Series Trailers profile was corrected |
--- ---
+50
View File
@@ -234,3 +234,53 @@ Every 15 minutes — watchdog_orchestrator.sh fires:
| `NETWORK_WATCHDOG_NPM_STATE_FILE` | `System/network_watchdog.sh` | itself | NPM proxy strike count | | `NETWORK_WATCHDOG_NPM_STATE_FILE` | `System/network_watchdog.sh` | itself | NPM proxy strike count |
All state files are in `$STATE_DIR` (adapts to storage mode). See `master.conf` for actual variable values. All state files are in `$STATE_DIR` (adapts to storage mode). See `master.conf` for actual variable values.
---
## ━━━ SAFEGUARDS COMMON TO THIS FOLDER ━━━
These run unattended every 15 minutes and hold the only irreversible remedy in the
ecosystem, so the guards matter as much as the checks. Each script's header documents its
full set; these are the folder-wide ones.
**Nothing acts on a single reading.** Every check that can trigger an action is strike-gated.
One bad sample is a reading; N consecutive samples is a condition. Strikes clear themselves
when the condition resolves, so a transient blip never accumulates toward an alarm across
unrelated weeks.
**Escalation is asymmetric on purpose.** Protective actions happen fast; undoing them happens
slowly. `resource_watchdog.sh` throttles before it pauses and pauses before it stops, then
requires `RW_RECOVER_CYCLES` clear cycles and de-escalates one level at a time. Recovering on
a single good reading would flap under sustained load.
**Ownership boundaries are enforced, not conventional.** Container health belongs to
`docker_watchdog.sh`; `stability_watchdog.sh` deliberately does not check it, and its Docker
daemon check writes a flag for `docker_watchdog.sh` rather than acting. Two watchdogs
remediating the same subsystem would race.
**Watchdogs coordinate through state files.** `resource_watchdog.sh` publishes
`mem_shutdown_active` so `docker_watchdog.sh` defers restarts — otherwise one would stop
containers to free RAM while the other restarted them to restore health. The reader applies a
staleness guard so a crashed writer cannot suppress restarts forever.
**Data safety outranks uptime.** `stability_watchdog.sh` aborts a reboot while a ZFS pool is
unhealthy, parity is running, or the mover is active. Only Tier 1 — conditions the system
cannot recover from and which worsen every cycle — bypasses that.
**Reboot loops end in shutdown, not more reboots.** No more than
`SYS_WATCHDOG_REBOOT_LIMIT` reboots within `SYS_WATCHDOG_REBOOT_WINDOW_HRS`. On hitting the
limit the host powers off: a fault that survives repeated reboots will not be fixed by more
of them, and a cleanly-down box is better than one cycling endlessly.
**An aborted reboot restores what it stopped.** An EXIT trap is armed the moment containers
start being stopped for a reboot and disarmed only once the reboot commits. If the script
dies in between, everything it stopped comes back — the failure mode is a running system,
never a host left stripped with no reboot.
**An unconfigured job list fails loudly.** An empty `WATCHDOG_ORCHESTRATOR_SCRIPTS` or
`SYSTEM_WATCHDOG_SCRIPTS` would otherwise report "0/0 passed" and exit 0 every cycle —
indistinguishable from healthy while nothing is monitored at all.
**Verify the path, not the process.** `network_watchdog.sh` checks NPM by fetching an external
URL rather than asking whether the container is running. A running container behind broken DNS
or a broken upstream still serves nothing.