Correct and extend folder docs for Docker Essentials, Watchdogs, Media and Arrs Stack
The docs had drifted from the scripts — a script that no longer exists, three wrong variable names, a reversed run order, and seven scheduled scripts that were never documented at all.
This commit is contained in:
Gmer4Lfe
@@ -200,6 +200,75 @@ Lidarr configured — no HOST1_LIDARR_URL means nothing runs.
|
||||
| `playback_aware_lidarr_discovery.sh` | Behavior-driven music discovery — Emby plays → Last.fm similar → Lidarr | Weekly |
|
||||
| `playback_aware_radarr_discovery.sh` | Behavior-driven movie discovery — Emby watches → TMDB recommendations → Radarr | Weekly |
|
||||
| `playback_aware_sonarr_discovery.sh` | Behavior-driven TV discovery — Emby episodes → TMDB TV recommendations → Sonarr | Weekly |
|
||||
| `arr_download_orphan_cleaner.sh` | Clear orphaned completed downloads out of the SABnzbd Completed folders | Daily |
|
||||
| `sonarr_classification_scan.sh` | Detect series sitting in the wrong root (anime / kids / general); `--move` acts | Daily |
|
||||
| `radarr_classification_scan.sh` | Same for movies, plus junk-metadata detection via `--remove-junk` | Daily |
|
||||
| `lidarr_duplicate_artist_cleanup.sh` | Remove phantom zero-file duplicate artists; flag real ones for review | Daily |
|
||||
| `arr_cache_prefill.sh` | Warm the shared tracked-data cache so consumers never read cold | Array start + every 4hr |
|
||||
| `arr_corruption_scan.sh` | ffprobe every tracked video for corrupt headers; `--remediate` deletes + re-searches | Weekly |
|
||||
| `arr_full_rescan.sh` | Force a real disk↔database reconciliation on all three arrs | Weekly |
|
||||
|
||||
---
|
||||
|
||||
## ━━━ THE NEWER LAYERS ━━━
|
||||
|
||||
The original folder was "delete what the arrs no longer track". These were added as distinct
|
||||
failure modes surfaced — each exists because something went wrong that the cleanups could not
|
||||
have caught.
|
||||
|
||||
### 🗑️ Download-Side Orphans — `arr_download_orphan_cleaner.sh`
|
||||
|
||||
Every cleanup script here walks the **library** side. Nothing walked the **download** side —
|
||||
so completed downloads the arrs had stopped tracking accumulated in SABnzbd's Completed
|
||||
folders indefinitely. Discovered as **755 GB of orphaned TV downloads, oldest from 2022**,
|
||||
filling the cache pool to 89%.
|
||||
|
||||
Classifies every entry as TRACKED / RECENT / JUNK / REDUNDANT / IMPORTABLE / UNMATCHED and
|
||||
acts only on the ones it can justify. The queue is a hard gate: if it cannot be read, the arr
|
||||
is skipped entirely, because without it there is no way to tell an active import from an
|
||||
orphan. A run total over `DOWNLOAD_ORPHAN_MAX_DELETE_GB` aborts — an abnormally large delete
|
||||
is the visible symptom of a partial queue fetch.
|
||||
|
||||
### 🎭 Wrong-Root Detection — `sonarr_classification_scan.sh` + `radarr_classification_scan.sh`
|
||||
|
||||
Overseerr lets any user request content into the wrong root folder — kids shows into general
|
||||
TV, anime into Kids_Tv_Shows. These classify every item from metadata alone (genre,
|
||||
certification, network/studio, original language) and report where the computed classification
|
||||
disagrees with the folder the item actually sits in.
|
||||
|
||||
Report-only by default. `--move` acts on forward misplacements and adult-content-in-kids-root
|
||||
leaks. It deliberately does **not** move non-anime content out of the anime root — deliberate
|
||||
style placements (Western animation grouped with anime by choice) are genuine judgment calls.
|
||||
|
||||
Both poll the arr's async move command to completion before verifying, because `moveFiles=true`
|
||||
flips the database instantly while the physical move is still queued behind others.
|
||||
|
||||
### 🎨 Phantom Artists — `lidarr_duplicate_artist_cleanup.sh`
|
||||
|
||||
MusicBrainz duplicates leave two Lidarr entries for one artist, one holding the files and one
|
||||
holding nothing. Removes only the zero-file side, with `deleteFiles=false` so nothing on disk
|
||||
is touched. Pairs where both sides hold files are flagged for review, never auto-resolved.
|
||||
|
||||
Gated on the tracked-count floor shared with `lidarr_cleanup.sh` — during a library-wide desync
|
||||
both sides of a real duplicate can read as zero-file phantoms.
|
||||
|
||||
### 🩺 Corruption + Reconciliation — `arr_corruption_scan.sh` + `arr_full_rescan.sh`
|
||||
|
||||
`arr_corruption_scan.sh` ffprobes tracked video files for corrupt headers. Report-only unless
|
||||
`--remediate`, which deletes the file record and triggers an explicit re-search. Requires
|
||||
repeat detections across separate runs before acting, so a transient probe failure cannot
|
||||
delete a healthy file.
|
||||
|
||||
`arr_full_rescan.sh` forces a genuine disk↔database reconciliation. Organic scans only touch
|
||||
files involved in an import, so an untouched library silently drifts — confirmed when Lidarr
|
||||
reported **~23% of its true track count** for 1,004 of 1,357 artists with no scan running and
|
||||
every file present on disk.
|
||||
|
||||
### ⚡ Cache Warmth — `arr_cache_prefill.sh`
|
||||
|
||||
Populates the shared tracked-data cache at array start and every 4 hours, so consumers never
|
||||
pay a cold fetch. Pure enhancement: nothing depends on it having run, and every consumer still
|
||||
writes through on a cold cache.
|
||||
|
||||
---
|
||||
|
||||
|
||||
@@ -94,20 +94,54 @@ HOST1_NETWORK_CONNECT_CONTAINERS=(
|
||||
|
||||
```bash
|
||||
# master.conf
|
||||
DAILY_CONTAINER_UPDATES=true # enable/disable daily image pull
|
||||
DAILY_CONTAINER_UPDATES=true # enable/disable the daily image pull
|
||||
# docker_daily_restart.sh still runs regardless
|
||||
# update and restart are independent
|
||||
|
||||
WEEKLY_REMAINING_UPDATES=true # enable/disable weekly remainder pull + prune
|
||||
# to disable: set false or remove from WEEKLY_MAINTENANCE_SCRIPTS
|
||||
WEEKLY_CONTAINER_UPDATES=true # enable/disable the weekly image pull
|
||||
# docker_weekly_restart.sh still runs regardless
|
||||
|
||||
MONTHLY_REMAINING_UPDATES=true # enable/disable the monthly remainder pull
|
||||
# to disable: set false, or remove
|
||||
# "docker_update.sh --remainder" from
|
||||
# MONTHLY_MAINTENANCE_SCRIPTS
|
||||
```
|
||||
|
||||
`docker_update.sh` in normal mode targets `DAILY_RESTART_CONTAINERS` — the same list
|
||||
used by `docker_daily_restart.sh`. No second list to maintain.
|
||||
There is one update script, `docker_update.sh`, with three modes. Each reuses the restart
|
||||
list it pairs with, so there is no second list to maintain:
|
||||
|
||||
`docker_update_remaining.sh` derives its target list automatically:
|
||||
all running containers minus `DAILY_RESTART_CONTAINERS` minus `WEEKLY_RESTART_CONTAINERS`.
|
||||
Everything gets updated at least once per week with no explicit configuration.
|
||||
| Mode | Targets | Runs |
|
||||
|------|---------|------|
|
||||
| *(default)* | `DAILY_RESTART_CONTAINERS` | Daily, before `docker_daily_restart.sh` |
|
||||
| `--weekly` | `WEEKLY_RESTART_CONTAINERS` | Weekly, before `docker_weekly_restart.sh` |
|
||||
| `--remainder` | derived, see below | Monthly, via `MONTHLY_MAINTENANCE_SCRIPTS` |
|
||||
|
||||
Remainder mode needs no configuration at all. It takes every **running** container and
|
||||
subtracts:
|
||||
|
||||
```
|
||||
DAILY_RESTART_CONTAINERS already updated daily
|
||||
WEEKLY_RESTART_CONTAINERS already updated weekly
|
||||
PROFILE_CRITICAL_CONTAINER_NAMES[emby] updated inline by the weekly sync window
|
||||
PROFILE_CRITICAL_CONTAINER_NAMES[critical-data] updated inline by the weekly sync window
|
||||
FALLBACK_<REMOTE>_TIER1..4 owned by the remote's update cycle
|
||||
```
|
||||
|
||||
Stopped containers are never targeted in any mode — pulling for a stopped container adds
|
||||
nothing, and it was most likely stopped deliberately.
|
||||
|
||||
The fallback exclusion is a correctness rule, not an optimisation. This server only runs
|
||||
those containers during a fallback; the remote owns their version. If remainder updated them
|
||||
independently and a handback then occurred, the remote's older image could meet data written
|
||||
by the newer one.
|
||||
|
||||
**Ordering matters.** The update always runs *before* its matching restart so the restart
|
||||
lands on the freshly pulled image. If a container's image actually changed, `docker_update.sh`
|
||||
rebuilds it from its template (a plain `docker restart` reuses the image ID baked in at
|
||||
creation time and would never pick up the new digest) and records it in
|
||||
`DOCKER_UPDATE_REBUILT_*_FILE`. The restart script reads that file and skips those containers
|
||||
rather than restarting them a second time — and discards the file as stale if it is older
|
||||
than `DOCKER_UPDATE_REBUILT_STALE_HOURS`.
|
||||
|
||||
---
|
||||
|
||||
@@ -142,12 +176,22 @@ HOST1_NETWORK_CONNECT_CONTAINERS=()
|
||||
# Shared — applies to both servers
|
||||
|
||||
# ── Container updates ──────────────────────────────────────────────────
|
||||
DAILY_CONTAINER_UPDATES=true
|
||||
WEEKLY_REMAINING_UPDATES=true
|
||||
DAILY_CONTAINER_UPDATES=true # daily pull (DAILY_RESTART_CONTAINERS)
|
||||
WEEKLY_CONTAINER_UPDATES=true # weekly pull (WEEKLY_RESTART_CONTAINERS)
|
||||
MONTHLY_REMAINING_UPDATES=true # monthly pull (everything else)
|
||||
|
||||
# Handoff between update and restart — written by docker_update.sh,
|
||||
# read by the matching restart script so it skips containers already
|
||||
# rebuilt onto a new image this run.
|
||||
DOCKER_UPDATE_REBUILT_DAILY_FILE
|
||||
DOCKER_UPDATE_REBUILT_WEEKLY_FILE
|
||||
DOCKER_UPDATE_REBUILT_STALE_HOURS=12 # older than this = discarded, restart all
|
||||
|
||||
# ── Retry behaviour (shared by restart scripts) ────────────────────────
|
||||
RETRY_COUNT=3 # retry attempts before marking failed
|
||||
SLEEP=5 # seconds between retry attempts
|
||||
CONTAINER_DELAY # seconds between a dependency and its dependents
|
||||
RESTART_VERIFY_WAIT=3 # settle time before verifying a restart stuck
|
||||
|
||||
# Watchdog thresholds (CPU, memory, HTTP, restart loop):
|
||||
# → see Watchdogs/Manual-Watchdogs.md
|
||||
@@ -211,7 +255,24 @@ All scripts support these standard flags:
|
||||
summary block: identity, duration, counts, and a status line. Per-container detail only
|
||||
appears with `--log`. Warnings and errors are always visible regardless of `--log`.
|
||||
|
||||
### `docker_update.sh --remainder`
|
||||
Switches to remainder mode — updates all running containers not in the managed daily/weekly
|
||||
lists. Called by `weekly_sync_maintenance.sh`. Can be run manually to sweep containers
|
||||
that haven't been updated recently.
|
||||
### Script-specific flags
|
||||
|
||||
| Script | Flag | What it does |
|
||||
|--------|------|-------------|
|
||||
| `docker_update.sh` | `--weekly` | Target `WEEKLY_RESTART_CONTAINERS`. Called by `weekly_sync_maintenance.sh` before the weekly restart. |
|
||||
| `docker_update.sh` | `--remainder` | Target every running container not in the daily list, weekly list, emby/critical-data profiles, or fallback tiers. Called by `monthly_maintenance.sh`. Safe to run manually to sweep anything missed. |
|
||||
| `media_cleaner.sh` *(Media/)* | `<anime\|media>` | Required positional profile — there is no default. |
|
||||
|
||||
`docker_update.sh` with no mode flag is normal mode: `DAILY_RESTART_CONTAINERS`, called by
|
||||
`daily_sync_maintenance.sh` before the daily restart.
|
||||
|
||||
### Exit codes
|
||||
|
||||
| Code | Meaning |
|
||||
|------|---------|
|
||||
| `0` | Success, or nothing to do (empty list, disabled toggle, no running containers) |
|
||||
| `1` | One or more containers failed — the summary names them |
|
||||
|
||||
`docker_update.sh` deliberately exits `0` even when pulls fail: a failed pull is not a reason
|
||||
to abort the restart that follows, which simply proceeds on the existing image. The failure
|
||||
is reported in the summary.
|
||||
|
||||
@@ -178,12 +178,28 @@ windows so any downtime from restarts is absorbed by the window that's already h
|
||||
|
||||
---
|
||||
|
||||
### 🔄 Image Currency — `docker_update.sh` + `docker_update_remaining.sh`
|
||||
### 🔄 Image Currency — `docker_update.sh`
|
||||
|
||||
Keeps all container images current without manual intervention. Daily updates for the
|
||||
auth/proxy stack (the containers that restart daily anyway — no extra downtime). Weekly
|
||||
remainder pass for everything else — derives the target list automatically from `docker ps`
|
||||
minus what was already updated, so there is no second list to maintain.
|
||||
One script, three modes — there is no separate remainder script.
|
||||
|
||||
| Mode | Targets | Runs |
|
||||
|------|---------|------|
|
||||
| *(default)* | `DAILY_RESTART_CONTAINERS` | Daily, **before** `docker_daily_restart.sh` |
|
||||
| `--weekly` | `WEEKLY_RESTART_CONTAINERS` | Weekly, **before** `docker_weekly_restart.sh` |
|
||||
| `--remainder` | Everything running that is in neither list | Monthly, via `monthly_maintenance.sh` |
|
||||
|
||||
The update always runs *before* its matching restart, so the restart lands on the freshly
|
||||
pulled image. Reversing that order would restart onto the old image and leave the new one
|
||||
sitting unused until the next window.
|
||||
|
||||
Each mode reuses the restart list it pairs with rather than maintaining its own — add a
|
||||
container to `DAILY_RESTART_CONTAINERS` once and it gets both the restart and the image pull.
|
||||
Remainder mode needs no list at all: it derives its targets from `docker ps` minus the daily
|
||||
list, the weekly list, the emby/critical-data sync-window profiles, and the fallback tiers.
|
||||
|
||||
Fallback containers are deliberately excluded from remainder mode. This server only runs them
|
||||
during a fallback; the remote owns their version. Updating them here would risk the remote's
|
||||
older image meeting data written by a newer one after a handback.
|
||||
|
||||
---
|
||||
|
||||
@@ -229,7 +245,7 @@ full coordination model between all four watchdogs.
|
||||
|--------|------|-------------|
|
||||
| `docker_daily_restart.sh` | Nightly proactive restart of degradation-prone containers | 1am via `daily_sync_maintenance.sh` |
|
||||
| `docker_weekly_restart.sh` | Weekly restart of less-critical services | 2:30am Sunday via `weekly_sync_maintenance.sh` |
|
||||
| `docker_update.sh` | Container image updates — daily list + weekly remainder mode | Daily before restart; weekly remainder pass |
|
||||
| `docker_update.sh` | Container image updates — three modes (default / `--weekly` / `--remainder`) | Daily and weekly before each restart; monthly for the remainder |
|
||||
| `docker_network_connect.sh` | Network existence + container connection enforcement | Every array start |
|
||||
| `docker_container_stop.sh` | Ordered container shutdown — graceful then forced | Called by `array_stopping.sh` |
|
||||
| `downloaders_reset.sh` | Download client hygiene — slskd / SABnzbd / qBittorrent | Every 30min via `critical_sync_maintenance.sh` |
|
||||
@@ -257,8 +273,13 @@ Daily maintenance window (1am):
|
||||
|
||||
Weekly maintenance window (2:30am Sunday):
|
||||
weekly_sync_maintenance.sh
|
||||
├── docker_weekly_restart.sh ──── restart less-critical services
|
||||
└── docker_update.sh --remainder ─ update containers not in managed lists
|
||||
├── docker_update.sh --weekly ─── pull latest (WEEKLY_RESTART_CONTAINERS)
|
||||
└── docker_weekly_restart.sh ──── restart onto the fresh image
|
||||
|
||||
Monthly maintenance window:
|
||||
monthly_maintenance.sh
|
||||
├── docker_update.sh --remainder update everything not in the managed lists
|
||||
└── Tools/docker_prune_images.sh --all
|
||||
|
||||
Critical maintenance window (every 30min):
|
||||
critical_sync_maintenance.sh
|
||||
@@ -266,5 +287,39 @@ Critical maintenance window (every 30min):
|
||||
|
||||
Array stopping:
|
||||
array_stopping.sh
|
||||
└── docker_container_stop.sh ──── ordered graceful shutdown
|
||||
└── docker_container_stop.sh ──── ordered graceful shutdown, verified per container
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## ━━━ SAFEGUARDS COMMON TO THIS FOLDER ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
|
||||
|
||||
Every script here talks to the Docker daemon, so they share the same protections. Each
|
||||
script's own header documents its full set; these are the ones worth knowing folder-wide.
|
||||
|
||||
**Daemon health is checked, not assumed.** A hung daemon returns an empty container list,
|
||||
which is indistinguishable from "no containers running". Without the check,
|
||||
`docker_container_stop.sh` would report a clean shutdown that never happened, and
|
||||
`docker_update.sh --remainder` would report "nothing to update" while doing nothing.
|
||||
|
||||
**Every docker call is timeout-wrapped.** A wedged daemon cannot stall a maintenance window
|
||||
or hold a lock open. The one deliberate exception is `docker pull` — a large image
|
||||
legitimately outlasts any sane timeout, and killing it mid-layer wastes the transfer.
|
||||
|
||||
**State is respected.** Running containers get restarted; stopped ones stay stopped. A
|
||||
stopped container was almost certainly stopped on purpose, and none of these scripts has the
|
||||
authority to overrule that.
|
||||
|
||||
**Restarts are verified, not assumed.** After each restart the container is re-checked once
|
||||
it has had time to settle. A container that starts and immediately crashes is recorded as a
|
||||
failure and notified — a restart that did not stick is never reported as success.
|
||||
|
||||
**Dependency ordering is shared with the watchdog.** Restarts follow
|
||||
`HOST*_WATCHDOG_DEPENDENCIES`, with `CONTAINER_DELAY` between a dependency and its dependents,
|
||||
so a dependent is never brought up while what it needs is still initialising.
|
||||
|
||||
**Locks prevent overlap.** Long windows can outlast their interval;
|
||||
`downloaders_reset.sh` uses wait-mode because it runs every 30 minutes and the previous pass
|
||||
may still be finishing.
|
||||
|
||||
---
|
||||
|
||||
+33
-7
@@ -86,10 +86,25 @@ MEDIA_FILE_PATTERNS=(
|
||||
### Play State Sync
|
||||
|
||||
```bash
|
||||
PLAY_STATE_SYNC_ENABLED=true # toggle entire sync
|
||||
PLAY_STATE_SYNC_LOOKBACK_DAYS=30 # history window for played items
|
||||
PLAY_SYNC_ENABLED=true # toggle entire sync
|
||||
PLAY_SYNC_REMOTE=true # sync across hosts via Tailscale
|
||||
# false = local servers only (this host's Emby + Jellyfin)
|
||||
PLAY_SYNC_TYPES="Movie,Episode" # item types to sync — Audio excluded, music library too large
|
||||
PLAY_SYNC_FAV_TYPES="MusicArtist,MusicAlbum,Movie,Series" # favourites, union sync, never unmarks
|
||||
|
||||
PLAY_SYNC_PROBE=true # skip per-item work when nothing changed since last run
|
||||
PLAY_SYNC_PROBE_MAX_AGE_HOURS=24 # force a full comparison when the fingerprint is older
|
||||
|
||||
PLAY_SYNC_HANDBACK_RETRIES=5 # fallback handback: attempts before DNS cutover proceeds
|
||||
PLAY_SYNC_HANDBACK_RETRY_DELAY=60 # seconds between those attempts
|
||||
```
|
||||
|
||||
> **There is no date/lookback filter, and one must not be re-added.** An earlier version
|
||||
> gated on played-date; it was removed once the real cost was measured — the 30-minute
|
||||
> runtime was fork overhead per item, not API volume or item count. The fix was jq epoch
|
||||
> parsing plus the response-hash probe below. Re-introducing a date window would reduce
|
||||
> correctness (older items silently stop syncing) without meaningfully reducing runtime.
|
||||
|
||||
Emby and Jellyfin servers configured per-host:
|
||||
|
||||
```bash
|
||||
@@ -194,13 +209,24 @@ Show every file examined, not just those removed.
|
||||
### play_state_sync.sh
|
||||
|
||||
`play_state_sync.sh`
|
||||
Sync played/unplayed state and resume positions from local Emby to the remote Emby.
|
||||
Only items played within PLAY_STATE_SYNC_LOOKBACK_DAYS are synced.
|
||||
Sync played/unplayed state, resume positions and favourites across **every** configured Emby
|
||||
and Jellyfin server — not just local→remote. Servers are discovered from every
|
||||
`HOST*_TRANSCODE_SERVERS` entry, with remote hosts' localhost URLs rewritten to their
|
||||
Tailscale IP. Newest `LastPlayedDate` wins; state only ever moves forward, never clears.
|
||||
|
||||
No age filter — every matched item is considered on every run. The **change probe** is what
|
||||
keeps that cheap: the raw API responses are hashed and compared against the fingerprint from
|
||||
the last successful run, and per-item processing is skipped entirely when nothing moved.
|
||||
Fetches still happen every run, so nothing can be missed by the probe.
|
||||
|
||||
`play_state_sync.sh --full`
|
||||
Ignore PLAY_STATE_SYNC_LOOKBACK_DAYS — sync all played items regardless of age. May be
|
||||
slow on large libraries. Use after a new Emby install or database restore to rebuild
|
||||
full play history.
|
||||
Bypass the change probe and force the full per-item comparison even when the fingerprint
|
||||
matches. Use after a new Emby install or database restore, or when debugging a sync that
|
||||
appears to be skipping work it should be doing.
|
||||
|
||||
`play_state_sync.sh --wait`
|
||||
Wait for an in-progress run instead of exiting. For manual runs that would otherwise be
|
||||
skipped by the scheduled every-30-minute pass. Used by `fallback.sh` during handback.
|
||||
|
||||
`play_state_sync.sh --dry-run`
|
||||
Show what would be synced without writing any state.
|
||||
|
||||
@@ -100,3 +100,43 @@ syncs between them.
|
||||
| `media_shares_permissions.sh` | Apply `nobody:users` ownership + correct permissions to all media shares | Daily — runs first |
|
||||
| `media_cleaner.sh` | Remove junk files (two profiles: `anime` + `media`) | Daily — runs before arr cleanup |
|
||||
| `play_state_sync.sh` | Sync watched/played state + resume positions across Emby + Jellyfin | Every 30 min |
|
||||
|
||||
---
|
||||
|
||||
## ━━━ THE ctime INVARIANT — READ BEFORE CHANGING PERMISSIONS ━━━
|
||||
|
||||
`media_shares_permissions.sh` applies every pass **conditionally** — it touches only entries
|
||||
whose owner or mode is actually wrong. That is not an optimisation, and it must stay that way.
|
||||
|
||||
`chown` and `chmod` rewrite an inode's ctime **even when the value does not change**. A
|
||||
blanket pass would therefore restamp every file in the library every night.
|
||||
|
||||
The arr cleanup scripts (`sonarr_cleanup.sh`, `radarr_cleanup.sh`, `lidarr_cleanup.sh`) gate
|
||||
orphan deletion on ctime. mtime cannot substitute: an import preserves the release's original
|
||||
timestamp, so mtime says nothing about when a file arrived here. Measured 2026-07-27 — of 400
|
||||
files imported that week, **all 400 had mtimes over 7 days old, one of them 9613 days.**
|
||||
|
||||
So:
|
||||
|
||||
```
|
||||
blanket chown/chmod → every ctime resets to today
|
||||
→ no file ever appears older than *_ORPHAN_AGE
|
||||
→ orphan collection silently stops
|
||||
→ nothing errors, nothing warns, disk just fills
|
||||
```
|
||||
|
||||
**The failure is invisible.** No script fails, no notification fires. The only symptom is
|
||||
orphans quietly accumulating until a pool fills — which is exactly how the 755 GB / 89%-full
|
||||
cache pool incident happened.
|
||||
|
||||
Two rules follow, and both are load-bearing:
|
||||
|
||||
1. **`media_shares_permissions.sh` passes stay conditional.** Making any of them unconditional
|
||||
breaks orphan collection ecosystem-wide.
|
||||
2. **`Tools/bulk_permissions_repair.sh` is unconditional by design** — it exists to repair
|
||||
known-wrong paths where correctness beats preserving a clock. That is precisely why it is a
|
||||
manual, targeted tool and not scheduled. Pointing it at a whole media root pauses orphan
|
||||
collection there for `*_ORPHAN_AGE` days.
|
||||
|
||||
Both scripts' headers carry this warning too. If you are reading this because you are about to
|
||||
"simplify" the permissions job, this is the thing that breaks.
|
||||
|
||||
@@ -234,3 +234,53 @@ Every 15 minutes — watchdog_orchestrator.sh fires:
|
||||
| `NETWORK_WATCHDOG_NPM_STATE_FILE` | `System/network_watchdog.sh` | itself | NPM proxy strike count |
|
||||
|
||||
All state files are in `$STATE_DIR` (adapts to storage mode). See `master.conf` for actual variable values.
|
||||
|
||||
---
|
||||
|
||||
## ━━━ SAFEGUARDS COMMON TO THIS FOLDER ━━━
|
||||
|
||||
These run unattended every 15 minutes and hold the only irreversible remedy in the
|
||||
ecosystem, so the guards matter as much as the checks. Each script's header documents its
|
||||
full set; these are the folder-wide ones.
|
||||
|
||||
**Nothing acts on a single reading.** Every check that can trigger an action is strike-gated.
|
||||
One bad sample is a reading; N consecutive samples is a condition. Strikes clear themselves
|
||||
when the condition resolves, so a transient blip never accumulates toward an alarm across
|
||||
unrelated weeks.
|
||||
|
||||
**Escalation is asymmetric on purpose.** Protective actions happen fast; undoing them happens
|
||||
slowly. `resource_watchdog.sh` throttles before it pauses and pauses before it stops, then
|
||||
requires `RW_RECOVER_CYCLES` clear cycles and de-escalates one level at a time. Recovering on
|
||||
a single good reading would flap under sustained load.
|
||||
|
||||
**Ownership boundaries are enforced, not conventional.** Container health belongs to
|
||||
`docker_watchdog.sh`; `stability_watchdog.sh` deliberately does not check it, and its Docker
|
||||
daemon check writes a flag for `docker_watchdog.sh` rather than acting. Two watchdogs
|
||||
remediating the same subsystem would race.
|
||||
|
||||
**Watchdogs coordinate through state files.** `resource_watchdog.sh` publishes
|
||||
`mem_shutdown_active` so `docker_watchdog.sh` defers restarts — otherwise one would stop
|
||||
containers to free RAM while the other restarted them to restore health. The reader applies a
|
||||
staleness guard so a crashed writer cannot suppress restarts forever.
|
||||
|
||||
**Data safety outranks uptime.** `stability_watchdog.sh` aborts a reboot while a ZFS pool is
|
||||
unhealthy, parity is running, or the mover is active. Only Tier 1 — conditions the system
|
||||
cannot recover from and which worsen every cycle — bypasses that.
|
||||
|
||||
**Reboot loops end in shutdown, not more reboots.** No more than
|
||||
`SYS_WATCHDOG_REBOOT_LIMIT` reboots within `SYS_WATCHDOG_REBOOT_WINDOW_HRS`. On hitting the
|
||||
limit the host powers off: a fault that survives repeated reboots will not be fixed by more
|
||||
of them, and a cleanly-down box is better than one cycling endlessly.
|
||||
|
||||
**An aborted reboot restores what it stopped.** An EXIT trap is armed the moment containers
|
||||
start being stopped for a reboot and disarmed only once the reboot commits. If the script
|
||||
dies in between, everything it stopped comes back — the failure mode is a running system,
|
||||
never a host left stripped with no reboot.
|
||||
|
||||
**An unconfigured job list fails loudly.** An empty `WATCHDOG_ORCHESTRATOR_SCRIPTS` or
|
||||
`SYSTEM_WATCHDOG_SCRIPTS` would otherwise report "0/0 passed" and exit 0 every cycle —
|
||||
indistinguishable from healthy while nothing is monitored at all.
|
||||
|
||||
**Verify the path, not the process.** `network_watchdog.sh` checks NPM by fetching an external
|
||||
URL rather than asking whether the container is running. A running container behind broken DNS
|
||||
or a broken upstream still serves nothing.
|
||||
|
||||
Reference in New Issue
Block a user