Correct and extend folder docs for Docker Essentials, Watchdogs, Media and Arrs Stack
The docs had drifted from the scripts — a script that no longer exists, three wrong variable names, a reversed run order, and seven scheduled scripts that were never documented at all.
This commit is contained in:
@@ -200,6 +200,75 @@ Lidarr configured — no HOST1_LIDARR_URL means nothing runs.
|
|||||||
| `playback_aware_lidarr_discovery.sh` | Behavior-driven music discovery — Emby plays → Last.fm similar → Lidarr | Weekly |
|
| `playback_aware_lidarr_discovery.sh` | Behavior-driven music discovery — Emby plays → Last.fm similar → Lidarr | Weekly |
|
||||||
| `playback_aware_radarr_discovery.sh` | Behavior-driven movie discovery — Emby watches → TMDB recommendations → Radarr | Weekly |
|
| `playback_aware_radarr_discovery.sh` | Behavior-driven movie discovery — Emby watches → TMDB recommendations → Radarr | Weekly |
|
||||||
| `playback_aware_sonarr_discovery.sh` | Behavior-driven TV discovery — Emby episodes → TMDB TV recommendations → Sonarr | Weekly |
|
| `playback_aware_sonarr_discovery.sh` | Behavior-driven TV discovery — Emby episodes → TMDB TV recommendations → Sonarr | Weekly |
|
||||||
|
| `arr_download_orphan_cleaner.sh` | Clear orphaned completed downloads out of the SABnzbd Completed folders | Daily |
|
||||||
|
| `sonarr_classification_scan.sh` | Detect series sitting in the wrong root (anime / kids / general); `--move` acts | Daily |
|
||||||
|
| `radarr_classification_scan.sh` | Same for movies, plus junk-metadata detection via `--remove-junk` | Daily |
|
||||||
|
| `lidarr_duplicate_artist_cleanup.sh` | Remove phantom zero-file duplicate artists; flag real ones for review | Daily |
|
||||||
|
| `arr_cache_prefill.sh` | Warm the shared tracked-data cache so consumers never read cold | Array start + every 4hr |
|
||||||
|
| `arr_corruption_scan.sh` | ffprobe every tracked video for corrupt headers; `--remediate` deletes + re-searches | Weekly |
|
||||||
|
| `arr_full_rescan.sh` | Force a real disk↔database reconciliation on all three arrs | Weekly |
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## ━━━ THE NEWER LAYERS ━━━
|
||||||
|
|
||||||
|
The original folder was "delete what the arrs no longer track". These were added as distinct
|
||||||
|
failure modes surfaced — each exists because something went wrong that the cleanups could not
|
||||||
|
have caught.
|
||||||
|
|
||||||
|
### 🗑️ Download-Side Orphans — `arr_download_orphan_cleaner.sh`
|
||||||
|
|
||||||
|
Every cleanup script here walks the **library** side. Nothing walked the **download** side —
|
||||||
|
so completed downloads the arrs had stopped tracking accumulated in SABnzbd's Completed
|
||||||
|
folders indefinitely. Discovered as **755 GB of orphaned TV downloads, oldest from 2022**,
|
||||||
|
filling the cache pool to 89%.
|
||||||
|
|
||||||
|
Classifies every entry as TRACKED / RECENT / JUNK / REDUNDANT / IMPORTABLE / UNMATCHED and
|
||||||
|
acts only on the ones it can justify. The queue is a hard gate: if it cannot be read, the arr
|
||||||
|
is skipped entirely, because without it there is no way to tell an active import from an
|
||||||
|
orphan. A run total over `DOWNLOAD_ORPHAN_MAX_DELETE_GB` aborts — an abnormally large delete
|
||||||
|
is the visible symptom of a partial queue fetch.
|
||||||
|
|
||||||
|
### 🎭 Wrong-Root Detection — `sonarr_classification_scan.sh` + `radarr_classification_scan.sh`
|
||||||
|
|
||||||
|
Overseerr lets any user request content into the wrong root folder — kids shows into general
|
||||||
|
TV, anime into Kids_Tv_Shows. These classify every item from metadata alone (genre,
|
||||||
|
certification, network/studio, original language) and report where the computed classification
|
||||||
|
disagrees with the folder the item actually sits in.
|
||||||
|
|
||||||
|
Report-only by default. `--move` acts on forward misplacements and adult-content-in-kids-root
|
||||||
|
leaks. It deliberately does **not** move non-anime content out of the anime root — deliberate
|
||||||
|
style placements (Western animation grouped with anime by choice) are genuine judgment calls.
|
||||||
|
|
||||||
|
Both poll the arr's async move command to completion before verifying, because `moveFiles=true`
|
||||||
|
flips the database instantly while the physical move is still queued behind others.
|
||||||
|
|
||||||
|
### 🎨 Phantom Artists — `lidarr_duplicate_artist_cleanup.sh`
|
||||||
|
|
||||||
|
MusicBrainz duplicates leave two Lidarr entries for one artist, one holding the files and one
|
||||||
|
holding nothing. Removes only the zero-file side, with `deleteFiles=false` so nothing on disk
|
||||||
|
is touched. Pairs where both sides hold files are flagged for review, never auto-resolved.
|
||||||
|
|
||||||
|
Gated on the tracked-count floor shared with `lidarr_cleanup.sh` — during a library-wide desync
|
||||||
|
both sides of a real duplicate can read as zero-file phantoms.
|
||||||
|
|
||||||
|
### 🩺 Corruption + Reconciliation — `arr_corruption_scan.sh` + `arr_full_rescan.sh`
|
||||||
|
|
||||||
|
`arr_corruption_scan.sh` ffprobes tracked video files for corrupt headers. Report-only unless
|
||||||
|
`--remediate`, which deletes the file record and triggers an explicit re-search. Requires
|
||||||
|
repeat detections across separate runs before acting, so a transient probe failure cannot
|
||||||
|
delete a healthy file.
|
||||||
|
|
||||||
|
`arr_full_rescan.sh` forces a genuine disk↔database reconciliation. Organic scans only touch
|
||||||
|
files involved in an import, so an untouched library silently drifts — confirmed when Lidarr
|
||||||
|
reported **~23% of its true track count** for 1,004 of 1,357 artists with no scan running and
|
||||||
|
every file present on disk.
|
||||||
|
|
||||||
|
### ⚡ Cache Warmth — `arr_cache_prefill.sh`
|
||||||
|
|
||||||
|
Populates the shared tracked-data cache at array start and every 4 hours, so consumers never
|
||||||
|
pay a cold fetch. Pure enhancement: nothing depends on it having run, and every consumer still
|
||||||
|
writes through on a cold cache.
|
||||||
|
|
||||||
---
|
---
|
||||||
|
|
||||||
|
|||||||
@@ -94,20 +94,54 @@ HOST1_NETWORK_CONNECT_CONTAINERS=(
|
|||||||
|
|
||||||
```bash
|
```bash
|
||||||
# master.conf
|
# master.conf
|
||||||
DAILY_CONTAINER_UPDATES=true # enable/disable daily image pull
|
DAILY_CONTAINER_UPDATES=true # enable/disable the daily image pull
|
||||||
# docker_daily_restart.sh still runs regardless
|
# docker_daily_restart.sh still runs regardless
|
||||||
# update and restart are independent
|
# update and restart are independent
|
||||||
|
|
||||||
WEEKLY_REMAINING_UPDATES=true # enable/disable weekly remainder pull + prune
|
WEEKLY_CONTAINER_UPDATES=true # enable/disable the weekly image pull
|
||||||
# to disable: set false or remove from WEEKLY_MAINTENANCE_SCRIPTS
|
# docker_weekly_restart.sh still runs regardless
|
||||||
|
|
||||||
|
MONTHLY_REMAINING_UPDATES=true # enable/disable the monthly remainder pull
|
||||||
|
# to disable: set false, or remove
|
||||||
|
# "docker_update.sh --remainder" from
|
||||||
|
# MONTHLY_MAINTENANCE_SCRIPTS
|
||||||
```
|
```
|
||||||
|
|
||||||
`docker_update.sh` in normal mode targets `DAILY_RESTART_CONTAINERS` — the same list
|
There is one update script, `docker_update.sh`, with three modes. Each reuses the restart
|
||||||
used by `docker_daily_restart.sh`. No second list to maintain.
|
list it pairs with, so there is no second list to maintain:
|
||||||
|
|
||||||
`docker_update_remaining.sh` derives its target list automatically:
|
| Mode | Targets | Runs |
|
||||||
all running containers minus `DAILY_RESTART_CONTAINERS` minus `WEEKLY_RESTART_CONTAINERS`.
|
|------|---------|------|
|
||||||
Everything gets updated at least once per week with no explicit configuration.
|
| *(default)* | `DAILY_RESTART_CONTAINERS` | Daily, before `docker_daily_restart.sh` |
|
||||||
|
| `--weekly` | `WEEKLY_RESTART_CONTAINERS` | Weekly, before `docker_weekly_restart.sh` |
|
||||||
|
| `--remainder` | derived, see below | Monthly, via `MONTHLY_MAINTENANCE_SCRIPTS` |
|
||||||
|
|
||||||
|
Remainder mode needs no configuration at all. It takes every **running** container and
|
||||||
|
subtracts:
|
||||||
|
|
||||||
|
```
|
||||||
|
DAILY_RESTART_CONTAINERS already updated daily
|
||||||
|
WEEKLY_RESTART_CONTAINERS already updated weekly
|
||||||
|
PROFILE_CRITICAL_CONTAINER_NAMES[emby] updated inline by the weekly sync window
|
||||||
|
PROFILE_CRITICAL_CONTAINER_NAMES[critical-data] updated inline by the weekly sync window
|
||||||
|
FALLBACK_<REMOTE>_TIER1..4 owned by the remote's update cycle
|
||||||
|
```
|
||||||
|
|
||||||
|
Stopped containers are never targeted in any mode — pulling for a stopped container adds
|
||||||
|
nothing, and it was most likely stopped deliberately.
|
||||||
|
|
||||||
|
The fallback exclusion is a correctness rule, not an optimisation. This server only runs
|
||||||
|
those containers during a fallback; the remote owns their version. If remainder updated them
|
||||||
|
independently and a handback then occurred, the remote's older image could meet data written
|
||||||
|
by the newer one.
|
||||||
|
|
||||||
|
**Ordering matters.** The update always runs *before* its matching restart so the restart
|
||||||
|
lands on the freshly pulled image. If a container's image actually changed, `docker_update.sh`
|
||||||
|
rebuilds it from its template (a plain `docker restart` reuses the image ID baked in at
|
||||||
|
creation time and would never pick up the new digest) and records it in
|
||||||
|
`DOCKER_UPDATE_REBUILT_*_FILE`. The restart script reads that file and skips those containers
|
||||||
|
rather than restarting them a second time — and discards the file as stale if it is older
|
||||||
|
than `DOCKER_UPDATE_REBUILT_STALE_HOURS`.
|
||||||
|
|
||||||
---
|
---
|
||||||
|
|
||||||
@@ -142,12 +176,22 @@ HOST1_NETWORK_CONNECT_CONTAINERS=()
|
|||||||
# Shared — applies to both servers
|
# Shared — applies to both servers
|
||||||
|
|
||||||
# ── Container updates ──────────────────────────────────────────────────
|
# ── Container updates ──────────────────────────────────────────────────
|
||||||
DAILY_CONTAINER_UPDATES=true
|
DAILY_CONTAINER_UPDATES=true # daily pull (DAILY_RESTART_CONTAINERS)
|
||||||
WEEKLY_REMAINING_UPDATES=true
|
WEEKLY_CONTAINER_UPDATES=true # weekly pull (WEEKLY_RESTART_CONTAINERS)
|
||||||
|
MONTHLY_REMAINING_UPDATES=true # monthly pull (everything else)
|
||||||
|
|
||||||
|
# Handoff between update and restart — written by docker_update.sh,
|
||||||
|
# read by the matching restart script so it skips containers already
|
||||||
|
# rebuilt onto a new image this run.
|
||||||
|
DOCKER_UPDATE_REBUILT_DAILY_FILE
|
||||||
|
DOCKER_UPDATE_REBUILT_WEEKLY_FILE
|
||||||
|
DOCKER_UPDATE_REBUILT_STALE_HOURS=12 # older than this = discarded, restart all
|
||||||
|
|
||||||
# ── Retry behaviour (shared by restart scripts) ────────────────────────
|
# ── Retry behaviour (shared by restart scripts) ────────────────────────
|
||||||
RETRY_COUNT=3 # retry attempts before marking failed
|
RETRY_COUNT=3 # retry attempts before marking failed
|
||||||
SLEEP=5 # seconds between retry attempts
|
SLEEP=5 # seconds between retry attempts
|
||||||
|
CONTAINER_DELAY # seconds between a dependency and its dependents
|
||||||
|
RESTART_VERIFY_WAIT=3 # settle time before verifying a restart stuck
|
||||||
|
|
||||||
# Watchdog thresholds (CPU, memory, HTTP, restart loop):
|
# Watchdog thresholds (CPU, memory, HTTP, restart loop):
|
||||||
# → see Watchdogs/Manual-Watchdogs.md
|
# → see Watchdogs/Manual-Watchdogs.md
|
||||||
@@ -211,7 +255,24 @@ All scripts support these standard flags:
|
|||||||
summary block: identity, duration, counts, and a status line. Per-container detail only
|
summary block: identity, duration, counts, and a status line. Per-container detail only
|
||||||
appears with `--log`. Warnings and errors are always visible regardless of `--log`.
|
appears with `--log`. Warnings and errors are always visible regardless of `--log`.
|
||||||
|
|
||||||
### `docker_update.sh --remainder`
|
### Script-specific flags
|
||||||
Switches to remainder mode — updates all running containers not in the managed daily/weekly
|
|
||||||
lists. Called by `weekly_sync_maintenance.sh`. Can be run manually to sweep containers
|
| Script | Flag | What it does |
|
||||||
that haven't been updated recently.
|
|--------|------|-------------|
|
||||||
|
| `docker_update.sh` | `--weekly` | Target `WEEKLY_RESTART_CONTAINERS`. Called by `weekly_sync_maintenance.sh` before the weekly restart. |
|
||||||
|
| `docker_update.sh` | `--remainder` | Target every running container not in the daily list, weekly list, emby/critical-data profiles, or fallback tiers. Called by `monthly_maintenance.sh`. Safe to run manually to sweep anything missed. |
|
||||||
|
| `media_cleaner.sh` *(Media/)* | `<anime\|media>` | Required positional profile — there is no default. |
|
||||||
|
|
||||||
|
`docker_update.sh` with no mode flag is normal mode: `DAILY_RESTART_CONTAINERS`, called by
|
||||||
|
`daily_sync_maintenance.sh` before the daily restart.
|
||||||
|
|
||||||
|
### Exit codes
|
||||||
|
|
||||||
|
| Code | Meaning |
|
||||||
|
|------|---------|
|
||||||
|
| `0` | Success, or nothing to do (empty list, disabled toggle, no running containers) |
|
||||||
|
| `1` | One or more containers failed — the summary names them |
|
||||||
|
|
||||||
|
`docker_update.sh` deliberately exits `0` even when pulls fail: a failed pull is not a reason
|
||||||
|
to abort the restart that follows, which simply proceeds on the existing image. The failure
|
||||||
|
is reported in the summary.
|
||||||
|
|||||||
@@ -178,12 +178,28 @@ windows so any downtime from restarts is absorbed by the window that's already h
|
|||||||
|
|
||||||
---
|
---
|
||||||
|
|
||||||
### 🔄 Image Currency — `docker_update.sh` + `docker_update_remaining.sh`
|
### 🔄 Image Currency — `docker_update.sh`
|
||||||
|
|
||||||
Keeps all container images current without manual intervention. Daily updates for the
|
One script, three modes — there is no separate remainder script.
|
||||||
auth/proxy stack (the containers that restart daily anyway — no extra downtime). Weekly
|
|
||||||
remainder pass for everything else — derives the target list automatically from `docker ps`
|
| Mode | Targets | Runs |
|
||||||
minus what was already updated, so there is no second list to maintain.
|
|------|---------|------|
|
||||||
|
| *(default)* | `DAILY_RESTART_CONTAINERS` | Daily, **before** `docker_daily_restart.sh` |
|
||||||
|
| `--weekly` | `WEEKLY_RESTART_CONTAINERS` | Weekly, **before** `docker_weekly_restart.sh` |
|
||||||
|
| `--remainder` | Everything running that is in neither list | Monthly, via `monthly_maintenance.sh` |
|
||||||
|
|
||||||
|
The update always runs *before* its matching restart, so the restart lands on the freshly
|
||||||
|
pulled image. Reversing that order would restart onto the old image and leave the new one
|
||||||
|
sitting unused until the next window.
|
||||||
|
|
||||||
|
Each mode reuses the restart list it pairs with rather than maintaining its own — add a
|
||||||
|
container to `DAILY_RESTART_CONTAINERS` once and it gets both the restart and the image pull.
|
||||||
|
Remainder mode needs no list at all: it derives its targets from `docker ps` minus the daily
|
||||||
|
list, the weekly list, the emby/critical-data sync-window profiles, and the fallback tiers.
|
||||||
|
|
||||||
|
Fallback containers are deliberately excluded from remainder mode. This server only runs them
|
||||||
|
during a fallback; the remote owns their version. Updating them here would risk the remote's
|
||||||
|
older image meeting data written by a newer one after a handback.
|
||||||
|
|
||||||
---
|
---
|
||||||
|
|
||||||
@@ -229,7 +245,7 @@ full coordination model between all four watchdogs.
|
|||||||
|--------|------|-------------|
|
|--------|------|-------------|
|
||||||
| `docker_daily_restart.sh` | Nightly proactive restart of degradation-prone containers | 1am via `daily_sync_maintenance.sh` |
|
| `docker_daily_restart.sh` | Nightly proactive restart of degradation-prone containers | 1am via `daily_sync_maintenance.sh` |
|
||||||
| `docker_weekly_restart.sh` | Weekly restart of less-critical services | 2:30am Sunday via `weekly_sync_maintenance.sh` |
|
| `docker_weekly_restart.sh` | Weekly restart of less-critical services | 2:30am Sunday via `weekly_sync_maintenance.sh` |
|
||||||
| `docker_update.sh` | Container image updates — daily list + weekly remainder mode | Daily before restart; weekly remainder pass |
|
| `docker_update.sh` | Container image updates — three modes (default / `--weekly` / `--remainder`) | Daily and weekly before each restart; monthly for the remainder |
|
||||||
| `docker_network_connect.sh` | Network existence + container connection enforcement | Every array start |
|
| `docker_network_connect.sh` | Network existence + container connection enforcement | Every array start |
|
||||||
| `docker_container_stop.sh` | Ordered container shutdown — graceful then forced | Called by `array_stopping.sh` |
|
| `docker_container_stop.sh` | Ordered container shutdown — graceful then forced | Called by `array_stopping.sh` |
|
||||||
| `downloaders_reset.sh` | Download client hygiene — slskd / SABnzbd / qBittorrent | Every 30min via `critical_sync_maintenance.sh` |
|
| `downloaders_reset.sh` | Download client hygiene — slskd / SABnzbd / qBittorrent | Every 30min via `critical_sync_maintenance.sh` |
|
||||||
@@ -257,8 +273,13 @@ Daily maintenance window (1am):
|
|||||||
|
|
||||||
Weekly maintenance window (2:30am Sunday):
|
Weekly maintenance window (2:30am Sunday):
|
||||||
weekly_sync_maintenance.sh
|
weekly_sync_maintenance.sh
|
||||||
├── docker_weekly_restart.sh ──── restart less-critical services
|
├── docker_update.sh --weekly ─── pull latest (WEEKLY_RESTART_CONTAINERS)
|
||||||
└── docker_update.sh --remainder ─ update containers not in managed lists
|
└── docker_weekly_restart.sh ──── restart onto the fresh image
|
||||||
|
|
||||||
|
Monthly maintenance window:
|
||||||
|
monthly_maintenance.sh
|
||||||
|
├── docker_update.sh --remainder update everything not in the managed lists
|
||||||
|
└── Tools/docker_prune_images.sh --all
|
||||||
|
|
||||||
Critical maintenance window (every 30min):
|
Critical maintenance window (every 30min):
|
||||||
critical_sync_maintenance.sh
|
critical_sync_maintenance.sh
|
||||||
@@ -266,5 +287,39 @@ Critical maintenance window (every 30min):
|
|||||||
|
|
||||||
Array stopping:
|
Array stopping:
|
||||||
array_stopping.sh
|
array_stopping.sh
|
||||||
└── docker_container_stop.sh ──── ordered graceful shutdown
|
└── docker_container_stop.sh ──── ordered graceful shutdown, verified per container
|
||||||
```
|
```
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## ━━━ SAFEGUARDS COMMON TO THIS FOLDER ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
|
||||||
|
|
||||||
|
Every script here talks to the Docker daemon, so they share the same protections. Each
|
||||||
|
script's own header documents its full set; these are the ones worth knowing folder-wide.
|
||||||
|
|
||||||
|
**Daemon health is checked, not assumed.** A hung daemon returns an empty container list,
|
||||||
|
which is indistinguishable from "no containers running". Without the check,
|
||||||
|
`docker_container_stop.sh` would report a clean shutdown that never happened, and
|
||||||
|
`docker_update.sh --remainder` would report "nothing to update" while doing nothing.
|
||||||
|
|
||||||
|
**Every docker call is timeout-wrapped.** A wedged daemon cannot stall a maintenance window
|
||||||
|
or hold a lock open. The one deliberate exception is `docker pull` — a large image
|
||||||
|
legitimately outlasts any sane timeout, and killing it mid-layer wastes the transfer.
|
||||||
|
|
||||||
|
**State is respected.** Running containers get restarted; stopped ones stay stopped. A
|
||||||
|
stopped container was almost certainly stopped on purpose, and none of these scripts has the
|
||||||
|
authority to overrule that.
|
||||||
|
|
||||||
|
**Restarts are verified, not assumed.** After each restart the container is re-checked once
|
||||||
|
it has had time to settle. A container that starts and immediately crashes is recorded as a
|
||||||
|
failure and notified — a restart that did not stick is never reported as success.
|
||||||
|
|
||||||
|
**Dependency ordering is shared with the watchdog.** Restarts follow
|
||||||
|
`HOST*_WATCHDOG_DEPENDENCIES`, with `CONTAINER_DELAY` between a dependency and its dependents,
|
||||||
|
so a dependent is never brought up while what it needs is still initialising.
|
||||||
|
|
||||||
|
**Locks prevent overlap.** Long windows can outlast their interval;
|
||||||
|
`downloaders_reset.sh` uses wait-mode because it runs every 30 minutes and the previous pass
|
||||||
|
may still be finishing.
|
||||||
|
|
||||||
|
---
|
||||||
|
|||||||
+33
-7
@@ -86,10 +86,25 @@ MEDIA_FILE_PATTERNS=(
|
|||||||
### Play State Sync
|
### Play State Sync
|
||||||
|
|
||||||
```bash
|
```bash
|
||||||
PLAY_STATE_SYNC_ENABLED=true # toggle entire sync
|
PLAY_SYNC_ENABLED=true # toggle entire sync
|
||||||
PLAY_STATE_SYNC_LOOKBACK_DAYS=30 # history window for played items
|
PLAY_SYNC_REMOTE=true # sync across hosts via Tailscale
|
||||||
|
# false = local servers only (this host's Emby + Jellyfin)
|
||||||
|
PLAY_SYNC_TYPES="Movie,Episode" # item types to sync — Audio excluded, music library too large
|
||||||
|
PLAY_SYNC_FAV_TYPES="MusicArtist,MusicAlbum,Movie,Series" # favourites, union sync, never unmarks
|
||||||
|
|
||||||
|
PLAY_SYNC_PROBE=true # skip per-item work when nothing changed since last run
|
||||||
|
PLAY_SYNC_PROBE_MAX_AGE_HOURS=24 # force a full comparison when the fingerprint is older
|
||||||
|
|
||||||
|
PLAY_SYNC_HANDBACK_RETRIES=5 # fallback handback: attempts before DNS cutover proceeds
|
||||||
|
PLAY_SYNC_HANDBACK_RETRY_DELAY=60 # seconds between those attempts
|
||||||
```
|
```
|
||||||
|
|
||||||
|
> **There is no date/lookback filter, and one must not be re-added.** An earlier version
|
||||||
|
> gated on played-date; it was removed once the real cost was measured — the 30-minute
|
||||||
|
> runtime was fork overhead per item, not API volume or item count. The fix was jq epoch
|
||||||
|
> parsing plus the response-hash probe below. Re-introducing a date window would reduce
|
||||||
|
> correctness (older items silently stop syncing) without meaningfully reducing runtime.
|
||||||
|
|
||||||
Emby and Jellyfin servers configured per-host:
|
Emby and Jellyfin servers configured per-host:
|
||||||
|
|
||||||
```bash
|
```bash
|
||||||
@@ -194,13 +209,24 @@ Show every file examined, not just those removed.
|
|||||||
### play_state_sync.sh
|
### play_state_sync.sh
|
||||||
|
|
||||||
`play_state_sync.sh`
|
`play_state_sync.sh`
|
||||||
Sync played/unplayed state and resume positions from local Emby to the remote Emby.
|
Sync played/unplayed state, resume positions and favourites across **every** configured Emby
|
||||||
Only items played within PLAY_STATE_SYNC_LOOKBACK_DAYS are synced.
|
and Jellyfin server — not just local→remote. Servers are discovered from every
|
||||||
|
`HOST*_TRANSCODE_SERVERS` entry, with remote hosts' localhost URLs rewritten to their
|
||||||
|
Tailscale IP. Newest `LastPlayedDate` wins; state only ever moves forward, never clears.
|
||||||
|
|
||||||
|
No age filter — every matched item is considered on every run. The **change probe** is what
|
||||||
|
keeps that cheap: the raw API responses are hashed and compared against the fingerprint from
|
||||||
|
the last successful run, and per-item processing is skipped entirely when nothing moved.
|
||||||
|
Fetches still happen every run, so nothing can be missed by the probe.
|
||||||
|
|
||||||
`play_state_sync.sh --full`
|
`play_state_sync.sh --full`
|
||||||
Ignore PLAY_STATE_SYNC_LOOKBACK_DAYS — sync all played items regardless of age. May be
|
Bypass the change probe and force the full per-item comparison even when the fingerprint
|
||||||
slow on large libraries. Use after a new Emby install or database restore to rebuild
|
matches. Use after a new Emby install or database restore, or when debugging a sync that
|
||||||
full play history.
|
appears to be skipping work it should be doing.
|
||||||
|
|
||||||
|
`play_state_sync.sh --wait`
|
||||||
|
Wait for an in-progress run instead of exiting. For manual runs that would otherwise be
|
||||||
|
skipped by the scheduled every-30-minute pass. Used by `fallback.sh` during handback.
|
||||||
|
|
||||||
`play_state_sync.sh --dry-run`
|
`play_state_sync.sh --dry-run`
|
||||||
Show what would be synced without writing any state.
|
Show what would be synced without writing any state.
|
||||||
|
|||||||
@@ -100,3 +100,43 @@ syncs between them.
|
|||||||
| `media_shares_permissions.sh` | Apply `nobody:users` ownership + correct permissions to all media shares | Daily — runs first |
|
| `media_shares_permissions.sh` | Apply `nobody:users` ownership + correct permissions to all media shares | Daily — runs first |
|
||||||
| `media_cleaner.sh` | Remove junk files (two profiles: `anime` + `media`) | Daily — runs before arr cleanup |
|
| `media_cleaner.sh` | Remove junk files (two profiles: `anime` + `media`) | Daily — runs before arr cleanup |
|
||||||
| `play_state_sync.sh` | Sync watched/played state + resume positions across Emby + Jellyfin | Every 30 min |
|
| `play_state_sync.sh` | Sync watched/played state + resume positions across Emby + Jellyfin | Every 30 min |
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## ━━━ THE ctime INVARIANT — READ BEFORE CHANGING PERMISSIONS ━━━
|
||||||
|
|
||||||
|
`media_shares_permissions.sh` applies every pass **conditionally** — it touches only entries
|
||||||
|
whose owner or mode is actually wrong. That is not an optimisation, and it must stay that way.
|
||||||
|
|
||||||
|
`chown` and `chmod` rewrite an inode's ctime **even when the value does not change**. A
|
||||||
|
blanket pass would therefore restamp every file in the library every night.
|
||||||
|
|
||||||
|
The arr cleanup scripts (`sonarr_cleanup.sh`, `radarr_cleanup.sh`, `lidarr_cleanup.sh`) gate
|
||||||
|
orphan deletion on ctime. mtime cannot substitute: an import preserves the release's original
|
||||||
|
timestamp, so mtime says nothing about when a file arrived here. Measured 2026-07-27 — of 400
|
||||||
|
files imported that week, **all 400 had mtimes over 7 days old, one of them 9613 days.**
|
||||||
|
|
||||||
|
So:
|
||||||
|
|
||||||
|
```
|
||||||
|
blanket chown/chmod → every ctime resets to today
|
||||||
|
→ no file ever appears older than *_ORPHAN_AGE
|
||||||
|
→ orphan collection silently stops
|
||||||
|
→ nothing errors, nothing warns, disk just fills
|
||||||
|
```
|
||||||
|
|
||||||
|
**The failure is invisible.** No script fails, no notification fires. The only symptom is
|
||||||
|
orphans quietly accumulating until a pool fills — which is exactly how the 755 GB / 89%-full
|
||||||
|
cache pool incident happened.
|
||||||
|
|
||||||
|
Two rules follow, and both are load-bearing:
|
||||||
|
|
||||||
|
1. **`media_shares_permissions.sh` passes stay conditional.** Making any of them unconditional
|
||||||
|
breaks orphan collection ecosystem-wide.
|
||||||
|
2. **`Tools/bulk_permissions_repair.sh` is unconditional by design** — it exists to repair
|
||||||
|
known-wrong paths where correctness beats preserving a clock. That is precisely why it is a
|
||||||
|
manual, targeted tool and not scheduled. Pointing it at a whole media root pauses orphan
|
||||||
|
collection there for `*_ORPHAN_AGE` days.
|
||||||
|
|
||||||
|
Both scripts' headers carry this warning too. If you are reading this because you are about to
|
||||||
|
"simplify" the permissions job, this is the thing that breaks.
|
||||||
|
|||||||
@@ -234,3 +234,53 @@ Every 15 minutes — watchdog_orchestrator.sh fires:
|
|||||||
| `NETWORK_WATCHDOG_NPM_STATE_FILE` | `System/network_watchdog.sh` | itself | NPM proxy strike count |
|
| `NETWORK_WATCHDOG_NPM_STATE_FILE` | `System/network_watchdog.sh` | itself | NPM proxy strike count |
|
||||||
|
|
||||||
All state files are in `$STATE_DIR` (adapts to storage mode). See `master.conf` for actual variable values.
|
All state files are in `$STATE_DIR` (adapts to storage mode). See `master.conf` for actual variable values.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## ━━━ SAFEGUARDS COMMON TO THIS FOLDER ━━━
|
||||||
|
|
||||||
|
These run unattended every 15 minutes and hold the only irreversible remedy in the
|
||||||
|
ecosystem, so the guards matter as much as the checks. Each script's header documents its
|
||||||
|
full set; these are the folder-wide ones.
|
||||||
|
|
||||||
|
**Nothing acts on a single reading.** Every check that can trigger an action is strike-gated.
|
||||||
|
One bad sample is a reading; N consecutive samples is a condition. Strikes clear themselves
|
||||||
|
when the condition resolves, so a transient blip never accumulates toward an alarm across
|
||||||
|
unrelated weeks.
|
||||||
|
|
||||||
|
**Escalation is asymmetric on purpose.** Protective actions happen fast; undoing them happens
|
||||||
|
slowly. `resource_watchdog.sh` throttles before it pauses and pauses before it stops, then
|
||||||
|
requires `RW_RECOVER_CYCLES` clear cycles and de-escalates one level at a time. Recovering on
|
||||||
|
a single good reading would flap under sustained load.
|
||||||
|
|
||||||
|
**Ownership boundaries are enforced, not conventional.** Container health belongs to
|
||||||
|
`docker_watchdog.sh`; `stability_watchdog.sh` deliberately does not check it, and its Docker
|
||||||
|
daemon check writes a flag for `docker_watchdog.sh` rather than acting. Two watchdogs
|
||||||
|
remediating the same subsystem would race.
|
||||||
|
|
||||||
|
**Watchdogs coordinate through state files.** `resource_watchdog.sh` publishes
|
||||||
|
`mem_shutdown_active` so `docker_watchdog.sh` defers restarts — otherwise one would stop
|
||||||
|
containers to free RAM while the other restarted them to restore health. The reader applies a
|
||||||
|
staleness guard so a crashed writer cannot suppress restarts forever.
|
||||||
|
|
||||||
|
**Data safety outranks uptime.** `stability_watchdog.sh` aborts a reboot while a ZFS pool is
|
||||||
|
unhealthy, parity is running, or the mover is active. Only Tier 1 — conditions the system
|
||||||
|
cannot recover from and which worsen every cycle — bypasses that.
|
||||||
|
|
||||||
|
**Reboot loops end in shutdown, not more reboots.** No more than
|
||||||
|
`SYS_WATCHDOG_REBOOT_LIMIT` reboots within `SYS_WATCHDOG_REBOOT_WINDOW_HRS`. On hitting the
|
||||||
|
limit the host powers off: a fault that survives repeated reboots will not be fixed by more
|
||||||
|
of them, and a cleanly-down box is better than one cycling endlessly.
|
||||||
|
|
||||||
|
**An aborted reboot restores what it stopped.** An EXIT trap is armed the moment containers
|
||||||
|
start being stopped for a reboot and disarmed only once the reboot commits. If the script
|
||||||
|
dies in between, everything it stopped comes back — the failure mode is a running system,
|
||||||
|
never a host left stripped with no reboot.
|
||||||
|
|
||||||
|
**An unconfigured job list fails loudly.** An empty `WATCHDOG_ORCHESTRATOR_SCRIPTS` or
|
||||||
|
`SYSTEM_WATCHDOG_SCRIPTS` would otherwise report "0/0 passed" and exit 0 every cycle —
|
||||||
|
indistinguishable from healthy while nothing is monitored at all.
|
||||||
|
|
||||||
|
**Verify the path, not the process.** `network_watchdog.sh` checks NPM by fetching an external
|
||||||
|
URL rather than asking whether the container is running. A running container behind broken DNS
|
||||||
|
or a broken upstream still serves nothing.
|
||||||
|
|||||||
Reference in New Issue
Block a user