Compare commits
3
Commits
| Author | SHA1 | Date | |
|---|---|---|---|
|
|
7a8fbbe06a | ||
|
|
53aa72d38c | ||
|
|
8a2707ee37 |
@@ -200,6 +200,75 @@ Lidarr configured — no HOST1_LIDARR_URL means nothing runs.
|
|||||||
| `playback_aware_lidarr_discovery.sh` | Behavior-driven music discovery — Emby plays → Last.fm similar → Lidarr | Weekly |
|
| `playback_aware_lidarr_discovery.sh` | Behavior-driven music discovery — Emby plays → Last.fm similar → Lidarr | Weekly |
|
||||||
| `playback_aware_radarr_discovery.sh` | Behavior-driven movie discovery — Emby watches → TMDB recommendations → Radarr | Weekly |
|
| `playback_aware_radarr_discovery.sh` | Behavior-driven movie discovery — Emby watches → TMDB recommendations → Radarr | Weekly |
|
||||||
| `playback_aware_sonarr_discovery.sh` | Behavior-driven TV discovery — Emby episodes → TMDB TV recommendations → Sonarr | Weekly |
|
| `playback_aware_sonarr_discovery.sh` | Behavior-driven TV discovery — Emby episodes → TMDB TV recommendations → Sonarr | Weekly |
|
||||||
|
| `arr_download_orphan_cleaner.sh` | Clear orphaned completed downloads out of the SABnzbd Completed folders | Daily |
|
||||||
|
| `sonarr_classification_scan.sh` | Detect series sitting in the wrong root (anime / kids / general); `--move` acts | Daily |
|
||||||
|
| `radarr_classification_scan.sh` | Same for movies, plus junk-metadata detection via `--remove-junk` | Daily |
|
||||||
|
| `lidarr_duplicate_artist_cleanup.sh` | Remove phantom zero-file duplicate artists; flag real ones for review | Daily |
|
||||||
|
| `arr_cache_prefill.sh` | Warm the shared tracked-data cache so consumers never read cold | Array start + every 4hr |
|
||||||
|
| `arr_corruption_scan.sh` | ffprobe every tracked video for corrupt headers; `--remediate` deletes + re-searches | Weekly |
|
||||||
|
| `arr_full_rescan.sh` | Force a real disk↔database reconciliation on all three arrs | Weekly |
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## ━━━ THE NEWER LAYERS ━━━
|
||||||
|
|
||||||
|
The original folder was "delete what the arrs no longer track". These were added as distinct
|
||||||
|
failure modes surfaced — each exists because something went wrong that the cleanups could not
|
||||||
|
have caught.
|
||||||
|
|
||||||
|
### 🗑️ Download-Side Orphans — `arr_download_orphan_cleaner.sh`
|
||||||
|
|
||||||
|
Every cleanup script here walks the **library** side. Nothing walked the **download** side —
|
||||||
|
so completed downloads the arrs had stopped tracking accumulated in SABnzbd's Completed
|
||||||
|
folders indefinitely. Discovered as **755 GB of orphaned TV downloads, oldest from 2022**,
|
||||||
|
filling the cache pool to 89%.
|
||||||
|
|
||||||
|
Classifies every entry as TRACKED / RECENT / JUNK / REDUNDANT / IMPORTABLE / UNMATCHED and
|
||||||
|
acts only on the ones it can justify. The queue is a hard gate: if it cannot be read, the arr
|
||||||
|
is skipped entirely, because without it there is no way to tell an active import from an
|
||||||
|
orphan. A run total over `DOWNLOAD_ORPHAN_MAX_DELETE_GB` aborts — an abnormally large delete
|
||||||
|
is the visible symptom of a partial queue fetch.
|
||||||
|
|
||||||
|
### 🎭 Wrong-Root Detection — `sonarr_classification_scan.sh` + `radarr_classification_scan.sh`
|
||||||
|
|
||||||
|
Overseerr lets any user request content into the wrong root folder — kids shows into general
|
||||||
|
TV, anime into Kids_Tv_Shows. These classify every item from metadata alone (genre,
|
||||||
|
certification, network/studio, original language) and report where the computed classification
|
||||||
|
disagrees with the folder the item actually sits in.
|
||||||
|
|
||||||
|
Report-only by default. `--move` acts on forward misplacements and adult-content-in-kids-root
|
||||||
|
leaks. It deliberately does **not** move non-anime content out of the anime root — deliberate
|
||||||
|
style placements (Western animation grouped with anime by choice) are genuine judgment calls.
|
||||||
|
|
||||||
|
Both poll the arr's async move command to completion before verifying, because `moveFiles=true`
|
||||||
|
flips the database instantly while the physical move is still queued behind others.
|
||||||
|
|
||||||
|
### 🎨 Phantom Artists — `lidarr_duplicate_artist_cleanup.sh`
|
||||||
|
|
||||||
|
MusicBrainz duplicates leave two Lidarr entries for one artist, one holding the files and one
|
||||||
|
holding nothing. Removes only the zero-file side, with `deleteFiles=false` so nothing on disk
|
||||||
|
is touched. Pairs where both sides hold files are flagged for review, never auto-resolved.
|
||||||
|
|
||||||
|
Gated on the tracked-count floor shared with `lidarr_cleanup.sh` — during a library-wide desync
|
||||||
|
both sides of a real duplicate can read as zero-file phantoms.
|
||||||
|
|
||||||
|
### 🩺 Corruption + Reconciliation — `arr_corruption_scan.sh` + `arr_full_rescan.sh`
|
||||||
|
|
||||||
|
`arr_corruption_scan.sh` ffprobes tracked video files for corrupt headers. Report-only unless
|
||||||
|
`--remediate`, which deletes the file record and triggers an explicit re-search. Requires
|
||||||
|
repeat detections across separate runs before acting, so a transient probe failure cannot
|
||||||
|
delete a healthy file.
|
||||||
|
|
||||||
|
`arr_full_rescan.sh` forces a genuine disk↔database reconciliation. Organic scans only touch
|
||||||
|
files involved in an import, so an untouched library silently drifts — confirmed when Lidarr
|
||||||
|
reported **~23% of its true track count** for 1,004 of 1,357 artists with no scan running and
|
||||||
|
every file present on disk.
|
||||||
|
|
||||||
|
### ⚡ Cache Warmth — `arr_cache_prefill.sh`
|
||||||
|
|
||||||
|
Populates the shared tracked-data cache at array start and every 4 hours, so consumers never
|
||||||
|
pay a cold fetch. Pure enhancement: nothing depends on it having run, and every consumer still
|
||||||
|
writes through on a cold cache.
|
||||||
|
|
||||||
---
|
---
|
||||||
|
|
||||||
|
|||||||
@@ -94,20 +94,54 @@ HOST1_NETWORK_CONNECT_CONTAINERS=(
|
|||||||
|
|
||||||
```bash
|
```bash
|
||||||
# master.conf
|
# master.conf
|
||||||
DAILY_CONTAINER_UPDATES=true # enable/disable daily image pull
|
DAILY_CONTAINER_UPDATES=true # enable/disable the daily image pull
|
||||||
# docker_daily_restart.sh still runs regardless
|
# docker_daily_restart.sh still runs regardless
|
||||||
# update and restart are independent
|
# update and restart are independent
|
||||||
|
|
||||||
WEEKLY_REMAINING_UPDATES=true # enable/disable weekly remainder pull + prune
|
WEEKLY_CONTAINER_UPDATES=true # enable/disable the weekly image pull
|
||||||
# to disable: set false or remove from WEEKLY_MAINTENANCE_SCRIPTS
|
# docker_weekly_restart.sh still runs regardless
|
||||||
|
|
||||||
|
MONTHLY_REMAINING_UPDATES=true # enable/disable the monthly remainder pull
|
||||||
|
# to disable: set false, or remove
|
||||||
|
# "docker_update.sh --remainder" from
|
||||||
|
# MONTHLY_MAINTENANCE_SCRIPTS
|
||||||
```
|
```
|
||||||
|
|
||||||
`docker_update.sh` in normal mode targets `DAILY_RESTART_CONTAINERS` — the same list
|
There is one update script, `docker_update.sh`, with three modes. Each reuses the restart
|
||||||
used by `docker_daily_restart.sh`. No second list to maintain.
|
list it pairs with, so there is no second list to maintain:
|
||||||
|
|
||||||
`docker_update_remaining.sh` derives its target list automatically:
|
| Mode | Targets | Runs |
|
||||||
all running containers minus `DAILY_RESTART_CONTAINERS` minus `WEEKLY_RESTART_CONTAINERS`.
|
|------|---------|------|
|
||||||
Everything gets updated at least once per week with no explicit configuration.
|
| *(default)* | `DAILY_RESTART_CONTAINERS` | Daily, before `docker_daily_restart.sh` |
|
||||||
|
| `--weekly` | `WEEKLY_RESTART_CONTAINERS` | Weekly, before `docker_weekly_restart.sh` |
|
||||||
|
| `--remainder` | derived, see below | Monthly, via `MONTHLY_MAINTENANCE_SCRIPTS` |
|
||||||
|
|
||||||
|
Remainder mode needs no configuration at all. It takes every **running** container and
|
||||||
|
subtracts:
|
||||||
|
|
||||||
|
```
|
||||||
|
DAILY_RESTART_CONTAINERS already updated daily
|
||||||
|
WEEKLY_RESTART_CONTAINERS already updated weekly
|
||||||
|
PROFILE_CRITICAL_CONTAINER_NAMES[emby] updated inline by the weekly sync window
|
||||||
|
PROFILE_CRITICAL_CONTAINER_NAMES[critical-data] updated inline by the weekly sync window
|
||||||
|
FALLBACK_<REMOTE>_TIER1..4 owned by the remote's update cycle
|
||||||
|
```
|
||||||
|
|
||||||
|
Stopped containers are never targeted in any mode — pulling for a stopped container adds
|
||||||
|
nothing, and it was most likely stopped deliberately.
|
||||||
|
|
||||||
|
The fallback exclusion is a correctness rule, not an optimisation. This server only runs
|
||||||
|
those containers during a fallback; the remote owns their version. If remainder updated them
|
||||||
|
independently and a handback then occurred, the remote's older image could meet data written
|
||||||
|
by the newer one.
|
||||||
|
|
||||||
|
**Ordering matters.** The update always runs *before* its matching restart so the restart
|
||||||
|
lands on the freshly pulled image. If a container's image actually changed, `docker_update.sh`
|
||||||
|
rebuilds it from its template (a plain `docker restart` reuses the image ID baked in at
|
||||||
|
creation time and would never pick up the new digest) and records it in
|
||||||
|
`DOCKER_UPDATE_REBUILT_*_FILE`. The restart script reads that file and skips those containers
|
||||||
|
rather than restarting them a second time — and discards the file as stale if it is older
|
||||||
|
than `DOCKER_UPDATE_REBUILT_STALE_HOURS`.
|
||||||
|
|
||||||
---
|
---
|
||||||
|
|
||||||
@@ -142,12 +176,22 @@ HOST1_NETWORK_CONNECT_CONTAINERS=()
|
|||||||
# Shared — applies to both servers
|
# Shared — applies to both servers
|
||||||
|
|
||||||
# ── Container updates ──────────────────────────────────────────────────
|
# ── Container updates ──────────────────────────────────────────────────
|
||||||
DAILY_CONTAINER_UPDATES=true
|
DAILY_CONTAINER_UPDATES=true # daily pull (DAILY_RESTART_CONTAINERS)
|
||||||
WEEKLY_REMAINING_UPDATES=true
|
WEEKLY_CONTAINER_UPDATES=true # weekly pull (WEEKLY_RESTART_CONTAINERS)
|
||||||
|
MONTHLY_REMAINING_UPDATES=true # monthly pull (everything else)
|
||||||
|
|
||||||
|
# Handoff between update and restart — written by docker_update.sh,
|
||||||
|
# read by the matching restart script so it skips containers already
|
||||||
|
# rebuilt onto a new image this run.
|
||||||
|
DOCKER_UPDATE_REBUILT_DAILY_FILE
|
||||||
|
DOCKER_UPDATE_REBUILT_WEEKLY_FILE
|
||||||
|
DOCKER_UPDATE_REBUILT_STALE_HOURS=12 # older than this = discarded, restart all
|
||||||
|
|
||||||
# ── Retry behaviour (shared by restart scripts) ────────────────────────
|
# ── Retry behaviour (shared by restart scripts) ────────────────────────
|
||||||
RETRY_COUNT=3 # retry attempts before marking failed
|
RETRY_COUNT=3 # retry attempts before marking failed
|
||||||
SLEEP=5 # seconds between retry attempts
|
SLEEP=5 # seconds between retry attempts
|
||||||
|
CONTAINER_DELAY # seconds between a dependency and its dependents
|
||||||
|
RESTART_VERIFY_WAIT=3 # settle time before verifying a restart stuck
|
||||||
|
|
||||||
# Watchdog thresholds (CPU, memory, HTTP, restart loop):
|
# Watchdog thresholds (CPU, memory, HTTP, restart loop):
|
||||||
# → see Watchdogs/Manual-Watchdogs.md
|
# → see Watchdogs/Manual-Watchdogs.md
|
||||||
@@ -211,7 +255,24 @@ All scripts support these standard flags:
|
|||||||
summary block: identity, duration, counts, and a status line. Per-container detail only
|
summary block: identity, duration, counts, and a status line. Per-container detail only
|
||||||
appears with `--log`. Warnings and errors are always visible regardless of `--log`.
|
appears with `--log`. Warnings and errors are always visible regardless of `--log`.
|
||||||
|
|
||||||
### `docker_update.sh --remainder`
|
### Script-specific flags
|
||||||
Switches to remainder mode — updates all running containers not in the managed daily/weekly
|
|
||||||
lists. Called by `weekly_sync_maintenance.sh`. Can be run manually to sweep containers
|
| Script | Flag | What it does |
|
||||||
that haven't been updated recently.
|
|--------|------|-------------|
|
||||||
|
| `docker_update.sh` | `--weekly` | Target `WEEKLY_RESTART_CONTAINERS`. Called by `weekly_sync_maintenance.sh` before the weekly restart. |
|
||||||
|
| `docker_update.sh` | `--remainder` | Target every running container not in the daily list, weekly list, emby/critical-data profiles, or fallback tiers. Called by `monthly_maintenance.sh`. Safe to run manually to sweep anything missed. |
|
||||||
|
| `media_cleaner.sh` *(Media/)* | `<anime\|media>` | Required positional profile — there is no default. |
|
||||||
|
|
||||||
|
`docker_update.sh` with no mode flag is normal mode: `DAILY_RESTART_CONTAINERS`, called by
|
||||||
|
`daily_sync_maintenance.sh` before the daily restart.
|
||||||
|
|
||||||
|
### Exit codes
|
||||||
|
|
||||||
|
| Code | Meaning |
|
||||||
|
|------|---------|
|
||||||
|
| `0` | Success, or nothing to do (empty list, disabled toggle, no running containers) |
|
||||||
|
| `1` | One or more containers failed — the summary names them |
|
||||||
|
|
||||||
|
`docker_update.sh` deliberately exits `0` even when pulls fail: a failed pull is not a reason
|
||||||
|
to abort the restart that follows, which simply proceeds on the existing image. The failure
|
||||||
|
is reported in the summary.
|
||||||
|
|||||||
@@ -178,12 +178,28 @@ windows so any downtime from restarts is absorbed by the window that's already h
|
|||||||
|
|
||||||
---
|
---
|
||||||
|
|
||||||
### 🔄 Image Currency — `docker_update.sh` + `docker_update_remaining.sh`
|
### 🔄 Image Currency — `docker_update.sh`
|
||||||
|
|
||||||
Keeps all container images current without manual intervention. Daily updates for the
|
One script, three modes — there is no separate remainder script.
|
||||||
auth/proxy stack (the containers that restart daily anyway — no extra downtime). Weekly
|
|
||||||
remainder pass for everything else — derives the target list automatically from `docker ps`
|
| Mode | Targets | Runs |
|
||||||
minus what was already updated, so there is no second list to maintain.
|
|------|---------|------|
|
||||||
|
| *(default)* | `DAILY_RESTART_CONTAINERS` | Daily, **before** `docker_daily_restart.sh` |
|
||||||
|
| `--weekly` | `WEEKLY_RESTART_CONTAINERS` | Weekly, **before** `docker_weekly_restart.sh` |
|
||||||
|
| `--remainder` | Everything running that is in neither list | Monthly, via `monthly_maintenance.sh` |
|
||||||
|
|
||||||
|
The update always runs *before* its matching restart, so the restart lands on the freshly
|
||||||
|
pulled image. Reversing that order would restart onto the old image and leave the new one
|
||||||
|
sitting unused until the next window.
|
||||||
|
|
||||||
|
Each mode reuses the restart list it pairs with rather than maintaining its own — add a
|
||||||
|
container to `DAILY_RESTART_CONTAINERS` once and it gets both the restart and the image pull.
|
||||||
|
Remainder mode needs no list at all: it derives its targets from `docker ps` minus the daily
|
||||||
|
list, the weekly list, the emby/critical-data sync-window profiles, and the fallback tiers.
|
||||||
|
|
||||||
|
Fallback containers are deliberately excluded from remainder mode. This server only runs them
|
||||||
|
during a fallback; the remote owns their version. Updating them here would risk the remote's
|
||||||
|
older image meeting data written by a newer one after a handback.
|
||||||
|
|
||||||
---
|
---
|
||||||
|
|
||||||
@@ -229,7 +245,7 @@ full coordination model between all four watchdogs.
|
|||||||
|--------|------|-------------|
|
|--------|------|-------------|
|
||||||
| `docker_daily_restart.sh` | Nightly proactive restart of degradation-prone containers | 1am via `daily_sync_maintenance.sh` |
|
| `docker_daily_restart.sh` | Nightly proactive restart of degradation-prone containers | 1am via `daily_sync_maintenance.sh` |
|
||||||
| `docker_weekly_restart.sh` | Weekly restart of less-critical services | 2:30am Sunday via `weekly_sync_maintenance.sh` |
|
| `docker_weekly_restart.sh` | Weekly restart of less-critical services | 2:30am Sunday via `weekly_sync_maintenance.sh` |
|
||||||
| `docker_update.sh` | Container image updates — daily list + weekly remainder mode | Daily before restart; weekly remainder pass |
|
| `docker_update.sh` | Container image updates — three modes (default / `--weekly` / `--remainder`) | Daily and weekly before each restart; monthly for the remainder |
|
||||||
| `docker_network_connect.sh` | Network existence + container connection enforcement | Every array start |
|
| `docker_network_connect.sh` | Network existence + container connection enforcement | Every array start |
|
||||||
| `docker_container_stop.sh` | Ordered container shutdown — graceful then forced | Called by `array_stopping.sh` |
|
| `docker_container_stop.sh` | Ordered container shutdown — graceful then forced | Called by `array_stopping.sh` |
|
||||||
| `downloaders_reset.sh` | Download client hygiene — slskd / SABnzbd / qBittorrent | Every 30min via `critical_sync_maintenance.sh` |
|
| `downloaders_reset.sh` | Download client hygiene — slskd / SABnzbd / qBittorrent | Every 30min via `critical_sync_maintenance.sh` |
|
||||||
@@ -257,8 +273,13 @@ Daily maintenance window (1am):
|
|||||||
|
|
||||||
Weekly maintenance window (2:30am Sunday):
|
Weekly maintenance window (2:30am Sunday):
|
||||||
weekly_sync_maintenance.sh
|
weekly_sync_maintenance.sh
|
||||||
├── docker_weekly_restart.sh ──── restart less-critical services
|
├── docker_update.sh --weekly ─── pull latest (WEEKLY_RESTART_CONTAINERS)
|
||||||
└── docker_update.sh --remainder ─ update containers not in managed lists
|
└── docker_weekly_restart.sh ──── restart onto the fresh image
|
||||||
|
|
||||||
|
Monthly maintenance window:
|
||||||
|
monthly_maintenance.sh
|
||||||
|
├── docker_update.sh --remainder update everything not in the managed lists
|
||||||
|
└── Tools/docker_prune_images.sh --all
|
||||||
|
|
||||||
Critical maintenance window (every 30min):
|
Critical maintenance window (every 30min):
|
||||||
critical_sync_maintenance.sh
|
critical_sync_maintenance.sh
|
||||||
@@ -266,5 +287,39 @@ Critical maintenance window (every 30min):
|
|||||||
|
|
||||||
Array stopping:
|
Array stopping:
|
||||||
array_stopping.sh
|
array_stopping.sh
|
||||||
└── docker_container_stop.sh ──── ordered graceful shutdown
|
└── docker_container_stop.sh ──── ordered graceful shutdown, verified per container
|
||||||
```
|
```
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## ━━━ SAFEGUARDS COMMON TO THIS FOLDER ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
|
||||||
|
|
||||||
|
Every script here talks to the Docker daemon, so they share the same protections. Each
|
||||||
|
script's own header documents its full set; these are the ones worth knowing folder-wide.
|
||||||
|
|
||||||
|
**Daemon health is checked, not assumed.** A hung daemon returns an empty container list,
|
||||||
|
which is indistinguishable from "no containers running". Without the check,
|
||||||
|
`docker_container_stop.sh` would report a clean shutdown that never happened, and
|
||||||
|
`docker_update.sh --remainder` would report "nothing to update" while doing nothing.
|
||||||
|
|
||||||
|
**Every docker call is timeout-wrapped.** A wedged daemon cannot stall a maintenance window
|
||||||
|
or hold a lock open. The one deliberate exception is `docker pull` — a large image
|
||||||
|
legitimately outlasts any sane timeout, and killing it mid-layer wastes the transfer.
|
||||||
|
|
||||||
|
**State is respected.** Running containers get restarted; stopped ones stay stopped. A
|
||||||
|
stopped container was almost certainly stopped on purpose, and none of these scripts has the
|
||||||
|
authority to overrule that.
|
||||||
|
|
||||||
|
**Restarts are verified, not assumed.** After each restart the container is re-checked once
|
||||||
|
it has had time to settle. A container that starts and immediately crashes is recorded as a
|
||||||
|
failure and notified — a restart that did not stick is never reported as success.
|
||||||
|
|
||||||
|
**Dependency ordering is shared with the watchdog.** Restarts follow
|
||||||
|
`HOST*_WATCHDOG_DEPENDENCIES`, with `CONTAINER_DELAY` between a dependency and its dependents,
|
||||||
|
so a dependent is never brought up while what it needs is still initialising.
|
||||||
|
|
||||||
|
**Locks prevent overlap.** Long windows can outlast their interval;
|
||||||
|
`downloaders_reset.sh` uses wait-mode because it runs every 30 minutes and the previous pass
|
||||||
|
may still be finishing.
|
||||||
|
|
||||||
|
---
|
||||||
|
|||||||
+42
-24
@@ -105,18 +105,36 @@ internet connectivity. Most hosts leave this empty.
|
|||||||
---
|
---
|
||||||
|
|
||||||
```
|
```
|
||||||
FALLBACK_HOST*_COVERS_HOST*_TIER1=(...)
|
FALLBACK_HOST*_TIER1=(...)
|
||||||
FALLBACK_HOST*_COVERS_HOST*_TIER2=(...)
|
FALLBACK_HOST*_TIER2=(...)
|
||||||
FALLBACK_HOST*_COVERS_HOST*_TIER3=(...)
|
FALLBACK_HOST*_TIER3=(...)
|
||||||
FALLBACK_HOST*_COVERS_HOST*_TIER4=(...)
|
FALLBACK_HOST*_TIER4=(...)
|
||||||
```
|
```
|
||||||
Containers this host starts for the remote host when the remote is down. TIER1 starts
|
|
||||||
immediately. TIER2–4 activate after the corresponding delay thresholds.
|
|
||||||
|
|
||||||
Variable pattern: `FALLBACK_${MY_ID}_COVERS_${REMOTE_ID}_TIER${N}`
|
**Each host declares its OWN services, in its OWN conf.** `FALLBACK_HOST2_TIER1` lives in
|
||||||
|
`host2.conf` and lists HOST2's vital containers — it is not a list HOST1 maintains.
|
||||||
|
|
||||||
The DDNS container for the remote's domain must be the first entry in TIER1 — DNS
|
At runtime `fallback.sh` reads its **partner's** list:
|
||||||
coverage before anything else.
|
|
||||||
|
```bash
|
||||||
|
get_tier_containers() {
|
||||||
|
local var_name="FALLBACK_${REMOTE_ID}_TIER${tier}" # note: REMOTE_ID, not MY_ID
|
||||||
|
```
|
||||||
|
|
||||||
|
So HOST1, covering HOST2, reads `FALLBACK_HOST2_TIER1` — a variable defined in `host2.conf`
|
||||||
|
and delivered to HOST1 through the partner conf cache (`conf_sync.sh`), because sparse
|
||||||
|
checkout means HOST1 never pulls `host2.conf` from git.
|
||||||
|
|
||||||
|
This is why the naming is what it is. The alternative — each host keeping a copy of its
|
||||||
|
partner's container list — would need editing on both machines every time either one changed
|
||||||
|
a service, and the two copies would silently diverge. Declaring once, on the host that owns
|
||||||
|
the services, means a host is always the authority on what covering it requires.
|
||||||
|
|
||||||
|
TIER1 starts immediately. TIER2–4 activate after their delay thresholds
|
||||||
|
(`HOST*_TIER2_DELAY` and friends, also in that host's own conf).
|
||||||
|
|
||||||
|
The DDNS container for that host's domain must be the **first entry in TIER1** — DNS coverage
|
||||||
|
before anything else.
|
||||||
|
|
||||||
---
|
---
|
||||||
|
|
||||||
@@ -181,15 +199,15 @@ FALLBACK_TEST_HANDBACK_WAIT=300
|
|||||||
|
|
||||||
---
|
---
|
||||||
|
|
||||||
### host2.conf — HOST2 covering HOST1
|
### host1.conf — HOST1's own services (started by HOST2 when HOST1 is down)
|
||||||
|
|
||||||
```bash
|
```bash
|
||||||
HOST2_DDNS_CONTAINERS=("Gmer4Lfe.us-DDNS")
|
HOST1_DDNS_CONTAINERS=("Gmer4Lfe.com-DDNS")
|
||||||
|
|
||||||
FALLBACK_HOST2_STOP_ON_NO_NET=()
|
FALLBACK_HOST1_STOP_ON_NO_NET=()
|
||||||
|
|
||||||
# Tier 1 — immediate (vital services + Live TV)
|
# Tier 1 — immediate (vital services + Live TV)
|
||||||
FALLBACK_HOST2_COVERS_HOST1_TIER1=(
|
FALLBACK_HOST1_TIER1=(
|
||||||
"Gmer4Lfe.com-DDNS" # ALWAYS FIRST — DNS coverage before anything else
|
"Gmer4Lfe.com-DDNS" # ALWAYS FIRST — DNS coverage before anything else
|
||||||
"Emby" # media server — people are watching
|
"Emby" # media server — people are watching
|
||||||
"NginxProxyManager" # reverse proxy — all external access routes through this
|
"NginxProxyManager" # reverse proxy — all external access routes through this
|
||||||
@@ -205,7 +223,7 @@ FALLBACK_HOST2_COVERS_HOST1_TIER1=(
|
|||||||
)
|
)
|
||||||
|
|
||||||
# Tier 2 — after 4 hours (shared productivity services)
|
# Tier 2 — after 4 hours (shared productivity services)
|
||||||
FALLBACK_HOST2_COVERS_HOST1_TIER2=(
|
FALLBACK_HOST1_TIER2=(
|
||||||
"Postgres-NextCloud" # must start before NextCloud
|
"Postgres-NextCloud" # must start before NextCloud
|
||||||
"NextCloud"
|
"NextCloud"
|
||||||
"PostgreSQL-Immich" # must start before Immich
|
"PostgreSQL-Immich" # must start before Immich
|
||||||
@@ -214,7 +232,7 @@ FALLBACK_HOST2_COVERS_HOST1_TIER2=(
|
|||||||
)
|
)
|
||||||
|
|
||||||
# Tier 3 — after 12 hours (secondary services)
|
# Tier 3 — after 12 hours (secondary services)
|
||||||
FALLBACK_HOST2_COVERS_HOST1_TIER3=(
|
FALLBACK_HOST1_TIER3=(
|
||||||
"Organizrv2-Gmer4Lfe"
|
"Organizrv2-Gmer4Lfe"
|
||||||
"AdGuard-Home"
|
"AdGuard-Home"
|
||||||
"UptimeKuma"
|
"UptimeKuma"
|
||||||
@@ -223,7 +241,7 @@ FALLBACK_HOST2_COVERS_HOST1_TIER3=(
|
|||||||
)
|
)
|
||||||
|
|
||||||
# Tier 4 — after 24 hours (arrs + downloaders)
|
# Tier 4 — after 24 hours (arrs + downloaders)
|
||||||
FALLBACK_HOST2_COVERS_HOST1_TIER4=(
|
FALLBACK_HOST1_TIER4=(
|
||||||
"Sonarr-Gmer4Lfe"
|
"Sonarr-Gmer4Lfe"
|
||||||
"Radarr-Gmer4Lfe"
|
"Radarr-Gmer4Lfe"
|
||||||
"Lidarr-Gmer4Lfe"
|
"Lidarr-Gmer4Lfe"
|
||||||
@@ -259,22 +277,22 @@ FALLBACK_HOST1_WRITEBACK_TIER3=(
|
|||||||
|
|
||||||
---
|
---
|
||||||
|
|
||||||
### host1.conf — HOST1 covering HOST2
|
### host2.conf — HOST2's own services (started by HOST1 when HOST2 is down)
|
||||||
|
|
||||||
```bash
|
```bash
|
||||||
HOST1_DDNS_CONTAINERS=("Gmer4Lfe.com-DDNS")
|
HOST2_DDNS_CONTAINERS=("Gmer4Lfe.us-DDNS")
|
||||||
|
|
||||||
FALLBACK_HOST1_STOP_ON_NO_NET=()
|
FALLBACK_HOST2_STOP_ON_NO_NET=()
|
||||||
|
|
||||||
# Tier 1 — immediate (HOST2's vital services)
|
# Tier 1 — immediate (HOST2's vital services)
|
||||||
FALLBACK_HOST1_COVERS_HOST2_TIER1=(
|
FALLBACK_HOST2_TIER1=(
|
||||||
"Gmer4Lfe.us-DDNS" # ALWAYS FIRST
|
"Gmer4Lfe.us-DDNS" # ALWAYS FIRST
|
||||||
# HOST2's Tier 1 services — fill per HOST2's stack
|
# HOST2's Tier 1 services — fill per HOST2's stack
|
||||||
)
|
)
|
||||||
|
|
||||||
FALLBACK_HOST1_COVERS_HOST2_TIER2=(...)
|
FALLBACK_HOST2_TIER2=(...)
|
||||||
FALLBACK_HOST1_COVERS_HOST2_TIER3=(...)
|
FALLBACK_HOST2_TIER3=(...)
|
||||||
FALLBACK_HOST1_COVERS_HOST2_TIER4=(...)
|
FALLBACK_HOST2_TIER4=(...)
|
||||||
|
|
||||||
# Tier delays for HOST2 outage
|
# Tier delays for HOST2 outage
|
||||||
HOST2_TIER2_DELAY=240
|
HOST2_TIER2_DELAY=240
|
||||||
@@ -492,7 +510,7 @@ Restart fallback.sh via User Scripts plugin. It will resume from NORMAL on its n
|
|||||||
|
|
||||||
1. Create the container on the covering server (stopped), with volume mounts pointing at
|
1. Create the container on the covering server (stopped), with volume mounts pointing at
|
||||||
the mirrored share path (e.g. `/mnt/user/Movies` must exist on the covering server)
|
the mirrored share path (e.g. `/mnt/user/Movies` must exist on the covering server)
|
||||||
2. Add the container name to `FALLBACK_HOST*_COVERS_HOST*_TIER*` in host*.conf
|
2. Add the container name to `FALLBACK_<THAT-HOST>_TIER*` in **that host's own** conf
|
||||||
in the appropriate tier position (dependency ordering — databases before apps)
|
in the appropriate tier position (dependency ordering — databases before apps)
|
||||||
3. Verify: `fallback.sh --status` shows the container in the expected tier list
|
3. Verify: `fallback.sh --status` shows the container in the expected tier list
|
||||||
4. Run `fallback_test.sh --dry-run` to confirm the full configuration is valid
|
4. Run `fallback_test.sh --dry-run` to confirm the full configuration is valid
|
||||||
|
|||||||
@@ -115,7 +115,7 @@ Both servers run `fallback.sh` independently as a continuous background process.
|
|||||||
makes all decisions from two pings every `FALLBACK_CHECK_INTERVAL` seconds:
|
makes all decisions from two pings every `FALLBACK_CHECK_INTERVAL` seconds:
|
||||||
|
|
||||||
```bash
|
```bash
|
||||||
ping REMOTE_TAILSCALE_IP # is the other server reachable?
|
ping "$(resolve_tailscale_ip "$REMOTE_SERVER_NAME")" # is the other server reachable?
|
||||||
ping EXTERNAL_IP # do I have internet? (default: 8.8.8.8)
|
ping EXTERNAL_IP # do I have internet? (default: 8.8.8.8)
|
||||||
```
|
```
|
||||||
|
|
||||||
@@ -220,8 +220,8 @@ determines which server is local and which is remote at runtime, then selects th
|
|||||||
container arrays and tier delays from config via MY_ID.
|
container arrays and tier delays from config via MY_ID.
|
||||||
|
|
||||||
```
|
```
|
||||||
HOST2 covers HOST1: FALLBACK_HOST2_COVERS_HOST1_TIER* (in host2.conf)
|
HOST2 covers HOST1: FALLBACK_HOST1_TIER* (in host2.conf)
|
||||||
HOST1 covers HOST2: FALLBACK_HOST1_COVERS_HOST2_TIER* (in host1.conf)
|
HOST1 covers HOST2: FALLBACK_HOST2_TIER* (in host1.conf)
|
||||||
```
|
```
|
||||||
|
|
||||||
Both servers run identical scripts. MY_ID selects the correct arrays. No hostname
|
Both servers run identical scripts. MY_ID selects the correct arrays. No hostname
|
||||||
|
|||||||
+33
-7
@@ -86,10 +86,25 @@ MEDIA_FILE_PATTERNS=(
|
|||||||
### Play State Sync
|
### Play State Sync
|
||||||
|
|
||||||
```bash
|
```bash
|
||||||
PLAY_STATE_SYNC_ENABLED=true # toggle entire sync
|
PLAY_SYNC_ENABLED=true # toggle entire sync
|
||||||
PLAY_STATE_SYNC_LOOKBACK_DAYS=30 # history window for played items
|
PLAY_SYNC_REMOTE=true # sync across hosts via Tailscale
|
||||||
|
# false = local servers only (this host's Emby + Jellyfin)
|
||||||
|
PLAY_SYNC_TYPES="Movie,Episode" # item types to sync — Audio excluded, music library too large
|
||||||
|
PLAY_SYNC_FAV_TYPES="MusicArtist,MusicAlbum,Movie,Series" # favourites, union sync, never unmarks
|
||||||
|
|
||||||
|
PLAY_SYNC_PROBE=true # skip per-item work when nothing changed since last run
|
||||||
|
PLAY_SYNC_PROBE_MAX_AGE_HOURS=24 # force a full comparison when the fingerprint is older
|
||||||
|
|
||||||
|
PLAY_SYNC_HANDBACK_RETRIES=5 # fallback handback: attempts before DNS cutover proceeds
|
||||||
|
PLAY_SYNC_HANDBACK_RETRY_DELAY=60 # seconds between those attempts
|
||||||
```
|
```
|
||||||
|
|
||||||
|
> **There is no date/lookback filter, and one must not be re-added.** An earlier version
|
||||||
|
> gated on played-date; it was removed once the real cost was measured — the 30-minute
|
||||||
|
> runtime was fork overhead per item, not API volume or item count. The fix was jq epoch
|
||||||
|
> parsing plus the response-hash probe below. Re-introducing a date window would reduce
|
||||||
|
> correctness (older items silently stop syncing) without meaningfully reducing runtime.
|
||||||
|
|
||||||
Emby and Jellyfin servers configured per-host:
|
Emby and Jellyfin servers configured per-host:
|
||||||
|
|
||||||
```bash
|
```bash
|
||||||
@@ -194,13 +209,24 @@ Show every file examined, not just those removed.
|
|||||||
### play_state_sync.sh
|
### play_state_sync.sh
|
||||||
|
|
||||||
`play_state_sync.sh`
|
`play_state_sync.sh`
|
||||||
Sync played/unplayed state and resume positions from local Emby to the remote Emby.
|
Sync played/unplayed state, resume positions and favourites across **every** configured Emby
|
||||||
Only items played within PLAY_STATE_SYNC_LOOKBACK_DAYS are synced.
|
and Jellyfin server — not just local→remote. Servers are discovered from every
|
||||||
|
`HOST*_TRANSCODE_SERVERS` entry, with remote hosts' localhost URLs rewritten to their
|
||||||
|
Tailscale IP. Newest `LastPlayedDate` wins; state only ever moves forward, never clears.
|
||||||
|
|
||||||
|
No age filter — every matched item is considered on every run. The **change probe** is what
|
||||||
|
keeps that cheap: the raw API responses are hashed and compared against the fingerprint from
|
||||||
|
the last successful run, and per-item processing is skipped entirely when nothing moved.
|
||||||
|
Fetches still happen every run, so nothing can be missed by the probe.
|
||||||
|
|
||||||
`play_state_sync.sh --full`
|
`play_state_sync.sh --full`
|
||||||
Ignore PLAY_STATE_SYNC_LOOKBACK_DAYS — sync all played items regardless of age. May be
|
Bypass the change probe and force the full per-item comparison even when the fingerprint
|
||||||
slow on large libraries. Use after a new Emby install or database restore to rebuild
|
matches. Use after a new Emby install or database restore, or when debugging a sync that
|
||||||
full play history.
|
appears to be skipping work it should be doing.
|
||||||
|
|
||||||
|
`play_state_sync.sh --wait`
|
||||||
|
Wait for an in-progress run instead of exiting. For manual runs that would otherwise be
|
||||||
|
skipped by the scheduled every-30-minute pass. Used by `fallback.sh` during handback.
|
||||||
|
|
||||||
`play_state_sync.sh --dry-run`
|
`play_state_sync.sh --dry-run`
|
||||||
Show what would be synced without writing any state.
|
Show what would be synced without writing any state.
|
||||||
|
|||||||
@@ -100,3 +100,43 @@ syncs between them.
|
|||||||
| `media_shares_permissions.sh` | Apply `nobody:users` ownership + correct permissions to all media shares | Daily — runs first |
|
| `media_shares_permissions.sh` | Apply `nobody:users` ownership + correct permissions to all media shares | Daily — runs first |
|
||||||
| `media_cleaner.sh` | Remove junk files (two profiles: `anime` + `media`) | Daily — runs before arr cleanup |
|
| `media_cleaner.sh` | Remove junk files (two profiles: `anime` + `media`) | Daily — runs before arr cleanup |
|
||||||
| `play_state_sync.sh` | Sync watched/played state + resume positions across Emby + Jellyfin | Every 30 min |
|
| `play_state_sync.sh` | Sync watched/played state + resume positions across Emby + Jellyfin | Every 30 min |
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## ━━━ THE ctime INVARIANT — READ BEFORE CHANGING PERMISSIONS ━━━
|
||||||
|
|
||||||
|
`media_shares_permissions.sh` applies every pass **conditionally** — it touches only entries
|
||||||
|
whose owner or mode is actually wrong. That is not an optimisation, and it must stay that way.
|
||||||
|
|
||||||
|
`chown` and `chmod` rewrite an inode's ctime **even when the value does not change**. A
|
||||||
|
blanket pass would therefore restamp every file in the library every night.
|
||||||
|
|
||||||
|
The arr cleanup scripts (`sonarr_cleanup.sh`, `radarr_cleanup.sh`, `lidarr_cleanup.sh`) gate
|
||||||
|
orphan deletion on ctime. mtime cannot substitute: an import preserves the release's original
|
||||||
|
timestamp, so mtime says nothing about when a file arrived here. Measured 2026-07-27 — of 400
|
||||||
|
files imported that week, **all 400 had mtimes over 7 days old, one of them 9613 days.**
|
||||||
|
|
||||||
|
So:
|
||||||
|
|
||||||
|
```
|
||||||
|
blanket chown/chmod → every ctime resets to today
|
||||||
|
→ no file ever appears older than *_ORPHAN_AGE
|
||||||
|
→ orphan collection silently stops
|
||||||
|
→ nothing errors, nothing warns, disk just fills
|
||||||
|
```
|
||||||
|
|
||||||
|
**The failure is invisible.** No script fails, no notification fires. The only symptom is
|
||||||
|
orphans quietly accumulating until a pool fills — which is exactly how the 755 GB / 89%-full
|
||||||
|
cache pool incident happened.
|
||||||
|
|
||||||
|
Two rules follow, and both are load-bearing:
|
||||||
|
|
||||||
|
1. **`media_shares_permissions.sh` passes stay conditional.** Making any of them unconditional
|
||||||
|
breaks orphan collection ecosystem-wide.
|
||||||
|
2. **`Tools/bulk_permissions_repair.sh` is unconditional by design** — it exists to repair
|
||||||
|
known-wrong paths where correctness beats preserving a clock. That is precisely why it is a
|
||||||
|
manual, targeted tool and not scheduled. Pointing it at a whole media root pauses orphan
|
||||||
|
collection there for `*_ORPHAN_AGE` days.
|
||||||
|
|
||||||
|
Both scripts' headers carry this warning too. If you are reading this because you are about to
|
||||||
|
"simplify" the permissions job, this is the thing that breaks.
|
||||||
|
|||||||
@@ -230,7 +230,7 @@ data — it just doesn't trigger a notification for that condition.
|
|||||||
| State File | Source | What It Shows |
|
| State File | Source | What It Shows |
|
||||||
|-----------|--------|---------------|
|
|-----------|--------|---------------|
|
||||||
| `FALLBACK_STATE_FILE` | `Fallback/fallback.sh` | Current fallback state (NORMAL/FALLBACK/etc.) |
|
| `FALLBACK_STATE_FILE` | `Fallback/fallback.sh` | Current fallback state (NORMAL/FALLBACK/etc.) |
|
||||||
| `SYS_WATCHDOG_FAILED_FILE` | `Watchdogs/docker_watchdog.sh` | Container skip list — needs human attention |
|
| `DOCKER_WATCHDOG_FAILED_FILE` | `Watchdogs/docker_watchdog.sh` | Container skip list — needs human attention |
|
||||||
| `WATCHDOG_STATE_FILE` | `Watchdogs/docker_watchdog.sh` | Active container strike counts |
|
| `WATCHDOG_STATE_FILE` | `Watchdogs/docker_watchdog.sh` | Active container strike counts |
|
||||||
| `SYS_WATCHDOG_STATE_FILE` | `Watchdogs/stability_watchdog.sh` | Active system watchdog strikes |
|
| `SYS_WATCHDOG_STATE_FILE` | `Watchdogs/stability_watchdog.sh` | Active system watchdog strikes |
|
||||||
| `BANDWIDTH_LOG` | `bandwidth_monitor.sh` | Yesterday's transfer history |
|
| `BANDWIDTH_LOG` | `bandwidth_monitor.sh` | Yesterday's transfer history |
|
||||||
|
|||||||
@@ -153,7 +153,7 @@ Daily 8am:
|
|||||||
weekly_health_digest.sh ── reads ──────────► FALLBACK_STATE_FILE
|
weekly_health_digest.sh ── reads ──────────► FALLBACK_STATE_FILE
|
||||||
── reads ──────────► WATCHDOG_STATE_FILE
|
── reads ──────────► WATCHDOG_STATE_FILE
|
||||||
── reads ──────────► SYS_WATCHDOG_STATE_FILE
|
── reads ──────────► SYS_WATCHDOG_STATE_FILE
|
||||||
── reads ──────────► SYS_WATCHDOG_FAILED_FILE
|
── reads ──────────► DOCKER_WATCHDOG_FAILED_FILE
|
||||||
── reads ──────────► BANDWIDTH_LOG
|
── reads ──────────► BANDWIDTH_LOG
|
||||||
── reads ──────────► TRANSCODE_DAILY_LOG
|
── reads ──────────► TRANSCODE_DAILY_LOG
|
||||||
── reads ──────────► TUNING_MONITOR_LOG
|
── reads ──────────► TUNING_MONITOR_LOG
|
||||||
|
|||||||
@@ -538,7 +538,7 @@ Runs on both servers; `detect_hosts()` determines which direction each sync goes
|
|||||||
# Drive temperature exit codes respected — skip share or abort all on CRIT.
|
# Drive temperature exit codes respected — skip share or abort all on CRIT.
|
||||||
#
|
#
|
||||||
# 3. Post-sync: DAILY_MAINTENANCE_SCRIPTS (everything except git pull)
|
# 3. Post-sync: DAILY_MAINTENANCE_SCRIPTS (everything except git pull)
|
||||||
# media_management.sh → permissions + cleaners + arr cleanup
|
# Media/ + Arrs_Stack/ scripts → permissions, cleaners, arr cleanup, classification
|
||||||
# docker_daily_restart.sh → nightly container restarts
|
# docker_daily_restart.sh → nightly container restarts
|
||||||
# ─────────────────────────────────────────────────────────────────────────────
|
# ─────────────────────────────────────────────────────────────────────────────
|
||||||
```
|
```
|
||||||
@@ -557,12 +557,12 @@ Runs on both servers; `detect_hosts()` determines which direction each sync goes
|
|||||||
#
|
#
|
||||||
# HOST1 runs this script at 1am:
|
# HOST1 runs this script at 1am:
|
||||||
# → pushes HOST1_DAILY_SYNC_SHARES (Movies, Tv_Shows, Music) → HOST2
|
# → pushes HOST1_DAILY_SYNC_SHARES (Movies, Tv_Shows, Music) → HOST2
|
||||||
# → media_management.sh on HOST1's shares
|
# → Media/ + Arrs_Stack/ scripts on HOST1's shares
|
||||||
# → docker_daily_restart.sh on HOST1's containers
|
# → docker_daily_restart.sh on HOST1's containers
|
||||||
#
|
#
|
||||||
# HOST2 runs this script at 1am:
|
# HOST2 runs this script at 1am:
|
||||||
# → pushes HOST2_DAILY_SYNC_SHARES (Anime_Shows, Anime_Movies) → HOST1
|
# → pushes HOST2_DAILY_SYNC_SHARES (Anime_Shows, Anime_Movies) → HOST1
|
||||||
# → media_management.sh on HOST2's shares
|
# → Media/ + Arrs_Stack/ scripts on HOST2's shares
|
||||||
# → docker_daily_restart.sh on HOST2's containers
|
# → docker_daily_restart.sh on HOST2's containers
|
||||||
# ─────────────────────────────────────────────────────────────────────────────
|
# ─────────────────────────────────────────────────────────────────────────────
|
||||||
```
|
```
|
||||||
|
|||||||
@@ -144,6 +144,8 @@ independently at 4-hour cadence.
|
|||||||
| `partnership_transfer.sh` | Transfer ownership from current owner to current mirror | Via `partnership_manager.sh --transfer`; owner only |
|
| `partnership_transfer.sh` | Transfer ownership from current owner to current mirror | Via `partnership_manager.sh --transfer`; owner only |
|
||||||
| `ssh_setup.sh` | SSH key generation, remote install, auth validation | Called by onboard; manually for re-keying or validation |
|
| `ssh_setup.sh` | SSH key generation, remote install, auth validation | Called by onboard; manually for re-keying or validation |
|
||||||
| `onboard_cancel.sh` | Remove SSH keys in one or both directions, clear setup flags | During cancelled or failed onboard; manual cleanup |
|
| `onboard_cancel.sh` | Remove SSH keys in one or both directions, clear setup flags | During cancelled or failed onboard; manual cleanup |
|
||||||
|
| `gitea_ssh_setup.sh` | Generate a Gitea keypair and register it via the Gitea API | Called during onboard; manually when re-keying a server |
|
||||||
|
| `share_setup.sh` | Create missing Unraid shares on the mirror from the owner's sync lists | Called during onboard; safe to re-run — existing shares are never modified |
|
||||||
|
|
||||||
---
|
---
|
||||||
|
|
||||||
|
|||||||
+70
-1
@@ -607,7 +607,7 @@ DEFAULT_RSYNC_OPTS="-az --no-perms --no-owner --no-group --inplace"
|
|||||||
BW_LIMIT=0 # KB/s, 0 = unlimited
|
BW_LIMIT=0 # KB/s, 0 = unlimited
|
||||||
RETRY_COUNT=3
|
RETRY_COUNT=3
|
||||||
SLEEP=60 # seconds between retries
|
SLEEP=60 # seconds between retries
|
||||||
ROOTFS_WARN_PCT=75 # abort if remote rootfs above this %
|
ROOTFS_WARN=75 # abort if remote rootfs above this %
|
||||||
|
|
||||||
# Shared with Monitors/
|
# Shared with Monitors/
|
||||||
BANDWIDTH_LOG="$DATA_DIR/bandwidth_history.db"
|
BANDWIDTH_LOG="$DATA_DIR/bandwidth_history.db"
|
||||||
@@ -696,3 +696,72 @@ name doesn't match any profile key, or when testing a specific profile.
|
|||||||
rsync.sh /mnt/user/appdata-Fallback/Critical-Data --profile=critical-data
|
rsync.sh /mnt/user/appdata-Fallback/Critical-Data --profile=critical-data
|
||||||
rsync.sh /mnt/user/Media_Server/Emby --profile=emby
|
rsync.sh /mnt/user/Media_Server/Emby --profile=emby
|
||||||
```
|
```
|
||||||
|
|
||||||
|
### --seed
|
||||||
|
|
||||||
|
Skip the empty-remote-share guard. Normally `check_remote_share()` aborts when the target
|
||||||
|
directory on the remote is missing or empty, because that usually means the array is not
|
||||||
|
mounted and syncing into it would write onto the underlying root filesystem. `--seed` is the
|
||||||
|
deliberate exception, for the first push of a genuinely new share.
|
||||||
|
|
||||||
|
```bash
|
||||||
|
rsync.sh /mnt/user/New_Share --seed
|
||||||
|
```
|
||||||
|
|
||||||
|
### --merge-run
|
||||||
|
|
||||||
|
Bidirectional merge. Two passes:
|
||||||
|
|
||||||
|
```
|
||||||
|
Pass 1 pull remote → local, with --ignore-existing
|
||||||
|
Pass 2 push local → remote, with --delete
|
||||||
|
```
|
||||||
|
|
||||||
|
Local ends up authoritative: the remote may contribute files the local lacks, but never a
|
||||||
|
competing version of a file the local already has. Also triggered automatically when the
|
||||||
|
pre-scan detects ≥75% directory overlap with the remote.
|
||||||
|
|
||||||
|
**`--delete` is applied only if pass 1 completed.** Its entire justification is "local is now
|
||||||
|
the authoritative superset", and a pull that was capped by `RSYNC_MAX_RUNTIME_HOURS` or failed
|
||||||
|
outright means it is not — the remote still holds content the local never received. On an
|
||||||
|
incomplete pull the push runs **without** `--delete`, warns, and notifies. Local content still
|
||||||
|
propagates; nothing remote-unique is destroyed; the delete happens on a later run whose pull
|
||||||
|
actually succeeded.
|
||||||
|
|
||||||
|
> This was a real defect, fixed 2026-08-01. `--delete` used to be unconditional, so a capped
|
||||||
|
> or failed pull would permanently delete remote content the pull had not yet retrieved — and
|
||||||
|
> the timeout branch's own comment promised it would "resume the pull next run", which was
|
||||||
|
> impossible once the data was gone. If you are changing this function, that interlock is the
|
||||||
|
> part that matters.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## ━━━ SOURCE PATH GUARDS ━━━
|
||||||
|
|
||||||
|
`rsync.sh` pushes to `root@remote` at the **same absolute path**, and `--merge-run` adds
|
||||||
|
`--delete`. The source argument is therefore a remote-side hazard, not just a local one, so
|
||||||
|
two guards run before anything else:
|
||||||
|
|
||||||
|
| Guard | Rejects |
|
||||||
|
|-------|---------|
|
||||||
|
| Absolute path, ≥3 components | `/`, `/mnt`, `/mnt/user`, empty |
|
||||||
|
| Directory must exist locally | typo'd paths |
|
||||||
|
|
||||||
|
Every real job path clears both — see `HOST*_DAILY_SYNC_SHARES` and friends, all of which are
|
||||||
|
`/mnt/user/<Share>` or deeper.
|
||||||
|
|
||||||
|
Without the existence check, a mistyped path under `--merge-run` would be *created* locally by
|
||||||
|
the pull pass, filled from the remote, then pushed back with `--delete`.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## ━━━ RUNTIME CEILING ━━━
|
||||||
|
|
||||||
|
No single transfer attempt may exceed `RSYNC_MAX_RUNTIME_HOURS` (default 23). On timeout the
|
||||||
|
transfer is terminated and resumes on the next scheduled run.
|
||||||
|
|
||||||
|
This is only safe because `--partial` is in `DEFAULT_RSYNC_OPTS` — a terminated transfer picks
|
||||||
|
up where it left off rather than restarting. The cap exists so one huge or stuck transfer
|
||||||
|
cannot hold its per-profile lock indefinitely and starve every other profile of a turn.
|
||||||
|
|
||||||
|
Both `--merge-run` passes are capped independently.
|
||||||
|
|||||||
@@ -77,7 +77,7 @@ Orchestrators/
|
|||||||
────────────────────────► inotify_tuning.sh (before docker_network_connect)
|
────────────────────────► inotify_tuning.sh (before docker_network_connect)
|
||||||
array_stopping.sh ───────────────────────► conf_cache_save.sh (first — while cache is fresh)
|
array_stopping.sh ───────────────────────► conf_cache_save.sh (first — while cache is fresh)
|
||||||
───────────────────────► rsync_stop.sh --rsync-only (stop transfers)
|
───────────────────────► rsync_stop.sh --rsync-only (stop transfers)
|
||||||
weekly_maintenance.sh ──────────────────► clear_logs.sh
|
weekly_sync_maintenance.sh ──────────────────► clear_logs.sh
|
||||||
|
|
||||||
Watchdogs/System/
|
Watchdogs/System/
|
||||||
conf_cache_watchdog.sh ─────────────────► maintains $PERSISTENT_CONF_CACHE while partner offline
|
conf_cache_watchdog.sh ─────────────────► maintains $PERSISTENT_CONF_CACHE while partner offline
|
||||||
@@ -109,7 +109,7 @@ Plugin/unraid/Watchdogs/System/
|
|||||||
| `conf_sync.sh` | Sync partner confs: pull → RAM cache, push own conf to partners | At array start + every 4 h (--pull-only) |
|
| `conf_sync.sh` | Sync partner confs: pull → RAM cache, push own conf to partners | At array start + every 4 h (--pull-only) |
|
||||||
| `conf_cache_save.sh` | Snapshot partner RAM cache → `$PERSISTENT_CONF_CACHE` | At array stop — first step |
|
| `conf_cache_save.sh` | Snapshot partner RAM cache → `$PERSISTENT_CONF_CACHE` | At array stop — first step |
|
||||||
| `conf_cache_restore.sh` | Load missing partner confs from persistent backup into RAM | At array start — after conf_sync |
|
| `conf_cache_restore.sh` | Load missing partner confs from persistent backup into RAM | At array start — after conf_sync |
|
||||||
| `clear_logs.sh` | Size-threshold log cleanup | Weekly via weekly_maintenance.sh |
|
| `clear_logs.sh` | Size-threshold log cleanup | Weekly via weekly_sync_maintenance.sh |
|
||||||
| `rsync_stop.sh` | Orchestrator-aware rsync stop | Manual / called by array_stopping.sh |
|
| `rsync_stop.sh` | Orchestrator-aware rsync stop | Manual / called by array_stopping.sh |
|
||||||
| `server_reboot.sh` | Graceful reboot with pre-flight warnings | Manual |
|
| `server_reboot.sh` | Graceful reboot with pre-flight warnings | Manual |
|
||||||
|
|
||||||
@@ -141,7 +141,7 @@ Every 15 minutes (watchdog_orchestrator.sh):
|
|||||||
Every 4 hours (intermediate_sync_maintenance.sh):
|
Every 4 hours (intermediate_sync_maintenance.sh):
|
||||||
└─ conf_sync.sh --pull-only ← refresh partner conf in RAM without pushing own conf
|
└─ conf_sync.sh --pull-only ← refresh partner conf in RAM without pushing own conf
|
||||||
|
|
||||||
Weekly (weekly_maintenance.sh):
|
Weekly (weekly_sync_maintenance.sh):
|
||||||
└─ clear_logs.sh
|
└─ clear_logs.sh
|
||||||
System logs: clear if > LOG_MIN_SIZE_MB
|
System logs: clear if > LOG_MIN_SIZE_MB
|
||||||
Docker logs: clear per-container if > LOG_DOCKER_MAX_MB
|
Docker logs: clear per-container if > LOG_DOCKER_MAX_MB
|
||||||
|
|||||||
@@ -250,7 +250,7 @@ docker start ContainerName
|
|||||||
# Both live on /boot/config — survive reboots intentionally.
|
# Both live on /boot/config — survive reboots intentionally.
|
||||||
# A container that was skip-listed before a reboot is still broken after it.
|
# A container that was skip-listed before a reboot is still broken after it.
|
||||||
|
|
||||||
$SYS_WATCHDOG_FAILED_FILE # persistent skip list
|
$DOCKER_WATCHDOG_FAILED_FILE # persistent skip list
|
||||||
$WATCHDOG_CONTAINER_RESTART_LOG # restart loop tracking
|
$WATCHDOG_CONTAINER_RESTART_LOG # restart loop tracking
|
||||||
```
|
```
|
||||||
|
|
||||||
|
|||||||
@@ -130,6 +130,7 @@ The relationship is one-way: Tools act on state that other scripts have written.
|
|||||||
| `emby_to_lidarr_sync.sh` | Add all Emby album artists not yet tracked in Lidarr | After Lidarr setup, database wipe, or when you suspect gaps |
|
| `emby_to_lidarr_sync.sh` | Add all Emby album artists not yet tracked in Lidarr | After Lidarr setup, database wipe, or when you suspect gaps |
|
||||||
| `emby_to_sonarr_sync.sh` | Add all Emby TV series not yet tracked in Sonarr | After Sonarr setup, database wipe, or when you suspect gaps |
|
| `emby_to_sonarr_sync.sh` | Add all Emby TV series not yet tracked in Sonarr | After Sonarr setup, database wipe, or when you suspect gaps |
|
||||||
| `emby_to_radarr_sync.sh` | Add all Emby movies not yet tracked in Radarr | After Radarr setup, database wipe, or when you suspect gaps |
|
| `emby_to_radarr_sync.sh` | Add all Emby movies not yet tracked in Radarr | After Radarr setup, database wipe, or when you suspect gaps |
|
||||||
|
| `trailer_folder_migration.sh` | Relocate TV trailers Emby was ingesting as fake episodes into `<show>/trailers/` | Once, after Trailarr's Series Trailers profile was corrected |
|
||||||
|
|
||||||
---
|
---
|
||||||
|
|
||||||
|
|||||||
@@ -234,3 +234,53 @@ Every 15 minutes — watchdog_orchestrator.sh fires:
|
|||||||
| `NETWORK_WATCHDOG_NPM_STATE_FILE` | `System/network_watchdog.sh` | itself | NPM proxy strike count |
|
| `NETWORK_WATCHDOG_NPM_STATE_FILE` | `System/network_watchdog.sh` | itself | NPM proxy strike count |
|
||||||
|
|
||||||
All state files are in `$STATE_DIR` (adapts to storage mode). See `master.conf` for actual variable values.
|
All state files are in `$STATE_DIR` (adapts to storage mode). See `master.conf` for actual variable values.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## ━━━ SAFEGUARDS COMMON TO THIS FOLDER ━━━
|
||||||
|
|
||||||
|
These run unattended every 15 minutes and hold the only irreversible remedy in the
|
||||||
|
ecosystem, so the guards matter as much as the checks. Each script's header documents its
|
||||||
|
full set; these are the folder-wide ones.
|
||||||
|
|
||||||
|
**Nothing acts on a single reading.** Every check that can trigger an action is strike-gated.
|
||||||
|
One bad sample is a reading; N consecutive samples is a condition. Strikes clear themselves
|
||||||
|
when the condition resolves, so a transient blip never accumulates toward an alarm across
|
||||||
|
unrelated weeks.
|
||||||
|
|
||||||
|
**Escalation is asymmetric on purpose.** Protective actions happen fast; undoing them happens
|
||||||
|
slowly. `resource_watchdog.sh` throttles before it pauses and pauses before it stops, then
|
||||||
|
requires `RW_RECOVER_CYCLES` clear cycles and de-escalates one level at a time. Recovering on
|
||||||
|
a single good reading would flap under sustained load.
|
||||||
|
|
||||||
|
**Ownership boundaries are enforced, not conventional.** Container health belongs to
|
||||||
|
`docker_watchdog.sh`; `stability_watchdog.sh` deliberately does not check it, and its Docker
|
||||||
|
daemon check writes a flag for `docker_watchdog.sh` rather than acting. Two watchdogs
|
||||||
|
remediating the same subsystem would race.
|
||||||
|
|
||||||
|
**Watchdogs coordinate through state files.** `resource_watchdog.sh` publishes
|
||||||
|
`mem_shutdown_active` so `docker_watchdog.sh` defers restarts — otherwise one would stop
|
||||||
|
containers to free RAM while the other restarted them to restore health. The reader applies a
|
||||||
|
staleness guard so a crashed writer cannot suppress restarts forever.
|
||||||
|
|
||||||
|
**Data safety outranks uptime.** `stability_watchdog.sh` aborts a reboot while a ZFS pool is
|
||||||
|
unhealthy, parity is running, or the mover is active. Only Tier 1 — conditions the system
|
||||||
|
cannot recover from and which worsen every cycle — bypasses that.
|
||||||
|
|
||||||
|
**Reboot loops end in shutdown, not more reboots.** No more than
|
||||||
|
`SYS_WATCHDOG_REBOOT_LIMIT` reboots within `SYS_WATCHDOG_REBOOT_WINDOW_HRS`. On hitting the
|
||||||
|
limit the host powers off: a fault that survives repeated reboots will not be fixed by more
|
||||||
|
of them, and a cleanly-down box is better than one cycling endlessly.
|
||||||
|
|
||||||
|
**An aborted reboot restores what it stopped.** An EXIT trap is armed the moment containers
|
||||||
|
start being stopped for a reboot and disarmed only once the reboot commits. If the script
|
||||||
|
dies in between, everything it stopped comes back — the failure mode is a running system,
|
||||||
|
never a host left stripped with no reboot.
|
||||||
|
|
||||||
|
**An unconfigured job list fails loudly.** An empty `WATCHDOG_ORCHESTRATOR_SCRIPTS` or
|
||||||
|
`SYSTEM_WATCHDOG_SCRIPTS` would otherwise report "0/0 passed" and exit 0 every cycle —
|
||||||
|
indistinguishable from healthy while nothing is monitored at all.
|
||||||
|
|
||||||
|
**Verify the path, not the process.** `network_watchdog.sh` checks NPM by fetching an external
|
||||||
|
URL rather than asking whether the container is running. A running container behind broken DNS
|
||||||
|
or a broken upstream still serves nothing.
|
||||||
|
|||||||
Reference in New Issue
Block a user