Compare commits
3
Commits
c377ddfcca
...
7a8fbbe06a
| Author | SHA1 | Date | |
|---|---|---|---|
 Gmer4Lfe
|
7a8fbbe06a | ||
|
Gmer4Lfe
|
53aa72d38c | ||
|
Gmer4Lfe
|
8a2707ee37 |
@@ -200,6 +200,75 @@ Lidarr configured — no HOST1_LIDARR_URL means nothing runs.
|
||||
| `playback_aware_lidarr_discovery.sh` | Behavior-driven music discovery — Emby plays → Last.fm similar → Lidarr | Weekly |
|
||||
| `playback_aware_radarr_discovery.sh` | Behavior-driven movie discovery — Emby watches → TMDB recommendations → Radarr | Weekly |
|
||||
| `playback_aware_sonarr_discovery.sh` | Behavior-driven TV discovery — Emby episodes → TMDB TV recommendations → Sonarr | Weekly |
|
||||
| `arr_download_orphan_cleaner.sh` | Clear orphaned completed downloads out of the SABnzbd Completed folders | Daily |
|
||||
| `sonarr_classification_scan.sh` | Detect series sitting in the wrong root (anime / kids / general); `--move` acts | Daily |
|
||||
| `radarr_classification_scan.sh` | Same for movies, plus junk-metadata detection via `--remove-junk` | Daily |
|
||||
| `lidarr_duplicate_artist_cleanup.sh` | Remove phantom zero-file duplicate artists; flag real ones for review | Daily |
|
||||
| `arr_cache_prefill.sh` | Warm the shared tracked-data cache so consumers never read cold | Array start + every 4hr |
|
||||
| `arr_corruption_scan.sh` | ffprobe every tracked video for corrupt headers; `--remediate` deletes + re-searches | Weekly |
|
||||
| `arr_full_rescan.sh` | Force a real disk↔database reconciliation on all three arrs | Weekly |
|
||||
|
||||
---
|
||||
|
||||
## ━━━ THE NEWER LAYERS ━━━
|
||||
|
||||
The original folder was "delete what the arrs no longer track". These were added as distinct
|
||||
failure modes surfaced — each exists because something went wrong that the cleanups could not
|
||||
have caught.
|
||||
|
||||
### 🗑️ Download-Side Orphans — `arr_download_orphan_cleaner.sh`
|
||||
|
||||
Every cleanup script here walks the **library** side. Nothing walked the **download** side —
|
||||
so completed downloads the arrs had stopped tracking accumulated in SABnzbd's Completed
|
||||
folders indefinitely. Discovered as **755 GB of orphaned TV downloads, oldest from 2022**,
|
||||
filling the cache pool to 89%.
|
||||
|
||||
Classifies every entry as TRACKED / RECENT / JUNK / REDUNDANT / IMPORTABLE / UNMATCHED and
|
||||
acts only on the ones it can justify. The queue is a hard gate: if it cannot be read, the arr
|
||||
is skipped entirely, because without it there is no way to tell an active import from an
|
||||
orphan. A run total over `DOWNLOAD_ORPHAN_MAX_DELETE_GB` aborts — an abnormally large delete
|
||||
is the visible symptom of a partial queue fetch.
|
||||
|
||||
### 🎭 Wrong-Root Detection — `sonarr_classification_scan.sh` + `radarr_classification_scan.sh`
|
||||
|
||||
Overseerr lets any user request content into the wrong root folder — kids shows into general
|
||||
TV, anime into Kids_Tv_Shows. These classify every item from metadata alone (genre,
|
||||
certification, network/studio, original language) and report where the computed classification
|
||||
disagrees with the folder the item actually sits in.
|
||||
|
||||
Report-only by default. `--move` acts on forward misplacements and adult-content-in-kids-root
|
||||
leaks. It deliberately does **not** move non-anime content out of the anime root — deliberate
|
||||
style placements (Western animation grouped with anime by choice) are genuine judgment calls.
|
||||
|
||||
Both poll the arr's async move command to completion before verifying, because `moveFiles=true`
|
||||
flips the database instantly while the physical move is still queued behind others.
|
||||
|
||||
### 🎨 Phantom Artists — `lidarr_duplicate_artist_cleanup.sh`
|
||||
|
||||
MusicBrainz duplicates leave two Lidarr entries for one artist, one holding the files and one
|
||||
holding nothing. Removes only the zero-file side, with `deleteFiles=false` so nothing on disk
|
||||
is touched. Pairs where both sides hold files are flagged for review, never auto-resolved.
|
||||
|
||||
Gated on the tracked-count floor shared with `lidarr_cleanup.sh` — during a library-wide desync
|
||||
both sides of a real duplicate can read as zero-file phantoms.
|
||||
|
||||
### 🩺 Corruption + Reconciliation — `arr_corruption_scan.sh` + `arr_full_rescan.sh`
|
||||
|
||||
`arr_corruption_scan.sh` ffprobes tracked video files for corrupt headers. Report-only unless
|
||||
`--remediate`, which deletes the file record and triggers an explicit re-search. Requires
|
||||
repeat detections across separate runs before acting, so a transient probe failure cannot
|
||||
delete a healthy file.
|
||||
|
||||
`arr_full_rescan.sh` forces a genuine disk↔database reconciliation. Organic scans only touch
|
||||
files involved in an import, so an untouched library silently drifts — confirmed when Lidarr
|
||||
reported **~23% of its true track count** for 1,004 of 1,357 artists with no scan running and
|
||||
every file present on disk.
|
||||
|
||||
### ⚡ Cache Warmth — `arr_cache_prefill.sh`
|
||||
|
||||
Populates the shared tracked-data cache at array start and every 4 hours, so consumers never
|
||||
pay a cold fetch. Pure enhancement: nothing depends on it having run, and every consumer still
|
||||
writes through on a cold cache.
|
||||
|
||||
---
|
||||
|
||||
|
||||
@@ -94,20 +94,54 @@ HOST1_NETWORK_CONNECT_CONTAINERS=(
|
||||
|
||||
```bash
|
||||
# master.conf
|
||||
DAILY_CONTAINER_UPDATES=true # enable/disable daily image pull
|
||||
DAILY_CONTAINER_UPDATES=true # enable/disable the daily image pull
|
||||
# docker_daily_restart.sh still runs regardless
|
||||
# update and restart are independent
|
||||
|
||||
WEEKLY_REMAINING_UPDATES=true # enable/disable weekly remainder pull + prune
|
||||
# to disable: set false or remove from WEEKLY_MAINTENANCE_SCRIPTS
|
||||
WEEKLY_CONTAINER_UPDATES=true # enable/disable the weekly image pull
|
||||
# docker_weekly_restart.sh still runs regardless
|
||||
|
||||
MONTHLY_REMAINING_UPDATES=true # enable/disable the monthly remainder pull
|
||||
# to disable: set false, or remove
|
||||
# "docker_update.sh --remainder" from
|
||||
# MONTHLY_MAINTENANCE_SCRIPTS
|
||||
```
|
||||
|
||||
`docker_update.sh` in normal mode targets `DAILY_RESTART_CONTAINERS` — the same list
|
||||
used by `docker_daily_restart.sh`. No second list to maintain.
|
||||
There is one update script, `docker_update.sh`, with three modes. Each reuses the restart
|
||||
list it pairs with, so there is no second list to maintain:
|
||||
|
||||
`docker_update_remaining.sh` derives its target list automatically:
|
||||
all running containers minus `DAILY_RESTART_CONTAINERS` minus `WEEKLY_RESTART_CONTAINERS`.
|
||||
Everything gets updated at least once per week with no explicit configuration.
|
||||
| Mode | Targets | Runs |
|
||||
|------|---------|------|
|
||||
| *(default)* | `DAILY_RESTART_CONTAINERS` | Daily, before `docker_daily_restart.sh` |
|
||||
| `--weekly` | `WEEKLY_RESTART_CONTAINERS` | Weekly, before `docker_weekly_restart.sh` |
|
||||
| `--remainder` | derived, see below | Monthly, via `MONTHLY_MAINTENANCE_SCRIPTS` |
|
||||
|
||||
Remainder mode needs no configuration at all. It takes every **running** container and
|
||||
subtracts:
|
||||
|
||||
```
|
||||
DAILY_RESTART_CONTAINERS already updated daily
|
||||
WEEKLY_RESTART_CONTAINERS already updated weekly
|
||||
PROFILE_CRITICAL_CONTAINER_NAMES[emby] updated inline by the weekly sync window
|
||||
PROFILE_CRITICAL_CONTAINER_NAMES[critical-data] updated inline by the weekly sync window
|
||||
FALLBACK_<REMOTE>_TIER1..4 owned by the remote's update cycle
|
||||
```
|
||||
|
||||
Stopped containers are never targeted in any mode — pulling for a stopped container adds
|
||||
nothing, and it was most likely stopped deliberately.
|
||||
|
||||
The fallback exclusion is a correctness rule, not an optimisation. This server only runs
|
||||
those containers during a fallback; the remote owns their version. If remainder updated them
|
||||
independently and a handback then occurred, the remote's older image could meet data written
|
||||
by the newer one.
|
||||
|
||||
**Ordering matters.** The update always runs *before* its matching restart so the restart
|
||||
lands on the freshly pulled image. If a container's image actually changed, `docker_update.sh`
|
||||
rebuilds it from its template (a plain `docker restart` reuses the image ID baked in at
|
||||
creation time and would never pick up the new digest) and records it in
|
||||
`DOCKER_UPDATE_REBUILT_*_FILE`. The restart script reads that file and skips those containers
|
||||
rather than restarting them a second time — and discards the file as stale if it is older
|
||||
than `DOCKER_UPDATE_REBUILT_STALE_HOURS`.
|
||||
|
||||
---
|
||||
|
||||
@@ -142,12 +176,22 @@ HOST1_NETWORK_CONNECT_CONTAINERS=()
|
||||
# Shared — applies to both servers
|
||||
|
||||
# ── Container updates ──────────────────────────────────────────────────
|
||||
DAILY_CONTAINER_UPDATES=true
|
||||
WEEKLY_REMAINING_UPDATES=true
|
||||
DAILY_CONTAINER_UPDATES=true # daily pull (DAILY_RESTART_CONTAINERS)
|
||||
WEEKLY_CONTAINER_UPDATES=true # weekly pull (WEEKLY_RESTART_CONTAINERS)
|
||||
MONTHLY_REMAINING_UPDATES=true # monthly pull (everything else)
|
||||
|
||||
# Handoff between update and restart — written by docker_update.sh,
|
||||
# read by the matching restart script so it skips containers already
|
||||
# rebuilt onto a new image this run.
|
||||
DOCKER_UPDATE_REBUILT_DAILY_FILE
|
||||
DOCKER_UPDATE_REBUILT_WEEKLY_FILE
|
||||
DOCKER_UPDATE_REBUILT_STALE_HOURS=12 # older than this = discarded, restart all
|
||||
|
||||
# ── Retry behaviour (shared by restart scripts) ────────────────────────
|
||||
RETRY_COUNT=3 # retry attempts before marking failed
|
||||
SLEEP=5 # seconds between retry attempts
|
||||
CONTAINER_DELAY # seconds between a dependency and its dependents
|
||||
RESTART_VERIFY_WAIT=3 # settle time before verifying a restart stuck
|
||||
|
||||
# Watchdog thresholds (CPU, memory, HTTP, restart loop):
|
||||
# → see Watchdogs/Manual-Watchdogs.md
|
||||
@@ -211,7 +255,24 @@ All scripts support these standard flags:
|
||||
summary block: identity, duration, counts, and a status line. Per-container detail only
|
||||
appears with `--log`. Warnings and errors are always visible regardless of `--log`.
|
||||
|
||||
### `docker_update.sh --remainder`
|
||||
Switches to remainder mode — updates all running containers not in the managed daily/weekly
|
||||
lists. Called by `weekly_sync_maintenance.sh`. Can be run manually to sweep containers
|
||||
that haven't been updated recently.
|
||||
### Script-specific flags
|
||||
|
||||
| Script | Flag | What it does |
|
||||
|--------|------|-------------|
|
||||
| `docker_update.sh` | `--weekly` | Target `WEEKLY_RESTART_CONTAINERS`. Called by `weekly_sync_maintenance.sh` before the weekly restart. |
|
||||
| `docker_update.sh` | `--remainder` | Target every running container not in the daily list, weekly list, emby/critical-data profiles, or fallback tiers. Called by `monthly_maintenance.sh`. Safe to run manually to sweep anything missed. |
|
||||
| `media_cleaner.sh` *(Media/)* | `<anime\|media>` | Required positional profile — there is no default. |
|
||||
|
||||
`docker_update.sh` with no mode flag is normal mode: `DAILY_RESTART_CONTAINERS`, called by
|
||||
`daily_sync_maintenance.sh` before the daily restart.
|
||||
|
||||
### Exit codes
|
||||
|
||||
| Code | Meaning |
|
||||
|------|---------|
|
||||
| `0` | Success, or nothing to do (empty list, disabled toggle, no running containers) |
|
||||
| `1` | One or more containers failed — the summary names them |
|
||||
|
||||
`docker_update.sh` deliberately exits `0` even when pulls fail: a failed pull is not a reason
|
||||
to abort the restart that follows, which simply proceeds on the existing image. The failure
|
||||
is reported in the summary.
|
||||
|
||||
@@ -178,12 +178,28 @@ windows so any downtime from restarts is absorbed by the window that's already h
|
||||
|
||||
---
|
||||
|
||||
### 🔄 Image Currency — `docker_update.sh` + `docker_update_remaining.sh`
|
||||
### 🔄 Image Currency — `docker_update.sh`
|
||||
|
||||
Keeps all container images current without manual intervention. Daily updates for the
|
||||
auth/proxy stack (the containers that restart daily anyway — no extra downtime). Weekly
|
||||
remainder pass for everything else — derives the target list automatically from `docker ps`
|
||||
minus what was already updated, so there is no second list to maintain.
|
||||
One script, three modes — there is no separate remainder script.
|
||||
|
||||
| Mode | Targets | Runs |
|
||||
|------|---------|------|
|
||||
| *(default)* | `DAILY_RESTART_CONTAINERS` | Daily, **before** `docker_daily_restart.sh` |
|
||||
| `--weekly` | `WEEKLY_RESTART_CONTAINERS` | Weekly, **before** `docker_weekly_restart.sh` |
|
||||
| `--remainder` | Everything running that is in neither list | Monthly, via `monthly_maintenance.sh` |
|
||||
|
||||
The update always runs *before* its matching restart, so the restart lands on the freshly
|
||||
pulled image. Reversing that order would restart onto the old image and leave the new one
|
||||
sitting unused until the next window.
|
||||
|
||||
Each mode reuses the restart list it pairs with rather than maintaining its own — add a
|
||||
container to `DAILY_RESTART_CONTAINERS` once and it gets both the restart and the image pull.
|
||||
Remainder mode needs no list at all: it derives its targets from `docker ps` minus the daily
|
||||
list, the weekly list, the emby/critical-data sync-window profiles, and the fallback tiers.
|
||||
|
||||
Fallback containers are deliberately excluded from remainder mode. This server only runs them
|
||||
during a fallback; the remote owns their version. Updating them here would risk the remote's
|
||||
older image meeting data written by a newer one after a handback.
|
||||
|
||||
---
|
||||
|
||||
@@ -229,7 +245,7 @@ full coordination model between all four watchdogs.
|
||||
|--------|------|-------------|
|
||||
| `docker_daily_restart.sh` | Nightly proactive restart of degradation-prone containers | 1am via `daily_sync_maintenance.sh` |
|
||||
| `docker_weekly_restart.sh` | Weekly restart of less-critical services | 2:30am Sunday via `weekly_sync_maintenance.sh` |
|
||||
| `docker_update.sh` | Container image updates — daily list + weekly remainder mode | Daily before restart; weekly remainder pass |
|
||||
| `docker_update.sh` | Container image updates — three modes (default / `--weekly` / `--remainder`) | Daily and weekly before each restart; monthly for the remainder |
|
||||
| `docker_network_connect.sh` | Network existence + container connection enforcement | Every array start |
|
||||
| `docker_container_stop.sh` | Ordered container shutdown — graceful then forced | Called by `array_stopping.sh` |
|
||||
| `downloaders_reset.sh` | Download client hygiene — slskd / SABnzbd / qBittorrent | Every 30min via `critical_sync_maintenance.sh` |
|
||||
@@ -257,8 +273,13 @@ Daily maintenance window (1am):
|
||||
|
||||
Weekly maintenance window (2:30am Sunday):
|
||||
weekly_sync_maintenance.sh
|
||||
├── docker_weekly_restart.sh ──── restart less-critical services
|
||||
└── docker_update.sh --remainder ─ update containers not in managed lists
|
||||
├── docker_update.sh --weekly ─── pull latest (WEEKLY_RESTART_CONTAINERS)
|
||||
└── docker_weekly_restart.sh ──── restart onto the fresh image
|
||||
|
||||
Monthly maintenance window:
|
||||
monthly_maintenance.sh
|
||||
├── docker_update.sh --remainder update everything not in the managed lists
|
||||
└── Tools/docker_prune_images.sh --all
|
||||
|
||||
Critical maintenance window (every 30min):
|
||||
critical_sync_maintenance.sh
|
||||
@@ -266,5 +287,39 @@ Critical maintenance window (every 30min):
|
||||
|
||||
Array stopping:
|
||||
array_stopping.sh
|
||||
└── docker_container_stop.sh ──── ordered graceful shutdown
|
||||
└── docker_container_stop.sh ──── ordered graceful shutdown, verified per container
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## ━━━ SAFEGUARDS COMMON TO THIS FOLDER ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
|
||||
|
||||
Every script here talks to the Docker daemon, so they share the same protections. Each
|
||||
script's own header documents its full set; these are the ones worth knowing folder-wide.
|
||||
|
||||
**Daemon health is checked, not assumed.** A hung daemon returns an empty container list,
|
||||
which is indistinguishable from "no containers running". Without the check,
|
||||
`docker_container_stop.sh` would report a clean shutdown that never happened, and
|
||||
`docker_update.sh --remainder` would report "nothing to update" while doing nothing.
|
||||
|
||||
**Every docker call is timeout-wrapped.** A wedged daemon cannot stall a maintenance window
|
||||
or hold a lock open. The one deliberate exception is `docker pull` — a large image
|
||||
legitimately outlasts any sane timeout, and killing it mid-layer wastes the transfer.
|
||||
|
||||
**State is respected.** Running containers get restarted; stopped ones stay stopped. A
|
||||
stopped container was almost certainly stopped on purpose, and none of these scripts has the
|
||||
authority to overrule that.
|
||||
|
||||
**Restarts are verified, not assumed.** After each restart the container is re-checked once
|
||||
it has had time to settle. A container that starts and immediately crashes is recorded as a
|
||||
failure and notified — a restart that did not stick is never reported as success.
|
||||
|
||||
**Dependency ordering is shared with the watchdog.** Restarts follow
|
||||
`HOST*_WATCHDOG_DEPENDENCIES`, with `CONTAINER_DELAY` between a dependency and its dependents,
|
||||
so a dependent is never brought up while what it needs is still initialising.
|
||||
|
||||
**Locks prevent overlap.** Long windows can outlast their interval;
|
||||
`downloaders_reset.sh` uses wait-mode because it runs every 30 minutes and the previous pass
|
||||
may still be finishing.
|
||||
|
||||
---
|
||||
|
||||
+42
-24
@@ -105,18 +105,36 @@ internet connectivity. Most hosts leave this empty.
|
||||
---
|
||||
|
||||
```
|
||||
FALLBACK_HOST*_COVERS_HOST*_TIER1=(...)
|
||||
FALLBACK_HOST*_COVERS_HOST*_TIER2=(...)
|
||||
FALLBACK_HOST*_COVERS_HOST*_TIER3=(...)
|
||||
FALLBACK_HOST*_COVERS_HOST*_TIER4=(...)
|
||||
FALLBACK_HOST*_TIER1=(...)
|
||||
FALLBACK_HOST*_TIER2=(...)
|
||||
FALLBACK_HOST*_TIER3=(...)
|
||||
FALLBACK_HOST*_TIER4=(...)
|
||||
```
|
||||
Containers this host starts for the remote host when the remote is down. TIER1 starts
|
||||
immediately. TIER2–4 activate after the corresponding delay thresholds.
|
||||
|
||||
Variable pattern: `FALLBACK_${MY_ID}_COVERS_${REMOTE_ID}_TIER${N}`
|
||||
**Each host declares its OWN services, in its OWN conf.** `FALLBACK_HOST2_TIER1` lives in
|
||||
`host2.conf` and lists HOST2's vital containers — it is not a list HOST1 maintains.
|
||||
|
||||
The DDNS container for the remote's domain must be the first entry in TIER1 — DNS
|
||||
coverage before anything else.
|
||||
At runtime `fallback.sh` reads its **partner's** list:
|
||||
|
||||
```bash
|
||||
get_tier_containers() {
|
||||
local var_name="FALLBACK_${REMOTE_ID}_TIER${tier}" # note: REMOTE_ID, not MY_ID
|
||||
```
|
||||
|
||||
So HOST1, covering HOST2, reads `FALLBACK_HOST2_TIER1` — a variable defined in `host2.conf`
|
||||
and delivered to HOST1 through the partner conf cache (`conf_sync.sh`), because sparse
|
||||
checkout means HOST1 never pulls `host2.conf` from git.
|
||||
|
||||
This is why the naming is what it is. The alternative — each host keeping a copy of its
|
||||
partner's container list — would need editing on both machines every time either one changed
|
||||
a service, and the two copies would silently diverge. Declaring once, on the host that owns
|
||||
the services, means a host is always the authority on what covering it requires.
|
||||
|
||||
TIER1 starts immediately. TIER2–4 activate after their delay thresholds
|
||||
(`HOST*_TIER2_DELAY` and friends, also in that host's own conf).
|
||||
|
||||
The DDNS container for that host's domain must be the **first entry in TIER1** — DNS coverage
|
||||
before anything else.
|
||||
|
||||
---
|
||||
|
||||
@@ -181,15 +199,15 @@ FALLBACK_TEST_HANDBACK_WAIT=300
|
||||
|
||||
---
|
||||
|
||||
### host2.conf — HOST2 covering HOST1
|
||||
### host1.conf — HOST1's own services (started by HOST2 when HOST1 is down)
|
||||
|
||||
```bash
|
||||
HOST2_DDNS_CONTAINERS=("Gmer4Lfe.us-DDNS")
|
||||
HOST1_DDNS_CONTAINERS=("Gmer4Lfe.com-DDNS")
|
||||
|
||||
FALLBACK_HOST2_STOP_ON_NO_NET=()
|
||||
FALLBACK_HOST1_STOP_ON_NO_NET=()
|
||||
|
||||
# Tier 1 — immediate (vital services + Live TV)
|
||||
FALLBACK_HOST2_COVERS_HOST1_TIER1=(
|
||||
FALLBACK_HOST1_TIER1=(
|
||||
"Gmer4Lfe.com-DDNS" # ALWAYS FIRST — DNS coverage before anything else
|
||||
"Emby" # media server — people are watching
|
||||
"NginxProxyManager" # reverse proxy — all external access routes through this
|
||||
@@ -205,7 +223,7 @@ FALLBACK_HOST2_COVERS_HOST1_TIER1=(
|
||||
)
|
||||
|
||||
# Tier 2 — after 4 hours (shared productivity services)
|
||||
FALLBACK_HOST2_COVERS_HOST1_TIER2=(
|
||||
FALLBACK_HOST1_TIER2=(
|
||||
"Postgres-NextCloud" # must start before NextCloud
|
||||
"NextCloud"
|
||||
"PostgreSQL-Immich" # must start before Immich
|
||||
@@ -214,7 +232,7 @@ FALLBACK_HOST2_COVERS_HOST1_TIER2=(
|
||||
)
|
||||
|
||||
# Tier 3 — after 12 hours (secondary services)
|
||||
FALLBACK_HOST2_COVERS_HOST1_TIER3=(
|
||||
FALLBACK_HOST1_TIER3=(
|
||||
"Organizrv2-Gmer4Lfe"
|
||||
"AdGuard-Home"
|
||||
"UptimeKuma"
|
||||
@@ -223,7 +241,7 @@ FALLBACK_HOST2_COVERS_HOST1_TIER3=(
|
||||
)
|
||||
|
||||
# Tier 4 — after 24 hours (arrs + downloaders)
|
||||
FALLBACK_HOST2_COVERS_HOST1_TIER4=(
|
||||
FALLBACK_HOST1_TIER4=(
|
||||
"Sonarr-Gmer4Lfe"
|
||||
"Radarr-Gmer4Lfe"
|
||||
"Lidarr-Gmer4Lfe"
|
||||
@@ -259,22 +277,22 @@ FALLBACK_HOST1_WRITEBACK_TIER3=(
|
||||
|
||||
---
|
||||
|
||||
### host1.conf — HOST1 covering HOST2
|
||||
### host2.conf — HOST2's own services (started by HOST1 when HOST2 is down)
|
||||
|
||||
```bash
|
||||
HOST1_DDNS_CONTAINERS=("Gmer4Lfe.com-DDNS")
|
||||
HOST2_DDNS_CONTAINERS=("Gmer4Lfe.us-DDNS")
|
||||
|
||||
FALLBACK_HOST1_STOP_ON_NO_NET=()
|
||||
FALLBACK_HOST2_STOP_ON_NO_NET=()
|
||||
|
||||
# Tier 1 — immediate (HOST2's vital services)
|
||||
FALLBACK_HOST1_COVERS_HOST2_TIER1=(
|
||||
FALLBACK_HOST2_TIER1=(
|
||||
"Gmer4Lfe.us-DDNS" # ALWAYS FIRST
|
||||
# HOST2's Tier 1 services — fill per HOST2's stack
|
||||
)
|
||||
|
||||
FALLBACK_HOST1_COVERS_HOST2_TIER2=(...)
|
||||
FALLBACK_HOST1_COVERS_HOST2_TIER3=(...)
|
||||
FALLBACK_HOST1_COVERS_HOST2_TIER4=(...)
|
||||
FALLBACK_HOST2_TIER2=(...)
|
||||
FALLBACK_HOST2_TIER3=(...)
|
||||
FALLBACK_HOST2_TIER4=(...)
|
||||
|
||||
# Tier delays for HOST2 outage
|
||||
HOST2_TIER2_DELAY=240
|
||||
@@ -492,7 +510,7 @@ Restart fallback.sh via User Scripts plugin. It will resume from NORMAL on its n
|
||||
|
||||
1. Create the container on the covering server (stopped), with volume mounts pointing at
|
||||
the mirrored share path (e.g. `/mnt/user/Movies` must exist on the covering server)
|
||||
2. Add the container name to `FALLBACK_HOST*_COVERS_HOST*_TIER*` in host*.conf
|
||||
2. Add the container name to `FALLBACK_<THAT-HOST>_TIER*` in **that host's own** conf
|
||||
in the appropriate tier position (dependency ordering — databases before apps)
|
||||
3. Verify: `fallback.sh --status` shows the container in the expected tier list
|
||||
4. Run `fallback_test.sh --dry-run` to confirm the full configuration is valid
|
||||
|
||||
@@ -115,7 +115,7 @@ Both servers run `fallback.sh` independently as a continuous background process.
|
||||
makes all decisions from two pings every `FALLBACK_CHECK_INTERVAL` seconds:
|
||||
|
||||
```bash
|
||||
ping REMOTE_TAILSCALE_IP # is the other server reachable?
|
||||
ping "$(resolve_tailscale_ip "$REMOTE_SERVER_NAME")" # is the other server reachable?
|
||||
ping EXTERNAL_IP # do I have internet? (default: 8.8.8.8)
|
||||
```
|
||||
|
||||
@@ -220,8 +220,8 @@ determines which server is local and which is remote at runtime, then selects th
|
||||
container arrays and tier delays from config via MY_ID.
|
||||
|
||||
```
|
||||
HOST2 covers HOST1: FALLBACK_HOST2_COVERS_HOST1_TIER* (in host2.conf)
|
||||
HOST1 covers HOST2: FALLBACK_HOST1_COVERS_HOST2_TIER* (in host1.conf)
|
||||
HOST2 covers HOST1: FALLBACK_HOST1_TIER* (in host2.conf)
|
||||
HOST1 covers HOST2: FALLBACK_HOST2_TIER* (in host1.conf)
|
||||
```
|
||||
|
||||
Both servers run identical scripts. MY_ID selects the correct arrays. No hostname
|
||||
|
||||
+33
-7
@@ -86,10 +86,25 @@ MEDIA_FILE_PATTERNS=(
|
||||
### Play State Sync
|
||||
|
||||
```bash
|
||||
PLAY_STATE_SYNC_ENABLED=true # toggle entire sync
|
||||
PLAY_STATE_SYNC_LOOKBACK_DAYS=30 # history window for played items
|
||||
PLAY_SYNC_ENABLED=true # toggle entire sync
|
||||
PLAY_SYNC_REMOTE=true # sync across hosts via Tailscale
|
||||
# false = local servers only (this host's Emby + Jellyfin)
|
||||
PLAY_SYNC_TYPES="Movie,Episode" # item types to sync — Audio excluded, music library too large
|
||||
PLAY_SYNC_FAV_TYPES="MusicArtist,MusicAlbum,Movie,Series" # favourites, union sync, never unmarks
|
||||
|
||||
PLAY_SYNC_PROBE=true # skip per-item work when nothing changed since last run
|
||||
PLAY_SYNC_PROBE_MAX_AGE_HOURS=24 # force a full comparison when the fingerprint is older
|
||||
|
||||
PLAY_SYNC_HANDBACK_RETRIES=5 # fallback handback: attempts before DNS cutover proceeds
|
||||
PLAY_SYNC_HANDBACK_RETRY_DELAY=60 # seconds between those attempts
|
||||
```
|
||||
|
||||
> **There is no date/lookback filter, and one must not be re-added.** An earlier version
|
||||
> gated on played-date; it was removed once the real cost was measured — the 30-minute
|
||||
> runtime was fork overhead per item, not API volume or item count. The fix was jq epoch
|
||||
> parsing plus the response-hash probe below. Re-introducing a date window would reduce
|
||||
> correctness (older items silently stop syncing) without meaningfully reducing runtime.
|
||||
|
||||
Emby and Jellyfin servers configured per-host:
|
||||
|
||||
```bash
|
||||
@@ -194,13 +209,24 @@ Show every file examined, not just those removed.
|
||||
### play_state_sync.sh
|
||||
|
||||
`play_state_sync.sh`
|
||||
Sync played/unplayed state and resume positions from local Emby to the remote Emby.
|
||||
Only items played within PLAY_STATE_SYNC_LOOKBACK_DAYS are synced.
|
||||
Sync played/unplayed state, resume positions and favourites across **every** configured Emby
|
||||
and Jellyfin server — not just local→remote. Servers are discovered from every
|
||||
`HOST*_TRANSCODE_SERVERS` entry, with remote hosts' localhost URLs rewritten to their
|
||||
Tailscale IP. Newest `LastPlayedDate` wins; state only ever moves forward, never clears.
|
||||
|
||||
No age filter — every matched item is considered on every run. The **change probe** is what
|
||||
keeps that cheap: the raw API responses are hashed and compared against the fingerprint from
|
||||
the last successful run, and per-item processing is skipped entirely when nothing moved.
|
||||
Fetches still happen every run, so nothing can be missed by the probe.
|
||||
|
||||
`play_state_sync.sh --full`
|
||||
Ignore PLAY_STATE_SYNC_LOOKBACK_DAYS — sync all played items regardless of age. May be
|
||||
slow on large libraries. Use after a new Emby install or database restore to rebuild
|
||||
full play history.
|
||||
Bypass the change probe and force the full per-item comparison even when the fingerprint
|
||||
matches. Use after a new Emby install or database restore, or when debugging a sync that
|
||||
appears to be skipping work it should be doing.
|
||||
|
||||
`play_state_sync.sh --wait`
|
||||
Wait for an in-progress run instead of exiting. For manual runs that would otherwise be
|
||||
skipped by the scheduled every-30-minute pass. Used by `fallback.sh` during handback.
|
||||
|
||||
`play_state_sync.sh --dry-run`
|
||||
Show what would be synced without writing any state.
|
||||
|
||||
@@ -100,3 +100,43 @@ syncs between them.
|
||||
| `media_shares_permissions.sh` | Apply `nobody:users` ownership + correct permissions to all media shares | Daily — runs first |
|
||||
| `media_cleaner.sh` | Remove junk files (two profiles: `anime` + `media`) | Daily — runs before arr cleanup |
|
||||
| `play_state_sync.sh` | Sync watched/played state + resume positions across Emby + Jellyfin | Every 30 min |
|
||||
|
||||
---
|
||||
|
||||
## ━━━ THE ctime INVARIANT — READ BEFORE CHANGING PERMISSIONS ━━━
|
||||
|
||||
`media_shares_permissions.sh` applies every pass **conditionally** — it touches only entries
|
||||
whose owner or mode is actually wrong. That is not an optimisation, and it must stay that way.
|
||||
|
||||
`chown` and `chmod` rewrite an inode's ctime **even when the value does not change**. A
|
||||
blanket pass would therefore restamp every file in the library every night.
|
||||
|
||||
The arr cleanup scripts (`sonarr_cleanup.sh`, `radarr_cleanup.sh`, `lidarr_cleanup.sh`) gate
|
||||
orphan deletion on ctime. mtime cannot substitute: an import preserves the release's original
|
||||
timestamp, so mtime says nothing about when a file arrived here. Measured 2026-07-27 — of 400
|
||||
files imported that week, **all 400 had mtimes over 7 days old, one of them 9613 days.**
|
||||
|
||||
So:
|
||||
|
||||
```
|
||||
blanket chown/chmod → every ctime resets to today
|
||||
→ no file ever appears older than *_ORPHAN_AGE
|
||||
→ orphan collection silently stops
|
||||
→ nothing errors, nothing warns, disk just fills
|
||||
```
|
||||
|
||||
**The failure is invisible.** No script fails, no notification fires. The only symptom is
|
||||
orphans quietly accumulating until a pool fills — which is exactly how the 755 GB / 89%-full
|
||||
cache pool incident happened.
|
||||
|
||||
Two rules follow, and both are load-bearing:
|
||||
|
||||
1. **`media_shares_permissions.sh` passes stay conditional.** Making any of them unconditional
|
||||
breaks orphan collection ecosystem-wide.
|
||||
2. **`Tools/bulk_permissions_repair.sh` is unconditional by design** — it exists to repair
|
||||
known-wrong paths where correctness beats preserving a clock. That is precisely why it is a
|
||||
manual, targeted tool and not scheduled. Pointing it at a whole media root pauses orphan
|
||||
collection there for `*_ORPHAN_AGE` days.
|
||||
|
||||
Both scripts' headers carry this warning too. If you are reading this because you are about to
|
||||
"simplify" the permissions job, this is the thing that breaks.
|
||||
|
||||
@@ -230,7 +230,7 @@ data — it just doesn't trigger a notification for that condition.
|
||||
| State File | Source | What It Shows |
|
||||
|-----------|--------|---------------|
|
||||
| `FALLBACK_STATE_FILE` | `Fallback/fallback.sh` | Current fallback state (NORMAL/FALLBACK/etc.) |
|
||||
| `SYS_WATCHDOG_FAILED_FILE` | `Watchdogs/docker_watchdog.sh` | Container skip list — needs human attention |
|
||||
| `DOCKER_WATCHDOG_FAILED_FILE` | `Watchdogs/docker_watchdog.sh` | Container skip list — needs human attention |
|
||||
| `WATCHDOG_STATE_FILE` | `Watchdogs/docker_watchdog.sh` | Active container strike counts |
|
||||
| `SYS_WATCHDOG_STATE_FILE` | `Watchdogs/stability_watchdog.sh` | Active system watchdog strikes |
|
||||
| `BANDWIDTH_LOG` | `bandwidth_monitor.sh` | Yesterday's transfer history |
|
||||
|
||||
@@ -153,7 +153,7 @@ Daily 8am:
|
||||
weekly_health_digest.sh ── reads ──────────► FALLBACK_STATE_FILE
|
||||
── reads ──────────► WATCHDOG_STATE_FILE
|
||||
── reads ──────────► SYS_WATCHDOG_STATE_FILE
|
||||
── reads ──────────► SYS_WATCHDOG_FAILED_FILE
|
||||
── reads ──────────► DOCKER_WATCHDOG_FAILED_FILE
|
||||
── reads ──────────► BANDWIDTH_LOG
|
||||
── reads ──────────► TRANSCODE_DAILY_LOG
|
||||
── reads ──────────► TUNING_MONITOR_LOG
|
||||
|
||||
@@ -538,7 +538,7 @@ Runs on both servers; `detect_hosts()` determines which direction each sync goes
|
||||
# Drive temperature exit codes respected — skip share or abort all on CRIT.
|
||||
#
|
||||
# 3. Post-sync: DAILY_MAINTENANCE_SCRIPTS (everything except git pull)
|
||||
# media_management.sh → permissions + cleaners + arr cleanup
|
||||
# Media/ + Arrs_Stack/ scripts → permissions, cleaners, arr cleanup, classification
|
||||
# docker_daily_restart.sh → nightly container restarts
|
||||
# ─────────────────────────────────────────────────────────────────────────────
|
||||
```
|
||||
@@ -557,12 +557,12 @@ Runs on both servers; `detect_hosts()` determines which direction each sync goes
|
||||
#
|
||||
# HOST1 runs this script at 1am:
|
||||
# → pushes HOST1_DAILY_SYNC_SHARES (Movies, Tv_Shows, Music) → HOST2
|
||||
# → media_management.sh on HOST1's shares
|
||||
# → Media/ + Arrs_Stack/ scripts on HOST1's shares
|
||||
# → docker_daily_restart.sh on HOST1's containers
|
||||
#
|
||||
# HOST2 runs this script at 1am:
|
||||
# → pushes HOST2_DAILY_SYNC_SHARES (Anime_Shows, Anime_Movies) → HOST1
|
||||
# → media_management.sh on HOST2's shares
|
||||
# → Media/ + Arrs_Stack/ scripts on HOST2's shares
|
||||
# → docker_daily_restart.sh on HOST2's containers
|
||||
# ─────────────────────────────────────────────────────────────────────────────
|
||||
```
|
||||
|
||||
@@ -144,6 +144,8 @@ independently at 4-hour cadence.
|
||||
| `partnership_transfer.sh` | Transfer ownership from current owner to current mirror | Via `partnership_manager.sh --transfer`; owner only |
|
||||
| `ssh_setup.sh` | SSH key generation, remote install, auth validation | Called by onboard; manually for re-keying or validation |
|
||||
| `onboard_cancel.sh` | Remove SSH keys in one or both directions, clear setup flags | During cancelled or failed onboard; manual cleanup |
|
||||
| `gitea_ssh_setup.sh` | Generate a Gitea keypair and register it via the Gitea API | Called during onboard; manually when re-keying a server |
|
||||
| `share_setup.sh` | Create missing Unraid shares on the mirror from the owner's sync lists | Called during onboard; safe to re-run — existing shares are never modified |
|
||||
|
||||
---
|
||||
|
||||
|
||||
+70
-1
@@ -607,7 +607,7 @@ DEFAULT_RSYNC_OPTS="-az --no-perms --no-owner --no-group --inplace"
|
||||
BW_LIMIT=0 # KB/s, 0 = unlimited
|
||||
RETRY_COUNT=3
|
||||
SLEEP=60 # seconds between retries
|
||||
ROOTFS_WARN_PCT=75 # abort if remote rootfs above this %
|
||||
ROOTFS_WARN=75 # abort if remote rootfs above this %
|
||||
|
||||
# Shared with Monitors/
|
||||
BANDWIDTH_LOG="$DATA_DIR/bandwidth_history.db"
|
||||
@@ -696,3 +696,72 @@ name doesn't match any profile key, or when testing a specific profile.
|
||||
rsync.sh /mnt/user/appdata-Fallback/Critical-Data --profile=critical-data
|
||||
rsync.sh /mnt/user/Media_Server/Emby --profile=emby
|
||||
```
|
||||
|
||||
### --seed
|
||||
|
||||
Skip the empty-remote-share guard. Normally `check_remote_share()` aborts when the target
|
||||
directory on the remote is missing or empty, because that usually means the array is not
|
||||
mounted and syncing into it would write onto the underlying root filesystem. `--seed` is the
|
||||
deliberate exception, for the first push of a genuinely new share.
|
||||
|
||||
```bash
|
||||
rsync.sh /mnt/user/New_Share --seed
|
||||
```
|
||||
|
||||
### --merge-run
|
||||
|
||||
Bidirectional merge. Two passes:
|
||||
|
||||
```
|
||||
Pass 1 pull remote → local, with --ignore-existing
|
||||
Pass 2 push local → remote, with --delete
|
||||
```
|
||||
|
||||
Local ends up authoritative: the remote may contribute files the local lacks, but never a
|
||||
competing version of a file the local already has. Also triggered automatically when the
|
||||
pre-scan detects ≥75% directory overlap with the remote.
|
||||
|
||||
**`--delete` is applied only if pass 1 completed.** Its entire justification is "local is now
|
||||
the authoritative superset", and a pull that was capped by `RSYNC_MAX_RUNTIME_HOURS` or failed
|
||||
outright means it is not — the remote still holds content the local never received. On an
|
||||
incomplete pull the push runs **without** `--delete`, warns, and notifies. Local content still
|
||||
propagates; nothing remote-unique is destroyed; the delete happens on a later run whose pull
|
||||
actually succeeded.
|
||||
|
||||
> This was a real defect, fixed 2026-08-01. `--delete` used to be unconditional, so a capped
|
||||
> or failed pull would permanently delete remote content the pull had not yet retrieved — and
|
||||
> the timeout branch's own comment promised it would "resume the pull next run", which was
|
||||
> impossible once the data was gone. If you are changing this function, that interlock is the
|
||||
> part that matters.
|
||||
|
||||
---
|
||||
|
||||
## ━━━ SOURCE PATH GUARDS ━━━
|
||||
|
||||
`rsync.sh` pushes to `root@remote` at the **same absolute path**, and `--merge-run` adds
|
||||
`--delete`. The source argument is therefore a remote-side hazard, not just a local one, so
|
||||
two guards run before anything else:
|
||||
|
||||
| Guard | Rejects |
|
||||
|-------|---------|
|
||||
| Absolute path, ≥3 components | `/`, `/mnt`, `/mnt/user`, empty |
|
||||
| Directory must exist locally | typo'd paths |
|
||||
|
||||
Every real job path clears both — see `HOST*_DAILY_SYNC_SHARES` and friends, all of which are
|
||||
`/mnt/user/<Share>` or deeper.
|
||||
|
||||
Without the existence check, a mistyped path under `--merge-run` would be *created* locally by
|
||||
the pull pass, filled from the remote, then pushed back with `--delete`.
|
||||
|
||||
---
|
||||
|
||||
## ━━━ RUNTIME CEILING ━━━
|
||||
|
||||
No single transfer attempt may exceed `RSYNC_MAX_RUNTIME_HOURS` (default 23). On timeout the
|
||||
transfer is terminated and resumes on the next scheduled run.
|
||||
|
||||
This is only safe because `--partial` is in `DEFAULT_RSYNC_OPTS` — a terminated transfer picks
|
||||
up where it left off rather than restarting. The cap exists so one huge or stuck transfer
|
||||
cannot hold its per-profile lock indefinitely and starve every other profile of a turn.
|
||||
|
||||
Both `--merge-run` passes are capped independently.
|
||||
|
||||
@@ -77,7 +77,7 @@ Orchestrators/
|
||||
────────────────────────► inotify_tuning.sh (before docker_network_connect)
|
||||
array_stopping.sh ───────────────────────► conf_cache_save.sh (first — while cache is fresh)
|
||||
───────────────────────► rsync_stop.sh --rsync-only (stop transfers)
|
||||
weekly_maintenance.sh ──────────────────► clear_logs.sh
|
||||
weekly_sync_maintenance.sh ──────────────────► clear_logs.sh
|
||||
|
||||
Watchdogs/System/
|
||||
conf_cache_watchdog.sh ─────────────────► maintains $PERSISTENT_CONF_CACHE while partner offline
|
||||
@@ -109,7 +109,7 @@ Plugin/unraid/Watchdogs/System/
|
||||
| `conf_sync.sh` | Sync partner confs: pull → RAM cache, push own conf to partners | At array start + every 4 h (--pull-only) |
|
||||
| `conf_cache_save.sh` | Snapshot partner RAM cache → `$PERSISTENT_CONF_CACHE` | At array stop — first step |
|
||||
| `conf_cache_restore.sh` | Load missing partner confs from persistent backup into RAM | At array start — after conf_sync |
|
||||
| `clear_logs.sh` | Size-threshold log cleanup | Weekly via weekly_maintenance.sh |
|
||||
| `clear_logs.sh` | Size-threshold log cleanup | Weekly via weekly_sync_maintenance.sh |
|
||||
| `rsync_stop.sh` | Orchestrator-aware rsync stop | Manual / called by array_stopping.sh |
|
||||
| `server_reboot.sh` | Graceful reboot with pre-flight warnings | Manual |
|
||||
|
||||
@@ -141,7 +141,7 @@ Every 15 minutes (watchdog_orchestrator.sh):
|
||||
Every 4 hours (intermediate_sync_maintenance.sh):
|
||||
└─ conf_sync.sh --pull-only ← refresh partner conf in RAM without pushing own conf
|
||||
|
||||
Weekly (weekly_maintenance.sh):
|
||||
Weekly (weekly_sync_maintenance.sh):
|
||||
└─ clear_logs.sh
|
||||
System logs: clear if > LOG_MIN_SIZE_MB
|
||||
Docker logs: clear per-container if > LOG_DOCKER_MAX_MB
|
||||
|
||||
@@ -250,7 +250,7 @@ docker start ContainerName
|
||||
# Both live on /boot/config — survive reboots intentionally.
|
||||
# A container that was skip-listed before a reboot is still broken after it.
|
||||
|
||||
$SYS_WATCHDOG_FAILED_FILE # persistent skip list
|
||||
$DOCKER_WATCHDOG_FAILED_FILE # persistent skip list
|
||||
$WATCHDOG_CONTAINER_RESTART_LOG # restart loop tracking
|
||||
```
|
||||
|
||||
|
||||
@@ -130,6 +130,7 @@ The relationship is one-way: Tools act on state that other scripts have written.
|
||||
| `emby_to_lidarr_sync.sh` | Add all Emby album artists not yet tracked in Lidarr | After Lidarr setup, database wipe, or when you suspect gaps |
|
||||
| `emby_to_sonarr_sync.sh` | Add all Emby TV series not yet tracked in Sonarr | After Sonarr setup, database wipe, or when you suspect gaps |
|
||||
| `emby_to_radarr_sync.sh` | Add all Emby movies not yet tracked in Radarr | After Radarr setup, database wipe, or when you suspect gaps |
|
||||
| `trailer_folder_migration.sh` | Relocate TV trailers Emby was ingesting as fake episodes into `<show>/trailers/` | Once, after Trailarr's Series Trailers profile was corrected |
|
||||
|
||||
---
|
||||
|
||||
|
||||
@@ -234,3 +234,53 @@ Every 15 minutes — watchdog_orchestrator.sh fires:
|
||||
| `NETWORK_WATCHDOG_NPM_STATE_FILE` | `System/network_watchdog.sh` | itself | NPM proxy strike count |
|
||||
|
||||
All state files are in `$STATE_DIR` (adapts to storage mode). See `master.conf` for actual variable values.
|
||||
|
||||
---
|
||||
|
||||
## ━━━ SAFEGUARDS COMMON TO THIS FOLDER ━━━
|
||||
|
||||
These run unattended every 15 minutes and hold the only irreversible remedy in the
|
||||
ecosystem, so the guards matter as much as the checks. Each script's header documents its
|
||||
full set; these are the folder-wide ones.
|
||||
|
||||
**Nothing acts on a single reading.** Every check that can trigger an action is strike-gated.
|
||||
One bad sample is a reading; N consecutive samples is a condition. Strikes clear themselves
|
||||
when the condition resolves, so a transient blip never accumulates toward an alarm across
|
||||
unrelated weeks.
|
||||
|
||||
**Escalation is asymmetric on purpose.** Protective actions happen fast; undoing them happens
|
||||
slowly. `resource_watchdog.sh` throttles before it pauses and pauses before it stops, then
|
||||
requires `RW_RECOVER_CYCLES` clear cycles and de-escalates one level at a time. Recovering on
|
||||
a single good reading would flap under sustained load.
|
||||
|
||||
**Ownership boundaries are enforced, not conventional.** Container health belongs to
|
||||
`docker_watchdog.sh`; `stability_watchdog.sh` deliberately does not check it, and its Docker
|
||||
daemon check writes a flag for `docker_watchdog.sh` rather than acting. Two watchdogs
|
||||
remediating the same subsystem would race.
|
||||
|
||||
**Watchdogs coordinate through state files.** `resource_watchdog.sh` publishes
|
||||
`mem_shutdown_active` so `docker_watchdog.sh` defers restarts — otherwise one would stop
|
||||
containers to free RAM while the other restarted them to restore health. The reader applies a
|
||||
staleness guard so a crashed writer cannot suppress restarts forever.
|
||||
|
||||
**Data safety outranks uptime.** `stability_watchdog.sh` aborts a reboot while a ZFS pool is
|
||||
unhealthy, parity is running, or the mover is active. Only Tier 1 — conditions the system
|
||||
cannot recover from and which worsen every cycle — bypasses that.
|
||||
|
||||
**Reboot loops end in shutdown, not more reboots.** No more than
|
||||
`SYS_WATCHDOG_REBOOT_LIMIT` reboots within `SYS_WATCHDOG_REBOOT_WINDOW_HRS`. On hitting the
|
||||
limit the host powers off: a fault that survives repeated reboots will not be fixed by more
|
||||
of them, and a cleanly-down box is better than one cycling endlessly.
|
||||
|
||||
**An aborted reboot restores what it stopped.** An EXIT trap is armed the moment containers
|
||||
start being stopped for a reboot and disarmed only once the reboot commits. If the script
|
||||
dies in between, everything it stopped comes back — the failure mode is a running system,
|
||||
never a host left stripped with no reboot.
|
||||
|
||||
**An unconfigured job list fails loudly.** An empty `WATCHDOG_ORCHESTRATOR_SCRIPTS` or
|
||||
`SYSTEM_WATCHDOG_SCRIPTS` would otherwise report "0/0 passed" and exit 0 every cycle —
|
||||
indistinguishable from healthy while nothing is monitored at all.
|
||||
|
||||
**Verify the path, not the process.** `network_watchdog.sh` checks NPM by fetching an external
|
||||
URL rather than asking whether the container is running. A running container behind broken DNS
|
||||
or a broken upstream still serves nothing.
|
||||
|
||||
Reference in New Issue
Block a user