doc audit — fix stale watchdog vars, fallback_state_reset, add monthly_maintenance and mesh_monitor sections

- Tools/Manual-Tools.md: fallback_state_reset.sh marked not yet built; manual
  workaround added; TOC entry updated; failover→fallback rename throughout
- Watchdogs/Manual-Watchdogs.md: System Watchdog config block updated — remove
  pre-split vars (MEM_WARN_GB, MEM_SHUTDOWN_GB, MEM_GB, MEM_RECOVER_GB,
  SYS_WATCHDOG_MEM_SHUTDOWN_EXCLUDED, SYS_WATCHDOG_REQUIRED_CONTAINERS);
  add SYS_WATCHDOG_MEM_GB (reboot trigger only); fix state file path names
  (watchdog_state.db → container_watchdog_state.db,
  sys_watchdog_state.db → system_watchdog_state.db)
- Orchestrators/README-Orchestrators.md: add monthly_maintenance.sh section
  with two-gate model, configuration reference, and usage examples
- Monitors/README-Monitors.md: add mesh_monitor.sh to both tables and diagram
- Monitors/Manual-Monitors.md: add mesh_monitor.sh section
- README-User_Script_Plug-in.md, Rsync/README-Rsync.md, Rsync/Manual-Rsync.md,
  README.md: emby-failover → emby-fallback (missed from prior batch)
This commit is contained in:
Gmer4Lfe
2026-05-22 22:07:46 -04:00
parent 070013cb4f
commit 2fa78a21b6
9 changed files with 131 additions and 52 deletions
+47 -3
View File
@@ -104,7 +104,7 @@ array_started.sh
# Cron — one entry per maintenance window:
*/3 * * * * transcode_management.sh
0 */6 * * * arrs_failed_stalled_recovery.sh
*/30 * * * * rsync.sh emby-failover ← not an orchestrator, direct call
*/30 * * * * rsync.sh emby-fallback ← not an orchestrator, direct call
0 1 * * * daily_sync_maintenance.sh
30 2 * * 0 weekly_sync_maintenance.sh
@@ -730,7 +730,7 @@ maintenance block before the 7am coffee report.
# ─────────────────────────────────────────────────────────────────────────────
# Two Emby syncs run in parallel — dirty and clean:
#
# emby-failover dirty sync (every 30 minutes, Emby running):
# emby-fallback dirty sync (every 30 minutes, Emby running):
# watch states, library deltas, user activity — continuous coverage
# WAL files excluded — safe to copy while Emby writes
# HOST2 always within 30 minutes of HOST1 on playback state
@@ -838,6 +838,50 @@ weekly_sync_maintenance.sh --status # show configured shares, jobs, update t
---
## ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
## 📆 monthly_maintenance.sh
## ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
Uptime-triggered orchestrator for long-running system tasks — ZFS scrub, SMART
long tests — that should only run on stable systems that have been up for at least
30 days. Called daily by cron; most invocations are silent no-ops.
Two gates must both pass before any job runs:
1. Server uptime ≥ `MONTHLY_UPTIME_THRESHOLD_DAYS`
2. Last run ≥ `MONTHLY_RUN_INTERVAL_DAYS` ago (state file on `/boot/config/` — survives reboots)
If either gate fails, the script exits 0 with no output. This is expected — it runs
daily and most days are no-ops.
`--force` bypasses both gates and runs the job list immediately. Use for testing or
when a scrub was missed and the server hasn't reached the uptime threshold yet.
### Configuration (master.conf)
```bash
MONTHLY_MAINTENANCE_SCRIPTS=(
#"Tools/zfs_pool_scrub.sh"
#"Tools/smart_long_test.sh"
)
MONTHLY_UPTIME_THRESHOLD_DAYS=30
MONTHLY_RUN_INTERVAL_DAYS=30
MONTHLY_LAST_RUN_FILE="/boot/config/monthly_maintenance_last_run.db"
```
Scripts are commented out by default — uncomment what applies to your hardware.
### Usage
```bash
monthly_maintenance.sh # normal run (daily cron — silent no-op when gates not met)
monthly_maintenance.sh --force # bypass both gates — run immediately
monthly_maintenance.sh --dry-run # show what would run without running it
monthly_maintenance.sh --status # show gate state: uptime, last run, next eligible
monthly_maintenance.sh --log # verbose output from each child script
```
---
## ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
## 🧹 media_management.sh
## ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
@@ -906,7 +950,7 @@ array_started.sh
# ─────────────────────────────────────────────────────────────────────────────
# Periodic — not an orchestrator, direct rsync call:
# ─────────────────────────────────────────────────────────────────────────────
*/30 * * * * rsync.sh /mnt/user/Media_Server/Emby --profile=emby-failover
*/30 * * * * rsync.sh /mnt/user/Media_Server/Emby --profile=emby-fallback
# ─────────────────────────────────────────────────────────────────────────────
# Every 6 hours: