diff --git a/Monitors/README-Monitors.md b/Monitors/README-Monitors.md index afcd8a0..e790180 100644 --- a/Monitors/README-Monitors.md +++ b/Monitors/README-Monitors.md @@ -82,7 +82,7 @@ drive SMART attributes, ZFS pool state, ARC statistics, kernel memory pressure. ``` Monitors/ ← observes and reports (this folder) Docker_Essentials/ ← acts on containers (docker_watchdog starts/stops) -unRAID_Essentials/ ← acts on the server (system_watchdog, inotify_tuning) +unRAID_Essentials/ ← acts on the server (stability_watchdog, inotify_tuning) Fallback/ ← acts on the full stack (failover, handback) Rsync/ ← calls bandwidth_monitor.sh (auto-logs each sync) ``` diff --git a/README-User_Script_Plug-in.md b/README-User_Script_Plug-in.md index 0174dda..395717c 100644 --- a/README-User_Script_Plug-in.md +++ b/README-User_Script_Plug-in.md @@ -111,14 +111,14 @@ docker_syslog_filter.sh — suppresses Docker veth noise before containers crea php_fpm_max_children.sh — sets PHP-FPM workers to 250 before WebGUI serves requests ramdisk_setup.sh — creates /mnt/ramdisk_transcodes and symlink before Emby starts docker_network_connect.sh— ensures custom networks exist + containers connected -system_watchdog.sh ← continuous — three-tier server stability watchdog +stability_watchdog.sh ← last-resort — three-tier server stability watchdog docker_watchdog.sh ← continuous — two-tier container healing watchdog failover.sh ← continuous — mutual failover state machine ``` **After running:** One-shot scripts exit cleanly. Three continuous processes run in the background until -the array stops. Check with: `pgrep -f "system_watchdog\|docker_watchdog\|failover"` +the array stops. Check with: `pgrep -f "stability_watchdog\|docker_watchdog\|failover"` --- @@ -899,7 +899,7 @@ Background: NO **What it shows:** Strike counts for all monitored containers. Current skip list with running/stopped state. Whether startup grace period is active and how long remains. Whether RAM emergency -deferral is active (mem_shutdown_active from system_watchdog). Last cycle timing. +deferral is active (mem_shutdown_active from stability_watchdog). Last cycle timing. --- @@ -1200,7 +1200,7 @@ flip count, session split ramdisk vs SSD). ## 🖥️ UNRAID_ESSENTIALS ## ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ -> **system_watchdog.sh is started by array_start.sh — not scheduled separately.** +> **stability_watchdog.sh is called by watchdog_orchestrator.sh — not scheduled separately.** > **webgui_watchdog.sh, system_tuning_monitor.sh, clear_logs.sh are scheduled below.** > **inotify_tuning.sh, php_fpm_max_children.sh, docker_syslog_filter.sh run via array_start.sh.** @@ -1264,11 +1264,11 @@ Active containers like Emby and SABnzbd grow fastest. Truncation not rotation --- -### ── system_watchdog_status ────────────────────────────────────────────────── +### ── stability_watchdog_status ────────────────────────────────────────────────── ``` Category: unRAID_Essentials -Name: system_watchdog_status +Name: stability_watchdog_status Schedule: None — run manually Background: NO ``` @@ -1632,7 +1632,7 @@ Correct startup setup: Wrong startup setup (old approach): ramdisk_setup — At Startup of Array docker_syslog_filter — At Startup of Array - system_watchdog — At Startup of Array + stability_watchdog — Every Minute via watchdog_orchestrator docker_watchdog — At Startup of Array failover — At Startup of Array (unpredictable order, no coordination) @@ -1707,5 +1707,5 @@ critical_sync_maintenance verify Critical-Data sync completes to HOST2 daily_sync_maintenance --dry-run verify remote connectivity and share list failover_status confirm NORMAL state on both servers docker_watchdog_status confirm required containers all running -system_watchdog_status confirm no threshold alerts on first check +stability_watchdog_status confirm no threshold alerts on first check ``` \ No newline at end of file diff --git a/README.md b/README.md index 8c2fe27..887dee1 100644 --- a/README.md +++ b/README.md @@ -448,8 +448,8 @@ Failover coverage: serves Gmer4Lfe.us via DDNS Monitoring: - Both servers resource/docker/storage/ resource/docker/storage/ - system_watchdog.sh system_watchdog.sh + Both servers resource/docker/system/ resource/docker/system/ + stability_watchdog.sh stability_watchdog.sh failover.sh failover.sh Sunday morning coffee report Sunday morning coffee report ``` @@ -476,7 +476,7 @@ Unraid_Scripts/ ├── Failover/ ← Mutual automatic failover — continuous background process │ README: README-Failover.md │ -├── Watchdogs/ ← All watchdog scripts: resource, docker, storage, system +├── Watchdogs/ ← All watchdog scripts: resource, docker, system, stability │ README: README-Watchdogs.md │ ├── Docker_Essentials/ ← Container lifecycle: restarts, updates, networks @@ -548,8 +548,8 @@ Every minute: watchdog_orchestrator.sh fires each watchdog in sequence → resource_watchdog.sh reduce pressure before healing attempts → docker_watchdog.sh two-tier container healing - → storage_watchdog.sh pool growth + runaway log detection - → system_watchdog.sh last resort — reboots when all else fails + → system_watchdog.sh system components: storage + WebGUI health + → stability_watchdog.sh last resort — reboots when all else fails Every 3 minutes: transcode_management.sh cleanup → manager (order non-negotiable) @@ -593,8 +593,8 @@ What actually happens on a typical day, from the ecosystem's perspective: Throughout the day (every minute via watchdog_orchestrator.sh): resource_watchdog.sh managing: system pressure (throttle/pause/stop) docker_watchdog.sh healing: memory leaks, HTTP failures, required containers - storage_watchdog.sh watching: appdata growth rate, runaway log files - system_watchdog.sh watching: RAM, CPU temp, rootfs, kernel, daemon + system_watchdog.sh watching: system components (storage growth, WebGUI) + stability_watchdog.sh watching: RAM, CPU temp, rootfs, kernel, daemon Throughout the day: failover.sh watching: remote server, internet connectivity @@ -639,13 +639,13 @@ Problem: Authelia crashed and won't restart → Tools: watchdog_skip_list_manager.sh to clear after fixing Problem: RAM drops to 3GB - → system_watchdog.sh: stop non-essential containers + → stability_watchdog.sh: stop non-essential containers → Wait for RAM to recover to 30GB → docker_watchdog.sh defers (reads mem_shutdown_active=true) → If RAM drops to 4GB AND 3+ OOM kills: bypass strikes → reboot Problem: Docker daemon hung - → system_watchdog.sh Tier 1: attempt /etc/rc.d/rc.docker restart + → stability_watchdog.sh Tier 1: attempt /etc/rc.d/rc.docker restart → If still hung: immediate reboot (no strikes needed — daemon can't be managed) Problem: HOST1 loses power @@ -713,7 +713,7 @@ Things that are different from what you might expect: # scheduled directly. array_start.sh is the only "At Startup of Array" entry. # 2. Scripts are silent when healthy. -# system_watchdog.sh runs 288 times per day. It produces zero output on clean +# stability_watchdog.sh runs 288 times per day. It produces zero output on clean # cycles. If you see output, something needs attention. If you see nothing, # the system is healthy. This is by design. diff --git a/Rsync/Manual-Rsync.md b/Rsync/Manual-Rsync.md index ef75b46..b6f5bbc 100644 --- a/Rsync/Manual-Rsync.md +++ b/Rsync/Manual-Rsync.md @@ -439,7 +439,7 @@ bash /mnt/user/appdata/unraid_scripts/Orchestrators/array_start.sh This single entry launches everything defined in ARRAY_START_SCRIPTS from master.conf: inotify_tuning, docker_syslog_filter, php_fpm_max_children, ramdisk_setup, -docker_network_connect, system_watchdog, docker_watchdog, failover. +docker_network_connect, stability_watchdog, docker_watchdog, failover. ### Cron Schedule diff --git a/Tools/Manual-Tools.md b/Tools/Manual-Tools.md index cbd7ba1..13131af 100644 --- a/Tools/Manual-Tools.md +++ b/Tools/Manual-Tools.md @@ -562,7 +562,7 @@ changes to any running process, container, or state file. ### What It Shows ``` -system_watchdog +stability_watchdog Running state, PID, uptime, approximate cycle count Active strikes, recent restart history Live snapshot: rootfs, RAM, ZFS ARC, load, zombie count, CPU temp diff --git a/Watchdogs/Manual-Watchdogs.md b/Watchdogs/Manual-Watchdogs.md index 587827a..b6713f1 100644 --- a/Watchdogs/Manual-Watchdogs.md +++ b/Watchdogs/Manual-Watchdogs.md @@ -1,7 +1,7 @@ # ━━━━━ WATCHDOGS — Manual ━━━━━ -Configuration reference, operational procedures, and troubleshooting for all four -watchdog scripts. For design philosophy and script relationships see `README-Watchdogs.md`. +Configuration reference, operational procedures, and troubleshooting for all watchdog +scripts. For design philosophy and script relationships see `README-Watchdogs.md`. For the orchestrator that calls these scripts see `Orchestrators/watchdog_orchestrator.sh`. --- @@ -11,8 +11,10 @@ For the orchestrator that calls these scripts see `Orchestrators/watchdog_orches - [Output Tiers](#output-tiers) - [resource_watchdog.sh](#resource_watchdogsh) - [docker_watchdog.sh](#docker_watchdogsh) -- [storage_watchdog.sh](#storage_watchdogsh) - [system_watchdog.sh](#system_watchdogsh) +- [System/storage_watchdog.sh](#systemstorage_watchdogsh) +- [System/webgui_watchdog.sh](#systemwebgui_watchdogsh) +- [stability_watchdog.sh](#stability_watchdogsh) - [Full Configuration Reference](#full-configuration-reference) - [Troubleshooting](#troubleshooting) @@ -23,7 +25,7 @@ For the orchestrator that calls these scripts see `Orchestrators/watchdog_orches All watchdog scripts use a two-tier output model: `echo` lines are always visible; `log` lines only appear when `--log` is passed. -All four watchdogs are **single-pass scripts** called once per minute by the orchestrator. +All watchdog scripts are **single-pass scripts** called once per minute by the orchestrator. Without `--log`, only state transitions, warnings, errors, and the conclusion line are visible. Per-check detail is suppressed on clean cycles. @@ -285,9 +287,26 @@ docker start ContainerName --- -## storage_watchdog.sh +## system_watchdog.sh -Runs third in the orchestrator sequence. Two independent checks per cycle: +Thin orchestrator — runs `SYSTEM_WATCHDOG_SCRIPTS` from master.conf sequentially each cycle. +Called third by `watchdog_orchestrator.sh`. Covers all system component watchdogs. +Can also be run standalone to check all system components at once. + +### Usage + +```bash +system_watchdog.sh # run all system component watchdogs +system_watchdog.sh --status # show configured scripts and their paths +system_watchdog.sh --dry-run # preview without executing anything +system_watchdog.sh --log # verbose output +``` + +--- + +## System/storage_watchdog.sh + +Called by `system_watchdog.sh` each cycle. Two independent checks per cycle: growth rate detection (automatic, zero config) and oversize log detection. Uses its own strike state file — independent from docker_watchdog. @@ -343,7 +362,7 @@ Log strikes auto-clear when the file drops below threshold. ### Usage ```bash -storage_watchdog.sh # single pass (called by watchdog_orchestrator.sh) +storage_watchdog.sh # single pass (called by system_watchdog.sh) storage_watchdog.sh --status # strikes, growth baseline age, suppress ceilings storage_watchdog.sh --dry-run # show what would be alerted/truncated storage_watchdog.sh --log # verbose per-container output @@ -351,7 +370,46 @@ storage_watchdog.sh --log # verbose per-container output --- -## system_watchdog.sh +## System/webgui_watchdog.sh + +Called by `system_watchdog.sh` each cycle. Monitors WebGUI availability and escalates +through three restart steps if unresponsive. Silent when healthy. + +### Escalation Path + +``` +WebGUI responding → exit 0 (silent) + +Not responding: + Step 1 — nginx restart → wait WEBGUI_NGINX_WAIT → recheck + Step 2 — php-fpm restart → wait WEBGUI_PHP_WAIT → recheck + Step 3 — emhttp restart → wait WEBGUI_EMHTTP_WAIT → recheck + All three failed → critical notify, manual intervention needed +``` + +### Configuration + +```bash +# master.conf +WEBGUI_URL="http://localhost" +WEBGUI_TIMEOUT=5 +WEBGUI_NGINX_WAIT=15 +WEBGUI_PHP_WAIT=10 +WEBGUI_EMHTTP_WAIT=30 +``` + +### Usage + +```bash +webgui_watchdog.sh # single pass (called by system_watchdog.sh) +webgui_watchdog.sh --status # current WebGUI state + nginx/php-fpm/emhttp status +webgui_watchdog.sh --dry-run # show which services would be restarted +webgui_watchdog.sh --log # verbose per-step output +``` + +--- + +## stability_watchdog.sh Runs last in the orchestrator sequence. The only script in the ecosystem authorized to reboot. Watches the server itself — not containers, not storage. Reboots only when @@ -425,10 +483,10 @@ reboots → something crashes again immediately → reboot again. ### Usage ```bash -system_watchdog.sh # single pass (called by watchdog_orchestrator.sh) -system_watchdog.sh --dry-run # run detection logic without rebooting -system_watchdog.sh --status # thresholds, current state, strike counts -system_watchdog.sh --log # verbose per-check output +stability_watchdog.sh # single pass (called by watchdog_orchestrator.sh) +stability_watchdog.sh --dry-run # run detection logic without rebooting +stability_watchdog.sh --status # thresholds, current state, strike counts +stability_watchdog.sh --log # verbose per-check output ``` --- @@ -616,7 +674,7 @@ storage_watchdog.sh --status # ) ``` -### system_watchdog Rebooted Unexpectedly +### stability_watchdog Rebooted Unexpectedly ```bash # Check the reboot log (survives reboots): @@ -624,13 +682,13 @@ cat /boot/config/system_watchdog_reboots.db # Shows timestamp and reason for each watchdog-triggered reboot # Check syslog near the reboot time: -grep "system_watchdog" /var/log/syslog | tail -20 +grep "stability_watchdog" /var/log/syslog | tail -20 ``` -### system_watchdog Not Responding / Watchdog Orchestrator Reports Timeout +### stability_watchdog Not Responding / Watchdog Orchestrator Reports Timeout ```bash -# All four watchdogs run as single-pass scripts — there is no background process to check. +# All watchdogs run as single-pass scripts — there is no background process to check. # If the orchestrator reports a timeout, one pass took longer than expected. # Check the orchestrator itself: diff --git a/Watchdogs/README-Watchdogs.md b/Watchdogs/README-Watchdogs.md index ee0bc29..071d9a1 100644 --- a/Watchdogs/README-Watchdogs.md +++ b/Watchdogs/README-Watchdogs.md @@ -1,9 +1,10 @@ # ━━━━━ WATCHDOGS ━━━━━ -**Four single-pass scripts that run every minute through `watchdog_orchestrator.sh`, -each with a clear lane:** reduce system pressure → heal containers → protect storage → -reboot if nothing else worked. They never run standalone loops. The orchestrator calls -them in order, once per cron cycle. +**Four tiers that run every minute through `watchdog_orchestrator.sh`, +each with a clear lane:** reduce system pressure → heal containers → check system components → +reboot if nothing else worked. The orchestrator calls them in order, once per cron cycle. +System component checks (storage, WebGUI) run inside a thin `system_watchdog.sh` orchestrator +that can also be run standalone. --- @@ -52,7 +53,7 @@ Rootfs fills. SSH stops accepting new connections. Docker can't write log files. files fail silently. The server is functionally dead but still technically running. Nothing in the container layer can detect or recover from this — it requires a reboot. -Fix: `system_watchdog.sh` — watches the server itself: RAM, CPU, disk, kernel, daemon +Fix: `stability_watchdog.sh` — watches the server itself: RAM, CPU, disk, kernel, daemon health. Only script in the stack authorized to reboot. Runs last in the orchestrator sequence so container healing and pressure reduction always get a chance first. @@ -60,24 +61,26 @@ sequence so container healing and pressure reduction always get a chance first. ## ━━━ WHAT THIS FOLDER DOES ━━━ -Four watchdogs. One purpose each. Fixed execution order via `watchdog_orchestrator.sh`. +Four tiers. Fixed execution order via `watchdog_orchestrator.sh`. ``` Pressure reduction resource_watchdog.sh — throttle/pause/stop before healing fails Container healing docker_watchdog.sh — memory, CPU, HTTP, required containers -Storage protection storage_watchdog.sh — pool growth rate + runaway log detection -Last resort system_watchdog.sh — reboot only when nothing else can recover +System components system_watchdog.sh — thin orchestrator: storage + WebGUI health + └─ System/ storage_watchdog.sh — pool growth rate + runaway log detection + webgui_watchdog.sh — WebGUI availability, nginx → php-fpm → emhttp +Last resort stability_watchdog.sh — reboot only when nothing else can recover ``` **The execution order is the design.** Resource pressure is reduced before docker_watchdog attempts restarts — containers restarted into a pressure-bound system just fail again. -Storage is checked after containers are healed — no false alerts from containers that -were already being restarted. System watchdog runs last — reboot is always the last -option, not the first. +System component checks run after containers are healed. Stability watchdog runs last — +reboot is always the last option, not the first. **None of these scripts run standalone loops.** Each is a single-pass script called once per minute by `Orchestrators/watchdog_orchestrator.sh`. The orchestrator handles -startup grace, overlap protection, heartbeat, and sequencing. +startup grace, overlap protection, heartbeat, and sequencing. `system_watchdog.sh` can +also be run standalone to check all system component watchdogs at once. --- @@ -85,10 +88,14 @@ startup grace, overlap protection, heartbeat, and sequencing. ``` Orchestrators/ - watchdog_orchestrator.sh ──────────────────► resource_watchdog.sh (1st — every minute) - ──────────────────► docker_watchdog.sh (2nd) - ──────────────────► storage_watchdog.sh (3rd) - ──────────────────► system_watchdog.sh (4th — last resort) + watchdog_orchestrator.sh ──────────────────► resource_watchdog.sh (1st — every minute) + ──────────────────► docker_watchdog.sh (2nd) + ──────────────────► system_watchdog.sh (3rd — thin orchestrator) + ──────────────────► stability_watchdog.sh (4th — last resort) + +Watchdogs/System/ ← called by system_watchdog.sh each cycle + storage_watchdog.sh pool growth rate + runaway log detection + webgui_watchdog.sh WebGUI availability — nginx → php-fpm → emhttp escalation Tools/ watchdog_skip_list_manager.sh ◄────────────── docker_watchdog.sh writes skip list @@ -97,10 +104,6 @@ Tools/ Docker_Essentials/ All container lifecycle scripts (daily restart, updates, network) — unaffected. docker_watchdog.sh coordinates with them via shared state, not direct calls. - -unRAID_Essentials/ - Server-level scripts (WebGUI restart, inotify tuning, log hygiene) — unaffected. - system_watchdog.sh runs in the same ecosystem but is independent of those scripts. ``` **watchdog_orchestrator.sh stays in Orchestrators/** — it's a job runner, not a watchdog. @@ -114,8 +117,10 @@ unRAID_Essentials/ |--------|------|-----------| | `resource_watchdog.sh` | Three-level pressure reduction — throttle, pause, stop | `watchdog_orchestrator.sh` — 1st every minute | | `docker_watchdog.sh` | Two-tier container healing — memory, CPU, HTTP, required | `watchdog_orchestrator.sh` — 2nd every minute | -| `storage_watchdog.sh` | Pool growth rate + runaway log detection and remediation | `watchdog_orchestrator.sh` — 3rd every minute | -| `system_watchdog.sh` | Last-resort server watchdog — reboots when healing has failed | `watchdog_orchestrator.sh` — 4th every minute | +| `system_watchdog.sh` | Thin orchestrator — runs System/ component watchdogs in sequence | `watchdog_orchestrator.sh` — 3rd every minute | +| `stability_watchdog.sh` | Last-resort server watchdog — reboots when healing has failed | `watchdog_orchestrator.sh` — 4th every minute | +| `System/storage_watchdog.sh` | Pool growth rate + runaway log detection and remediation | `system_watchdog.sh` — every minute | +| `System/webgui_watchdog.sh` | WebGUI availability — nginx → php-fpm → emhttp escalation | `system_watchdog.sh` — every minute | > `watchdog_orchestrator.sh` is in `Orchestrators/`. `watchdog_skip_list_manager.sh` > is in `Tools/`. Neither is a watchdog — they sit at the edges of this system. @@ -149,13 +154,23 @@ Every minute — watchdog_orchestrator.sh fires: │ │ (skip list management) → Tools/watchdog_skip_list_manager.sh │ - Step 3 — storage_watchdog.sh - │ growth rate scan: du -sm appdata/* → compare to previous cycle baseline - │ growth > WATCHDOG_APPDATA_GROWTH_GB → 3-strike warn → alert - │ log file scan: find *.log > WATCHDOG_APPDATA_LOG_MAX_GB - │ oversize log found → 3-strike warn → truncate (if enabled) or alert + Step 3 — system_watchdog.sh (thin orchestrator) + │ runs SYSTEM_WATCHDOG_SCRIPTS from master.conf sequentially: │ - Step 4 — system_watchdog.sh + │ └─ storage_watchdog.sh + │ growth rate scan: du -sm appdata/* → compare to previous cycle baseline + │ growth > WATCHDOG_APPDATA_GROWTH_GB → 3-strike warn → alert + │ log file scan: find *.log > WATCHDOG_APPDATA_LOG_MAX_GB + │ oversize log found → 3-strike warn → truncate (if enabled) or alert + │ + │ └─ webgui_watchdog.sh + │ curl check → WebGUI responding → exit 0 (silent) + │ not responding → nginx restart → wait → recheck + │ still down → php-fpm restart → wait → recheck + │ still down → emhttp restart → wait → recheck + │ all three failed → critical notify, manual intervention needed + │ + Step 4 — stability_watchdog.sh checks the server itself — RAM, CPU temp, rootfs, FDs, kernel, daemon Tier 1 CRITICAL → immediate reboot (no strikes) Tier 2 URGENT → reboot if OOM confirmed @@ -169,8 +184,8 @@ Every minute — watchdog_orchestrator.sh fires: | State File | Written By | Read By | Purpose | |-----------|-----------|---------|---------| | `RW_STATE_FILE` | `resource_watchdog.sh` | `docker_watchdog.sh` | `mem_shutdown_active` flag — defer restarts during RAM emergency | -| `SYS_WATCHDOG_STATE_FILE` | `system_watchdog.sh` | `docker_watchdog.sh` | `watchdog_cycle` heartbeat — stale guard (2hr timeout) | +| `SYS_WATCHDOG_STATE_FILE` | `stability_watchdog.sh` | `docker_watchdog.sh` | `watchdog_cycle` heartbeat — stale guard (2hr timeout) | | `WATCHDOG_STATE_FILE` | `docker_watchdog.sh` | itself | CPU/HTTP strike counts per container | | `SYS_WATCHDOG_FAILED_FILE` | `docker_watchdog.sh` | `watchdog_skip_list_manager.sh` | Container skip list | -| `STORAGE_WATCHDOG_STATE_FILE` | `storage_watchdog.sh` | itself | Growth + log strike counts | -| `WATCHDOG_APPDATA_GROWTH_FILE` | `storage_watchdog.sh` | itself | Per-container size baseline for growth rate | +| `STORAGE_WATCHDOG_STATE_FILE` | `System/storage_watchdog.sh` | itself | Growth + log strike counts | +| `WATCHDOG_APPDATA_GROWTH_FILE` | `System/storage_watchdog.sh` | itself | Per-container size baseline for growth rate | diff --git a/unRAID_Essentials/Manual-Unraid_Essentials.md b/unRAID_Essentials/Manual-Unraid_Essentials.md index 5367ab7..2d3132d 100644 --- a/unRAID_Essentials/Manual-Unraid_Essentials.md +++ b/unRAID_Essentials/Manual-Unraid_Essentials.md @@ -4,7 +4,7 @@ Configuration reference, operational procedures, and troubleshooting for system-level scripts. Read the ARRAY_START_SCRIPTS order section before adding or reordering scripts at array start. -> **Watchdog scripts have moved.** `system_watchdog.sh` and `resource_watchdog.sh` +> **Watchdog scripts have moved.** `stability_watchdog.sh` and `resource_watchdog.sh` > now live in `Watchdogs/`. Their configuration reference and troubleshooting > procedures are in `Watchdogs/Manual-Watchdogs.md`. @@ -452,8 +452,8 @@ server_reboot.sh --reason="disk work" # include reason in notification ## Full Configuration Reference -> Watchdog configuration (`system_watchdog.sh`, `resource_watchdog.sh`, -> `docker_watchdog.sh`, `storage_watchdog.sh`) lives in `Watchdogs/Manual-Watchdogs.md`. +> Watchdog configuration (`stability_watchdog.sh`, `resource_watchdog.sh`, +> `docker_watchdog.sh`, `System/storage_watchdog.sh`) lives in `Watchdogs/Manual-Watchdogs.md`. ```bash # master.conf @@ -494,8 +494,8 @@ REBOOT_VM_WAIT=30 ## Troubleshooting -> Watchdog troubleshooting (system_watchdog, resource_watchdog, docker_watchdog, -> storage_watchdog) is in `Watchdogs/Manual-Watchdogs.md`. +> Watchdog troubleshooting (stability_watchdog, resource_watchdog, docker_watchdog, +> System/System/storage_watchdog) is in `Watchdogs/Manual-Watchdogs.md`. ### rsync_stop Killed the Wrong Thing diff --git a/unRAID_Essentials/README-Unraid_Essentials.md b/unRAID_Essentials/README-Unraid_Essentials.md index 7603b59..591215e 100644 --- a/unRAID_Essentials/README-Unraid_Essentials.md +++ b/unRAID_Essentials/README-Unraid_Essentials.md @@ -16,7 +16,7 @@ The restart hangs. The server needs a reboot, but nothing in the ecosystem is authorized to call one. Or: rootfs fills to 99%. SSH stops working. Docker can't write log files. The server is functionally dead but still technically running. -Fix: `system_watchdog.sh` — three-tier response: immediate reboot on critical +Fix: `stability_watchdog.sh` — three-tier response: immediate reboot on critical failures, OOM-confirmed bypass for RAM crises, strike system for sustained threshold breaches. Last line of defense before a hard crash. @@ -82,7 +82,7 @@ Graceful operations mover_stop.sh — clean mover stop server_reboot.sh — clean reboot with pre-flight warnings ``` -> `system_watchdog.sh` and `resource_watchdog.sh` have moved to `Watchdogs/`. +> `stability_watchdog.sh` and `resource_watchdog.sh` have moved to `Watchdogs/`. > See `Watchdogs/README-Watchdogs.md` for the full watchdog suite. --- @@ -99,9 +99,9 @@ Orchestrators/ server_reboot.sh ────────────────────────► user_scripts_stop.sh (called internally) Watchdogs/ - system_watchdog.sh and resource_watchdog.sh now live here. + stability_watchdog.sh and resource_watchdog.sh now live here. See Watchdogs/README-Watchdogs.md for how they relate to each other - and to docker_watchdog.sh and storage_watchdog.sh. + and to docker_watchdog.sh and System/storage_watchdog.sh. ``` ---