Update all docs to reflect watchdog restructure
README-Watchdogs.md: new tier diagram, updated script table, execution flow, state file table, and folder relationship diagram Manual-Watchdogs.md: added system_watchdog (thin), System/storage_watchdog, System/webgui_watchdog, and stability_watchdog sections; updated TOC and troubleshooting README.md: updated schedule listing, folder description, daily cycle examples README-User_Script_Plug-in.md: stability_watchdog references throughout unRAID_Essentials/README + Manual: stability_watchdog, updated storage path Monitors/README, Rsync/Manual, Tools/Manual: lock name list updates
This commit is contained in:
Gmer4Lfe
@@ -82,7 +82,7 @@ drive SMART attributes, ZFS pool state, ARC statistics, kernel memory pressure.
|
||||
```
|
||||
Monitors/ ← observes and reports (this folder)
|
||||
Docker_Essentials/ ← acts on containers (docker_watchdog starts/stops)
|
||||
unRAID_Essentials/ ← acts on the server (system_watchdog, inotify_tuning)
|
||||
unRAID_Essentials/ ← acts on the server (stability_watchdog, inotify_tuning)
|
||||
Fallback/ ← acts on the full stack (failover, handback)
|
||||
Rsync/ ← calls bandwidth_monitor.sh (auto-logs each sync)
|
||||
```
|
||||
|
||||
@@ -111,14 +111,14 @@ docker_syslog_filter.sh — suppresses Docker veth noise before containers crea
|
||||
php_fpm_max_children.sh — sets PHP-FPM workers to 250 before WebGUI serves requests
|
||||
ramdisk_setup.sh — creates /mnt/ramdisk_transcodes and symlink before Emby starts
|
||||
docker_network_connect.sh— ensures custom networks exist + containers connected
|
||||
system_watchdog.sh ← continuous — three-tier server stability watchdog
|
||||
stability_watchdog.sh ← last-resort — three-tier server stability watchdog
|
||||
docker_watchdog.sh ← continuous — two-tier container healing watchdog
|
||||
failover.sh ← continuous — mutual failover state machine
|
||||
```
|
||||
|
||||
**After running:**
|
||||
One-shot scripts exit cleanly. Three continuous processes run in the background until
|
||||
the array stops. Check with: `pgrep -f "system_watchdog\|docker_watchdog\|failover"`
|
||||
the array stops. Check with: `pgrep -f "stability_watchdog\|docker_watchdog\|failover"`
|
||||
|
||||
---
|
||||
|
||||
@@ -899,7 +899,7 @@ Background: NO
|
||||
**What it shows:**
|
||||
Strike counts for all monitored containers. Current skip list with running/stopped state.
|
||||
Whether startup grace period is active and how long remains. Whether RAM emergency
|
||||
deferral is active (mem_shutdown_active from system_watchdog). Last cycle timing.
|
||||
deferral is active (mem_shutdown_active from stability_watchdog). Last cycle timing.
|
||||
|
||||
---
|
||||
|
||||
@@ -1200,7 +1200,7 @@ flip count, session split ramdisk vs SSD).
|
||||
## 🖥️ UNRAID_ESSENTIALS
|
||||
## ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
|
||||
|
||||
> **system_watchdog.sh is started by array_start.sh — not scheduled separately.**
|
||||
> **stability_watchdog.sh is called by watchdog_orchestrator.sh — not scheduled separately.**
|
||||
> **webgui_watchdog.sh, system_tuning_monitor.sh, clear_logs.sh are scheduled below.**
|
||||
> **inotify_tuning.sh, php_fpm_max_children.sh, docker_syslog_filter.sh run via array_start.sh.**
|
||||
|
||||
@@ -1264,11 +1264,11 @@ Active containers like Emby and SABnzbd grow fastest. Truncation not rotation
|
||||
|
||||
---
|
||||
|
||||
### ── system_watchdog_status ──────────────────────────────────────────────────
|
||||
### ── stability_watchdog_status ──────────────────────────────────────────────────
|
||||
|
||||
```
|
||||
Category: unRAID_Essentials
|
||||
Name: system_watchdog_status
|
||||
Name: stability_watchdog_status
|
||||
Schedule: None — run manually
|
||||
Background: NO
|
||||
```
|
||||
@@ -1632,7 +1632,7 @@ Correct startup setup:
|
||||
Wrong startup setup (old approach):
|
||||
ramdisk_setup — At Startup of Array
|
||||
docker_syslog_filter — At Startup of Array
|
||||
system_watchdog — At Startup of Array
|
||||
stability_watchdog — Every Minute via watchdog_orchestrator
|
||||
docker_watchdog — At Startup of Array
|
||||
failover — At Startup of Array
|
||||
(unpredictable order, no coordination)
|
||||
@@ -1707,5 +1707,5 @@ critical_sync_maintenance verify Critical-Data sync completes to HOST2
|
||||
daily_sync_maintenance --dry-run verify remote connectivity and share list
|
||||
failover_status confirm NORMAL state on both servers
|
||||
docker_watchdog_status confirm required containers all running
|
||||
system_watchdog_status confirm no threshold alerts on first check
|
||||
stability_watchdog_status confirm no threshold alerts on first check
|
||||
```
|
||||
@@ -448,8 +448,8 @@ Failover coverage:
|
||||
serves Gmer4Lfe.us via DDNS
|
||||
|
||||
Monitoring:
|
||||
Both servers resource/docker/storage/ resource/docker/storage/
|
||||
system_watchdog.sh system_watchdog.sh
|
||||
Both servers resource/docker/system/ resource/docker/system/
|
||||
stability_watchdog.sh stability_watchdog.sh
|
||||
failover.sh failover.sh
|
||||
Sunday morning coffee report Sunday morning coffee report
|
||||
```
|
||||
@@ -476,7 +476,7 @@ Unraid_Scripts/
|
||||
├── Failover/ ← Mutual automatic failover — continuous background process
|
||||
│ README: README-Failover.md
|
||||
│
|
||||
├── Watchdogs/ ← All watchdog scripts: resource, docker, storage, system
|
||||
├── Watchdogs/ ← All watchdog scripts: resource, docker, system, stability
|
||||
│ README: README-Watchdogs.md
|
||||
│
|
||||
├── Docker_Essentials/ ← Container lifecycle: restarts, updates, networks
|
||||
@@ -548,8 +548,8 @@ Every minute:
|
||||
watchdog_orchestrator.sh fires each watchdog in sequence
|
||||
→ resource_watchdog.sh reduce pressure before healing attempts
|
||||
→ docker_watchdog.sh two-tier container healing
|
||||
→ storage_watchdog.sh pool growth + runaway log detection
|
||||
→ system_watchdog.sh last resort — reboots when all else fails
|
||||
→ system_watchdog.sh system components: storage + WebGUI health
|
||||
→ stability_watchdog.sh last resort — reboots when all else fails
|
||||
|
||||
Every 3 minutes:
|
||||
transcode_management.sh cleanup → manager (order non-negotiable)
|
||||
@@ -593,8 +593,8 @@ What actually happens on a typical day, from the ecosystem's perspective:
|
||||
Throughout the day (every minute via watchdog_orchestrator.sh):
|
||||
resource_watchdog.sh managing: system pressure (throttle/pause/stop)
|
||||
docker_watchdog.sh healing: memory leaks, HTTP failures, required containers
|
||||
storage_watchdog.sh watching: appdata growth rate, runaway log files
|
||||
system_watchdog.sh watching: RAM, CPU temp, rootfs, kernel, daemon
|
||||
system_watchdog.sh watching: system components (storage growth, WebGUI)
|
||||
stability_watchdog.sh watching: RAM, CPU temp, rootfs, kernel, daemon
|
||||
|
||||
Throughout the day:
|
||||
failover.sh watching: remote server, internet connectivity
|
||||
@@ -639,13 +639,13 @@ Problem: Authelia crashed and won't restart
|
||||
→ Tools: watchdog_skip_list_manager.sh to clear after fixing
|
||||
|
||||
Problem: RAM drops to 3GB
|
||||
→ system_watchdog.sh: stop non-essential containers
|
||||
→ stability_watchdog.sh: stop non-essential containers
|
||||
→ Wait for RAM to recover to 30GB
|
||||
→ docker_watchdog.sh defers (reads mem_shutdown_active=true)
|
||||
→ If RAM drops to 4GB AND 3+ OOM kills: bypass strikes → reboot
|
||||
|
||||
Problem: Docker daemon hung
|
||||
→ system_watchdog.sh Tier 1: attempt /etc/rc.d/rc.docker restart
|
||||
→ stability_watchdog.sh Tier 1: attempt /etc/rc.d/rc.docker restart
|
||||
→ If still hung: immediate reboot (no strikes needed — daemon can't be managed)
|
||||
|
||||
Problem: HOST1 loses power
|
||||
@@ -713,7 +713,7 @@ Things that are different from what you might expect:
|
||||
# scheduled directly. array_start.sh is the only "At Startup of Array" entry.
|
||||
|
||||
# 2. Scripts are silent when healthy.
|
||||
# system_watchdog.sh runs 288 times per day. It produces zero output on clean
|
||||
# stability_watchdog.sh runs 288 times per day. It produces zero output on clean
|
||||
# cycles. If you see output, something needs attention. If you see nothing,
|
||||
# the system is healthy. This is by design.
|
||||
|
||||
|
||||
@@ -439,7 +439,7 @@ bash /mnt/user/appdata/unraid_scripts/Orchestrators/array_start.sh
|
||||
|
||||
This single entry launches everything defined in ARRAY_START_SCRIPTS from master.conf:
|
||||
inotify_tuning, docker_syslog_filter, php_fpm_max_children, ramdisk_setup,
|
||||
docker_network_connect, system_watchdog, docker_watchdog, failover.
|
||||
docker_network_connect, stability_watchdog, docker_watchdog, failover.
|
||||
|
||||
### Cron Schedule
|
||||
|
||||
|
||||
@@ -562,7 +562,7 @@ changes to any running process, container, or state file.
|
||||
### What It Shows
|
||||
|
||||
```
|
||||
system_watchdog
|
||||
stability_watchdog
|
||||
Running state, PID, uptime, approximate cycle count
|
||||
Active strikes, recent restart history
|
||||
Live snapshot: rootfs, RAM, ZFS ARC, load, zombie count, CPU temp
|
||||
|
||||
@@ -1,7 +1,7 @@
|
||||
# ━━━━━ WATCHDOGS — Manual ━━━━━
|
||||
|
||||
Configuration reference, operational procedures, and troubleshooting for all four
|
||||
watchdog scripts. For design philosophy and script relationships see `README-Watchdogs.md`.
|
||||
Configuration reference, operational procedures, and troubleshooting for all watchdog
|
||||
scripts. For design philosophy and script relationships see `README-Watchdogs.md`.
|
||||
For the orchestrator that calls these scripts see `Orchestrators/watchdog_orchestrator.sh`.
|
||||
|
||||
---
|
||||
@@ -11,8 +11,10 @@ For the orchestrator that calls these scripts see `Orchestrators/watchdog_orches
|
||||
- [Output Tiers](#output-tiers)
|
||||
- [resource_watchdog.sh](#resource_watchdogsh)
|
||||
- [docker_watchdog.sh](#docker_watchdogsh)
|
||||
- [storage_watchdog.sh](#storage_watchdogsh)
|
||||
- [system_watchdog.sh](#system_watchdogsh)
|
||||
- [System/storage_watchdog.sh](#systemstorage_watchdogsh)
|
||||
- [System/webgui_watchdog.sh](#systemwebgui_watchdogsh)
|
||||
- [stability_watchdog.sh](#stability_watchdogsh)
|
||||
- [Full Configuration Reference](#full-configuration-reference)
|
||||
- [Troubleshooting](#troubleshooting)
|
||||
|
||||
@@ -23,7 +25,7 @@ For the orchestrator that calls these scripts see `Orchestrators/watchdog_orches
|
||||
All watchdog scripts use a two-tier output model: `echo` lines are always visible;
|
||||
`log` lines only appear when `--log` is passed.
|
||||
|
||||
All four watchdogs are **single-pass scripts** called once per minute by the orchestrator.
|
||||
All watchdog scripts are **single-pass scripts** called once per minute by the orchestrator.
|
||||
Without `--log`, only state transitions, warnings, errors, and the conclusion line are
|
||||
visible. Per-check detail is suppressed on clean cycles.
|
||||
|
||||
@@ -285,9 +287,26 @@ docker start ContainerName
|
||||
|
||||
---
|
||||
|
||||
## storage_watchdog.sh
|
||||
## system_watchdog.sh
|
||||
|
||||
Runs third in the orchestrator sequence. Two independent checks per cycle:
|
||||
Thin orchestrator — runs `SYSTEM_WATCHDOG_SCRIPTS` from master.conf sequentially each cycle.
|
||||
Called third by `watchdog_orchestrator.sh`. Covers all system component watchdogs.
|
||||
Can also be run standalone to check all system components at once.
|
||||
|
||||
### Usage
|
||||
|
||||
```bash
|
||||
system_watchdog.sh # run all system component watchdogs
|
||||
system_watchdog.sh --status # show configured scripts and their paths
|
||||
system_watchdog.sh --dry-run # preview without executing anything
|
||||
system_watchdog.sh --log # verbose output
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## System/storage_watchdog.sh
|
||||
|
||||
Called by `system_watchdog.sh` each cycle. Two independent checks per cycle:
|
||||
growth rate detection (automatic, zero config) and oversize log detection.
|
||||
Uses its own strike state file — independent from docker_watchdog.
|
||||
|
||||
@@ -343,7 +362,7 @@ Log strikes auto-clear when the file drops below threshold.
|
||||
### Usage
|
||||
|
||||
```bash
|
||||
storage_watchdog.sh # single pass (called by watchdog_orchestrator.sh)
|
||||
storage_watchdog.sh # single pass (called by system_watchdog.sh)
|
||||
storage_watchdog.sh --status # strikes, growth baseline age, suppress ceilings
|
||||
storage_watchdog.sh --dry-run # show what would be alerted/truncated
|
||||
storage_watchdog.sh --log # verbose per-container output
|
||||
@@ -351,7 +370,46 @@ storage_watchdog.sh --log # verbose per-container output
|
||||
|
||||
---
|
||||
|
||||
## system_watchdog.sh
|
||||
## System/webgui_watchdog.sh
|
||||
|
||||
Called by `system_watchdog.sh` each cycle. Monitors WebGUI availability and escalates
|
||||
through three restart steps if unresponsive. Silent when healthy.
|
||||
|
||||
### Escalation Path
|
||||
|
||||
```
|
||||
WebGUI responding → exit 0 (silent)
|
||||
|
||||
Not responding:
|
||||
Step 1 — nginx restart → wait WEBGUI_NGINX_WAIT → recheck
|
||||
Step 2 — php-fpm restart → wait WEBGUI_PHP_WAIT → recheck
|
||||
Step 3 — emhttp restart → wait WEBGUI_EMHTTP_WAIT → recheck
|
||||
All three failed → critical notify, manual intervention needed
|
||||
```
|
||||
|
||||
### Configuration
|
||||
|
||||
```bash
|
||||
# master.conf
|
||||
WEBGUI_URL="http://localhost"
|
||||
WEBGUI_TIMEOUT=5
|
||||
WEBGUI_NGINX_WAIT=15
|
||||
WEBGUI_PHP_WAIT=10
|
||||
WEBGUI_EMHTTP_WAIT=30
|
||||
```
|
||||
|
||||
### Usage
|
||||
|
||||
```bash
|
||||
webgui_watchdog.sh # single pass (called by system_watchdog.sh)
|
||||
webgui_watchdog.sh --status # current WebGUI state + nginx/php-fpm/emhttp status
|
||||
webgui_watchdog.sh --dry-run # show which services would be restarted
|
||||
webgui_watchdog.sh --log # verbose per-step output
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## stability_watchdog.sh
|
||||
|
||||
Runs last in the orchestrator sequence. The only script in the ecosystem authorized
|
||||
to reboot. Watches the server itself — not containers, not storage. Reboots only when
|
||||
@@ -425,10 +483,10 @@ reboots → something crashes again immediately → reboot again.
|
||||
### Usage
|
||||
|
||||
```bash
|
||||
system_watchdog.sh # single pass (called by watchdog_orchestrator.sh)
|
||||
system_watchdog.sh --dry-run # run detection logic without rebooting
|
||||
system_watchdog.sh --status # thresholds, current state, strike counts
|
||||
system_watchdog.sh --log # verbose per-check output
|
||||
stability_watchdog.sh # single pass (called by watchdog_orchestrator.sh)
|
||||
stability_watchdog.sh --dry-run # run detection logic without rebooting
|
||||
stability_watchdog.sh --status # thresholds, current state, strike counts
|
||||
stability_watchdog.sh --log # verbose per-check output
|
||||
```
|
||||
|
||||
---
|
||||
@@ -616,7 +674,7 @@ storage_watchdog.sh --status
|
||||
# )
|
||||
```
|
||||
|
||||
### system_watchdog Rebooted Unexpectedly
|
||||
### stability_watchdog Rebooted Unexpectedly
|
||||
|
||||
```bash
|
||||
# Check the reboot log (survives reboots):
|
||||
@@ -624,13 +682,13 @@ cat /boot/config/system_watchdog_reboots.db
|
||||
# Shows timestamp and reason for each watchdog-triggered reboot
|
||||
|
||||
# Check syslog near the reboot time:
|
||||
grep "system_watchdog" /var/log/syslog | tail -20
|
||||
grep "stability_watchdog" /var/log/syslog | tail -20
|
||||
```
|
||||
|
||||
### system_watchdog Not Responding / Watchdog Orchestrator Reports Timeout
|
||||
### stability_watchdog Not Responding / Watchdog Orchestrator Reports Timeout
|
||||
|
||||
```bash
|
||||
# All four watchdogs run as single-pass scripts — there is no background process to check.
|
||||
# All watchdogs run as single-pass scripts — there is no background process to check.
|
||||
# If the orchestrator reports a timeout, one pass took longer than expected.
|
||||
|
||||
# Check the orchestrator itself:
|
||||
|
||||
@@ -1,9 +1,10 @@
|
||||
# ━━━━━ WATCHDOGS ━━━━━
|
||||
|
||||
**Four single-pass scripts that run every minute through `watchdog_orchestrator.sh`,
|
||||
each with a clear lane:** reduce system pressure → heal containers → protect storage →
|
||||
reboot if nothing else worked. They never run standalone loops. The orchestrator calls
|
||||
them in order, once per cron cycle.
|
||||
**Four tiers that run every minute through `watchdog_orchestrator.sh`,
|
||||
each with a clear lane:** reduce system pressure → heal containers → check system components →
|
||||
reboot if nothing else worked. The orchestrator calls them in order, once per cron cycle.
|
||||
System component checks (storage, WebGUI) run inside a thin `system_watchdog.sh` orchestrator
|
||||
that can also be run standalone.
|
||||
|
||||
---
|
||||
|
||||
@@ -52,7 +53,7 @@ Rootfs fills. SSH stops accepting new connections. Docker can't write log files.
|
||||
files fail silently. The server is functionally dead but still technically running.
|
||||
Nothing in the container layer can detect or recover from this — it requires a reboot.
|
||||
|
||||
Fix: `system_watchdog.sh` — watches the server itself: RAM, CPU, disk, kernel, daemon
|
||||
Fix: `stability_watchdog.sh` — watches the server itself: RAM, CPU, disk, kernel, daemon
|
||||
health. Only script in the stack authorized to reboot. Runs last in the orchestrator
|
||||
sequence so container healing and pressure reduction always get a chance first.
|
||||
|
||||
@@ -60,24 +61,26 @@ sequence so container healing and pressure reduction always get a chance first.
|
||||
|
||||
## ━━━ WHAT THIS FOLDER DOES ━━━
|
||||
|
||||
Four watchdogs. One purpose each. Fixed execution order via `watchdog_orchestrator.sh`.
|
||||
Four tiers. Fixed execution order via `watchdog_orchestrator.sh`.
|
||||
|
||||
```
|
||||
Pressure reduction resource_watchdog.sh — throttle/pause/stop before healing fails
|
||||
Container healing docker_watchdog.sh — memory, CPU, HTTP, required containers
|
||||
Storage protection storage_watchdog.sh — pool growth rate + runaway log detection
|
||||
Last resort system_watchdog.sh — reboot only when nothing else can recover
|
||||
System components system_watchdog.sh — thin orchestrator: storage + WebGUI health
|
||||
└─ System/ storage_watchdog.sh — pool growth rate + runaway log detection
|
||||
webgui_watchdog.sh — WebGUI availability, nginx → php-fpm → emhttp
|
||||
Last resort stability_watchdog.sh — reboot only when nothing else can recover
|
||||
```
|
||||
|
||||
**The execution order is the design.** Resource pressure is reduced before docker_watchdog
|
||||
attempts restarts — containers restarted into a pressure-bound system just fail again.
|
||||
Storage is checked after containers are healed — no false alerts from containers that
|
||||
were already being restarted. System watchdog runs last — reboot is always the last
|
||||
option, not the first.
|
||||
System component checks run after containers are healed. Stability watchdog runs last —
|
||||
reboot is always the last option, not the first.
|
||||
|
||||
**None of these scripts run standalone loops.** Each is a single-pass script called
|
||||
once per minute by `Orchestrators/watchdog_orchestrator.sh`. The orchestrator handles
|
||||
startup grace, overlap protection, heartbeat, and sequencing.
|
||||
startup grace, overlap protection, heartbeat, and sequencing. `system_watchdog.sh` can
|
||||
also be run standalone to check all system component watchdogs at once.
|
||||
|
||||
---
|
||||
|
||||
@@ -85,10 +88,14 @@ startup grace, overlap protection, heartbeat, and sequencing.
|
||||
|
||||
```
|
||||
Orchestrators/
|
||||
watchdog_orchestrator.sh ──────────────────► resource_watchdog.sh (1st — every minute)
|
||||
──────────────────► docker_watchdog.sh (2nd)
|
||||
──────────────────► storage_watchdog.sh (3rd)
|
||||
──────────────────► system_watchdog.sh (4th — last resort)
|
||||
watchdog_orchestrator.sh ──────────────────► resource_watchdog.sh (1st — every minute)
|
||||
──────────────────► docker_watchdog.sh (2nd)
|
||||
──────────────────► system_watchdog.sh (3rd — thin orchestrator)
|
||||
──────────────────► stability_watchdog.sh (4th — last resort)
|
||||
|
||||
Watchdogs/System/ ← called by system_watchdog.sh each cycle
|
||||
storage_watchdog.sh pool growth rate + runaway log detection
|
||||
webgui_watchdog.sh WebGUI availability — nginx → php-fpm → emhttp escalation
|
||||
|
||||
Tools/
|
||||
watchdog_skip_list_manager.sh ◄────────────── docker_watchdog.sh writes skip list
|
||||
@@ -97,10 +104,6 @@ Tools/
|
||||
Docker_Essentials/
|
||||
All container lifecycle scripts (daily restart, updates, network) — unaffected.
|
||||
docker_watchdog.sh coordinates with them via shared state, not direct calls.
|
||||
|
||||
unRAID_Essentials/
|
||||
Server-level scripts (WebGUI restart, inotify tuning, log hygiene) — unaffected.
|
||||
system_watchdog.sh runs in the same ecosystem but is independent of those scripts.
|
||||
```
|
||||
|
||||
**watchdog_orchestrator.sh stays in Orchestrators/** — it's a job runner, not a watchdog.
|
||||
@@ -114,8 +117,10 @@ unRAID_Essentials/
|
||||
|--------|------|-----------|
|
||||
| `resource_watchdog.sh` | Three-level pressure reduction — throttle, pause, stop | `watchdog_orchestrator.sh` — 1st every minute |
|
||||
| `docker_watchdog.sh` | Two-tier container healing — memory, CPU, HTTP, required | `watchdog_orchestrator.sh` — 2nd every minute |
|
||||
| `storage_watchdog.sh` | Pool growth rate + runaway log detection and remediation | `watchdog_orchestrator.sh` — 3rd every minute |
|
||||
| `system_watchdog.sh` | Last-resort server watchdog — reboots when healing has failed | `watchdog_orchestrator.sh` — 4th every minute |
|
||||
| `system_watchdog.sh` | Thin orchestrator — runs System/ component watchdogs in sequence | `watchdog_orchestrator.sh` — 3rd every minute |
|
||||
| `stability_watchdog.sh` | Last-resort server watchdog — reboots when healing has failed | `watchdog_orchestrator.sh` — 4th every minute |
|
||||
| `System/storage_watchdog.sh` | Pool growth rate + runaway log detection and remediation | `system_watchdog.sh` — every minute |
|
||||
| `System/webgui_watchdog.sh` | WebGUI availability — nginx → php-fpm → emhttp escalation | `system_watchdog.sh` — every minute |
|
||||
|
||||
> `watchdog_orchestrator.sh` is in `Orchestrators/`. `watchdog_skip_list_manager.sh`
|
||||
> is in `Tools/`. Neither is a watchdog — they sit at the edges of this system.
|
||||
@@ -149,13 +154,23 @@ Every minute — watchdog_orchestrator.sh fires:
|
||||
│
|
||||
│ (skip list management) → Tools/watchdog_skip_list_manager.sh
|
||||
│
|
||||
Step 3 — storage_watchdog.sh
|
||||
│ growth rate scan: du -sm appdata/* → compare to previous cycle baseline
|
||||
│ growth > WATCHDOG_APPDATA_GROWTH_GB → 3-strike warn → alert
|
||||
│ log file scan: find *.log > WATCHDOG_APPDATA_LOG_MAX_GB
|
||||
│ oversize log found → 3-strike warn → truncate (if enabled) or alert
|
||||
Step 3 — system_watchdog.sh (thin orchestrator)
|
||||
│ runs SYSTEM_WATCHDOG_SCRIPTS from master.conf sequentially:
|
||||
│
|
||||
Step 4 — system_watchdog.sh
|
||||
│ └─ storage_watchdog.sh
|
||||
│ growth rate scan: du -sm appdata/* → compare to previous cycle baseline
|
||||
│ growth > WATCHDOG_APPDATA_GROWTH_GB → 3-strike warn → alert
|
||||
│ log file scan: find *.log > WATCHDOG_APPDATA_LOG_MAX_GB
|
||||
│ oversize log found → 3-strike warn → truncate (if enabled) or alert
|
||||
│
|
||||
│ └─ webgui_watchdog.sh
|
||||
│ curl check → WebGUI responding → exit 0 (silent)
|
||||
│ not responding → nginx restart → wait → recheck
|
||||
│ still down → php-fpm restart → wait → recheck
|
||||
│ still down → emhttp restart → wait → recheck
|
||||
│ all three failed → critical notify, manual intervention needed
|
||||
│
|
||||
Step 4 — stability_watchdog.sh
|
||||
checks the server itself — RAM, CPU temp, rootfs, FDs, kernel, daemon
|
||||
Tier 1 CRITICAL → immediate reboot (no strikes)
|
||||
Tier 2 URGENT → reboot if OOM confirmed
|
||||
@@ -169,8 +184,8 @@ Every minute — watchdog_orchestrator.sh fires:
|
||||
| State File | Written By | Read By | Purpose |
|
||||
|-----------|-----------|---------|---------|
|
||||
| `RW_STATE_FILE` | `resource_watchdog.sh` | `docker_watchdog.sh` | `mem_shutdown_active` flag — defer restarts during RAM emergency |
|
||||
| `SYS_WATCHDOG_STATE_FILE` | `system_watchdog.sh` | `docker_watchdog.sh` | `watchdog_cycle` heartbeat — stale guard (2hr timeout) |
|
||||
| `SYS_WATCHDOG_STATE_FILE` | `stability_watchdog.sh` | `docker_watchdog.sh` | `watchdog_cycle` heartbeat — stale guard (2hr timeout) |
|
||||
| `WATCHDOG_STATE_FILE` | `docker_watchdog.sh` | itself | CPU/HTTP strike counts per container |
|
||||
| `SYS_WATCHDOG_FAILED_FILE` | `docker_watchdog.sh` | `watchdog_skip_list_manager.sh` | Container skip list |
|
||||
| `STORAGE_WATCHDOG_STATE_FILE` | `storage_watchdog.sh` | itself | Growth + log strike counts |
|
||||
| `WATCHDOG_APPDATA_GROWTH_FILE` | `storage_watchdog.sh` | itself | Per-container size baseline for growth rate |
|
||||
| `STORAGE_WATCHDOG_STATE_FILE` | `System/storage_watchdog.sh` | itself | Growth + log strike counts |
|
||||
| `WATCHDOG_APPDATA_GROWTH_FILE` | `System/storage_watchdog.sh` | itself | Per-container size baseline for growth rate |
|
||||
|
||||
@@ -4,7 +4,7 @@ Configuration reference, operational procedures, and troubleshooting for
|
||||
system-level scripts. Read the ARRAY_START_SCRIPTS order section before
|
||||
adding or reordering scripts at array start.
|
||||
|
||||
> **Watchdog scripts have moved.** `system_watchdog.sh` and `resource_watchdog.sh`
|
||||
> **Watchdog scripts have moved.** `stability_watchdog.sh` and `resource_watchdog.sh`
|
||||
> now live in `Watchdogs/`. Their configuration reference and troubleshooting
|
||||
> procedures are in `Watchdogs/Manual-Watchdogs.md`.
|
||||
|
||||
@@ -452,8 +452,8 @@ server_reboot.sh --reason="disk work" # include reason in notification
|
||||
|
||||
## Full Configuration Reference
|
||||
|
||||
> Watchdog configuration (`system_watchdog.sh`, `resource_watchdog.sh`,
|
||||
> `docker_watchdog.sh`, `storage_watchdog.sh`) lives in `Watchdogs/Manual-Watchdogs.md`.
|
||||
> Watchdog configuration (`stability_watchdog.sh`, `resource_watchdog.sh`,
|
||||
> `docker_watchdog.sh`, `System/storage_watchdog.sh`) lives in `Watchdogs/Manual-Watchdogs.md`.
|
||||
|
||||
```bash
|
||||
# master.conf
|
||||
@@ -494,8 +494,8 @@ REBOOT_VM_WAIT=30
|
||||
|
||||
## Troubleshooting
|
||||
|
||||
> Watchdog troubleshooting (system_watchdog, resource_watchdog, docker_watchdog,
|
||||
> storage_watchdog) is in `Watchdogs/Manual-Watchdogs.md`.
|
||||
> Watchdog troubleshooting (stability_watchdog, resource_watchdog, docker_watchdog,
|
||||
> System/System/storage_watchdog) is in `Watchdogs/Manual-Watchdogs.md`.
|
||||
|
||||
### rsync_stop Killed the Wrong Thing
|
||||
|
||||
|
||||
@@ -16,7 +16,7 @@ The restart hangs. The server needs a reboot, but nothing in the ecosystem is
|
||||
authorized to call one. Or: rootfs fills to 99%. SSH stops working. Docker can't
|
||||
write log files. The server is functionally dead but still technically running.
|
||||
|
||||
Fix: `system_watchdog.sh` — three-tier response: immediate reboot on critical
|
||||
Fix: `stability_watchdog.sh` — three-tier response: immediate reboot on critical
|
||||
failures, OOM-confirmed bypass for RAM crises, strike system for sustained
|
||||
threshold breaches. Last line of defense before a hard crash.
|
||||
|
||||
@@ -82,7 +82,7 @@ Graceful operations mover_stop.sh — clean mover stop
|
||||
server_reboot.sh — clean reboot with pre-flight warnings
|
||||
```
|
||||
|
||||
> `system_watchdog.sh` and `resource_watchdog.sh` have moved to `Watchdogs/`.
|
||||
> `stability_watchdog.sh` and `resource_watchdog.sh` have moved to `Watchdogs/`.
|
||||
> See `Watchdogs/README-Watchdogs.md` for the full watchdog suite.
|
||||
|
||||
---
|
||||
@@ -99,9 +99,9 @@ Orchestrators/
|
||||
server_reboot.sh ────────────────────────► user_scripts_stop.sh (called internally)
|
||||
|
||||
Watchdogs/
|
||||
system_watchdog.sh and resource_watchdog.sh now live here.
|
||||
stability_watchdog.sh and resource_watchdog.sh now live here.
|
||||
See Watchdogs/README-Watchdogs.md for how they relate to each other
|
||||
and to docker_watchdog.sh and storage_watchdog.sh.
|
||||
and to docker_watchdog.sh and System/storage_watchdog.sh.
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
Reference in New Issue
Block a user