diff --git a/Watchdogs/Manual-Watchdogs.md b/Watchdogs/Manual-Watchdogs.md index 232efdf..cc0598d 100644 --- a/Watchdogs/Manual-Watchdogs.md +++ b/Watchdogs/Manual-Watchdogs.md @@ -524,14 +524,14 @@ healing at every other layer has failed or when the failure is non-recoverable. | /boot read-only | write test fails | Config writes silently failing | **Tier 2 — URGENT (bypass strikes with OOM confirmation)** -RAM below `MEM_GB` AND OOM kills this cycle ≥ `SYS_WATCHDOG_OOM_LIMIT`. +RAM below `SYS_WATCHDOG_MEM_GB` AND OOM kills this cycle ≥ `SYS_WATCHDOG_OOM_LIMIT`. Both conditions required — RAM alone uses the standard strike system. OOM confirms the system is dying faster than watchdogs can heal. **Tier 3 — STANDARD (`SYS_WATCHDOG_STRIKES` consecutive failures → reboot)** | Check | Threshold | |-------|-----------| -| Free RAM | `MEM_WARN_GB` → `MEM_SHUTDOWN_GB` → `MEM_GB` | +| Free RAM | `SYS_WATCHDOG_MEM_GB` (reboot trigger — earlier tiers handled by resource_watchdog) | | Load average | `SYS_WATCHDOG_LOAD_MULTIPLIER` × cpu_count | | CPU temperature | `SYS_WATCHDOG_CPU_TEMP` | | Zombie processes | `SYS_WATCHDOG_ZOMBIES` | @@ -539,19 +539,23 @@ OOM confirms the system is dying faster than watchdogs can heal. | /tmp usage | `SYS_WATCHDOG_TMP_PCT` | | Array disk errors | mdstat error delta > 0 | | NIC state | interface operstate != "up" | -| Required containers | containers in `SYS_WATCHDOG_REQUIRED_CONTAINERS` | +| Required containers | containers in `HOST*_WATCHDOG_REQUIRED_CONTAINERS` (host*.conf) | ### RAM Tiers +RAM pressure is a graduated response split across resource_watchdog and stability_watchdog: + ``` -MEM_WARN_GB (10GB) → warn + notify, no action -MEM_SHUTDOWN_GB (6GB) → stop non-essential containers, wait for recovery -MEM_GB (4GB) → strike → reboot (URGENT bypass with OOM) -MEM_RECOVER_GB (30GB) → RAM must reach this before stopped containers restart +RW_RAM_SOFT_GB (12GB) → throttle downloads, reduce background load (resource_watchdog) +RW_RAM_MEDIUM_GB (8GB) → pause background containers (resource_watchdog) +RW_RAM_HARD_GB (6GB) → stop optional containers, wait for recovery (resource_watchdog) +SYS_WATCHDOG_MEM_GB (4GB) → strike → reboot (last resort) (stability_watchdog) +RW_RAM_RECOVER_GB (20GB) → RAM must reach this before stopped containers restart ``` -At `MEM_SHUTDOWN_GB`, all containers NOT listed in `SYS_WATCHDOG_MEM_SHUTDOWN_EXCLUDED` -are stopped. Adjust in master.conf for your critical services. +`SYS_WATCHDOG_MEM_GB` must always be below `RW_RAM_HARD_GB` — resource_watchdog acts first. +Containers stopped at the hard tier use `HOST*_RW_STOP_CONTAINERS` in host*.conf. +Containers paused at the medium tier use `HOST*_RW_PAUSE_CONTAINERS` in host*.conf. ### Abort Conditions