From 2d0c91ea13b0097fcc84eb03ee04c8bf7c61c2f9 Mon Sep 17 00:00:00 2001 From: Gmer4Lfe Date: Fri, 22 May 2026 22:09:13 -0400 Subject: [PATCH] fix stale RAM tier vars in stability_watchdog section of Watchdogs manual Update the Three-Tier Response and RAM Tiers sections to reflect the resource_watchdog/stability_watchdog split: MEM_WARN_GB/MEM_SHUTDOWN_GB/ MEM_RECOVER_GB moved to RW_RAM_SOFT/MEDIUM/HARD/RECOVER_GB in resource_watchdog; stability_watchdog now only holds SYS_WATCHDOG_MEM_GB (reboot trigger). Update required containers reference from SYS_WATCHDOG_REQUIRED_CONTAINERS (removed) to HOST*_WATCHDOG_REQUIRED_CONTAINERS in host*.conf. --- Watchdogs/Manual-Watchdogs.md | 22 +++++++++++++--------- 1 file changed, 13 insertions(+), 9 deletions(-) diff --git a/Watchdogs/Manual-Watchdogs.md b/Watchdogs/Manual-Watchdogs.md index 232efdf..cc0598d 100644 --- a/Watchdogs/Manual-Watchdogs.md +++ b/Watchdogs/Manual-Watchdogs.md @@ -524,14 +524,14 @@ healing at every other layer has failed or when the failure is non-recoverable. | /boot read-only | write test fails | Config writes silently failing | **Tier 2 — URGENT (bypass strikes with OOM confirmation)** -RAM below `MEM_GB` AND OOM kills this cycle ≥ `SYS_WATCHDOG_OOM_LIMIT`. +RAM below `SYS_WATCHDOG_MEM_GB` AND OOM kills this cycle ≥ `SYS_WATCHDOG_OOM_LIMIT`. Both conditions required — RAM alone uses the standard strike system. OOM confirms the system is dying faster than watchdogs can heal. **Tier 3 — STANDARD (`SYS_WATCHDOG_STRIKES` consecutive failures → reboot)** | Check | Threshold | |-------|-----------| -| Free RAM | `MEM_WARN_GB` → `MEM_SHUTDOWN_GB` → `MEM_GB` | +| Free RAM | `SYS_WATCHDOG_MEM_GB` (reboot trigger — earlier tiers handled by resource_watchdog) | | Load average | `SYS_WATCHDOG_LOAD_MULTIPLIER` × cpu_count | | CPU temperature | `SYS_WATCHDOG_CPU_TEMP` | | Zombie processes | `SYS_WATCHDOG_ZOMBIES` | @@ -539,19 +539,23 @@ OOM confirms the system is dying faster than watchdogs can heal. | /tmp usage | `SYS_WATCHDOG_TMP_PCT` | | Array disk errors | mdstat error delta > 0 | | NIC state | interface operstate != "up" | -| Required containers | containers in `SYS_WATCHDOG_REQUIRED_CONTAINERS` | +| Required containers | containers in `HOST*_WATCHDOG_REQUIRED_CONTAINERS` (host*.conf) | ### RAM Tiers +RAM pressure is a graduated response split across resource_watchdog and stability_watchdog: + ``` -MEM_WARN_GB (10GB) → warn + notify, no action -MEM_SHUTDOWN_GB (6GB) → stop non-essential containers, wait for recovery -MEM_GB (4GB) → strike → reboot (URGENT bypass with OOM) -MEM_RECOVER_GB (30GB) → RAM must reach this before stopped containers restart +RW_RAM_SOFT_GB (12GB) → throttle downloads, reduce background load (resource_watchdog) +RW_RAM_MEDIUM_GB (8GB) → pause background containers (resource_watchdog) +RW_RAM_HARD_GB (6GB) → stop optional containers, wait for recovery (resource_watchdog) +SYS_WATCHDOG_MEM_GB (4GB) → strike → reboot (last resort) (stability_watchdog) +RW_RAM_RECOVER_GB (20GB) → RAM must reach this before stopped containers restart ``` -At `MEM_SHUTDOWN_GB`, all containers NOT listed in `SYS_WATCHDOG_MEM_SHUTDOWN_EXCLUDED` -are stopped. Adjust in master.conf for your critical services. +`SYS_WATCHDOG_MEM_GB` must always be below `RW_RAM_HARD_GB` — resource_watchdog acts first. +Containers stopped at the hard tier use `HOST*_RW_STOP_CONTAINERS` in host*.conf. +Containers paused at the medium tier use `HOST*_RW_PAUSE_CONTAINERS` in host*.conf. ### Abort Conditions