Commit Graph
61 Commits
Author SHA1 Message Date
Gmer4Lfe b78fdddb04 fix(watchdogs): Tailscale false positive + system watchdog banner
network_watchdog: Tailscale JSON uses "BackendState": "Running" (with space)
but grep pattern matched "BackendState":"Running" (no space) — always failed.
Switched to grep -qE with \s* to handle both forms.

system_watchdog: add echo banner at run start so its output is visually
separated from docker_watchdog in orchestrator logs. Previously appeared
as a continuation of the docker section since docker_watchdog has a banner
but system_watchdog did not.
2026-05-24 17:11:54 -04:00
Gmer4Lfe f85493cf35 watchdogs: fix Tailscale error propagation and docker watchdog output order
- network_watchdog: remove ISSUES++ for Tailscale — it's notify-only by
  design, no restart possible; cascading to system watchdog exit 1 was noise
- docker_watchdog: move banner to top of cycle so body errors appear under
  the correct section header, not visually inside Resource Manager output
2026-05-24 16:56:32 -04:00
Gmer4Lfe 2d0c91ea13 fix stale RAM tier vars in stability_watchdog section of Watchdogs manual
Update the Three-Tier Response and RAM Tiers sections to reflect the
resource_watchdog/stability_watchdog split: MEM_WARN_GB/MEM_SHUTDOWN_GB/
MEM_RECOVER_GB moved to RW_RAM_SOFT/MEDIUM/HARD/RECOVER_GB in resource_watchdog;
stability_watchdog now only holds SYS_WATCHDOG_MEM_GB (reboot trigger). Update
required containers reference from SYS_WATCHDOG_REQUIRED_CONTAINERS (removed)
to HOST*_WATCHDOG_REQUIRED_CONTAINERS in host*.conf.
2026-05-22 22:09:13 -04:00
Gmer4Lfe 2fa78a21b6 doc audit — fix stale watchdog vars, fallback_state_reset, add monthly_maintenance and mesh_monitor sections
- Tools/Manual-Tools.md: fallback_state_reset.sh marked not yet built; manual
  workaround added; TOC entry updated; failover→fallback rename throughout
- Watchdogs/Manual-Watchdogs.md: System Watchdog config block updated — remove
  pre-split vars (MEM_WARN_GB, MEM_SHUTDOWN_GB, MEM_GB, MEM_RECOVER_GB,
  SYS_WATCHDOG_MEM_SHUTDOWN_EXCLUDED, SYS_WATCHDOG_REQUIRED_CONTAINERS);
  add SYS_WATCHDOG_MEM_GB (reboot trigger only); fix state file path names
  (watchdog_state.db → container_watchdog_state.db,
  sys_watchdog_state.db → system_watchdog_state.db)
- Orchestrators/README-Orchestrators.md: add monthly_maintenance.sh section
  with two-gate model, configuration reference, and usage examples
- Monitors/README-Monitors.md: add mesh_monitor.sh to both tables and diagram
- Monitors/Manual-Monitors.md: add mesh_monitor.sh section
- README-User_Script_Plug-in.md, Rsync/README-Rsync.md, Rsync/Manual-Rsync.md,
  README.md: emby-failover → emby-fallback (missed from prior batch)
2026-05-22 22:07:46 -04:00
Gmer4Lfe d05df84d00 Docs: add network_watchdog to README-Watchdogs and Manual-Watchdogs
README-Watchdogs: tier diagram, folder relationship, script table,
  execution flow, state file table
Manual-Watchdogs: TOC entry, full System/network_watchdog.sh section
  (check sequence, config, usage, troubleshooting), config reference block
2026-05-22 20:54:04 -04:00
Gmer4Lfe 4535804c5d Add System/network_watchdog.sh — services-layer connectivity watchdog
Checks:
  1. Internet reachability (gates all remaining checks)
  2. DDNS sync — public IP vs Cloudflare DNS record, restarts Gmer4Lfe.com container on mismatch
  3. Tailscale status — notify only, no auto-restart
  4. NPM proxy — external curl to https://gmer4lfe.com, 2-strike system before NginxProxyManager restart

Config: master.conf NETWORK_WATCHDOG_* block, host1.conf HOST1_NETWORK_WATCHDOG_* values
Added to SYSTEM_WATCHDOG_SCRIPTS — called by system_watchdog.sh each cycle

Fix: storage_watchdog.sh was calling get_strikes/set_strikes without defining them —
added local definitions (same pattern as docker_watchdog.sh and stability_watchdog.sh)
2026-05-22 20:51:22 -04:00
Gmer4Lfe 13ce8497f3 Update all docs to reflect watchdog restructure
README-Watchdogs.md: new tier diagram, updated script table, execution flow,
  state file table, and folder relationship diagram
Manual-Watchdogs.md: added system_watchdog (thin), System/storage_watchdog,
  System/webgui_watchdog, and stability_watchdog sections; updated TOC and troubleshooting
README.md: updated schedule listing, folder description, daily cycle examples
README-User_Script_Plug-in.md: stability_watchdog references throughout
unRAID_Essentials/README + Manual: stability_watchdog, updated storage path
Monitors/README, Rsync/Manual, Tools/Manual: lock name list updates
2026-05-22 20:26:36 -04:00
Gmer4Lfe cf21efaeea Structural reorganization: watchdog taxonomy + server_reboot integration
Watchdog renames and moves:
  system_watchdog.sh → stability_watchdog.sh (last line of defense — reboots)
  storage_watchdog.sh → Watchdogs/System/storage_watchdog.sh
  webgui_restart.sh → Watchdogs/System/webgui_watchdog.sh (renamed to match folder convention)

New thin orchestrator:
  Watchdogs/system_watchdog.sh — runs SYSTEM_WATCHDOG_SCRIPTS from master.conf
  Sits between docker_watchdog and stability_watchdog in the orchestrator tier chain
  System/ subfolder is the growth seam for future system component watchdogs

master.conf:
  WATCHDOG_ORCHESTRATOR_SCRIPTS updated — storage removed, system_watchdog added as tier
  SYSTEM_WATCHDOG_SCRIPTS array added — storage + webgui

server_reboot.sh:
  Calls array_stopping.sh before VM shutdown for guaranteed safe array stop
  Removed raw rc.docker stop and exit trap — orchestrator owns container shutdown
2026-05-22 20:17:22 -04:00
Gmer4Lfe 8a3e22c9b8 Watchdogs/ docs: new README + Manual, update affected folders
New docs:
  Watchdogs/README-Watchdogs.md  — design, relationships, script table, state file map
  Watchdogs/Manual-Watchdogs.md  — full config reference for all 4 watchdogs

Docker_Essentials/:
  README — remove docker_watchdog, update folder description and diagrams
  Manual  — strip watchdog config sections, add pointer to Watchdogs/Manual

unRAID_Essentials/:
  README — remove system/resource watchdog, update diagrams and script table
  Manual  — strip system/resource watchdog sections, update TOC + config reference

README.md:
  Add Watchdogs/ to folder structure
  Fix "WHAT RUNS WHEN" — watchdogs run via orchestrator every minute, not array start
  Fix daily cycle and monitoring diagrams
2026-05-22 17:26:21 -04:00
Gmer4Lfe ec79a926e8 Revert orchestrator + skip_list_manager to original folders
watchdog_orchestrator.sh stays in Orchestrators/ — it's an orchestrator, not a watchdog.
watchdog_skip_list_manager.sh stays in Tools/ — it's a management utility.

Only the 4 watchdog scripts belong in Watchdogs/:
  docker_watchdog.sh, resource_watchdog.sh, storage_watchdog.sh, system_watchdog.sh
2026-05-22 17:10:53 -04:00
Gmer4Lfe 95151c2278 Watchdogs/ folder + host conf rename
Move all watchdog scripts to a dedicated Watchdogs/ folder:
  Docker_Essentials/docker_watchdog.sh   → Watchdogs/
  unRAID_Essentials/system_watchdog.sh   → Watchdogs/
  unRAID_Essentials/resource_watchdog.sh → Watchdogs/
  Orchestrators/watchdog_orchestrator.sh → Watchdogs/
  Tools/watchdog_skip_list_manager.sh    → Watchdogs/

Rename host config files:
  master_host1.conf → host1.conf
  master_host2.conf → host2.conf

Update all references across the ecosystem:
  master.conf: WATCHDOG_ORCHESTRATOR_SCRIPTS paths → Watchdogs/
  load_config.sh: host*.conf glob + all comments
  git_pull_execute.sh: sparse checkout glob + all comments
  Partnership/ssh_setup.sh: HOST_CONF path construction
  user_script_plug-in.sh: all script paths + per-host conf path
  common.sh, README.md, README-User_Script_Plug-in.md: comment refs
  All Partnership, Fallback, Monitors, Transcodes, Tools scripts: comment refs
2026-05-22 17:08:36 -04:00