Codebase-wide audit pass: fixed real bugs (SSH hangs missing BatchMode,
local-outside-function no-ops, variable name collisions, a truncated
ratio calc, wrong state-dir path, DARK vs NO_INTERNET drift, and more),
then pulled logic that was duplicated across multiple scripts — arr
cleanup safety gates, docker restart ordering, container maintenance
stop/restart, watchdog state-file helpers, partnership role resolution,
cert expiry checks, remote node discovery, and TMDB discovery scoring —
into common.sh so each now has a single implementation.
Fix discrepancies found by auditing docs against headers:
- stability_watchdog.sh: wrong variable name (STRIKES → STRIKE_LIMIT) and wrong default (2hr → 12hr) in header
- Watchdogs manual: REBOOT_WINDOW_HRS example value was 2, should be 12
- Transcodes README: transcode_cleanup/manager table said "Every 3 min", should be "Every 7 minutes"
- Tools manual: fallback_state_reset.sh still marked "not yet built" — rewrite section to reflect current script
- Tools manual: claude_startup --setup flag doesn't exist; modes were inverted — fix and add --launch
- Tools README + manual: add docker_prune_images.sh (existed but undocumented)
- Tools manual: add play_state_sync.sh --full to flag reference
- Orchestrators README: arrs_failed_stalled_recovery runs via intermediate_sync (every 4hr), not standalone every 6hr
- Plugin README: add build.sh to scripts table
- Partnership README + manual: add partnership_transfer.sh and onboard_cancel.sh to script tables and flag reference
stability_watchdog.sh mover check now uses platform_is_mover_running()
instead of raw pgrep, matching server_reboot.sh.
check_unraid_version_parity() renamed to check_os_version_parity() —
the function internals were already platform-neutral after the adapter
refactor; the name was the last Unraid artifact.
All bash scripts are now platform-neutral. Unraid-specific paths, commands,
and service checks moved to Plugin/unraid/adapter.sh. Core scripts call
platform_*() functions exclusively — no direct OS paths in runtime logic.
New adapter functions: platform_storage_path, platform_webui_install_path,
platform_scripts_dir_probe_cmd, platform_setup_db_path, platform_storage_healthy,
platform_is_service_enabled, platform_get_temp_thresholds, platform_disk_states_path,
platform_rebuild_container, platform_push_conf, platform_push_setup_state,
platform_get_templates_dir, platform_send_os_notification.
Partnership services stack (Emby/Jellyfin/Seerr/SeerrFin) added as third
onboarding stack alongside auth and arr stacks.
docker_watchdog: config dump at startup (thresholds/limits), skip list shown when active, per-container healthy log for Tier1 required + mem/CPU monitored containers
stability_watchdog: config dump with all tier thresholds, log() on pass for rootfs/log/tmp/load/zombies/NIC checks (previously silent on clean)
resource_watchdog: config dump with all pressure thresholds and container lists, log normal pressure state with live RAM/load values
system_watchdog: per-script timing on each child script run
network_watchdog: config dump (internet URL, DDNS domain/container, NPM URL, strike limit)
storage_watchdog: config dump (growth threshold, log max, paths, suppress ceilings)
webgui_watchdog: log nginx worker and php-fpm worker counts on healthy check
- Remove dead DOCKER_TIMEOUT=10 from stability_watchdog (daemon check
is delegated to docker_watchdog; variable was never used post-refactor)
- Fix parity check detection: parity-date.txt is gone in Unraid 7.3;
now reads mdResync from var.ini (non-zero = check/sync in progress),
with parity-date.txt fallback for older versions
- Same is_parity_running() fix in docker_watchdog
- DOCKER_TIMEOUT in docker_watchdog now reads WATCHDOG_DAEMON_TIMEOUT
from master.conf (default 20s) instead of hardcoded 15s
- WATCHDOG_DAEMON_RESTART_WAIT now reads from master.conf
(default 900s/15min) instead of hardcoded 30s
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
common.sh: add is_docker_enabled() and is_vm_manager_enabled() helpers
reading /boot/config/docker.cfg and /boot/config/domain.cfg.
docker_watchdog: exit cleanly if Docker not enabled in Unraid settings.
stability_watchdog: skip Docker daemon check and Docker container stop
if Docker not enabled; skip virsh VM shutdown if VM Manager not enabled.
server_reboot: skip VM shutdown and libvirt stop if VM Manager not enabled.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
stability_watchdog: remove SYS_WATCHDOG_FAILED_FILE from state file
touch list — skip list is owned by docker_watchdog, not stability.
docker_watchdog: rename check_system_watchdog_state →
check_resource_watchdog_state (reads RW_STATE_FILE from resource_watchdog,
not SYS_WATCHDOG_STATE_FILE from stability_watchdog). Update all comments
and state file docs to reflect correct ownership.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
docker_watchdog now writes daemon_confirmed_down=true to its state file
when restart is attempted and daemon is still unresponsive. Clears the
flag on recovery.
stability_watchdog removes the duplicate CRITICAL daemon check (which
bypassed all strikes, all abort conditions, and had its own rc.docker
restart). Replaced with a standard strike check reading the
daemon_confirmed_down flag from docker_watchdog's state file.
Result: docker_watchdog owns all daemon restart logic. stability_watchdog
escalates to reboot only after N consecutive strike cycles confirming the
daemon is truly unrecoverable — same pattern as all other standard checks.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
stability_watchdog: container health is owned by docker_watchdog — rebooting
when docker_watchdog skip-lists a container creates a reboot loop. Removed
SYS_WATCHDOG_CHECK_CONTAINERS check entirely. docker_watchdog handles
strikes, restart attempts, skip-listing, and notifications.
varaverk.plg: warn on USB/removable boot device at install time. Plugin is
designed for internal NVMe/SSD boot. Warning is non-blocking but clear.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
All watchdog scripts now open with a consistent echo banner:
━━━ <icon> <Name> Watchdog — <timestamp> ━━━
Previously network and storage used a quiet log() line (no separator),
and stability_watchdog used warn() which made every run show ⚠️.
Also: network_watchdog now logs "Network healthy ✅" on clean exit,
and stability_watchdog's reboot-trigger block now says "Stability"
instead of the old "System Watchdog" label.
Watchdog renames and moves:
system_watchdog.sh → stability_watchdog.sh (last line of defense — reboots)
storage_watchdog.sh → Watchdogs/System/storage_watchdog.sh
webgui_restart.sh → Watchdogs/System/webgui_watchdog.sh (renamed to match folder convention)
New thin orchestrator:
Watchdogs/system_watchdog.sh — runs SYSTEM_WATCHDOG_SCRIPTS from master.conf
Sits between docker_watchdog and stability_watchdog in the orchestrator tier chain
System/ subfolder is the growth seam for future system component watchdogs
master.conf:
WATCHDOG_ORCHESTRATOR_SCRIPTS updated — storage removed, system_watchdog added as tier
SYSTEM_WATCHDOG_SCRIPTS array added — storage + webgui
server_reboot.sh:
Calls array_stopping.sh before VM shutdown for guaranteed safe array stop
Removed raw rc.docker stop and exit trap — orchestrator owns container shutdown
Gmer4Lfe