lm-sensors prints high= and crit= on the same line as the reading, so scraping
the line for its largest number reported a flat 100C on Intel while the shell's
last-field parse got a bare ")" and never fired at all. One parse in the
adapter now, preferring Tdie over Tctl since Tctl carries a +27C offset.
Every existing exemption lasts until someone remembers to undo it, and nobody
does — Healarr has sat in a pressure list since it was uninstalled and seven
ignore entries name containers that are gone. A mute states when it ends and
then ends, capped by WATCHDOG_MUTE_MAX_HOURS so temporary is enforced rather
than intended. Applied where IGNORE_MAP is built, so all five check sites
inherit it, and shown with its countdown because an invisible suppression is
the thing being fixed.
The state file is keyed both ways — a bare container name for the container check,
container plus suffix for the HTTP, API, CPU and docker checks — so inspecting the key
asked docker about Emby_http and deleted a live counter every cycle, putting the
two-strike limits out of reach.
The 2026-07-19 prune covered the skip list and intentional stops but not the
per-container strike counts, which have the same failure: set_strikes 0 only
fires when a container is seen running again, so an uninstalled one keeps its
count forever and every consumer reads the host as unhealthy. Reserved daemon_*
keys are excluded.
The docs had drifted from the scripts — a script that no longer exists, three wrong variable
names, a reversed run order, and seven scheduled scripts that were never documented at all.
Finishes the pass: every script now documents its safeguards, and the deliberate absences
in the sourced libraries are recorded so they are not "corrected" later.
MY_ID is not exported and is unset until detect_hosts runs, so the DDNS and NPM checks
were silently resolving empty config and skipping themselves every cycle.
Neither remove_from_skip_list() nor clear_intentional_stop() can ever
fire for a container that's been fully uninstalled — both only trigger
on seeing it running again. Confirmed live 2026-07-19: Healarr sat on
the skip list nagging "manual intervention needed" every single cycle
for weeks after being removed. Now checks docker inspect for each
entry every cycle and drops it automatically if the container no
longer exists at all.
Both had a basic HTTP check (Emby) or no coverage at all (Jellyfin), but
neither would have caught today's real incident: Jellyfin's SQLite
database locked up hard (repeated 'database table is locked' errors,
30s+ query timeouts) while its own /System/Info endpoint kept responding
200 the whole time — a basic HTTP check on that endpoint would never have
tripped. /Users forces an actual DB round-trip and was confirmed live to
hang during the exact incident.
Generalized the API check's success condition to also accept array-shaped
responses (/Users returns an array; the existing check only recognized
object fields like .ServerName/.Id/.Version, which would error when
applied to an array) — benefits any future array-returning endpoint, not
just this one. Also corrected the host.conf.template's API_CHECKS format
comment, which described a 3-field format the code never actually used.
Codebase-wide audit pass: fixed real bugs (SSH hangs missing BatchMode,
local-outside-function no-ops, variable name collisions, a truncated
ratio calc, wrong state-dir path, DARK vs NO_INTERNET drift, and more),
then pulled logic that was duplicated across multiple scripts — arr
cleanup safety gates, docker restart ordering, container maintenance
stop/restart, watchdog state-file helpers, partnership role resolution,
cert expiry checks, remote node discovery, and TMDB discovery scoring —
into common.sh so each now has a single implementation.
TRANSCODE_LOG_RETENTION was documented as trimming the daily transcode log
but never referenced — log grew unbounded. TRANSCODE_ORPHAN_AGE was shown in
--status but never used, so empty session folders were deleted immediately
instead of after the documented grace period, risking a race against ffmpeg
creating a folder just before writing its first segment.
docker_watchdog.sh's daemon-health thresholds were only hardcoded fallbacks
despite comments claiming they were master.conf-configurable, and it
referenced a heartbeat feature that was never implemented (that's owned by
watchdog_orchestrator.sh, its caller) — added the three thresholds to
master.conf for real and removed the stale heartbeat claim.
Also added the missing HOSTN_PARTNERSHIP_SERVICES_STACK block to
host.conf.template (containers.sh already read it via detect_hosts, just
never had a template entry) and corrected play_state_sync.sh's doc comment
for PLAY_SYNC_TYPES' actual default.
Same audit as the orchestrator standardization pass (2a062e5), extended to
every remaining script. Found the same class of bug independently recurring:
ramdisk_stop.sh checked $LOG (nothing assigns it, should be $ENABLE_LOGGING),
partnership_onboard.sh checked $LOG_MODE (same issue), emby_session_report.sh
checked $TRANSCODE_PCT which was never computed so the high-transcode alert
could never fire, and storage_migrate.sh never called detect_hosts() so
$MY_ID was empty, silently breaking the post-migration host*.conf update.
partnership_manager.sh used `local` at top-level script scope (invalid outside
a function) and had two master.conf path references missing "Configurations/".
Along the way: several scripts (share_setup.sh, conf_sync.sh,
downloaders_reset.sh, transcode_cleanup.sh, transcode_manager.sh,
remote_arr_cache_writer.sh, upgrade_webhook_handler.sh) had no explicit
trailing exit code, so they always reported success regardless of real
failures. play_state_sync.sh was missing the partnership gate its own header
documented, so remote play-state sync ran even with PARTNERSHIP_ENABLED=false;
it also always exited 0 on sync errors. arr_profile_enforcer.sh and
webhook_setup.sh hand-rolled their own flag parsing instead of common.sh's
parse_args, so --log silently did nothing on either.
system_watchdog.sh was itself an un-standardized mini-orchestrator — converted
to the shared run_orch_child()/JOB_PASS/JOB_FAIL pattern, added the missing
failure notification, and fixed dry-run to pass --dry-run down to children
instead of skipping them outright. Also fixed a stale webgui_watchdog.sh path
in master.conf.template that would break system_watchdog.sh on any fresh
install.
Closed a sibling-drift gap: radarr_cleanup.sh and sonarr_cleanup.sh were
missing lidarr_cleanup.sh's tracked-count percentage-drop safety gate and its
"not configured on this host, skip cleanly" guard — both now match Lidarr's
7-gate model.
Exit code 0 on a required container (docker stop, UI stop) now reads as
intentional — no strike, no restart. Non-zero exits still trigger the
existing strike → restart path.
Adds --pause / --resume management commands and a persistent intentional-
stops state file for maintenance windows where even the exit-code heuristic
isn't enough. Containers auto-cleared from the list when seen running again.
Every script now has the established header format: PURPOSE with ─────── separator,
OPERATIONAL MODEL, DESIGN PRINCIPLES, OPERATIONAL SAFEGUARDS, CONFIGURATION, and
RUNTIME MODES — structured with full ====== banner sections throughout.
Orchestrators converted from compact ── inline format to full banners. Stale
emby-fallback and dirty sync references removed from Plugin/user_script_plug-in.sh.
All state/data file paths in scripts and PHP now resolve via STATE_DIR / DATA_DIR /
PERSISTENT_CONF_CACHE instead of hardcoded /boot/config/ or /tmp/ paths, so the
ecosystem works in both internal and appdata storage modes.
PHP layer (watchdog.php, partnership.php, fallback.php, monitor.php, snapshot.php,
config.php): all state reads switched to STATE_DIR constant; remote state reads use
the new vv_remote_state_cmd() helper which resolves the remote's SCRIPTS_DIR via
their varaverk.cfg before building the path.
conf_sync.sh: fixed SCRIPTS_ROOT → SCRIPTS_DIR bug on MY_CONF path; added
_remote_scripts_dir() to resolve partner's SCRIPTS_DIR before SCP pull.
fallback.php page: added controls card (PARTNERSHIP_ENABLED, FALLBACK_ENABLED,
FALLBACK_RSYNC_ENABLED toggles), status grid, and settings card.
README and Manual updated for System_Essentials, Watchdogs, Fallback, Rsync,
Media, Monitors, Orchestrators, Partnership: added new scripts (conf_sync,
conf_cache_save/restore, conf_cache_watchdog, play_state_sync, start_webhook_listener,
upgrade_webhook_handler), corrected all stale /boot/config/ state file paths to
$STATE_DIR/$DATA_DIR, noted webgui/php_fpm/mover/user_scripts scripts moved to
Plugin/unraid/System_Essentials, fixed start_webhook_listener.sh header (Node.js,
not PHP -S).
Adds require_partnership() to common.sh — exits cleanly when PARTNERSHIP_ENABLED=false.
Removes FALLBACK_PARTNERSHIP_REQUIRED toggle — partnership is now always required,
not optional. Cross-server scripts (rsync, conf sync, fallback, arr sync, play state,
backup verify) all call require_partnership after detect_hosts.
Writes partner confs from RAM cache to /boot/config/.cache/vv/d/ while
remote is offline, and removes the backup when remote comes back. Called
each minute via SYSTEM_WATCHDOG_SCRIPTS so crashes and power loss are
covered — not just graceful shutdowns.
Fix discrepancies found by auditing docs against headers:
- stability_watchdog.sh: wrong variable name (STRIKES → STRIKE_LIMIT) and wrong default (2hr → 12hr) in header
- Watchdogs manual: REBOOT_WINDOW_HRS example value was 2, should be 12
- Transcodes README: transcode_cleanup/manager table said "Every 3 min", should be "Every 7 minutes"
- Tools manual: fallback_state_reset.sh still marked "not yet built" — rewrite section to reflect current script
- Tools manual: claude_startup --setup flag doesn't exist; modes were inverted — fix and add --launch
- Tools README + manual: add docker_prune_images.sh (existed but undocumented)
- Tools manual: add play_state_sync.sh --full to flag reference
- Orchestrators README: arrs_failed_stalled_recovery runs via intermediate_sync (every 4hr), not standalone every 6hr
- Plugin README: add build.sh to scripts table
- Partnership README + manual: add partnership_transfer.sh and onboard_cancel.sh to script tables and flag reference
stability_watchdog.sh mover check now uses platform_is_mover_running()
instead of raw pgrep, matching server_reboot.sh.
check_unraid_version_parity() renamed to check_os_version_parity() —
the function internals were already platform-neutral after the adapter
refactor; the name was the last Unraid artifact.
All bash scripts are now platform-neutral. Unraid-specific paths, commands,
and service checks moved to Plugin/unraid/adapter.sh. Core scripts call
platform_*() functions exclusively — no direct OS paths in runtime logic.
New adapter functions: platform_storage_path, platform_webui_install_path,
platform_scripts_dir_probe_cmd, platform_setup_db_path, platform_storage_healthy,
platform_is_service_enabled, platform_get_temp_thresholds, platform_disk_states_path,
platform_rebuild_container, platform_push_conf, platform_push_setup_state,
platform_get_templates_dir, platform_send_os_notification.
Partnership services stack (Emby/Jellyfin/Seerr/SeerrFin) added as third
onboarding stack alongside auth and arr stacks.
- Auth stack: fold cert monitor into Auth Stack page as fourth tab (Certs);
remove standalone cert page and top-level tab
- cert_monitor.sh: write JSON status cache to State_Files/cert_status.json
after each run; expose per-domain days/expiry via _CERT_DAYS/_CERT_EXPIRY globals
- api/cert.php: new — serves cached cert status; falls back to configured
domains as UNKN when no cache exists; POST action=run triggers live check
- arrs db fallbacks: vv_arr_cleanup_stats/discovery_stats/recovery_stats now
read from data/*.db files when log JSON files don't yet exist
- config.php vv_conf_vars(): unescape bash \$ → $ so passwords with dollar
signs read correctly from conf files
- host1.conf: fill in HOST1_NPM_USER/PASS and HOST1_LLDAP_USER/PASS
- Partnership adapter pattern: Unraid-specific container logic extracted to
Plugin/unraid/Partnership/; platform-agnostic structure stays in Partnership/
- First-run wizard: uniform multi-step flow for all hosts; HOST2 pull moved
to checklist; auto SSH keygen and API key creation on save
- api/checklist.php: live setup checklist with pull_master action
- Fullscreen toggle: hide Unraid header/menu; state persists via localStorage
play_state_sync: fix TVDB episode lookup for Jellyfin 10.x — AnyProviderIdEquals
returns the entire library for TVDB queries; switch to season+episode search with
ProviderIds.Tvdb validation to find the correct episode. Also fix pkey extraction
that was pulling s7e2 instead of the TVDB ID from tvdb:ep:5618559:s7e2.
docker_watchdog: add automatic manual-stop detection — containers stopped cleanly
(exit 0/143) are tracked in docker_watchdog_manual_stop.db and skipped until
restarted, removing the need to add manually-stopped containers to the exclusion
list. Auto-clears when the container is seen running again.
docker_daily_restart: remove bare `local` declarations outside a function that
were printing an error for every container restarted.
docker_watchdog: config dump at startup (thresholds/limits), skip list shown when active, per-container healthy log for Tier1 required + mem/CPU monitored containers
stability_watchdog: config dump with all tier thresholds, log() on pass for rootfs/log/tmp/load/zombies/NIC checks (previously silent on clean)
resource_watchdog: config dump with all pressure thresholds and container lists, log normal pressure state with live RAM/load values
system_watchdog: per-script timing on each child script run
network_watchdog: config dump (internet URL, DDNS domain/container, NPM URL, strike limit)
storage_watchdog: config dump (growth threshold, log max, paths, suppress ceilings)
webgui_watchdog: log nginx worker and php-fpm worker counts on healthy check
Two root causes of the May 31 19:30 hang (orchestrator stuck for 45min):
1. /etc/rc.d/rc.docker restart had no timeout — on a host with many
containers, stopping them all before daemon restart can take 30+
minutes. Added timeout 180 so it's bounded.
2. WATCHDOG_DAEMON_RESTART_WAIT=900 (15min sleep after restart) —
unnecessary. The daemon itself comes up in seconds; containers are
handled by subsequent docker_watchdog cycles. Reduced to 90s in
master.conf, cutting worst-case orchestrator block from 45min to ~5min.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
- Remove dead DOCKER_TIMEOUT=10 from stability_watchdog (daemon check
is delegated to docker_watchdog; variable was never used post-refactor)
- Fix parity check detection: parity-date.txt is gone in Unraid 7.3;
now reads mdResync from var.ini (non-zero = check/sync in progress),
with parity-date.txt fallback for older versions
- Same is_parity_running() fix in docker_watchdog
- DOCKER_TIMEOUT in docker_watchdog now reads WATCHDOG_DAEMON_TIMEOUT
from master.conf (default 20s) instead of hardcoded 15s
- WATCHDOG_DAEMON_RESTART_WAIT now reads from master.conf
(default 900s/15min) instead of hardcoded 30s
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
common.sh: add is_docker_enabled() and is_vm_manager_enabled() helpers
reading /boot/config/docker.cfg and /boot/config/domain.cfg.
docker_watchdog: exit cleanly if Docker not enabled in Unraid settings.
stability_watchdog: skip Docker daemon check and Docker container stop
if Docker not enabled; skip virsh VM shutdown if VM Manager not enabled.
server_reboot: skip VM shutdown and libvirt stop if VM Manager not enabled.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Skip list belongs to docker_watchdog, not stability_watchdog (legacy
naming from when docker+system were the only two watchdogs). Rename
variable and physical file system_watchdog_failed.db →
docker_watchdog_failed.db across all files.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
stability_watchdog: remove SYS_WATCHDOG_FAILED_FILE from state file
touch list — skip list is owned by docker_watchdog, not stability.
docker_watchdog: rename check_system_watchdog_state →
check_resource_watchdog_state (reads RW_STATE_FILE from resource_watchdog,
not SYS_WATCHDOG_STATE_FILE from stability_watchdog). Update all comments
and state file docs to reflect correct ownership.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Previously only cleared inside the strikes>0 block. After a reboot,
strikes reset to 0 but the flag persists on flash — daemon could be
healthy but flag stays true, causing stability_watchdog to keep striking.
Now cleared on every healthy docker info check regardless of strike count.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
docker_watchdog now writes daemon_confirmed_down=true to its state file
when restart is attempted and daemon is still unresponsive. Clears the
flag on recovery.
stability_watchdog removes the duplicate CRITICAL daemon check (which
bypassed all strikes, all abort conditions, and had its own rc.docker
restart). Replaced with a standard strike check reading the
daemon_confirmed_down flag from docker_watchdog's state file.
Result: docker_watchdog owns all daemon restart logic. stability_watchdog
escalates to reboot only after N consecutive strike cycles confirming the
daemon is truly unrecoverable — same pattern as all other standard checks.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
stability_watchdog: container health is owned by docker_watchdog — rebooting
when docker_watchdog skip-lists a container creates a reboot loop. Removed
SYS_WATCHDOG_CHECK_CONTAINERS check entirely. docker_watchdog handles
strikes, restart attempts, skip-listing, and notifications.
varaverk.plg: warn on USB/removable boot device at install time. Plugin is
designed for internal NVMe/SSD boot. Warning is non-blocking but clear.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Scripts calling acquire_lock followed by acquire_rsync_lock (rsync.sh)
or a custom EXIT trap (resource_watchdog.sh) would overwrite the lock
release trap, leaving the first lock file behind on exit.
Replace per-acquire trap with a _LOCK_FILES registry. _register_lock()
appends each lock file to the array and sets a single _release_all_locks
trap. All acquired locks release together on exit regardless of how many
traps are subsequently set or overwritten.
resource_watchdog.sh: chain _release_all_locks into both its custom trap
and the state-persisted disarm path so the lock releases in all exit paths.
log() is gated on ENABLE_LOGGING — silent on normal runs. Healthy
confirmations (All healthy, Network healthy, Storage healthy, WebGUI
healthy) were invisible, making banners appear with nothing after them.
Switched all four to plain echo so they show every cycle.
Gmer4Lfe