FailedProxy/Varaverk
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Platform adapter: rename System_Essentials, add Plugin/unraid/adapter.sh, wire call sites

Browse Source 
- Rename unRAID_Essentials/ → System_Essentials/ (git detects as rename)
- Add Plugin/unraid/adapter.sh: 13 platform_*() functions providing OS-agnostic API
  for storage health, service management, mover, user scripts, notifications,
  disk temps, and platform command validation
- Update load_config.sh: detect PLATFORM (unraid/truenas/unknown), export SCRIPTS_DIR,
  auto-source Plugin/$PLATFORM/adapter.sh after common.sh
- Wire all call sites: replace direct rc.d, pgrep/pkill, var.ini, dynamix.cfg,
  disks.ini, and validate_unraid_cmd calls with platform_*() functions across
  watchdogs, orchestrators, and System_Essentials scripts
- Update all documentation: rename refs, update webgui escalation logic,
  add platform adapter section to Plugin README, update main README with
  portability vision and corrected self-healing stack description
This commit is contained in:
Gmer4Lfe
2026-06-04 18:14:34 -04:00
parent de50a01ab2
commit 369a9e6c19
73 changed files with 522 additions and 228 deletions
Download Patch File Download Diff File Expand all files Collapse all files
Deployment/conf_templates/master.conf
+7 -7
View File
@@ -282,9 +282,9 @@
# watchdog_orchestrator.sh — NOT launched here.
ARRAY_START_SCRIPTS=(
"Transcodes/ramdisk_setup.sh" # creates ramdisk + symlink before Emby starts
"unRAID_Essentials/docker_syslog_filter.sh" # suppress veth noise before logs fill
"unRAID_Essentials/php_fpm_max_children.sh" # WebGUI performance tuning
"unRAID_Essentials/inotify_tuning.sh" # bump inotify limits — containers miss events if exhausted
"System_Essentials/docker_syslog_filter.sh" # suppress veth noise before logs fill
"System_Essentials/php_fpm_max_children.sh" # WebGUI performance tuning
"System_Essentials/inotify_tuning.sh" # bump inotify limits — containers miss events if exhausted
"Docker_Essentials/docker_network_connect.sh" # ensure networks exist + connect containers
"Fallback/fallback.sh" # mutual failover — continuous
)
@@ -294,10 +294,10 @@
# Run sequentially (foreground) — each must complete before the next starts.
# Order matters: user scripts first (prevents new ops), then data movement, then containers.
ARRAY_STOP_SCRIPTS=(
"unRAID_Essentials/user_scripts_stop.sh" # stop background scripts before they start new ops
"System_Essentials/user_scripts_stop.sh" # stop background scripts before they start new ops
"Fallback/fallback.sh --stop" # gracefully stop fallback (not caught by user_scripts_stop)
"unRAID_Essentials/rsync_stop.sh --rsync-only" # kill rsync; skip container recovery (handled below)
"unRAID_Essentials/mover_stop.sh" # stop mover after rsync (they conflict on same files)
"System_Essentials/rsync_stop.sh --rsync-only" # kill rsync; skip container recovery (handled below)
"System_Essentials/mover_stop.sh" # stop mover after rsync (they conflict on same files)
"Docker_Essentials/docker_container_stop.sh" # stop all containers last
)
@@ -393,7 +393,7 @@
WEEKLY_MAINTENANCE_SCRIPTS=(
"Docker_Essentials/docker_weekly_restart.sh" # weekly container restarts after sync
"Docker_Essentials/docker_update_remaining.sh" # pull updates for all other containers
"unRAID_Essentials/clear_logs.sh" # purge aged logs — Sunday only, low priority
"System_Essentials/clear_logs.sh" # purge aged logs — Sunday only, low priority
"Media/playback_aware_lidarr_discovery.sh" # behavior-driven music discovery using weekly Emby playback history
"Media/playback_aware_radarr_discovery.sh" # behavior-driven movie discovery using TMDB recommendations
"Media/playback_aware_sonarr_discovery.sh" # behavior-driven TV discovery using TMDB recommendations
Docker_Essentials/README-Docker_Essentials.md
+1 -1
View File
@@ -153,7 +153,7 @@ its history after you've fixed the problem. No manual file editing required.
```
Docker_Essentials/ ← acts on containers (this folder)
unRAID_Essentials/ ← acts on the server itself
System_Essentials/ ← acts on the server itself
Watchdogs/ ← reactive monitoring + last-resort stability
Monitors/ ← observes, measures, reports
Rsync/ ← moves data between servers
Docker_Essentials/docker_network_connect.sh
+1 -1
View File
@@ -131,7 +131,7 @@ fi
detect_hosts
# Validate unRAID notify script — used for network creation alerts
validate_unraid_cmd "/usr/local/emhttp/plugins/dynamix/scripts/notify" "" "" "unRAID notify script" || warn "unRAID notify script not found — native notifications disabled"
platform_require_cmd "/usr/local/emhttp/plugins/dynamix/scripts/notify" "" "" "unRAID notify script" || warn "unRAID notify script not found — native notifications disabled"
# Docker daemon check — network operations are useless if daemon is hung
DOCKER_TIMEOUT=15
Fallback/fallback.sh
+1 -1
View File
@@ -292,7 +292,7 @@ detect_hosts
resolve_remote_ip
# Validate unRAID notify script — used throughout for state change notifications
validate_unraid_cmd \
platform_require_cmd \
"/usr/local/emhttp/plugins/dynamix/scripts/notify" \
"" "" \
"unRAID notify script" || warn "unRAID notify script not found — native notifications disabled"
Fallback/fallback_test.sh
+2 -2
View File
@@ -166,12 +166,12 @@ detect_hosts
resolve_remote_ip
# Validate commands used by this script
validate_unraid_cmd \
platform_require_cmd \
"$(which iptables 2>/dev/null || echo /sbin/iptables)" \
"--version" "iptables" \
"iptables" || { error "iptables not found — required for connectivity simulation"; exit 1; }
validate_unraid_cmd \
platform_require_cmd \
"/usr/local/emhttp/plugins/dynamix/scripts/notify" \
"" "" \
"unRAID notify script" || warn "unRAID notify script not found — native notifications disabled"
Media/arrs_failed_stalled_recovery.sh
+1 -1
View File
@@ -112,7 +112,7 @@ if ! command -v jq >/dev/null 2>&1; then
fi
log "jq found"
validate_unraid_cmd \
platform_require_cmd \
"/usr/local/emhttp/plugins/dynamix/scripts/notify" \
"" "" \
"unRAID notify script" || warn "unRAID notify script not found — native notifications disabled"
Media/lidarr_cleanup.sh
+2 -2
View File
@@ -43,7 +43,7 @@
# jq + curl validation — exits if either tool missing
# DOCKER_TIMEOUT — container checks protected against daemon hangs
# Duplicate detection — temp file of tracked paths, grep before delete
# validate_unraid_cmd — notify script validated before use
# platform_require_cmd — notify script validated before use
# Silent by default — orphans/junk warn(), clean library logs silently
#
# ==============================================================================================
@@ -153,7 +153,7 @@ if ! command -v jq >/dev/null 2>&1; then
exit 1
fi
validate_unraid_cmd \
platform_require_cmd \
"/usr/local/emhttp/plugins/dynamix/scripts/notify" \
"" "" \
"unRAID notify script" || warn "unRAID notify script not found — native notifications disabled"
Media/media_cleaner.sh
+2 -2
View File
@@ -50,7 +50,7 @@
# detect_hosts() — correct folder lists per host via MY_ID aliases
# Empty array guards — warns and exits cleanly if no folders or patterns configured
# Folder existence — skips missing folders with warning, continues others
# validate_unraid_cmd — notify script validated before use
# platform_require_cmd — notify script validated before use
#
# ==============================================================================================
# CONFIGURATION
@@ -117,7 +117,7 @@ acquire_lock "wait"
# detect_hosts() sets MY_ID and aliases HOST*_ANIME/MEDIA_CLEAN_FOLDERS
detect_hosts
validate_unraid_cmd \
platform_require_cmd \
"/usr/local/emhttp/plugins/dynamix/scripts/notify" \
"" "" \
"unRAID notify script" || warn "unRAID notify script not found — native notifications disabled"
Media/media_shares_permissions.sh
+2 -2
View File
@@ -22,7 +22,7 @@
# Empty array guard — warns and exits cleanly if no shares configured
# Folder existence — skips missing shares with warning, continues others
# Separate passes — directories and files chmod'd separately for correctness
# validate_unraid_cmd — notify script validated before use
# platform_require_cmd — notify script validated before use
# Silent by default — only failures produce output, success is silent
#
# Diagnostic — high corrected count on every run means a container has wrong PUID/PGID:
@@ -70,7 +70,7 @@ if [[ "$EUID" -ne 0 ]]; then
exit 1
fi
validate_unraid_cmd \
platform_require_cmd \
"/usr/local/emhttp/plugins/dynamix/scripts/notify" \
"" "" \
"unRAID notify script" || warn "unRAID notify script not found — native notifications disabled"
Media/radarr_cleanup.sh
+2 -2
View File
@@ -46,7 +46,7 @@
# jq + curl validation — exits if either tool missing
# DOCKER_TIMEOUT — container checks protected against daemon hangs
# notify_emby_scan() — triggers Emby clean after deletion
# validate_unraid_cmd — notify script validated before use
# platform_require_cmd — notify script validated before use
# Silent by default — orphans/junk warn(), clean library logs silently
#
# ==============================================================================================
@@ -141,7 +141,7 @@ if ! command -v jq >/dev/null 2>&1; then
exit 1
fi
validate_unraid_cmd \
platform_require_cmd \
"/usr/local/emhttp/plugins/dynamix/scripts/notify" \
"" "" \
"unRAID notify script" || warn "unRAID notify script not found — native notifications disabled"
Media/sonarr_cleanup.sh
+2 -2
View File
@@ -46,7 +46,7 @@
# jq + curl validation — exits if either tool missing
# DOCKER_TIMEOUT — container checks protected against daemon hangs
# notify_emby_scan() — triggers Emby clean after deletion
# validate_unraid_cmd — notify script validated before use
# platform_require_cmd — notify script validated before use
# Silent by default — orphans/junk warn(), clean library logs silently
#
# ==============================================================================================
@@ -141,7 +141,7 @@ if ! command -v jq >/dev/null 2>&1; then
exit 1
fi
validate_unraid_cmd \
platform_require_cmd \
"/usr/local/emhttp/plugins/dynamix/scripts/notify" \
"" "" \
"unRAID notify script" || warn "unRAID notify script not found — native notifications disabled"
Monitors/Manual-Monitors.md
+3 -3
View File
@@ -285,7 +285,7 @@ higher than 90% — at 100% utilisation new inotify watches silently fail.
**`PHP_FPM_WARN_PCT`**: 80% means the WebGUI is using most of its workers. At 100%
new requests queue (WebGUI feels sluggish) or time out.
**`PHP_MAX_CHILDREN`**: Set by `php_fpm_max_children.sh` in `unRAID_Essentials/` — do
**`PHP_MAX_CHILDREN`**: Set by `php_fpm_max_children.sh` in `System_Essentials/` — do
not set manually here.
### Reading the Weekly Digest Data
@@ -297,7 +297,7 @@ shows for the week:
A few warning snapshots per week is normal. A rising peak or many warnings per week
means the limits should be adjusted — use `inotify_tuning.sh` or `php_fpm_max_children.sh`
in `unRAID_Essentials/`.
in `System_Essentials/`.
---
@@ -566,7 +566,7 @@ SILENT_MODE=false # monitor script — output is the point
parse_args "$@"
# root check if needed
# tool validation (validate_unraid_cmd)
# tool validation (platform_require_cmd)
acquire_lock
detect_hosts # if host-specific config needed
Monitors/README-Monitors.md
+2 -2
View File
@@ -82,13 +82,13 @@ drive SMART attributes, ZFS pool state, ARC statistics, kernel memory pressure.
```
Monitors/ ← observes and reports (this folder)
Docker_Essentials/ ← acts on containers (docker_watchdog starts/stops)
unRAID_Essentials/ ← acts on the server (stability_watchdog, inotify_tuning)
System_Essentials/ ← acts on the server (stability_watchdog, inotify_tuning)
Fallback/ ← acts on the full stack (fallback, handback)
Rsync/ ← calls bandwidth_monitor.sh (auto-logs each sync)
```
`weekly_health_digest.sh` reads state files written by scripts in Docker_Essentials,
unRAID_Essentials, Fallback, and Rsync. It is the only script in this folder with
System_Essentials, Fallback, and Rsync. It is the only script in this folder with
runtime dependencies on other folders' output — everything else is fully independent.
---
Monitors/backup_verify.sh
+2 -2
View File
@@ -57,7 +57,7 @@
# SSH_TIMEOUT caps all SSH calls. One hung connection does not block the run.
#
# Notification Validated
# validate_unraid_cmd confirms the notify script is present before use.
# platform_require_cmd confirms the notify script is present before use.
#
# ==============================================================================================
# CONFIGURATION
@@ -119,7 +119,7 @@ if [[ "$EUID" -ne 0 ]]; then
exit 1
fi
validate_unraid_cmd \
platform_require_cmd \
"/usr/local/emhttp/plugins/dynamix/scripts/notify" \
"" "" \
"unRAID notify script" || warn "unRAID notify script not found — native notifications disabled"
Monitors/bandwidth_monitor.sh
+2 -2
View File
@@ -47,7 +47,7 @@
# Creates the log directory if it doesn't exist. Exits cleanly if unwritable.
#
# Notification Validated
# validate_unraid_cmd confirms the notify script is present before use.
# platform_require_cmd confirms the notify script is present before use.
#
# ==============================================================================================
# CONFIGURATION
@@ -134,7 +134,7 @@ if [[ "$EUID" -ne 0 ]]; then
exit 1
fi
validate_unraid_cmd \
platform_require_cmd \
"/usr/local/emhttp/plugins/dynamix/scripts/notify" \
"" "" \
"unRAID notify script" || warn "unRAID notify script not found — native notifications disabled"
Monitors/cert_monitor.sh
+3 -3
View File
@@ -45,7 +45,7 @@
# does not block the remaining domains.
#
# Notification Validated
# validate_unraid_cmd confirms openssl and notify script are present before use.
# platform_require_cmd confirms openssl and notify script are present before use.
#
# ==============================================================================================
# CONFIGURATION
@@ -105,12 +105,12 @@ if [[ "$EUID" -ne 0 ]]; then
fi
# Validate openssl — required for all cert checks
validate_unraid_cmd \
platform_require_cmd \
"$(command -v openssl 2>/dev/null || echo /usr/bin/openssl)" \
"version" "OpenSSL" \
"openssl" || { error "openssl not found — required for certificate checks"; exit 1; }
validate_unraid_cmd \
platform_require_cmd \
"/usr/local/emhttp/plugins/dynamix/scripts/notify" \
"" "" \
"unRAID notify script" || warn "unRAID notify script not found — native notifications disabled"
Monitors/emby_session_report.sh
+2 -2
View File
@@ -36,7 +36,7 @@
# Each server reports on its own Emby instance automatically.
#
# Notification Validated
# validate_unraid_cmd confirms the notify script is present before use.
# platform_require_cmd confirms the notify script is present before use.
#
# ==============================================================================================
# CONFIGURATION
@@ -111,7 +111,7 @@ if ! command -v jq >/dev/null 2>&1; then
exit 1
fi
validate_unraid_cmd \
platform_require_cmd \
"/usr/local/emhttp/plugins/dynamix/scripts/notify" \
"" "" \
"unRAID notify script" || warn "unRAID notify script not found — native notifications disabled"
Monitors/smart_health.sh
+3 -3
View File
@@ -37,7 +37,7 @@
# dashboard use the same thresholds. Falls back to master.conf values if not found.
#
# Notifications Validated
# validate_unraid_cmd confirms smartctl and notify script are present before use.
# platform_require_cmd confirms smartctl and notify script are present before use.
#
# ==============================================================================================
# CONFIGURATION
@@ -94,7 +94,7 @@ if [[ "$EUID" -ne 0 ]]; then
fi
# Validate smartctl — required for all drive checks
validate_unraid_cmd \
platform_require_cmd \
"$(command -v smartctl 2>/dev/null || echo /usr/bin/smartctl)" \
"--version" "smartmontools" \
"smartctl" || {
@@ -104,7 +104,7 @@ validate_unraid_cmd \
exit 1
}
validate_unraid_cmd \
platform_require_cmd \
"/usr/local/emhttp/plugins/dynamix/scripts/notify" \
"" "" \
"unRAID notify script" || warn "unRAID notify script not found — native notifications disabled"
Monitors/system_tuning_monitor.sh
+3 -3
View File
@@ -48,7 +48,7 @@
# the accumulated history.
#
# Notification Validated
# validate_unraid_cmd confirms the notify script is present before use.
# platform_require_cmd confirms the notify script is present before use.
#
# ==============================================================================================
# STATE FILES
@@ -72,7 +72,7 @@
# Warn if php-fpm active workers exceed this percentage of PHP_MAX_CHILDREN. (default: 80)
#
# PHP_MAX_CHILDREN
# Maximum php-fpm workers — set by php_fpm_max_children.sh in unRAID_Essentials.
# Maximum php-fpm workers — set by php_fpm_max_children.sh in System_Essentials.
#
# TUNING_MONITOR_LOG
# Log file path. (default: DATA_DIR/tuning_monitor.db)
@@ -113,7 +113,7 @@ if [[ "$EUID" -ne 0 ]]; then
exit 1
fi
validate_unraid_cmd \
platform_require_cmd \
"/usr/local/emhttp/plugins/dynamix/scripts/notify" \
"" "" \
"unRAID notify script" || warn "unRAID notify script not found — native notifications disabled"
Monitors/weekly_health_digest.sh
+3 -3
View File
@@ -59,7 +59,7 @@
# reporting is found.
#
# Notifications Validated
# validate_unraid_cmd confirms notify and openssl are present before use.
# platform_require_cmd confirms notify and openssl are present before use.
#
# ==============================================================================================
# CONFIGURATION
@@ -126,12 +126,12 @@ if [[ "$EUID" -ne 0 ]]; then
exit 1
fi
validate_unraid_cmd \
platform_require_cmd \
"/usr/local/emhttp/plugins/dynamix/scripts/notify" \
"" "" \
"unRAID notify script" || warn "unRAID notify script not found — native notifications disabled"
validate_unraid_cmd \
platform_require_cmd \
"$(command -v openssl 2>/dev/null || echo /usr/bin/openssl)" \
"version" "OpenSSL" \
"openssl" || warn "openssl not found — SSL cert checks will be skipped"
Monitors/zfs_memory_snapshot.sh
+2 -2
View File
@@ -52,7 +52,7 @@
# DOCKER_TIMEOUT caps docker stats calls. A hung daemon does not block the report.
#
# Notification Validated
# validate_unraid_cmd confirms the notify script is present before use.
# platform_require_cmd confirms the notify script is present before use.
#
# ==============================================================================================
# STATE FILES
@@ -125,7 +125,7 @@ if [[ "$EUID" -ne 0 ]]; then
exit 1
fi
validate_unraid_cmd \
platform_require_cmd \
"/usr/local/emhttp/plugins/dynamix/scripts/notify" \
"" "" \
"unRAID notify script" || warn "unRAID notify script not found — native notifications disabled"
Orchestrators/README-Orchestrators.md
+4 -4
View File
@@ -205,16 +205,16 @@ background process — and reports which succeeded and which failed.
#
ARRAY_START_SCRIPTS=(
# ── One-shot scripts — run and exit naturally ─────────────────────────────
"unRAID_Essentials/unraid_api_key_renew.sh" # re-register API key FIRST — unraid-api
"System_Essentials/unraid_api_key_renew.sh" # re-register API key FIRST — unraid-api
# registry is ephemeral, lost on service restart
"unRAID_Essentials/inotify_tuning.sh" # raise inotify BEFORE containers start
"System_Essentials/inotify_tuning.sh" # raise inotify BEFORE containers start
# containers inherit limits at startup —
# if Code-Server starts with low limits
# it keeps them until restart
"unRAID_Essentials/docker_syslog_filter.sh" # suppress veth noise BEFORE containers create
"System_Essentials/docker_syslog_filter.sh" # suppress veth noise BEFORE containers create
# veth interfaces — otherwise the first boot
# always has unfiltered veth spam
"unRAID_Essentials/php_fpm_max_children.sh" # WebGUI tuning — before any WebGUI requests
"System_Essentials/php_fpm_max_children.sh" # WebGUI tuning — before any WebGUI requests
"Transcodes/ramdisk_setup.sh" # create tmpfs + symlink BEFORE Emby starts —
# Emby needs the transcode path to exist
"Docker_Essentials/docker_network_connect.sh" # ensure networks + connections BEFORE
Orchestrators/array_started.sh
+6 -6
View File
@@ -12,10 +12,10 @@
# Current order (order matters — see below):
#
# ONE-SHOT (run and exit naturally):
# unRAID_Essentials/unraid_api_key_renew.sh — re-register Varaverk API key at boot
# unRAID_Essentials/inotify_tuning.sh — raise inotify limits before containers start
# unRAID_Essentials/docker_syslog_filter.sh — suppress veth log noise before logs fill
# unRAID_Essentials/php_fpm_max_children.sh — WebGUI performance tuning
# System_Essentials/unraid_api_key_renew.sh — re-register Varaverk API key at boot
# System_Essentials/inotify_tuning.sh — raise inotify limits before containers start
# System_Essentials/docker_syslog_filter.sh — suppress veth log noise before logs fill
# System_Essentials/php_fpm_max_children.sh — WebGUI performance tuning
# Transcodes/ramdisk_setup.sh — create tmpfs + symlink before Emby starts
# Docker_Essentials/docker_network_connect.sh — ensure networks + container connections
#
@@ -44,7 +44,7 @@
# Root check — all launched scripts require root
# acquire_lock — prevents duplicate array start launches
# detect_hosts() — MY_ID in notifications
# validate_unraid_cmd — notify validated before use
# platform_require_cmd — notify validated before use
# chmod +x auto-fix — non-executable scripts fixed before launch
# Full path on failure — shows exact path for debugging
# notify on failures — alert if any script fails to launch
@@ -74,7 +74,7 @@ if [[ "$EUID" -ne 0 ]]; then
exit 1
fi
validate_unraid_cmd \
platform_require_cmd \
"/usr/local/emhttp/plugins/dynamix/scripts/notify" \
"" "" \
"unRAID notify script" || warn "unRAID notify script not found — native notifications disabled"
Orchestrators/array_stopping.sh
+2 -2
View File
@@ -24,7 +24,7 @@
# Root check — all stop scripts require root
# acquire_lock — prevents concurrent array stop runs
# detect_hosts() — MY_ID in notifications and logs
# validate_unraid_cmd — notify validated before use
# platform_require_cmd — notify validated before use
# Non-fatal steps — a failed step is logged but remaining steps still run
# notify on failures — alert if any stop script fails
#
@@ -53,7 +53,7 @@ if [[ "$EUID" -ne 0 ]]; then
exit 1
fi
validate_unraid_cmd \
platform_require_cmd \
"/usr/local/emhttp/plugins/dynamix/scripts/notify" \
"" "" \
"unRAID notify script" || warn "unRAID notify script not found — native notifications disabled"
Orchestrators/critical_sync_maintenance.sh
+1 -1
View File
@@ -59,7 +59,7 @@ if [[ "$EUID" -ne 0 ]]; then
exit 1
fi
validate_unraid_cmd \
platform_require_cmd \
"/usr/local/emhttp/plugins/dynamix/scripts/notify" \
"" "" \
"unRAID notify script" || warn "unRAID notify script not found — native notifications disabled"
Orchestrators/daily_sync_maintenance.sh
+1 -1
View File
@@ -83,7 +83,7 @@ if [[ "$EUID" -ne 0 ]]; then
exit 1
fi
validate_unraid_cmd \
platform_require_cmd \
"/usr/local/emhttp/plugins/dynamix/scripts/notify" \
"" "" \
"unRAID notify script" || warn "unRAID notify script not found — native notifications disabled"
Orchestrators/intermediate_sync_maintenance.sh
+1 -1
View File
@@ -70,7 +70,7 @@ if [[ "$EUID" -ne 0 ]]; then
exit 1
fi
validate_unraid_cmd \
platform_require_cmd \
"/usr/local/emhttp/plugins/dynamix/scripts/notify" \
"" "" \
"unRAID notify script" || warn "unRAID notify script not found — native notifications disabled"
Orchestrators/watchdog_orchestrator.sh
+1 -1
View File
@@ -121,7 +121,7 @@ fi
# ==============================================================================================
# ━━━ Array Check ━━━
# ==============================================================================================
if ! df --output=fstype /mnt/user 2>/dev/null | grep -q shfs; then
if ! platform_storage_healthy; then
echo "Array not started — skipping watchdog cycle"
exit 0
fi
Orchestrators/weekly_sync_maintenance.sh
+2 -2
View File
@@ -39,7 +39,7 @@
# check_remote_rootfs — aborts if remote rootfs nearly full
# DOCKER_TIMEOUT — all docker calls protected
# SSH_TIMEOUT — all SSH calls protected
# validate_unraid_cmd — notify validated before use
# platform_require_cmd — notify validated before use
# Silent on success — runs weekly, only failures warrant notification
#
# ── CONFIGURATION (master.conf) ───────────────────────────────────────────────────────────────
@@ -76,7 +76,7 @@ if [[ "$EUID" -ne 0 ]]; then
exit 1
fi
validate_unraid_cmd \
platform_require_cmd \
"/usr/local/emhttp/plugins/dynamix/scripts/notify" \
"" "" \
"unRAID notify script" || warn "unRAID notify script not found — native notifications disabled"
Partnership/gitea_ssh_setup.sh
+1 -1
View File
@@ -135,7 +135,7 @@ if [[ "$EUID" -ne 0 ]]; then
exit 1
fi
validate_unraid_cmd \
platform_require_cmd \
"/usr/local/emhttp/plugins/dynamix/scripts/notify" \
"" "" \
"unRAID notify script" || warn "unRAID notify script not found — native notifications disabled"
Partnership/partnership_manager.sh
+1 -1
View File
@@ -235,7 +235,7 @@ if [[ "$EUID" -ne 0 ]]; then
exit 1
fi
validate_unraid_cmd \
platform_require_cmd \
"/usr/local/emhttp/plugins/dynamix/scripts/notify" \
"" "" \
"unRAID notify script" || warn "unRAID notify script not found — native notifications disabled"
Partnership/partnership_transfer.sh
+1 -1
View File
@@ -126,7 +126,7 @@ if ! command -v docker &>/dev/null; then
exit 1
fi
validate_unraid_cmd \
platform_require_cmd \
"/usr/local/emhttp/plugins/dynamix/scripts/notify" \
"" "" \
"unRAID notify script" || warn "unRAID notify script not found — native notifications disabled"
Partnership/ssh_setup.sh
+1 -1
View File
@@ -68,7 +68,7 @@ fi
acquire_lock
validate_unraid_cmd \
platform_require_cmd \
"/usr/local/emhttp/plugins/dynamix/scripts/notify" \
"" "" \
"unRAID notify script" || warn "unRAID notify script not found — native notifications disabled"
Plugin/README-Plugin.md
+37 -1
View File
@@ -33,7 +33,8 @@ no separate monitoring stack, no third-party dashboards.
Plugin/
├── dev_install.sh # One-time developer setup: symlinks plugin into web server
├── Icons/ # Source icon assets (1024px master files)
└── unraid/ # The Unraid plugin application
└── unraid/ # The Unraid platform adapter + plugin application
├── adapter.sh # Platform adapter — provides platform_*() API to all scripts
├── Varaverk.page # Main plugin entry point (Tasks menu)
├── VaraverkSettings.page # Unraid Settings → Other Settings entry
├── api/ # PHP API endpoints (called by JS via fetch)
@@ -45,6 +46,10 @@ Plugin/
├── js/ # Frontend JavaScript
├── pages/ # Per-tab page includes (monitor, scheduler, docker, ...)
└── run_job.sh # Script runner invoked by the Scheduler
# Future platform adapters follow the same structure:
# Plugin/truenas/adapter.sh — TrueNAS adapter (future)
# Plugin/ubuntu/adapter.sh — Ubuntu/Debian adapter (future)
```
---
@@ -74,6 +79,37 @@ so behaviour stays consistent without a shell dependency.
---
## ━━━ THE PLATFORM ADAPTER ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
`Plugin/unraid/adapter.sh` is the Unraid platform adapter. It is sourced automatically
by `load_config.sh` whenever `PLATFORM=unraid` is detected (via `/etc/unraid-version`).
Every bash script in the ecosystem calls `platform_*()` functions instead of OS-specific
commands directly. The adapter translates those calls into Unraid-specific implementations.
```
platform_storage_healthy # is the array up and shfs mounted?
platform_is_maintenance_running # parity check or sync in progress?
platform_is_service_running # is a named service process alive?
platform_restart_service # restart via rc.d (Unraid) or systemctl (future)
platform_stop_service # stop a named service
platform_is_mover_running # Unraid mover active?
platform_get_mover_pid # PID of the mover process
platform_stop_user_scripts # kill Unraid user.scripts background jobs
platform_send_os_notification # dynamix notify (Unraid) or equivalent
platform_get_disk_states # reads disks.ini (Unraid) or equivalent
platform_get_temp_thresholds # reads dynamix.cfg (Unraid) or equivalent
platform_is_service_enabled # docker.cfg / domain.cfg enabled check
platform_require_cmd # verify a platform command exists
```
**Adding a new platform:** Create `Plugin/<platform>/adapter.sh` implementing the same
function names. `load_config.sh` detects the OS at runtime and sources the correct adapter.
No other files need changing.
---
## ━━━ UNRAID INTEGRATION POINTS ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
| File | Where it appears in Unraid |
Plugin/unraid/adapter.sh
Executable
+211
View File
@@ -0,0 +1,211 @@
#!/bin/bash
# ==============================================================================================
# ================================= Unraid Platform Adapter ====================================
# ==============================================================================================
# Sourced by load_config.sh when PLATFORM=unraid.
# Provides the platform_*() API — bash scripts call these instead of OS-specific commands.
#
# ── API CONTRACT ──────────────────────────────────────────────────────────────────────────────
# Every function returns 0 on success / 1 on failure unless noted.
# Functions that produce output write to stdout; callers capture with $().
# No function calls exit — callers decide what failure means for their flow.
#
# ── ADDING A PLATFORM ─────────────────────────────────────────────────────────────────────────
# Create Plugin/truenas/adapter.sh (or ubuntu/adapter.sh) implementing the same function names.
# load_config.sh sources Plugin/$PLATFORM/adapter.sh — no other changes needed.
#
# ── FUNCTIONS ─────────────────────────────────────────────────────────────────────────────────
# platform_require_cmd — verify a platform command exists and is executable
# platform_storage_healthy — array mounted and shfs active on /mnt/user
# platform_get_disk_states — raw disks.ini content from emhttp state
# platform_get_temp_thresholds — disk warn/crit °C from dynamix.cfg
# platform_is_maintenance_running — parity check/sync in progress
# platform_is_service_enabled — docker or libvirt enabled in boot config
# platform_restart_service — restart a named service via rc.d
# platform_stop_service — stop a named service via rc.d
# platform_is_service_running — check if a named service process is alive
# platform_is_mover_running — unRAID mover process check
# platform_stop_user_scripts — kill all user.scripts background processes
# platform_send_os_notification — native unRAID notify (dynamix)
# ==============================================================================================
# ──────────────────────────────────────────────────────────────────────────────────────────────
# Internal: map a logical service name → its rc.d script path
# ──────────────────────────────────────────────────────────────────────────────────────────────
_platform_rc_script() {
case "$1" in
docker) echo "/etc/rc.d/rc.docker" ;;
sshd) echo "/etc/rc.d/rc.sshd" ;;
libvirt) echo "/etc/rc.d/rc.libvirt" ;;
rsyslog) echo "/etc/rc.d/rc.rsyslogd" ;;
nginx) echo "/etc/rc.d/rc.nginx" ;;
php-fpm) echo "/etc/rc.d/rc.php-fpm" ;;
emhttp) echo "/etc/rc.d/rc.emhttp" ;;
*) echo "/etc/rc.d/rc.$1" ;;
esac
}
# ──────────────────────────────────────────────────────────────────────────────────────────────
# platform_require_cmd <path> [test_arg] [expected_output] [label]
# Replaces platform_require_cmd. Returns 0 if the command exists and (optionally) its output
# matches expected_output. Returns 1 and prints a warning if not.
# ──────────────────────────────────────────────────────────────────────────────────────────────
platform_require_cmd() {
local cmd="$1"
local test_arg="${2:-}"
local expected="${3:-}"
local label="${4:-$cmd}"
if [[ ! -x "$cmd" ]]; then
return 1
fi
if [[ -n "$test_arg" && -n "$expected" ]]; then
"$cmd" $test_arg 2>&1 | grep -q "$expected" || return 1
fi
return 0
}
# ──────────────────────────────────────────────────────────────────────────────────────────────
# platform_storage_healthy
# Returns 0 if /mnt/user is mounted as shfs (array is up and healthy).
# Returns 1 if the mount is absent or is not shfs (array stopped / degraded).
# ──────────────────────────────────────────────────────────────────────────────────────────────
platform_storage_healthy() {
df --output=fstype /mnt/user 2>/dev/null | grep -q shfs
}
# ──────────────────────────────────────────────────────────────────────────────────────────────
# platform_get_disk_states
# Writes raw /var/local/emhttp/disks.ini to stdout.
# Returns 1 if the file is absent (array not started or emhttp not running).
# ──────────────────────────────────────────────────────────────────────────────────────────────
platform_get_disk_states() {
local disks_ini="/var/local/emhttp/disks.ini"
[[ -f "$disks_ini" ]] || return 1
cat "$disks_ini"
}
# ──────────────────────────────────────────────────────────────────────────────────────────────
# platform_get_temp_thresholds
# Writes two space-separated values to stdout: WARN_TEMP CRIT_TEMP (°C integers).
# Falls back to 45 55 if dynamix.cfg is absent or the keys are missing.
# ──────────────────────────────────────────────────────────────────────────────────────────────
platform_get_temp_thresholds() {
local cfg="/boot/config/plugins/dynamix/dynamix.cfg"
local warn crit
warn=$(grep -m1 '^diskWarn=' "$cfg" 2>/dev/null | cut -d= -f2 | tr -d '"')
crit=$(grep -m1 '^diskCrit=' "$cfg" 2>/dev/null | cut -d= -f2 | tr -d '"')
echo "${warn:-45} ${crit:-55}"
}
# ──────────────────────────────────────────────────────────────────────────────────────────────
# platform_is_maintenance_running
# Returns 0 if a parity check or sync is currently in progress.
# Checks var.ini (Unraid 7.3+) then falls back to parity-date.txt (older).
# ──────────────────────────────────────────────────────────────────────────────────────────────
platform_is_maintenance_running() {
local resync
resync=$(awk -F'"' '/^mdResync=/{print $2}' /var/local/emhttp/var.ini 2>/dev/null)
if [[ -n "$resync" && "$resync" != "0" ]]; then
return 0
fi
grep -q "progress" /var/local/emhttp/parity-date.txt 2>/dev/null
}
# ──────────────────────────────────────────────────────────────────────────────────────────────
# platform_is_service_enabled <service>
# service: docker | libvirt
# Returns 0 if the service is enabled in the Unraid boot config.
# ──────────────────────────────────────────────────────────────────────────────────────────────
platform_is_service_enabled() {
case "$1" in
docker) grep -q '^DOCKER_ENABLED=yes' /boot/config/docker.cfg 2>/dev/null ;;
libvirt) grep -q '^DOMAIN_ENABLE=yes' /boot/config/domain.cfg 2>/dev/null ;;
*) return 1 ;;
esac
}
# ──────────────────────────────────────────────────────────────────────────────────────────────
# platform_restart_service <service>
# Restarts the named service via its rc.d script.
# Returns 1 if the rc.d script does not exist or is not executable.
# ──────────────────────────────────────────────────────────────────────────────────────────────
platform_restart_service() {
local rc
rc=$(_platform_rc_script "$1")
[[ -x "$rc" ]] || return 1
"$rc" restart >/dev/null 2>&1
}
# ──────────────────────────────────────────────────────────────────────────────────────────────
# platform_stop_service <service>
# Stops the named service via its rc.d script.
# Returns 1 if the rc.d script does not exist or is not executable.
# ──────────────────────────────────────────────────────────────────────────────────────────────
platform_stop_service() {
local rc
rc=$(_platform_rc_script "$1")
[[ -x "$rc" ]] || return 1
"$rc" stop >/dev/null 2>&1
}
# ──────────────────────────────────────────────────────────────────────────────────────────────
# platform_is_service_running <service>
# Returns 0 if the service's main process is alive.
# ──────────────────────────────────────────────────────────────────────────────────────────────
platform_is_service_running() {
case "$1" in
docker) pgrep -x dockerd >/dev/null 2>&1 ;;
emhttp) pgrep -x emhttpd >/dev/null 2>&1 ;;
nginx) pgrep -x nginx >/dev/null 2>&1 ;;
php-fpm) pgrep -x php-fpm >/dev/null 2>&1 ;;
sshd) pgrep -x sshd >/dev/null 2>&1 ;;
rsyslog) pgrep -x rsyslogd >/dev/null 2>&1 ;;
libvirt) pgrep -x libvirtd >/dev/null 2>&1 ;;
*) pgrep -x "$1" >/dev/null 2>&1 ;;
esac
}
# ──────────────────────────────────────────────────────────────────────────────────────────────
# platform_is_mover_running
# Returns 0 if the Unraid mover is currently active.
# ──────────────────────────────────────────────────────────────────────────────────────────────
platform_is_mover_running() {
pgrep -f "emhttp.*Mover" >/dev/null 2>&1
}
# ──────────────────────────────────────────────────────────────────────────────────────────────
# platform_get_mover_pid
# Writes the mover's PID to stdout. Returns 1 if the mover is not running.
# ──────────────────────────────────────────────────────────────────────────────────────────────
platform_get_mover_pid() {
local pid
pid=$(pgrep -f "emhttp.*Mover" | head -1)
[[ -n "$pid" ]] || return 1
echo "$pid"
}
# ──────────────────────────────────────────────────────────────────────────────────────────────
# platform_stop_user_scripts
# Kills all Unraid user.scripts background processes.
# Returns 0 whether or not any processes were found (pkill exits 1 on no match).
# ──────────────────────────────────────────────────────────────────────────────────────────────
platform_stop_user_scripts() {
pkill -f "/tmp/user.scripts" 2>/dev/null || true
}
# ──────────────────────────────────────────────────────────────────────────────────────────────
# platform_send_os_notification <message> [subject] [severity]
# Sends a native Unraid notification via the dynamix notify script.
# severity: normal | warning | alert (default: normal)
# Returns 1 if the notify script is absent.
# ──────────────────────────────────────────────────────────────────────────────────────────────
platform_send_os_notification() {
local message="$1"
local subject="${2:-Varaverk}"
local severity="${3:-normal}"
local notify_script="/usr/local/emhttp/plugins/dynamix/scripts/notify"
[[ -x "$notify_script" ]] || return 1
"$notify_script" -e "Varaverk" -s "$subject" -d "$message" -i "$severity" 2>/dev/null
}
Plugin/unraid/api/storage.php
+1 -1
View File
@@ -141,7 +141,7 @@ if ($action === 'api_status') {
// ── Setup/renew API keys (local + all partners via SSH) ───────────────────────
if ($action === 'setup_apikeys' && $_SERVER['REQUEST_METHOD'] === 'POST') {
$script = SCRIPTS_DIR . '/unRAID_Essentials/unraid_api_key_renew.sh';
$script = SCRIPTS_DIR . '/System_Essentials/unraid_api_key_renew.sh';
if (!file_exists($script)) {
echo json_encode(['ok' => false, 'error' => 'unraid_api_key_renew.sh not found']); exit;
}
README-User_Script_Plug-in.md
+14 -14
View File
@@ -72,7 +72,7 @@ Rsync — manual per-profile sync triggers
Docker_Essentials — container management tools (manual use)
Media — library maintenance tools (manual use)
Transcodes — ramdisk and transcode tools (manual use)
unRAID_Essentials — server management tools (manual use)
System_Essentials — server management tools (manual use)
Partnership — two-server relationship tools (manual use)
Tools — repair, recovery, and one-time utilities
Git — script deployment
@@ -1190,7 +1190,7 @@ flip count, session split ramdisk vs SSD).
### ── webgui_watchdog ───────────────────────────────────────────────────────────
```
Category: unRAID_Essentials
Category: System_Essentials
Name: webgui_watchdog
Schedule: */10 * * * * (every 10 minutes)
Background: YES
@@ -1221,7 +1221,7 @@ A notification fires on any restart so you know what recovered and when.
### ── clear_logs ───────────────────────────────────────────────────────────────
```
Category: unRAID_Essentials
Category: System_Essentials
Name: clear_logs
Schedule: 0 5 * * 0 (Sunday 5am — before Sunday maintenance window)
Background: YES
@@ -1229,7 +1229,7 @@ Background: YES
```bash
#!/bin/bash
/boot/config/plugins/varaverk/unRAID_Essentials/clear_logs.sh
/boot/config/plugins/varaverk/System_Essentials/clear_logs.sh
```
**What it does:**
@@ -1248,7 +1248,7 @@ Active containers like Emby and SABnzbd grow fastest. Truncation not rotation
### ── stability_watchdog_status ──────────────────────────────────────────────────
```
Category: unRAID_Essentials
Category: System_Essentials
Name: stability_watchdog_status
Schedule: None — run manually
Background: NO
@@ -1269,7 +1269,7 @@ watchdog-triggered reboots. All 18 per-host toggle states.
### ── mover_stop ───────────────────────────────────────────────────────────────
```
Category: unRAID_Essentials
Category: System_Essentials
Name: mover_stop
Schedule: None — run manually (before array operations needing mover stopped)
Background: NO
@@ -1277,7 +1277,7 @@ Background: NO
```bash
#!/bin/bash
/boot/config/plugins/varaverk/unRAID_Essentials/mover_stop.sh
/boot/config/plugins/varaverk/System_Essentials/mover_stop.sh
```
**What it does:**
@@ -1290,7 +1290,7 @@ files). Waits + verifies. SIGKILL if still running (warns that partial files are
### ── rsync_stop ────────────────────────────────────────────────────────────────
```
Category: unRAID_Essentials
Category: System_Essentials
Name: rsync_stop
Schedule: None — run manually (when rsync needs stopping)
Background: NO
@@ -1298,7 +1298,7 @@ Background: NO
```bash
#!/bin/bash
/boot/config/plugins/varaverk/unRAID_Essentials/rsync_stop.sh --status
/boot/config/plugins/varaverk/System_Essentials/rsync_stop.sh --status
```
**What it does:**
@@ -1312,7 +1312,7 @@ exits cleanly. No orphaned lock files. Containers recover via docker_watchdog.sh
### ── user_scripts_stop ────────────────────────────────────────────────────────
```
Category: unRAID_Essentials
Category: System_Essentials
Name: user_scripts_stop
Schedule: None — run manually (before reboots, or when a script hangs)
Background: NO
@@ -1320,7 +1320,7 @@ Background: NO
```bash
#!/bin/bash
/boot/config/plugins/varaverk/unRAID_Essentials/user_scripts_stop.sh --status
/boot/config/plugins/varaverk/System_Essentials/user_scripts_stop.sh --status
```
**What it does:**
@@ -1334,7 +1334,7 @@ by server_reboot.sh before reboot.
### ── server_reboot ────────────────────────────────────────────────────────────
```
Category: unRAID_Essentials
Category: System_Essentials
Name: server_reboot
Schedule: None — run manually (planned maintenance reboots)
Background: YES — sequence takes 1-2 minutes
@@ -1342,7 +1342,7 @@ Background: YES — sequence takes 1-2 minutes
```bash
#!/bin/bash
/boot/config/plugins/varaverk/unRAID_Essentials/server_reboot.sh --dry-run
/boot/config/plugins/varaverk/System_Essentials/server_reboot.sh --dry-run
```
**What it does:**
@@ -1587,7 +1587,7 @@ Background: NO
```bash
#!/bin/bash
/boot/config/plugins/varaverk/unRAID_Essentials/git_pull_execute.sh
/boot/config/plugins/varaverk/System_Essentials/git_pull_execute.sh
```
**What it does:**
README.md
+51 -14
View File
@@ -2,18 +2,24 @@
# 🏠 VARAVERK
# ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
**A complete self-healing, self-maintaining, mutually-redundant two-server unRAID
**A complete self-healing, self-maintaining, mutually-redundant two-server home server
ecosystem.** One codebase runs on both servers. Both servers run their own lives
independently. When one goes down the other covers it — not because of conventional
fallback infrastructure, but because both servers already have copies of everything
that matters and both know how to run it.
> **This is not conventional failover.** There is no primary and standby. There is no
> shared storage. There is no cluster manager. There are two fully independent unRAID
> servers, living in different locations on different power utilities, each running
> their own household's media stack — and a layer of scripts that keeps them
> synchronised, coordinated, and mutually covering. The goal is minimal disruption to
> a media stack, not enterprise high availability.
> shared storage. There is no cluster manager. There are two fully independent servers,
> living in different locations on different power utilities, each running their own
> household's media stack — and a layer of scripts that keeps them synchronised,
> coordinated, and mutually covering. The goal is minimal disruption to a media stack,
> not enterprise high availability.
**Started on Unraid, built for portability.** The ecosystem runs on Unraid today. The
platform adapter layer (`Plugin/unraid/adapter.sh`) isolates every OS-specific call so
bash scripts stay clean. A `Plugin/truenas/adapter.sh` or `Plugin/ubuntu/adapter.sh`
provides the same function names — `load_config.sh` detects the OS and sources the
correct adapter. The scripts themselves never branch on the OS.
---
@@ -481,7 +487,15 @@ varaverk/
│ ├── host1.conf ← HOST1-specific: share lists, container names, API keys
│ └── host2.conf ← HOST2-specific: same structure, different values
├── common.sh ← Shared library — all functions used by every script
├── load_config.sh ← Sources Configurations/ and common.sh at startup
├── load_config.sh ← Detects OS platform → sources Configurations/, common.sh, adapter
│
├── Plugin/ ← Platform adapters + Unraid web UI
│ └── unraid/
│ ├── adapter.sh ← Unraid adapter: platform_*() functions (rc.d, emhttp, dynamix)
│ ├── api/ ← PHP API endpoints
│ ├── include/ ← PHP business logic
│ ├── pages/ ← Per-tab page includes
│ └── ... ← Full web UI (scheduler, monitor, docker, arrs, fallback)
│
├── Orchestrators/ ← Scheduled job runners — what Varaverk scheduler executes
│ README: README-Orchestrators.md
@@ -498,8 +512,8 @@ varaverk/
├── Docker_Essentials/ ← Container lifecycle: restarts, updates, networks
│ README: README-Docker_Essentials.md
│
├── unRAID_Essentials/ ← Server-level: WebGUI, log hygiene, kernel tuning
│ README: README-Unraid_Essentials.md
├── System_Essentials/ ← Server-level: WebGUI recovery, log hygiene, kernel tuning, reboot
│ README: README-System_Essentials.md
│
├── Media/ ← Library health + behavior-driven discovery: permissions, junk cleanup, orphan removal, weekly arr adds
│ README: README-Media.md
@@ -529,6 +543,13 @@ varaverk/
# All three conf files live in Configurations/. load_config.sh sources them.
# Sparse checkout ensures each server only receives its own host*.conf.
#
# load_config.sh does five things in order:
# 1. Detects OS platform (/etc/unraid-version → PLATFORM=unraid)
# 2. Sources master.conf
# 3. Auto-discovers and sources all host*.conf present
# 4. Sources common.sh
# 5. Sources Plugin/$PLATFORM/adapter.sh (platform_*() functions)
#
# master.conf — shared across both servers
# Everything that applies equally to both: thresholds, schedules, profile
# definitions, watchdog settings, arr cleanup config, DDNS timing, etc.
@@ -547,6 +568,11 @@ varaverk/
# Aliases HOST2_DAILY_SYNC_SHARES → DAILY_SYNC_SHARES (if on HOST2)
# Every script uses the unprefixed name. The same script on either server
# automatically uses the correct values. No hostname comparisons in scripts.
#
# Plugin/$PLATFORM/adapter.sh — OS-specific function implementations
# Scripts call platform_restart_service(), platform_storage_healthy(), etc.
# The adapter maps those to rc.d scripts, emhttp commands, and dynamix files
# on Unraid — or to systemctl, mount checks, etc. on a future platform.
# ─────────────────────────────────────────────────────────────────────────────
```
@@ -667,8 +693,8 @@ Problem: RAM drops to 3GB
→ If RAM drops to 4GB AND 3+ OOM kills: bypass strikes → reboot
Problem: Docker daemon hung
→ stability_watchdog.sh Tier 1: attempt /etc/rc.d/rc.docker restart
→ If still hung: immediate reboot (no strikes needed — daemon can't be managed)
→ docker_watchdog.sh: attempt platform_restart_service docker (rc.d on Unraid)
→ If still hung: sets daemon_confirmed_down flag → stability_watchdog escalates → reboot
Problem: HOST1 loses power
→ fallback.sh on HOST2 detects at next 2-minute check
@@ -677,7 +703,7 @@ Problem: HOST1 loses power
```
**→ Container healing: [README-Docker_Essentials.md](Docker_Essentials/README-Docker_Essentials.md)**
**→ System stability: [README-Unraid_Essentials.md](unRAID_Essentials/README-Unraid_Essentials.md)**
**→ System stability: [README-System_Essentials.md](System_Essentials/README-System_Essentials.md)**
---
@@ -795,15 +821,15 @@ If you're setting this up from scratch on two servers:
|--------|---------------|
| [README-Fallback.md](Fallback/README-Fallback.md) | DDNS sequencing, tiered fallback, handback, split brain prevention, fallback_test.sh |
| [README-Docker_Essentials.md](Docker_Essentials/README-Docker_Essentials.md) | Two-tier container watchdog, memory limits, dependency ordering, skip list, daily/weekly restarts |
| [README-Unraid_Essentials.md](unRAID_Essentials/README-Unraid_Essentials.md) | Three-tier system watchdog, WebGUI recovery, inotify tuning, PHP-FPM, log hygiene |
| [README-System_Essentials.md](System_Essentials/README-System_Essentials.md) | WebGUI recovery, inotify tuning, PHP-FPM, log hygiene, mover/reboot/rsync stop |
| [README-Orchestrators.md](Orchestrators/README-Orchestrators.md) | What runs when, execution order, daily/weekly windows, adding jobs |
| [README-Media.md](Media/README-Media.md) | Permissions model, junk cleanup, arr orphan removal, safety layers, behavior-driven discovery, testing procedure |
| [README-Transcoding.md](Transcodes/README-Transcoding.md) | Ramdisk design, symlink architecture, Docker mount requirement, SSD fallback |
| [README-Monitors.md](Monitors/README-Monitors.md) | Cert monitoring, SMART health, bandwidth tracking, health digest profiles |
| [README-Partnership.md](Partnership/README-Partnership.md) | Auth stack sharing, onboard/offboard/transfer lifecycle, deferred offboard |
| [README-Plugin.md](Plugin/README-Plugin.md) | Platform adapter API, Unraid web UI, scheduler/monitor/docker pages |
| [Manual.md](Manual.md) | Complete setup guide — Tailscale, SSH keys, git clone, master.conf, Varaverk scheduler |
| [README-Tools.md](Tools/README-Tools.md) | fallback_state_reset, skip list manager, emby_database_repair, container export |
| [README-Unraid_Essentials.md](unRAID_Essentials/README-Unraid_Essentials.md) | `git_pull_execute.sh`, `server_reboot.sh`, `mover_stop.sh`, `user_scripts_stop.sh` |
---
@@ -848,6 +874,17 @@ Things that are different from what you might expect:
# lidarr_cleanup.sh, sonarr_cleanup.sh, radarr_cleanup.sh permanently delete
# orphaned files. Safety layers prevent catastrophic runs but always test with
# --dry-run --log first on a new system. README-Media.md has the full procedure.
# 9. Scripts never call OS commands directly.
# All platform-specific operations go through plugin_*() functions defined in
# Plugin/$PLATFORM/adapter.sh. rc.d scripts, emhttp, dynamix, disks.ini — all
# isolated in the adapter. This is what makes the codebase portable.
# Adding a new OS = writing one adapter file, not patching dozens of scripts.
# 10. System_Essentials/ is platform-agnostic at the script level.
# Scripts like server_reboot.sh, mover_stop.sh, and webgui_watchdog.sh call
# adapter functions (platform_is_mover_running, platform_restart_service, etc.)
# The adapter handles what those mean on the actual OS. The scripts are clean.
# ─────────────────────────────────────────────────────────────────────────────
```
Rsync/rsync.sh
+2 -2
View File
@@ -73,7 +73,7 @@
# Global concurrent limit prevents too many simultaneous rsync processes.
#
# Notification Validated
# validate_unraid_cmd confirms the notify script is present before use.
# platform_require_cmd confirms the notify script is present before use.
#
# ==============================================================================================
# CONFIGURATION
@@ -170,7 +170,7 @@ if ! command -v docker &>/dev/null; then
exit 1
fi
validate_unraid_cmd \
platform_require_cmd \
"/usr/local/emhttp/plugins/dynamix/scripts/notify" \
"" "" \
"unRAID notify script" || warn "unRAID notify script not found — native notifications disabled"
unRAID_Essentials/Manual-Unraid_Essentials.md → System_Essentials/Manual-System_Essentials.md
+17 -10
View File
@@ -1,4 +1,4 @@
# ━━━━━ UNRAID ESSENTIALS — Manual ━━━━━
# ━━━━━ SYSTEM ESSENTIALS — Manual ━━━━━
Configuration reference, operational procedures, and troubleshooting for
system-level scripts. Read the ARRAY_START_SCRIPTS order section before
@@ -84,15 +84,15 @@ generates veth messages — these will appear in syslog if the filter isn't acti
curl $WEBGUI_URL → 200 OK → exit 0 (silent)
Not responding:
1. /etc/rc.d/rc.nginx restart
1. platform_restart_service nginx
wait WEBGUI_NGINX_WAIT (15s) → recheck
→ recovered: notify, exit 0
2. /etc/rc.d/rc.php-fpm restart
2. platform_restart_service php-fpm
wait WEBGUI_PHP_WAIT (10s) → recheck
→ recovered: notify, exit 0
3. /usr/local/sbin/emhttp stop && start
3. platform_restart_service emhttp
wait WEBGUI_EMHTTP_WAIT (30s) → recheck
→ recovered: notify, exit 0
@@ -115,16 +115,23 @@ WEBGUI_EMHTTP_WAIT=30 # seconds after emhttp restart before recheck
# Check which services are running:
webgui_watchdog.sh --status
# Try manual restart sequence (same as the script):
/etc/rc.d/rc.nginx restart
# Try manual restart sequence (mirrors what the script calls):
# Source the ecosystem first to get platform functions:
source /boot/config/plugins/varaverk/load_config.sh
platform_restart_service nginx
# wait 15s, then:
curl -sf --max-time 5 http://localhost >/dev/null && echo "OK" || echo "still down"
# If nginx didn't fix it, php-fpm:
/etc/rc.d/rc.php-fpm restart
# If nginx did not fix it, php-fpm:
platform_restart_service php-fpm
# If still down, emhttp:
/usr/local/sbin/emhttp stop && /usr/local/sbin/emhttp start
platform_restart_service emhttp
# Raw equivalents (no source needed — paste directly into terminal):
# /etc/rc.d/rc.nginx restart
# /etc/rc.d/rc.php-fpm restart
# /usr/local/sbin/emhttp stop && /usr/local/sbin/emhttp start
# If all three failed:
server_reboot.sh --status # check for active sessions first
@@ -296,7 +303,7 @@ du -sh /var/lib/docker/containers/*/*.log 2>/dev/null | sort -rh | head -5
### Stop Sequence
```
1. Check if mover is running (pgrep "emhttp.*Mover") → exit cleanly if not
1. Check if mover is running (platform_is_mover_running) → exit cleanly if not
2. Wall message to all logged-in terminal users
3. Wait MOVER_STOP_TIMEOUT seconds (default: 30)
4. SIGTERM — mover finishes its current file operation, then stops
unRAID_Essentials/README-Unraid_Essentials.md → System_Essentials/README-System_Essentials.md
+2 -2
View File
@@ -1,6 +1,6 @@
# ━━━━━ UNRAID ESSENTIALS ━━━━━
# ━━━━━ SYSTEM ESSENTIALS ━━━━━
**System-level scripts that act on the unRAID server itself — not containers,
**System-level scripts that act on the server itself — not containers,
not media, not monitoring.** Keeping the server stable under load, recovering a
frozen WebGUI, tuning kernel limits, suppressing log noise, and handling graceful
shutdowns with proper warning sequences.
unRAID_Essentials/clear_logs.sh → System_Essentials/clear_logs.sh
+1 -1
View File
@@ -93,7 +93,7 @@ if [[ "$EUID" -ne 0 ]]; then
exit 1
fi
validate_unraid_cmd \
platform_require_cmd \
"/usr/local/emhttp/plugins/dynamix/scripts/notify" \
"" "" \
"unRAID notify script" || warn "unRAID notify script not found — native notifications disabled"
unRAID_Essentials/conf_sync.sh → System_Essentials/conf_sync.sh
View File
unRAID_Essentials/docker_syslog_filter.sh → System_Essentials/docker_syslog_filter.sh
+4 -4
View File
@@ -93,7 +93,7 @@ if [[ "$EUID" -ne 0 ]]; then
exit 1
fi
validate_unraid_cmd \
platform_require_cmd \
"/usr/local/emhttp/plugins/dynamix/scripts/notify" \
"" "" \
"unRAID notify script" || warn "unRAID notify script not found — native notifications disabled"
@@ -134,7 +134,7 @@ if [[ "$SHOW_STATUS" == true ]]; then
echo ""
echo "━━━ rsyslog Status ━━━"
if pgrep -x rsyslogd >/dev/null 2>&1; then
if platform_is_service_running rsyslog; then
echo " rsyslogd: running ✅"
else
echo " rsyslogd: NOT running"
@@ -189,10 +189,10 @@ if [[ "$DRY_RUN" == true ]]; then
warn "DRY RUN — would restart rsyslog"
else
echo "Restarting rsyslog..."
if /etc/rc.d/rc.rsyslogd restart >/dev/null 2>&1; then
if platform_restart_service rsyslog; then
sleep 2
# Verify rsyslog actually running after restart
if pgrep -x rsyslogd >/dev/null 2>&1; then
if platform_is_service_running rsyslog; then
echo "rsyslog restarted and running ✅"
else
error "rsyslog not running after restart"
unRAID_Essentials/inotify_tuning.sh → System_Essentials/inotify_tuning.sh
+1 -1
View File
@@ -99,7 +99,7 @@ if [[ "$EUID" -ne 0 ]]; then
exit 1
fi
validate_unraid_cmd \
platform_require_cmd \
"/usr/local/emhttp/plugins/dynamix/scripts/notify" \
"" "" \
"unRAID notify script" || warn "unRAID notify script not found — native notifications disabled"
unRAID_Essentials/mover_stop.sh → System_Essentials/mover_stop.sh
+9 -9
View File
@@ -86,7 +86,7 @@ if [[ "$EUID" -ne 0 ]]; then
exit 1
fi
validate_unraid_cmd \
platform_require_cmd \
"/usr/local/emhttp/plugins/dynamix/scripts/notify" \
"" "" \
"unRAID notify script" || warn "unRAID notify script not found — native notifications disabled"
@@ -109,8 +109,8 @@ if [[ "$SHOW_STATUS" == true ]]; then
echo "$ICON_MOVER Timeout: ${MOVER_STOP_TIMEOUT}s"
echo "$ICON_GEAR Dry Run: $DRY_RUN"
echo ""
if pgrep -f "emhttp.*Mover" >/dev/null 2>&1; then
MOVER_PID=$(pgrep -f "emhttp.*Mover" | head -1)
if platform_is_mover_running; then
MOVER_PID=$(platform_get_mover_pid)
MOVER_START=$(ps -o lstart= -p "$MOVER_PID" 2>/dev/null | xargs)
echo " $ICON_MOVER Mover: RUNNING (PID $MOVER_PID)"
[[ -n "$MOVER_START" ]] && echo " $ICON_TIME Started: $MOVER_START"
@@ -126,12 +126,12 @@ fi
# ==============================================================================================
START=$(date +%s)
if ! pgrep -f "emhttp.*Mover" >/dev/null 2>&1; then
if ! platform_is_mover_running; then
echo "Mover is not running — nothing to do"
exit 0
fi
MOVER_PID=$(pgrep -f "emhttp.*Mover" | head -1)
MOVER_PID=$(platform_get_mover_pid)
MOVER_START=$(ps -o lstart= -p "$MOVER_PID" 2>/dev/null | xargs)
MOVER_ELAPSED=$(ps -o etimes= -p "$MOVER_PID" 2>/dev/null | tr -d ' ')
warn "Mover is running (PID $MOVER_PID) — stopping in ${MOVER_STOP_TIMEOUT}s"
@@ -151,20 +151,20 @@ if [[ "$DRY_RUN" == true ]]; then
warn "DRY RUN — would send SIGTERM to mover (PID $MOVER_PID)"
else
log "Sending SIGTERM to mover (PID $MOVER_PID)..."
pkill -TERM -f "emhttp.*Mover" 2>/dev/null || true
kill -TERM "$MOVER_PID" 2>/dev/null || true
sleep 5
# Verify stopped after SIGTERM
if ! pgrep -f "emhttp.*Mover" >/dev/null 2>&1; then
if ! platform_is_mover_running; then
warn "Mover stopped cleanly (SIGTERM) ✅"
else
# ── SIGKILL — forced stop ─────────────────────────────────────────────────────────────
warn "Mover still running after SIGTERM — sending SIGKILL (may leave partial files)"
pkill -KILL -f "emhttp.*Mover" 2>/dev/null || true
kill -KILL "$MOVER_PID" 2>/dev/null || true
sleep 2
# Final verify
if pgrep -f "emhttp.*Mover" >/dev/null 2>&1; then
if platform_is_mover_running; then
error "Mover still running after SIGKILL — manual intervention needed"
notify "Mover stop failed on $(hostname) ($MY_ID) — process unkillable" \
"Mover Stop" "warning"
unRAID_Essentials/php_fpm_max_children.sh → System_Essentials/php_fpm_max_children.sh
+2 -2
View File
@@ -103,7 +103,7 @@ if [[ "$EUID" -ne 0 ]]; then
exit 1
fi
validate_unraid_cmd \
platform_require_cmd \
"/usr/local/emhttp/plugins/dynamix/scripts/notify" \
"" "" \
"unRAID notify script" || warn "unRAID notify script not found — native notifications disabled"
@@ -200,7 +200,7 @@ log "Config updated"
# ── Restart PHP-FPM ──────────────────────────────────────────────────────────────────────────
log "Restarting PHP-FPM..."
if ! /etc/rc.d/rc.php-fpm restart >/dev/null 2>&1; then
if ! platform_restart_service php-fpm; then
error "PHP-FPM restart command failed"
notify "PHP-FPM restart failed on $(hostname) ($MY_ID)" \
"PHP-FPM" "warning"
unRAID_Essentials/rsync_stop.sh → System_Essentials/rsync_stop.sh
+1 -1
View File
@@ -114,7 +114,7 @@ if [[ "$EUID" -ne 0 ]]; then
exit 1
fi
validate_unraid_cmd \
platform_require_cmd \
"/usr/local/emhttp/plugins/dynamix/scripts/notify" \
"" "" \
"unRAID notify script" || warn "unRAID notify script not found — native notifications disabled"
unRAID_Essentials/server_reboot.sh → System_Essentials/server_reboot.sh
+4 -4
View File
@@ -110,7 +110,7 @@ if [[ "$EUID" -ne 0 ]]; then
exit 1
fi
validate_unraid_cmd \
platform_require_cmd \
"/usr/local/emhttp/plugins/dynamix/scripts/notify" \
"" "" \
"unRAID notify script" || warn "unRAID notify script not found — native notifications disabled"
@@ -140,7 +140,7 @@ if [[ "$SHOW_STATUS" == true ]]; then
warn " rsync: RUNNING — partial files if rebooted now" || \
log " rsync: not running"
pgrep -f "emhttp.*Mover" >/dev/null 2>&1 && \
platform_is_mover_running && \
warn " mover: RUNNING — files may be left mid-move" || \
log " mover: not running"
@@ -175,7 +175,7 @@ if pgrep -x rsync >/dev/null 2>&1; then
fi
# mover check — files may be left mid-move
if pgrep -f "emhttp.*Mover" >/dev/null 2>&1; then
if platform_is_mover_running; then
warn "Mover is running — files may be left mid-move on cache or array"
warn "Consider: mover_stop.sh before rebooting"
WARNINGS+=("mover running")
@@ -297,7 +297,7 @@ if ! is_vm_manager_enabled; then
elif [[ "$DRY_RUN" == true ]]; then
warn "DRY RUN — would stop VM Manager (libvirt)"
else
if /etc/rc.d/rc.libvirt stop >/dev/null 2>&1; then
if platform_stop_service libvirt; then
warn "VM Manager stopped ✅"
else
warn "VM Manager stop returned non-zero — may already be stopped"
unRAID_Essentials/unraid_api_key_renew.sh → System_Essentials/unraid_api_key_renew.sh
View File
unRAID_Essentials/user_scripts_stop.sh → System_Essentials/user_scripts_stop.sh
+1 -1
View File
@@ -85,7 +85,7 @@ if [[ "$EUID" -ne 0 ]]; then
exit 1
fi
validate_unraid_cmd \
platform_require_cmd \
"/usr/local/emhttp/plugins/dynamix/scripts/notify" \
"" "" \
"unRAID notify script" || warn "unRAID notify script not found — native notifications disabled"
Tools/Manual-Tools.md
+1 -1
View File
@@ -759,7 +759,7 @@ parse_args "$@"
if [[ "$EUID" -ne 0 ]]; then error "Must be run as root"; exit 1; fi
validate_unraid_cmd \
platform_require_cmd \
"/usr/local/emhttp/plugins/dynamix/scripts/notify" \
"" "" "unRAID notify script" || warn "notify not found — notifications disabled"
Tools/README-Tools.md
+2 -2
View File
@@ -86,7 +86,7 @@ something new, write the tool. Store it here. Find it at 2am next time.
## ━━━ RELATIONSHIP TO OTHER FOLDERS ━━━
```
unRAID_Essentials/ ← regular system maintenance — scheduled
System_Essentials/ ← regular system maintenance — scheduled
Docker_Essentials/ ← regular container management — scheduled
Monitors/ ← regular health reporting — scheduled
Orchestrators/ ← regular maintenance windows — scheduled
@@ -103,7 +103,7 @@ Fallback/
Docker_Essentials/
docker_watchdog.sh ── writes skip list + history ──► watchdog_skip_list_manager.sh manages them
Docker_Essentials/ + unRAID_Essentials/ + Fallback/
Docker_Essentials/ + System_Essentials/ + Fallback/
All continuous scripts ──────────────────────────► continuous_scripts_status.sh reads their state
```
Tools/bulk_permissions_repair.sh
+2 -2
View File
@@ -47,7 +47,7 @@
# are skipped rather than silently passing.
#
# Notification Validated
# validate_unraid_cmd confirms the notify script is present before use.
# platform_require_cmd confirms the notify script is present before use.
#
# Silent on Success
# Only failures and the wrong-owner diagnostic produce visible output.
@@ -98,7 +98,7 @@ fi
acquire_lock
validate_unraid_cmd \
platform_require_cmd \
"/usr/local/emhttp/plugins/dynamix/scripts/notify" \
"" "" \
"unRAID notify script" || warn "unRAID notify script not found — native notifications disabled"
Tools/conf_populate.sh
+1 -1
View File
@@ -222,5 +222,5 @@ echo "━━━━━━━━━━━━━━━━━━━━━"
if [[ "$UPDATED" -gt 0 ]] && [[ "$DRY_RUN" == false ]] && [[ "$NO_PUSH" == false ]]; then
echo ""
info "Pushing updated conf to partners..."
bash "$SCRIPTS_ROOT/unRAID_Essentials/conf_sync.sh" --push-only "${EXTRA_FLAGS[@]}" || true
bash "$SCRIPTS_ROOT/System_Essentials/conf_sync.sh" --push-only "${EXTRA_FLAGS[@]}" || true
fi
Tools/container_data_export.sh
+2 -2
View File
@@ -47,7 +47,7 @@
# daemon blocking the script indefinitely.
#
# Notification Validated
# validate_unraid_cmd confirms the notify script is present before use.
# platform_require_cmd confirms the notify script is present before use.
#
# ==============================================================================================
# RUNTIME MODES
@@ -93,7 +93,7 @@ if ! command -v docker &>/dev/null; then
exit 1
fi
validate_unraid_cmd \
platform_require_cmd \
"/usr/local/emhttp/plugins/dynamix/scripts/notify" \
"" "" \
"unRAID notify script" || warn "unRAID notify script not found — native notifications disabled"
Tools/emby_database_repair.sh
+3 -3
View File
@@ -41,7 +41,7 @@
# Emby can take time to stop cleanly — 30s is intentionally generous.
#
# Tool Validation
# validate_unraid_cmd confirms sqlite3 and the notify script are present
# platform_require_cmd confirms sqlite3 and the notify script are present
# before use. jq is checked separately — required for config path detection.
#
# Post-Restart Verify
@@ -96,12 +96,12 @@ if [[ "$EUID" -ne 0 ]]; then
fi
# Validate required tools
validate_unraid_cmd \
platform_require_cmd \
"$(command -v sqlite3 2>/dev/null || echo /usr/bin/sqlite3)" \
"--version" "." \
"sqlite3" || { error "sqlite3 not found — install sqlite package"; exit 1; }
validate_unraid_cmd \
platform_require_cmd \
"/usr/local/emhttp/plugins/dynamix/scripts/notify" \
"" "" \
"unRAID notify script" || warn "unRAID notify script not found — native notifications disabled"
Tools/fallback_state_reset.sh
+2 -2
View File
@@ -41,7 +41,7 @@
# non-interactive contexts (cron, scripts).
#
# Notification Validated
# validate_unraid_cmd confirms the notify script is present before use.
# platform_require_cmd confirms the notify script is present before use.
#
# ==============================================================================================
# RUNTIME MODES
@@ -88,7 +88,7 @@ if [[ "$EUID" -ne 0 ]]; then
exit 1
fi
validate_unraid_cmd \
platform_require_cmd \
"/usr/local/emhttp/plugins/dynamix/scripts/notify" \
"" "" \
"unRAID notify script" || warn "unRAID notify script not found — native notifications disabled"
Tools/ramdisk_stop.sh
+1 -1
View File
@@ -101,7 +101,7 @@ if [[ "$EUID" -ne 0 ]]; then
exit 1
fi
validate_unraid_cmd \
platform_require_cmd \
"/usr/local/emhttp/plugins/dynamix/scripts/notify" \
"" "" \
"unRAID notify script" || warn "unRAID notify script not found — native notifications disabled"
Tools/recreate_shares.sh
+2 -2
View File
@@ -55,7 +55,7 @@
# /boot/config/shares/ was not restored.
#
# Notification Validated
# validate_unraid_cmd confirms the notify script is present before use.
# platform_require_cmd confirms the notify script is present before use.
#
# ==============================================================================================
# RUNTIME MODES
@@ -96,7 +96,7 @@ if [[ "$EUID" -ne 0 ]]; then
exit 1
fi
validate_unraid_cmd \
platform_require_cmd \
"/usr/local/emhttp/plugins/dynamix/scripts/notify" \
"" "" \
"unRAID notify script" || warn "unRAID notify script not found — native notifications disabled"
Tools/watchdog_skip_list_manager.sh
+1 -1
View File
@@ -109,7 +109,7 @@ if [[ "$EUID" -ne 0 ]]; then
exit 1
fi
validate_unraid_cmd \
platform_require_cmd \
"/usr/local/emhttp/plugins/dynamix/scripts/notify" \
"" "" \
"unRAID notify script" || warn "unRAID notify script not found — native notifications disabled"
Tools/zfs_pool_scrub.sh
+3 -3
View File
@@ -41,7 +41,7 @@
# The poll loop exits cleanly on signal. The ZFS scrub continues regardless.
#
# Tool Validation
# validate_unraid_cmd confirms zpool and the notify script are present before use.
# platform_require_cmd confirms zpool and the notify script are present before use.
#
# Silent When Clean
# Only errors produce visible output and a notification.
@@ -95,7 +95,7 @@ if [[ "$EUID" -ne 0 ]]; then
exit 1
fi
validate_unraid_cmd \
platform_require_cmd \
"$(command -v zpool 2>/dev/null || echo /sbin/zpool)" \
"--version" "" \
"zpool" || {
@@ -103,7 +103,7 @@ validate_unraid_cmd \
exit 1
}
validate_unraid_cmd \
platform_require_cmd \
"/usr/local/emhttp/plugins/dynamix/scripts/notify" \
"" "" \
"unRAID notify script" || warn "unRAID notify script not found — native notifications disabled"
Transcodes/README-Transcoding.md
+1 -1
View File
@@ -110,7 +110,7 @@ path is a symlink whose target is managed at runtime. Sessions in progress never
## ━━━ RELATIONSHIP TO OTHER FOLDERS ━━━
```
unRAID_Essentials/
System_Essentials/
array_started.sh ──────────────────────────────► ramdisk_setup.sh (at array start)
Orchestrators/
Transcodes/ramdisk_setup.sh
+3 -3
View File
@@ -7,7 +7,7 @@
# ─────────────────────────────────────────────────────────────────────────────
# Creates the tmpfs ramdisk, SSD fallback directory, transcode symlink, and
# pre-creates transcoding-temp on the ramdisk. Run once at array start via
# array_started.sh (unRAID_Essentials/). Idempotent — already-mounted ramdisk
# array_started.sh (System_Essentials/). Idempotent — already-mounted ramdisk
# reports status and exits cleanly. Always resets the symlink to the ramdisk
# on boot, ensuring a clean state regardless of what state it was in before
# shutdown.
@@ -44,7 +44,7 @@
# without attempting to remount or changing anything.
#
# Notification Validated
# validate_unraid_cmd confirms the notify script is present before use.
# platform_require_cmd confirms the notify script is present before use.
#
# Silent on Success
# Startup script runs on every boot — no output when healthy.
@@ -117,7 +117,7 @@ if [[ "$EUID" -ne 0 ]]; then
exit 1
fi
validate_unraid_cmd \
platform_require_cmd \
"/usr/local/emhttp/plugins/dynamix/scripts/notify" \
"" "" \
"unRAID notify script" || warn "unRAID notify script not found — native notifications disabled"
Transcodes/transcode_cleanup.sh
+1 -1
View File
@@ -112,7 +112,7 @@ if [[ "$EUID" -ne 0 ]]; then
exit 1
fi
validate_unraid_cmd \
platform_require_cmd \
"/usr/local/emhttp/plugins/dynamix/scripts/notify" \
"" "" \
"unRAID notify script" || warn "unRAID notify script not found — native notifications disabled"
Transcodes/transcode_manager.sh
+2 -2
View File
@@ -56,7 +56,7 @@
# DOCKER_TIMEOUT caps all docker calls against a hung daemon.
#
# Notification Validated
# validate_unraid_cmd confirms the notify script is present before use.
# platform_require_cmd confirms the notify script is present before use.
#
# Silent by Default
# Runs every 7 minutes — only speaks when something changes or needs attention.
@@ -150,7 +150,7 @@ if [[ "$EUID" -ne 0 ]]; then
exit 1
fi
validate_unraid_cmd \
platform_require_cmd \
"/usr/local/emhttp/plugins/dynamix/scripts/notify" \
"" "" \
"unRAID notify script" || warn "unRAID notify script not found — native notifications disabled"
Watchdogs/System/network_watchdog.sh
+1 -1
View File
@@ -112,7 +112,7 @@ if [[ "$EUID" -ne 0 ]]; then
exit 1
fi
validate_unraid_cmd \
platform_require_cmd \
"/usr/local/emhttp/plugins/dynamix/scripts/notify" \
"" "" \
"unRAID notify script" || warn "unRAID notify script not found — native notifications disabled"
Watchdogs/System/webgui_watchdog.sh
+9 -9
View File
@@ -104,7 +104,7 @@ if [[ "$EUID" -ne 0 ]]; then
exit 1
fi
validate_unraid_cmd \
platform_require_cmd \
"/usr/local/emhttp/plugins/dynamix/scripts/notify" \
"" "" \
"unRAID notify script" || warn "unRAID notify script not found — native notifications disabled"
@@ -132,16 +132,16 @@ if [[ "$SHOW_STATUS" == true ]]; then
echo " $ICON_ERROR WebGUI: NOT responding"
fi
pgrep -x nginx >/dev/null 2>&1 && \
platform_is_service_running nginx && \
echo " $ICON_SUCCESS nginx: running ✅" || \
echo " $ICON_ERROR nginx: NOT running"
pgrep -f "php-fpm" >/dev/null 2>&1 && \
platform_is_service_running php-fpm && \
FPM_COUNT=$(pgrep -fc "php-fpm" 2>/dev/null || echo "?") && \
echo " $ICON_SUCCESS php-fpm: running ($FPM_COUNT workers) ✅" || \
echo " $ICON_ERROR php-fpm: NOT running"
pgrep emhttpd >/dev/null 2>&1 && \
platform_is_service_running emhttp && \
echo " $ICON_SUCCESS emhttp: running ✅" || \
echo " $ICON_ERROR emhttp: NOT running"
echo "━━━━━━━━━━━━━━━━━━━━━━━"
@@ -185,10 +185,10 @@ if [[ "$DRY_RUN" == true ]]; then
warn "DRY RUN — would restart nginx"
else
warn "Restarting nginx..."
if /etc/rc.d/rc.nginx restart >/dev/null 2>&1; then
if platform_restart_service nginx; then
# Verify nginx actually running
sleep 2
if pgrep -x nginx >/dev/null 2>&1; then
if platform_is_service_running nginx; then
warn "nginx restarted ✅"
else
error "nginx not running after restart command"
@@ -216,9 +216,9 @@ if [[ "$RECOVERY_OK" == false ]]; then
if [[ "$DRY_RUN" == true ]]; then
warn "DRY RUN — would restart php-fpm"
else
if /etc/rc.d/rc.php-fpm restart >/dev/null 2>&1; then
if platform_restart_service php-fpm; then
sleep 2
if pgrep -f "php-fpm" >/dev/null 2>&1; then
if platform_is_service_running php-fpm; then
warn "php-fpm restarted ✅"
else
error "php-fpm not running after restart command"
@@ -249,7 +249,7 @@ if [[ "$RECOVERY_OK" == false ]]; then
else
if /usr/local/sbin/emhttp stop >/dev/null 2>&1 && /usr/local/sbin/emhttp start >/dev/null 2>&1; then
sleep 2
if pgrep emhttpd >/dev/null 2>&1; then
if platform_is_service_running emhttp; then
warn "emhttp restarted ✅"
else
error "emhttp not running after restart command"
Watchdogs/docker_watchdog.sh
+9 -19
View File
@@ -256,9 +256,9 @@ log "$ICON_CONTAINERS Tier1: watched=${#WATCHDOG_CONTAINERS[@]} required=${#WAT
# Validate unRAID-specific commands used by this script
# If rc.docker is missing or changed, daemon restart will fail — better to know now
validate_unraid_cmd "/etc/rc.d/rc.docker" "" "" "Docker rc.d script" || warn "rc.docker not found — daemon restart unavailable if needed"
platform_require_cmd "/etc/rc.d/rc.docker" "" "" "Docker rc.d script" || warn "rc.docker not found — daemon restart unavailable if needed"
validate_unraid_cmd "/usr/local/emhttp/plugins/dynamix/scripts/notify" "" "" "unRAID notify script" || warn "unRAID notify script not found — native notifications disabled"
platform_require_cmd "/usr/local/emhttp/plugins/dynamix/scripts/notify" "" "" "unRAID notify script" || warn "unRAID notify script not found — native notifications disabled"
# Ensure state files exist
touch "$WATCHDOG_STATE_FILE" "$WATCHDOG_CONTAINER_RESTART_LOG" \
@@ -453,15 +453,8 @@ flush_notify() {
NOTIFY_EVENTS=()
}
# Returns 0 if parity check is currently running
# Unraid 7.3+: mdResync in var.ini (non-zero = check/sync in progress)
# Older: parity-date.txt contained "progress" — check both for compatibility
is_parity_running() {
local resync
resync=$(awk -F'"' '/^mdResync=/{print $2}' /var/local/emhttp/var.ini 2>/dev/null)
[[ -n "$resync" && "$resync" != "0" ]] && return 0
grep -q "progress" /var/local/emhttp/parity-date.txt 2>/dev/null
}
# is_parity_running — delegates to adapter
is_parity_running() { platform_is_maintenance_running; }
# ==============================================================================================
# ── DOCKER DAEMON HEALTH CHECK ────────────────────────────────────────────────────────────────
@@ -567,16 +560,14 @@ check_docker_daemon() {
notify "Docker daemon hung on $(hostname) — attempting restart" "Docker Watchdog" "warning"
if [[ "$DRY_RUN" == true ]]; then
warn "DRY RUN — would restart Docker daemon via /etc/rc.d/rc.docker restart"
warn "DRY RUN — would restart Docker daemon via platform_restart_service docker"
return 1
fi
# Restart daemon — unRAID uses rc.d scripts, not systemd
# timeout 180: rc.docker stops all containers before restarting — without a limit this
# can block for 30+ min on a busy host, holding the orchestrator lock the entire time.
# timeout 180: Docker daemon stop can block for 30+ min on a busy host.
WATCHDOG_DAEMON_RESTARTED=true
set_strikes "daemon_restarted_flag" "true" "$WATCHDOG_STATE_FILE"
if timeout 180 /etc/rc.d/rc.docker restart >/dev/null 2>&1; then
if timeout 180 platform_restart_service docker; then
log "Docker daemon restart issued — waiting ${WATCHDOG_DAEMON_RESTART_WAIT}s..."
sleep "$WATCHDOG_DAEMON_RESTART_WAIT"
@@ -598,8 +589,8 @@ check_docker_daemon() {
return 1
fi
else
# rc.docker not found or failed — flag immediately, stability_watchdog escalates
error "Failed to issue Docker daemon restart — /etc/rc.d/rc.docker not found or failed"
# platform_restart_service returned non-zero — flag immediately, stability_watchdog escalates
error "Failed to issue Docker daemon restart — platform_restart_service docker failed"
set_strikes "daemon_confirmed_down" "true" "$WATCHDOG_STATE_FILE"
queue_notify "Docker daemon restart command failed on $(hostname) — stability_watchdog escalating" "critical"
flush_notify
@@ -632,7 +623,6 @@ CYCLE_START=$(date +%s)
done
# ── Skip list visibility ─────────────────────────────────────────────────────────────────
local _skip_contents
_skip_contents=$(cat "$DOCKER_WATCHDOG_FAILED_FILE" 2>/dev/null | tr '\n' ' ' | xargs)
[[ -n "$_skip_contents" ]] && warn "$ICON_SKIP Skip list active: $_skip_contents — manual intervention needed"
Watchdogs/stability_watchdog.sh
+6 -10
View File
@@ -119,7 +119,7 @@ if [[ "$EUID" -ne 0 ]]; then
exit 1
fi
validate_unraid_cmd \
platform_require_cmd \
"/usr/local/emhttp/plugins/dynamix/scripts/notify" \
"" "" \
"unRAID notify script" || warn "unRAID notify script not found — native notifications disabled"
@@ -302,12 +302,8 @@ check_abort_conditions() {
fi
fi
# Unraid 7.3+: parity state is in var.ini (mdResync != 0 means check/sync in progress)
# Older: parity-date.txt contained "progress" — check both for compatibility
_md_resync=$(awk -F'"' '/^mdResync=/{print $2}' /var/local/emhttp/var.ini 2>/dev/null)
_parity_running=false
[[ -n "$_md_resync" && "$_md_resync" != "0" ]] && _parity_running=true
grep -q "progress" /var/local/emhttp/parity-date.txt 2>/dev/null && _parity_running=true
platform_is_maintenance_running && _parity_running=true
if [[ "$_parity_running" == true ]]; then
if [[ "$SYS_WATCHDOG_ABORT_ON_PARITY" == true ]]; then
error "Parity check running — aborting reboot"
@@ -451,7 +447,7 @@ do_reboot() {
fi
warn "Stopping User Scripts..."
pkill -f "/tmp/user.scripts" 2>/dev/null || true
platform_stop_user_scripts
warn "Syncing disks..."
sync
@@ -726,12 +722,12 @@ echo "━━━ $ICON_REBOOT Stability Watchdog — $(date '+%Y-%m-%d %H:%M:%S')
# ── sshd — try restart before escalating ─────────────────────────────────────────────────
if [[ "$SYS_WATCHDOG_CHECK_SSHD" == true ]]; then
if ! pgrep -x sshd >/dev/null 2>&1; then
if ! platform_is_service_running sshd; then
warn "sshd not running — attempting restart..."
if [[ "$DRY_RUN" == false ]]; then
/etc/rc.d/rc.sshd start >/dev/null 2>&1
platform_restart_service sshd
sleep 3
if pgrep -x sshd >/dev/null 2>&1; then
if platform_is_service_running sshd; then
warn "sshd restarted successfully ✅"
reset_strikes "sshd"
notify "sshd was down on $(hostname) ($MY_ID) — restarted automatically" \
Watchdogs/system_watchdog.sh
+1 -1
View File
@@ -45,7 +45,7 @@ if [[ "$EUID" -ne 0 ]]; then
exit 1
fi
validate_unraid_cmd \
platform_require_cmd \
"/usr/local/emhttp/plugins/dynamix/scripts/notify" \
"" "" \
"unRAID notify script" || warn "unRAID notify script not found — native notifications disabled"
load_config.sh
+21 -4
View File
@@ -6,10 +6,12 @@
# Every script sources this file instead of sourcing master.conf files directly.
#
# ── HOW IT WORKS ──────────────────────────────────────────────────────────────────────────────
# 1. Sources master.conf (shared config — hostnames, thresholds, toggles, profiles, job lists)
# 2. Auto-discovers and sources all host*.conf files in the same directory
# 1. Detects OS platform → PLATFORM=unraid|truenas|unknown; exports SCRIPTS_DIR
# 2. Sources master.conf (shared config — hostnames, thresholds, toggles, profiles, job lists)
# 3. Auto-discovers and sources all host*.conf files in the same directory
# Each host conf extends the shared profile arrays and adds host-specific credentials
# 3. Sources common.sh (shared functions — detect_hosts, logging, notifications etc.)
# 4. Sources common.sh (shared functions — detect_hosts, logging, notifications etc.)
# 5. Sources Plugin/<platform>/adapter.sh (platform_*() functions for OS-specific ops)
#
# ── WHY THIS EXISTS ───────────────────────────────────────────────────────────────────────────
# Without this loader every script had to explicitly source each conf file:
@@ -53,6 +55,15 @@
# Scripts call it from subdirectories using ../ — resolve to the actual root.
LOAD_CONFIG_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
# ━━━ Platform detection ━━━
# PLATFORM drives the adapter layer — Plugin/<platform>/adapter.sh
# Each platform adapter provides the same function API; scripts stay OS-agnostic.
if [[ -f /etc/unraid-version ]]; then PLATFORM="unraid"
elif [[ -f /etc/truenas ]]; then PLATFORM="truenas"
else PLATFORM="unknown"
fi
export PLATFORM SCRIPTS_DIR="$LOAD_CONFIG_DIR"
# ━━━ Source shared config ━━━
# master.conf must be sourced first — it declares the shared PROFILE_* arrays
# that Host confs extend. Sourcing host confs before master.conf would fail.
@@ -93,5 +104,11 @@
fi
source "$LOAD_CONFIG_DIR/common.sh"
# ━━━ Source platform adapter ━━━
# Provides platform_*() functions used by common.sh and scripts.
# Guard lets the ecosystem run before Plugin/<platform>/adapter.sh exists.
_adapter="$LOAD_CONFIG_DIR/Plugin/$PLATFORM/adapter.sh"
[[ -f "$_adapter" ]] && source "$_adapter"
# ━━━ Cleanup ━━━
unset _conf _host_confs_loaded LOAD_CONFIG_DIR
unset _conf _host_confs_loaded _adapter LOAD_CONFIG_DIR
user_script_plug-in.sh
+20 -20
View File
@@ -551,9 +551,9 @@
# --full-stop: kills orchestrator first then rsync — emergency full stop.
# --rsync-only: stop rsync, skip container recovery check (used by partnership_manager --offboard).
#
# bash /boot/config/plugins/varaverk/unRAID_Essentials/rsync_stop.sh --status
# bash /boot/config/plugins/varaverk/unRAID_Essentials/rsync_stop.sh
# bash /boot/config/plugins/varaverk/unRAID_Essentials/rsync_stop.sh --full-stop
# bash /boot/config/plugins/varaverk/System_Essentials/rsync_stop.sh --status
# bash /boot/config/plugins/varaverk/System_Essentials/rsync_stop.sh
# bash /boot/config/plugins/varaverk/System_Essentials/rsync_stop.sh --full-stop
# ──────────────────────────────────────────────────────────────────────────────────────────────
@@ -793,8 +793,8 @@
# If Code-Server started before this ran → docker restart Code-Server to inherit new limits.
# Exhaustion symptom: downloads complete but arrs don't detect them, Live TV stutters.
#
# bash /boot/config/plugins/varaverk/unRAID_Essentials/inotify_tuning.sh --status
# bash /boot/config/plugins/varaverk/unRAID_Essentials/inotify_tuning.sh
# bash /boot/config/plugins/varaverk/System_Essentials/inotify_tuning.sh --status
# bash /boot/config/plugins/varaverk/System_Essentials/inotify_tuning.sh
# php_fpm_max_children.sh — set PHP-FPM pm.max_children at array start
# Called by array_started.sh. Default is 4-8 workers — inadequate for a busy multi-user server.
@@ -802,8 +802,8 @@
# Symptom of saturation: WebGUI slow, settings saves hang, container UI starts timeout.
# Resets on each reboot — reapplied at array start. Idempotent: silent when already correct.
#
# bash /boot/config/plugins/varaverk/unRAID_Essentials/php_fpm_max_children.sh --status
# bash /boot/config/plugins/varaverk/unRAID_Essentials/php_fpm_max_children.sh
# bash /boot/config/plugins/varaverk/System_Essentials/php_fpm_max_children.sh --status
# bash /boot/config/plugins/varaverk/System_Essentials/php_fpm_max_children.sh
# docker_syslog_filter.sh — suppress Docker veth/docker0 interface log noise
# Called by array_started.sh before containers start. Creates rsyslog drop rule.
@@ -811,8 +811,8 @@
# Real events (mount failures, permission errors) are invisible in that noise.
# Idempotent: compares expected filter content exactly — only writes when changed.
#
# bash /boot/config/plugins/varaverk/unRAID_Essentials/docker_syslog_filter.sh --status
# bash /boot/config/plugins/varaverk/unRAID_Essentials/docker_syslog_filter.sh
# bash /boot/config/plugins/varaverk/System_Essentials/docker_syslog_filter.sh --status
# bash /boot/config/plugins/varaverk/System_Essentials/docker_syslog_filter.sh
# webgui_watchdog.sh — WebGUI availability watchdog (called by system_watchdog via SYSTEM_WATCHDOG_SCRIPTS)
# Not scheduled directly — runs as part of the every-15-minute watchdog chain.
@@ -823,17 +823,17 @@
# clear_logs.sh — size-threshold log cleanup (called by weekly_sync_maintenance via WEEKLY_MAINTENANCE_SCRIPTS)
# Not scheduled directly — runs as part of the Sunday 2:30am weekly window.
# System logs only cleared if > LOG_MIN_SIZE_MB (10MB). Docker logs only if > LOG_DOCKER_MAX_MB (100MB).
# bash /boot/config/plugins/varaverk/unRAID_Essentials/clear_logs.sh --dry-run
# bash /boot/config/plugins/varaverk/unRAID_Essentials/clear_logs.sh --status
# bash /boot/config/plugins/varaverk/System_Essentials/clear_logs.sh --dry-run
# bash /boot/config/plugins/varaverk/System_Essentials/clear_logs.sh --status
# mover_stop.sh — stop unRAID mover cleanly before array operations
# Wall message to logged-in users. Waits MOVER_STOP_TIMEOUT seconds.
# SIGTERM (allows mover to finish current file — no partial files). Verify. SIGKILL last resort.
# Use before: planned reboots with mover running, disk replacement, array maintenance.
#
# bash /boot/config/plugins/varaverk/unRAID_Essentials/mover_stop.sh --status
# bash /boot/config/plugins/varaverk/unRAID_Essentials/mover_stop.sh --dry-run
# bash /boot/config/plugins/varaverk/unRAID_Essentials/mover_stop.sh
# bash /boot/config/plugins/varaverk/System_Essentials/mover_stop.sh --status
# bash /boot/config/plugins/varaverk/System_Essentials/mover_stop.sh --dry-run
# bash /boot/config/plugins/varaverk/System_Essentials/mover_stop.sh
# server_reboot.sh — graceful reboot with pre-flight warnings and clean shutdown sequence
# Pre-flight warnings (inform not block): rsync running, mover running, active Emby sessions.
@@ -841,9 +841,9 @@
# → wait REBOOT_VM_WAIT → stop libvirt → stop Docker → sync → /sbin/reboot
# --dry-run walks the full sequence without rebooting — shows all active processes.
#
# bash /boot/config/plugins/varaverk/unRAID_Essentials/server_reboot.sh --dry-run
# bash /boot/config/plugins/varaverk/unRAID_Essentials/server_reboot.sh --status
# bash /boot/config/plugins/varaverk/unRAID_Essentials/server_reboot.sh
# bash /boot/config/plugins/varaverk/System_Essentials/server_reboot.sh --dry-run
# bash /boot/config/plugins/varaverk/System_Essentials/server_reboot.sh --status
# bash /boot/config/plugins/varaverk/System_Essentials/server_reboot.sh
# user_scripts_stop.sh — stop all running User Script processes
# Finds processes by /tmp/user.scripts path signature. Shows script names, not just PIDs.
@@ -851,9 +851,9 @@
# Use when plugin Abort button didn't work, or before a reboot to clean up running scripts.
# Called automatically by server_reboot.sh before reboot.
#
# bash /boot/config/plugins/varaverk/unRAID_Essentials/user_scripts_stop.sh --status
# bash /boot/config/plugins/varaverk/unRAID_Essentials/user_scripts_stop.sh --dry-run
# bash /boot/config/plugins/varaverk/unRAID_Essentials/user_scripts_stop.sh
# bash /boot/config/plugins/varaverk/System_Essentials/user_scripts_stop.sh --status
# bash /boot/config/plugins/varaverk/System_Essentials/user_scripts_stop.sh --dry-run
# bash /boot/config/plugins/varaverk/System_Essentials/user_scripts_stop.sh
# git_pull_execute.sh — pull latest scripts from Gitea and set execute permissions
# Deployment mechanism for the ecosystem. Push from VS Code → Gitea → run on both servers.