22 scripts used docker without verifying it's present. Consistent with the
pattern already established in Docker_Essentials scripts. Added after acquire_lock
and before detect_hosts in each script's setup section.
Consistent naming with the watchdog family (system_watchdog, docker_watchdog).
File renamed, all RM_ config variables and internal references updated to RW_
across master.conf, master_host1.conf, common.sh, docker_watchdog.sh,
system_watchdog.sh, watchdog_orchestrator.sh, and sunday_morning_coffee_report.sh.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Introduce a four-layer self-healing stack replacing the continuous-loop watchdogs:
- resource_manager.sh (new): single-pass pressure reduction layer; throttles
SABnzbd/qBit at level 1, docker-pauses background containers at level 2,
docker-stops optional containers and signals docker_watchdog to defer at
level 3; graduated recovery with hysteresis
- watchdog_orchestrator.sh (new, Orchestrators/): runs resource_manager →
docker_watchdog → system_watchdog in sequence; intended for per-minute cron
via User Scripts; startup grace, acquire_lock to prevent pile-up, heartbeat
- docker_watchdog.sh: de-looped to single-pass; daemon strikes persisted to
state file across runs; cross-script coordination reads RM_STATE_FILE instead
of SYS_WATCHDOG_STATE_FILE
- system_watchdog.sh: de-looped to single-pass; stripped of all container
management (shutdown_non_essential_containers removed); reboot-only last resort
- master.conf: removed system_watchdog and docker_watchdog from
ARRAY_START_SCRIPTS; added WATCHDOG ORCHESTRATOR and RESOURCE MANAGER sections
- master_host1.conf: added RM_PAUSE_CONTAINERS and RM_STOP_CONTAINERS arrays
- common.sh: aliased RM_PAUSE_CONTAINERS and RM_STOP_CONTAINERS via detect_hosts()
- continuous_scripts_status.sh: moved to Tools/ (preserved for future use)
- sunday_morning_coffee_report.sh: watchdog section updated to use state file
mtime checks instead of is_running; added Resource Manager subsection;
fixed mem_shutdown grep filter pointing to wrong state file
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
- Rename array_start.sh → array_started.sh, array_stop.sh → array_stopping.sh
to clarify these are event-driven (array has started/is stopping), not imperative
- Update all references across 9 files (master.conf, user_script_plug-in.sh,
watchdogs, continuous_scripts_status.sh, claude_startup.sh)
- Add --remainder mode to docker_update.sh: updates all running containers
excluding daily containers, weekly sync-window containers (emby + critical-data),
and fallback coverage containers (FALLBACK_${MY_ID}_COVERS_${REMOTE_ID}_TIER*)
Fallback containers excluded because the remote server owns their version —
independent updates risk writeback incompatibility on handback
- weekly_sync_maintenance.sh calls docker_update.sh --remainder as final step
- git_pull_execute.sh: add safe.directory config to fix dubious ownership error
when running as root on a directory owned by uid 1000
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
common.sh:
- Add resolve_tailscale_ip() helper — tries `tailscale ip -4` first, falls back to
parsing `tailscale status` output; handles hosts where MagicDNS short-name resolution
is not active
- Add PARTNERSHIP_OWN_CONTAINERS alias in detect_hosts()
- Add aliasing for 4 Emby provisioning vars (PARTNERSHIP_PROVISION_EMBY_ADMIN,
PARTNERSHIP_EMBY_ADMIN_USER, PARTNERSHIP_EMBY_ADMIN_PASS, PARTNERSHIP_EMBY_PORT)
Partnership/partnership_manager.sh:
- Replace 9 bare `tailscale ip -4` calls with resolve_tailscale_ip()
- Add read_remote_conf_var() and read_remote_conf_array() — SSH to mirror, source its
own load_config.sh + detect_hosts(), return aliased variable; solves sparse-checkout
problem where HOST1 cannot read master_host2.conf directly
- Add derive_short_name() — strips unraid- prefix, capitalises first char
- Add cleanup_partner_containers() — removes partner containers via FolderView3 folder
if enabled, else falls back to FALLBACK_*_COVERS_*_TIER* arrays
- Add cleanup_owner_containers_on_mirror() — SSH to mirror, stops and removes containers
matching *-${OWNER_SHORT} naming convention
- Add start_own_stack() and start_mirror_own_stack() — restart own containers locally
or on mirror via SSH using PARTNERSHIP_OWN_CONTAINERS
- Add provision_emby_admin() — reads mirror credentials via read_remote_conf_var, checks
for username collision, creates user + sets password + grants admin policy via Emby API
- Add revoke_emby_admin() — looks up mirror username on local Emby, deletes via REST API
- Wire offboard paths (both mirror-initiated and owner-initiated) to call container
cleanup and stack restart; update --check finalisation paths accordingly
- Fix write_state_file in --onboard not gated on DRY_RUN (was writing ACTIVE state on
dry runs)
master_host1.conf:
- Add HOST1_PARTNERSHIP_OWN_CONTAINERS array
- Add partnership Emby provisioning config (toggle + port + per-host credentials)
master_host2.conf:
- Add HOST2_PARTNERSHIP_OWN_CONTAINERS array
- Add HOST2_PARTNERSHIP_EMBY_ADMIN_USER and HOST2_PARTNERSHIP_EMBY_ADMIN_PASS
Tailscale fix applied to:
- Initial_run/ssh_setup.sh (2 callsites)
- unRAID_Essentials/rsync_stop.sh (1 callsite)
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Add intermediate_sync_maintenance.sh (0 */4 * * *) — arr library sync as
fixed first step, optional mid-day rsync (INTERMEDIATE_SYNC_SHARES, empty
by default), then INTERMEDIATE_MAINTENANCE_SCRIPTS. Wired with full rsync
infrastructure (temperature abort, check_rsync_enabled, INTERMEDIATE_RSYNC_ENABLED
toggle) matching daily/weekly pattern. lidarr_missing_art.sh scheduled here.
arr_sync.sh: blocklist --add now end-to-end — looks up real display name
from local arr API, writes TSV, deletes from local arr API (deleteFiles=false),
SSHes each remote node and deletes from their arr API. Files become orphans
for arr_cleanup safety pass.
master.conf: DEFAULT_RSYNC_OPTS updated (--partial, --timeout=60,
--numeric-ids; --no-whole-file removed). Media share comments updated to
mesh model.
master_host1.conf + master_host2.conf: both hosts now push all media shares
bidirectionally (true mesh — no ownership per share, arr_cleanup enforces truth).
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Arr sync (new):
- Media/arr_sync.sh — full mesh bidirectional sync across all HOST* nodes
- Lidarr (MusicBrainz), Sonarr (TVDB), Radarr (TMDB) all handled in one script
- Remote API keys read live from config.xml via SSH — never stored in conf files
- Shared blocklist (DATA_DIR/arr_sync_blocklist.tsv) merged from all nodes at runtime
- Graceful skip if arr not configured locally or not reachable on a remote node
- --blocklist-add / --blocklist-remove / --blocklist-list management flags
- daily_sync_maintenance.sh — arr sync runs as explicit phase before rsync
- partnership_onboard.sh — Step 3 bootstraps merged library on both sides at onboard
- master.conf — ARR_SYNC_* config block, DOCKER_APPDATA_BASE
Rsync / cleanup:
- DEFAULT_RSYNC_OPTS — removed --delete; arr_cleanup.sh owns orphan enforcement
- lidarr_cleanup.sh — removed HOST1-only guard; runs on any node with Lidarr configured
Config architecture:
- HOST1/HOST2 hostnames moved from master_host*.conf → master.conf (not credentials)
- Sparse checkout now works correctly: each server only needs its own host conf
- detect_hosts() still resolves MY_ID + REMOTE_ID via master.conf hostname values
Bug fix:
- common.sh line 493 — watchdog toggle eval had broken quoting; all SYS_WATCHDOG_CHECK_*
globals were silently set to empty instead of their configured values
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Gmer4Lfe
FailedProxy