Checks:
1. Internet reachability (gates all remaining checks)
2. DDNS sync — public IP vs Cloudflare DNS record, restarts Gmer4Lfe.com container on mismatch
3. Tailscale status — notify only, no auto-restart
4. NPM proxy — external curl to https://gmer4lfe.com, 2-strike system before NginxProxyManager restart
Config: master.conf NETWORK_WATCHDOG_* block, host1.conf HOST1_NETWORK_WATCHDOG_* values
Added to SYSTEM_WATCHDOG_SCRIPTS — called by system_watchdog.sh each cycle
Fix: storage_watchdog.sh was calling get_strikes/set_strikes without defining them —
added local definitions (same pattern as docker_watchdog.sh and stability_watchdog.sh)
Watchdog renames and moves:
system_watchdog.sh → stability_watchdog.sh (last line of defense — reboots)
storage_watchdog.sh → Watchdogs/System/storage_watchdog.sh
webgui_restart.sh → Watchdogs/System/webgui_watchdog.sh (renamed to match folder convention)
New thin orchestrator:
Watchdogs/system_watchdog.sh — runs SYSTEM_WATCHDOG_SCRIPTS from master.conf
Sits between docker_watchdog and stability_watchdog in the orchestrator tier chain
System/ subfolder is the growth seam for future system component watchdogs
master.conf:
WATCHDOG_ORCHESTRATOR_SCRIPTS updated — storage removed, system_watchdog added as tier
SYSTEM_WATCHDOG_SCRIPTS array added — storage + webgui
server_reboot.sh:
Calls array_stopping.sh before VM shutdown for guaranteed safe array stop
Removed raw rc.docker stop and exit trap — orchestrator owns container shutdown
master.conf, host1.conf, host2.conf relocated from repo root to Configurations/.
load_config.sh updated to source from the new path — no other scripts need changes.
Note: sparse checkout rules on HOST1 and HOST2 need updating to use
Configurations/host1.conf and Configurations/host2.conf respectively.
README.md: header updated to VARAVERK. All "User Scripts plugin" references
replaced with "Arbiter". Project name section: "Working name: Arbiter" → "Varaverk".
Gitea corrected to Tier 1 (was shown as Tier 3, updated to Tier 1 this session).
master.conf: GITEA_REPO_PATH updated FailedProxy → Varaverk org name.
Script internals (user_scripts_stop.sh, script headers, other docs) reference
the /tmp/user.scripts working path which is Arbiter-specific — those will be
updated in a separate pass once Arbiter's working directory is confirmed.
Partnership/gitea_ssh_setup.sh: generates ed25519 keypair, registers public key
with Gitea API (POST /api/v1/user/keys), tests SSH connection, writes ~/.ssh/config
entry. Idempotent — skips steps already done. --force regenerates and re-registers.
Resolves Gitea endpoint from container IP first, falls back to GITEA_DOMAIN.
common.sh: alias HOST*_GITEA_API_TOKEN → GITEA_API_TOKEN in detect_hosts().
Add to detect_hosts() doc comment.
master.conf: add GITEA_HTTP_PORT=3000 for API endpoint construction.
host1.conf: add HOST1_GITEA_API_TOKEN (fill in from Gitea Settings → Applications).
Add my-Gitea.xml to HOST1_PARTNERSHIP_AUTH_STACK — onboard pushes it to HOST2.
host2.conf: add Gitea to FALLBACK_HOST2_COVERS_HOST1_TIER1 — starts immediately
when HOST1 goes down, making the source of truth reachable independently of
HOST1's auth stack.
host1.conf: HOST1_RAMDISK_SIZE 8G→10G, WARN_GB 6.8→8.5, LOW_GB 5.5→7.
ramdisk_stop.sh: new Tools/ script — redirects symlink to SSD before unmount
so Emby keeps writing during maintenance, auto-falls back to lazy umount
when only directory handles are open (no active writes).
common.sh: fix `info` call at line 595 (undefined) → `log`.
Tools docs: added ramdisk_stop.sh to README table, categories, diagram, and Manual.
watchdog_orchestrator.sh stays in Orchestrators/ — it's an orchestrator, not a watchdog.
watchdog_skip_list_manager.sh stays in Tools/ — it's a management utility.
Only the 4 watchdog scripts belong in Watchdogs/:
docker_watchdog.sh, resource_watchdog.sh, storage_watchdog.sh, system_watchdog.sh
No setup banners to remove (Tools folder was already clean). Fix log→echo
on summary status/conclusion lines across container_data_export, emby_database_repair,
recreate_shares, zfs_pool_scrub, and fallback_state_reset (safety check + two summary
detail lines). Fix log→echo on status display lines in watchdog_skip_list_manager
(skip list empty, restart history lines). Move per-item "to add" preview echoes to
log in emby_to_lidarr/radarr/sonarr_sync; fix nothing-to-do log→echo in all three.
Add output tiers section to Manual-Tools.md.
Fix log→echo on success status line in summary block. Add output tiers
section to Manual-Rsync.md. No setup banner (script was already clean);
all section headers and transfer identity block correctly remain echo.
Remove setup banners from all 8 scripts that had them (discovery scripts
use comment-only setup sections). Move per-item/detail echoes to log in
arr_sync, lidarr_missing_art, lidarr/radarr/sonarr_cleanup (scan progress,
API query status, root/orphan config lines), and per-title lines in
radarr_tmdb_removed / sonarr_tvdb_removed. Fix log→echo on clean-status
summary lines in arrs_failed_stalled_recovery, media_cleaner, and both
removed-title scripts. Add output tiers section to Manual-Media.md.
Consistent two-tier output model across the entire folder:
- Per-container banners, action lines, and list details → log (ENABLE_LOGGING=true only)
- Section headers, summaries, counts, and status conclusions → echo (always visible)
- Warnings and errors always visible regardless of log setting
- Blank echo lines inside loops removed
All scripts: added Lock Acquisition and Host Detection entries to OPERATIONAL SAFEGUARDS.
Setup banners removed from docker_watchdog.sh, docker_weekly_restart.sh,
downloaders_reset.sh (noise before any work happens).
docker_watchdog.sh: removed success "Running as root" / "Docker found" setup lines;
info() → log() for daemon-recovered and per-cycle header; removed per-cycle echo separator.
docker_update.sh + docker_update_remaining.sh: docker pull stdout suppressed when
ENABLE_LOGGING=false to prevent orphaned Status: lines appearing without container context;
restructured pull block to use PIPESTATUS for exit code capture.
downloaders_reset.sh: 11× info() → log() for all per-item API operation lines.
docker_weekly_restart.sh: removed duplicate restart-order echo (build_restart_order
already logs it internally).
Manual-Docker_Essentials.md: documented the two-tier output model and watchdog
silent-when-healthy exception in the flag reference section.
22 scripts used docker without verifying it's present. Consistent with the
pattern already established in Docker_Essentials scripts. Added after acquire_lock
and before detect_hosts in each script's setup section.
docker_container_stop, docker_update, docker_update_remaining — concurrent Docker
operations on the same containers would conflict; now locked.
rsync.sh — two rsync processes running against the same share simultaneously
would produce incomplete or corrupted mirrors; now locked.
partnership_onboard, ssh_setup — one-shot setup scripts that mutate SSH config and
deploy containers; concurrent runs would produce undefined state; now locked.
Old_Arch_Still_Works: arr_cleanup, continuous_scripts_status — legacy scripts still
sourcing load_config.sh; added lock for consistency even in old-arch context.
partnership_manager.sh intentionally left unchanged — it uses a conditional lock
that excludes read-only "check" mode and "offboard" mode (which delegates to
partnership_offboard.sh, which has its own lock).
arr_sync.sh had acquire_lock documented in OPERATIONAL SAFEGUARDS header but the
call was never present in the code — two sync instances could run simultaneously.
bulk_permissions_repair.sh and container_data_export.sh had no lock at all.
All three are now consistent with every other script in Media/ and Tools/.
Script headers (6 files):
- Add OPERATIONAL SAFEGUARDS section to all discovery + emby sync scripts
- Add STATE FILES section to all 3 discovery scripts
- Add CONFIGURATION section to emby_to_*_sync.sh
- Rename USAGE → RUNTIME MODES in emby_to_*_sync.sh (doc standard)
- Fix SONARR_DISCOVERY_MONITOR_MODE default doc: "future" → "all"
Media/README-Media.md:
- Remove "WIP prototype" label; describe all 3 discovery scripts
- Add discovery scripts to scripts table and HOW THE SCRIPTS RELATE diagram
- Add weekly discovery schedule block to EXECUTION ORDER section
- Update intro to mention behavior-driven discovery
Media/Manual-Media.md:
- Add configuration reference sections for all 3 discovery scripts
- Add flag reference sections for all 3 discovery scripts
Tools/README-Tools.md:
- Add "Library Sync Bootstrap" category and emby_to_*_sync.sh to scripts table
- Add emby sync scripts to HOW THE SCRIPTS RELATE diagram
Tools/Manual-Tools.md:
- Update emby sync notes: search commands now fire immediately after add
README.md:
- Update Media folder description to include discovery
- Update weekly schedule to show discovery scripts
- Update README table entry for Media
future only monitors seasons not yet aired — shows with all existing
seasons would add as unmonitored with nothing to search. all monitors
everything so SeriesSearch actually finds episodes to grab.
- New: Media/playback_aware_sonarr_discovery.sh — two-stage TV show
discovery using Emby episode play history → TMDB TV recommendations.
Multi-user aware: diversity score (0-50) is primary seed driver so one
user binge-watching alone cannot dominate. Per-user episode cap (default 8)
limits single-user volume contribution. TMDB→TVDB via external_ids for
reliable Sonarr lookup. monitor=future by default.
- Add to WEEKLY_MAINTENANCE_SCRIPTS; Sonarr discovery config block in
master.conf (threshold=52, lookback=14d, max_adds=3, user_ep_cap=8).
- Fix: all arr add scripts were not triggering searches after adding.
Added _radarr_command/_lidarr_command/_sonarr_command helpers and fire
MoviesSearch/ArtistSearch/SeriesSearch immediately after each successful
add. Affects: playback_aware_radarr_discovery.sh, emby_to_lidarr_sync.sh,
emby_to_radarr_sync.sh, emby_to_sonarr_sync.sh.
- Stage 1 rewritten to use Emby activity log (playback.stop events)
with ItemId batch-fetching instead of per-library UserData sort;
SortBy=DatePlayed requires UserId context and errored server-wide
- Scoring simplified: recency (0-50) + play frequency across all
users (0-50); CommunityRating is not exposed by Emby Items API
so TMDB rating scoring moved entirely to Stage 2
- Batch IDs in groups of 100 — 1500+ unique item IDs in a 30-day
window exceeded GET URL limits on a single request
- Threshold lowered to 50 (from 60); with diverse-genre seeds,
recommendations rarely appear across multiple seeds so breadth
score stays at 10/40 — threshold 50 yields ~5 adds per run at
good quality (rating 7.0+, 1k+ votes)
- HOST1_TMDB_API_KEY moved to sit under Radarr section in
master_host1.conf where it logically belongs
- playback_aware_radarr_discovery.sh: new two-stage movie discovery
using recently watched Emby movies as seeds and TMDB recommendations
as the similarity engine; scores on recency + rating + vote count;
adds HOST*_TMDB_API_KEY alias to detect_hosts() in common.sh
- emby_to_sonarr_sync.sh: restrict to SONARR_EMBY_LIBRARIES allowlist
(Anime, Kids Shows, Stand-Up Comedy, TV shows) — excludes Youtube
and Recorded Sports libraries; add garbage title filter for multi-
season folder names (S01-S03) and bare bracket tags ([Prof])
- emby_to_radarr_sync.sh: add _is_episode_title filter to block anime
episode files stored in Movies library (underscore/dot patterns,
fansub bracket format, episode markers, codec metadata strings)
- emby_to_lidarr_sync.sh: add U+FFFD Unicode replacement character
filter alongside the existing ASCII ? filter for encoding corruption
- master.conf: RADARR_DISCOVERY_* settings, SONARR/RADARR_EMBY_LIBRARIES
allowlists, Radarr discovery added to WEEKLY_MAINTENANCE_SCRIPTS
Lidarr Discovery (playback_aware_lidarr_discovery.sh):
- Complete rewrite to two-stage pipeline: Stage 1 scores weekly Emby
plays → top seeds; Stage 2 runs Last.fm getSimilar on seeds → scores
candidates → adds top 0-5 to Lidarr
- Per-user influence cap (35%) prevents single listener dominating discovery
- Requires 2+ seeds to accept a candidate (single-seed skipped)
- Last.fm similarity limit 10 for tighter, higher-quality candidates
- 30-day reject cooldown history; MAX_ADDS=5 cap enforced
- Root folder fetched from Lidarr API at runtime (no config path)
- Added to WEEKLY_MAINTENANCE_SCRIPTS (uncommented)
Emby→arr sync tools (Tools/):
- emby_to_lidarr_sync.sh: finds Emby music artists not in Lidarr, adds them
- Dirty tag filter: comma-list, feat./ft., &, vs, " - " patterns skipped
- Root folder fetched from Lidarr API at runtime
- emby_to_sonarr_sync.sh: finds Emby series not in Sonarr, adds them
- TVDB ID matching with title fallback
- emby_to_radarr_sync.sh: finds Emby movies not in Radarr, adds them
- TMDB ID matching with title fallback
- All three: searchForMissing*: false (monitoring only, no searches triggered)
- All three documented in Tools/Manual-Tools.md
- downloaders_reset: connection check block before slskd API sections;
triggers PUT /api/v0/server reconnect if disconnected, polls 60s,
gates Stuck Searches and Dead Transfer Records on SLSKD_CONNECTED
- Sync all modified/new/deleted files from v2 refactor across Docker_Essentials,
Media, Monitors, Partnership, Rsync, Tools, Transcodes, unRAID_Essentials,
common.sh, master confs, and new Manual/README docs
Searches: grep was potentially matching nested IDs from search result
objects. Switch to splitting JSON at { boundaries and requiring
"searchText" presence — only top-level search objects have that field.
Transfers: DELETE /api/v0/transfers/downloads/{username} returns 405
(endpoint does not exist). Replace with per-user GET then per-file
DELETE /api/v0/transfers/downloads/{username}/{id}, using the same
{-split + state grep approach to extract terminal-state file IDs.
Fallback runs as a bare subprocess (cmdline: bash .../Fallback/fallback.sh),
not via /tmp/user.scripts, so user_scripts_stop.sh cannot find it.
Adding --stop mode closes that gap in the array shutdown sequence.
fallback.sh --stop: reads lock file (/tmp/unraid_locks/fallback.lock) to
get the running PID, sends SIGTERM (hits the existing trap that sets
FALLBACK_RUNNING=false), waits up to 10s, SIGKILLs if still alive.
master.conf: insert Fallback/fallback.sh --stop as step 2 in
ARRAY_STOP_SCRIPTS (after user_scripts_stop, before rsync_stop).
user_script_plug-in.sh: update ARRAY STOPPING block to show 5-step order
and add v2.5 changelog entry.
It's a Media/ individual script, not an orchestrator — belongs in the
individual scripts section only.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Add Old_Arch_Still_Works/ folder as staging area for scripts awaiting
native platform support or retirement. Python arr_cleanup.sh/.py parked
here until Unraid ships Python natively — fully built and config-driven,
ready to activate. continuous_scripts_status.sh retired from Tools.
Consistent naming with the watchdog family (system_watchdog, docker_watchdog).
File renamed, all RM_ config variables and internal references updated to RW_
across master.conf, master_host1.conf, common.sh, docker_watchdog.sh,
system_watchdog.sh, watchdog_orchestrator.sh, and sunday_morning_coffee_report.sh.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Introduce a four-layer self-healing stack replacing the continuous-loop watchdogs:
- resource_manager.sh (new): single-pass pressure reduction layer; throttles
SABnzbd/qBit at level 1, docker-pauses background containers at level 2,
docker-stops optional containers and signals docker_watchdog to defer at
level 3; graduated recovery with hysteresis
- watchdog_orchestrator.sh (new, Orchestrators/): runs resource_manager →
docker_watchdog → system_watchdog in sequence; intended for per-minute cron
via User Scripts; startup grace, acquire_lock to prevent pile-up, heartbeat
- docker_watchdog.sh: de-looped to single-pass; daemon strikes persisted to
state file across runs; cross-script coordination reads RM_STATE_FILE instead
of SYS_WATCHDOG_STATE_FILE
- system_watchdog.sh: de-looped to single-pass; stripped of all container
management (shutdown_non_essential_containers removed); reboot-only last resort
- master.conf: removed system_watchdog and docker_watchdog from
ARRAY_START_SCRIPTS; added WATCHDOG ORCHESTRATOR and RESOURCE MANAGER sections
- master_host1.conf: added RM_PAUSE_CONTAINERS and RM_STOP_CONTAINERS arrays
- common.sh: aliased RM_PAUSE_CONTAINERS and RM_STOP_CONTAINERS via detect_hosts()
- continuous_scripts_status.sh: moved to Tools/ (preserved for future use)
- sunday_morning_coffee_report.sh: watchdog section updated to use state file
mtime checks instead of is_running; added Resource Manager subsection;
fixed mem_shutdown grep filter pointing to wrong state file
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
- Rename array_start.sh → array_started.sh, array_stop.sh → array_stopping.sh
to clarify these are event-driven (array has started/is stopping), not imperative
- Update all references across 9 files (master.conf, user_script_plug-in.sh,
watchdogs, continuous_scripts_status.sh, claude_startup.sh)
- Add --remainder mode to docker_update.sh: updates all running containers
excluding daily containers, weekly sync-window containers (emby + critical-data),
and fallback coverage containers (FALLBACK_${MY_ID}_COVERS_${REMOTE_ID}_TIER*)
Fallback containers excluded because the remote server owns their version —
independent updates risk writeback incompatibility on handback
- weekly_sync_maintenance.sh calls docker_update.sh --remainder as final step
- git_pull_execute.sh: add safe.directory config to fix dubious ownership error
when running as root on a directory owned by uid 1000
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
common.sh:
- Add resolve_tailscale_ip() helper — tries `tailscale ip -4` first, falls back to
parsing `tailscale status` output; handles hosts where MagicDNS short-name resolution
is not active
- Add PARTNERSHIP_OWN_CONTAINERS alias in detect_hosts()
- Add aliasing for 4 Emby provisioning vars (PARTNERSHIP_PROVISION_EMBY_ADMIN,
PARTNERSHIP_EMBY_ADMIN_USER, PARTNERSHIP_EMBY_ADMIN_PASS, PARTNERSHIP_EMBY_PORT)
Partnership/partnership_manager.sh:
- Replace 9 bare `tailscale ip -4` calls with resolve_tailscale_ip()
- Add read_remote_conf_var() and read_remote_conf_array() — SSH to mirror, source its
own load_config.sh + detect_hosts(), return aliased variable; solves sparse-checkout
problem where HOST1 cannot read master_host2.conf directly
- Add derive_short_name() — strips unraid- prefix, capitalises first char
- Add cleanup_partner_containers() — removes partner containers via FolderView3 folder
if enabled, else falls back to FALLBACK_*_COVERS_*_TIER* arrays
- Add cleanup_owner_containers_on_mirror() — SSH to mirror, stops and removes containers
matching *-${OWNER_SHORT} naming convention
- Add start_own_stack() and start_mirror_own_stack() — restart own containers locally
or on mirror via SSH using PARTNERSHIP_OWN_CONTAINERS
- Add provision_emby_admin() — reads mirror credentials via read_remote_conf_var, checks
for username collision, creates user + sets password + grants admin policy via Emby API
- Add revoke_emby_admin() — looks up mirror username on local Emby, deletes via REST API
- Wire offboard paths (both mirror-initiated and owner-initiated) to call container
cleanup and stack restart; update --check finalisation paths accordingly
- Fix write_state_file in --onboard not gated on DRY_RUN (was writing ACTIVE state on
dry runs)
master_host1.conf:
- Add HOST1_PARTNERSHIP_OWN_CONTAINERS array
- Add partnership Emby provisioning config (toggle + port + per-host credentials)
master_host2.conf:
- Add HOST2_PARTNERSHIP_OWN_CONTAINERS array
- Add HOST2_PARTNERSHIP_EMBY_ADMIN_USER and HOST2_PARTNERSHIP_EMBY_ADMIN_PASS
Tailscale fix applied to:
- Initial_run/ssh_setup.sh (2 callsites)
- unRAID_Essentials/rsync_stop.sh (1 callsite)
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Gmer4Lfe
FailedProxy