Documents the known issue where an unclean shutdown leaves the --mount
bind-propagation entry in a broken state that prevents Emby from starting.
Recovery: remove mount → start → wait for load → re-add mount → restart.
Follows the same extraction pattern as partnership_offboard.sh.
partnership_manager.sh --transfer now exec-forwards to the new script.
Legacy inline block preserved under `if false` for reference.
Fixes in partnership_transfer.sh vs the original inline code:
- WebUI reconfiguration corrected: SSH to new owner (mirror) to set localhost;
reconfigure_local_webuis() on current server to point to new owner IP
- master.conf remote update uses scp push instead of self-SSH with wrong path
- SSH key uses $SSH_KEY directly (was using undefined {HOST*}_SSH_KEY expansion)
partnership_manager.sh:
- fix auto-offboard threshold: × 96 → × 48 (was triggering at 2× configured days)
- fix do_final_sync(): use CRITICAL_SYNC_SHARES if configured; fall back to hardcoded paths
- replace inline --check offboard blocks with background partnership_offboard.sh calls:
• owner path was missing remote container cleanup (steps 8+9) and blocklist
• mirror path was missing blocklist, SSH revocation, CRITICAL_RSYNC_ENABLED=false
• local grace_seconds outside a function (bash error) eliminated with the block removal
• 6hr Tailscale grace sleep was blocking the cron for the full grace window
partnership_offboard.sh:
- reorder owner path: state write moved from step 5 to step 10 (last) so re-running
after a mid-offboard crash restarts from scratch instead of exiting as INACTIVE;
state is now written before the Tailscale grace sleep so --check stops re-triggering
- add FolderView3 remote folder cleanup in step 7 (mirror's fallback folder persisted)
- track step 1+2 outcome with STEP_STOP_OK / STEP_SYNC_OK; summary now reflects actual
result instead of hardcoding ✅ regardless of dry-run or failure
- update header step list to match new step order
master.conf:
- reorder ORCHESTRATORS section by run frequency; add missing entries
- remove heartbeat vars (watchdogs no longer run in continuous cycle)
- remove dead REMOTE_DOCKER_RETRY_WAIT and display-only watchdog interval vars
- move INTERMEDIATE_RSYNC_ENABLED into RSYNC two-tier block; reorder by frequency
common.sh:
- add missing info() and success() definitions — both were called throughout but never defined
- update output tier description to list all 5 functions
- fix double-icon in check_local_disk_temps() error/warn calls
- fix smashed curl/ssh command lines in notify_emby_scan() and check_remote_docker_daemon()
- update check_rsync_enabled() comment to include INTERMEDIATE, ordered by frequency
emby_to_radarr_sync.sh / emby_to_sonarr_sync.sh:
- wire up RADARR_EMBY_LIBRARIES / SONARR_EMBY_LIBRARIES; empty array now scans all libraries
appdata-Failover → appdata-Fallback:
All .md files updated — scripts and confs already used appdata-Fallback.
Share was renamed; docs were the only stale references.
critical_sync_maintenance.sh: */15 → */30, 96/day → 48/day:
critical_sync_maintenance.sh header, master.conf, host1/2.conf comments,
Orchestrators README, User Script Plug-in README, Partnership README/manual/script,
Docker_Essentials README/downloaders_reset.sh, Fallback README,
weekly_sync_maintenance.sh, user_script_plug-in.sh all updated.
rsync_emby_failover User Script superseded:
Emby dirty sync was already in HOST1_CRITICAL_SYNC_SHARES — it now runs inside
critical_sync_maintenance.sh every 30min alongside auth sync and partnership check.
Section in User Script Plug-in README marked for deletion from plugin.
transcode_management.sh: */3 → */7:
transcode_management.sh, transcode_manager.sh, transcode_cleanup.sh headers,
Transcodes README/manual, Orchestrators README, User Script Plug-in README,
user_script_plug-in.sh all updated.
Update the Three-Tier Response and RAM Tiers sections to reflect the
resource_watchdog/stability_watchdog split: MEM_WARN_GB/MEM_SHUTDOWN_GB/
MEM_RECOVER_GB moved to RW_RAM_SOFT/MEDIUM/HARD/RECOVER_GB in resource_watchdog;
stability_watchdog now only holds SYS_WATCHDOG_MEM_GB (reboot trigger). Update
required containers reference from SYS_WATCHDOG_REQUIRED_CONTAINERS (removed)
to HOST*_WATCHDOG_REQUIRED_CONTAINERS in host*.conf.
array_start.sh → array_started.sh (bulk rename — actual file has always been
array_started.sh; docs consistently used the wrong name across 9 files)
failover.sh → fallback.sh (bulk rename — script is Fallback/fallback.sh;
docs used the old name throughout)
FAILOVER_HOST2_RUNS_FOR_HOST1_TIER1 → FALLBACK_HOST2_COVERS_HOST1_TIER1
(README.md — wrong variable name, actual vars use FALLBACK_ prefix and
COVERS_ not RUNS_FOR_)
docker_watchdog.sh attribution: "started by array_started.sh" → "called every
minute by watchdog_orchestrator.sh" (docker_watchdog is not in ARRAY_START_SCRIPTS)
server_reboot.sh shutdown sequence: step 7 "rc.docker stop" → calls
array_stopping.sh (user_scripts, fallback, rsync, mover, docker_container_stop)
Rsync manual ARRAY_START_SCRIPTS list: removed stability_watchdog and
docker_watchdog (now in watchdog_orchestrator); failover → fallback
Failover/ was renamed to Fallback/ — all script paths and README links
updated across README.md, Orchestrators README, and User Script Plug-in README.
array_start.sh → array_started.sh in Orchestrators scripts table (wrong name).
fallback_state_reset.sh: script is referenced in User Script Plug-in README
but does not exist in Fallback/. Entry updated to document it as not yet built
and provides the manual workaround (direct state file write).
Note: /mnt/user/appdata-Failover/ share path references left unchanged —
that is the actual unRAID share name on the server, not the script folder.
arr_sync.sh bidirectional union replaced the old split-ownership model.
Old: HOST1 owns Movies/TV/Music, HOST2 owns Anime — each arr manages
different shares to avoid conflicts.
New: arr_sync.sh keeps all arr databases in union. Either server can
download to any share at any time. The union is the source of truth.
Auth stack references left intact — HOST1 still owns auth config
(one-way sync to HOST2). Only media library ownership language updated.
Files changed:
README.md — server block + share annotations
Fallback/README-Fallback.md — role line + Split Source of Truth section
Orchestrators/README-Orchestrators — daily sync share comments
Rsync/Manual-Rsync.md — DEFAULT_RSYNC_OPTS comment block
Media/README-Media.md — HOST AWARENESS section
Old model: HOST1 owns Movies/TV/Music, HOST2 owns Anime — each arr
manages different shares to prevent conflicts.
New model: arr_sync.sh syncs all arr databases bidirectionally before
every rsync cycle. Either server can download to any share at any time.
The databases converge (union, not overwrite), then rsync spreads files
additively. Neither server owns a share. The union is the source of truth.
Also updates share name annotations and removes the "read-only mirror"
framing from the share naming section.
Two-gate design: server must have ≥30 days uptime AND last run must
be ≥30 days ago. Both gates must pass before any scripts fire. Called
daily at 3am via cron — script self-gates, calling more often is safe.
State file on /boot/config (survives reboots): the interval gate is
independent of uptime. A reboot resets uptime but does not reset when
maintenance last ran — both gates must independently pass.
MONTHLY_MAINTENANCE_SCRIPTS added to master.conf in ORCHESTRATORS
section. zfs_pool_scrub.sh and smart_long_test.sh listed but commented
(neither script exists yet). Also commits mesh_monitor.sh move to
Monitors/ that was staged from prior session.
Supports --force to bypass both gates for manual runs.
Checks:
1. Internet reachability (gates all remaining checks)
2. DDNS sync — public IP vs Cloudflare DNS record, restarts Gmer4Lfe.com container on mismatch
3. Tailscale status — notify only, no auto-restart
4. NPM proxy — external curl to https://gmer4lfe.com, 2-strike system before NginxProxyManager restart
Config: master.conf NETWORK_WATCHDOG_* block, host1.conf HOST1_NETWORK_WATCHDOG_* values
Added to SYSTEM_WATCHDOG_SCRIPTS — called by system_watchdog.sh each cycle
Fix: storage_watchdog.sh was calling get_strikes/set_strikes without defining them —
added local definitions (same pattern as docker_watchdog.sh and stability_watchdog.sh)
Watchdog renames and moves:
system_watchdog.sh → stability_watchdog.sh (last line of defense — reboots)
storage_watchdog.sh → Watchdogs/System/storage_watchdog.sh
webgui_restart.sh → Watchdogs/System/webgui_watchdog.sh (renamed to match folder convention)
New thin orchestrator:
Watchdogs/system_watchdog.sh — runs SYSTEM_WATCHDOG_SCRIPTS from master.conf
Sits between docker_watchdog and stability_watchdog in the orchestrator tier chain
System/ subfolder is the growth seam for future system component watchdogs
master.conf:
WATCHDOG_ORCHESTRATOR_SCRIPTS updated — storage removed, system_watchdog added as tier
SYSTEM_WATCHDOG_SCRIPTS array added — storage + webgui
server_reboot.sh:
Calls array_stopping.sh before VM shutdown for guaranteed safe array stop
Removed raw rc.docker stop and exit trap — orchestrator owns container shutdown
master.conf, host1.conf, host2.conf relocated from repo root to Configurations/.
load_config.sh updated to source from the new path — no other scripts need changes.
Note: sparse checkout rules on HOST1 and HOST2 need updating to use
Configurations/host1.conf and Configurations/host2.conf respectively.
README.md: header updated to VARAVERK. All "User Scripts plugin" references
replaced with "Arbiter". Project name section: "Working name: Arbiter" → "Varaverk".
Gitea corrected to Tier 1 (was shown as Tier 3, updated to Tier 1 this session).
master.conf: GITEA_REPO_PATH updated FailedProxy → Varaverk org name.
Script internals (user_scripts_stop.sh, script headers, other docs) reference
the /tmp/user.scripts working path which is Arbiter-specific — those will be
updated in a separate pass once Arbiter's working directory is confirmed.
Partnership/gitea_ssh_setup.sh: generates ed25519 keypair, registers public key
with Gitea API (POST /api/v1/user/keys), tests SSH connection, writes ~/.ssh/config
entry. Idempotent — skips steps already done. --force regenerates and re-registers.
Resolves Gitea endpoint from container IP first, falls back to GITEA_DOMAIN.
common.sh: alias HOST*_GITEA_API_TOKEN → GITEA_API_TOKEN in detect_hosts().
Add to detect_hosts() doc comment.
master.conf: add GITEA_HTTP_PORT=3000 for API endpoint construction.
host1.conf: add HOST1_GITEA_API_TOKEN (fill in from Gitea Settings → Applications).
Add my-Gitea.xml to HOST1_PARTNERSHIP_AUTH_STACK — onboard pushes it to HOST2.
host2.conf: add Gitea to FALLBACK_HOST2_COVERS_HOST1_TIER1 — starts immediately
when HOST1 goes down, making the source of truth reachable independently of
HOST1's auth stack.
host1.conf: HOST1_RAMDISK_SIZE 8G→10G, WARN_GB 6.8→8.5, LOW_GB 5.5→7.
ramdisk_stop.sh: new Tools/ script — redirects symlink to SSD before unmount
so Emby keeps writing during maintenance, auto-falls back to lazy umount
when only directory handles are open (no active writes).
common.sh: fix `info` call at line 595 (undefined) → `log`.
Tools docs: added ramdisk_stop.sh to README table, categories, diagram, and Manual.
watchdog_orchestrator.sh stays in Orchestrators/ — it's an orchestrator, not a watchdog.
watchdog_skip_list_manager.sh stays in Tools/ — it's a management utility.
Only the 4 watchdog scripts belong in Watchdogs/:
docker_watchdog.sh, resource_watchdog.sh, storage_watchdog.sh, system_watchdog.sh
No setup banners to remove (Tools folder was already clean). Fix log→echo
on summary status/conclusion lines across container_data_export, emby_database_repair,
recreate_shares, zfs_pool_scrub, and fallback_state_reset (safety check + two summary
detail lines). Fix log→echo on status display lines in watchdog_skip_list_manager
(skip list empty, restart history lines). Move per-item "to add" preview echoes to
log in emby_to_lidarr/radarr/sonarr_sync; fix nothing-to-do log→echo in all three.
Add output tiers section to Manual-Tools.md.
Fix log→echo on success status line in summary block. Add output tiers
section to Manual-Rsync.md. No setup banner (script was already clean);
all section headers and transfer identity block correctly remain echo.
Remove setup banners from all 8 scripts that had them (discovery scripts
use comment-only setup sections). Move per-item/detail echoes to log in
arr_sync, lidarr_missing_art, lidarr/radarr/sonarr_cleanup (scan progress,
API query status, root/orphan config lines), and per-title lines in
radarr_tmdb_removed / sonarr_tvdb_removed. Fix log→echo on clean-status
summary lines in arrs_failed_stalled_recovery, media_cleaner, and both
removed-title scripts. Add output tiers section to Manual-Media.md.
Consistent two-tier output model across the entire folder:
- Per-container banners, action lines, and list details → log (ENABLE_LOGGING=true only)
- Section headers, summaries, counts, and status conclusions → echo (always visible)
- Warnings and errors always visible regardless of log setting
- Blank echo lines inside loops removed
All scripts: added Lock Acquisition and Host Detection entries to OPERATIONAL SAFEGUARDS.
Setup banners removed from docker_watchdog.sh, docker_weekly_restart.sh,
downloaders_reset.sh (noise before any work happens).
docker_watchdog.sh: removed success "Running as root" / "Docker found" setup lines;
info() → log() for daemon-recovered and per-cycle header; removed per-cycle echo separator.
docker_update.sh + docker_update_remaining.sh: docker pull stdout suppressed when
ENABLE_LOGGING=false to prevent orphaned Status: lines appearing without container context;
restructured pull block to use PIPESTATUS for exit code capture.
downloaders_reset.sh: 11× info() → log() for all per-item API operation lines.
docker_weekly_restart.sh: removed duplicate restart-order echo (build_restart_order
already logs it internally).
Manual-Docker_Essentials.md: documented the two-tier output model and watchdog
silent-when-healthy exception in the flag reference section.
22 scripts used docker without verifying it's present. Consistent with the
pattern already established in Docker_Essentials scripts. Added after acquire_lock
and before detect_hosts in each script's setup section.
docker_container_stop, docker_update, docker_update_remaining — concurrent Docker
operations on the same containers would conflict; now locked.
rsync.sh — two rsync processes running against the same share simultaneously
would produce incomplete or corrupted mirrors; now locked.
partnership_onboard, ssh_setup — one-shot setup scripts that mutate SSH config and
deploy containers; concurrent runs would produce undefined state; now locked.
Old_Arch_Still_Works: arr_cleanup, continuous_scripts_status — legacy scripts still
sourcing load_config.sh; added lock for consistency even in old-arch context.
partnership_manager.sh intentionally left unchanged — it uses a conditional lock
that excludes read-only "check" mode and "offboard" mode (which delegates to
partnership_offboard.sh, which has its own lock).
arr_sync.sh had acquire_lock documented in OPERATIONAL SAFEGUARDS header but the
call was never present in the code — two sync instances could run simultaneously.
bulk_permissions_repair.sh and container_data_export.sh had no lock at all.
All three are now consistent with every other script in Media/ and Tools/.
Script headers (6 files):
- Add OPERATIONAL SAFEGUARDS section to all discovery + emby sync scripts
- Add STATE FILES section to all 3 discovery scripts
- Add CONFIGURATION section to emby_to_*_sync.sh
- Rename USAGE → RUNTIME MODES in emby_to_*_sync.sh (doc standard)
- Fix SONARR_DISCOVERY_MONITOR_MODE default doc: "future" → "all"
Media/README-Media.md:
- Remove "WIP prototype" label; describe all 3 discovery scripts
- Add discovery scripts to scripts table and HOW THE SCRIPTS RELATE diagram
- Add weekly discovery schedule block to EXECUTION ORDER section
- Update intro to mention behavior-driven discovery
Media/Manual-Media.md:
- Add configuration reference sections for all 3 discovery scripts
- Add flag reference sections for all 3 discovery scripts
Tools/README-Tools.md:
- Add "Library Sync Bootstrap" category and emby_to_*_sync.sh to scripts table
- Add emby sync scripts to HOW THE SCRIPTS RELATE diagram
Tools/Manual-Tools.md:
- Update emby sync notes: search commands now fire immediately after add
README.md:
- Update Media folder description to include discovery
- Update weekly schedule to show discovery scripts
- Update README table entry for Media
future only monitors seasons not yet aired — shows with all existing
seasons would add as unmonitored with nothing to search. all monitors
everything so SeriesSearch actually finds episodes to grab.
- New: Media/playback_aware_sonarr_discovery.sh — two-stage TV show
discovery using Emby episode play history → TMDB TV recommendations.
Multi-user aware: diversity score (0-50) is primary seed driver so one
user binge-watching alone cannot dominate. Per-user episode cap (default 8)
limits single-user volume contribution. TMDB→TVDB via external_ids for
reliable Sonarr lookup. monitor=future by default.
- Add to WEEKLY_MAINTENANCE_SCRIPTS; Sonarr discovery config block in
master.conf (threshold=52, lookback=14d, max_adds=3, user_ep_cap=8).
- Fix: all arr add scripts were not triggering searches after adding.
Added _radarr_command/_lidarr_command/_sonarr_command helpers and fire
MoviesSearch/ArtistSearch/SeriesSearch immediately after each successful
add. Affects: playback_aware_radarr_discovery.sh, emby_to_lidarr_sync.sh,
emby_to_radarr_sync.sh, emby_to_sonarr_sync.sh.
- Stage 1 rewritten to use Emby activity log (playback.stop events)
with ItemId batch-fetching instead of per-library UserData sort;
SortBy=DatePlayed requires UserId context and errored server-wide
- Scoring simplified: recency (0-50) + play frequency across all
users (0-50); CommunityRating is not exposed by Emby Items API
so TMDB rating scoring moved entirely to Stage 2
- Batch IDs in groups of 100 — 1500+ unique item IDs in a 30-day
window exceeded GET URL limits on a single request
- Threshold lowered to 50 (from 60); with diverse-genre seeds,
recommendations rarely appear across multiple seeds so breadth
score stays at 10/40 — threshold 50 yields ~5 adds per run at
good quality (rating 7.0+, 1k+ votes)
- HOST1_TMDB_API_KEY moved to sit under Radarr section in
master_host1.conf where it logically belongs
- playback_aware_radarr_discovery.sh: new two-stage movie discovery
using recently watched Emby movies as seeds and TMDB recommendations
as the similarity engine; scores on recency + rating + vote count;
adds HOST*_TMDB_API_KEY alias to detect_hosts() in common.sh
- emby_to_sonarr_sync.sh: restrict to SONARR_EMBY_LIBRARIES allowlist
(Anime, Kids Shows, Stand-Up Comedy, TV shows) — excludes Youtube
and Recorded Sports libraries; add garbage title filter for multi-
season folder names (S01-S03) and bare bracket tags ([Prof])
- emby_to_radarr_sync.sh: add _is_episode_title filter to block anime
episode files stored in Movies library (underscore/dot patterns,
fansub bracket format, episode markers, codec metadata strings)
- emby_to_lidarr_sync.sh: add U+FFFD Unicode replacement character
filter alongside the existing ASCII ? filter for encoding corruption
- master.conf: RADARR_DISCOVERY_* settings, SONARR/RADARR_EMBY_LIBRARIES
allowlists, Radarr discovery added to WEEKLY_MAINTENANCE_SCRIPTS
Lidarr Discovery (playback_aware_lidarr_discovery.sh):
- Complete rewrite to two-stage pipeline: Stage 1 scores weekly Emby
plays → top seeds; Stage 2 runs Last.fm getSimilar on seeds → scores
candidates → adds top 0-5 to Lidarr
- Per-user influence cap (35%) prevents single listener dominating discovery
- Requires 2+ seeds to accept a candidate (single-seed skipped)
- Last.fm similarity limit 10 for tighter, higher-quality candidates
- 30-day reject cooldown history; MAX_ADDS=5 cap enforced
- Root folder fetched from Lidarr API at runtime (no config path)
- Added to WEEKLY_MAINTENANCE_SCRIPTS (uncommented)
Emby→arr sync tools (Tools/):
- emby_to_lidarr_sync.sh: finds Emby music artists not in Lidarr, adds them
- Dirty tag filter: comma-list, feat./ft., &, vs, " - " patterns skipped
- Root folder fetched from Lidarr API at runtime
- emby_to_sonarr_sync.sh: finds Emby series not in Sonarr, adds them
- TVDB ID matching with title fallback
- emby_to_radarr_sync.sh: finds Emby movies not in Radarr, adds them
- TMDB ID matching with title fallback
- All three: searchForMissing*: false (monitoring only, no searches triggered)
- All three documented in Tools/Manual-Tools.md
Gmer4Lfe