audit echo vs log across all scripts — outcomes always visible, verbose for per-item loops
This commit is contained in:
Gmer4Lfe
@@ -0,0 +1,151 @@
|
||||
# Varaverk — Claude Code Context
|
||||
|
||||
## Working Rules (read first)
|
||||
|
||||
- **Workspace is always** `/boot/config/plugins/varaverk` — every edit goes here.
|
||||
- **Never touch** `/mnt/user/Important Shit/Git/Development/Varaverk` — stale dev folder, ignore it.
|
||||
- **No Co-Authored-By** in commit messages unless explicitly asked.
|
||||
- **No comments** unless the WHY is genuinely non-obvious.
|
||||
- The `.plg` symlinks the installed plugin location directly to this workspace — one copy, no drift.
|
||||
|
||||
---
|
||||
|
||||
## Project: What Varaverk Is
|
||||
|
||||
Self-healing, self-maintaining, mutually-redundant two-server Unraid home media ecosystem.
|
||||
One codebase runs on both servers. No primary/standby — both run independently and cover each other.
|
||||
|
||||
**HOST1 — unRAID-Gmer4Lfe** (`gmer4lfe@gmail.com`)
|
||||
- Hardware: Threadripper 1950X, 128 GB RAM, ZFS cache pools
|
||||
- Domain: Gmer4Lfe.com
|
||||
- Runs: full arr stack (Sonarr/Radarr/Lidarr), auth stack (source of truth), Emby primary
|
||||
|
||||
**HOST2 — unRAID-Jayred365**
|
||||
- Hardware: Intel i5 10th gen, 64 GB RAM
|
||||
- Domain: Gmer4Lfe.us
|
||||
- Status: being rebuilt — most host2.conf sections scaffolded, not yet fully online
|
||||
|
||||
Networking between hosts: Tailscale mesh. No hardcoded IPs — hostnames resolve via Tailscale.
|
||||
|
||||
---
|
||||
|
||||
## Configuration System (three-file model)
|
||||
|
||||
Every script sources all three at startup:
|
||||
|
||||
```
|
||||
master.conf ← shared: thresholds, toggles, profiles, orchestrator job lists
|
||||
host1.conf ← HOST1 credentials, shares, container names, keys
|
||||
host2.conf ← HOST2 credentials, shares, container names, keys
|
||||
```
|
||||
|
||||
Sparse checkout (git) means each server only pulls its own `host*.conf`.
|
||||
HOST1 never sees HOST2 credentials and vice versa.
|
||||
|
||||
**Rule:** thresholds/toggles → `master.conf`; credentials/paths/container names → `host*.conf`.
|
||||
|
||||
`detect_hosts()` in `common.sh` matches `$(hostname)` against `HOST1`/`HOST2` in `master.conf`
|
||||
and sets `MY_ID` / `REMOTE_ID` for the rest of the script.
|
||||
|
||||
---
|
||||
|
||||
## Platform Adapter Layer
|
||||
|
||||
`Plugin/unraid/adapter.sh` isolates all OS-specific calls.
|
||||
Scripts never branch on OS directly — always call adapter functions.
|
||||
This is intentional architecture — don't bypass it.
|
||||
|
||||
---
|
||||
|
||||
## Key Paths
|
||||
|
||||
| Path | Purpose |
|
||||
|------|---------|
|
||||
| `master.conf` | Shared config — all thresholds, toggles, profiles |
|
||||
| `host1.conf` / `host2.conf` | Per-host credentials, shares, container lists |
|
||||
| `common.sh` | Shared functions — `detect_hosts()`, `log()`, `notify()`, etc. |
|
||||
| `load_config.sh` | Sources all three conf files + common.sh |
|
||||
| `State_Files/` | Runtime state (watchdogs, fallback, transcode) — survives reboots |
|
||||
| `data/` | Historical logs and stats |
|
||||
| `Plugin/unraid/` | Unraid WebGUI plugin (PHP pages, API endpoints, adapter) |
|
||||
| `Orchestrators/` | Top-level schedulers (array_started, daily, weekly, watchdog) |
|
||||
| `Watchdogs/` | docker_watchdog, system_watchdog, resource_watchdog, stability |
|
||||
| `Fallback/` | Mutual container failover logic |
|
||||
| `Rsync/` | rsync.sh + profile system |
|
||||
| `Media/` | Arr cleanup, discovery, permissions, play state sync |
|
||||
| `Tools/` | Manual one-off tools including `claude_startup.sh` |
|
||||
|
||||
---
|
||||
|
||||
## Orchestrator Schedule
|
||||
|
||||
| When | What |
|
||||
|------|------|
|
||||
| Array start | `Orchestrators/array_started.sh` → runs `ARRAY_START_SCRIPTS` |
|
||||
| Every minute | `watchdog_orchestrator.sh` → resource → docker → system → stability watchdogs |
|
||||
| Every 30 min | `critical_sync_maintenance.sh` → downloaders_reset, play_state_sync, critical rsync |
|
||||
| Every 4 hours | `intermediate_sync_maintenance.sh` → arr_sync, arrs_failed_stalled_recovery |
|
||||
| Daily 1am | `daily_sync_maintenance.sh` → git pull, permissions, cleaners, arr cleanup, docker updates |
|
||||
| Sunday 2:30am | `weekly_sync_maintenance.sh` → full Emby + Critical-Data sync, weekly restarts |
|
||||
| Sunday 3am+ | `monthly_maintenance.sh` (self-gated on 30-day uptime) → ZFS scrub, SMART tests |
|
||||
| Sunday 7am | `sunday_morning_coffee_report.sh` → ZFS, SMART, certs, backup verify, bandwidth, Emby report |
|
||||
|
||||
---
|
||||
|
||||
## Rsync Toggle State (current)
|
||||
|
||||
```bash
|
||||
RSYNC_ENABLED=true
|
||||
CRITICAL_RSYNC_ENABLED=true
|
||||
INTERMEDIATE_RSYNC_ENABLED=true
|
||||
DAILY_RSYNC_ENABLED=false # HOST2 rebuild in progress — re-enable when ready
|
||||
WEEKLY_RSYNC_ENABLED=true
|
||||
FALLBACK_RSYNC_ENABLED=true
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Fallback System
|
||||
|
||||
`fallback.sh` runs continuously from array start.
|
||||
States: `NORMAL | FALLBACK | NO_INTERNET | DARK`
|
||||
|
||||
DDNS rules are absolute:
|
||||
- Internet loss → stop own DDNS immediately
|
||||
- Failover → start remote's DDNS as Tier 1 first
|
||||
- Handback → stop remote DDNS → rsync → start containers → start local DDNS last
|
||||
|
||||
Tier delays before activating higher tiers are in `host*.conf` (`HOST1_TIER*_DELAY`, `HOST2_TIER*_DELAY`).
|
||||
|
||||
---
|
||||
|
||||
## Port Notes
|
||||
|
||||
- **NPM admin API (`HOST1_NPM_URL`)** — port **7818**. Port 81 is the partnership WebUI port (`HOST1_PARTNERSHIP_AUTH_WEBUIS`), not the API. Easy to confuse.
|
||||
- **HOST1_NETWORK_WATCHDOG_NPM_URL** — external HTTPS domain, completely separate from the admin API.
|
||||
|
||||
## Known Gaps / Active Work
|
||||
|
||||
- HOST2 NPM/lldap credentials (`HOST2_NPM_USER`, `HOST2_NPM_PASS`, `HOST2_LLDAP_PASS`) are empty in `host2.conf` — fill in when HOST2 is back online.
|
||||
- `PARTNERSHIP_ENABLED=false` — not yet active.
|
||||
- `FALLBACK_ENABLED=true` — fallback is running.
|
||||
- `DAILY_RSYNC_ENABLED=false` — paused during HOST2 rebuild.
|
||||
|
||||
---
|
||||
|
||||
## Claude Code Persistence on Unraid
|
||||
|
||||
`/root` is a RAM filesystem — wiped on every reboot.
|
||||
`Tools/claude_startup.sh` runs at array start (via `ARRAY_START_SCRIPTS`) and:
|
||||
- Symlinks `/root/.claude` → `/mnt/user/appdata/claude-code/.claude`
|
||||
- Symlinks `/root/.local/share/claude` → `/mnt/user/appdata/claude-code/local/share/claude`
|
||||
- Symlinks `/root/CLAUDE.md` → `/boot/config/plugins/varaverk/CLAUDE.md` (this file)
|
||||
|
||||
This file lives on `/boot` (USB flash) and is always available regardless of array state.
|
||||
|
||||
---
|
||||
|
||||
## Commit Style
|
||||
|
||||
Plain, concise messages. No Co-Authored-By trailers. No bullet-point summaries in the body.
|
||||
One sentence on the why, not the what.
|
||||
@@ -0,0 +1,150 @@
|
||||
# Varaverk — Claude Code Context
|
||||
|
||||
## Working Rules (read first)
|
||||
|
||||
- **Workspace is always** `/boot/config/plugins/varaverk` — every edit goes here.
|
||||
- **Never touch** `/mnt/user/Important Shit/Git/Development/Varaverk` — stale dev folder, ignore it.
|
||||
- **No Co-Authored-By** in commit messages unless explicitly asked.
|
||||
- **No comments** unless the WHY is genuinely non-obvious.
|
||||
- The `.plg` symlinks the installed plugin location directly to this workspace — one copy, no drift.
|
||||
|
||||
---
|
||||
|
||||
## Project: What Varaverk Is
|
||||
|
||||
Self-healing, self-maintaining, mutually-redundant two-server Unraid home media ecosystem.
|
||||
One codebase runs on both servers. No primary/standby — both run independently and cover each other.
|
||||
|
||||
**HOST1 — unRAID-Gmer4Lfe** (`gmer4lfe@gmail.com`)
|
||||
- Hardware: Threadripper 1950X, 128 GB RAM, ZFS cache pools
|
||||
- Domain: Gmer4Lfe.com
|
||||
- Runs: full arr stack (Sonarr/Radarr/Lidarr), auth stack (source of truth), Emby primary
|
||||
|
||||
**HOST2 — unRAID-Jayred365**
|
||||
- Hardware: Intel i5 10th gen, 64 GB RAM
|
||||
- Domain: Gmer4Lfe.us
|
||||
- Status: being rebuilt — most host2.conf sections scaffolded, not yet fully online
|
||||
|
||||
Networking between hosts: Tailscale mesh. No hardcoded IPs — hostnames resolve via Tailscale.
|
||||
|
||||
---
|
||||
|
||||
## Configuration System (three-file model)
|
||||
|
||||
Every script sources all three at startup:
|
||||
|
||||
```
|
||||
master.conf ← shared: thresholds, toggles, profiles, orchestrator job lists
|
||||
host1.conf ← HOST1 credentials, shares, container names, keys
|
||||
host2.conf ← HOST2 credentials, shares, container names, keys
|
||||
```
|
||||
|
||||
Sparse checkout (git) means each server only pulls its own `host*.conf`.
|
||||
HOST1 never sees HOST2 credentials and vice versa.
|
||||
|
||||
**Rule:** thresholds/toggles → `master.conf`; credentials/paths/container names → `host*.conf`.
|
||||
|
||||
`detect_hosts()` in `common.sh` matches `$(hostname)` against `HOST1`/`HOST2` in `master.conf`
|
||||
and sets `MY_ID` / `REMOTE_ID` for the rest of the script.
|
||||
|
||||
---
|
||||
|
||||
## Platform Adapter Layer
|
||||
|
||||
`Plugin/unraid/adapter.sh` isolates all OS-specific calls.
|
||||
Scripts never branch on OS directly — always call adapter functions.
|
||||
This is intentional architecture — don't bypass it.
|
||||
|
||||
---
|
||||
|
||||
## Key Paths
|
||||
|
||||
| Path | Purpose |
|
||||
|------|---------|
|
||||
| `master.conf` | Shared config — all thresholds, toggles, profiles |
|
||||
| `host1.conf` / `host2.conf` | Per-host credentials, shares, container lists |
|
||||
| `common.sh` | Shared functions — `detect_hosts()`, `log()`, `notify()`, etc. |
|
||||
| `load_config.sh` | Sources all three conf files + common.sh |
|
||||
| `State_Files/` | Runtime state (watchdogs, fallback, transcode) — survives reboots |
|
||||
| `data/` | Historical logs and stats |
|
||||
| `Plugin/unraid/` | Unraid WebGUI plugin (PHP pages, API endpoints, adapter) |
|
||||
| `Orchestrators/` | Top-level schedulers (array_started, daily, weekly, watchdog) |
|
||||
| `Watchdogs/` | docker_watchdog, system_watchdog, resource_watchdog, stability |
|
||||
| `Fallback/` | Mutual container failover logic |
|
||||
| `Rsync/` | rsync.sh + profile system |
|
||||
| `Media/` | Arr cleanup, discovery, permissions, play state sync |
|
||||
| `Tools/` | Manual one-off tools including `claude_startup.sh` |
|
||||
|
||||
---
|
||||
|
||||
## Orchestrator Schedule
|
||||
|
||||
| When | What |
|
||||
|------|------|
|
||||
| Array start | `Orchestrators/array_started.sh` → runs `ARRAY_START_SCRIPTS` |
|
||||
| Every minute | `watchdog_orchestrator.sh` → resource → docker → system → stability watchdogs |
|
||||
| Every 30 min | `critical_sync_maintenance.sh` → downloaders_reset, play_state_sync, critical rsync |
|
||||
| Every 4 hours | `intermediate_sync_maintenance.sh` → arr_sync, arrs_failed_stalled_recovery |
|
||||
| Daily 1am | `daily_sync_maintenance.sh` → git pull, permissions, cleaners, arr cleanup, docker updates |
|
||||
| Sunday 2:30am | `weekly_sync_maintenance.sh` → full Emby + Critical-Data sync, weekly restarts |
|
||||
| Sunday 3am+ | `monthly_maintenance.sh` (self-gated on 30-day uptime) → ZFS scrub, SMART tests |
|
||||
| Sunday 7am | `sunday_morning_coffee_report.sh` → ZFS, SMART, certs, backup verify, bandwidth, Emby report |
|
||||
|
||||
---
|
||||
|
||||
## Rsync Toggle State (current)
|
||||
|
||||
```bash
|
||||
RSYNC_ENABLED=true
|
||||
CRITICAL_RSYNC_ENABLED=true
|
||||
INTERMEDIATE_RSYNC_ENABLED=true
|
||||
DAILY_RSYNC_ENABLED=true
|
||||
WEEKLY_RSYNC_ENABLED=true
|
||||
FALLBACK_RSYNC_ENABLED=true
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Fallback System
|
||||
|
||||
`fallback.sh` runs continuously from array start.
|
||||
States: `NORMAL | FALLBACK | NO_INTERNET | DARK`
|
||||
|
||||
DDNS rules are absolute:
|
||||
- Internet loss → stop own DDNS immediately
|
||||
- Failover → start remote's DDNS as Tier 1 first
|
||||
- Handback → stop remote DDNS → rsync → start containers → start local DDNS last
|
||||
|
||||
Tier delays before activating higher tiers are in `host*.conf` (`HOST1_TIER*_DELAY`, `HOST2_TIER*_DELAY`).
|
||||
|
||||
---
|
||||
|
||||
## Port Notes
|
||||
|
||||
- **NPM admin API (`HOST1_NPM_URL`)** — port **7818**. Port 81 is the partnership WebUI port (`HOST1_PARTNERSHIP_AUTH_WEBUIS`), not the API. Easy to confuse.
|
||||
- **HOST1_NETWORK_WATCHDOG_NPM_URL** — external HTTPS domain, completely separate from the admin API.
|
||||
|
||||
## Known Gaps / Active Work
|
||||
|
||||
- HOST2 NPM/lldap credentials (`HOST2_NPM_USER`, `HOST2_NPM_PASS`, `HOST2_LLDAP_PASS`) are empty in `host2.conf` — fill in when HOST2 is back online.
|
||||
- `PARTNERSHIP_ENABLED=false` — not yet active.
|
||||
- `FALLBACK_ENABLED=true` — fallback is running.
|
||||
|
||||
---
|
||||
|
||||
## Claude Code Persistence on Unraid
|
||||
|
||||
`/root` is a RAM filesystem — wiped on every reboot.
|
||||
`Tools/claude_startup.sh` runs at array start (via `ARRAY_START_SCRIPTS`) and:
|
||||
- Symlinks `/root/.claude` → `/mnt/user/appdata/claude-code/.claude`
|
||||
- Symlinks `/root/.local/share/claude` → `/mnt/user/appdata/claude-code/local/share/claude`
|
||||
- Symlinks `/root/CLAUDE.md` → `/boot/config/plugins/varaverk/CLAUDE.md` (this file)
|
||||
|
||||
This file lives on `/boot` (USB flash) and is always available regardless of array state.
|
||||
|
||||
---
|
||||
|
||||
## Commit Style
|
||||
|
||||
Plain, concise messages. No Co-Authored-By trailers. No bullet-point summaries in the body.
|
||||
One sentence on the why, not the what.
|
||||
@@ -0,0 +1,771 @@
|
||||
#!/bin/bash
|
||||
# ==============================================================================================
|
||||
# ========================== HOST1 CONFIGURATION — unRAID-Gmer4Lfe ============================
|
||||
# ==============================================================================================
|
||||
# HOST1-specific variables — credentials, container names, share paths, failover lists.
|
||||
# Sourced after master.conf — values here extend shared profile arrays and add HOST1-specific
|
||||
# identity, credentials, and container configuration.
|
||||
#
|
||||
# Sparse checkout (git) ensures HOST2 never receives this file.
|
||||
# HOST2 never sees HOST1 credentials — clean separation at the file level.
|
||||
#
|
||||
# DO NOT put shared config here — thresholds, toggles, profiles belong in master.conf.
|
||||
# DO NOT put HOST2 variables here — they belong in host2.conf.
|
||||
#
|
||||
# ── INDEX ─────────────────────────────────────────────────────────────────────────────────────
|
||||
#
|
||||
# ── IDENTITY & CONNECTIVITY ────────────────────────────────────────────────────────────────
|
||||
# IDENTITY hostname, SSH key
|
||||
# EMBY container name, URL, API key
|
||||
# NOTIFICATIONS Discord webhook
|
||||
# PARTNERSHIP auth containers, backup paths
|
||||
#
|
||||
# ── RSYNC ──────────────────────────────────────────────────────────────────────────────────
|
||||
# DAILY SYNC SHARES media shares HOST1 owns and pushes to HOST2
|
||||
# WEEKLY SYNC SHARES appdata shares synced weekly (Sunday 2:30am)
|
||||
# CRITICAL SYNC SHARES appdata shares synced every 30 minutes
|
||||
# BACKUP VERIFY shares for checksum verification against remote
|
||||
# HOST1 RSYNC PROFILE host1-appdata profile for HOST1-specific appdata syncs
|
||||
#
|
||||
# ── DOCKER ─────────────────────────────────────────────────────────────────────────────────
|
||||
# DOCKER DAILY RESTART containers restarted daily
|
||||
# DOCKER WEEKLY RESTART containers restarted weekly
|
||||
# DOCKER WATCHDOG memory limits, health URLs, required containers, ignore list
|
||||
# DOCKER NETWORK CONNECT networks and containers for docker_network_connect.sh
|
||||
#
|
||||
# ── FALLBACK ───────────────────────────────────────────────────────────────────────────────
|
||||
# DDNS DDNS containers managed by HOST1
|
||||
# INTERNET LOSS containers stopped when internet is lost
|
||||
# FALLBACK TIERS what HOST1 runs for HOST2 per tier
|
||||
# TIER DELAYS how long HOST1 must be down before each tier activates on HOST2
|
||||
# RSYNC WRITEBACK HOST1 appdata synced back on handback
|
||||
#
|
||||
# ── MEDIA ──────────────────────────────────────────────────────────────────────────────────
|
||||
# MEDIA PERMISSIONS share list for media_shares_permissions.sh
|
||||
# MEDIA CLEANER folder lists for media_cleaner.sh
|
||||
#
|
||||
# ── MONITORS ───────────────────────────────────────────────────────────────────────────────
|
||||
# CERTIFICATE MONITOR domains checked for SSL expiry
|
||||
# SMART HEALTH drives to skip in SMART monitoring
|
||||
# ZFS REPORT pools to exclude from ZFS health report
|
||||
#
|
||||
# ── TRANSCODES ─────────────────────────────────────────────────────────────────────────────
|
||||
# TRANSCODES ramdisk size, thresholds, SSD path, server array
|
||||
#
|
||||
# ── ARR STACK ──────────────────────────────────────────────────────────────────────────────
|
||||
# DOWNLOADERS slskd, SABnzbd, qBittorrent credentials and URLs
|
||||
# LIDARR URL, API key, path map
|
||||
# SONARR URL, API key, path map
|
||||
# RADARR URL, API key, path map
|
||||
# ARR RECOVERY per-arr recovery toggles
|
||||
#
|
||||
# ── SYSTEM WATCHDOG ────────────────────────────────────────────────────────────────────────
|
||||
# SYSTEM WATCHDOG per-host check toggles and NIC configuration
|
||||
#
|
||||
# ── RESOURCE MANAGER ───────────────────────────────────────────────────────────────────────
|
||||
# RESOURCE MANAGER containers paused/stopped under memory pressure
|
||||
#
|
||||
# ==============================================================================================
|
||||
|
||||
# ==============================================================================================
|
||||
# ── IDENTITY & CONNECTIVITY ───────────────────────────────────────────────────────────────────
|
||||
# ==============================================================================================
|
||||
|
||||
# ━━━ Identity ━━━
|
||||
# HOST1 hostname lives in master.conf (not a credential — safe for all servers).
|
||||
# SSH key used for all server-to-server operations — rsync, failover container commands.
|
||||
# Must be in /root/.ssh/ and authorised in HOST2's /root/.ssh/authorized_keys.
|
||||
HOST1_SSH_KEY="/root/.ssh/gmer4lfe_rsync_automation"
|
||||
HOST1_OWNER="gmer4lfe"
|
||||
HOST1_OWNER_EMAIL="gmer4lfe@gmail.com"
|
||||
|
||||
# ━━━ Emby ━━━
|
||||
# Referenced by transcode_manager.sh, emby_session_report.sh, emby_database_repair.sh,
|
||||
# weekly_sync_maintenance.sh, and HOST1_TRANSCODE_SERVERS below.
|
||||
# API key: Emby Dashboard → API Keys → + New Key
|
||||
HOST1_EMBY_CONTAINER="Emby"
|
||||
HOST1_EMBY_URL="http://localhost:8096"
|
||||
HOST1_EMBY_API_KEY="0c27448d93a7431f9ac63569f7655829"
|
||||
|
||||
# ━━━ Jellyfin ━━━
|
||||
# API key: Jellyfin Dashboard → Administration → API Keys → + New Key
|
||||
HOST1_JELLYFIN_CONTAINER="Jellyfin"
|
||||
HOST1_JELLYFIN_URL="http://localhost:8095"
|
||||
HOST1_JELLYFIN_API_KEY="4e820e7df74c4933acec212b1996314e"
|
||||
|
||||
# ━━━ Gitea ━━━
|
||||
# Personal access token for gitea_ssh_setup.sh — registers this server's SSH public key
|
||||
# with Gitea so git operations use key auth instead of passwords.
|
||||
# Create in Gitea: Settings → Applications → Generate Token → scope: write:user
|
||||
HOST1_GITEA_API_TOKEN=""
|
||||
|
||||
# ━━━ Notifications ━━━
|
||||
# Discord webhook — leave blank to disable.
|
||||
# Per-host so HOST1 and HOST2 can post to different channels or only one server notifies.
|
||||
HOST1_DISCORD_WEBHOOK=""
|
||||
|
||||
# ━━━ Partnership ━━━
|
||||
# HOST1 is always the owner (source of truth) unless --transfer has been run.
|
||||
# See README-Partnership.md and master.conf PARTNERSHIP section for full lifecycle docs.
|
||||
|
||||
# Auth containers reconfigured on onboard/offboard.
|
||||
# Format: "ContainerName|WebUIPort"
|
||||
# On onboard → WebUI pointed at owner's Tailscale IP (mirror clicks NPM, gets owner's auth)
|
||||
# On offboard → WebUI pointed back at localhost
|
||||
HOST1_PARTNERSHIP_AUTH_WEBUIS=(
|
||||
"NginxProxyManager|81"
|
||||
"Lldap-Gmer4Lfe|17170"
|
||||
"Authelia|9091"
|
||||
"Authelia-Secondary|9092"
|
||||
)
|
||||
|
||||
# XML templates (from this server's templates-user/) pushed to mirror during onboard.
|
||||
# These become the mirror's active auth stack, backed by the rsync-synced appdata.
|
||||
# Update filename if Lldap is renamed to drop the host suffix.
|
||||
HOST1_PARTNERSHIP_AUTH_STACK=(
|
||||
# Dependencies first — Mariadb/Redis must be healthy before Authelia starts
|
||||
"my-Mariadb-Authelia.xml"
|
||||
"my-Mariadb-Authelia-Secondary.xml"
|
||||
"my-Redis-Authelia.xml"
|
||||
"my-Redis-Authelia-Secondary.xml"
|
||||
# Auth apps — deployed after their deps are confirmed healthy
|
||||
"my-Authelia.xml"
|
||||
"my-Authelia-Secondary.xml"
|
||||
"my-NginxProxyManager.xml"
|
||||
"my-Lldap-Gmer4Lfe.xml"
|
||||
# Source of truth — must be available on HOST2 independently of the auth stack
|
||||
"my-Gitea.xml"
|
||||
)
|
||||
|
||||
# XML templates pushed to mirror for the arr stack during onboard.
|
||||
# Deps (e.g. databases) first if any — same ordering rule as auth stack.
|
||||
HOST1_PARTNERSHIP_ARR_STACK=(
|
||||
# "my-Sonarr.xml"
|
||||
# "my-Radarr.xml"
|
||||
# "my-Lidarr.xml"
|
||||
# "my-Prowlarr.xml"
|
||||
# "my-Bazarr.xml"
|
||||
)
|
||||
|
||||
# Paths HOST2 should collect during the grace window after offboard.
|
||||
# Notified on offboard — no auto-deletion, HOST2 must collect manually within PARTNERSHIP_GRACE_HOURS.
|
||||
HOST1_PARTNERSHIP_MIRROR_BACKUPS=(
|
||||
# "/mnt/user/appdata-Fallback/Jayred365-Emby"
|
||||
)
|
||||
|
||||
# Containers parked on this server when partnership is active.
|
||||
# Stopped on onboard (owner deploys its stack instead), restarted on offboard.
|
||||
HOST1_PARTNERSHIP_OWN_CONTAINERS=(
|
||||
# "Emby"
|
||||
# "NginxProxyManager"
|
||||
)
|
||||
|
||||
# Emby admin provisioning — toggle is owner-only, credentials are per-host.
|
||||
# Owner enables/disables the feature. Each host sets the account they want on the shared Emby.
|
||||
# On onboard: owner reads mirror's HOST*_PARTNERSHIP_EMBY_ADMIN_* and creates that account.
|
||||
# On offboard: account is deleted. Username collision → onboard exits with error.
|
||||
HOST1_PARTNERSHIP_PROVISION_EMBY_ADMIN=false # owner controls whether Emby is shared
|
||||
HOST1_PARTNERSHIP_EMBY_PORT=8096
|
||||
HOST1_PARTNERSHIP_EMBY_ADMIN_USER="" # this server's desired Emby username
|
||||
HOST1_PARTNERSHIP_EMBY_ADMIN_PASS="" # this server's desired Emby password
|
||||
|
||||
# ==============================================================================================
|
||||
# ── RSYNC ─────────────────────────────────────────────────────────────────────────────────────
|
||||
# ==============================================================================================
|
||||
|
||||
# ━━━ Daily Sync Shares ━━━
|
||||
# Shares HOST1 pushes to all other nodes every night (1am via daily_sync_maintenance.sh).
|
||||
# Mesh model: every node pushes every media share — no ownership, no mirrors.
|
||||
# arr_sync ensures all arr libraries converge (union). rsync spreads files (additive, no --delete).
|
||||
# arr_cleanup removes true orphans based on local arr state.
|
||||
# Any node can download content to any share — it propagates to all nodes on the next cycle.
|
||||
# Nextcloud is intentionally one-directional (HOST1→HOST2 offsite backup — not arr-managed).
|
||||
# Uses DEFAULT_RSYNC_OPTS from master.conf — no profile needed.
|
||||
# For shares needing container stops or custom options — add a profile in master.conf.
|
||||
HOST1_DAILY_SYNC_SHARES=(
|
||||
/mnt/user/Books
|
||||
/mnt/user/Intros
|
||||
/mnt/user/Kids_Movies
|
||||
/mnt/user/Kids_Tv_Shows
|
||||
/mnt/user/Movies
|
||||
/mnt/user/Music
|
||||
/mnt/user/Music_Videos
|
||||
/mnt/user/Nextcloud
|
||||
/mnt/user/stand-up_comedy
|
||||
/mnt/user/Sports
|
||||
# /mnt/user/Tv_Shows
|
||||
/mnt/user/Anime_Shows-Old
|
||||
/mnt/user/Anime_Movies-Old
|
||||
/mnt/user/Anime_Movies
|
||||
/mnt/user/Anime_Shows
|
||||
)
|
||||
|
||||
# Personal encrypted shares — synced for offsite backup, independent of media shares.
|
||||
# ZFS encrypted at dataset level — remote receives encrypted blocks, cannot read content.
|
||||
# See README-Rsync_Setup.md for ZFS encryption setup before uncommenting.
|
||||
HOST1_PERSONAL_SHARES=(
|
||||
# /mnt/user/HOST1-Personal # uncomment after creating encrypted dataset
|
||||
)
|
||||
|
||||
# ━━━ Weekly Sync Shares ━━━
|
||||
# Appdata shares synced during the weekly maintenance window (Sunday 2:30am).
|
||||
# Containers stopped both sides before sync — full clean state guaranteed.
|
||||
# Profiles drive container stops, excludes, and options — configured in master.conf RSYNC section.
|
||||
# Order matters — Emby first (larger transfer), then Critical-Data (auth stack).
|
||||
HOST1_WEEKLY_SYNC_SHARES=(
|
||||
"/mnt/user/Media_Server/Emby" # emby profile — full clean mirror
|
||||
"/mnt/user/appdata-Fallback/Critical-Data" # critical-data profile — auth stack
|
||||
)
|
||||
|
||||
# ━━━ Intermediate Sync Shares ━━━
|
||||
# Shares synced every 4 hours by intermediate_sync_maintenance.sh.
|
||||
# Uses DEFAULT_RSYNC_OPTS (no --delete) — for sub-daily propagation of metadata or watch state.
|
||||
# Full media share sync stays in the daily window. Leave empty to skip mid-day rsync entirely.
|
||||
HOST1_INTERMEDIATE_SYNC_SHARES=(
|
||||
# Add shares here to enable mid-day rsync
|
||||
# Example: "/mnt/user/Emby_Metadata"
|
||||
)
|
||||
|
||||
# ━━━ Critical Sync Shares ━━━
|
||||
# Appdata shares synced every 30 minutes by critical_sync_maintenance.sh.
|
||||
# Format: "/path/to/share" or "/path/to/share|profile-name"
|
||||
# Order matters — Critical-Data first (auth stack), then Emby dirty sync.
|
||||
HOST1_CRITICAL_SYNC_SHARES=(
|
||||
"/mnt/user/appdata-Fallback/Critical-Data|critical-fallback" # auth dirty sync — stays running
|
||||
"/mnt/user/Media_Server/Emby|emby-fallback" # Emby dirty sync — stays running
|
||||
)
|
||||
|
||||
# ━━━ Backup Verify ━━━
|
||||
# Shares verified by backup_verify.sh — random file checksum comparison against remote.
|
||||
# Leave empty to use HOST1_DAILY_SYNC_SHARES automatically.
|
||||
# Sample size and minimum file size defined in master.conf.
|
||||
HOST1_BACKUP_VERIFY_SHARES=(
|
||||
# leave empty to use HOST1_DAILY_SYNC_SHARES automatically
|
||||
)
|
||||
|
||||
# ━━━ HOST1 Rsync Profile — host1-appdata ━━━
|
||||
# HOST1-specific appdata sync profile — extends the shared PROFILE_* arrays in master.conf.
|
||||
# Use for appdata unique to HOST1 (Organizrv2, VaultWarden, UptimeKuma etc.)
|
||||
# Shared appdata (auth stack, Emby) use dedicated profiles defined in master.conf.
|
||||
# Run manually: bash Rsync/rsync.sh /mnt/user/appdata-Fallback/HOST1-Appdata --profile=host1-appdata
|
||||
PROFILE_RSYNC_OPTS[host1-appdata]="-av --info=progress2 --bwlimit=${PROFILE_BW_LIMIT[host1-appdata]:-8000}"
|
||||
PROFILE_BW_LIMIT[host1-appdata]=8000
|
||||
PROFILE_RETRY_COUNT[host1-appdata]=3
|
||||
PROFILE_SLEEP[host1-appdata]=300
|
||||
PROFILE_CRITICAL_CONTAINER_NAMES[host1-appdata]="Organizrv2-Gmer4Lfe UptimeKuma-Gmer4Lfe VaultWarden-Gmer4Lfe"
|
||||
PROFILE_DELAYED_CONTAINERS[host1-appdata]=""
|
||||
PROFILE_CONTAINER_DELAY[host1-appdata]=5
|
||||
PROFILE_EXCLUDE_DIRS[host1-appdata]="logs *.tmp"
|
||||
|
||||
# ==============================================================================================
|
||||
# ── DOCKER ────────────────────────────────────────────────────────────────────────────────────
|
||||
# ==============================================================================================
|
||||
|
||||
# ━━━ Docker Daily Restart ━━━
|
||||
# Containers restarted every day via DAILY_MAINTENANCE_SCRIPTS.
|
||||
# Dispatcharr degrades over time without restart — daily is intentional, not just housekeeping.
|
||||
# Order matters — auth stack first, then media services.
|
||||
HOST1_DAILY_RESTART_CONTAINERS=(
|
||||
"NginxProxyManager"
|
||||
"Lldap-Gmer4Lfe"
|
||||
"Authelia"
|
||||
"Authelia-Secondary"
|
||||
"Dispatcharr-Iptv-Users"
|
||||
"Dispatcharr" # Live TV scheduler — degrades without daily restart
|
||||
"Dispatcharr-Basic"
|
||||
"ErsatzTV-Emby"
|
||||
"Slskd" # Soulseek connection drops after extended uptime; restart refreshes share index
|
||||
)
|
||||
|
||||
# ━━━ Docker Weekly Restart ━━━
|
||||
# Less critical services restarted weekly via WEEKLY_MAINTENANCE_SCRIPTS (Sunday 2:30am).
|
||||
# Containers already stopped for weekly sync — restart adds zero extra downtime.
|
||||
HOST1_WEEKLY_RESTART_CONTAINERS=(
|
||||
"NextCloud"
|
||||
"Organizrv2-Gmer4Lfe"
|
||||
"AdGuard-Home"
|
||||
"Immich-Gmer4Lfe"
|
||||
)
|
||||
|
||||
# ━━━ Docker Watchdog ━━━
|
||||
# Per-HOST1 container configuration for docker_watchdog.sh.
|
||||
# Shared thresholds and toggles live in master.conf.
|
||||
|
||||
# Memory hard limits in MB — immediate restart if exceeded.
|
||||
# Set at "container is clearly broken" not "container is busy".
|
||||
# 20GB=20480 18GB=18432 16GB=16384 12GB=12288 8GB=8192 4GB=4096 2GB=2048 1GB=1024
|
||||
declare -A HOST1_WATCHDOG_CONTAINERS=(
|
||||
["Emby"]=20480 # 20GB — large library + active transcodes
|
||||
["LidaTube"]=6144 # 6GB — memory leak over time
|
||||
["Tdarr"]=6144 # 6GB — encoding is memory intensive
|
||||
["Code-Server"]=1024 # 1GB — should never need more
|
||||
)
|
||||
|
||||
# HTTP health check URLs — checked every cycle, strike system before restart.
|
||||
# Only add containers with a meaningful web interface to check.
|
||||
declare -A HOST1_WATCHDOG_CONTAINER_URLS=(
|
||||
["Emby"]="http://localhost:8096"
|
||||
)
|
||||
|
||||
# Required containers — must always be running on HOST1.
|
||||
# Strike system before restart — repeated failures go on skip list, auto-clears on recovery.
|
||||
# Listed in dependency order — dependencies before dependents.
|
||||
HOST1_WATCHDOG_REQUIRED_CONTAINERS=(
|
||||
"NginxProxyManager"
|
||||
"Lldap-Gmer4Lfe"
|
||||
"Mariadb-Authelia"
|
||||
"Mariadb-Authelia-Secondary"
|
||||
"Redis-Authelia"
|
||||
"Redis-Authelia-Secondary"
|
||||
"Authelia"
|
||||
"Authelia-Secondary"
|
||||
)
|
||||
|
||||
# Containers to skip in Tier 2 global scan — legitimately stopped or frequently restarting.
|
||||
# Watchdog leaves these alone entirely — no restart attempts, no crash loop tracking.
|
||||
HOST1_WATCHDOG_SCAN_IGNORE=(
|
||||
"DashGate"
|
||||
"PIA-WG-Config-Generator"
|
||||
"Aperture"
|
||||
"Aperture-Kids"
|
||||
"pgvector-18-Apeture-Kids"
|
||||
"Pgvector18-Aperture"
|
||||
"emby-test" # broken test container (exit 127 — bad image)
|
||||
)
|
||||
|
||||
# Dependency ordering — skip restarting a container if its dependency is also down.
|
||||
# Prevents watchdog from restarting Authelia before Mariadb is back up.
|
||||
# SPACE-SEPARATED STRINGS — converted to array at runtime.
|
||||
declare -A HOST1_WATCHDOG_DEPENDENCIES=(
|
||||
["Authelia"]="Mariadb-Authelia Redis-Authelia"
|
||||
["Authelia-Secondary"]="Mariadb-Authelia Redis-Authelia-Secondary"
|
||||
["NextCloud"]="Postgres-NextCloud"
|
||||
)
|
||||
|
||||
# Per-container appdata growth suppress ceilings in MB.
|
||||
# ONLY needed in specific cases — growth rate detection covers all containers automatically.
|
||||
# Use this when a container legitimately has large stable data and you want to guarantee
|
||||
# it never triggers a false-positive growth alert. Growth warnings are suppressed while the
|
||||
# container's dir stays below this ceiling; above it, warnings resume as normal.
|
||||
# 50GB=51200 25GB=25600 20GB=20480 15GB=15360 10GB=10240 5GB=5120
|
||||
declare -A HOST1_WATCHDOG_APPDATA_SIZES=(
|
||||
["Tdarr"]="25600" # 25GB — transcode cache grows legitimately during active jobs
|
||||
["7dtd"]="20480" # 20GB — game server world data, expected to be large
|
||||
)
|
||||
|
||||
# ━━━ Network Watchdog ━━━
|
||||
# Host-specific connectivity config for Watchdogs/System/network_watchdog.sh.
|
||||
HOST1_NETWORK_WATCHDOG_DDNS_DOMAIN="gmer4lfe.com"
|
||||
HOST1_NETWORK_WATCHDOG_DDNS_CONTAINER="Gmer4Lfe.com"
|
||||
HOST1_NETWORK_WATCHDOG_NPM_URL="https://gmer4lfe.com"
|
||||
|
||||
# ━━━ Docker Network Connect ━━━
|
||||
# Containers connected to custom networks at array start by docker_network_connect.sh.
|
||||
# Networks created if they don't exist — idempotent, safe to re-run.
|
||||
HOST1_NETWORK_CONNECT_CONTAINERS=(
|
||||
"memcached"
|
||||
"Npm-CrowdSec"
|
||||
)
|
||||
|
||||
HOST1_NETWORK_CONNECT_NETWORKS=(
|
||||
"high-availability"
|
||||
)
|
||||
|
||||
# ==============================================================================================
|
||||
# ── FALLBACK ──────────────────────────────────────────────────────────────────────────────────
|
||||
# ==============================================================================================
|
||||
|
||||
# ━━━ DDNS ━━━
|
||||
# DDNS containers HOST1 manages — started/stopped by fallback.sh per DDNS absolute rules:
|
||||
# Internet loss → stop immediately
|
||||
# Failover → HOST2 starts HOST1's DDNS as Tier 1 (before any other containers)
|
||||
# Handback → stop HOST1's DDNS on HOST2 → rsync → start containers → start local DDNS last
|
||||
HOST1_DDNS_CONTAINERS=(
|
||||
"Gmer4Lfe.com"
|
||||
)
|
||||
|
||||
# ━━━ Internet Loss ━━━
|
||||
# Containers stopped immediately on HOST1 when internet connection is lost.
|
||||
# Prevents external-facing services from operating without connectivity.
|
||||
FALLBACK_HOST1_STOP_ON_NO_NET=(
|
||||
"Gmer4Lfe.com"
|
||||
)
|
||||
|
||||
# ━━━ Fallback Tiers — HOST1 Runs for HOST2 ━━━
|
||||
# Containers HOST1 starts when HOST2 goes down.
|
||||
# Tier 1 is always immediate — vital services cannot wait.
|
||||
# Higher tiers activate after HOST2_TIER*_DELAY minutes (set in host2.conf).
|
||||
FALLBACK_HOST1_COVERS_HOST2_TIER1=(
|
||||
"Gmer4Lfe.us"
|
||||
"VaultWarden-Jayred365"
|
||||
)
|
||||
|
||||
FALLBACK_HOST1_COVERS_HOST2_TIER2=(
|
||||
# "container-placeholder"
|
||||
)
|
||||
|
||||
FALLBACK_HOST1_COVERS_HOST2_TIER3=(
|
||||
# "container-placeholder"
|
||||
)
|
||||
|
||||
FALLBACK_HOST1_COVERS_HOST2_TIER4=(
|
||||
# "container-placeholder"
|
||||
)
|
||||
|
||||
# ━━━ Tier Delays — HOST1's Containers on HOST2 ━━━
|
||||
# How long HOST1 must be down before each tier activates on HOST2 — in minutes.
|
||||
# Tier 1 is always immediate — no delay var needed.
|
||||
HOST1_TIER2_DELAY=240 # 4 hours — NextCloud, Immich
|
||||
HOST1_TIER3_DELAY=720 # 12 hours — secondary services
|
||||
HOST1_TIER4_DELAY=1440 # 24 hours — arrs + downloaders
|
||||
|
||||
# ━━━ Rsync Writeback — HOST1 Appdata Back on Handback ━━━
|
||||
# Syncs HOST1 appdata BACK to HOST1 when it comes back online after a failover.
|
||||
# Containers stopped before writeback — clean source, no competing writes.
|
||||
#
|
||||
# HOST1_TIER1_WRITEBACK_DELAY: short outages skip Tier 1 writeback — primary state
|
||||
# is more reliable than dirty sync data for brief outages.
|
||||
HOST1_TIER1_WRITEBACK_DELAY=60 # skip Emby writeback if outage under 1hr
|
||||
|
||||
# Tier 4 automatically syncs HOST1_DAILY_SYNC_SHARES — only list paths NOT in that array.
|
||||
FALLBACK_HOST1_WRITEBACK_TIER1=(
|
||||
"/mnt/user/Media_Server/Emby" # watch states built up during outage
|
||||
)
|
||||
|
||||
FALLBACK_HOST1_WRITEBACK_TIER2=(
|
||||
"/mnt/user/appdata-Fallback/Important-Data" # NextCloud + Postgres — files added during outage
|
||||
)
|
||||
|
||||
FALLBACK_HOST1_WRITEBACK_TIER3=(
|
||||
# "location-placeholder"
|
||||
)
|
||||
|
||||
FALLBACK_HOST1_WRITEBACK_TIER4=(
|
||||
"/mnt/user/appdata-Fallback/Arrs_Stack" # arr databases — downloads queued during outage
|
||||
)
|
||||
|
||||
# ==============================================================================================
|
||||
# ── MEDIA ─────────────────────────────────────────────────────────────────────────────────────
|
||||
# ==============================================================================================
|
||||
|
||||
# ━━━ Media Permissions ━━━
|
||||
# Shares that media_shares_permissions.sh applies PERMISSIONS_MODE and PERMISSIONS_OWNER to.
|
||||
# Runs first in DAILY_MAINTENANCE_SCRIPTS — arr cleanup depends on correct ownership.
|
||||
HOST1_MEDIA_PERMISSION_SHARES=(
|
||||
/mnt/user/Anime_Movies
|
||||
/mnt/user/Anime_Movies-Old
|
||||
/mnt/user/Anime_Shows
|
||||
/mnt/user/Anime_Shows-Old
|
||||
/mnt/user/appcache
|
||||
/mnt/user/Books
|
||||
/mnt/user/Downloads
|
||||
/mnt/user/Games
|
||||
/mnt/user/Intros
|
||||
/mnt/user/Kids_Movies
|
||||
/mnt/user/Kids_Tv_Shows
|
||||
/mnt/user/Movie_Recordings
|
||||
/mnt/user/Movies
|
||||
/mnt/user/Music
|
||||
/mnt/user/Music_Videos
|
||||
/mnt/user/Photo
|
||||
/mnt/user/Sports
|
||||
/mnt/user/stand-up_comedy
|
||||
/mnt/user/Tv_Recordings
|
||||
/mnt/user/Tv_Shows
|
||||
/mnt/user/YouTube
|
||||
)
|
||||
|
||||
# ━━━ Media Cleaner ━━━
|
||||
# Folder lists for media_cleaner.sh — two profiles: anime and media.
|
||||
# File patterns shared across all servers — defined in master.conf.
|
||||
# Called via DAILY_MAINTENANCE_SCRIPTS. Run manually: Media/media_cleaner.sh anime|media
|
||||
HOST1_ANIME_CLEAN_FOLDERS=(
|
||||
/mnt/user/Anime_Movies
|
||||
/mnt/user/Anime_Movies-Old
|
||||
/mnt/user/Anime_Shows
|
||||
/mnt/user/Anime_Shows-Old
|
||||
)
|
||||
|
||||
HOST1_MEDIA_CLEAN_FOLDERS=(
|
||||
/mnt/user/Kids_Movies
|
||||
/mnt/user/Kids_Tv_Shows
|
||||
/mnt/user/Movies
|
||||
/mnt/user/Music
|
||||
/mnt/user/Sports
|
||||
/mnt/user/stand-up_comedy
|
||||
/mnt/user/Tv_Shows
|
||||
)
|
||||
|
||||
# ==============================================================================================
|
||||
# ── MONITORS ──────────────────────────────────────────────────────────────────────────────────
|
||||
# ==============================================================================================
|
||||
|
||||
# ━━━ Certificate Monitor ━━━
|
||||
# Domains checked via direct openssl connection — not relying on NPM's certificate state.
|
||||
# Checks the actual certificate served, not what NPM thinks it has.
|
||||
# Thresholds (CERT_WARN_DAYS, CERT_CRIT_DAYS) defined in master.conf.
|
||||
HOST1_CERT_MONITOR_DOMAINS=(
|
||||
"Gmer4Lfe.com"
|
||||
"Gmer4Lfe.us"
|
||||
)
|
||||
|
||||
# ━━━ SMART Health ━━━
|
||||
# Drives skipped in SMART attribute monitoring — hardware is server-specific.
|
||||
# Thresholds read from dynamix.cfg at runtime — fallbacks in master.conf.
|
||||
HOST1_SMART_IGNORE_DRIVES=(
|
||||
"sda" # boot USB — SMART not meaningful on flash drives
|
||||
)
|
||||
|
||||
# ━━━ ZFS Report ━━━
|
||||
# Pools excluded from the weekly ZFS health report — reduces noise from single-disk array pools.
|
||||
# These are individual array disks formatted as ZFS — converting to XFS over time via unBalance.
|
||||
# Pool health thresholds defined in master.conf.
|
||||
HOST1_ZFS_REPORT_IGNORE_POOLS=(
|
||||
"disk5"
|
||||
"disk6"
|
||||
"disk8"
|
||||
"disk9"
|
||||
"disk10"
|
||||
)
|
||||
|
||||
# ==============================================================================================
|
||||
# ── TRANSCODES ────────────────────────────────────────────────────────────────────────────────
|
||||
# ==============================================================================================
|
||||
|
||||
# Ramdisk size ceiling — tmpfs only uses RAM actually needed, not the full size upfront.
|
||||
# Real-world: 9 streams peaked at ~5.5GB — 10G gives generous headroom on 128GB RAM.
|
||||
HOST1_RAMDISK_SIZE="10G"
|
||||
|
||||
# Usage thresholds — coupled to HOST1_RAMDISK_SIZE, adjust all three together if size changes.
|
||||
# Hysteresis gap (8.5 - 7 = 1.5GB) prevents flip-flop between ramdisk and SSD.
|
||||
HOST1_RAMDISK_WARN_GB=8.5 # flip to SSD when ramdisk usage reaches this
|
||||
HOST1_RAMDISK_LOW_GB=7 # flip back to ramdisk when usage drops to this
|
||||
|
||||
# SSD fallback path — where transcodes land when ramdisk exceeds HOST1_RAMDISK_WARN_GB.
|
||||
# Must be on cache pool — array disks too slow for active transcode writes.
|
||||
HOST1_TRANSCODE_SSD="/mnt/cache/Temp_Storage/Emby/Transcodes/"
|
||||
|
||||
# Media servers sharing the ramdisk transcode space on HOST1.
|
||||
# Format: "ContainerName|URL|APIKey|Type" — Type: emby | jellyfin | plex
|
||||
# Entries with placeholder API keys are skipped automatically.
|
||||
# ⚠️ Tdarr does NOT belong here — keep Tdarr on SSD, not ramdisk.
|
||||
HOST1_TRANSCODE_SERVERS=(
|
||||
"${HOST1_EMBY_CONTAINER}|${HOST1_EMBY_URL}|${HOST1_EMBY_API_KEY}|emby"
|
||||
"${HOST1_JELLYFIN_CONTAINER}|${HOST1_JELLYFIN_URL}|${HOST1_JELLYFIN_API_KEY}|jellyfin"
|
||||
)
|
||||
|
||||
# ==============================================================================================
|
||||
# ── ARR STACK ─────────────────────────────────────────────────────────────────────────────────
|
||||
# ==============================================================================================
|
||||
# Used by arr cleanup scripts and arrs_failed_stalled_recovery.sh.
|
||||
# detect_hosts() selects HOST1 vars when running on HOST1.
|
||||
#
|
||||
# PATH MAPS — container path → host path translation.
|
||||
# Arr stores file paths using container-internal paths — scripts need host paths to scan.
|
||||
# Add one entry per root folder in arr Settings → Media Management → Root Folders.
|
||||
|
||||
# ━━━ Downloaders ━━━
|
||||
# Used by downloaders_reset.sh — runs every 30min via CRITICAL_MAINTENANCE_SCRIPTS.
|
||||
# Clears stuck states, purges old history, prepares each client for a clean cycle.
|
||||
|
||||
# slskd — clears stuck searches, dead transfers, purges expired failed imports.
|
||||
# SLSKD_FAILED_IMPORTS_DIR: where Soularr moves albums Lidarr rejected.
|
||||
HOST1_SLSKD_URL="http://localhost:8980"
|
||||
HOST1_SLSKD_API_KEY="4bF9kL2mNpQrT7vWxYz1A3dEgHjKoRsU"
|
||||
HOST1_SLSKD_FAILED_IMPORTS_DIR="/mnt/user/Temp_Storage/Slskd/completed/failed_imports"
|
||||
|
||||
# SABnzbd
|
||||
HOST1_SABNZBD_URL="http://localhost:8180"
|
||||
HOST1_SABNZBD_API_KEY="8bfefe41d83b4d50883e32859b55ca9a"
|
||||
|
||||
# qBittorrent — deleteFiles=false removes torrent from qBit but leaves files on disk.
|
||||
# Radarr/Sonarr manage actual files independently.
|
||||
HOST1_QBIT_URL="http://localhost:8080"
|
||||
HOST1_QBIT_USERNAME="root"
|
||||
HOST1_QBIT_PASSWORD="Stay0utD!ck"
|
||||
|
||||
# ━━━ Lidarr — HOST1 only ━━━
|
||||
# HOST2 does not run Lidarr — HOST1_LIDARR_RECOVERY flag handles the exit cleanly.
|
||||
HOST1_LIDARR_URL="http://localhost:8686"
|
||||
HOST1_LIDARR_API_KEY="b2977e71ef074bc0a0529d9fcce3b2dc"
|
||||
HOST1_LIDARR_MUSIC_ROOT="/mnt/user/Music-New"
|
||||
HOST1_FANART_API_KEY="Yd7147a43b692df0b364b94dc47efb81"
|
||||
HOST1_LASTFM_API_KEY="be6dc169c33ae263e690c30d18b7491d"
|
||||
|
||||
declare -A HOST1_LIDARR_PATH_MAP=(
|
||||
["/ext-music"]="/mnt/user/Music-New"
|
||||
)
|
||||
|
||||
# ━━━ Sonarr ━━━
|
||||
HOST1_SONARR_URL="http://localhost:8989"
|
||||
HOST1_SONARR_API_KEY="130decd3db5b4c25afad64864cd03f9f"
|
||||
HOST1_SONARR_TV_ROOT="/mnt/user/Tv_Shows"
|
||||
|
||||
# Note: stand-up_comedy in both Sonarr + Radarr — TV specials and movie specials, one folder
|
||||
declare -A HOST1_SONARR_PATH_MAP=(
|
||||
["/tv"]="/mnt/user/Tv_Shows"
|
||||
["/ext-standup-comedy"]="/mnt/user/stand-up_comedy/series"
|
||||
["/kids tv"]="/mnt/user/Kids_Tv_Shows"
|
||||
["/ext-anime-shows"]="/mnt/user/Anime_Shows-Old"
|
||||
)
|
||||
|
||||
# ━━━ Radarr ━━━
|
||||
HOST1_RADARR_URL="http://localhost:7878"
|
||||
HOST1_RADARR_API_KEY="d43a3ec6cf1549edb4af0cc63f98b2a9"
|
||||
HOST1_TMDB_API_KEY="3dac5e2e49b5540472d2eafec4f01260"
|
||||
HOST1_RADARR_MOVIES_ROOT="/mnt/user/Movies"
|
||||
|
||||
# Note: stand-up_comedy in both Radarr + Sonarr — movie specials and TV specials, one folder
|
||||
declare -A HOST1_RADARR_PATH_MAP=(
|
||||
["/movies"]="/mnt/user/Movies"
|
||||
["/kids movies"]="/mnt/user/Kids_Movies"
|
||||
["/ext-stand-up-comedy"]="/mnt/user/stand-up_comedy/specials"
|
||||
["/anime-movies"]="/mnt/user/Anime_Movies-Old"
|
||||
["/ext-anime-movies"]="/mnt/user/Anime_Movies-Old"
|
||||
)
|
||||
|
||||
# ━━━ Arr Recovery Toggles ━━━
|
||||
# false = skip that arr on this host — exits cleanly without error
|
||||
HOST1_SONARR_RECOVERY=true
|
||||
HOST1_RADARR_RECOVERY=true
|
||||
HOST1_LIDARR_RECOVERY=true # HOST1 only — exits cleanly on HOST2
|
||||
|
||||
# ==============================================================================================
|
||||
# ── SYSTEM WATCHDOG ───────────────────────────────────────────────────────────────────────────
|
||||
# ==============================================================================================
|
||||
# Per-host check toggles and NIC config for system_watchdog.sh.
|
||||
# Aliased by detect_hosts() — script uses unprefixed SYS_WATCHDOG_* names.
|
||||
# HOST1: TR1950X 128GB — full media server, active transcoding, ZFS cache pools.
|
||||
#
|
||||
# Three-tier response — all critical checks enabled by default on HOST1:
|
||||
# Tier 1 (bypass strikes, reboot now): docker daemon, rootfs full, kernel oops, FD, /boot
|
||||
# Tier 2 (bypass strikes with OOM): RAM critical + OOM kills in cycle
|
||||
# Tier 3 (standard strike system): everything else
|
||||
#
|
||||
# RAM tiers, OOM limits, and reboot loop settings in master.conf System Watchdog section.
|
||||
|
||||
# ━━━ Primary NIC ━━━
|
||||
# Network interface for NIC state check — verify with: ip link show | grep "^[0-9]"
|
||||
# Common values: eth0, bond0, br0, eno1
|
||||
HOST1_SYS_WATCHDOG_NIC="eth0"
|
||||
|
||||
# ━━━ Tier 1 — Critical Checks ━━━
|
||||
# These bypass the strike system — a single hit triggers immediate reboot.
|
||||
# Disabling any of these is not recommended — they protect against acute system failure.
|
||||
|
||||
# Docker daemon unresponsive → try restart, reboot if restart fails.
|
||||
# Without a working daemon docker_watchdog.sh is blind and containers cannot be managed.
|
||||
HOST1_SYS_WATCHDOG_CHECK_DOCKER_DAEMON=true
|
||||
|
||||
# rootfs at critical threshold (ROOTFS_CRITICAL_PCT=99) → reboot immediately.
|
||||
# At 99% rootfs writes fail silently — logs stop, Docker errors out, SSH may stop working.
|
||||
# Standard 95% threshold still uses strike system — only 99%+ is critical tier.
|
||||
HOST1_SYS_WATCHDOG_CHECK_ROOTFS=true
|
||||
|
||||
# Kernel BUG/Oops in dmesg delta since last cycle → reboot immediately.
|
||||
# A kernel oops means the kernel ran with a corrupted state — stability is not guaranteed.
|
||||
HOST1_SYS_WATCHDOG_CHECK_KERNEL_OOPS=true
|
||||
|
||||
# File descriptor exhaustion at FD_CRITICAL_PCT (95%) → reboot immediately.
|
||||
# At 95% FD: new connections fail, Docker can't spawn processes, SSH drops.
|
||||
HOST1_SYS_WATCHDOG_CHECK_FD=true
|
||||
|
||||
# /boot read-only detected → reboot immediately.
|
||||
# Unexpected read-only /boot means state files and config writes are silently failing.
|
||||
# Fallback state, watchdog reboot log, and lock files all go stale silently.
|
||||
HOST1_SYS_WATCHDOG_CHECK_BOOT=true
|
||||
|
||||
# ━━━ Tier 2 — Urgent OOM Check ━━━
|
||||
# Bypass strikes when RAM is critically low AND OOM kill rate confirms active crisis.
|
||||
# Both must be enabled for Tier 2 bypass to function — disable either to always use strikes.
|
||||
|
||||
# Track kernel OOM kills each cycle via /proc/vmstat oom_kill delta.
|
||||
# Also provides diagnostic context in reboot messages (which processes were killed).
|
||||
HOST1_SYS_WATCHDOG_CHECK_OOM=true
|
||||
|
||||
# Free RAM check — required for both Tier 2 bypass and RAM tier logic.
|
||||
# Tiers: MEM_WARN_GB(10) → notify | MEM_SHUTDOWN_GB(6) → stop containers | MEM_GB(4) → strikes
|
||||
HOST1_SYS_WATCHDOG_CHECK_RAM=true
|
||||
|
||||
# ━━━ Tier 3 — Standard Checks (strike system) ━━━
|
||||
# Each check must fail SYS_WATCHDOG_STRIKE_LIMIT consecutive cycles before action is taken.
|
||||
# Single spikes are ignored — sustained problems trigger reboot.
|
||||
|
||||
# /var/log filesystem usage above SYS_WATCHDOG_LOG_PCT.
|
||||
# Log spam (Docker log storms, syslog loops) fills rootfs — indicates something broken.
|
||||
HOST1_SYS_WATCHDOG_CHECK_LOG=true
|
||||
|
||||
# ZFS ARC memory pinned above SYS_WATCHDOG_ARC_PINNED_PCT after cache drop.
|
||||
# Enabled on HOST1 — ZFS cache pools actively used. Disable on hosts without ZFS.
|
||||
HOST1_SYS_WATCHDOG_CHECK_ARC=true
|
||||
|
||||
# CPU temperature above SYS_WATCHDOG_CPU_TEMP_MAX (95°C).
|
||||
# Sustained high temp causes kernel throttling or panic. Requires lm-sensors.
|
||||
HOST1_SYS_WATCHDOG_CHECK_CPU_TEMP=true
|
||||
|
||||
# Load average above SYS_WATCHDOG_LOAD_MULTIPLIER × core count.
|
||||
# DISABLED on HOST1 — Tdarr and Emby cause legitimate sustained load spikes during encoding.
|
||||
# Enable on idle servers or adjust SYS_WATCHDOG_LOAD_MULTIPLIER if load is always high.
|
||||
HOST1_SYS_WATCHDOG_CHECK_LOAD=false
|
||||
|
||||
# Zombie process count above SYS_WATCHDOG_ZOMBIE_LIMIT (50).
|
||||
# Large zombie counts indicate serious process management failure — something is stuck.
|
||||
HOST1_SYS_WATCHDOG_CHECK_ZOMBIES=true
|
||||
|
||||
# Check docker_watchdog.sh persistent skip list — required containers on skip list.
|
||||
# Cross-watchdog coordination: if docker_watchdog gave up, system_watchdog escalates.
|
||||
# ENABLED — HOST1 fully built and operational, skip list is meaningful.
|
||||
HOST1_SYS_WATCHDOG_CHECK_CONTAINERS=true
|
||||
|
||||
# /tmp filesystem usage above SYS_WATCHDOG_TMP_PCT with auto-clear attempt.
|
||||
# Script tries to clear aged /tmp files first — only strikes if clear fails.
|
||||
# Lock files, rsync temp files, and Docker ops use /tmp — 100% means lock failures.
|
||||
HOST1_SYS_WATCHDOG_CHECK_TMP=true
|
||||
|
||||
# Array disk error count delta in /proc/mdstat — accumulating errors = disk failing now.
|
||||
# Triggers on SYS_WATCHDOG_MDSTAT_ERROR_LIMIT new errors in one cycle.
|
||||
HOST1_SYS_WATCHDOG_CHECK_MDSTAT=true
|
||||
|
||||
# Primary NIC operstate — detects NIC going down (physical or driver failure).
|
||||
# Uses HOST1_SYS_WATCHDOG_NIC above. Strike system — brief flaps don't trigger reboot.
|
||||
HOST1_SYS_WATCHDOG_CHECK_NETWORK=true
|
||||
|
||||
# sshd running check — attempts restart before escalating.
|
||||
# sshd down = no remote access. Script tries rc.sshd start, notifies, strikes on failure.
|
||||
HOST1_SYS_WATCHDOG_CHECK_SSHD=true
|
||||
|
||||
# Runaway process detection — single process above SYS_WATCHDOG_RUNAWAY_CPU_PCT sustained.
|
||||
# DISABLED — Tdarr encoding and Emby transcoding legitimately peg CPU for extended periods.
|
||||
# Enable only if HOST1 has no CPU-intensive workloads.
|
||||
HOST1_SYS_WATCHDOG_CHECK_RUNAWAY=false
|
||||
|
||||
# ==============================================================================================
|
||||
# ── RESOURCE MANAGER ──────────────────────────────────────────────────────────────────────────
|
||||
# ==============================================================================================
|
||||
# Containers to manage under pressure — see master.conf RW_CRITICAL_CONTAINERS for exclusions.
|
||||
|
||||
# docker pause at medium pressure (RAM < RW_RAM_MEDIUM_GB or load > medium threshold)
|
||||
# Suspended in-place — instant to pause/unpause, no state lost, no restart delay.
|
||||
HOST1_RW_PAUSE_CONTAINERS=(
|
||||
"Huntarr" # arr search automation — safe to suspend
|
||||
"Cleanuparr" # download cleanup — safe to suspend
|
||||
"Healarr" # arr health checks — safe to suspend
|
||||
"Soularr" # Slskd automation — background only
|
||||
"ChannelTube" # YouTube archiver — background only
|
||||
"Pinchflat" # YouTube archiver — background only
|
||||
)
|
||||
|
||||
# docker stop at hard pressure (RAM < RW_RAM_HARD_GB)
|
||||
# Full stop — these are optional/heavy services that free significant RAM when stopped.
|
||||
# resource_watchdog.sh restarts them when pressure fully clears (RAM >= RW_RAM_RECOVER_GB).
|
||||
HOST1_RW_STOP_CONTAINERS=(
|
||||
"LocalAI" # GPU/CPU heavy — largest RAM consumer when idle
|
||||
"7DaysToDie" # game server — optional
|
||||
"V-Rising" # game server — optional
|
||||
"Code-Server" # IDE — not needed during pressure events
|
||||
)
|
||||
|
||||
# ==============================================================================================
|
||||
# ──────────────────────── End Of HOST1 Variables ──────────────────────────────────────────────
|
||||
# ==============================================================================================
|
||||
HOST1_UNRAID_API_KEY="1825c3a2e03ea5089974f4da2e171aa2d5907a1dea23cc479c33e492c8ff4dbb"
|
||||
@@ -0,0 +1,771 @@
|
||||
#!/bin/bash
|
||||
# ==============================================================================================
|
||||
# ========================== HOST1 CONFIGURATION — unRAID-Gmer4Lfe ============================
|
||||
# ==============================================================================================
|
||||
# HOST1-specific variables — credentials, container names, share paths, failover lists.
|
||||
# Sourced after master.conf — values here extend shared profile arrays and add HOST1-specific
|
||||
# identity, credentials, and container configuration.
|
||||
#
|
||||
# Sparse checkout (git) ensures HOST2 never receives this file.
|
||||
# HOST2 never sees HOST1 credentials — clean separation at the file level.
|
||||
#
|
||||
# DO NOT put shared config here — thresholds, toggles, profiles belong in master.conf.
|
||||
# DO NOT put HOST2 variables here — they belong in host2.conf.
|
||||
#
|
||||
# ── INDEX ─────────────────────────────────────────────────────────────────────────────────────
|
||||
#
|
||||
# ── IDENTITY & CONNECTIVITY ────────────────────────────────────────────────────────────────
|
||||
# IDENTITY hostname, SSH key
|
||||
# EMBY container name, URL, API key
|
||||
# NOTIFICATIONS Discord webhook
|
||||
# PARTNERSHIP auth containers, backup paths
|
||||
#
|
||||
# ── RSYNC ──────────────────────────────────────────────────────────────────────────────────
|
||||
# DAILY SYNC SHARES media shares HOST1 owns and pushes to HOST2
|
||||
# WEEKLY SYNC SHARES appdata shares synced weekly (Sunday 2:30am)
|
||||
# CRITICAL SYNC SHARES appdata shares synced every 30 minutes
|
||||
# BACKUP VERIFY shares for checksum verification against remote
|
||||
# HOST1 RSYNC PROFILE host1-appdata profile for HOST1-specific appdata syncs
|
||||
#
|
||||
# ── DOCKER ─────────────────────────────────────────────────────────────────────────────────
|
||||
# DOCKER DAILY RESTART containers restarted daily
|
||||
# DOCKER WEEKLY RESTART containers restarted weekly
|
||||
# DOCKER WATCHDOG memory limits, health URLs, required containers, ignore list
|
||||
# DOCKER NETWORK CONNECT networks and containers for docker_network_connect.sh
|
||||
#
|
||||
# ── FALLBACK ───────────────────────────────────────────────────────────────────────────────
|
||||
# DDNS DDNS containers managed by HOST1
|
||||
# INTERNET LOSS containers stopped when internet is lost
|
||||
# FALLBACK TIERS what HOST1 runs for HOST2 per tier
|
||||
# TIER DELAYS how long HOST1 must be down before each tier activates on HOST2
|
||||
# RSYNC WRITEBACK HOST1 appdata synced back on handback
|
||||
#
|
||||
# ── MEDIA ──────────────────────────────────────────────────────────────────────────────────
|
||||
# MEDIA PERMISSIONS share list for media_shares_permissions.sh
|
||||
# MEDIA CLEANER folder lists for media_cleaner.sh
|
||||
#
|
||||
# ── MONITORS ───────────────────────────────────────────────────────────────────────────────
|
||||
# CERTIFICATE MONITOR domains checked for SSL expiry
|
||||
# SMART HEALTH drives to skip in SMART monitoring
|
||||
# ZFS REPORT pools to exclude from ZFS health report
|
||||
#
|
||||
# ── TRANSCODES ─────────────────────────────────────────────────────────────────────────────
|
||||
# TRANSCODES ramdisk size, thresholds, SSD path, server array
|
||||
#
|
||||
# ── ARR STACK ──────────────────────────────────────────────────────────────────────────────
|
||||
# DOWNLOADERS slskd, SABnzbd, qBittorrent credentials and URLs
|
||||
# LIDARR URL, API key, path map
|
||||
# SONARR URL, API key, path map
|
||||
# RADARR URL, API key, path map
|
||||
# ARR RECOVERY per-arr recovery toggles
|
||||
#
|
||||
# ── SYSTEM WATCHDOG ────────────────────────────────────────────────────────────────────────
|
||||
# SYSTEM WATCHDOG per-host check toggles and NIC configuration
|
||||
#
|
||||
# ── RESOURCE MANAGER ───────────────────────────────────────────────────────────────────────
|
||||
# RESOURCE MANAGER containers paused/stopped under memory pressure
|
||||
#
|
||||
# ==============================================================================================
|
||||
|
||||
# ==============================================================================================
|
||||
# ── IDENTITY & CONNECTIVITY ───────────────────────────────────────────────────────────────────
|
||||
# ==============================================================================================
|
||||
|
||||
# ━━━ Identity ━━━
|
||||
# HOST1 hostname lives in master.conf (not a credential — safe for all servers).
|
||||
# SSH key used for all server-to-server operations — rsync, failover container commands.
|
||||
# Must be in /root/.ssh/ and authorised in HOST2's /root/.ssh/authorized_keys.
|
||||
HOST1_SSH_KEY="/root/.ssh/gmer4lfe_rsync_automation"
|
||||
HOST1_OWNER="gmer4lfe"
|
||||
HOST1_OWNER_EMAIL="gmer4lfe@gmail.com"
|
||||
|
||||
# ━━━ Emby ━━━
|
||||
# Referenced by transcode_manager.sh, emby_session_report.sh, emby_database_repair.sh,
|
||||
# weekly_sync_maintenance.sh, and HOST1_TRANSCODE_SERVERS below.
|
||||
# API key: Emby Dashboard → API Keys → + New Key
|
||||
HOST1_EMBY_CONTAINER="Emby"
|
||||
HOST1_EMBY_URL="http://localhost:8096"
|
||||
HOST1_EMBY_API_KEY="0c27448d93a7431f9ac63569f7655829"
|
||||
|
||||
# ━━━ Jellyfin ━━━
|
||||
# API key: Jellyfin Dashboard → Administration → API Keys → + New Key
|
||||
HOST1_JELLYFIN_CONTAINER="Jellyfin"
|
||||
HOST1_JELLYFIN_URL="http://localhost:8095"
|
||||
HOST1_JELLYFIN_API_KEY="4e820e7df74c4933acec212b1996314e"
|
||||
|
||||
# ━━━ Gitea ━━━
|
||||
# Personal access token for gitea_ssh_setup.sh — registers this server's SSH public key
|
||||
# with Gitea so git operations use key auth instead of passwords.
|
||||
# Create in Gitea: Settings → Applications → Generate Token → scope: write:user
|
||||
HOST1_GITEA_API_TOKEN=""
|
||||
|
||||
# ━━━ Notifications ━━━
|
||||
# Discord webhook — leave blank to disable.
|
||||
# Per-host so HOST1 and HOST2 can post to different channels or only one server notifies.
|
||||
HOST1_DISCORD_WEBHOOK=""
|
||||
|
||||
# ━━━ Partnership ━━━
|
||||
# HOST1 is always the owner (source of truth) unless --transfer has been run.
|
||||
# See README-Partnership.md and master.conf PARTNERSHIP section for full lifecycle docs.
|
||||
|
||||
# Auth containers reconfigured on onboard/offboard.
|
||||
# Format: "ContainerName|WebUIPort"
|
||||
# On onboard → WebUI pointed at owner's Tailscale IP (mirror clicks NPM, gets owner's auth)
|
||||
# On offboard → WebUI pointed back at localhost
|
||||
HOST1_PARTNERSHIP_AUTH_WEBUIS=(
|
||||
"NginxProxyManager|81"
|
||||
"Lldap-Gmer4Lfe|17170"
|
||||
"Authelia|9091"
|
||||
"Authelia-Secondary|9092"
|
||||
)
|
||||
|
||||
# XML templates (from this server's templates-user/) pushed to mirror during onboard.
|
||||
# These become the mirror's active auth stack, backed by the rsync-synced appdata.
|
||||
# Update filename if Lldap is renamed to drop the host suffix.
|
||||
HOST1_PARTNERSHIP_AUTH_STACK=(
|
||||
# Dependencies first — Mariadb/Redis must be healthy before Authelia starts
|
||||
"my-Mariadb-Authelia.xml"
|
||||
"my-Mariadb-Authelia-Secondary.xml"
|
||||
"my-Redis-Authelia.xml"
|
||||
"my-Redis-Authelia-Secondary.xml"
|
||||
# Auth apps — deployed after their deps are confirmed healthy
|
||||
"my-Authelia.xml"
|
||||
"my-Authelia-Secondary.xml"
|
||||
"my-NginxProxyManager.xml"
|
||||
"my-Lldap-Gmer4Lfe.xml"
|
||||
# Source of truth — must be available on HOST2 independently of the auth stack
|
||||
"my-Gitea.xml"
|
||||
)
|
||||
|
||||
# XML templates pushed to mirror for the arr stack during onboard.
|
||||
# Deps (e.g. databases) first if any — same ordering rule as auth stack.
|
||||
HOST1_PARTNERSHIP_ARR_STACK=(
|
||||
# "my-Sonarr.xml"
|
||||
# "my-Radarr.xml"
|
||||
# "my-Lidarr.xml"
|
||||
# "my-Prowlarr.xml"
|
||||
# "my-Bazarr.xml"
|
||||
)
|
||||
|
||||
# Paths HOST2 should collect during the grace window after offboard.
|
||||
# Notified on offboard — no auto-deletion, HOST2 must collect manually within PARTNERSHIP_GRACE_HOURS.
|
||||
HOST1_PARTNERSHIP_MIRROR_BACKUPS=(
|
||||
# "/mnt/user/appdata-Fallback/Jayred365-Emby"
|
||||
)
|
||||
|
||||
# Containers parked on this server when partnership is active.
|
||||
# Stopped on onboard (owner deploys its stack instead), restarted on offboard.
|
||||
HOST1_PARTNERSHIP_OWN_CONTAINERS=(
|
||||
# "Emby"
|
||||
# "NginxProxyManager"
|
||||
)
|
||||
|
||||
# Emby admin provisioning — toggle is owner-only, credentials are per-host.
|
||||
# Owner enables/disables the feature. Each host sets the account they want on the shared Emby.
|
||||
# On onboard: owner reads mirror's HOST*_PARTNERSHIP_EMBY_ADMIN_* and creates that account.
|
||||
# On offboard: account is deleted. Username collision → onboard exits with error.
|
||||
HOST1_PARTNERSHIP_PROVISION_EMBY_ADMIN=false # owner controls whether Emby is shared
|
||||
HOST1_PARTNERSHIP_EMBY_PORT=8096
|
||||
HOST1_PARTNERSHIP_EMBY_ADMIN_USER="" # this server's desired Emby username
|
||||
HOST1_PARTNERSHIP_EMBY_ADMIN_PASS="" # this server's desired Emby password
|
||||
|
||||
# ==============================================================================================
|
||||
# ── RSYNC ─────────────────────────────────────────────────────────────────────────────────────
|
||||
# ==============================================================================================
|
||||
|
||||
# ━━━ Daily Sync Shares ━━━
|
||||
# Shares HOST1 pushes to all other nodes every night (1am via daily_sync_maintenance.sh).
|
||||
# Mesh model: every node pushes every media share — no ownership, no mirrors.
|
||||
# arr_sync ensures all arr libraries converge (union). rsync spreads files (additive, no --delete).
|
||||
# arr_cleanup removes true orphans based on local arr state.
|
||||
# Any node can download content to any share — it propagates to all nodes on the next cycle.
|
||||
# Nextcloud is intentionally one-directional (HOST1→HOST2 offsite backup — not arr-managed).
|
||||
# Uses DEFAULT_RSYNC_OPTS from master.conf — no profile needed.
|
||||
# For shares needing container stops or custom options — add a profile in master.conf.
|
||||
HOST1_DAILY_SYNC_SHARES=(
|
||||
/mnt/user/Books
|
||||
/mnt/user/Intros
|
||||
/mnt/user/Kids_Movies
|
||||
/mnt/user/Kids_Tv_Shows
|
||||
/mnt/user/Movies
|
||||
/mnt/user/Music
|
||||
/mnt/user/Music_Videos
|
||||
/mnt/user/Nextcloud
|
||||
/mnt/user/stand-up_comedy
|
||||
/mnt/user/Sports
|
||||
# /mnt/user/Tv_Shows
|
||||
/mnt/user/Anime_Shows-Old
|
||||
/mnt/user/Anime_Movies-Old
|
||||
/mnt/user/Anime_Movies
|
||||
/mnt/user/Anime_Shows
|
||||
)
|
||||
|
||||
# Personal encrypted shares — synced for offsite backup, independent of media shares.
|
||||
# ZFS encrypted at dataset level — remote receives encrypted blocks, cannot read content.
|
||||
# See README-Rsync_Setup.md for ZFS encryption setup before uncommenting.
|
||||
HOST1_PERSONAL_SHARES=(
|
||||
# /mnt/user/HOST1-Personal # uncomment after creating encrypted dataset
|
||||
)
|
||||
|
||||
# ━━━ Weekly Sync Shares ━━━
|
||||
# Appdata shares synced during the weekly maintenance window (Sunday 2:30am).
|
||||
# Containers stopped both sides before sync — full clean state guaranteed.
|
||||
# Profiles drive container stops, excludes, and options — configured in master.conf RSYNC section.
|
||||
# Order matters — Emby first (larger transfer), then Critical-Data (auth stack).
|
||||
HOST1_WEEKLY_SYNC_SHARES=(
|
||||
"/mnt/user/Media_Server/Emby" # emby profile — full clean mirror
|
||||
"/mnt/user/appdata-Fallback/Critical-Data" # critical-data profile — auth stack
|
||||
)
|
||||
|
||||
# ━━━ Intermediate Sync Shares ━━━
|
||||
# Shares synced every 4 hours by intermediate_sync_maintenance.sh.
|
||||
# Uses DEFAULT_RSYNC_OPTS (no --delete) — for sub-daily propagation of metadata or watch state.
|
||||
# Full media share sync stays in the daily window. Leave empty to skip mid-day rsync entirely.
|
||||
HOST1_INTERMEDIATE_SYNC_SHARES=(
|
||||
# Add shares here to enable mid-day rsync
|
||||
# Example: "/mnt/user/Emby_Metadata"
|
||||
)
|
||||
|
||||
# ━━━ Critical Sync Shares ━━━
|
||||
# Appdata shares synced every 30 minutes by critical_sync_maintenance.sh.
|
||||
# Format: "/path/to/share" or "/path/to/share|profile-name"
|
||||
# Order matters — Critical-Data first (auth stack), then Emby dirty sync.
|
||||
HOST1_CRITICAL_SYNC_SHARES=(
|
||||
"/mnt/user/appdata-Fallback/Critical-Data|critical-fallback" # auth dirty sync — stays running
|
||||
"/mnt/user/Media_Server/Emby|emby-fallback" # Emby dirty sync — stays running
|
||||
)
|
||||
|
||||
# ━━━ Backup Verify ━━━
|
||||
# Shares verified by backup_verify.sh — random file checksum comparison against remote.
|
||||
# Leave empty to use HOST1_DAILY_SYNC_SHARES automatically.
|
||||
# Sample size and minimum file size defined in master.conf.
|
||||
HOST1_BACKUP_VERIFY_SHARES=(
|
||||
# leave empty to use HOST1_DAILY_SYNC_SHARES automatically
|
||||
)
|
||||
|
||||
# ━━━ HOST1 Rsync Profile — host1-appdata ━━━
|
||||
# HOST1-specific appdata sync profile — extends the shared PROFILE_* arrays in master.conf.
|
||||
# Use for appdata unique to HOST1 (Organizrv2, VaultWarden, UptimeKuma etc.)
|
||||
# Shared appdata (auth stack, Emby) use dedicated profiles defined in master.conf.
|
||||
# Run manually: bash Rsync/rsync.sh /mnt/user/appdata-Fallback/HOST1-Appdata --profile=host1-appdata
|
||||
PROFILE_RSYNC_OPTS[host1-appdata]="-av --info=progress2 --bwlimit=${PROFILE_BW_LIMIT[host1-appdata]:-8000}"
|
||||
PROFILE_BW_LIMIT[host1-appdata]=8000
|
||||
PROFILE_RETRY_COUNT[host1-appdata]=3
|
||||
PROFILE_SLEEP[host1-appdata]=300
|
||||
PROFILE_CRITICAL_CONTAINER_NAMES[host1-appdata]="Organizrv2-Gmer4Lfe UptimeKuma-Gmer4Lfe VaultWarden-Gmer4Lfe"
|
||||
PROFILE_DELAYED_CONTAINERS[host1-appdata]=""
|
||||
PROFILE_CONTAINER_DELAY[host1-appdata]=5
|
||||
PROFILE_EXCLUDE_DIRS[host1-appdata]="logs *.tmp"
|
||||
|
||||
# ==============================================================================================
|
||||
# ── DOCKER ────────────────────────────────────────────────────────────────────────────────────
|
||||
# ==============================================================================================
|
||||
|
||||
# ━━━ Docker Daily Restart ━━━
|
||||
# Containers restarted every day via DAILY_MAINTENANCE_SCRIPTS.
|
||||
# Dispatcharr degrades over time without restart — daily is intentional, not just housekeeping.
|
||||
# Order matters — auth stack first, then media services.
|
||||
HOST1_DAILY_RESTART_CONTAINERS=(
|
||||
"NginxProxyManager"
|
||||
"Lldap-Gmer4Lfe"
|
||||
"Authelia"
|
||||
"Authelia-Secondary"
|
||||
"Dispatcharr-Iptv-Users"
|
||||
"Dispatcharr" # Live TV scheduler — degrades without daily restart
|
||||
"Dispatcharr-Basic"
|
||||
"ErsatzTV-Emby"
|
||||
"slskd" # Soulseek connection drops after extended uptime; restart refreshes share index
|
||||
)
|
||||
|
||||
# ━━━ Docker Weekly Restart ━━━
|
||||
# Less critical services restarted weekly via WEEKLY_MAINTENANCE_SCRIPTS (Sunday 2:30am).
|
||||
# Containers already stopped for weekly sync — restart adds zero extra downtime.
|
||||
HOST1_WEEKLY_RESTART_CONTAINERS=(
|
||||
"NextCloud"
|
||||
"Organizrv2-Gmer4Lfe"
|
||||
"AdGuard-Home"
|
||||
"Immich-Gmer4Lfe"
|
||||
)
|
||||
|
||||
# ━━━ Docker Watchdog ━━━
|
||||
# Per-HOST1 container configuration for docker_watchdog.sh.
|
||||
# Shared thresholds and toggles live in master.conf.
|
||||
|
||||
# Memory hard limits in MB — immediate restart if exceeded.
|
||||
# Set at "container is clearly broken" not "container is busy".
|
||||
# 20GB=20480 18GB=18432 16GB=16384 12GB=12288 8GB=8192 4GB=4096 2GB=2048 1GB=1024
|
||||
declare -A HOST1_WATCHDOG_CONTAINERS=(
|
||||
["Emby"]=20480 # 20GB — large library + active transcodes
|
||||
["LidaTube"]=6144 # 6GB — memory leak over time
|
||||
["Tdarr"]=6144 # 6GB — encoding is memory intensive
|
||||
["Code-Server"]=1024 # 1GB — should never need more
|
||||
)
|
||||
|
||||
# HTTP health check URLs — checked every cycle, strike system before restart.
|
||||
# Only add containers with a meaningful web interface to check.
|
||||
declare -A HOST1_WATCHDOG_CONTAINER_URLS=(
|
||||
["Emby"]="http://localhost:8096"
|
||||
)
|
||||
|
||||
# Required containers — must always be running on HOST1.
|
||||
# Strike system before restart — repeated failures go on skip list, auto-clears on recovery.
|
||||
# Listed in dependency order — dependencies before dependents.
|
||||
HOST1_WATCHDOG_REQUIRED_CONTAINERS=(
|
||||
"NginxProxyManager"
|
||||
"Lldap-Gmer4Lfe"
|
||||
"Mariadb-Authelia"
|
||||
"Mariadb-Authelia-Secondary"
|
||||
"Redis-Authelia"
|
||||
"Redis-Authelia-Secondary"
|
||||
"Authelia"
|
||||
"Authelia-Secondary"
|
||||
)
|
||||
|
||||
# Containers to skip in Tier 2 global scan — legitimately stopped or frequently restarting.
|
||||
# Watchdog leaves these alone entirely — no restart attempts, no crash loop tracking.
|
||||
HOST1_WATCHDOG_SCAN_IGNORE=(
|
||||
"DashGate"
|
||||
"PIA-WG-Config-Generator"
|
||||
"Aperture"
|
||||
"Aperture-Kids"
|
||||
"pgvector-18-Apeture-Kids"
|
||||
"Pgvector18-Aperture"
|
||||
"emby-test" # broken test container (exit 127 — bad image)
|
||||
)
|
||||
|
||||
# Dependency ordering — skip restarting a container if its dependency is also down.
|
||||
# Prevents watchdog from restarting Authelia before Mariadb is back up.
|
||||
# SPACE-SEPARATED STRINGS — converted to array at runtime.
|
||||
declare -A HOST1_WATCHDOG_DEPENDENCIES=(
|
||||
["Authelia"]="Mariadb-Authelia Redis-Authelia"
|
||||
["Authelia-Secondary"]="Mariadb-Authelia Redis-Authelia-Secondary"
|
||||
["NextCloud"]="Postgres-NextCloud"
|
||||
)
|
||||
|
||||
# Per-container appdata growth suppress ceilings in MB.
|
||||
# ONLY needed in specific cases — growth rate detection covers all containers automatically.
|
||||
# Use this when a container legitimately has large stable data and you want to guarantee
|
||||
# it never triggers a false-positive growth alert. Growth warnings are suppressed while the
|
||||
# container's dir stays below this ceiling; above it, warnings resume as normal.
|
||||
# 50GB=51200 25GB=25600 20GB=20480 15GB=15360 10GB=10240 5GB=5120
|
||||
declare -A HOST1_WATCHDOG_APPDATA_SIZES=(
|
||||
["Tdarr"]="25600" # 25GB — transcode cache grows legitimately during active jobs
|
||||
["7dtd"]="20480" # 20GB — game server world data, expected to be large
|
||||
)
|
||||
|
||||
# ━━━ Network Watchdog ━━━
|
||||
# Host-specific connectivity config for Watchdogs/System/network_watchdog.sh.
|
||||
HOST1_NETWORK_WATCHDOG_DDNS_DOMAIN="gmer4lfe.com"
|
||||
HOST1_NETWORK_WATCHDOG_DDNS_CONTAINER="Gmer4Lfe.com"
|
||||
HOST1_NETWORK_WATCHDOG_NPM_URL="https://gmer4lfe.com"
|
||||
|
||||
# ━━━ Docker Network Connect ━━━
|
||||
# Containers connected to custom networks at array start by docker_network_connect.sh.
|
||||
# Networks created if they don't exist — idempotent, safe to re-run.
|
||||
HOST1_NETWORK_CONNECT_CONTAINERS=(
|
||||
"memcached"
|
||||
"Npm-CrowdSec"
|
||||
)
|
||||
|
||||
HOST1_NETWORK_CONNECT_NETWORKS=(
|
||||
"high-availability"
|
||||
)
|
||||
|
||||
# ==============================================================================================
|
||||
# ── FALLBACK ──────────────────────────────────────────────────────────────────────────────────
|
||||
# ==============================================================================================
|
||||
|
||||
# ━━━ DDNS ━━━
|
||||
# DDNS containers HOST1 manages — started/stopped by fallback.sh per DDNS absolute rules:
|
||||
# Internet loss → stop immediately
|
||||
# Failover → HOST2 starts HOST1's DDNS as Tier 1 (before any other containers)
|
||||
# Handback → stop HOST1's DDNS on HOST2 → rsync → start containers → start local DDNS last
|
||||
HOST1_DDNS_CONTAINERS=(
|
||||
"Gmer4Lfe.com"
|
||||
)
|
||||
|
||||
# ━━━ Internet Loss ━━━
|
||||
# Containers stopped immediately on HOST1 when internet connection is lost.
|
||||
# Prevents external-facing services from operating without connectivity.
|
||||
FALLBACK_HOST1_STOP_ON_NO_NET=(
|
||||
"Gmer4Lfe.com"
|
||||
)
|
||||
|
||||
# ━━━ Fallback Tiers — HOST1 Runs for HOST2 ━━━
|
||||
# Containers HOST1 starts when HOST2 goes down.
|
||||
# Tier 1 is always immediate — vital services cannot wait.
|
||||
# Higher tiers activate after HOST2_TIER*_DELAY minutes (set in host2.conf).
|
||||
FALLBACK_HOST1_COVERS_HOST2_TIER1=(
|
||||
"Gmer4Lfe.us"
|
||||
"VaultWarden-Jayred365"
|
||||
)
|
||||
|
||||
FALLBACK_HOST1_COVERS_HOST2_TIER2=(
|
||||
# "container-placeholder"
|
||||
)
|
||||
|
||||
FALLBACK_HOST1_COVERS_HOST2_TIER3=(
|
||||
# "container-placeholder"
|
||||
)
|
||||
|
||||
FALLBACK_HOST1_COVERS_HOST2_TIER4=(
|
||||
# "container-placeholder"
|
||||
)
|
||||
|
||||
# ━━━ Tier Delays — HOST1's Containers on HOST2 ━━━
|
||||
# How long HOST1 must be down before each tier activates on HOST2 — in minutes.
|
||||
# Tier 1 is always immediate — no delay var needed.
|
||||
HOST1_TIER2_DELAY=240 # 4 hours — NextCloud, Immich
|
||||
HOST1_TIER3_DELAY=720 # 12 hours — secondary services
|
||||
HOST1_TIER4_DELAY=1440 # 24 hours — arrs + downloaders
|
||||
|
||||
# ━━━ Rsync Writeback — HOST1 Appdata Back on Handback ━━━
|
||||
# Syncs HOST1 appdata BACK to HOST1 when it comes back online after a failover.
|
||||
# Containers stopped before writeback — clean source, no competing writes.
|
||||
#
|
||||
# HOST1_TIER1_WRITEBACK_DELAY: short outages skip Tier 1 writeback — primary state
|
||||
# is more reliable than dirty sync data for brief outages.
|
||||
HOST1_TIER1_WRITEBACK_DELAY=60 # skip Emby writeback if outage under 1hr
|
||||
|
||||
# Tier 4 automatically syncs HOST1_DAILY_SYNC_SHARES — only list paths NOT in that array.
|
||||
FALLBACK_HOST1_WRITEBACK_TIER1=(
|
||||
"/mnt/user/Media_Server/Emby" # watch states built up during outage
|
||||
)
|
||||
|
||||
FALLBACK_HOST1_WRITEBACK_TIER2=(
|
||||
"/mnt/user/appdata-Fallback/Important-Data" # NextCloud + Postgres — files added during outage
|
||||
)
|
||||
|
||||
FALLBACK_HOST1_WRITEBACK_TIER3=(
|
||||
# "location-placeholder"
|
||||
)
|
||||
|
||||
FALLBACK_HOST1_WRITEBACK_TIER4=(
|
||||
"/mnt/user/appdata-Fallback/Arrs_Stack" # arr databases — downloads queued during outage
|
||||
)
|
||||
|
||||
# ==============================================================================================
|
||||
# ── MEDIA ─────────────────────────────────────────────────────────────────────────────────────
|
||||
# ==============================================================================================
|
||||
|
||||
# ━━━ Media Permissions ━━━
|
||||
# Shares that media_shares_permissions.sh applies PERMISSIONS_MODE and PERMISSIONS_OWNER to.
|
||||
# Runs first in DAILY_MAINTENANCE_SCRIPTS — arr cleanup depends on correct ownership.
|
||||
HOST1_MEDIA_PERMISSION_SHARES=(
|
||||
/mnt/user/Anime_Movies
|
||||
/mnt/user/Anime_Movies-Old
|
||||
/mnt/user/Anime_Shows
|
||||
/mnt/user/Anime_Shows-Old
|
||||
/mnt/user/appcache
|
||||
/mnt/user/Books
|
||||
/mnt/user/Downloads
|
||||
/mnt/user/Games
|
||||
/mnt/user/Intros
|
||||
/mnt/user/Kids_Movies
|
||||
/mnt/user/Kids_Tv_Shows
|
||||
/mnt/user/Movie_Recordings
|
||||
/mnt/user/Movies
|
||||
/mnt/user/Music
|
||||
/mnt/user/Music_Videos
|
||||
/mnt/user/Photo
|
||||
/mnt/user/Sports
|
||||
/mnt/user/stand-up_comedy
|
||||
/mnt/user/Tv_Recordings
|
||||
/mnt/user/Tv_Shows
|
||||
/mnt/user/YouTube
|
||||
)
|
||||
|
||||
# ━━━ Media Cleaner ━━━
|
||||
# Folder lists for media_cleaner.sh — two profiles: anime and media.
|
||||
# File patterns shared across all servers — defined in master.conf.
|
||||
# Called via DAILY_MAINTENANCE_SCRIPTS. Run manually: Media/media_cleaner.sh anime|media
|
||||
HOST1_ANIME_CLEAN_FOLDERS=(
|
||||
/mnt/user/Anime_Movies
|
||||
/mnt/user/Anime_Movies-Old
|
||||
/mnt/user/Anime_Shows
|
||||
/mnt/user/Anime_Shows-Old
|
||||
)
|
||||
|
||||
HOST1_MEDIA_CLEAN_FOLDERS=(
|
||||
/mnt/user/Kids_Movies
|
||||
/mnt/user/Kids_Tv_Shows
|
||||
/mnt/user/Movies
|
||||
/mnt/user/Music
|
||||
/mnt/user/Sports
|
||||
/mnt/user/stand-up_comedy
|
||||
/mnt/user/Tv_Shows
|
||||
)
|
||||
|
||||
# ==============================================================================================
|
||||
# ── MONITORS ──────────────────────────────────────────────────────────────────────────────────
|
||||
# ==============================================================================================
|
||||
|
||||
# ━━━ Certificate Monitor ━━━
|
||||
# Domains checked via direct openssl connection — not relying on NPM's certificate state.
|
||||
# Checks the actual certificate served, not what NPM thinks it has.
|
||||
# Thresholds (CERT_WARN_DAYS, CERT_CRIT_DAYS) defined in master.conf.
|
||||
HOST1_CERT_MONITOR_DOMAINS=(
|
||||
"Gmer4Lfe.com"
|
||||
"Gmer4Lfe.us"
|
||||
)
|
||||
|
||||
# ━━━ SMART Health ━━━
|
||||
# Drives skipped in SMART attribute monitoring — hardware is server-specific.
|
||||
# Thresholds read from dynamix.cfg at runtime — fallbacks in master.conf.
|
||||
HOST1_SMART_IGNORE_DRIVES=(
|
||||
"sda" # boot USB — SMART not meaningful on flash drives
|
||||
)
|
||||
|
||||
# ━━━ ZFS Report ━━━
|
||||
# Pools excluded from the weekly ZFS health report — reduces noise from single-disk array pools.
|
||||
# These are individual array disks formatted as ZFS — converting to XFS over time via unBalance.
|
||||
# Pool health thresholds defined in master.conf.
|
||||
HOST1_ZFS_REPORT_IGNORE_POOLS=(
|
||||
"disk5"
|
||||
"disk6"
|
||||
"disk8"
|
||||
"disk9"
|
||||
"disk10"
|
||||
)
|
||||
|
||||
# ==============================================================================================
|
||||
# ── TRANSCODES ────────────────────────────────────────────────────────────────────────────────
|
||||
# ==============================================================================================
|
||||
|
||||
# Ramdisk size ceiling — tmpfs only uses RAM actually needed, not the full size upfront.
|
||||
# Real-world: 9 streams peaked at ~5.5GB — 10G gives generous headroom on 128GB RAM.
|
||||
HOST1_RAMDISK_SIZE="10G"
|
||||
|
||||
# Usage thresholds — coupled to HOST1_RAMDISK_SIZE, adjust all three together if size changes.
|
||||
# Hysteresis gap (8.5 - 7 = 1.5GB) prevents flip-flop between ramdisk and SSD.
|
||||
HOST1_RAMDISK_WARN_GB=8.5 # flip to SSD when ramdisk usage reaches this
|
||||
HOST1_RAMDISK_LOW_GB=7 # flip back to ramdisk when usage drops to this
|
||||
|
||||
# SSD fallback path — where transcodes land when ramdisk exceeds HOST1_RAMDISK_WARN_GB.
|
||||
# Must be on cache pool — array disks too slow for active transcode writes.
|
||||
HOST1_TRANSCODE_SSD="/mnt/cache/Temp_Storage/Emby/Transcodes/"
|
||||
|
||||
# Media servers sharing the ramdisk transcode space on HOST1.
|
||||
# Format: "ContainerName|URL|APIKey|Type" — Type: emby | jellyfin | plex
|
||||
# Entries with placeholder API keys are skipped automatically.
|
||||
# ⚠️ Tdarr does NOT belong here — keep Tdarr on SSD, not ramdisk.
|
||||
HOST1_TRANSCODE_SERVERS=(
|
||||
"${HOST1_EMBY_CONTAINER}|${HOST1_EMBY_URL}|${HOST1_EMBY_API_KEY}|emby"
|
||||
"${HOST1_JELLYFIN_CONTAINER}|${HOST1_JELLYFIN_URL}|${HOST1_JELLYFIN_API_KEY}|jellyfin"
|
||||
)
|
||||
|
||||
# ==============================================================================================
|
||||
# ── ARR STACK ─────────────────────────────────────────────────────────────────────────────────
|
||||
# ==============================================================================================
|
||||
# Used by arr cleanup scripts and arrs_failed_stalled_recovery.sh.
|
||||
# detect_hosts() selects HOST1 vars when running on HOST1.
|
||||
#
|
||||
# PATH MAPS — container path → host path translation.
|
||||
# Arr stores file paths using container-internal paths — scripts need host paths to scan.
|
||||
# Add one entry per root folder in arr Settings → Media Management → Root Folders.
|
||||
|
||||
# ━━━ Downloaders ━━━
|
||||
# Used by downloaders_reset.sh — runs every 30min via CRITICAL_MAINTENANCE_SCRIPTS.
|
||||
# Clears stuck states, purges old history, prepares each client for a clean cycle.
|
||||
|
||||
# slskd — clears stuck searches, dead transfers, purges expired failed imports.
|
||||
# SLSKD_FAILED_IMPORTS_DIR: where Soularr moves albums Lidarr rejected.
|
||||
HOST1_SLSKD_URL="http://localhost:8980"
|
||||
HOST1_SLSKD_API_KEY="4bF9kL2mNpQrT7vWxYz1A3dEgHjKoRsU"
|
||||
HOST1_SLSKD_FAILED_IMPORTS_DIR="/mnt/user/Temp_Storage/Slskd/completed/failed_imports"
|
||||
|
||||
# SABnzbd
|
||||
HOST1_SABNZBD_URL="http://localhost:8180"
|
||||
HOST1_SABNZBD_API_KEY="8bfefe41d83b4d50883e32859b55ca9a"
|
||||
|
||||
# qBittorrent — deleteFiles=false removes torrent from qBit but leaves files on disk.
|
||||
# Radarr/Sonarr manage actual files independently.
|
||||
HOST1_QBIT_URL="http://localhost:8080"
|
||||
HOST1_QBIT_USERNAME="root"
|
||||
HOST1_QBIT_PASSWORD="Stay0utD!ck"
|
||||
|
||||
# ━━━ Lidarr — HOST1 only ━━━
|
||||
# HOST2 does not run Lidarr — HOST1_LIDARR_RECOVERY flag handles the exit cleanly.
|
||||
HOST1_LIDARR_URL="http://localhost:8686"
|
||||
HOST1_LIDARR_API_KEY="b2977e71ef074bc0a0529d9fcce3b2dc"
|
||||
HOST1_LIDARR_MUSIC_ROOT="/mnt/user/Music-New"
|
||||
HOST1_FANART_API_KEY="Yd7147a43b692df0b364b94dc47efb81"
|
||||
HOST1_LASTFM_API_KEY="be6dc169c33ae263e690c30d18b7491d"
|
||||
|
||||
declare -A HOST1_LIDARR_PATH_MAP=(
|
||||
["/ext-music"]="/mnt/user/Music-New"
|
||||
)
|
||||
|
||||
# ━━━ Sonarr ━━━
|
||||
HOST1_SONARR_URL="http://localhost:8989"
|
||||
HOST1_SONARR_API_KEY="130decd3db5b4c25afad64864cd03f9f"
|
||||
HOST1_SONARR_TV_ROOT="/mnt/user/Tv_Shows"
|
||||
|
||||
# Note: stand-up_comedy in both Sonarr + Radarr — TV specials and movie specials, one folder
|
||||
declare -A HOST1_SONARR_PATH_MAP=(
|
||||
["/tv"]="/mnt/user/Tv_Shows"
|
||||
["/ext-standup-comedy"]="/mnt/user/stand-up_comedy/series"
|
||||
["/kids tv"]="/mnt/user/Kids_Tv_Shows"
|
||||
["/ext-anime-shows"]="/mnt/user/Anime_Shows-Old"
|
||||
)
|
||||
|
||||
# ━━━ Radarr ━━━
|
||||
HOST1_RADARR_URL="http://localhost:7878"
|
||||
HOST1_RADARR_API_KEY="d43a3ec6cf1549edb4af0cc63f98b2a9"
|
||||
HOST1_TMDB_API_KEY="3dac5e2e49b5540472d2eafec4f01260"
|
||||
HOST1_RADARR_MOVIES_ROOT="/mnt/user/Movies"
|
||||
|
||||
# Note: stand-up_comedy in both Radarr + Sonarr — movie specials and TV specials, one folder
|
||||
declare -A HOST1_RADARR_PATH_MAP=(
|
||||
["/movies"]="/mnt/user/Movies"
|
||||
["/kids movies"]="/mnt/user/Kids_Movies"
|
||||
["/ext-stand-up-comedy"]="/mnt/user/stand-up_comedy/specials"
|
||||
["/anime-movies"]="/mnt/user/Anime_Movies-Old"
|
||||
["/ext-anime-movies"]="/mnt/user/Anime_Movies-Old"
|
||||
)
|
||||
|
||||
# ━━━ Arr Recovery Toggles ━━━
|
||||
# false = skip that arr on this host — exits cleanly without error
|
||||
HOST1_SONARR_RECOVERY=true
|
||||
HOST1_RADARR_RECOVERY=true
|
||||
HOST1_LIDARR_RECOVERY=true # HOST1 only — exits cleanly on HOST2
|
||||
|
||||
# ==============================================================================================
|
||||
# ── SYSTEM WATCHDOG ───────────────────────────────────────────────────────────────────────────
|
||||
# ==============================================================================================
|
||||
# Per-host check toggles and NIC config for system_watchdog.sh.
|
||||
# Aliased by detect_hosts() — script uses unprefixed SYS_WATCHDOG_* names.
|
||||
# HOST1: TR1950X 128GB — full media server, active transcoding, ZFS cache pools.
|
||||
#
|
||||
# Three-tier response — all critical checks enabled by default on HOST1:
|
||||
# Tier 1 (bypass strikes, reboot now): docker daemon, rootfs full, kernel oops, FD, /boot
|
||||
# Tier 2 (bypass strikes with OOM): RAM critical + OOM kills in cycle
|
||||
# Tier 3 (standard strike system): everything else
|
||||
#
|
||||
# RAM tiers, OOM limits, and reboot loop settings in master.conf System Watchdog section.
|
||||
|
||||
# ━━━ Primary NIC ━━━
|
||||
# Network interface for NIC state check — verify with: ip link show | grep "^[0-9]"
|
||||
# Common values: eth0, bond0, br0, eno1
|
||||
HOST1_SYS_WATCHDOG_NIC="eth0"
|
||||
|
||||
# ━━━ Tier 1 — Critical Checks ━━━
|
||||
# These bypass the strike system — a single hit triggers immediate reboot.
|
||||
# Disabling any of these is not recommended — they protect against acute system failure.
|
||||
|
||||
# Docker daemon unresponsive → try restart, reboot if restart fails.
|
||||
# Without a working daemon docker_watchdog.sh is blind and containers cannot be managed.
|
||||
HOST1_SYS_WATCHDOG_CHECK_DOCKER_DAEMON=true
|
||||
|
||||
# rootfs at critical threshold (ROOTFS_CRITICAL_PCT=99) → reboot immediately.
|
||||
# At 99% rootfs writes fail silently — logs stop, Docker errors out, SSH may stop working.
|
||||
# Standard 95% threshold still uses strike system — only 99%+ is critical tier.
|
||||
HOST1_SYS_WATCHDOG_CHECK_ROOTFS=true
|
||||
|
||||
# Kernel BUG/Oops in dmesg delta since last cycle → reboot immediately.
|
||||
# A kernel oops means the kernel ran with a corrupted state — stability is not guaranteed.
|
||||
HOST1_SYS_WATCHDOG_CHECK_KERNEL_OOPS=true
|
||||
|
||||
# File descriptor exhaustion at FD_CRITICAL_PCT (95%) → reboot immediately.
|
||||
# At 95% FD: new connections fail, Docker can't spawn processes, SSH drops.
|
||||
HOST1_SYS_WATCHDOG_CHECK_FD=true
|
||||
|
||||
# /boot read-only detected → reboot immediately.
|
||||
# Unexpected read-only /boot means state files and config writes are silently failing.
|
||||
# Fallback state, watchdog reboot log, and lock files all go stale silently.
|
||||
HOST1_SYS_WATCHDOG_CHECK_BOOT=true
|
||||
|
||||
# ━━━ Tier 2 — Urgent OOM Check ━━━
|
||||
# Bypass strikes when RAM is critically low AND OOM kill rate confirms active crisis.
|
||||
# Both must be enabled for Tier 2 bypass to function — disable either to always use strikes.
|
||||
|
||||
# Track kernel OOM kills each cycle via /proc/vmstat oom_kill delta.
|
||||
# Also provides diagnostic context in reboot messages (which processes were killed).
|
||||
HOST1_SYS_WATCHDOG_CHECK_OOM=true
|
||||
|
||||
# Free RAM check — required for both Tier 2 bypass and RAM tier logic.
|
||||
# Tiers: MEM_WARN_GB(10) → notify | MEM_SHUTDOWN_GB(6) → stop containers | MEM_GB(4) → strikes
|
||||
HOST1_SYS_WATCHDOG_CHECK_RAM=true
|
||||
|
||||
# ━━━ Tier 3 — Standard Checks (strike system) ━━━
|
||||
# Each check must fail SYS_WATCHDOG_STRIKE_LIMIT consecutive cycles before action is taken.
|
||||
# Single spikes are ignored — sustained problems trigger reboot.
|
||||
|
||||
# /var/log filesystem usage above SYS_WATCHDOG_LOG_PCT.
|
||||
# Log spam (Docker log storms, syslog loops) fills rootfs — indicates something broken.
|
||||
HOST1_SYS_WATCHDOG_CHECK_LOG=true
|
||||
|
||||
# ZFS ARC memory pinned above SYS_WATCHDOG_ARC_PINNED_PCT after cache drop.
|
||||
# Enabled on HOST1 — ZFS cache pools actively used. Disable on hosts without ZFS.
|
||||
HOST1_SYS_WATCHDOG_CHECK_ARC=true
|
||||
|
||||
# CPU temperature above SYS_WATCHDOG_CPU_TEMP_MAX (95°C).
|
||||
# Sustained high temp causes kernel throttling or panic. Requires lm-sensors.
|
||||
HOST1_SYS_WATCHDOG_CHECK_CPU_TEMP=true
|
||||
|
||||
# Load average above SYS_WATCHDOG_LOAD_MULTIPLIER × core count.
|
||||
# DISABLED on HOST1 — Tdarr and Emby cause legitimate sustained load spikes during encoding.
|
||||
# Enable on idle servers or adjust SYS_WATCHDOG_LOAD_MULTIPLIER if load is always high.
|
||||
HOST1_SYS_WATCHDOG_CHECK_LOAD=false
|
||||
|
||||
# Zombie process count above SYS_WATCHDOG_ZOMBIE_LIMIT (50).
|
||||
# Large zombie counts indicate serious process management failure — something is stuck.
|
||||
HOST1_SYS_WATCHDOG_CHECK_ZOMBIES=true
|
||||
|
||||
# Check docker_watchdog.sh persistent skip list — required containers on skip list.
|
||||
# Cross-watchdog coordination: if docker_watchdog gave up, system_watchdog escalates.
|
||||
# ENABLED — HOST1 fully built and operational, skip list is meaningful.
|
||||
HOST1_SYS_WATCHDOG_CHECK_CONTAINERS=true
|
||||
|
||||
# /tmp filesystem usage above SYS_WATCHDOG_TMP_PCT with auto-clear attempt.
|
||||
# Script tries to clear aged /tmp files first — only strikes if clear fails.
|
||||
# Lock files, rsync temp files, and Docker ops use /tmp — 100% means lock failures.
|
||||
HOST1_SYS_WATCHDOG_CHECK_TMP=true
|
||||
|
||||
# Array disk error count delta in /proc/mdstat — accumulating errors = disk failing now.
|
||||
# Triggers on SYS_WATCHDOG_MDSTAT_ERROR_LIMIT new errors in one cycle.
|
||||
HOST1_SYS_WATCHDOG_CHECK_MDSTAT=true
|
||||
|
||||
# Primary NIC operstate — detects NIC going down (physical or driver failure).
|
||||
# Uses HOST1_SYS_WATCHDOG_NIC above. Strike system — brief flaps don't trigger reboot.
|
||||
HOST1_SYS_WATCHDOG_CHECK_NETWORK=true
|
||||
|
||||
# sshd running check — attempts restart before escalating.
|
||||
# sshd down = no remote access. Script tries rc.sshd start, notifies, strikes on failure.
|
||||
HOST1_SYS_WATCHDOG_CHECK_SSHD=true
|
||||
|
||||
# Runaway process detection — single process above SYS_WATCHDOG_RUNAWAY_CPU_PCT sustained.
|
||||
# DISABLED — Tdarr encoding and Emby transcoding legitimately peg CPU for extended periods.
|
||||
# Enable only if HOST1 has no CPU-intensive workloads.
|
||||
HOST1_SYS_WATCHDOG_CHECK_RUNAWAY=false
|
||||
|
||||
# ==============================================================================================
|
||||
# ── RESOURCE MANAGER ──────────────────────────────────────────────────────────────────────────
|
||||
# ==============================================================================================
|
||||
# Containers to manage under pressure — see master.conf RW_CRITICAL_CONTAINERS for exclusions.
|
||||
|
||||
# docker pause at medium pressure (RAM < RW_RAM_MEDIUM_GB or load > medium threshold)
|
||||
# Suspended in-place — instant to pause/unpause, no state lost, no restart delay.
|
||||
HOST1_RW_PAUSE_CONTAINERS=(
|
||||
"Huntarr" # arr search automation — safe to suspend
|
||||
"Cleanuparr" # download cleanup — safe to suspend
|
||||
"Healarr" # arr health checks — safe to suspend
|
||||
"Soularr" # Slskd automation — background only
|
||||
"ChannelTube" # YouTube archiver — background only
|
||||
"Pinchflat" # YouTube archiver — background only
|
||||
)
|
||||
|
||||
# docker stop at hard pressure (RAM < RW_RAM_HARD_GB)
|
||||
# Full stop — these are optional/heavy services that free significant RAM when stopped.
|
||||
# resource_watchdog.sh restarts them when pressure fully clears (RAM >= RW_RAM_RECOVER_GB).
|
||||
HOST1_RW_STOP_CONTAINERS=(
|
||||
"LocalAI" # GPU/CPU heavy — largest RAM consumer when idle
|
||||
"7DaysToDie" # game server — optional
|
||||
"V-Rising" # game server — optional
|
||||
"Code-Server" # IDE — not needed during pressure events
|
||||
)
|
||||
|
||||
# ==============================================================================================
|
||||
# ──────────────────────── End Of HOST1 Variables ──────────────────────────────────────────────
|
||||
# ==============================================================================================
|
||||
HOST1_UNRAID_API_KEY="1825c3a2e03ea5089974f4da2e171aa2d5907a1dea23cc479c33e492c8ff4dbb"
|
||||
+461
@@ -0,0 +1,461 @@
|
||||
#!/bin/bash
|
||||
# ==============================================================================================
|
||||
# ============================= Rsync Core Script ==============================================
|
||||
# ==============================================================================================
|
||||
#
|
||||
# PURPOSE
|
||||
# ─────────────────────────────────────────────────────────────────────────────
|
||||
# Core rsync engine for the two-server ecosystem. Called per share or per
|
||||
# appdata profile by orchestrators (daily_sync_maintenance, weekly_sync_maintenance,
|
||||
# critical_sync_maintenance) and directly for manual or scheduled dirty syncs.
|
||||
#
|
||||
# Profile is inferred from the directory basename (lowercased). Override with
|
||||
# --profile=name for explicit selection. If no profile matches, global defaults
|
||||
# from master.conf apply and no containers are stopped.
|
||||
#
|
||||
# After each sync, logs transfer data to bandwidth_monitor.sh for the weekly
|
||||
# bandwidth report. Silent on success — only failures produce visible output.
|
||||
#
|
||||
# ==============================================================================================
|
||||
# OPERATIONAL MODEL
|
||||
# ==============================================================================================
|
||||
#
|
||||
# Profiles define per-share behavior:
|
||||
# PROFILE_CRITICAL_CONTAINER_NAMES — containers stopped on both servers before sync
|
||||
# PROFILE_DELAYED_CONTAINERS — containers with a delay before restart after sync
|
||||
# PROFILE_CONTAINER_DELAY — seconds before delayed containers start
|
||||
# PROFILE_RSYNC_OPTS — rsync flags (does not inherit DEFAULT_RSYNC_OPTS)
|
||||
# PROFILE_BW_LIMIT — bandwidth limit in KB/s
|
||||
# PROFILE_RETRY_COUNT — retry attempts on failure
|
||||
# PROFILE_SLEEP — seconds between retry attempts
|
||||
# PROFILE_EXCLUDE_DIRS — paths excluded from transfer
|
||||
# PROFILE_REMOTE_RESTART_CONTAINERS — containers restarted on remote after dirty sync
|
||||
# Was running → restart. Was stopped → leave stopped.
|
||||
#
|
||||
# Two-tier rsync enable/disable:
|
||||
# Tier 1: RSYNC_ENABLED=false → all rsync stops immediately (checked by this script)
|
||||
# Tier 2: per-orchestrator flag (DAILY_RSYNC_ENABLED etc.) → checked by caller
|
||||
#
|
||||
# Bandwidth logging: after each sync, logs profile/duration/status/bytes to
|
||||
# bandwidth_monitor.sh --log-transfer. Bytes captured from rsync --stats via awk
|
||||
# using version-stable field names.
|
||||
#
|
||||
# ==============================================================================================
|
||||
# OPERATIONAL SAFEGUARDS
|
||||
# ==============================================================================================
|
||||
#
|
||||
# Global Rsync Gate
|
||||
# check_rsync_enabled() — RSYNC_ENABLED=false exits cleanly before any operation.
|
||||
#
|
||||
# Partnership Blocklist
|
||||
# Refuses to sync if REMOTE_SERVER_NAME appears in the partnership blocklist.
|
||||
# Written at offboard — prevents stale access after a partnership ends.
|
||||
#
|
||||
# Version Parity
|
||||
# check_unraid_version_parity — refuses sync if servers on incompatible unRAID versions.
|
||||
#
|
||||
# Remote Health Pre-flights
|
||||
# check_connectivity() — Tailscale IP reachable before any SSH
|
||||
# check_remote_rootfs() — aborts if remote rootfs exceeds ROOTFS_WARN_PCT
|
||||
# check_remote_share() — aborts if target directory missing or empty on remote
|
||||
# check_remote_disks() — verifies all backing disks online on remote
|
||||
#
|
||||
# Drive Temperature Check
|
||||
# check_local_disk_temps() — runs before any transfer. Exit 1 = skip this profile,
|
||||
# exit 2 = abort all remaining syncs (CRITICAL temperature).
|
||||
#
|
||||
# Remote Docker Daemon Check
|
||||
# check_remote_docker_daemon — verified before any container stop/start operations.
|
||||
# If daemon unresponsive: container operations skipped, rsync proceeds without stopping.
|
||||
#
|
||||
# Per-Profile Concurrency Lock
|
||||
# acquire_rsync_lock() — per-profile lock prevents parallel runs of the same profile.
|
||||
# Global concurrent limit prevents too many simultaneous rsync processes.
|
||||
#
|
||||
# Notification Validated
|
||||
# platform_require_cmd confirms the notify script is present before use.
|
||||
#
|
||||
# ==============================================================================================
|
||||
# CONFIGURATION
|
||||
# ==============================================================================================
|
||||
#
|
||||
# master.conf
|
||||
#
|
||||
# RSYNC_ENABLED
|
||||
# Global on/off toggle for all rsync operations. (default: true)
|
||||
#
|
||||
# DEFAULT_RSYNC_OPTS
|
||||
# Base rsync flags for unproiled shares. Note: --delete is intentionally absent —
|
||||
# media shares spread files only, arr cleanup scripts own deletions. Profile-specific
|
||||
# opts set --delete explicitly where needed.
|
||||
#
|
||||
# BW_LIMIT
|
||||
# Default bandwidth cap in KB/s when no PROFILE_BW_LIMIT is set. (default: 0 = unlimited)
|
||||
#
|
||||
# RETRY_COUNT
|
||||
# Default retry attempts on rsync failure. (default: 3)
|
||||
#
|
||||
# SLEEP
|
||||
# Default seconds between retry attempts. (default: 60)
|
||||
#
|
||||
# ROOTFS_WARN_PCT
|
||||
# Abort threshold for remote rootfs percentage full. (default: 75)
|
||||
#
|
||||
# PROFILES["profile_KEY"]
|
||||
# Profile definitions — one entry per PROFILE_* key per profile name.
|
||||
# See OPERATIONAL MODEL above for all supported keys.
|
||||
#
|
||||
# BANDWIDTH_LOG / BANDWIDTH_WARN_GB
|
||||
# Shared with bandwidth_monitor.sh — set once, used by both.
|
||||
#
|
||||
# ==============================================================================================
|
||||
# RUNTIME MODES
|
||||
# ==============================================================================================
|
||||
#
|
||||
# rsync.sh /path/to/share
|
||||
# Sync the given path to the remote. Profile inferred from directory basename.
|
||||
#
|
||||
# rsync.sh /path/to/share --profile=name
|
||||
# Sync with explicit profile override — bypasses basename inference.
|
||||
#
|
||||
# rsync.sh /path/to/share --dry-run
|
||||
# Run all pre-flight checks and show what rsync would transfer. No transfer,
|
||||
# no container stops.
|
||||
#
|
||||
# rsync.sh /path/to/share --status
|
||||
# Show resolved profile, remote identity, and configuration. Then exit.
|
||||
#
|
||||
# rsync.sh /path/to/share --log
|
||||
# Verbose output throughout — every decision logged.
|
||||
#
|
||||
# ==============================================================================================
|
||||
|
||||
SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
|
||||
|
||||
source "$SCRIPT_DIR/../load_config.sh"
|
||||
|
||||
# ── Separate positional directory arg from flags ───────────────────────────────────────────────
|
||||
DIRECTORY=""
|
||||
PROFILE_OVERRIDE=""
|
||||
RAW_ARGS=()
|
||||
|
||||
for ARG in "$@"; do
|
||||
case "$ARG" in
|
||||
--profile=*) PROFILE_OVERRIDE="${ARG#--profile=}" ;;
|
||||
--*|*=*) RAW_ARGS+=("$ARG") ;;
|
||||
*) [[ -z "$DIRECTORY" ]] && DIRECTORY="$ARG" || RAW_ARGS+=("$ARG") ;;
|
||||
esac
|
||||
done
|
||||
|
||||
parse_args "${RAW_ARGS[@]}"
|
||||
|
||||
[[ -z "$DIRECTORY" ]] && {
|
||||
error "No directory specified"
|
||||
error "Usage: rsync.sh <dir> [--dry-run] [--log] [--profile=name]"
|
||||
exit 1
|
||||
}
|
||||
|
||||
# ==============================================================================================
|
||||
# ━━━ Setup ━━━
|
||||
# ==============================================================================================
|
||||
if [[ "$EUID" -ne 0 ]]; then
|
||||
error "Must be run as root"
|
||||
exit 1
|
||||
fi
|
||||
|
||||
acquire_lock
|
||||
|
||||
if ! command -v docker &>/dev/null; then
|
||||
error "Docker command not found"
|
||||
exit 1
|
||||
fi
|
||||
|
||||
platform_require_cmd \
|
||||
"/usr/local/emhttp/plugins/dynamix/scripts/notify" \
|
||||
"" "" \
|
||||
"unRAID notify script" || warn "unRAID notify script not found — native notifications disabled"
|
||||
|
||||
detect_hosts
|
||||
|
||||
# Tier 1 global gate — Tier 2 (per-orchestrator) checked by caller
|
||||
if ! check_rsync_enabled; then
|
||||
warn "RSYNC_ENABLED=false — exiting cleanly"
|
||||
exit 0
|
||||
fi
|
||||
|
||||
# Blocklist gate — refuse to sync with a partner blocked after offboard
|
||||
BLOCKLIST_FILE="${PARTNERSHIP_BLOCKLIST_FILE:-${STATE_DIR:-/boot/config}/partnership_blocklist.db}"
|
||||
if [[ -f "$BLOCKLIST_FILE" ]] && grep -q "^${REMOTE_SERVER_NAME}|" "$BLOCKLIST_FILE" 2>/dev/null; then
|
||||
error "Rsync blocked — $REMOTE_SERVER_NAME is on the partnership blocklist"
|
||||
error "Re-onboard the partnership to restore access: partnership_manager.sh --onboard"
|
||||
exit 1
|
||||
fi
|
||||
|
||||
resolve_remote_ip
|
||||
|
||||
# ── Profile inference ─────────────────────────────────────────────────────────────────────────
|
||||
if [[ -n "$PROFILE_OVERRIDE" ]]; then
|
||||
PROFILE_NAME="$PROFILE_OVERRIDE"
|
||||
log "Profile override: $PROFILE_NAME"
|
||||
else
|
||||
PROFILE_NAME=$(basename "$DIRECTORY" | tr '[:upper:]' '[:lower:]')
|
||||
log "Profile inferred: $PROFILE_NAME"
|
||||
fi
|
||||
|
||||
# Acquire per-profile lock and check global concurrent limit
|
||||
acquire_rsync_lock "$PROFILE_NAME"
|
||||
|
||||
# Tee all output to a live log file for the Varaverk UI
|
||||
VV_LIVE_LOG="/tmp/unraid_locks/rsync_${PROFILE_NAME}.log"
|
||||
VV_LAST_LOG="/tmp/unraid_locks/rsync_${PROFILE_NAME}.last.log"
|
||||
: > "$VV_LIVE_LOG"
|
||||
exec 1> >(tee -a "$VV_LIVE_LOG") 2>&1
|
||||
|
||||
# ── Load profile settings ─────────────────────────────────────────────────────────────────────
|
||||
BW_LIMIT=${PROFILE_BW_LIMIT[$PROFILE_NAME]:-$BW_LIMIT}
|
||||
RETRY_COUNT=${PROFILE_RETRY_COUNT[$PROFILE_NAME]:-$RETRY_COUNT}
|
||||
SLEEP=${PROFILE_SLEEP[$PROFILE_NAME]:-$SLEEP}
|
||||
CONTAINER_DELAY=${PROFILE_CONTAINER_DELAY[$PROFILE_NAME]:-$CONTAINER_DELAY}
|
||||
|
||||
read -r -a CRITICAL_CONTAINER_NAMES <<< "${PROFILE_CRITICAL_CONTAINER_NAMES[$PROFILE_NAME]:-}"
|
||||
read -r -a DELAYED_CONTAINERS <<< "${PROFILE_DELAYED_CONTAINERS[$PROFILE_NAME]:-}"
|
||||
read -r -a EXCLUDE_DIRS <<< "${PROFILE_EXCLUDE_DIRS[$PROFILE_NAME]:-}"
|
||||
read -r -a REMOTE_RESTART_CONTAINERS <<< "${PROFILE_REMOTE_RESTART_CONTAINERS[$PROFILE_NAME]:-}"
|
||||
|
||||
# Local containers use same names as remote (mirrored naming scheme)
|
||||
LOCAL_CRITICAL_CONTAINER_NAMES=("${CRITICAL_CONTAINER_NAMES[@]}")
|
||||
|
||||
log "$ICON_GEAR Config: profile=${PROFILE_NAME} bw-limit=${BW_LIMIT}KB/s retry=${RETRY_COUNT} sleep=${SLEEP}s container-delay=${CONTAINER_DELAY}s"
|
||||
log "$ICON_GEAR Containers: critical=${CRITICAL_CONTAINER_NAMES[*]:-none} delayed=${DELAYED_CONTAINERS[*]:-none} remote-restart=${REMOTE_RESTART_CONTAINERS[*]:-none}"
|
||||
|
||||
[[ "$SHOW_STATUS" == true ]] && show_status && exit 0
|
||||
[[ "$DRY_RUN" == true ]] && warn "DRY RUN — no changes will be made"
|
||||
|
||||
# ==============================================================================================
|
||||
# ━━━ Pre-flight Checks ━━━
|
||||
# ==============================================================================================
|
||||
echo ""
|
||||
echo "━━━ $ICON_SHIELD Pre-flight ━━━"
|
||||
|
||||
# Disk temp — before touching remote or moving data
|
||||
# Exit 1 = skip this profile | Exit 2 = abort all remaining profiles
|
||||
check_local_disk_temps
|
||||
TEMP_RESULT=$?
|
||||
if [[ "$TEMP_RESULT" -eq 2 ]]; then
|
||||
error "Drive temps CRITICAL — aborting all remaining syncs"
|
||||
exit 2
|
||||
elif [[ "$TEMP_RESULT" -eq 1 ]]; then
|
||||
warn "Drive temps high — skipping profile [$PROFILE_NAME]"
|
||||
exit 1
|
||||
else
|
||||
log "Drive temps OK — $TEMP_CHECK_RESULT"
|
||||
fi
|
||||
|
||||
# Version parity — refuse if servers on incompatible unRAID versions
|
||||
check_unraid_version_parity || exit 1
|
||||
|
||||
check_connectivity
|
||||
check_remote_rootfs
|
||||
check_remote_share "$DIRECTORY"
|
||||
check_remote_disks "$DIRECTORY"
|
||||
|
||||
# Remote Docker daemon — check before attempting container operations
|
||||
if [[ ${#CRITICAL_CONTAINER_NAMES[@]} -gt 0 ]] || [[ ${#REMOTE_RESTART_CONTAINERS[@]} -gt 0 ]]; then
|
||||
check_remote_docker_daemon || {
|
||||
warn "Remote Docker daemon unresponsive — skipping container operations"
|
||||
warn "Proceeding with rsync only — containers will not be stopped or restarted"
|
||||
CRITICAL_CONTAINER_NAMES=()
|
||||
LOCAL_CRITICAL_CONTAINER_NAMES=()
|
||||
REMOTE_RESTART_CONTAINERS=()
|
||||
}
|
||||
fi
|
||||
|
||||
# ==============================================================================================
|
||||
# ━━━ Stop Containers ━━━
|
||||
# ==============================================================================================
|
||||
if [[ ${#CRITICAL_CONTAINER_NAMES[@]} -gt 0 || ${#LOCAL_CRITICAL_CONTAINER_NAMES[@]} -gt 0 ]]; then
|
||||
echo ""
|
||||
echo "━━━ $ICON_STOP $ICON_CONTAINERS Stop Containers ━━━"
|
||||
if [[ "$DRY_RUN" == true ]]; then
|
||||
for c in "${LOCAL_CRITICAL_CONTAINER_NAMES[@]:-}"; do
|
||||
[[ -n "$c" ]] && warn "DRY RUN — would stop local: $c"
|
||||
done
|
||||
for c in "${CRITICAL_CONTAINER_NAMES[@]:-}"; do
|
||||
[[ -n "$c" ]] && warn "DRY RUN — would stop remote: $c"
|
||||
done
|
||||
else
|
||||
# Local first — flush local databases before pushing
|
||||
stop_local_containers
|
||||
# Remote next — prevent writes while receiving
|
||||
stop_containers
|
||||
fi
|
||||
fi
|
||||
|
||||
# ==============================================================================================
|
||||
# ━━━ Transfer ━━━
|
||||
# ==============================================================================================
|
||||
echo ""
|
||||
echo "━━━ $ICON_SYNC Transfer ━━━"
|
||||
echo "$ICON_RUN Source: $DIRECTORY"
|
||||
echo "$ICON_NET Remote: $REMOTE_SERVER:$DIRECTORY"
|
||||
echo "$ICON_GEAR Profile: $PROFILE_NAME"
|
||||
echo "$ICON_HOST Identity: $MY_ID → $REMOTE_ID"
|
||||
echo ""
|
||||
|
||||
get_rsync_opts
|
||||
|
||||
# Append profile excludes
|
||||
for ex in "${EXCLUDE_DIRS[@]:-}"; do
|
||||
[[ -n "$ex" ]] && RSYNC_OPTS+=(--exclude="$ex")
|
||||
done
|
||||
|
||||
# Add --stats to capture bytes transferred for bandwidth logging
|
||||
RSYNC_OPTS+=(--stats)
|
||||
|
||||
[[ "$DRY_RUN" == true ]] && RSYNC_OPTS+=("--dry-run")
|
||||
|
||||
START=$(date +%s)
|
||||
RSYNC_SUCCESS=false
|
||||
BYTES_TRANSFERRED=0
|
||||
ATTEMPT=0
|
||||
|
||||
for (( ATTEMPT=1; ATTEMPT<=RETRY_COUNT; ATTEMPT++ )); do
|
||||
log "$ICON_RETRY Attempt $ATTEMPT of $RETRY_COUNT..."
|
||||
echo "$ICON_SYNC Rsync running — this may take a while..."
|
||||
|
||||
RSYNC_OUTPUT=$(rsync "${RSYNC_OPTS[@]}" \
|
||||
-e "ssh -i $SSH_KEY -T -o Compression=no -o IPQoS=throughput" \
|
||||
"$DIRECTORY" "root@${REMOTE_SERVER}:$(dirname "$DIRECTORY")/" 2>&1)
|
||||
|
||||
RSYNC_EXIT=$?
|
||||
|
||||
if [[ "$RSYNC_EXIT" -eq 0 ]]; then
|
||||
# Parse bytes transferred from --stats output
|
||||
BYTES_TRANSFERRED=$(echo "$RSYNC_OUTPUT" | \
|
||||
awk '/Total transferred file size:/{gsub(/,/,"",$NF); gsub(/[^0-9]/,"",$NF); print $NF+0}')
|
||||
BYTES_TRANSFERRED="${BYTES_TRANSFERRED:-0}"
|
||||
|
||||
log "$ICON_DONE Rsync complete — $BYTES_TRANSFERRED bytes transferred"
|
||||
RSYNC_SUCCESS=true
|
||||
break
|
||||
else
|
||||
warn "$ICON_RETRY Rsync failed (attempt $ATTEMPT/$RETRY_COUNT)"
|
||||
log "Exit code: $RSYNC_EXIT"
|
||||
if [[ "$ATTEMPT" -lt "$RETRY_COUNT" ]]; then
|
||||
log "Retrying in ${SLEEP}s..."
|
||||
sleep "$SLEEP"
|
||||
fi
|
||||
fi
|
||||
done
|
||||
|
||||
# ==============================================================================================
|
||||
# ━━━ Start Containers ━━━
|
||||
# ==============================================================================================
|
||||
if [[ ${#CRITICAL_CONTAINER_NAMES[@]} -gt 0 || ${#LOCAL_CRITICAL_CONTAINER_NAMES[@]} -gt 0 ]]; then
|
||||
echo ""
|
||||
echo "━━━ $ICON_START $ICON_CONTAINERS Start Containers ━━━"
|
||||
if [[ "$DRY_RUN" == true ]]; then
|
||||
for c in "${CRITICAL_CONTAINER_NAMES[@]:-}"; do
|
||||
[[ -n "$c" ]] && warn "DRY RUN — would start remote: $c"
|
||||
done
|
||||
for c in "${LOCAL_CRITICAL_CONTAINER_NAMES[@]:-}"; do
|
||||
[[ -n "$c" ]] && warn "DRY RUN — would start local: $c"
|
||||
done
|
||||
else
|
||||
# Remote first — can be coming up while local restarts
|
||||
start_containers
|
||||
# Local next
|
||||
start_local_containers
|
||||
fi
|
||||
fi
|
||||
|
||||
# ==============================================================================================
|
||||
# ━━━ Remote Restart (dirty sync profiles) ━━━
|
||||
# ==============================================================================================
|
||||
# For dirty sync profiles (critical-fallback, emby-fallback) — restart containers on remote
|
||||
# that were running before sync so they pick up config changes from the dirty sync window.
|
||||
# Was running → restart. Was stopped → leave stopped.
|
||||
if [[ ${#REMOTE_RESTART_CONTAINERS[@]} -gt 0 ]]; then
|
||||
echo ""
|
||||
echo "━━━ $ICON_START $ICON_CONTAINERS Remote Restart (post dirty sync) ━━━"
|
||||
log "Restarting configured containers on $REMOTE_SERVER_NAME..."
|
||||
|
||||
for container in "${REMOTE_RESTART_CONTAINERS[@]}"; do
|
||||
[[ -z "$container" ]] && continue
|
||||
|
||||
# Check if container was running before sync (still tracked via RUNNING_CONTAINERS)
|
||||
WAS_RUNNING=false
|
||||
for prev in "${RUNNING_CONTAINERS[@]:-}"; do
|
||||
[[ "$prev" == "$container" ]] && WAS_RUNNING=true && break
|
||||
done
|
||||
|
||||
if [[ "$WAS_RUNNING" == false ]]; then
|
||||
# Not in stop list — check current remote state
|
||||
REMOTE_STATUS=$(timeout 15 ssh -i "$SSH_KEY" \
|
||||
-o ConnectTimeout=10 root@"$REMOTE_SERVER" \
|
||||
"docker inspect -f '{{.State.Running}}' $container 2>/dev/null" 2>/dev/null)
|
||||
[[ "$REMOTE_STATUS" != "true" ]] && \
|
||||
log "$container not running on $REMOTE_SERVER_NAME — skipping remote restart" && \
|
||||
continue
|
||||
fi
|
||||
|
||||
if [[ "$DRY_RUN" == true ]]; then
|
||||
warn "DRY RUN — would restart $container on $REMOTE_SERVER_NAME"
|
||||
continue
|
||||
fi
|
||||
|
||||
timeout 15 ssh -i "$SSH_KEY" -o ConnectTimeout=10 root@"$REMOTE_SERVER" \
|
||||
"docker restart $container" >/dev/null 2>&1 && \
|
||||
log "$ICON_STARTED $container restarted on $REMOTE_SERVER_NAME ✅" || \
|
||||
warn "Failed to restart $container on $REMOTE_SERVER_NAME"
|
||||
done
|
||||
fi
|
||||
|
||||
END=$(date +%s)
|
||||
DURATION=$(( END - START ))
|
||||
|
||||
# ==============================================================================================
|
||||
# ━━━ Bandwidth Logging ━━━
|
||||
# ==============================================================================================
|
||||
# Logs to bandwidth_monitor.sh — new format includes bytes transferred and warn flag.
|
||||
# Only logs on actual runs (not dry-run) and only when bandwidth_monitor.sh exists.
|
||||
BANDWIDTH_MONITOR="$SCRIPT_DIR/../Monitors/bandwidth_monitor.sh"
|
||||
|
||||
if [[ "$DRY_RUN" == false ]] && [[ -f "$BANDWIDTH_MONITOR" ]]; then
|
||||
STATUS="success"
|
||||
[[ "$RSYNC_SUCCESS" == false ]] && STATUS="failed"
|
||||
bash "$BANDWIDTH_MONITOR" --log-transfer \
|
||||
"$PROFILE_NAME" "$DURATION" "$STATUS" "$BYTES_TRANSFERRED"
|
||||
log "$ICON_BANDWIDTH Transfer logged to bandwidth monitor ($BYTES_TRANSFERRED bytes)"
|
||||
fi
|
||||
|
||||
# ==============================================================================================
|
||||
# ━━━ Summary ━━━
|
||||
# ==============================================================================================
|
||||
echo ""
|
||||
echo "━━━━━ $ICON_SUMMARY RSYNC SUMMARY ━━━━━"
|
||||
echo "$ICON_HOST Identity: $MY_ID ($LOCAL_SERVER_NAME)"
|
||||
echo "$ICON_RUN Directory: $DIRECTORY"
|
||||
echo "$ICON_GEAR Profile: $PROFILE_NAME"
|
||||
echo "$ICON_TIME Duration: $(format_duration $DURATION)"
|
||||
[[ "$BYTES_TRANSFERRED" -gt 0 ]] && \
|
||||
echo "$ICON_BANDWIDTH Transferred: $(awk "BEGIN {printf \"%.2fGB\", $BYTES_TRANSFERRED / 1073741824}")"
|
||||
|
||||
if [[ "$DRY_RUN" == true ]]; then
|
||||
warn "DRY RUN — no changes made"
|
||||
elif [[ "$RSYNC_SUCCESS" == true ]]; then
|
||||
echo "$ICON_DONE Status: $ICON_SUCCESS DONE"
|
||||
else
|
||||
echo "$ICON_ERROR Status: FAILED after $RETRY_COUNT attempts"
|
||||
notify "Rsync FAILED — $DIRECTORY ($PROFILE_NAME) after $RETRY_COUNT attempts on $(hostname)" \
|
||||
"Rsync" "warning"
|
||||
fi
|
||||
echo "━━━━━━━━━━━━━━━━━━━━━━━"
|
||||
|
||||
# Flush tee and preserve log for UI
|
||||
exec 1>&-; wait
|
||||
cp "$VV_LIVE_LOG" "$VV_LAST_LOG" 2>/dev/null
|
||||
rm -f "$VV_LIVE_LOG"
|
||||
|
||||
[[ "$RSYNC_SUCCESS" == false ]] && [[ "$DRY_RUN" == false ]] && exit 1
|
||||
exit 0
|
||||
+461
@@ -0,0 +1,461 @@
|
||||
#!/bin/bash
|
||||
# ==============================================================================================
|
||||
# ============================= Rsync Core Script ==============================================
|
||||
# ==============================================================================================
|
||||
#
|
||||
# PURPOSE
|
||||
# ─────────────────────────────────────────────────────────────────────────────
|
||||
# Core rsync engine for the two-server ecosystem. Called per share or per
|
||||
# appdata profile by orchestrators (daily_sync_maintenance, weekly_sync_maintenance,
|
||||
# critical_sync_maintenance) and directly for manual or scheduled dirty syncs.
|
||||
#
|
||||
# Profile is inferred from the directory basename (lowercased). Override with
|
||||
# --profile=name for explicit selection. If no profile matches, global defaults
|
||||
# from master.conf apply and no containers are stopped.
|
||||
#
|
||||
# After each sync, logs transfer data to bandwidth_monitor.sh for the weekly
|
||||
# bandwidth report. Silent on success — only failures produce visible output.
|
||||
#
|
||||
# ==============================================================================================
|
||||
# OPERATIONAL MODEL
|
||||
# ==============================================================================================
|
||||
#
|
||||
# Profiles define per-share behavior:
|
||||
# PROFILE_CRITICAL_CONTAINER_NAMES — containers stopped on both servers before sync
|
||||
# PROFILE_DELAYED_CONTAINERS — containers with a delay before restart after sync
|
||||
# PROFILE_CONTAINER_DELAY — seconds before delayed containers start
|
||||
# PROFILE_RSYNC_OPTS — rsync flags (does not inherit DEFAULT_RSYNC_OPTS)
|
||||
# PROFILE_BW_LIMIT — bandwidth limit in KB/s
|
||||
# PROFILE_RETRY_COUNT — retry attempts on failure
|
||||
# PROFILE_SLEEP — seconds between retry attempts
|
||||
# PROFILE_EXCLUDE_DIRS — paths excluded from transfer
|
||||
# PROFILE_REMOTE_RESTART_CONTAINERS — containers restarted on remote after dirty sync
|
||||
# Was running → restart. Was stopped → leave stopped.
|
||||
#
|
||||
# Two-tier rsync enable/disable:
|
||||
# Tier 1: RSYNC_ENABLED=false → all rsync stops immediately (checked by this script)
|
||||
# Tier 2: per-orchestrator flag (DAILY_RSYNC_ENABLED etc.) → checked by caller
|
||||
#
|
||||
# Bandwidth logging: after each sync, logs profile/duration/status/bytes to
|
||||
# bandwidth_monitor.sh --log-transfer. Bytes captured from rsync --stats via awk
|
||||
# using version-stable field names.
|
||||
#
|
||||
# ==============================================================================================
|
||||
# OPERATIONAL SAFEGUARDS
|
||||
# ==============================================================================================
|
||||
#
|
||||
# Global Rsync Gate
|
||||
# check_rsync_enabled() — RSYNC_ENABLED=false exits cleanly before any operation.
|
||||
#
|
||||
# Partnership Blocklist
|
||||
# Refuses to sync if REMOTE_SERVER_NAME appears in the partnership blocklist.
|
||||
# Written at offboard — prevents stale access after a partnership ends.
|
||||
#
|
||||
# Version Parity
|
||||
# check_unraid_version_parity — refuses sync if servers on incompatible unRAID versions.
|
||||
#
|
||||
# Remote Health Pre-flights
|
||||
# check_connectivity() — Tailscale IP reachable before any SSH
|
||||
# check_remote_rootfs() — aborts if remote rootfs exceeds ROOTFS_WARN_PCT
|
||||
# check_remote_share() — aborts if target directory missing or empty on remote
|
||||
# check_remote_disks() — verifies all backing disks online on remote
|
||||
#
|
||||
# Drive Temperature Check
|
||||
# check_local_disk_temps() — runs before any transfer. Exit 1 = skip this profile,
|
||||
# exit 2 = abort all remaining syncs (CRITICAL temperature).
|
||||
#
|
||||
# Remote Docker Daemon Check
|
||||
# check_remote_docker_daemon — verified before any container stop/start operations.
|
||||
# If daemon unresponsive: container operations skipped, rsync proceeds without stopping.
|
||||
#
|
||||
# Per-Profile Concurrency Lock
|
||||
# acquire_rsync_lock() — per-profile lock prevents parallel runs of the same profile.
|
||||
# Global concurrent limit prevents too many simultaneous rsync processes.
|
||||
#
|
||||
# Notification Validated
|
||||
# platform_require_cmd confirms the notify script is present before use.
|
||||
#
|
||||
# ==============================================================================================
|
||||
# CONFIGURATION
|
||||
# ==============================================================================================
|
||||
#
|
||||
# master.conf
|
||||
#
|
||||
# RSYNC_ENABLED
|
||||
# Global on/off toggle for all rsync operations. (default: true)
|
||||
#
|
||||
# DEFAULT_RSYNC_OPTS
|
||||
# Base rsync flags for unproiled shares. Note: --delete is intentionally absent —
|
||||
# media shares spread files only, arr cleanup scripts own deletions. Profile-specific
|
||||
# opts set --delete explicitly where needed.
|
||||
#
|
||||
# BW_LIMIT
|
||||
# Default bandwidth cap in KB/s when no PROFILE_BW_LIMIT is set. (default: 0 = unlimited)
|
||||
#
|
||||
# RETRY_COUNT
|
||||
# Default retry attempts on rsync failure. (default: 3)
|
||||
#
|
||||
# SLEEP
|
||||
# Default seconds between retry attempts. (default: 60)
|
||||
#
|
||||
# ROOTFS_WARN_PCT
|
||||
# Abort threshold for remote rootfs percentage full. (default: 75)
|
||||
#
|
||||
# PROFILES["profile_KEY"]
|
||||
# Profile definitions — one entry per PROFILE_* key per profile name.
|
||||
# See OPERATIONAL MODEL above for all supported keys.
|
||||
#
|
||||
# BANDWIDTH_LOG / BANDWIDTH_WARN_GB
|
||||
# Shared with bandwidth_monitor.sh — set once, used by both.
|
||||
#
|
||||
# ==============================================================================================
|
||||
# RUNTIME MODES
|
||||
# ==============================================================================================
|
||||
#
|
||||
# rsync.sh /path/to/share
|
||||
# Sync the given path to the remote. Profile inferred from directory basename.
|
||||
#
|
||||
# rsync.sh /path/to/share --profile=name
|
||||
# Sync with explicit profile override — bypasses basename inference.
|
||||
#
|
||||
# rsync.sh /path/to/share --dry-run
|
||||
# Run all pre-flight checks and show what rsync would transfer. No transfer,
|
||||
# no container stops.
|
||||
#
|
||||
# rsync.sh /path/to/share --status
|
||||
# Show resolved profile, remote identity, and configuration. Then exit.
|
||||
#
|
||||
# rsync.sh /path/to/share --log
|
||||
# Verbose output throughout — every decision logged.
|
||||
#
|
||||
# ==============================================================================================
|
||||
|
||||
SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
|
||||
|
||||
source "$SCRIPT_DIR/../load_config.sh"
|
||||
|
||||
# ── Separate positional directory arg from flags ───────────────────────────────────────────────
|
||||
DIRECTORY=""
|
||||
PROFILE_OVERRIDE=""
|
||||
RAW_ARGS=()
|
||||
|
||||
for ARG in "$@"; do
|
||||
case "$ARG" in
|
||||
--profile=*) PROFILE_OVERRIDE="${ARG#--profile=}" ;;
|
||||
--*|*=*) RAW_ARGS+=("$ARG") ;;
|
||||
*) [[ -z "$DIRECTORY" ]] && DIRECTORY="$ARG" || RAW_ARGS+=("$ARG") ;;
|
||||
esac
|
||||
done
|
||||
|
||||
parse_args "${RAW_ARGS[@]}"
|
||||
|
||||
[[ -z "$DIRECTORY" ]] && {
|
||||
error "No directory specified"
|
||||
error "Usage: rsync.sh <dir> [--dry-run] [--log] [--profile=name]"
|
||||
exit 1
|
||||
}
|
||||
|
||||
# ==============================================================================================
|
||||
# ━━━ Setup ━━━
|
||||
# ==============================================================================================
|
||||
if [[ "$EUID" -ne 0 ]]; then
|
||||
error "Must be run as root"
|
||||
exit 1
|
||||
fi
|
||||
|
||||
acquire_lock
|
||||
|
||||
if ! command -v docker &>/dev/null; then
|
||||
error "Docker command not found"
|
||||
exit 1
|
||||
fi
|
||||
|
||||
platform_require_cmd \
|
||||
"/usr/local/emhttp/plugins/dynamix/scripts/notify" \
|
||||
"" "" \
|
||||
"unRAID notify script" || warn "unRAID notify script not found — native notifications disabled"
|
||||
|
||||
detect_hosts
|
||||
|
||||
# Tier 1 global gate — Tier 2 (per-orchestrator) checked by caller
|
||||
if ! check_rsync_enabled; then
|
||||
warn "RSYNC_ENABLED=false — exiting cleanly"
|
||||
exit 0
|
||||
fi
|
||||
|
||||
# Blocklist gate — refuse to sync with a partner blocked after offboard
|
||||
BLOCKLIST_FILE="${PARTNERSHIP_BLOCKLIST_FILE:-${STATE_DIR:-/boot/config}/partnership_blocklist.db}"
|
||||
if [[ -f "$BLOCKLIST_FILE" ]] && grep -q "^${REMOTE_SERVER_NAME}|" "$BLOCKLIST_FILE" 2>/dev/null; then
|
||||
error "Rsync blocked — $REMOTE_SERVER_NAME is on the partnership blocklist"
|
||||
error "Re-onboard the partnership to restore access: partnership_manager.sh --onboard"
|
||||
exit 1
|
||||
fi
|
||||
|
||||
resolve_remote_ip
|
||||
|
||||
# ── Profile inference ─────────────────────────────────────────────────────────────────────────
|
||||
if [[ -n "$PROFILE_OVERRIDE" ]]; then
|
||||
PROFILE_NAME="$PROFILE_OVERRIDE"
|
||||
log "Profile override: $PROFILE_NAME"
|
||||
else
|
||||
PROFILE_NAME=$(basename "$DIRECTORY" | tr '[:upper:]' '[:lower:]')
|
||||
log "Profile inferred: $PROFILE_NAME"
|
||||
fi
|
||||
|
||||
# Acquire per-profile lock and check global concurrent limit
|
||||
acquire_rsync_lock "$PROFILE_NAME"
|
||||
|
||||
# Tee all output to a live log file for the Varaverk UI
|
||||
VV_LIVE_LOG="/tmp/unraid_locks/rsync_${PROFILE_NAME}.log"
|
||||
VV_LAST_LOG="/tmp/unraid_locks/rsync_${PROFILE_NAME}.last.log"
|
||||
: > "$VV_LIVE_LOG"
|
||||
exec 1> >(tee -a "$VV_LIVE_LOG") 2>&1
|
||||
|
||||
# ── Load profile settings ─────────────────────────────────────────────────────────────────────
|
||||
BW_LIMIT=${PROFILE_BW_LIMIT[$PROFILE_NAME]:-$BW_LIMIT}
|
||||
RETRY_COUNT=${PROFILE_RETRY_COUNT[$PROFILE_NAME]:-$RETRY_COUNT}
|
||||
SLEEP=${PROFILE_SLEEP[$PROFILE_NAME]:-$SLEEP}
|
||||
CONTAINER_DELAY=${PROFILE_CONTAINER_DELAY[$PROFILE_NAME]:-$CONTAINER_DELAY}
|
||||
|
||||
read -r -a CRITICAL_CONTAINER_NAMES <<< "${PROFILE_CRITICAL_CONTAINER_NAMES[$PROFILE_NAME]:-}"
|
||||
read -r -a DELAYED_CONTAINERS <<< "${PROFILE_DELAYED_CONTAINERS[$PROFILE_NAME]:-}"
|
||||
read -r -a EXCLUDE_DIRS <<< "${PROFILE_EXCLUDE_DIRS[$PROFILE_NAME]:-}"
|
||||
read -r -a REMOTE_RESTART_CONTAINERS <<< "${PROFILE_REMOTE_RESTART_CONTAINERS[$PROFILE_NAME]:-}"
|
||||
|
||||
# Local containers use same names as remote (mirrored naming scheme)
|
||||
LOCAL_CRITICAL_CONTAINER_NAMES=("${CRITICAL_CONTAINER_NAMES[@]}")
|
||||
|
||||
log "$ICON_GEAR Config: profile=${PROFILE_NAME} bw-limit=${BW_LIMIT}KB/s retry=${RETRY_COUNT} sleep=${SLEEP}s container-delay=${CONTAINER_DELAY}s"
|
||||
log "$ICON_GEAR Containers: critical=${CRITICAL_CONTAINER_NAMES[*]:-none} delayed=${DELAYED_CONTAINERS[*]:-none} remote-restart=${REMOTE_RESTART_CONTAINERS[*]:-none}"
|
||||
|
||||
[[ "$SHOW_STATUS" == true ]] && show_status && exit 0
|
||||
[[ "$DRY_RUN" == true ]] && warn "DRY RUN — no changes will be made"
|
||||
|
||||
# ==============================================================================================
|
||||
# ━━━ Pre-flight Checks ━━━
|
||||
# ==============================================================================================
|
||||
echo ""
|
||||
echo "━━━ $ICON_SHIELD Pre-flight ━━━"
|
||||
|
||||
# Disk temp — before touching remote or moving data
|
||||
# Exit 1 = skip this profile | Exit 2 = abort all remaining profiles
|
||||
check_local_disk_temps
|
||||
TEMP_RESULT=$?
|
||||
if [[ "$TEMP_RESULT" -eq 2 ]]; then
|
||||
error "Drive temps CRITICAL — aborting all remaining syncs"
|
||||
exit 2
|
||||
elif [[ "$TEMP_RESULT" -eq 1 ]]; then
|
||||
warn "Drive temps high — skipping profile [$PROFILE_NAME]"
|
||||
exit 1
|
||||
else
|
||||
log "Drive temps OK — $TEMP_CHECK_RESULT"
|
||||
fi
|
||||
|
||||
# Version parity — refuse if servers on incompatible unRAID versions
|
||||
check_unraid_version_parity || exit 1
|
||||
|
||||
check_connectivity
|
||||
check_remote_rootfs
|
||||
check_remote_share "$DIRECTORY"
|
||||
check_remote_disks "$DIRECTORY"
|
||||
|
||||
# Remote Docker daemon — check before attempting container operations
|
||||
if [[ ${#CRITICAL_CONTAINER_NAMES[@]} -gt 0 ]] || [[ ${#REMOTE_RESTART_CONTAINERS[@]} -gt 0 ]]; then
|
||||
check_remote_docker_daemon || {
|
||||
warn "Remote Docker daemon unresponsive — skipping container operations"
|
||||
warn "Proceeding with rsync only — containers will not be stopped or restarted"
|
||||
CRITICAL_CONTAINER_NAMES=()
|
||||
LOCAL_CRITICAL_CONTAINER_NAMES=()
|
||||
REMOTE_RESTART_CONTAINERS=()
|
||||
}
|
||||
fi
|
||||
|
||||
# ==============================================================================================
|
||||
# ━━━ Stop Containers ━━━
|
||||
# ==============================================================================================
|
||||
if [[ ${#CRITICAL_CONTAINER_NAMES[@]} -gt 0 || ${#LOCAL_CRITICAL_CONTAINER_NAMES[@]} -gt 0 ]]; then
|
||||
echo ""
|
||||
echo "━━━ $ICON_STOP $ICON_CONTAINERS Stop Containers ━━━"
|
||||
if [[ "$DRY_RUN" == true ]]; then
|
||||
for c in "${LOCAL_CRITICAL_CONTAINER_NAMES[@]:-}"; do
|
||||
[[ -n "$c" ]] && warn "DRY RUN — would stop local: $c"
|
||||
done
|
||||
for c in "${CRITICAL_CONTAINER_NAMES[@]:-}"; do
|
||||
[[ -n "$c" ]] && warn "DRY RUN — would stop remote: $c"
|
||||
done
|
||||
else
|
||||
# Local first — flush local databases before pushing
|
||||
stop_local_containers
|
||||
# Remote next — prevent writes while receiving
|
||||
stop_containers
|
||||
fi
|
||||
fi
|
||||
|
||||
# ==============================================================================================
|
||||
# ━━━ Transfer ━━━
|
||||
# ==============================================================================================
|
||||
echo ""
|
||||
echo "━━━ $ICON_SYNC Transfer ━━━"
|
||||
echo "$ICON_RUN Source: $DIRECTORY"
|
||||
echo "$ICON_NET Remote: $REMOTE_SERVER:$DIRECTORY"
|
||||
echo "$ICON_GEAR Profile: $PROFILE_NAME"
|
||||
echo "$ICON_HOST Identity: $MY_ID → $REMOTE_ID"
|
||||
echo ""
|
||||
|
||||
get_rsync_opts
|
||||
|
||||
# Append profile excludes
|
||||
for ex in "${EXCLUDE_DIRS[@]:-}"; do
|
||||
[[ -n "$ex" ]] && RSYNC_OPTS+=(--exclude="$ex")
|
||||
done
|
||||
|
||||
# Add --stats to capture bytes transferred for bandwidth logging
|
||||
RSYNC_OPTS+=(--stats)
|
||||
|
||||
[[ "$DRY_RUN" == true ]] && RSYNC_OPTS+=("--dry-run")
|
||||
|
||||
START=$(date +%s)
|
||||
RSYNC_SUCCESS=false
|
||||
BYTES_TRANSFERRED=0
|
||||
ATTEMPT=0
|
||||
|
||||
for (( ATTEMPT=1; ATTEMPT<=RETRY_COUNT; ATTEMPT++ )); do
|
||||
log "$ICON_RETRY Attempt $ATTEMPT of $RETRY_COUNT..."
|
||||
echo "$ICON_SYNC Rsync running — this may take a while..."
|
||||
|
||||
RSYNC_OUTPUT=$(rsync "${RSYNC_OPTS[@]}" \
|
||||
-e "ssh -i $SSH_KEY -T -o Compression=no -o IPQoS=throughput" \
|
||||
"$DIRECTORY" "root@${REMOTE_SERVER}:$(dirname "$DIRECTORY")/" 2>&1)
|
||||
|
||||
RSYNC_EXIT=$?
|
||||
|
||||
if [[ "$RSYNC_EXIT" -eq 0 ]]; then
|
||||
# Parse bytes transferred from --stats output
|
||||
BYTES_TRANSFERRED=$(echo "$RSYNC_OUTPUT" | \
|
||||
awk '/Total transferred file size:/{gsub(/,/,"",$NF); gsub(/[^0-9]/,"",$NF); print $NF+0}')
|
||||
BYTES_TRANSFERRED="${BYTES_TRANSFERRED:-0}"
|
||||
|
||||
log "$ICON_DONE Rsync complete — $BYTES_TRANSFERRED bytes transferred"
|
||||
RSYNC_SUCCESS=true
|
||||
break
|
||||
else
|
||||
warn "$ICON_RETRY Rsync failed (attempt $ATTEMPT/$RETRY_COUNT)"
|
||||
log "Exit code: $RSYNC_EXIT"
|
||||
if [[ "$ATTEMPT" -lt "$RETRY_COUNT" ]]; then
|
||||
log "Retrying in ${SLEEP}s..."
|
||||
sleep "$SLEEP"
|
||||
fi
|
||||
fi
|
||||
done
|
||||
|
||||
# ==============================================================================================
|
||||
# ━━━ Start Containers ━━━
|
||||
# ==============================================================================================
|
||||
if [[ ${#CRITICAL_CONTAINER_NAMES[@]} -gt 0 || ${#LOCAL_CRITICAL_CONTAINER_NAMES[@]} -gt 0 ]]; then
|
||||
echo ""
|
||||
echo "━━━ $ICON_START $ICON_CONTAINERS Start Containers ━━━"
|
||||
if [[ "$DRY_RUN" == true ]]; then
|
||||
for c in "${CRITICAL_CONTAINER_NAMES[@]:-}"; do
|
||||
[[ -n "$c" ]] && warn "DRY RUN — would start remote: $c"
|
||||
done
|
||||
for c in "${LOCAL_CRITICAL_CONTAINER_NAMES[@]:-}"; do
|
||||
[[ -n "$c" ]] && warn "DRY RUN — would start local: $c"
|
||||
done
|
||||
else
|
||||
# Remote first — can be coming up while local restarts
|
||||
start_containers
|
||||
# Local next
|
||||
start_local_containers
|
||||
fi
|
||||
fi
|
||||
|
||||
# ==============================================================================================
|
||||
# ━━━ Remote Restart (dirty sync profiles) ━━━
|
||||
# ==============================================================================================
|
||||
# For dirty sync profiles (critical-fallback, emby-fallback) — restart containers on remote
|
||||
# that were running before sync so they pick up config changes from the dirty sync window.
|
||||
# Was running → restart. Was stopped → leave stopped.
|
||||
if [[ ${#REMOTE_RESTART_CONTAINERS[@]} -gt 0 ]]; then
|
||||
echo ""
|
||||
echo "━━━ $ICON_START $ICON_CONTAINERS Remote Restart (post dirty sync) ━━━"
|
||||
log "Restarting configured containers on $REMOTE_SERVER_NAME..."
|
||||
|
||||
for container in "${REMOTE_RESTART_CONTAINERS[@]}"; do
|
||||
[[ -z "$container" ]] && continue
|
||||
|
||||
# Check if container was running before sync (still tracked via RUNNING_CONTAINERS)
|
||||
WAS_RUNNING=false
|
||||
for prev in "${RUNNING_CONTAINERS[@]:-}"; do
|
||||
[[ "$prev" == "$container" ]] && WAS_RUNNING=true && break
|
||||
done
|
||||
|
||||
if [[ "$WAS_RUNNING" == false ]]; then
|
||||
# Not in stop list — check current remote state
|
||||
REMOTE_STATUS=$(timeout 15 ssh -i "$SSH_KEY" \
|
||||
-o ConnectTimeout=10 root@"$REMOTE_SERVER" \
|
||||
"docker inspect -f '{{.State.Running}}' $container 2>/dev/null" 2>/dev/null)
|
||||
[[ "$REMOTE_STATUS" != "true" ]] && \
|
||||
log "$container not running on $REMOTE_SERVER_NAME — skipping remote restart" && \
|
||||
continue
|
||||
fi
|
||||
|
||||
if [[ "$DRY_RUN" == true ]]; then
|
||||
warn "DRY RUN — would restart $container on $REMOTE_SERVER_NAME"
|
||||
continue
|
||||
fi
|
||||
|
||||
timeout 15 ssh -i "$SSH_KEY" -o ConnectTimeout=10 root@"$REMOTE_SERVER" \
|
||||
"docker restart $container" >/dev/null 2>&1 && \
|
||||
log "$ICON_STARTED $container restarted on $REMOTE_SERVER_NAME ✅" || \
|
||||
warn "Failed to restart $container on $REMOTE_SERVER_NAME"
|
||||
done
|
||||
fi
|
||||
|
||||
END=$(date +%s)
|
||||
DURATION=$(( END - START ))
|
||||
|
||||
# ==============================================================================================
|
||||
# ━━━ Bandwidth Logging ━━━
|
||||
# ==============================================================================================
|
||||
# Logs to bandwidth_monitor.sh — new format includes bytes transferred and warn flag.
|
||||
# Only logs on actual runs (not dry-run) and only when bandwidth_monitor.sh exists.
|
||||
BANDWIDTH_MONITOR="$SCRIPT_DIR/../Monitors/bandwidth_monitor.sh"
|
||||
|
||||
if [[ "$DRY_RUN" == false ]] && [[ -f "$BANDWIDTH_MONITOR" ]]; then
|
||||
STATUS="success"
|
||||
[[ "$RSYNC_SUCCESS" == false ]] && STATUS="failed"
|
||||
bash "$BANDWIDTH_MONITOR" --log-transfer \
|
||||
"$PROFILE_NAME" "$DURATION" "$STATUS" "$BYTES_TRANSFERRED"
|
||||
log "$ICON_BANDWIDTH Transfer logged to bandwidth monitor ($BYTES_TRANSFERRED bytes)"
|
||||
fi
|
||||
|
||||
# ==============================================================================================
|
||||
# ━━━ Summary ━━━
|
||||
# ==============================================================================================
|
||||
echo ""
|
||||
echo "━━━━━ $ICON_SUMMARY RSYNC SUMMARY ━━━━━"
|
||||
echo "$ICON_HOST Identity: $MY_ID ($LOCAL_SERVER_NAME)"
|
||||
echo "$ICON_RUN Directory: $DIRECTORY"
|
||||
echo "$ICON_GEAR Profile: $PROFILE_NAME"
|
||||
echo "$ICON_TIME Duration: $(format_duration $DURATION)"
|
||||
[[ "$BYTES_TRANSFERRED" -gt 0 ]] && \
|
||||
echo "$ICON_BANDWIDTH Transferred: $(awk "BEGIN {printf \"%.2fGB\", $BYTES_TRANSFERRED / 1073741824}")"
|
||||
|
||||
if [[ "$DRY_RUN" == true ]]; then
|
||||
warn "DRY RUN — no changes made"
|
||||
elif [[ "$RSYNC_SUCCESS" == true ]]; then
|
||||
echo "$ICON_DONE Status: $ICON_SUCCESS DONE"
|
||||
else
|
||||
echo "$ICON_ERROR Status: FAILED after $RETRY_COUNT attempts"
|
||||
notify "Rsync FAILED — $DIRECTORY ($PROFILE_NAME) after $RETRY_COUNT attempts on $(hostname)" \
|
||||
"Rsync" "warning"
|
||||
fi
|
||||
echo "━━━━━━━━━━━━━━━━━━━━━━━"
|
||||
|
||||
# Flush tee and preserve log for UI — close both ends of the pipe so tee gets EOF
|
||||
exec 1>&- 2>&-; wait
|
||||
cp "$VV_LIVE_LOG" "$VV_LAST_LOG" 2>/dev/null
|
||||
rm -f "$VV_LIVE_LOG"
|
||||
|
||||
[[ "$RSYNC_SUCCESS" == false ]] && [[ "$DRY_RUN" == false ]] && exit 1
|
||||
exit 0
|
||||
+386
@@ -0,0 +1,386 @@
|
||||
#!/bin/bash
|
||||
# ==============================================================================================
|
||||
# ================================= Docker Update ==============================================
|
||||
# ==============================================================================================
|
||||
#
|
||||
# PURPOSE
|
||||
# ─────────────────────────────────────────────────────────────────────────────
|
||||
# Pulls the latest images for configured containers. Two modes: normal (daily)
|
||||
# and remainder (weekly).
|
||||
#
|
||||
# Normal mode is called by daily_sync_maintenance.sh before docker_daily_restart.sh.
|
||||
# Containers stay running during the pull — no extra downtime beyond what the
|
||||
# nightly restart already causes.
|
||||
#
|
||||
# Remainder mode is called by weekly_sync_maintenance.sh as the final update step.
|
||||
# It catches everything that normal mode and the weekly sync window did not already
|
||||
# update — derived automatically from docker ps, nothing to configure.
|
||||
#
|
||||
# ==============================================================================================
|
||||
# OPERATIONAL MODEL
|
||||
# ==============================================================================================
|
||||
#
|
||||
# Normal mode (daily):
|
||||
# Targets DAILY_RESTART_CONTAINERS — same list used by docker_daily_restart.sh.
|
||||
# Pull → compare old vs new image ID → mark updated or already current.
|
||||
# docker_daily_restart.sh runs after — containers restart onto the fresh image.
|
||||
#
|
||||
# Remainder mode (weekly):
|
||||
# Targets all currently running containers NOT in:
|
||||
# DAILY_RESTART_CONTAINERS — already updated daily
|
||||
# emby + critical-data profiles — updated inline by the weekly sync window
|
||||
# FALLBACK_*_TIER* — owned by the remote server's update cycle
|
||||
# Pull → compare → prune dangling images.
|
||||
#
|
||||
# ==============================================================================================
|
||||
# DESIGN PRINCIPLES
|
||||
# ==============================================================================================
|
||||
#
|
||||
# Single List
|
||||
# Normal mode reuses DAILY_RESTART_CONTAINERS rather than maintaining a
|
||||
# separate update list. Adding or removing a container from the restart list
|
||||
# automatically updates the image pull list — one change, both places.
|
||||
#
|
||||
# Version Ownership
|
||||
# Fallback containers are excluded from remainder mode. This server only runs
|
||||
# them during a fallback. The remote server owns their version — if remainder
|
||||
# updates them independently and a handback occurs, the remote's older image
|
||||
# may not handle data written by the newer version.
|
||||
#
|
||||
# State Respect
|
||||
# Stopped containers are never targeted. Pulling while stopped adds no value
|
||||
# and a stopped container was likely halted intentionally.
|
||||
#
|
||||
# ==============================================================================================
|
||||
# OPERATIONAL SAFEGUARDS
|
||||
# ==============================================================================================
|
||||
#
|
||||
# Lock Acquisition
|
||||
# Prevents concurrent execution via acquire_lock(). Safe to call from
|
||||
# maintenance scripts without risk of overlap.
|
||||
#
|
||||
# Host Detection
|
||||
# detect_hosts() identifies which server is running the script and aliases
|
||||
# HOST*_DAILY_RESTART_CONTAINERS to the correct host's values.
|
||||
#
|
||||
# Root Enforcement
|
||||
# Docker operations require root privileges.
|
||||
#
|
||||
# DAILY_CONTAINER_UPDATES Toggle
|
||||
# Normal mode exits cleanly when disabled. docker_daily_restart.sh still runs
|
||||
# regardless — update and restart are independent operations.
|
||||
#
|
||||
# Fallback Exclusion
|
||||
# Remainder mode excludes containers owned by the remote server's update cycle
|
||||
# to prevent version divergence across the fallback boundary.
|
||||
#
|
||||
# Running-Only Filter
|
||||
# Stopped containers excluded from remainder mode — intentionally down.
|
||||
#
|
||||
# Image ID Comparison
|
||||
# Containers not restarted unless their image actually changed. Pulls that
|
||||
# result in "already up to date" produce no restart.
|
||||
#
|
||||
# ==============================================================================================
|
||||
# CONFIGURATION
|
||||
# ==============================================================================================
|
||||
#
|
||||
# master.conf
|
||||
#
|
||||
# DAILY_CONTAINER_UPDATES
|
||||
# Enable or disable normal mode. docker_daily_restart.sh runs regardless.
|
||||
# (default: true)
|
||||
#
|
||||
# PROFILE_CRITICAL_CONTAINER_NAMES[emby|critical-data]
|
||||
# Container names for emby and critical-data profiles — excluded from
|
||||
# remainder mode (already updated by the weekly sync window)
|
||||
#
|
||||
# host*.conf
|
||||
#
|
||||
# HOST*_DAILY_RESTART_CONTAINERS
|
||||
# Containers updated in normal mode. Aliased by detect_hosts() →
|
||||
# DAILY_RESTART_CONTAINERS
|
||||
#
|
||||
# ==============================================================================================
|
||||
# RUNTIME MODES
|
||||
# ==============================================================================================
|
||||
#
|
||||
# docker_update.sh
|
||||
# Normal mode — pull latest images for DAILY_RESTART_CONTAINERS
|
||||
#
|
||||
# docker_update.sh --remainder
|
||||
# Remainder mode — pull all running containers not in managed lists,
|
||||
# restart those that received updates, prune dangling images
|
||||
#
|
||||
# docker_update.sh --dry-run
|
||||
# Preview which containers would be pulled without making changes
|
||||
#
|
||||
# docker_update.sh --status
|
||||
# Show configuration and container list for current mode
|
||||
#
|
||||
# docker_update.sh --log
|
||||
# Verbose per-container pull and comparison output
|
||||
#
|
||||
# ==============================================================================================
|
||||
|
||||
SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
|
||||
|
||||
source "$SCRIPT_DIR/../load_config.sh"
|
||||
|
||||
# Pre-parse --remainder before parse_args (unknown args pass through to PARSED_ARGS)
|
||||
REMAINDER_MODE=false
|
||||
_filtered_args=()
|
||||
for _arg in "$@"; do
|
||||
if [[ "$_arg" == "--remainder" ]]; then
|
||||
REMAINDER_MODE=true
|
||||
else
|
||||
_filtered_args+=("$_arg")
|
||||
fi
|
||||
done
|
||||
unset _arg
|
||||
|
||||
parse_args "${_filtered_args[@]}"
|
||||
unset _filtered_args
|
||||
|
||||
# ==============================================================================================
|
||||
# ━━━ Setup ━━━
|
||||
# ==============================================================================================
|
||||
if [[ "$EUID" -ne 0 ]]; then
|
||||
error "Must be run as root"
|
||||
exit 1
|
||||
fi
|
||||
|
||||
acquire_lock
|
||||
|
||||
if ! command -v docker &>/dev/null; then
|
||||
error "Docker command not found"
|
||||
exit 1
|
||||
fi
|
||||
|
||||
detect_hosts
|
||||
|
||||
# ==============================================================================================
|
||||
# ━━━ Container Discovery ━━━
|
||||
# ==============================================================================================
|
||||
if [[ "$REMAINDER_MODE" == true ]]; then
|
||||
declare -A _exclude=()
|
||||
|
||||
# Daily containers — updated by docker_update.sh normal mode
|
||||
for _c in "${DAILY_RESTART_CONTAINERS[@]}"; do
|
||||
[[ -n "$_c" ]] && _exclude["$_c"]=1
|
||||
done
|
||||
|
||||
# Weekly sync-window containers (emby + critical-data) — updated inline by weekly_sync_maintenance.sh
|
||||
_weekly_str="${PROFILE_CRITICAL_CONTAINER_NAMES[emby]:-} ${PROFILE_CRITICAL_CONTAINER_NAMES[critical-data]:-}"
|
||||
read -r -a _weekly_arr <<< "$_weekly_str"
|
||||
for _c in "${_weekly_arr[@]}"; do
|
||||
[[ -n "$_c" ]] && _exclude["$_c"]=1
|
||||
done
|
||||
unset _weekly_str _weekly_arr
|
||||
|
||||
# Fallback coverage containers — owned by the remote server's update cycle.
|
||||
# This server runs them during fallback but should never update them independently.
|
||||
# Updating them here risks version divergence: if remote's writeback after handback
|
||||
# encounters data written by a newer version, it may not handle it correctly.
|
||||
for _tier in 1 2 3 4; do
|
||||
_tier_var="FALLBACK_${MY_ID}_COVERS_${REMOTE_ID}_TIER${_tier}"
|
||||
eval "_tier_arr=(\"\${${_tier_var}[@]:-}\")" 2>/dev/null
|
||||
for _c in "${_tier_arr[@]}"; do
|
||||
[[ -n "$_c" ]] && _exclude["$_c"]=1
|
||||
done
|
||||
done
|
||||
unset _tier _tier_var _tier_arr _c
|
||||
|
||||
mapfile -t _all_running < <(docker ps --format '{{.Names}}' | sort)
|
||||
TARGET_CONTAINERS=()
|
||||
for _c in "${_all_running[@]}"; do
|
||||
[[ -z "${_exclude[$_c]+x}" ]] && TARGET_CONTAINERS+=("$_c")
|
||||
done
|
||||
unset _all_running _exclude _c
|
||||
else
|
||||
if [[ "${DAILY_CONTAINER_UPDATES:-true}" != "true" ]]; then
|
||||
echo "DAILY_CONTAINER_UPDATES=false — skipping container updates"
|
||||
exit 0
|
||||
fi
|
||||
|
||||
if [[ ${#DAILY_RESTART_CONTAINERS[@]} -eq 0 ]]; then
|
||||
warn "DAILY_RESTART_CONTAINERS is empty for $MY_ID — nothing to update"
|
||||
warn "Check HOST${MY_ID#HOST}_DAILY_RESTART_CONTAINERS in host*.conf"
|
||||
exit 0
|
||||
fi
|
||||
|
||||
TARGET_CONTAINERS=("${DAILY_RESTART_CONTAINERS[@]}")
|
||||
fi
|
||||
|
||||
# ==============================================================================================
|
||||
# ━━━ Status ━━━
|
||||
# ==============================================================================================
|
||||
if [[ "$SHOW_STATUS" == true ]]; then
|
||||
echo ""
|
||||
echo "━━━━━ $ICON_SUMMARY STATUS ━━━━━"
|
||||
echo "$ICON_HOST Identity: $MY_ID ($LOCAL_SERVER_NAME)"
|
||||
echo "$ICON_GEAR Mode: $([[ "$REMAINDER_MODE" == true ]] && echo "remainder" || echo "normal (daily)")"
|
||||
if [[ "$REMAINDER_MODE" == true ]]; then
|
||||
echo "$ICON_CONTAINERS Containers: ${#TARGET_CONTAINERS[@]} running (excluding daily, weekly sync, and fallback)"
|
||||
for _c in "${TARGET_CONTAINERS[@]}"; do echo " $_c"; done
|
||||
else
|
||||
echo "$ICON_CONTAINERS Containers: ${TARGET_CONTAINERS[*]}"
|
||||
echo "$ICON_GEAR Enabled: ${DAILY_CONTAINER_UPDATES:-true}"
|
||||
fi
|
||||
echo "$ICON_GEAR Dry Run: $DRY_RUN"
|
||||
echo "━━━━━━━━━━━━━━━━━━━━━━━"
|
||||
exit 0
|
||||
fi
|
||||
|
||||
[[ "$DRY_RUN" == true ]] && warn "DRY RUN — no images will be pulled"
|
||||
|
||||
if [[ ${#TARGET_CONTAINERS[@]} -eq 0 ]]; then
|
||||
echo "No containers to update"
|
||||
exit 0
|
||||
fi
|
||||
|
||||
# ==============================================================================================
|
||||
# ━━━ Pull Updates ━━━
|
||||
# ==============================================================================================
|
||||
echo ""
|
||||
if [[ "$REMAINDER_MODE" == true ]]; then
|
||||
echo "━━━ $ICON_CONTAINERS Docker Update (remainder) — $(date '+%Y-%m-%d %H:%M:%S') ━━━"
|
||||
echo "$ICON_CONTAINERS Updating ${#TARGET_CONTAINERS[@]} container(s) (not in daily or weekly sync)"
|
||||
log "$ICON_CONTAINERS Remainder targets: ${TARGET_CONTAINERS[*]}"
|
||||
else
|
||||
echo "━━━ $ICON_CONTAINERS Docker Update — $(date '+%Y-%m-%d %H:%M:%S') ━━━"
|
||||
log "$ICON_CONTAINERS Containers: ${TARGET_CONTAINERS[*]}"
|
||||
fi
|
||||
echo ""
|
||||
|
||||
START=$(date +%s)
|
||||
UPDATED=()
|
||||
UP_TO_DATE=()
|
||||
FAILED=()
|
||||
SKIPPED=()
|
||||
|
||||
for container in "${TARGET_CONTAINERS[@]}"; do
|
||||
[[ -z "$container" ]] && continue
|
||||
log "━━━ $ICON_CONTAINERS $container ━━━"
|
||||
|
||||
if ! docker inspect "$container" &>/dev/null; then
|
||||
warn "$container — not found, skipping"
|
||||
SKIPPED+=("$container")
|
||||
continue
|
||||
fi
|
||||
|
||||
IMAGE=$(docker inspect --format='{{.Config.Image}}' "$container" 2>/dev/null)
|
||||
if [[ -z "$IMAGE" ]]; then
|
||||
warn "$container — could not determine image, skipping"
|
||||
SKIPPED+=("$container")
|
||||
continue
|
||||
fi
|
||||
|
||||
log "$container — image: $IMAGE"
|
||||
|
||||
if [[ "$DRY_RUN" == true ]]; then
|
||||
warn "DRY RUN — would pull: $IMAGE"
|
||||
UPDATED+=("$container")
|
||||
continue
|
||||
fi
|
||||
|
||||
# Capture image ID before pull to detect whether an update landed
|
||||
OLD_ID=$(docker image inspect "$IMAGE" --format='{{.Id}}' 2>/dev/null || echo "")
|
||||
|
||||
log "$ICON_SYNC Pulling $IMAGE..."
|
||||
if [[ "$ENABLE_LOGGING" == "true" ]]; then
|
||||
docker pull "$IMAGE" 2>&1 | grep -E "^(Status:|Digest:|Error|error)" | sed 's/^/ /'
|
||||
_pull_rc=${PIPESTATUS[0]}
|
||||
else
|
||||
docker pull "$IMAGE" >/dev/null 2>&1
|
||||
_pull_rc=$?
|
||||
fi
|
||||
NEW_ID=$(docker image inspect "$IMAGE" --format='{{.Id}}' 2>/dev/null || echo "")
|
||||
|
||||
if [[ $_pull_rc -eq 0 ]]; then
|
||||
if [[ -n "$OLD_ID" ]] && [[ "$OLD_ID" != "$NEW_ID" ]]; then
|
||||
log "$ICON_DONE $container — updated ✅ (${OLD_ID:7:12} → ${NEW_ID:7:12})"
|
||||
UPDATED+=("$container")
|
||||
else
|
||||
log "$container — already up to date (${NEW_ID:7:12})"
|
||||
UP_TO_DATE+=("$container")
|
||||
fi
|
||||
else
|
||||
warn "$container — pull failed ($IMAGE)"
|
||||
FAILED+=("$container")
|
||||
fi
|
||||
|
||||
done
|
||||
|
||||
# ── Recreate containers that received a new image ────────────────────────────
|
||||
# docker restart uses the image ID baked in at creation time — it never picks
|
||||
# up the new digest. rebuild_container reads the stored XML template, stops the
|
||||
# old container, recreates it (new image, same config), then prunes the old image.
|
||||
REBUILT=()
|
||||
REBUILD_FAILED=()
|
||||
if [[ ${#UPDATED[@]} -gt 0 ]]; then
|
||||
for container in "${UPDATED[@]}"; do
|
||||
[[ -z "$container" ]] && continue
|
||||
if [[ "$DRY_RUN" == true ]]; then
|
||||
warn "DRY RUN — would rebuild $container from template"
|
||||
REBUILT+=("$container")
|
||||
continue
|
||||
fi
|
||||
log "$ICON_SYNC Rebuilding $container from template on new image..."
|
||||
if /usr/local/emhttp/plugins/dynamix.docker.manager/scripts/rebuild_container "$container" >/dev/null 2>&1; then
|
||||
log "$ICON_DONE $container rebuilt ✅"
|
||||
REBUILT+=("$container")
|
||||
else
|
||||
error "Failed to rebuild $container — will be picked up by docker_daily_restart.sh"
|
||||
notify "$container failed to rebuild after image update on $(hostname)" "Docker Update" "warning"
|
||||
REBUILD_FAILED+=("$container")
|
||||
fi
|
||||
done
|
||||
fi
|
||||
|
||||
# ── Prune dangling images ─────────────────────────────────────────────────────
|
||||
# Old images become dangling after a pull lands a new digest. Prune here so
|
||||
# they don't accumulate across daily runs.
|
||||
if [[ "$DRY_RUN" == true ]]; then
|
||||
warn "DRY RUN — would prune dangling images"
|
||||
PRUNED_SUMMARY="(dry run)"
|
||||
else
|
||||
PRUNED_OUTPUT=$(docker image prune -f 2>&1)
|
||||
[[ "$ENABLE_LOGGING" == "true" ]] && echo "$PRUNED_OUTPUT" | sed 's/^/ /'
|
||||
PRUNED_SUMMARY=$(echo "$PRUNED_OUTPUT" | grep -E "^Total reclaimed" || echo "nothing reclaimed")
|
||||
fi
|
||||
|
||||
END=$(date +%s)
|
||||
|
||||
# ==============================================================================================
|
||||
# ━━━ Summary ━━━
|
||||
# ==============================================================================================
|
||||
if [[ "$REMAINDER_MODE" == true ]]; then
|
||||
echo "━━━━━ $ICON_SUMMARY DOCKER UPDATE (REMAINDER) SUMMARY ━━━━━"
|
||||
else
|
||||
echo "━━━━━ $ICON_SUMMARY DOCKER UPDATE SUMMARY ━━━━━"
|
||||
fi
|
||||
echo "$ICON_HOST Identity: $MY_ID ($LOCAL_SERVER_NAME)"
|
||||
echo "$ICON_TIME Duration: $(format_duration $(( END - START )))"
|
||||
if [[ ${#UPDATED[@]} -gt 0 ]]; then
|
||||
echo "$ICON_DONE Updated: ${#UPDATED[@]}"
|
||||
log " ${UPDATED[*]}"
|
||||
fi
|
||||
[[ ${#REBUILT[@]} -gt 0 ]] && echo "$ICON_SYNC Rebuilt: ${#REBUILT[@]}"
|
||||
[[ ${#REBUILD_FAILED[@]} -gt 0 ]] && echo "$ICON_ERROR Rebuild fail:${REBUILD_FAILED[*]}"
|
||||
[[ ${#UP_TO_DATE[@]} -gt 0 ]] && log "$ICON_RUNNING Up to date: ${UP_TO_DATE[*]}"
|
||||
[[ ${#SKIPPED[@]} -gt 0 ]] && log "$ICON_WARN Skipped: ${SKIPPED[*]}"
|
||||
[[ ${#FAILED[@]} -gt 0 ]] && echo "$ICON_ERROR Failed: ${FAILED[*]}"
|
||||
echo "$ICON_SYNC Pruned: ${PRUNED_SUMMARY:-none}"
|
||||
|
||||
if [[ "$DRY_RUN" == true ]]; then
|
||||
warn "DRY RUN — no images pulled"
|
||||
elif [[ ${#FAILED[@]} -eq 0 ]]; then
|
||||
echo "$ICON_DONE Status: done ✅ — ${#UPDATED[@]} updated, ${#UP_TO_DATE[@]} current"
|
||||
else
|
||||
warn "Status: ${#FAILED[@]} pull(s) failed — restart will proceed with existing images"
|
||||
fi
|
||||
echo "━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━"
|
||||
|
||||
# Pull failures are non-fatal — restart proceeds regardless
|
||||
exit 0
|
||||
+392
@@ -0,0 +1,392 @@
|
||||
#!/bin/bash
|
||||
# ==============================================================================================
|
||||
# ================================= Docker Update ==============================================
|
||||
# ==============================================================================================
|
||||
#
|
||||
# PURPOSE
|
||||
# ─────────────────────────────────────────────────────────────────────────────
|
||||
# Pulls the latest images for configured containers. Two modes: normal (daily)
|
||||
# and remainder (weekly).
|
||||
#
|
||||
# Normal mode is called by daily_sync_maintenance.sh before docker_daily_restart.sh.
|
||||
# Containers stay running during the pull — no extra downtime beyond what the
|
||||
# nightly restart already causes.
|
||||
#
|
||||
# Remainder mode is called by weekly_sync_maintenance.sh as the final update step.
|
||||
# It catches everything that normal mode and the weekly sync window did not already
|
||||
# update — derived automatically from docker ps, nothing to configure.
|
||||
#
|
||||
# ==============================================================================================
|
||||
# OPERATIONAL MODEL
|
||||
# ==============================================================================================
|
||||
#
|
||||
# Normal mode (daily):
|
||||
# Targets DAILY_RESTART_CONTAINERS — same list used by docker_daily_restart.sh.
|
||||
# Pull → compare old vs new image ID → mark updated or already current.
|
||||
# docker_daily_restart.sh runs after — containers restart onto the fresh image.
|
||||
#
|
||||
# Remainder mode (weekly):
|
||||
# Targets all currently running containers NOT in:
|
||||
# DAILY_RESTART_CONTAINERS — already updated daily
|
||||
# emby + critical-data profiles — updated inline by the weekly sync window
|
||||
# FALLBACK_*_TIER* — owned by the remote server's update cycle
|
||||
# Pull → compare → prune dangling images.
|
||||
#
|
||||
# ==============================================================================================
|
||||
# DESIGN PRINCIPLES
|
||||
# ==============================================================================================
|
||||
#
|
||||
# Single List
|
||||
# Normal mode reuses DAILY_RESTART_CONTAINERS rather than maintaining a
|
||||
# separate update list. Adding or removing a container from the restart list
|
||||
# automatically updates the image pull list — one change, both places.
|
||||
#
|
||||
# Version Ownership
|
||||
# Fallback containers are excluded from remainder mode. This server only runs
|
||||
# them during a fallback. The remote server owns their version — if remainder
|
||||
# updates them independently and a handback occurs, the remote's older image
|
||||
# may not handle data written by the newer version.
|
||||
#
|
||||
# State Respect
|
||||
# Stopped containers are never targeted. Pulling while stopped adds no value
|
||||
# and a stopped container was likely halted intentionally.
|
||||
#
|
||||
# ==============================================================================================
|
||||
# OPERATIONAL SAFEGUARDS
|
||||
# ==============================================================================================
|
||||
#
|
||||
# Lock Acquisition
|
||||
# Prevents concurrent execution via acquire_lock(). Safe to call from
|
||||
# maintenance scripts without risk of overlap.
|
||||
#
|
||||
# Host Detection
|
||||
# detect_hosts() identifies which server is running the script and aliases
|
||||
# HOST*_DAILY_RESTART_CONTAINERS to the correct host's values.
|
||||
#
|
||||
# Root Enforcement
|
||||
# Docker operations require root privileges.
|
||||
#
|
||||
# DAILY_CONTAINER_UPDATES Toggle
|
||||
# Normal mode exits cleanly when disabled. docker_daily_restart.sh still runs
|
||||
# regardless — update and restart are independent operations.
|
||||
#
|
||||
# Fallback Exclusion
|
||||
# Remainder mode excludes containers owned by the remote server's update cycle
|
||||
# to prevent version divergence across the fallback boundary.
|
||||
#
|
||||
# Running-Only Filter
|
||||
# Stopped containers excluded from remainder mode — intentionally down.
|
||||
#
|
||||
# Image ID Comparison
|
||||
# Containers not restarted unless their image actually changed. Pulls that
|
||||
# result in "already up to date" produce no restart.
|
||||
#
|
||||
# ==============================================================================================
|
||||
# CONFIGURATION
|
||||
# ==============================================================================================
|
||||
#
|
||||
# master.conf
|
||||
#
|
||||
# DAILY_CONTAINER_UPDATES
|
||||
# Enable or disable normal mode. docker_daily_restart.sh runs regardless.
|
||||
# (default: true)
|
||||
#
|
||||
# PROFILE_CRITICAL_CONTAINER_NAMES[emby|critical-data]
|
||||
# Container names for emby and critical-data profiles — excluded from
|
||||
# remainder mode (already updated by the weekly sync window)
|
||||
#
|
||||
# host*.conf
|
||||
#
|
||||
# HOST*_DAILY_RESTART_CONTAINERS
|
||||
# Containers updated in normal mode. Aliased by detect_hosts() →
|
||||
# DAILY_RESTART_CONTAINERS
|
||||
#
|
||||
# ==============================================================================================
|
||||
# RUNTIME MODES
|
||||
# ==============================================================================================
|
||||
#
|
||||
# docker_update.sh
|
||||
# Normal mode — pull latest images for DAILY_RESTART_CONTAINERS
|
||||
#
|
||||
# docker_update.sh --remainder
|
||||
# Remainder mode — pull all running containers not in managed lists,
|
||||
# restart those that received updates, prune dangling images
|
||||
#
|
||||
# docker_update.sh --dry-run
|
||||
# Preview which containers would be pulled without making changes
|
||||
#
|
||||
# docker_update.sh --status
|
||||
# Show configuration and container list for current mode
|
||||
#
|
||||
# docker_update.sh --log
|
||||
# Verbose per-container pull and comparison output
|
||||
#
|
||||
# ==============================================================================================
|
||||
|
||||
SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
|
||||
|
||||
source "$SCRIPT_DIR/../load_config.sh"
|
||||
|
||||
# Pre-parse --remainder before parse_args (unknown args pass through to PARSED_ARGS)
|
||||
REMAINDER_MODE=false
|
||||
_filtered_args=()
|
||||
for _arg in "$@"; do
|
||||
if [[ "$_arg" == "--remainder" ]]; then
|
||||
REMAINDER_MODE=true
|
||||
else
|
||||
_filtered_args+=("$_arg")
|
||||
fi
|
||||
done
|
||||
unset _arg
|
||||
|
||||
parse_args "${_filtered_args[@]}"
|
||||
unset _filtered_args
|
||||
|
||||
# ==============================================================================================
|
||||
# ━━━ Setup ━━━
|
||||
# ==============================================================================================
|
||||
if [[ "$EUID" -ne 0 ]]; then
|
||||
error "Must be run as root"
|
||||
exit 1
|
||||
fi
|
||||
|
||||
acquire_lock
|
||||
|
||||
if ! command -v docker &>/dev/null; then
|
||||
error "Docker command not found"
|
||||
exit 1
|
||||
fi
|
||||
|
||||
detect_hosts
|
||||
|
||||
# ==============================================================================================
|
||||
# ━━━ Container Discovery ━━━
|
||||
# ==============================================================================================
|
||||
if [[ "$REMAINDER_MODE" == true ]]; then
|
||||
declare -A _exclude=()
|
||||
|
||||
# Daily containers — updated by docker_update.sh normal mode
|
||||
for _c in "${DAILY_RESTART_CONTAINERS[@]}"; do
|
||||
[[ -n "$_c" ]] && _exclude["$_c"]=1
|
||||
done
|
||||
|
||||
# Weekly sync-window containers (emby + critical-data) — updated inline by weekly_sync_maintenance.sh
|
||||
_weekly_str="${PROFILE_CRITICAL_CONTAINER_NAMES[emby]:-} ${PROFILE_CRITICAL_CONTAINER_NAMES[critical-data]:-}"
|
||||
read -r -a _weekly_arr <<< "$_weekly_str"
|
||||
for _c in "${_weekly_arr[@]}"; do
|
||||
[[ -n "$_c" ]] && _exclude["$_c"]=1
|
||||
done
|
||||
unset _weekly_str _weekly_arr
|
||||
|
||||
# Fallback coverage containers — owned by the remote server's update cycle.
|
||||
# This server runs them during fallback but should never update them independently.
|
||||
# Updating them here risks version divergence: if remote's writeback after handback
|
||||
# encounters data written by a newer version, it may not handle it correctly.
|
||||
for _tier in 1 2 3 4; do
|
||||
_tier_var="FALLBACK_${MY_ID}_COVERS_${REMOTE_ID}_TIER${_tier}"
|
||||
eval "_tier_arr=(\"\${${_tier_var}[@]:-}\")" 2>/dev/null
|
||||
for _c in "${_tier_arr[@]}"; do
|
||||
[[ -n "$_c" ]] && _exclude["$_c"]=1
|
||||
done
|
||||
done
|
||||
unset _tier _tier_var _tier_arr _c
|
||||
|
||||
mapfile -t _all_running < <(docker ps --format '{{.Names}}' | sort)
|
||||
TARGET_CONTAINERS=()
|
||||
for _c in "${_all_running[@]}"; do
|
||||
[[ -z "${_exclude[$_c]+x}" ]] && TARGET_CONTAINERS+=("$_c")
|
||||
done
|
||||
unset _all_running _exclude _c
|
||||
else
|
||||
if [[ "${DAILY_CONTAINER_UPDATES:-true}" != "true" ]]; then
|
||||
echo "DAILY_CONTAINER_UPDATES=false — skipping container updates"
|
||||
exit 0
|
||||
fi
|
||||
|
||||
if [[ ${#DAILY_RESTART_CONTAINERS[@]} -eq 0 ]]; then
|
||||
warn "DAILY_RESTART_CONTAINERS is empty for $MY_ID — nothing to update"
|
||||
warn "Check HOST${MY_ID#HOST}_DAILY_RESTART_CONTAINERS in host*.conf"
|
||||
exit 0
|
||||
fi
|
||||
|
||||
TARGET_CONTAINERS=("${DAILY_RESTART_CONTAINERS[@]}")
|
||||
fi
|
||||
|
||||
# ==============================================================================================
|
||||
# ━━━ Status ━━━
|
||||
# ==============================================================================================
|
||||
if [[ "$SHOW_STATUS" == true ]]; then
|
||||
echo ""
|
||||
echo "━━━━━ $ICON_SUMMARY STATUS ━━━━━"
|
||||
echo "$ICON_HOST Identity: $MY_ID ($LOCAL_SERVER_NAME)"
|
||||
echo "$ICON_GEAR Mode: $([[ "$REMAINDER_MODE" == true ]] && echo "remainder" || echo "normal (daily)")"
|
||||
if [[ "$REMAINDER_MODE" == true ]]; then
|
||||
echo "$ICON_CONTAINERS Containers: ${#TARGET_CONTAINERS[@]} running (excluding daily, weekly sync, and fallback)"
|
||||
for _c in "${TARGET_CONTAINERS[@]}"; do echo " $_c"; done
|
||||
else
|
||||
echo "$ICON_CONTAINERS Containers: ${TARGET_CONTAINERS[*]}"
|
||||
echo "$ICON_GEAR Enabled: ${DAILY_CONTAINER_UPDATES:-true}"
|
||||
fi
|
||||
echo "$ICON_GEAR Dry Run: $DRY_RUN"
|
||||
echo "━━━━━━━━━━━━━━━━━━━━━━━"
|
||||
exit 0
|
||||
fi
|
||||
|
||||
[[ "$DRY_RUN" == true ]] && warn "DRY RUN — no images will be pulled"
|
||||
|
||||
if [[ ${#TARGET_CONTAINERS[@]} -eq 0 ]]; then
|
||||
echo "No containers to update"
|
||||
exit 0
|
||||
fi
|
||||
|
||||
# ==============================================================================================
|
||||
# ━━━ Pull Updates ━━━
|
||||
# ==============================================================================================
|
||||
echo ""
|
||||
if [[ "$REMAINDER_MODE" == true ]]; then
|
||||
echo "━━━ $ICON_CONTAINERS Docker Update (remainder) — $(date '+%Y-%m-%d %H:%M:%S') ━━━"
|
||||
echo "$ICON_CONTAINERS Updating ${#TARGET_CONTAINERS[@]} container(s) (not in daily or weekly sync)"
|
||||
log "$ICON_CONTAINERS Remainder targets: ${TARGET_CONTAINERS[*]}"
|
||||
else
|
||||
echo "━━━ $ICON_CONTAINERS Docker Update — $(date '+%Y-%m-%d %H:%M:%S') ━━━"
|
||||
log "$ICON_CONTAINERS Containers: ${TARGET_CONTAINERS[*]}"
|
||||
fi
|
||||
echo ""
|
||||
|
||||
START=$(date +%s)
|
||||
UPDATED=()
|
||||
UP_TO_DATE=()
|
||||
FAILED=()
|
||||
OLD_IMAGE_IDS=() # old image IDs to explicitly remove after rebuilds
|
||||
SKIPPED=()
|
||||
|
||||
for container in "${TARGET_CONTAINERS[@]}"; do
|
||||
[[ -z "$container" ]] && continue
|
||||
log "━━━ $ICON_CONTAINERS $container ━━━"
|
||||
|
||||
if ! docker inspect "$container" &>/dev/null; then
|
||||
warn "$container — not found, skipping"
|
||||
SKIPPED+=("$container")
|
||||
continue
|
||||
fi
|
||||
|
||||
IMAGE=$(docker inspect --format='{{.Config.Image}}' "$container" 2>/dev/null)
|
||||
if [[ -z "$IMAGE" ]]; then
|
||||
warn "$container — could not determine image, skipping"
|
||||
SKIPPED+=("$container")
|
||||
continue
|
||||
fi
|
||||
|
||||
log "$container — image: $IMAGE"
|
||||
|
||||
if [[ "$DRY_RUN" == true ]]; then
|
||||
warn "DRY RUN — would pull: $IMAGE"
|
||||
UPDATED+=("$container")
|
||||
continue
|
||||
fi
|
||||
|
||||
# Capture image ID before pull to detect whether an update landed
|
||||
OLD_ID=$(docker image inspect "$IMAGE" --format='{{.Id}}' 2>/dev/null || echo "")
|
||||
|
||||
log "$ICON_SYNC Pulling $IMAGE..."
|
||||
if [[ "$ENABLE_LOGGING" == "true" ]]; then
|
||||
docker pull "$IMAGE" 2>&1 | grep -E "^(Status:|Digest:|Error|error)" | sed 's/^/ /'
|
||||
_pull_rc=${PIPESTATUS[0]}
|
||||
else
|
||||
docker pull "$IMAGE" >/dev/null 2>&1
|
||||
_pull_rc=$?
|
||||
fi
|
||||
NEW_ID=$(docker image inspect "$IMAGE" --format='{{.Id}}' 2>/dev/null || echo "")
|
||||
|
||||
if [[ $_pull_rc -eq 0 ]]; then
|
||||
if [[ -n "$OLD_ID" ]] && [[ "$OLD_ID" != "$NEW_ID" ]]; then
|
||||
log "$ICON_DONE $container — updated ✅ (${OLD_ID:7:12} → ${NEW_ID:7:12})"
|
||||
UPDATED+=("$container")
|
||||
OLD_IMAGE_IDS+=("$OLD_ID")
|
||||
else
|
||||
log "$container — already up to date (${NEW_ID:7:12})"
|
||||
UP_TO_DATE+=("$container")
|
||||
fi
|
||||
else
|
||||
warn "$container — pull failed ($IMAGE)"
|
||||
FAILED+=("$container")
|
||||
fi
|
||||
|
||||
done
|
||||
|
||||
# ── Recreate containers that received a new image ────────────────────────────
|
||||
# docker restart uses the image ID baked in at creation time — it never picks
|
||||
# up the new digest. rebuild_container reads the stored XML template, stops the
|
||||
# old container, recreates it (new image, same config), then prunes the old image.
|
||||
REBUILT=()
|
||||
REBUILD_FAILED=()
|
||||
if [[ ${#UPDATED[@]} -gt 0 ]]; then
|
||||
for container in "${UPDATED[@]}"; do
|
||||
[[ -z "$container" ]] && continue
|
||||
if [[ "$DRY_RUN" == true ]]; then
|
||||
warn "DRY RUN — would rebuild $container from template"
|
||||
REBUILT+=("$container")
|
||||
continue
|
||||
fi
|
||||
log "$ICON_SYNC Rebuilding $container from template on new image..."
|
||||
if /usr/local/emhttp/plugins/dynamix.docker.manager/scripts/rebuild_container "$container" >/dev/null 2>&1; then
|
||||
log "$ICON_DONE $container rebuilt ✅"
|
||||
REBUILT+=("$container")
|
||||
else
|
||||
error "Failed to rebuild $container — will be picked up by docker_daily_restart.sh"
|
||||
notify "$container failed to rebuild after image update on $(hostname)" "Docker Update" "warning"
|
||||
REBUILD_FAILED+=("$container")
|
||||
fi
|
||||
done
|
||||
fi
|
||||
|
||||
# ── Remove old images ────────────────────────────────────────────────────────
|
||||
# Explicitly rmi by the IDs captured before each pull. Tagged images are never
|
||||
# caught by dangling-only prune, so this is the only reliable cleanup path.
|
||||
# Fall through to dangling prune to catch any leftovers from other update paths.
|
||||
if [[ "$DRY_RUN" == true ]]; then
|
||||
warn "DRY RUN — would remove ${#OLD_IMAGE_IDS[@]} old image(s) and prune dangling"
|
||||
PRUNED_SUMMARY="(dry run)"
|
||||
else
|
||||
for _old_id in "${OLD_IMAGE_IDS[@]}"; do
|
||||
docker rmi "$_old_id" >/dev/null 2>&1 || true
|
||||
done
|
||||
PRUNED_OUTPUT=$(docker image prune -f 2>&1)
|
||||
[[ "$ENABLE_LOGGING" == "true" ]] && echo "$PRUNED_OUTPUT" | sed 's/^/ /'
|
||||
PRUNED_SUMMARY=$(echo "$PRUNED_OUTPUT" | grep -E "^Total reclaimed" || echo "nothing reclaimed")
|
||||
fi
|
||||
|
||||
END=$(date +%s)
|
||||
|
||||
# ==============================================================================================
|
||||
# ━━━ Summary ━━━
|
||||
# ==============================================================================================
|
||||
if [[ "$REMAINDER_MODE" == true ]]; then
|
||||
echo "━━━━━ $ICON_SUMMARY DOCKER UPDATE (REMAINDER) SUMMARY ━━━━━"
|
||||
else
|
||||
echo "━━━━━ $ICON_SUMMARY DOCKER UPDATE SUMMARY ━━━━━"
|
||||
fi
|
||||
echo "$ICON_HOST Identity: $MY_ID ($LOCAL_SERVER_NAME)"
|
||||
echo "$ICON_TIME Duration: $(format_duration $(( END - START )))"
|
||||
if [[ ${#UPDATED[@]} -gt 0 ]]; then
|
||||
echo "$ICON_DONE Updated: ${#UPDATED[@]}"
|
||||
log " ${UPDATED[*]}"
|
||||
fi
|
||||
[[ ${#REBUILT[@]} -gt 0 ]] && echo "$ICON_SYNC Rebuilt: ${#REBUILT[@]}"
|
||||
[[ ${#REBUILD_FAILED[@]} -gt 0 ]] && echo "$ICON_ERROR Rebuild fail:${REBUILD_FAILED[*]}"
|
||||
[[ ${#UP_TO_DATE[@]} -gt 0 ]] && log "$ICON_RUNNING Up to date: ${UP_TO_DATE[*]}"
|
||||
[[ ${#SKIPPED[@]} -gt 0 ]] && log "$ICON_WARN Skipped: ${SKIPPED[*]}"
|
||||
[[ ${#FAILED[@]} -gt 0 ]] && echo "$ICON_ERROR Failed: ${FAILED[*]}"
|
||||
echo "$ICON_SYNC Pruned: ${PRUNED_SUMMARY:-none}"
|
||||
|
||||
if [[ "$DRY_RUN" == true ]]; then
|
||||
warn "DRY RUN — no images pulled"
|
||||
elif [[ ${#FAILED[@]} -eq 0 ]]; then
|
||||
echo "$ICON_DONE Status: done ✅ — ${#UPDATED[@]} updated, ${#UP_TO_DATE[@]} current"
|
||||
else
|
||||
warn "Status: ${#FAILED[@]} pull(s) failed — restart will proceed with existing images"
|
||||
fi
|
||||
echo "━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━"
|
||||
|
||||
# Pull failures are non-fatal — restart proceeds regardless
|
||||
exit 0
|
||||
+399
@@ -0,0 +1,399 @@
|
||||
#!/bin/bash
|
||||
# ==============================================================================================
|
||||
# ================================= Docker Update ==============================================
|
||||
# ==============================================================================================
|
||||
#
|
||||
# PURPOSE
|
||||
# ─────────────────────────────────────────────────────────────────────────────
|
||||
# Pulls the latest images for configured containers. Two modes: normal (daily)
|
||||
# and remainder (weekly).
|
||||
#
|
||||
# Normal mode is called by daily_sync_maintenance.sh before docker_daily_restart.sh.
|
||||
# Containers stay running during the pull — no extra downtime beyond what the
|
||||
# nightly restart already causes.
|
||||
#
|
||||
# Remainder mode is called by weekly_sync_maintenance.sh as the final update step.
|
||||
# It catches everything that normal mode and the weekly sync window did not already
|
||||
# update — derived automatically from docker ps, nothing to configure.
|
||||
#
|
||||
# ==============================================================================================
|
||||
# OPERATIONAL MODEL
|
||||
# ==============================================================================================
|
||||
#
|
||||
# Normal mode (daily):
|
||||
# Targets DAILY_RESTART_CONTAINERS — same list used by docker_daily_restart.sh.
|
||||
# Pull → compare old vs new image ID → mark updated or already current.
|
||||
# docker_daily_restart.sh runs after — containers restart onto the fresh image.
|
||||
#
|
||||
# Remainder mode (weekly):
|
||||
# Targets all currently running containers NOT in:
|
||||
# DAILY_RESTART_CONTAINERS — already updated daily
|
||||
# emby + critical-data profiles — updated inline by the weekly sync window
|
||||
# FALLBACK_*_TIER* — owned by the remote server's update cycle
|
||||
# Pull → compare → prune dangling images.
|
||||
#
|
||||
# ==============================================================================================
|
||||
# DESIGN PRINCIPLES
|
||||
# ==============================================================================================
|
||||
#
|
||||
# Single List
|
||||
# Normal mode reuses DAILY_RESTART_CONTAINERS rather than maintaining a
|
||||
# separate update list. Adding or removing a container from the restart list
|
||||
# automatically updates the image pull list — one change, both places.
|
||||
#
|
||||
# Version Ownership
|
||||
# Fallback containers are excluded from remainder mode. This server only runs
|
||||
# them during a fallback. The remote server owns their version — if remainder
|
||||
# updates them independently and a handback occurs, the remote's older image
|
||||
# may not handle data written by the newer version.
|
||||
#
|
||||
# State Respect
|
||||
# Stopped containers are never targeted. Pulling while stopped adds no value
|
||||
# and a stopped container was likely halted intentionally.
|
||||
#
|
||||
# ==============================================================================================
|
||||
# OPERATIONAL SAFEGUARDS
|
||||
# ==============================================================================================
|
||||
#
|
||||
# Lock Acquisition
|
||||
# Prevents concurrent execution via acquire_lock(). Safe to call from
|
||||
# maintenance scripts without risk of overlap.
|
||||
#
|
||||
# Host Detection
|
||||
# detect_hosts() identifies which server is running the script and aliases
|
||||
# HOST*_DAILY_RESTART_CONTAINERS to the correct host's values.
|
||||
#
|
||||
# Root Enforcement
|
||||
# Docker operations require root privileges.
|
||||
#
|
||||
# DAILY_CONTAINER_UPDATES Toggle
|
||||
# Normal mode exits cleanly when disabled. docker_daily_restart.sh still runs
|
||||
# regardless — update and restart are independent operations.
|
||||
#
|
||||
# Fallback Exclusion
|
||||
# Remainder mode excludes containers owned by the remote server's update cycle
|
||||
# to prevent version divergence across the fallback boundary.
|
||||
#
|
||||
# Running-Only Filter
|
||||
# Stopped containers excluded from remainder mode — intentionally down.
|
||||
#
|
||||
# Image ID Comparison
|
||||
# Containers not restarted unless their image actually changed. Pulls that
|
||||
# result in "already up to date" produce no restart.
|
||||
#
|
||||
# ==============================================================================================
|
||||
# CONFIGURATION
|
||||
# ==============================================================================================
|
||||
#
|
||||
# master.conf
|
||||
#
|
||||
# DAILY_CONTAINER_UPDATES
|
||||
# Enable or disable normal mode. docker_daily_restart.sh runs regardless.
|
||||
# (default: true)
|
||||
#
|
||||
# PROFILE_CRITICAL_CONTAINER_NAMES[emby|critical-data]
|
||||
# Container names for emby and critical-data profiles — excluded from
|
||||
# remainder mode (already updated by the weekly sync window)
|
||||
#
|
||||
# host*.conf
|
||||
#
|
||||
# HOST*_DAILY_RESTART_CONTAINERS
|
||||
# Containers updated in normal mode. Aliased by detect_hosts() →
|
||||
# DAILY_RESTART_CONTAINERS
|
||||
#
|
||||
# ==============================================================================================
|
||||
# RUNTIME MODES
|
||||
# ==============================================================================================
|
||||
#
|
||||
# docker_update.sh
|
||||
# Normal mode — pull latest images for DAILY_RESTART_CONTAINERS
|
||||
#
|
||||
# docker_update.sh --remainder
|
||||
# Remainder mode — pull all running containers not in managed lists,
|
||||
# restart those that received updates, prune dangling images
|
||||
#
|
||||
# docker_update.sh --dry-run
|
||||
# Preview which containers would be pulled without making changes
|
||||
#
|
||||
# docker_update.sh --status
|
||||
# Show configuration and container list for current mode
|
||||
#
|
||||
# docker_update.sh --log
|
||||
# Verbose per-container pull and comparison output
|
||||
#
|
||||
# ==============================================================================================
|
||||
|
||||
SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
|
||||
|
||||
source "$SCRIPT_DIR/../load_config.sh"
|
||||
|
||||
# Pre-parse --remainder before parse_args (unknown args pass through to PARSED_ARGS)
|
||||
REMAINDER_MODE=false
|
||||
_filtered_args=()
|
||||
for _arg in "$@"; do
|
||||
if [[ "$_arg" == "--remainder" ]]; then
|
||||
REMAINDER_MODE=true
|
||||
else
|
||||
_filtered_args+=("$_arg")
|
||||
fi
|
||||
done
|
||||
unset _arg
|
||||
|
||||
parse_args "${_filtered_args[@]}"
|
||||
unset _filtered_args
|
||||
|
||||
# ==============================================================================================
|
||||
# ━━━ Setup ━━━
|
||||
# ==============================================================================================
|
||||
if [[ "$EUID" -ne 0 ]]; then
|
||||
error "Must be run as root"
|
||||
exit 1
|
||||
fi
|
||||
|
||||
acquire_lock
|
||||
|
||||
if ! command -v docker &>/dev/null; then
|
||||
error "Docker command not found"
|
||||
exit 1
|
||||
fi
|
||||
|
||||
detect_hosts
|
||||
|
||||
# ==============================================================================================
|
||||
# ━━━ Container Discovery ━━━
|
||||
# ==============================================================================================
|
||||
if [[ "$REMAINDER_MODE" == true ]]; then
|
||||
declare -A _exclude=()
|
||||
|
||||
# Daily containers — updated by docker_update.sh normal mode
|
||||
for _c in "${DAILY_RESTART_CONTAINERS[@]}"; do
|
||||
[[ -n "$_c" ]] && _exclude["$_c"]=1
|
||||
done
|
||||
|
||||
# Weekly sync-window containers (emby + critical-data) — updated inline by weekly_sync_maintenance.sh
|
||||
_weekly_str="${PROFILE_CRITICAL_CONTAINER_NAMES[emby]:-} ${PROFILE_CRITICAL_CONTAINER_NAMES[critical-data]:-}"
|
||||
read -r -a _weekly_arr <<< "$_weekly_str"
|
||||
for _c in "${_weekly_arr[@]}"; do
|
||||
[[ -n "$_c" ]] && _exclude["$_c"]=1
|
||||
done
|
||||
unset _weekly_str _weekly_arr
|
||||
|
||||
# Fallback coverage containers — owned by the remote server's update cycle.
|
||||
# This server runs them during fallback but should never update them independently.
|
||||
# Updating them here risks version divergence: if remote's writeback after handback
|
||||
# encounters data written by a newer version, it may not handle it correctly.
|
||||
for _tier in 1 2 3 4; do
|
||||
_tier_var="FALLBACK_${MY_ID}_COVERS_${REMOTE_ID}_TIER${_tier}"
|
||||
eval "_tier_arr=(\"\${${_tier_var}[@]:-}\")" 2>/dev/null
|
||||
for _c in "${_tier_arr[@]}"; do
|
||||
[[ -n "$_c" ]] && _exclude["$_c"]=1
|
||||
done
|
||||
done
|
||||
unset _tier _tier_var _tier_arr _c
|
||||
|
||||
mapfile -t _all_running < <(docker ps --format '{{.Names}}' | sort)
|
||||
TARGET_CONTAINERS=()
|
||||
for _c in "${_all_running[@]}"; do
|
||||
[[ -z "${_exclude[$_c]+x}" ]] && TARGET_CONTAINERS+=("$_c")
|
||||
done
|
||||
unset _all_running _exclude _c
|
||||
else
|
||||
if [[ "${DAILY_CONTAINER_UPDATES:-true}" != "true" ]]; then
|
||||
echo "DAILY_CONTAINER_UPDATES=false — skipping container updates"
|
||||
exit 0
|
||||
fi
|
||||
|
||||
if [[ ${#DAILY_RESTART_CONTAINERS[@]} -eq 0 ]]; then
|
||||
warn "DAILY_RESTART_CONTAINERS is empty for $MY_ID — nothing to update"
|
||||
warn "Check HOST${MY_ID#HOST}_DAILY_RESTART_CONTAINERS in host*.conf"
|
||||
exit 0
|
||||
fi
|
||||
|
||||
TARGET_CONTAINERS=("${DAILY_RESTART_CONTAINERS[@]}")
|
||||
fi
|
||||
|
||||
# ==============================================================================================
|
||||
# ━━━ Status ━━━
|
||||
# ==============================================================================================
|
||||
if [[ "$SHOW_STATUS" == true ]]; then
|
||||
echo ""
|
||||
echo "━━━━━ $ICON_SUMMARY STATUS ━━━━━"
|
||||
echo "$ICON_HOST Identity: $MY_ID ($LOCAL_SERVER_NAME)"
|
||||
echo "$ICON_GEAR Mode: $([[ "$REMAINDER_MODE" == true ]] && echo "remainder" || echo "normal (daily)")"
|
||||
if [[ "$REMAINDER_MODE" == true ]]; then
|
||||
echo "$ICON_CONTAINERS Containers: ${#TARGET_CONTAINERS[@]} running (excluding daily, weekly sync, and fallback)"
|
||||
for _c in "${TARGET_CONTAINERS[@]}"; do echo " $_c"; done
|
||||
else
|
||||
echo "$ICON_CONTAINERS Containers: ${TARGET_CONTAINERS[*]}"
|
||||
echo "$ICON_GEAR Enabled: ${DAILY_CONTAINER_UPDATES:-true}"
|
||||
fi
|
||||
echo "$ICON_GEAR Dry Run: $DRY_RUN"
|
||||
echo "━━━━━━━━━━━━━━━━━━━━━━━"
|
||||
exit 0
|
||||
fi
|
||||
|
||||
[[ "$DRY_RUN" == true ]] && warn "DRY RUN — no images will be pulled"
|
||||
|
||||
if [[ ${#TARGET_CONTAINERS[@]} -eq 0 ]]; then
|
||||
echo "No containers to update"
|
||||
exit 0
|
||||
fi
|
||||
|
||||
# ==============================================================================================
|
||||
# ━━━ Pull Updates ━━━
|
||||
# ==============================================================================================
|
||||
echo ""
|
||||
if [[ "$REMAINDER_MODE" == true ]]; then
|
||||
echo "━━━ $ICON_CONTAINERS Docker Update (remainder) — $(date '+%Y-%m-%d %H:%M:%S') ━━━"
|
||||
echo "$ICON_CONTAINERS Updating ${#TARGET_CONTAINERS[@]} container(s) (not in daily or weekly sync)"
|
||||
log "$ICON_CONTAINERS Remainder targets: ${TARGET_CONTAINERS[*]}"
|
||||
else
|
||||
echo "━━━ $ICON_CONTAINERS Docker Update — $(date '+%Y-%m-%d %H:%M:%S') ━━━"
|
||||
log "$ICON_CONTAINERS Containers: ${TARGET_CONTAINERS[*]}"
|
||||
fi
|
||||
echo ""
|
||||
|
||||
START=$(date +%s)
|
||||
UPDATED=()
|
||||
UP_TO_DATE=()
|
||||
FAILED=()
|
||||
OLD_IMAGE_IDS=() # old image IDs to explicitly remove after rebuilds
|
||||
SKIPPED=()
|
||||
|
||||
for container in "${TARGET_CONTAINERS[@]}"; do
|
||||
[[ -z "$container" ]] && continue
|
||||
log "━━━ $ICON_CONTAINERS $container ━━━"
|
||||
|
||||
if ! docker inspect "$container" &>/dev/null; then
|
||||
warn "$container — not found, skipping"
|
||||
SKIPPED+=("$container")
|
||||
continue
|
||||
fi
|
||||
|
||||
IMAGE=$(docker inspect --format='{{.Config.Image}}' "$container" 2>/dev/null)
|
||||
if [[ -z "$IMAGE" ]]; then
|
||||
warn "$container — could not determine image, skipping"
|
||||
SKIPPED+=("$container")
|
||||
continue
|
||||
fi
|
||||
|
||||
log "$container — image: $IMAGE"
|
||||
|
||||
if [[ "$DRY_RUN" == true ]]; then
|
||||
warn "DRY RUN — would pull: $IMAGE"
|
||||
UPDATED+=("$container")
|
||||
continue
|
||||
fi
|
||||
|
||||
# Capture the image ID the container is currently running on, and the
|
||||
# image ID :latest points to before the pull. After pulling, we rebuild if
|
||||
# either a new digest landed OR the container is behind what :latest is now.
|
||||
CONTAINER_IMAGE_ID=$(docker inspect "$container" --format='{{.Image}}' 2>/dev/null || echo "")
|
||||
OLD_ID=$(docker image inspect "$IMAGE" --format='{{.Id}}' 2>/dev/null || echo "")
|
||||
|
||||
log "$ICON_SYNC Pulling $IMAGE..."
|
||||
if [[ "$ENABLE_LOGGING" == "true" ]]; then
|
||||
docker pull "$IMAGE" 2>&1 | grep -E "^(Status:|Digest:|Error|error)" | sed 's/^/ /'
|
||||
_pull_rc=${PIPESTATUS[0]}
|
||||
else
|
||||
docker pull "$IMAGE" >/dev/null 2>&1
|
||||
_pull_rc=$?
|
||||
fi
|
||||
NEW_ID=$(docker image inspect "$IMAGE" --format='{{.Id}}' 2>/dev/null || echo "")
|
||||
|
||||
if [[ $_pull_rc -eq 0 ]]; then
|
||||
_pull_new=$( [[ -n "$OLD_ID" && "$OLD_ID" != "$NEW_ID" ]] && echo true || echo false)
|
||||
_container_behind=$([[ -n "$CONTAINER_IMAGE_ID" && -n "$NEW_ID" && "$CONTAINER_IMAGE_ID" != "$NEW_ID" ]] && echo true || echo false)
|
||||
|
||||
if [[ "$_pull_new" == true || "$_container_behind" == true ]]; then
|
||||
[[ "$_pull_new" == true ]] && log "$ICON_DONE $container — new image (${OLD_ID:7:12} → ${NEW_ID:7:12})"
|
||||
[[ "$_container_behind" == true && "$_pull_new" == false ]] && log "$ICON_DONE $container — image already pulled, container behind (${CONTAINER_IMAGE_ID:7:12} → ${NEW_ID:7:12})"
|
||||
UPDATED+=("$container")
|
||||
OLD_IMAGE_IDS+=("$CONTAINER_IMAGE_ID")
|
||||
else
|
||||
log "$container — up to date (${NEW_ID:7:12})"
|
||||
UP_TO_DATE+=("$container")
|
||||
fi
|
||||
else
|
||||
warn "$container — pull failed ($IMAGE)"
|
||||
FAILED+=("$container")
|
||||
fi
|
||||
|
||||
done
|
||||
|
||||
# ── Recreate containers that received a new image ────────────────────────────
|
||||
# docker restart uses the image ID baked in at creation time — it never picks
|
||||
# up the new digest. rebuild_container reads the stored XML template, stops the
|
||||
# old container, recreates it (new image, same config), then prunes the old image.
|
||||
REBUILT=()
|
||||
REBUILD_FAILED=()
|
||||
if [[ ${#UPDATED[@]} -gt 0 ]]; then
|
||||
for container in "${UPDATED[@]}"; do
|
||||
[[ -z "$container" ]] && continue
|
||||
if [[ "$DRY_RUN" == true ]]; then
|
||||
warn "DRY RUN — would rebuild $container from template"
|
||||
REBUILT+=("$container")
|
||||
continue
|
||||
fi
|
||||
log "$ICON_SYNC Rebuilding $container from template on new image..."
|
||||
if /usr/local/emhttp/plugins/dynamix.docker.manager/scripts/rebuild_container "$container" >/dev/null 2>&1; then
|
||||
log "$ICON_DONE $container rebuilt ✅"
|
||||
REBUILT+=("$container")
|
||||
else
|
||||
error "Failed to rebuild $container — will be picked up by docker_daily_restart.sh"
|
||||
notify "$container failed to rebuild after image update on $(hostname)" "Docker Update" "warning"
|
||||
REBUILD_FAILED+=("$container")
|
||||
fi
|
||||
done
|
||||
fi
|
||||
|
||||
# ── Remove old images ────────────────────────────────────────────────────────
|
||||
# Explicitly rmi by the IDs captured before each pull. Tagged images are never
|
||||
# caught by dangling-only prune, so this is the only reliable cleanup path.
|
||||
# Fall through to dangling prune to catch any leftovers from other update paths.
|
||||
if [[ "$DRY_RUN" == true ]]; then
|
||||
warn "DRY RUN — would remove ${#OLD_IMAGE_IDS[@]} old image(s) and prune dangling"
|
||||
PRUNED_SUMMARY="(dry run)"
|
||||
else
|
||||
for _old_id in "${OLD_IMAGE_IDS[@]}"; do
|
||||
docker rmi "$_old_id" >/dev/null 2>&1 || true
|
||||
done
|
||||
PRUNED_OUTPUT=$(docker image prune -f 2>&1)
|
||||
[[ "$ENABLE_LOGGING" == "true" ]] && echo "$PRUNED_OUTPUT" | sed 's/^/ /'
|
||||
PRUNED_SUMMARY=$(echo "$PRUNED_OUTPUT" | grep -E "^Total reclaimed" || echo "nothing reclaimed")
|
||||
fi
|
||||
|
||||
END=$(date +%s)
|
||||
|
||||
# ==============================================================================================
|
||||
# ━━━ Summary ━━━
|
||||
# ==============================================================================================
|
||||
if [[ "$REMAINDER_MODE" == true ]]; then
|
||||
echo "━━━━━ $ICON_SUMMARY DOCKER UPDATE (REMAINDER) SUMMARY ━━━━━"
|
||||
else
|
||||
echo "━━━━━ $ICON_SUMMARY DOCKER UPDATE SUMMARY ━━━━━"
|
||||
fi
|
||||
echo "$ICON_HOST Identity: $MY_ID ($LOCAL_SERVER_NAME)"
|
||||
echo "$ICON_TIME Duration: $(format_duration $(( END - START )))"
|
||||
if [[ ${#UPDATED[@]} -gt 0 ]]; then
|
||||
echo "$ICON_DONE Updated: ${#UPDATED[@]}"
|
||||
log " ${UPDATED[*]}"
|
||||
fi
|
||||
[[ ${#REBUILT[@]} -gt 0 ]] && echo "$ICON_SYNC Rebuilt: ${#REBUILT[@]}"
|
||||
[[ ${#REBUILD_FAILED[@]} -gt 0 ]] && echo "$ICON_ERROR Rebuild fail:${REBUILD_FAILED[*]}"
|
||||
[[ ${#UP_TO_DATE[@]} -gt 0 ]] && log "$ICON_RUNNING Up to date: ${UP_TO_DATE[*]}"
|
||||
[[ ${#SKIPPED[@]} -gt 0 ]] && log "$ICON_WARN Skipped: ${SKIPPED[*]}"
|
||||
[[ ${#FAILED[@]} -gt 0 ]] && echo "$ICON_ERROR Failed: ${FAILED[*]}"
|
||||
echo "$ICON_SYNC Pruned: ${PRUNED_SUMMARY:-none}"
|
||||
|
||||
if [[ "$DRY_RUN" == true ]]; then
|
||||
warn "DRY RUN — no images pulled"
|
||||
elif [[ ${#FAILED[@]} -eq 0 ]]; then
|
||||
echo "$ICON_DONE Status: done ✅ — ${#UPDATED[@]} updated, ${#UP_TO_DATE[@]} current"
|
||||
else
|
||||
warn "Status: ${#FAILED[@]} pull(s) failed — restart will proceed with existing images"
|
||||
fi
|
||||
echo "━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━"
|
||||
|
||||
# Pull failures are non-fatal — restart proceeds regardless
|
||||
exit 0
|
||||
+315
@@ -0,0 +1,315 @@
|
||||
#!/bin/bash
|
||||
# ==============================================================================================
|
||||
# ============================= Docker Update — Remaining ======================================
|
||||
# ==============================================================================================
|
||||
#
|
||||
# PURPOSE
|
||||
# ─────────────────────────────────────────────────────────────────────────────
|
||||
# Weekly sweep that pulls the latest image for every running container not
|
||||
# already covered by the daily or weekly managed update cycles. Restarts
|
||||
# containers that received a new image, then prunes dangling images.
|
||||
#
|
||||
# Called by weekly_sync_maintenance.sh as the final step in the weekly window.
|
||||
# Derives its target list automatically from docker ps minus the two managed
|
||||
# lists — there is nothing to configure for this script.
|
||||
#
|
||||
# Together with docker_update.sh (normal + remainder modes), every deployed
|
||||
# container receives at least one image pull per week without any per-container
|
||||
# configuration required here.
|
||||
#
|
||||
# ==============================================================================================
|
||||
# OPERATIONAL SAFEGUARDS
|
||||
# ==============================================================================================
|
||||
#
|
||||
# Lock Acquisition
|
||||
# Prevents concurrent execution via acquire_lock(). Safe to call from
|
||||
# weekly maintenance scripts without risk of overlap.
|
||||
#
|
||||
# Host Detection
|
||||
# detect_hosts() identifies which server is running the script and aliases
|
||||
# HOST*_DAILY_RESTART_CONTAINERS and HOST*_WEEKLY_RESTART_CONTAINERS to
|
||||
# the correct host's values for exclusion.
|
||||
#
|
||||
# Root Enforcement
|
||||
# Docker operations require root privileges.
|
||||
#
|
||||
# WEEKLY_REMAINING_UPDATES Toggle
|
||||
# Exits cleanly when disabled via master.conf.
|
||||
#
|
||||
# Running-Only Filter
|
||||
# Stopped containers excluded — intentionally down, pulling adds no value.
|
||||
#
|
||||
# Image ID Comparison
|
||||
# Containers not restarted unless their image actually changed.
|
||||
#
|
||||
# ==============================================================================================
|
||||
# CONFIGURATION
|
||||
# ==============================================================================================
|
||||
#
|
||||
# master.conf
|
||||
#
|
||||
# WEEKLY_REMAINING_UPDATES
|
||||
# Enable or disable this script. (default: true)
|
||||
# To disable without the toggle: remove from WEEKLY_MAINTENANCE_SCRIPTS.
|
||||
#
|
||||
# host*.conf
|
||||
#
|
||||
# HOST*_DAILY_RESTART_CONTAINERS
|
||||
# Excluded from this script — already updated daily. Aliased by
|
||||
# detect_hosts() → DAILY_RESTART_CONTAINERS
|
||||
#
|
||||
# HOST*_WEEKLY_RESTART_CONTAINERS
|
||||
# Excluded from this script — already updated by weekly sync window.
|
||||
# Aliased by detect_hosts() → WEEKLY_RESTART_CONTAINERS
|
||||
#
|
||||
# ==============================================================================================
|
||||
# RUNTIME MODES
|
||||
# ==============================================================================================
|
||||
#
|
||||
# docker_update_remaining.sh
|
||||
# Pull all remaining running containers, restart those updated, prune images
|
||||
#
|
||||
# docker_update_remaining.sh --dry-run
|
||||
# Preview which containers would be pulled and restarted
|
||||
#
|
||||
# docker_update_remaining.sh --status
|
||||
# Show exclusion lists and current remaining container count
|
||||
#
|
||||
# docker_update_remaining.sh --log
|
||||
# Verbose per-container pull and restart output
|
||||
#
|
||||
# ==============================================================================================
|
||||
|
||||
SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
|
||||
|
||||
source "$SCRIPT_DIR/../load_config.sh"
|
||||
|
||||
parse_args "$@"
|
||||
|
||||
# ==============================================================================================
|
||||
# ━━━ Setup ━━━
|
||||
# ==============================================================================================
|
||||
if [[ "$EUID" -ne 0 ]]; then
|
||||
error "Must be run as root"
|
||||
exit 1
|
||||
fi
|
||||
|
||||
acquire_lock
|
||||
|
||||
if ! command -v docker &>/dev/null; then
|
||||
error "Docker command not found"
|
||||
exit 1
|
||||
fi
|
||||
|
||||
detect_hosts
|
||||
|
||||
if [[ "${WEEKLY_REMAINING_UPDATES:-true}" != "true" ]]; then
|
||||
echo "WEEKLY_REMAINING_UPDATES=false — skipping remaining container updates"
|
||||
exit 0
|
||||
fi
|
||||
|
||||
# ── Build exclusion set from daily + weekly managed lists ─────────────────────────────────────
|
||||
declare -A EXCLUDED
|
||||
for c in "${DAILY_RESTART_CONTAINERS[@]}" "${WEEKLY_RESTART_CONTAINERS[@]}"; do
|
||||
[[ -n "$c" ]] && EXCLUDED["$c"]=1
|
||||
done
|
||||
|
||||
# ── Get all running containers ────────────────────────────────────────────────────────────────
|
||||
mapfile -t ALL_RUNNING < <(docker ps --format '{{.Names}}' 2>/dev/null | sort)
|
||||
|
||||
# ── Derive remainder: running minus excluded ──────────────────────────────────────────────────
|
||||
REMAINING=()
|
||||
for c in "${ALL_RUNNING[@]}"; do
|
||||
[[ -z "$c" ]] && continue
|
||||
[[ -n "${EXCLUDED[$c]:-}" ]] && continue
|
||||
REMAINING+=("$c")
|
||||
done
|
||||
|
||||
# ==============================================================================================
|
||||
# ━━━ Status ━━━
|
||||
# ==============================================================================================
|
||||
if [[ "$SHOW_STATUS" == true ]]; then
|
||||
echo ""
|
||||
echo "━━━━━ $ICON_SUMMARY STATUS ━━━━━"
|
||||
echo "$ICON_HOST Identity: $MY_ID ($LOCAL_SERVER_NAME)"
|
||||
echo "$ICON_GEAR Enabled: ${WEEKLY_REMAINING_UPDATES:-true}"
|
||||
echo "$ICON_CONTAINERS All running: ${#ALL_RUNNING[@]}"
|
||||
echo "$ICON_CONTAINERS Excluded: ${!EXCLUDED[*]}"
|
||||
echo "$ICON_CONTAINERS Remaining: ${REMAINING[*]:-none}"
|
||||
echo "━━━━━━━━━━━━━━━━━━━━━━━"
|
||||
exit 0
|
||||
fi
|
||||
|
||||
if [[ ${#REMAINING[@]} -eq 0 ]]; then
|
||||
echo "No remaining containers to update — all running containers are covered by daily/weekly lists"
|
||||
exit 0
|
||||
fi
|
||||
|
||||
[[ "$DRY_RUN" == true ]] && warn "DRY RUN — no images will be pulled or containers restarted"
|
||||
|
||||
# ==============================================================================================
|
||||
# ── FUNCTIONS ─────────────────────────────────────────────────────────────────────────────────
|
||||
# ==============================================================================================
|
||||
|
||||
# docker_cmd, retry_docker, verify_running — defined in common.sh
|
||||
|
||||
# ==============================================================================================
|
||||
# ━━━ Pull Updates ━━━
|
||||
# ==============================================================================================
|
||||
echo ""
|
||||
echo "━━━ $ICON_CONTAINERS Docker Update (Remaining) — $(date '+%Y-%m-%d %H:%M:%S') ━━━"
|
||||
log "$ICON_CONTAINERS Containers: ${REMAINING[*]}"
|
||||
log "$ICON_CONTAINERS Excluded (managed elsewhere): ${!EXCLUDED[*]}"
|
||||
echo ""
|
||||
|
||||
START=$(date +%s)
|
||||
UPDATED=()
|
||||
UP_TO_DATE=()
|
||||
FAILED=()
|
||||
|
||||
for container in "${REMAINING[@]}"; do
|
||||
[[ -z "$container" ]] && continue
|
||||
log "━━━ $ICON_CONTAINERS $container ━━━"
|
||||
|
||||
IMAGE=$(docker inspect --format='{{.Config.Image}}' "$container" 2>/dev/null)
|
||||
if [[ -z "$IMAGE" ]]; then
|
||||
warn "$container — could not determine image, skipping"
|
||||
FAILED+=("$container")
|
||||
continue
|
||||
fi
|
||||
|
||||
log "$container — image: $IMAGE"
|
||||
|
||||
if [[ "$DRY_RUN" == true ]]; then
|
||||
warn "DRY RUN — would pull: $IMAGE"
|
||||
UPDATED+=("$container")
|
||||
continue
|
||||
fi
|
||||
|
||||
OLD_ID=$(docker image inspect "$IMAGE" --format='{{.Id}}' 2>/dev/null || echo "")
|
||||
|
||||
log "$ICON_SYNC Pulling $IMAGE..."
|
||||
if [[ "$ENABLE_LOGGING" == "true" ]]; then
|
||||
docker pull "$IMAGE" 2>&1 | grep -E "^(Status:|Digest:|Error|error)" | sed 's/^/ /'
|
||||
_pull_rc=${PIPESTATUS[0]}
|
||||
else
|
||||
docker pull "$IMAGE" >/dev/null 2>&1
|
||||
_pull_rc=$?
|
||||
fi
|
||||
NEW_ID=$(docker image inspect "$IMAGE" --format='{{.Id}}' 2>/dev/null || echo "")
|
||||
|
||||
if [[ $_pull_rc -eq 0 ]]; then
|
||||
if [[ -n "$OLD_ID" ]] && [[ "$OLD_ID" != "$NEW_ID" ]]; then
|
||||
log "$ICON_DONE $container — updated ✅ (${OLD_ID:7:12} → ${NEW_ID:7:12})"
|
||||
UPDATED+=("$container")
|
||||
else
|
||||
log "$container — already up to date (${NEW_ID:7:12})"
|
||||
UP_TO_DATE+=("$container")
|
||||
fi
|
||||
else
|
||||
warn "$container — pull failed ($IMAGE)"
|
||||
FAILED+=("$container")
|
||||
fi
|
||||
|
||||
done
|
||||
|
||||
# ==============================================================================================
|
||||
# ━━━ Restart Updated Containers ━━━
|
||||
# ==============================================================================================
|
||||
RESTARTED=()
|
||||
RESTART_FAILED=()
|
||||
SKIPPED_STOPPED=()
|
||||
|
||||
if [[ ${#UPDATED[@]} -gt 0 ]]; then
|
||||
echo ""
|
||||
echo "━━━ $ICON_CONTAINERS Restarting Updated Containers — $(date '+%Y-%m-%d %H:%M:%S') ━━━"
|
||||
log "$ICON_CONTAINERS Containers with new image: ${UPDATED[*]}"
|
||||
|
||||
for container in "${UPDATED[@]}"; do
|
||||
[[ -z "$container" ]] && continue
|
||||
log "━━━ $ICON_CONTAINERS $container ━━━"
|
||||
|
||||
STATUS=$(timeout "$DOCKER_TIMEOUT" docker inspect -f '{{.State.Running}}' "$container" 2>/dev/null)
|
||||
|
||||
if [[ "$STATUS" != "true" ]]; then
|
||||
log "$ICON_NOT_RUNNING $container is stopped — skipping restart (respecting stopped state)"
|
||||
SKIPPED_STOPPED+=("$container")
|
||||
continue
|
||||
fi
|
||||
|
||||
if [[ "$DRY_RUN" == true ]]; then
|
||||
warn "DRY RUN — would restart $container"
|
||||
RESTARTED+=("$container")
|
||||
continue
|
||||
fi
|
||||
|
||||
log "$ICON_RUNNING $container — recreating from template on new image..."
|
||||
if /usr/local/emhttp/plugins/dynamix.docker.manager/scripts/rebuild_container "$container" >/dev/null 2>&1; then
|
||||
if verify_running "$container"; then
|
||||
log "$ICON_DONE $container recreated and running ✅"
|
||||
RESTARTED+=("$container")
|
||||
else
|
||||
error "$container recreated but not running — may be intentionally stopped"
|
||||
RESTARTED+=("$container")
|
||||
fi
|
||||
else
|
||||
error "Failed to rebuild $container from template"
|
||||
notify "$container failed to rebuild after update on $(hostname)" "Docker Update Remaining" "warning"
|
||||
RESTART_FAILED+=("$container")
|
||||
fi
|
||||
done
|
||||
else
|
||||
log "No containers received a new image — nothing to restart"
|
||||
fi
|
||||
|
||||
# ==============================================================================================
|
||||
# ━━━ Prune Old Images ━━━
|
||||
# ==============================================================================================
|
||||
echo ""
|
||||
echo "━━━ $ICON_SYNC Pruning Dangling Images — $(date '+%Y-%m-%d %H:%M:%S') ━━━"
|
||||
|
||||
if [[ "$DRY_RUN" == true ]]; then
|
||||
warn "DRY RUN — would prune dangling images"
|
||||
PRUNED_SUMMARY="(dry run)"
|
||||
else
|
||||
PRUNED_OUTPUT=$(docker image prune -f 2>&1)
|
||||
[[ "$ENABLE_LOGGING" == "true" ]] && echo "$PRUNED_OUTPUT" | sed 's/^/ /'
|
||||
PRUNED_SUMMARY=$(echo "$PRUNED_OUTPUT" | grep -E "^Total reclaimed" || echo "nothing reclaimed")
|
||||
fi
|
||||
|
||||
END=$(date +%s)
|
||||
|
||||
# ==============================================================================================
|
||||
# ━━━ Summary ━━━
|
||||
# ==============================================================================================
|
||||
echo ""
|
||||
echo "━━━━━ $ICON_SUMMARY DOCKER UPDATE (REMAINING) SUMMARY ━━━━━"
|
||||
echo "$ICON_HOST Identity: $MY_ID ($LOCAL_SERVER_NAME)"
|
||||
echo "$ICON_TIME Duration: $(format_duration $(( END - START )))"
|
||||
echo "$ICON_CONTAINERS Scope: ${#ALL_RUNNING[@]} running — ${#EXCLUDED[@]} managed = ${#REMAINING[@]} checked"
|
||||
if [[ ${#UPDATED[@]} -gt 0 ]]; then
|
||||
echo "$ICON_DONE New image: ${#UPDATED[@]}"
|
||||
log " ${UPDATED[*]}"
|
||||
fi
|
||||
[[ ${#UP_TO_DATE[@]} -gt 0 ]] && log "$ICON_RUNNING Up to date: ${UP_TO_DATE[*]}"
|
||||
[[ ${#FAILED[@]} -gt 0 ]] && echo "$ICON_ERROR Pull failed: ${FAILED[*]}"
|
||||
if [[ ${#RESTARTED[@]} -gt 0 ]]; then
|
||||
echo "$ICON_DONE Restarted: ${#RESTARTED[@]}"
|
||||
log " ${RESTARTED[*]}"
|
||||
fi
|
||||
[[ ${#SKIPPED_STOPPED[@]} -gt 0 ]] && log "$ICON_WARN Not running: ${SKIPPED_STOPPED[*]} (skipped restart)"
|
||||
[[ ${#RESTART_FAILED[@]} -gt 0 ]] && echo "$ICON_ERROR Restart fail:${RESTART_FAILED[*]}"
|
||||
echo "$ICON_SYNC Pruned: ${PRUNED_SUMMARY:-none}"
|
||||
|
||||
ALL_FAILED=$(( ${#FAILED[@]} + ${#RESTART_FAILED[@]} ))
|
||||
if [[ "$DRY_RUN" == true ]]; then
|
||||
warn "DRY RUN — no changes made"
|
||||
elif [[ "$ALL_FAILED" -eq 0 ]]; then
|
||||
echo "$ICON_DONE Status: done ✅ — ${#RESTARTED[@]} restarted, ${#UP_TO_DATE[@]} current"
|
||||
else
|
||||
warn "Status: $ALL_FAILED error(s) — ${#FAILED[@]} pull failure(s), ${#RESTART_FAILED[@]} restart failure(s)"
|
||||
fi
|
||||
echo "━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━"
|
||||
|
||||
[[ "$ALL_FAILED" -gt 0 ]] && exit 1
|
||||
exit 0
|
||||
+320
@@ -0,0 +1,320 @@
|
||||
#!/bin/bash
|
||||
# ==============================================================================================
|
||||
# ============================= Docker Update — Remaining ======================================
|
||||
# ==============================================================================================
|
||||
#
|
||||
# PURPOSE
|
||||
# ─────────────────────────────────────────────────────────────────────────────
|
||||
# Weekly sweep that pulls the latest image for every running container not
|
||||
# already covered by the daily or weekly managed update cycles. Restarts
|
||||
# containers that received a new image, then prunes dangling images.
|
||||
#
|
||||
# Called by weekly_sync_maintenance.sh as the final step in the weekly window.
|
||||
# Derives its target list automatically from docker ps minus the two managed
|
||||
# lists — there is nothing to configure for this script.
|
||||
#
|
||||
# Together with docker_update.sh (normal + remainder modes), every deployed
|
||||
# container receives at least one image pull per week without any per-container
|
||||
# configuration required here.
|
||||
#
|
||||
# ==============================================================================================
|
||||
# OPERATIONAL SAFEGUARDS
|
||||
# ==============================================================================================
|
||||
#
|
||||
# Lock Acquisition
|
||||
# Prevents concurrent execution via acquire_lock(). Safe to call from
|
||||
# weekly maintenance scripts without risk of overlap.
|
||||
#
|
||||
# Host Detection
|
||||
# detect_hosts() identifies which server is running the script and aliases
|
||||
# HOST*_DAILY_RESTART_CONTAINERS and HOST*_WEEKLY_RESTART_CONTAINERS to
|
||||
# the correct host's values for exclusion.
|
||||
#
|
||||
# Root Enforcement
|
||||
# Docker operations require root privileges.
|
||||
#
|
||||
# WEEKLY_REMAINING_UPDATES Toggle
|
||||
# Exits cleanly when disabled via master.conf.
|
||||
#
|
||||
# Running-Only Filter
|
||||
# Stopped containers excluded — intentionally down, pulling adds no value.
|
||||
#
|
||||
# Image ID Comparison
|
||||
# Containers not restarted unless their image actually changed.
|
||||
#
|
||||
# ==============================================================================================
|
||||
# CONFIGURATION
|
||||
# ==============================================================================================
|
||||
#
|
||||
# master.conf
|
||||
#
|
||||
# WEEKLY_REMAINING_UPDATES
|
||||
# Enable or disable this script. (default: true)
|
||||
# To disable without the toggle: remove from WEEKLY_MAINTENANCE_SCRIPTS.
|
||||
#
|
||||
# host*.conf
|
||||
#
|
||||
# HOST*_DAILY_RESTART_CONTAINERS
|
||||
# Excluded from this script — already updated daily. Aliased by
|
||||
# detect_hosts() → DAILY_RESTART_CONTAINERS
|
||||
#
|
||||
# HOST*_WEEKLY_RESTART_CONTAINERS
|
||||
# Excluded from this script — already updated by weekly sync window.
|
||||
# Aliased by detect_hosts() → WEEKLY_RESTART_CONTAINERS
|
||||
#
|
||||
# ==============================================================================================
|
||||
# RUNTIME MODES
|
||||
# ==============================================================================================
|
||||
#
|
||||
# docker_update_remaining.sh
|
||||
# Pull all remaining running containers, restart those updated, prune images
|
||||
#
|
||||
# docker_update_remaining.sh --dry-run
|
||||
# Preview which containers would be pulled and restarted
|
||||
#
|
||||
# docker_update_remaining.sh --status
|
||||
# Show exclusion lists and current remaining container count
|
||||
#
|
||||
# docker_update_remaining.sh --log
|
||||
# Verbose per-container pull and restart output
|
||||
#
|
||||
# ==============================================================================================
|
||||
|
||||
SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
|
||||
|
||||
source "$SCRIPT_DIR/../load_config.sh"
|
||||
|
||||
parse_args "$@"
|
||||
|
||||
# ==============================================================================================
|
||||
# ━━━ Setup ━━━
|
||||
# ==============================================================================================
|
||||
if [[ "$EUID" -ne 0 ]]; then
|
||||
error "Must be run as root"
|
||||
exit 1
|
||||
fi
|
||||
|
||||
acquire_lock
|
||||
|
||||
if ! command -v docker &>/dev/null; then
|
||||
error "Docker command not found"
|
||||
exit 1
|
||||
fi
|
||||
|
||||
detect_hosts
|
||||
|
||||
if [[ "${WEEKLY_REMAINING_UPDATES:-true}" != "true" ]]; then
|
||||
echo "WEEKLY_REMAINING_UPDATES=false — skipping remaining container updates"
|
||||
exit 0
|
||||
fi
|
||||
|
||||
# ── Build exclusion set from daily + weekly managed lists ─────────────────────────────────────
|
||||
declare -A EXCLUDED
|
||||
for c in "${DAILY_RESTART_CONTAINERS[@]}" "${WEEKLY_RESTART_CONTAINERS[@]}"; do
|
||||
[[ -n "$c" ]] && EXCLUDED["$c"]=1
|
||||
done
|
||||
|
||||
# ── Get all running containers ────────────────────────────────────────────────────────────────
|
||||
mapfile -t ALL_RUNNING < <(docker ps --format '{{.Names}}' 2>/dev/null | sort)
|
||||
|
||||
# ── Derive remainder: running minus excluded ──────────────────────────────────────────────────
|
||||
REMAINING=()
|
||||
for c in "${ALL_RUNNING[@]}"; do
|
||||
[[ -z "$c" ]] && continue
|
||||
[[ -n "${EXCLUDED[$c]:-}" ]] && continue
|
||||
REMAINING+=("$c")
|
||||
done
|
||||
|
||||
# ==============================================================================================
|
||||
# ━━━ Status ━━━
|
||||
# ==============================================================================================
|
||||
if [[ "$SHOW_STATUS" == true ]]; then
|
||||
echo ""
|
||||
echo "━━━━━ $ICON_SUMMARY STATUS ━━━━━"
|
||||
echo "$ICON_HOST Identity: $MY_ID ($LOCAL_SERVER_NAME)"
|
||||
echo "$ICON_GEAR Enabled: ${WEEKLY_REMAINING_UPDATES:-true}"
|
||||
echo "$ICON_CONTAINERS All running: ${#ALL_RUNNING[@]}"
|
||||
echo "$ICON_CONTAINERS Excluded: ${!EXCLUDED[*]}"
|
||||
echo "$ICON_CONTAINERS Remaining: ${REMAINING[*]:-none}"
|
||||
echo "━━━━━━━━━━━━━━━━━━━━━━━"
|
||||
exit 0
|
||||
fi
|
||||
|
||||
if [[ ${#REMAINING[@]} -eq 0 ]]; then
|
||||
echo "No remaining containers to update — all running containers are covered by daily/weekly lists"
|
||||
exit 0
|
||||
fi
|
||||
|
||||
[[ "$DRY_RUN" == true ]] && warn "DRY RUN — no images will be pulled or containers restarted"
|
||||
|
||||
# ==============================================================================================
|
||||
# ── FUNCTIONS ─────────────────────────────────────────────────────────────────────────────────
|
||||
# ==============================================================================================
|
||||
|
||||
# docker_cmd, retry_docker, verify_running — defined in common.sh
|
||||
|
||||
# ==============================================================================================
|
||||
# ━━━ Pull Updates ━━━
|
||||
# ==============================================================================================
|
||||
echo ""
|
||||
echo "━━━ $ICON_CONTAINERS Docker Update (Remaining) — $(date '+%Y-%m-%d %H:%M:%S') ━━━"
|
||||
log "$ICON_CONTAINERS Containers: ${REMAINING[*]}"
|
||||
log "$ICON_CONTAINERS Excluded (managed elsewhere): ${!EXCLUDED[*]}"
|
||||
echo ""
|
||||
|
||||
START=$(date +%s)
|
||||
UPDATED=()
|
||||
UP_TO_DATE=()
|
||||
FAILED=()
|
||||
OLD_IMAGE_IDS=()
|
||||
|
||||
for container in "${REMAINING[@]}"; do
|
||||
[[ -z "$container" ]] && continue
|
||||
log "━━━ $ICON_CONTAINERS $container ━━━"
|
||||
|
||||
IMAGE=$(docker inspect --format='{{.Config.Image}}' "$container" 2>/dev/null)
|
||||
if [[ -z "$IMAGE" ]]; then
|
||||
warn "$container — could not determine image, skipping"
|
||||
FAILED+=("$container")
|
||||
continue
|
||||
fi
|
||||
|
||||
log "$container — image: $IMAGE"
|
||||
|
||||
if [[ "$DRY_RUN" == true ]]; then
|
||||
warn "DRY RUN — would pull: $IMAGE"
|
||||
UPDATED+=("$container")
|
||||
continue
|
||||
fi
|
||||
|
||||
OLD_ID=$(docker image inspect "$IMAGE" --format='{{.Id}}' 2>/dev/null || echo "")
|
||||
|
||||
log "$ICON_SYNC Pulling $IMAGE..."
|
||||
if [[ "$ENABLE_LOGGING" == "true" ]]; then
|
||||
docker pull "$IMAGE" 2>&1 | grep -E "^(Status:|Digest:|Error|error)" | sed 's/^/ /'
|
||||
_pull_rc=${PIPESTATUS[0]}
|
||||
else
|
||||
docker pull "$IMAGE" >/dev/null 2>&1
|
||||
_pull_rc=$?
|
||||
fi
|
||||
NEW_ID=$(docker image inspect "$IMAGE" --format='{{.Id}}' 2>/dev/null || echo "")
|
||||
|
||||
if [[ $_pull_rc -eq 0 ]]; then
|
||||
if [[ -n "$OLD_ID" ]] && [[ "$OLD_ID" != "$NEW_ID" ]]; then
|
||||
log "$ICON_DONE $container — updated ✅ (${OLD_ID:7:12} → ${NEW_ID:7:12})"
|
||||
UPDATED+=("$container")
|
||||
OLD_IMAGE_IDS+=("$OLD_ID")
|
||||
else
|
||||
log "$container — already up to date (${NEW_ID:7:12})"
|
||||
UP_TO_DATE+=("$container")
|
||||
fi
|
||||
else
|
||||
warn "$container — pull failed ($IMAGE)"
|
||||
FAILED+=("$container")
|
||||
fi
|
||||
|
||||
done
|
||||
|
||||
# ==============================================================================================
|
||||
# ━━━ Restart Updated Containers ━━━
|
||||
# ==============================================================================================
|
||||
RESTARTED=()
|
||||
RESTART_FAILED=()
|
||||
SKIPPED_STOPPED=()
|
||||
|
||||
if [[ ${#UPDATED[@]} -gt 0 ]]; then
|
||||
echo ""
|
||||
echo "━━━ $ICON_CONTAINERS Restarting Updated Containers — $(date '+%Y-%m-%d %H:%M:%S') ━━━"
|
||||
log "$ICON_CONTAINERS Containers with new image: ${UPDATED[*]}"
|
||||
|
||||
for container in "${UPDATED[@]}"; do
|
||||
[[ -z "$container" ]] && continue
|
||||
log "━━━ $ICON_CONTAINERS $container ━━━"
|
||||
|
||||
STATUS=$(timeout "$DOCKER_TIMEOUT" docker inspect -f '{{.State.Running}}' "$container" 2>/dev/null)
|
||||
|
||||
if [[ "$STATUS" != "true" ]]; then
|
||||
log "$ICON_NOT_RUNNING $container is stopped — skipping restart (respecting stopped state)"
|
||||
SKIPPED_STOPPED+=("$container")
|
||||
continue
|
||||
fi
|
||||
|
||||
if [[ "$DRY_RUN" == true ]]; then
|
||||
warn "DRY RUN — would restart $container"
|
||||
RESTARTED+=("$container")
|
||||
continue
|
||||
fi
|
||||
|
||||
log "$ICON_RUNNING $container — recreating from template on new image..."
|
||||
if /usr/local/emhttp/plugins/dynamix.docker.manager/scripts/rebuild_container "$container" >/dev/null 2>&1; then
|
||||
if verify_running "$container"; then
|
||||
log "$ICON_DONE $container recreated and running ✅"
|
||||
RESTARTED+=("$container")
|
||||
else
|
||||
error "$container recreated but not running — may be intentionally stopped"
|
||||
RESTARTED+=("$container")
|
||||
fi
|
||||
else
|
||||
error "Failed to rebuild $container from template"
|
||||
notify "$container failed to rebuild after update on $(hostname)" "Docker Update Remaining" "warning"
|
||||
RESTART_FAILED+=("$container")
|
||||
fi
|
||||
done
|
||||
else
|
||||
log "No containers received a new image — nothing to restart"
|
||||
fi
|
||||
|
||||
# ==============================================================================================
|
||||
# ━━━ Prune Old Images ━━━
|
||||
# ==============================================================================================
|
||||
echo ""
|
||||
echo "━━━ $ICON_SYNC Pruning Dangling Images — $(date '+%Y-%m-%d %H:%M:%S') ━━━"
|
||||
|
||||
if [[ "$DRY_RUN" == true ]]; then
|
||||
warn "DRY RUN — would remove ${#OLD_IMAGE_IDS[@]} old image(s) and prune dangling"
|
||||
PRUNED_SUMMARY="(dry run)"
|
||||
else
|
||||
for _old_id in "${OLD_IMAGE_IDS[@]}"; do
|
||||
docker rmi "$_old_id" >/dev/null 2>&1 || true
|
||||
done
|
||||
PRUNED_OUTPUT=$(docker image prune -f 2>&1)
|
||||
[[ "$ENABLE_LOGGING" == "true" ]] && echo "$PRUNED_OUTPUT" | sed 's/^/ /'
|
||||
PRUNED_SUMMARY=$(echo "$PRUNED_OUTPUT" | grep -E "^Total reclaimed" || echo "nothing reclaimed")
|
||||
fi
|
||||
|
||||
END=$(date +%s)
|
||||
|
||||
# ==============================================================================================
|
||||
# ━━━ Summary ━━━
|
||||
# ==============================================================================================
|
||||
echo ""
|
||||
echo "━━━━━ $ICON_SUMMARY DOCKER UPDATE (REMAINING) SUMMARY ━━━━━"
|
||||
echo "$ICON_HOST Identity: $MY_ID ($LOCAL_SERVER_NAME)"
|
||||
echo "$ICON_TIME Duration: $(format_duration $(( END - START )))"
|
||||
echo "$ICON_CONTAINERS Scope: ${#ALL_RUNNING[@]} running — ${#EXCLUDED[@]} managed = ${#REMAINING[@]} checked"
|
||||
if [[ ${#UPDATED[@]} -gt 0 ]]; then
|
||||
echo "$ICON_DONE New image: ${#UPDATED[@]}"
|
||||
log " ${UPDATED[*]}"
|
||||
fi
|
||||
[[ ${#UP_TO_DATE[@]} -gt 0 ]] && log "$ICON_RUNNING Up to date: ${UP_TO_DATE[*]}"
|
||||
[[ ${#FAILED[@]} -gt 0 ]] && echo "$ICON_ERROR Pull failed: ${FAILED[*]}"
|
||||
if [[ ${#RESTARTED[@]} -gt 0 ]]; then
|
||||
echo "$ICON_DONE Restarted: ${#RESTARTED[@]}"
|
||||
log " ${RESTARTED[*]}"
|
||||
fi
|
||||
[[ ${#SKIPPED_STOPPED[@]} -gt 0 ]] && log "$ICON_WARN Not running: ${SKIPPED_STOPPED[*]} (skipped restart)"
|
||||
[[ ${#RESTART_FAILED[@]} -gt 0 ]] && echo "$ICON_ERROR Restart fail:${RESTART_FAILED[*]}"
|
||||
echo "$ICON_SYNC Pruned: ${PRUNED_SUMMARY:-none}"
|
||||
|
||||
ALL_FAILED=$(( ${#FAILED[@]} + ${#RESTART_FAILED[@]} ))
|
||||
if [[ "$DRY_RUN" == true ]]; then
|
||||
warn "DRY RUN — no changes made"
|
||||
elif [[ "$ALL_FAILED" -eq 0 ]]; then
|
||||
echo "$ICON_DONE Status: done ✅ — ${#RESTARTED[@]} restarted, ${#UP_TO_DATE[@]} current"
|
||||
else
|
||||
warn "Status: $ALL_FAILED error(s) — ${#FAILED[@]} pull failure(s), ${#RESTART_FAILED[@]} restart failure(s)"
|
||||
fi
|
||||
echo "━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━"
|
||||
|
||||
[[ "$ALL_FAILED" -gt 0 ]] && exit 1
|
||||
exit 0
|
||||
+325
@@ -0,0 +1,325 @@
|
||||
#!/bin/bash
|
||||
# ==============================================================================================
|
||||
# ============================= Docker Update — Remaining ======================================
|
||||
# ==============================================================================================
|
||||
#
|
||||
# PURPOSE
|
||||
# ─────────────────────────────────────────────────────────────────────────────
|
||||
# Weekly sweep that pulls the latest image for every running container not
|
||||
# already covered by the daily or weekly managed update cycles. Restarts
|
||||
# containers that received a new image, then prunes dangling images.
|
||||
#
|
||||
# Called by weekly_sync_maintenance.sh as the final step in the weekly window.
|
||||
# Derives its target list automatically from docker ps minus the two managed
|
||||
# lists — there is nothing to configure for this script.
|
||||
#
|
||||
# Together with docker_update.sh (normal + remainder modes), every deployed
|
||||
# container receives at least one image pull per week without any per-container
|
||||
# configuration required here.
|
||||
#
|
||||
# ==============================================================================================
|
||||
# OPERATIONAL SAFEGUARDS
|
||||
# ==============================================================================================
|
||||
#
|
||||
# Lock Acquisition
|
||||
# Prevents concurrent execution via acquire_lock(). Safe to call from
|
||||
# weekly maintenance scripts without risk of overlap.
|
||||
#
|
||||
# Host Detection
|
||||
# detect_hosts() identifies which server is running the script and aliases
|
||||
# HOST*_DAILY_RESTART_CONTAINERS and HOST*_WEEKLY_RESTART_CONTAINERS to
|
||||
# the correct host's values for exclusion.
|
||||
#
|
||||
# Root Enforcement
|
||||
# Docker operations require root privileges.
|
||||
#
|
||||
# WEEKLY_REMAINING_UPDATES Toggle
|
||||
# Exits cleanly when disabled via master.conf.
|
||||
#
|
||||
# Running-Only Filter
|
||||
# Stopped containers excluded — intentionally down, pulling adds no value.
|
||||
#
|
||||
# Image ID Comparison
|
||||
# Containers not restarted unless their image actually changed.
|
||||
#
|
||||
# ==============================================================================================
|
||||
# CONFIGURATION
|
||||
# ==============================================================================================
|
||||
#
|
||||
# master.conf
|
||||
#
|
||||
# WEEKLY_REMAINING_UPDATES
|
||||
# Enable or disable this script. (default: true)
|
||||
# To disable without the toggle: remove from WEEKLY_MAINTENANCE_SCRIPTS.
|
||||
#
|
||||
# host*.conf
|
||||
#
|
||||
# HOST*_DAILY_RESTART_CONTAINERS
|
||||
# Excluded from this script — already updated daily. Aliased by
|
||||
# detect_hosts() → DAILY_RESTART_CONTAINERS
|
||||
#
|
||||
# HOST*_WEEKLY_RESTART_CONTAINERS
|
||||
# Excluded from this script — already updated by weekly sync window.
|
||||
# Aliased by detect_hosts() → WEEKLY_RESTART_CONTAINERS
|
||||
#
|
||||
# ==============================================================================================
|
||||
# RUNTIME MODES
|
||||
# ==============================================================================================
|
||||
#
|
||||
# docker_update_remaining.sh
|
||||
# Pull all remaining running containers, restart those updated, prune images
|
||||
#
|
||||
# docker_update_remaining.sh --dry-run
|
||||
# Preview which containers would be pulled and restarted
|
||||
#
|
||||
# docker_update_remaining.sh --status
|
||||
# Show exclusion lists and current remaining container count
|
||||
#
|
||||
# docker_update_remaining.sh --log
|
||||
# Verbose per-container pull and restart output
|
||||
#
|
||||
# ==============================================================================================
|
||||
|
||||
SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
|
||||
|
||||
source "$SCRIPT_DIR/../load_config.sh"
|
||||
|
||||
parse_args "$@"
|
||||
|
||||
# ==============================================================================================
|
||||
# ━━━ Setup ━━━
|
||||
# ==============================================================================================
|
||||
if [[ "$EUID" -ne 0 ]]; then
|
||||
error "Must be run as root"
|
||||
exit 1
|
||||
fi
|
||||
|
||||
acquire_lock
|
||||
|
||||
if ! command -v docker &>/dev/null; then
|
||||
error "Docker command not found"
|
||||
exit 1
|
||||
fi
|
||||
|
||||
detect_hosts
|
||||
|
||||
if [[ "${WEEKLY_REMAINING_UPDATES:-true}" != "true" ]]; then
|
||||
echo "WEEKLY_REMAINING_UPDATES=false — skipping remaining container updates"
|
||||
exit 0
|
||||
fi
|
||||
|
||||
# ── Build exclusion set from daily + weekly managed lists ─────────────────────────────────────
|
||||
declare -A EXCLUDED
|
||||
for c in "${DAILY_RESTART_CONTAINERS[@]}" "${WEEKLY_RESTART_CONTAINERS[@]}"; do
|
||||
[[ -n "$c" ]] && EXCLUDED["$c"]=1
|
||||
done
|
||||
|
||||
# ── Get all running containers ────────────────────────────────────────────────────────────────
|
||||
mapfile -t ALL_RUNNING < <(docker ps --format '{{.Names}}' 2>/dev/null | sort)
|
||||
|
||||
# ── Derive remainder: running minus excluded ──────────────────────────────────────────────────
|
||||
REMAINING=()
|
||||
for c in "${ALL_RUNNING[@]}"; do
|
||||
[[ -z "$c" ]] && continue
|
||||
[[ -n "${EXCLUDED[$c]:-}" ]] && continue
|
||||
REMAINING+=("$c")
|
||||
done
|
||||
|
||||
# ==============================================================================================
|
||||
# ━━━ Status ━━━
|
||||
# ==============================================================================================
|
||||
if [[ "$SHOW_STATUS" == true ]]; then
|
||||
echo ""
|
||||
echo "━━━━━ $ICON_SUMMARY STATUS ━━━━━"
|
||||
echo "$ICON_HOST Identity: $MY_ID ($LOCAL_SERVER_NAME)"
|
||||
echo "$ICON_GEAR Enabled: ${WEEKLY_REMAINING_UPDATES:-true}"
|
||||
echo "$ICON_CONTAINERS All running: ${#ALL_RUNNING[@]}"
|
||||
echo "$ICON_CONTAINERS Excluded: ${!EXCLUDED[*]}"
|
||||
echo "$ICON_CONTAINERS Remaining: ${REMAINING[*]:-none}"
|
||||
echo "━━━━━━━━━━━━━━━━━━━━━━━"
|
||||
exit 0
|
||||
fi
|
||||
|
||||
if [[ ${#REMAINING[@]} -eq 0 ]]; then
|
||||
echo "No remaining containers to update — all running containers are covered by daily/weekly lists"
|
||||
exit 0
|
||||
fi
|
||||
|
||||
[[ "$DRY_RUN" == true ]] && warn "DRY RUN — no images will be pulled or containers restarted"
|
||||
|
||||
# ==============================================================================================
|
||||
# ── FUNCTIONS ─────────────────────────────────────────────────────────────────────────────────
|
||||
# ==============================================================================================
|
||||
|
||||
# docker_cmd, retry_docker, verify_running — defined in common.sh
|
||||
|
||||
# ==============================================================================================
|
||||
# ━━━ Pull Updates ━━━
|
||||
# ==============================================================================================
|
||||
echo ""
|
||||
echo "━━━ $ICON_CONTAINERS Docker Update (Remaining) — $(date '+%Y-%m-%d %H:%M:%S') ━━━"
|
||||
log "$ICON_CONTAINERS Containers: ${REMAINING[*]}"
|
||||
log "$ICON_CONTAINERS Excluded (managed elsewhere): ${!EXCLUDED[*]}"
|
||||
echo ""
|
||||
|
||||
START=$(date +%s)
|
||||
UPDATED=()
|
||||
UP_TO_DATE=()
|
||||
FAILED=()
|
||||
OLD_IMAGE_IDS=()
|
||||
|
||||
for container in "${REMAINING[@]}"; do
|
||||
[[ -z "$container" ]] && continue
|
||||
log "━━━ $ICON_CONTAINERS $container ━━━"
|
||||
|
||||
IMAGE=$(docker inspect --format='{{.Config.Image}}' "$container" 2>/dev/null)
|
||||
if [[ -z "$IMAGE" ]]; then
|
||||
warn "$container — could not determine image, skipping"
|
||||
FAILED+=("$container")
|
||||
continue
|
||||
fi
|
||||
|
||||
log "$container — image: $IMAGE"
|
||||
|
||||
if [[ "$DRY_RUN" == true ]]; then
|
||||
warn "DRY RUN — would pull: $IMAGE"
|
||||
UPDATED+=("$container")
|
||||
continue
|
||||
fi
|
||||
|
||||
CONTAINER_IMAGE_ID=$(docker inspect "$container" --format='{{.Image}}' 2>/dev/null || echo "")
|
||||
OLD_ID=$(docker image inspect "$IMAGE" --format='{{.Id}}' 2>/dev/null || echo "")
|
||||
|
||||
log "$ICON_SYNC Pulling $IMAGE..."
|
||||
if [[ "$ENABLE_LOGGING" == "true" ]]; then
|
||||
docker pull "$IMAGE" 2>&1 | grep -E "^(Status:|Digest:|Error|error)" | sed 's/^/ /'
|
||||
_pull_rc=${PIPESTATUS[0]}
|
||||
else
|
||||
docker pull "$IMAGE" >/dev/null 2>&1
|
||||
_pull_rc=$?
|
||||
fi
|
||||
NEW_ID=$(docker image inspect "$IMAGE" --format='{{.Id}}' 2>/dev/null || echo "")
|
||||
|
||||
if [[ $_pull_rc -eq 0 ]]; then
|
||||
_pull_new=$( [[ -n "$OLD_ID" && "$OLD_ID" != "$NEW_ID" ]] && echo true || echo false)
|
||||
_container_behind=$([[ -n "$CONTAINER_IMAGE_ID" && -n "$NEW_ID" && "$CONTAINER_IMAGE_ID" != "$NEW_ID" ]] && echo true || echo false)
|
||||
|
||||
if [[ "$_pull_new" == true || "$_container_behind" == true ]]; then
|
||||
[[ "$_pull_new" == true ]] && log "$ICON_DONE $container — new image (${OLD_ID:7:12} → ${NEW_ID:7:12})"
|
||||
[[ "$_container_behind" == true && "$_pull_new" == false ]] && log "$ICON_DONE $container — image already pulled, container behind (${CONTAINER_IMAGE_ID:7:12} → ${NEW_ID:7:12})"
|
||||
UPDATED+=("$container")
|
||||
OLD_IMAGE_IDS+=("$CONTAINER_IMAGE_ID")
|
||||
else
|
||||
log "$container — up to date (${NEW_ID:7:12})"
|
||||
UP_TO_DATE+=("$container")
|
||||
fi
|
||||
else
|
||||
warn "$container — pull failed ($IMAGE)"
|
||||
FAILED+=("$container")
|
||||
fi
|
||||
|
||||
done
|
||||
|
||||
# ==============================================================================================
|
||||
# ━━━ Restart Updated Containers ━━━
|
||||
# ==============================================================================================
|
||||
RESTARTED=()
|
||||
RESTART_FAILED=()
|
||||
SKIPPED_STOPPED=()
|
||||
|
||||
if [[ ${#UPDATED[@]} -gt 0 ]]; then
|
||||
echo ""
|
||||
echo "━━━ $ICON_CONTAINERS Restarting Updated Containers — $(date '+%Y-%m-%d %H:%M:%S') ━━━"
|
||||
log "$ICON_CONTAINERS Containers with new image: ${UPDATED[*]}"
|
||||
|
||||
for container in "${UPDATED[@]}"; do
|
||||
[[ -z "$container" ]] && continue
|
||||
log "━━━ $ICON_CONTAINERS $container ━━━"
|
||||
|
||||
STATUS=$(timeout "$DOCKER_TIMEOUT" docker inspect -f '{{.State.Running}}' "$container" 2>/dev/null)
|
||||
|
||||
if [[ "$STATUS" != "true" ]]; then
|
||||
log "$ICON_NOT_RUNNING $container is stopped — skipping restart (respecting stopped state)"
|
||||
SKIPPED_STOPPED+=("$container")
|
||||
continue
|
||||
fi
|
||||
|
||||
if [[ "$DRY_RUN" == true ]]; then
|
||||
warn "DRY RUN — would restart $container"
|
||||
RESTARTED+=("$container")
|
||||
continue
|
||||
fi
|
||||
|
||||
log "$ICON_RUNNING $container — recreating from template on new image..."
|
||||
if /usr/local/emhttp/plugins/dynamix.docker.manager/scripts/rebuild_container "$container" >/dev/null 2>&1; then
|
||||
if verify_running "$container"; then
|
||||
log "$ICON_DONE $container recreated and running ✅"
|
||||
RESTARTED+=("$container")
|
||||
else
|
||||
error "$container recreated but not running — may be intentionally stopped"
|
||||
RESTARTED+=("$container")
|
||||
fi
|
||||
else
|
||||
error "Failed to rebuild $container from template"
|
||||
notify "$container failed to rebuild after update on $(hostname)" "Docker Update Remaining" "warning"
|
||||
RESTART_FAILED+=("$container")
|
||||
fi
|
||||
done
|
||||
else
|
||||
log "No containers received a new image — nothing to restart"
|
||||
fi
|
||||
|
||||
# ==============================================================================================
|
||||
# ━━━ Prune Old Images ━━━
|
||||
# ==============================================================================================
|
||||
echo ""
|
||||
echo "━━━ $ICON_SYNC Pruning Dangling Images — $(date '+%Y-%m-%d %H:%M:%S') ━━━"
|
||||
|
||||
if [[ "$DRY_RUN" == true ]]; then
|
||||
warn "DRY RUN — would remove ${#OLD_IMAGE_IDS[@]} old image(s) and prune dangling"
|
||||
PRUNED_SUMMARY="(dry run)"
|
||||
else
|
||||
for _old_id in "${OLD_IMAGE_IDS[@]}"; do
|
||||
docker rmi "$_old_id" >/dev/null 2>&1 || true
|
||||
done
|
||||
PRUNED_OUTPUT=$(docker image prune -f 2>&1)
|
||||
[[ "$ENABLE_LOGGING" == "true" ]] && echo "$PRUNED_OUTPUT" | sed 's/^/ /'
|
||||
PRUNED_SUMMARY=$(echo "$PRUNED_OUTPUT" | grep -E "^Total reclaimed" || echo "nothing reclaimed")
|
||||
fi
|
||||
|
||||
END=$(date +%s)
|
||||
|
||||
# ==============================================================================================
|
||||
# ━━━ Summary ━━━
|
||||
# ==============================================================================================
|
||||
echo ""
|
||||
echo "━━━━━ $ICON_SUMMARY DOCKER UPDATE (REMAINING) SUMMARY ━━━━━"
|
||||
echo "$ICON_HOST Identity: $MY_ID ($LOCAL_SERVER_NAME)"
|
||||
echo "$ICON_TIME Duration: $(format_duration $(( END - START )))"
|
||||
echo "$ICON_CONTAINERS Scope: ${#ALL_RUNNING[@]} running — ${#EXCLUDED[@]} managed = ${#REMAINING[@]} checked"
|
||||
if [[ ${#UPDATED[@]} -gt 0 ]]; then
|
||||
echo "$ICON_DONE New image: ${#UPDATED[@]}"
|
||||
log " ${UPDATED[*]}"
|
||||
fi
|
||||
[[ ${#UP_TO_DATE[@]} -gt 0 ]] && log "$ICON_RUNNING Up to date: ${UP_TO_DATE[*]}"
|
||||
[[ ${#FAILED[@]} -gt 0 ]] && echo "$ICON_ERROR Pull failed: ${FAILED[*]}"
|
||||
if [[ ${#RESTARTED[@]} -gt 0 ]]; then
|
||||
echo "$ICON_DONE Restarted: ${#RESTARTED[@]}"
|
||||
log " ${RESTARTED[*]}"
|
||||
fi
|
||||
[[ ${#SKIPPED_STOPPED[@]} -gt 0 ]] && log "$ICON_WARN Not running: ${SKIPPED_STOPPED[*]} (skipped restart)"
|
||||
[[ ${#RESTART_FAILED[@]} -gt 0 ]] && echo "$ICON_ERROR Restart fail:${RESTART_FAILED[*]}"
|
||||
echo "$ICON_SYNC Pruned: ${PRUNED_SUMMARY:-none}"
|
||||
|
||||
ALL_FAILED=$(( ${#FAILED[@]} + ${#RESTART_FAILED[@]} ))
|
||||
if [[ "$DRY_RUN" == true ]]; then
|
||||
warn "DRY RUN — no changes made"
|
||||
elif [[ "$ALL_FAILED" -eq 0 ]]; then
|
||||
echo "$ICON_DONE Status: done ✅ — ${#RESTARTED[@]} restarted, ${#UP_TO_DATE[@]} current"
|
||||
else
|
||||
warn "Status: $ALL_FAILED error(s) — ${#FAILED[@]} pull failure(s), ${#RESTART_FAILED[@]} restart failure(s)"
|
||||
fi
|
||||
echo "━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━"
|
||||
|
||||
[[ "$ALL_FAILED" -gt 0 ]] && exit 1
|
||||
exit 0
|
||||
File diff suppressed because it is too large
Load Diff
File diff suppressed because it is too large
Load Diff
+385
@@ -0,0 +1,385 @@
|
||||
#!/bin/bash
|
||||
# ==============================================================================================
|
||||
# ============================= Health Digest ==================================================
|
||||
# ==============================================================================================
|
||||
#
|
||||
# PURPOSE
|
||||
# ─────────────────────────────────────────────────────────────────────────────
|
||||
# Full ecosystem health aggregation from existing state files. Scheduled daily
|
||||
# (8am). DIGEST_PROFILE controls when notifications actually send — the cron
|
||||
# schedule never changes, only the profile in master.conf.
|
||||
#
|
||||
# Reads state files from across the system (watchdog strikes, fallback state,
|
||||
# skip list, bandwidth history, transcode stats, cert status) and compiles them
|
||||
# into a single digest. Reads only — writes nothing, changes nothing.
|
||||
#
|
||||
# ==============================================================================================
|
||||
# OPERATIONAL MODEL
|
||||
# ==============================================================================================
|
||||
#
|
||||
# Three profiles — switch by changing DIGEST_PROFILE in master.conf:
|
||||
#
|
||||
# always — sends every run regardless of findings
|
||||
# Use: daily digest of everything, even when healthy
|
||||
#
|
||||
# smart — sends only when something worth reporting was found
|
||||
# Stays silent on clean days. DIGEST_SMART_ON_* toggles control
|
||||
# what triggers a send — all independently configurable.
|
||||
#
|
||||
# weekly — sends once per week on DIGEST_DAY (default Sunday), silent all other days
|
||||
# Use: one weekly summary without daily noise
|
||||
#
|
||||
# Data sources (reads only):
|
||||
# FALLBACK_STATE_FILE — current fallback state
|
||||
# DOCKER_WATCHDOG_FAILED_FILE — container skip list (manual intervention needed)
|
||||
# WATCHDOG_STATE_FILE — active container watchdog strikes
|
||||
# SYS_WATCHDOG_STATE_FILE — active system watchdog strikes
|
||||
# BANDWIDTH_LOG — yesterday's transfer totals
|
||||
# TRANSCODE_DAILY_LOG — weekly transcode statistics
|
||||
# CERT_MONITOR_DOMAINS — live SSL cert check via openssl
|
||||
# RAMDISK_PATH / TRANSCODE_LINK — current transcode location and usage
|
||||
#
|
||||
# ==============================================================================================
|
||||
# OPERATIONAL SAFEGUARDS
|
||||
# ==============================================================================================
|
||||
#
|
||||
# Single Instance Lock
|
||||
# acquire_lock prevents duplicate reports — report generation takes time.
|
||||
#
|
||||
# Per-Host Variables
|
||||
# detect_hosts() aliases CERT_MONITOR_DOMAINS, RAMDISK_WARN_GB, RAMDISK_SIZE,
|
||||
# RAMDISK_LOW_GB, and all other host-specific vars used in the report.
|
||||
#
|
||||
# Per-Section Guards
|
||||
# Each data source section checks whether its state file exists before reading.
|
||||
# A missing state file is skipped cleanly — it does not abort the report.
|
||||
#
|
||||
# Silent Smart Profile
|
||||
# smart profile produces no output and no notification when nothing worth
|
||||
# reporting is found.
|
||||
#
|
||||
# Notifications Validated
|
||||
# platform_require_cmd confirms notify and openssl are present before use.
|
||||
#
|
||||
# ==============================================================================================
|
||||
# CONFIGURATION
|
||||
# ==============================================================================================
|
||||
#
|
||||
# master.conf
|
||||
#
|
||||
# DIGEST_PROFILE
|
||||
# Notification frequency: always | smart | weekly. (default: weekly)
|
||||
#
|
||||
# DIGEST_DAY
|
||||
# Day name for weekly profile — must match `date +%A` output. (default: Sunday)
|
||||
#
|
||||
# DIGEST_SMART_ON_WATCHDOG
|
||||
# Send smart profile notification if any active watchdog strikes. (default: true)
|
||||
#
|
||||
# DIGEST_SMART_ON_FALLBACK
|
||||
# Send smart profile notification if fallback state is not NORMAL. (default: true)
|
||||
#
|
||||
# DIGEST_SMART_ON_CERT_WARN
|
||||
# Send smart profile notification if any cert is within CERT_WARN_DAYS. (default: true)
|
||||
#
|
||||
# DIGEST_SMART_ON_BANDWIDTH
|
||||
# Send smart profile notification if any transfer exceeded BANDWIDTH_WARN_GB. (default: true)
|
||||
#
|
||||
# CERT_WARN_DAYS / CERT_CRIT_DAYS / CERT_TIMEOUT
|
||||
# Cert check thresholds — shared with cert_monitor.sh.
|
||||
#
|
||||
# BANDWIDTH_WARN_GB
|
||||
# High-transfer threshold — shared with bandwidth_monitor.sh.
|
||||
#
|
||||
# TRANSCODE_DAILY_LOG
|
||||
# Path to the transcode statistics log.
|
||||
#
|
||||
# ==============================================================================================
|
||||
# RUNTIME MODES
|
||||
# ==============================================================================================
|
||||
#
|
||||
# weekly_health_digest.sh
|
||||
# Run digest. DIGEST_PROFILE determines whether a notification is sent.
|
||||
#
|
||||
# weekly_health_digest.sh --dry-run
|
||||
# Generate and display digest output. No notification sent regardless of profile.
|
||||
#
|
||||
# weekly_health_digest.sh --status
|
||||
# Show profile, day, and smart trigger configuration. Then exit.
|
||||
#
|
||||
# weekly_health_digest.sh --log
|
||||
# Verbose per-section output during digest generation.
|
||||
#
|
||||
# ==============================================================================================
|
||||
|
||||
SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
|
||||
|
||||
source "$SCRIPT_DIR/../load_config.sh"
|
||||
|
||||
parse_args "$@"
|
||||
|
||||
# ==============================================================================================
|
||||
# ━━━ Setup ━━━
|
||||
# ==============================================================================================
|
||||
if [[ "$EUID" -ne 0 ]]; then
|
||||
error "Must be run as root"
|
||||
exit 1
|
||||
fi
|
||||
|
||||
platform_require_cmd \
|
||||
"/usr/local/emhttp/plugins/dynamix/scripts/notify" \
|
||||
"" "" \
|
||||
"unRAID notify script" || warn "unRAID notify script not found — native notifications disabled"
|
||||
|
||||
platform_require_cmd \
|
||||
"$(command -v openssl 2>/dev/null || echo /usr/bin/openssl)" \
|
||||
"version" "OpenSSL" \
|
||||
"openssl" || warn "openssl not found — SSL cert checks will be skipped"
|
||||
|
||||
acquire_lock
|
||||
|
||||
# detect_hosts() sets MY_ID and aliases all host-specific vars used in this report
|
||||
detect_hosts
|
||||
|
||||
log "$ICON_GEAR Config: profile=${DIGEST_PROFILE} day=${DIGEST_DAY}"
|
||||
log "$ICON_GEAR Smart triggers: watchdog=${DIGEST_SMART_ON_WATCHDOG} fallback=${DIGEST_SMART_ON_FALLBACK} cert=${DIGEST_SMART_ON_CERT_WARN} bandwidth=${DIGEST_SMART_ON_BANDWIDTH}"
|
||||
|
||||
[[ "$DRY_RUN" == true ]] && warn "DRY RUN — report generated but no notification sent"
|
||||
|
||||
# ==============================================================================================
|
||||
# ━━━ Status ━━━
|
||||
# ==============================================================================================
|
||||
if [[ "$SHOW_STATUS" == true ]]; then
|
||||
echo ""
|
||||
echo "━━━━━ $ICON_SUMMARY STATUS ━━━━━"
|
||||
echo "$ICON_HOST Identity: $MY_ID ($LOCAL_SERVER_NAME)"
|
||||
echo "$ICON_DIGEST Profile: $DIGEST_PROFILE"
|
||||
echo "$ICON_DIGEST Digest day: $DIGEST_DAY"
|
||||
echo "$ICON_DIGEST Smart triggers: watchdog=$DIGEST_SMART_ON_WATCHDOG fallback=$DIGEST_SMART_ON_FALLBACK cert=$DIGEST_SMART_ON_CERT_WARN bandwidth=$DIGEST_SMART_ON_BANDWIDTH"
|
||||
echo "$ICON_CERT Cert domains: ${CERT_MONITOR_DOMAINS[*]:-none}"
|
||||
echo "$ICON_BANDWIDTH Bandwidth warn: ${BANDWIDTH_WARN_GB}GB"
|
||||
echo "$ICON_GEAR Dry Run: $DRY_RUN"
|
||||
echo "━━━━━━━━━━━━━━━━━━━━━━━"
|
||||
exit 0
|
||||
fi
|
||||
|
||||
# ==============================================================================================
|
||||
# ── Profile gate — should we send today? ──────────────────────────────────────────────────────
|
||||
# ==============================================================================================
|
||||
SHOULD_SEND=false
|
||||
|
||||
case "$DIGEST_PROFILE" in
|
||||
always)
|
||||
SHOULD_SEND=true
|
||||
log "Profile: always — will send"
|
||||
;;
|
||||
weekly)
|
||||
TODAY_NAME=$(date '+%A')
|
||||
if [[ "$TODAY_NAME" == "$DIGEST_DAY" ]]; then
|
||||
SHOULD_SEND=true
|
||||
log "Profile: weekly — today is $DIGEST_DAY — will send"
|
||||
else
|
||||
echo "Profile: weekly — today is $TODAY_NAME, digest day is $DIGEST_DAY — no-op"
|
||||
exit 0
|
||||
fi
|
||||
;;
|
||||
smart)
|
||||
log "Profile: smart — evaluating findings before deciding"
|
||||
SHOULD_SEND=false
|
||||
;;
|
||||
*)
|
||||
warn "Unknown DIGEST_PROFILE: $DIGEST_PROFILE — defaulting to weekly"
|
||||
TODAY_NAME=$(date '+%A')
|
||||
[[ "$TODAY_NAME" == "$DIGEST_DAY" ]] && SHOULD_SEND=true
|
||||
;;
|
||||
esac
|
||||
|
||||
# ==============================================================================================
|
||||
# ── Data Gathering ────────────────────────────────────────────────────────────────────────────
|
||||
# ==============================================================================================
|
||||
FINDINGS=() # notable but not critical
|
||||
ISSUES=() # need attention
|
||||
DIGEST_LINES=() # full report lines
|
||||
|
||||
# ── fallback State ────────────────────────────────────────────────────────────────────────────
|
||||
log "Reading: fallback=$FALLBACK_STATE_FILE skip=$DOCKER_WATCHDOG_FAILED_FILE watchdog=$WATCHDOG_STATE_FILE sys=$SYS_WATCHDOG_STATE_FILE"
|
||||
if [[ -f "$FALLBACK_STATE_FILE" ]]; then
|
||||
FALLBACK_STATE=$(grep "^state=" "$FALLBACK_STATE_FILE" 2>/dev/null | cut -d= -f2)
|
||||
if [[ -n "$FALLBACK_STATE" ]]; then
|
||||
DIGEST_LINES+=("$ICON_FALLBACK Fallback: $FALLBACK_STATE")
|
||||
if [[ "$FALLBACK_STATE" != "NORMAL" ]]; then
|
||||
ISSUES+=("Fallback state: $FALLBACK_STATE")
|
||||
[[ "$DIGEST_SMART_ON_FALLBACK" == true ]] && SHOULD_SEND=true
|
||||
fi
|
||||
fi
|
||||
else
|
||||
DIGEST_LINES+=("$ICON_FALLBACK Fallback: state file not found")
|
||||
fi
|
||||
|
||||
# ── Container Skip List ───────────────────────────────────────────────────────────────────────
|
||||
if [[ -f "$DOCKER_WATCHDOG_FAILED_FILE" ]] && [[ -s "$DOCKER_WATCHDOG_FAILED_FILE" ]]; then
|
||||
SKIP_COUNT=$(wc -l < "$DOCKER_WATCHDOG_FAILED_FILE")
|
||||
SKIP_LIST=$(cat "$DOCKER_WATCHDOG_FAILED_FILE" | tr '\n' ' ')
|
||||
DIGEST_LINES+=("$ICON_NOT_RUNNING Skip list: $SKIP_COUNT containers — $SKIP_LIST")
|
||||
ISSUES+=("Containers on skip list (manual intervention needed): $SKIP_LIST")
|
||||
SHOULD_SEND=true
|
||||
else
|
||||
DIGEST_LINES+=("$ICON_RUNNING Skip list: empty ✅")
|
||||
fi
|
||||
|
||||
# ── Container Watchdog Strikes ────────────────────────────────────────────────────────────────
|
||||
if [[ -f "$WATCHDOG_STATE_FILE" ]]; then
|
||||
ACTIVE_STRIKES=$(grep -v ":0$" "$WATCHDOG_STATE_FILE" 2>/dev/null | grep -c ".")
|
||||
if [[ "$ACTIVE_STRIKES" -gt 0 ]]; then
|
||||
STRIKE_LIST=$(grep -v ":0$" "$WATCHDOG_STATE_FILE" 2>/dev/null | tr '\n' ' ')
|
||||
DIGEST_LINES+=("$ICON_WATCHDOG Container strikes: $ACTIVE_STRIKES active — $STRIKE_LIST")
|
||||
FINDINGS+=("Container watchdog: $ACTIVE_STRIKES active strikes")
|
||||
[[ "$DIGEST_SMART_ON_WATCHDOG" == true ]] && SHOULD_SEND=true
|
||||
else
|
||||
DIGEST_LINES+=("$ICON_WATCHDOG Container watchdog: no active strikes ✅")
|
||||
fi
|
||||
fi
|
||||
|
||||
# ── System Watchdog Strikes ───────────────────────────────────────────────────────────────────
|
||||
if [[ -f "$SYS_WATCHDOG_STATE_FILE" ]]; then
|
||||
SYS_STRIKES=$(grep -v ":0$" "$SYS_WATCHDOG_STATE_FILE" 2>/dev/null | grep -c ".")
|
||||
if [[ "$SYS_STRIKES" -gt 0 ]]; then
|
||||
SYS_STRIKE_LIST=$(grep -v ":0$" "$SYS_WATCHDOG_STATE_FILE" 2>/dev/null | tr '\n' ' ')
|
||||
DIGEST_LINES+=("$ICON_REBOOT_SMART System strikes: $SYS_STRIKES active — $SYS_STRIKE_LIST")
|
||||
FINDINGS+=("System watchdog: $SYS_STRIKES active strikes")
|
||||
[[ "$DIGEST_SMART_ON_WATCHDOG" == true ]] && SHOULD_SEND=true
|
||||
else
|
||||
DIGEST_LINES+=("$ICON_REBOOT_SMART System watchdog: no active strikes ✅")
|
||||
fi
|
||||
fi
|
||||
|
||||
# ── Transcode Ramdisk ─────────────────────────────────────────────────────────────────────────
|
||||
if mountpoint -q "$RAMDISK_PATH" 2>/dev/null; then
|
||||
RAMDISK_USED_KB=$(df "$RAMDISK_PATH" --output=used 2>/dev/null | tail -1 | tr -d ' ')
|
||||
RAMDISK_USED_GB=$(awk "BEGIN {printf \"%.2f\", ${RAMDISK_USED_KB:-0} / 1048576}")
|
||||
SYMLINK_TARGET=$(readlink "$TRANSCODE_LINK" 2>/dev/null || echo "unknown")
|
||||
DIGEST_LINES+=("$ICON_RAM Transcodes: ${RAMDISK_USED_GB}GB used → $SYMLINK_TARGET")
|
||||
|
||||
# Weekly transcode stats from TRANSCODE_DAILY_LOG
|
||||
if [[ -f "${TRANSCODE_DAILY_LOG:-}" ]]; then
|
||||
WEEK_CUTOFF=$(date -d "7 days ago" '+%Y-%m-%d')
|
||||
WEEK_PEAK=$(awk -F'|' -v c="$WEEK_CUTOFF" \
|
||||
'$1>=c{if($2>max)max=$2} END{printf "%.2f",max+0}' "$TRANSCODE_DAILY_LOG")
|
||||
WEEK_FLIPS=$(awk -F'|' -v c="$WEEK_CUTOFF" \
|
||||
'$1>=c{sum+=$3} END{print sum+0}' "$TRANSCODE_DAILY_LOG")
|
||||
WEEK_RAM=$(awk -F'|' -v c="$WEEK_CUTOFF" \
|
||||
'$1>=c{sum+=$4} END{print sum+0}' "$TRANSCODE_DAILY_LOG")
|
||||
WEEK_SSD=$(awk -F'|' -v c="$WEEK_CUTOFF" \
|
||||
'$1>=c{sum+=$5} END{print sum+0}' "$TRANSCODE_DAILY_LOG")
|
||||
WEEK_FILES=$(awk -F'|' -v c="$WEEK_CUTOFF" \
|
||||
'$1>=c{sum+=$6} END{print sum+0}' "$TRANSCODE_DAILY_LOG")
|
||||
|
||||
DIGEST_LINES+=("$ICON_RAM Transcode week: peak ${WEEK_PEAK}GB | flips: $WEEK_FLIPS | sessions: ${WEEK_RAM} ramdisk / ${WEEK_SSD} SSD | cleaned: ${WEEK_FILES} files")
|
||||
|
||||
PEAK_INT=$(printf "%.0f" "$WEEK_PEAK")
|
||||
WARN_INT=$(printf "%.0f" "${RAMDISK_WARN_GB:-6.8}")
|
||||
if [[ "$PEAK_INT" -ge "$WARN_INT" ]]; then
|
||||
DIGEST_LINES+=("$ICON_WARN Peak ${WEEK_PEAK}GB near threshold ${RAMDISK_WARN_GB}GB — consider increasing HOST*_RAMDISK_SIZE")
|
||||
FINDINGS+=("Transcode ramdisk near threshold: ${WEEK_PEAK}GB / ${RAMDISK_WARN_GB}GB")
|
||||
fi
|
||||
fi
|
||||
else
|
||||
DIGEST_LINES+=("$ICON_RAM Transcodes: ramdisk not mounted")
|
||||
ISSUES+=("Ramdisk not mounted at $RAMDISK_PATH")
|
||||
SHOULD_SEND=true
|
||||
fi
|
||||
|
||||
# ── Bandwidth ─────────────────────────────────────────────────────────────────────────────────
|
||||
# Updated for new log format: date|time|profile|duration|status|bytes|warn_flag
|
||||
if [[ -f "${BANDWIDTH_LOG:-}" ]] && [[ -s "$BANDWIDTH_LOG" ]]; then
|
||||
YESTERDAY=$(date -d "yesterday" '+%Y-%m-%d')
|
||||
YESTERDAY_BYTES=$(awk -F'|' -v d="$YESTERDAY" '$1==d{sum+=$6} END{print sum+0}' \
|
||||
"$BANDWIDTH_LOG")
|
||||
YESTERDAY_GB=$(awk "BEGIN {printf \"%.2f\", ${YESTERDAY_BYTES:-0} / 1073741824}")
|
||||
YESTERDAY_LARGE=$(awk -F'|' -v d="$YESTERDAY" '$1==d && $7=="LARGE"' \
|
||||
"$BANDWIDTH_LOG" | wc -l)
|
||||
|
||||
if [[ "${YESTERDAY_LARGE:-0}" -gt 0 ]]; then
|
||||
DIGEST_LINES+=("$ICON_BANDWIDTH Yesterday: ${YESTERDAY_GB}GB — $YESTERDAY_LARGE large transfer(s) exceeded ${BANDWIDTH_WARN_GB}GB")
|
||||
FINDINGS+=("High bandwidth day: ${YESTERDAY_GB}GB transferred")
|
||||
[[ "$DIGEST_SMART_ON_BANDWIDTH" == true ]] && SHOULD_SEND=true
|
||||
else
|
||||
DIGEST_LINES+=("$ICON_BANDWIDTH Yesterday: ${YESTERDAY_GB}GB")
|
||||
fi
|
||||
else
|
||||
DIGEST_LINES+=("$ICON_BANDWIDTH Bandwidth: no data yet")
|
||||
fi
|
||||
|
||||
# ── SSL Certificates ──────────────────────────────────────────────────────────────────────────
|
||||
if [[ ${#CERT_MONITOR_DOMAINS[@]} -gt 0 ]] && command -v openssl >/dev/null 2>&1; then
|
||||
CERT_ISSUES=()
|
||||
for domain in "${CERT_MONITOR_DOMAINS[@]}"; do
|
||||
[[ -z "$domain" ]] && continue
|
||||
expiry_str=$(echo | timeout "${CERT_TIMEOUT:-10}" openssl s_client \
|
||||
-connect "${domain}:443" -servername "$domain" \
|
||||
2>/dev/null | openssl x509 -noout -enddate 2>/dev/null | cut -d= -f2)
|
||||
if [[ -n "$expiry_str" ]]; then
|
||||
expiry_epoch=$(date -d "$expiry_str" +%s 2>/dev/null)
|
||||
days_remaining=$(( (expiry_epoch - $(date +%s)) / 86400 ))
|
||||
if [[ "$days_remaining" -le "${CERT_CRIT_DAYS:-7}" ]]; then
|
||||
CERT_ISSUES+=("$domain: ${days_remaining}d CRITICAL")
|
||||
SHOULD_SEND=true
|
||||
elif [[ "$days_remaining" -le "${CERT_WARN_DAYS:-30}" ]]; then
|
||||
CERT_ISSUES+=("$domain: ${days_remaining}d warning")
|
||||
[[ "$DIGEST_SMART_ON_CERT_WARN" == true ]] && SHOULD_SEND=true
|
||||
else
|
||||
log "$ICON_CERT $domain: ${days_remaining}d remaining ✅"
|
||||
fi
|
||||
fi
|
||||
done
|
||||
if [[ ${#CERT_ISSUES[@]} -gt 0 ]]; then
|
||||
DIGEST_LINES+=("$ICON_CERT Certificates: ${CERT_ISSUES[*]}")
|
||||
FINDINGS+=("Cert issues: ${CERT_ISSUES[*]}")
|
||||
else
|
||||
DIGEST_LINES+=("$ICON_CERT Certificates: all healthy ✅")
|
||||
fi
|
||||
fi
|
||||
|
||||
# ==============================================================================================
|
||||
# ── Smart profile — exit silently if nothing to report ────────────────────────────────────────
|
||||
# ==============================================================================================
|
||||
if [[ "$DIGEST_PROFILE" == "smart" && "$SHOULD_SEND" == false ]]; then
|
||||
echo "Profile: smart — no findings worth reporting — silent exit"
|
||||
exit 0
|
||||
fi
|
||||
|
||||
# ==============================================================================================
|
||||
# ━━━ Build and Send Digest ━━━
|
||||
# ==============================================================================================
|
||||
echo ""
|
||||
echo "━━━ $ICON_DIGEST Health Digest — $(date '+%Y-%m-%d %H:%M:%S') ━━━"
|
||||
echo "$ICON_HOST $MY_ID ($LOCAL_SERVER_NAME)"
|
||||
echo ""
|
||||
|
||||
for line in "${DIGEST_LINES[@]}"; do
|
||||
echo " $line"
|
||||
done
|
||||
|
||||
echo ""
|
||||
echo "━━━━━ $ICON_SUMMARY DIGEST SUMMARY ━━━━━"
|
||||
echo "$ICON_HOST Identity: $MY_ID ($LOCAL_SERVER_NAME)"
|
||||
echo "$ICON_DIGEST Profile: $DIGEST_PROFILE"
|
||||
echo "$ICON_ERROR Issues: ${#ISSUES[@]}"
|
||||
echo "$ICON_WARN Findings: ${#FINDINGS[@]}"
|
||||
echo "━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━"
|
||||
|
||||
# Build notification message
|
||||
NOTIFY_MSG="Health Digest — $MY_ID ($LOCAL_SERVER_NAME)"
|
||||
[[ ${#ISSUES[@]} -gt 0 ]] && NOTIFY_MSG+=" | Issues: ${ISSUES[*]}"
|
||||
[[ ${#FINDINGS[@]} -gt 0 ]] && NOTIFY_MSG+=" | Findings: ${FINDINGS[*]}"
|
||||
[[ ${#ISSUES[@]} -eq 0 && ${#FINDINGS[@]} -eq 0 ]] && NOTIFY_MSG+=" | All systems healthy"
|
||||
|
||||
NOTIFY_SEV="normal"
|
||||
[[ ${#ISSUES[@]} -gt 0 ]] && NOTIFY_SEV="warning"
|
||||
|
||||
if [[ "$DRY_RUN" == true ]]; then
|
||||
warn "DRY RUN — digest generated but not sent"
|
||||
elif [[ "$SHOULD_SEND" == true ]]; then
|
||||
notify "$NOTIFY_MSG" "Health Digest" "$NOTIFY_SEV"
|
||||
echo "Digest sent"
|
||||
fi
|
||||
+385
@@ -0,0 +1,385 @@
|
||||
#!/bin/bash
|
||||
# ==============================================================================================
|
||||
# ============================= Health Digest ==================================================
|
||||
# ==============================================================================================
|
||||
#
|
||||
# PURPOSE
|
||||
# ─────────────────────────────────────────────────────────────────────────────
|
||||
# Full ecosystem health aggregation from existing state files. Scheduled daily
|
||||
# (8am). DIGEST_PROFILE controls when notifications actually send — the cron
|
||||
# schedule never changes, only the profile in master.conf.
|
||||
#
|
||||
# Reads state files from across the system (watchdog strikes, fallback state,
|
||||
# skip list, bandwidth history, transcode stats, cert status) and compiles them
|
||||
# into a single digest. Reads only — writes nothing, changes nothing.
|
||||
#
|
||||
# ==============================================================================================
|
||||
# OPERATIONAL MODEL
|
||||
# ==============================================================================================
|
||||
#
|
||||
# Three profiles — switch by changing DIGEST_PROFILE in master.conf:
|
||||
#
|
||||
# always — sends every run regardless of findings
|
||||
# Use: daily digest of everything, even when healthy
|
||||
#
|
||||
# smart — sends only when something worth reporting was found
|
||||
# Stays silent on clean days. DIGEST_SMART_ON_* toggles control
|
||||
# what triggers a send — all independently configurable.
|
||||
#
|
||||
# weekly — sends once per week on DIGEST_DAY (default Sunday), silent all other days
|
||||
# Use: one weekly summary without daily noise
|
||||
#
|
||||
# Data sources (reads only):
|
||||
# FALLBACK_STATE_FILE — current fallback state
|
||||
# DOCKER_WATCHDOG_FAILED_FILE — container skip list (manual intervention needed)
|
||||
# WATCHDOG_STATE_FILE — active container watchdog strikes
|
||||
# SYS_WATCHDOG_STATE_FILE — active system watchdog strikes
|
||||
# BANDWIDTH_LOG — yesterday's transfer totals
|
||||
# TRANSCODE_DAILY_LOG — weekly transcode statistics
|
||||
# CERT_MONITOR_DOMAINS — live SSL cert check via openssl
|
||||
# RAMDISK_PATH / TRANSCODE_LINK — current transcode location and usage
|
||||
#
|
||||
# ==============================================================================================
|
||||
# OPERATIONAL SAFEGUARDS
|
||||
# ==============================================================================================
|
||||
#
|
||||
# Single Instance Lock
|
||||
# acquire_lock prevents duplicate reports — report generation takes time.
|
||||
#
|
||||
# Per-Host Variables
|
||||
# detect_hosts() aliases CERT_MONITOR_DOMAINS, RAMDISK_WARN_GB, RAMDISK_SIZE,
|
||||
# RAMDISK_LOW_GB, and all other host-specific vars used in the report.
|
||||
#
|
||||
# Per-Section Guards
|
||||
# Each data source section checks whether its state file exists before reading.
|
||||
# A missing state file is skipped cleanly — it does not abort the report.
|
||||
#
|
||||
# Silent Smart Profile
|
||||
# smart profile produces no output and no notification when nothing worth
|
||||
# reporting is found.
|
||||
#
|
||||
# Notifications Validated
|
||||
# platform_require_cmd confirms notify and openssl are present before use.
|
||||
#
|
||||
# ==============================================================================================
|
||||
# CONFIGURATION
|
||||
# ==============================================================================================
|
||||
#
|
||||
# master.conf
|
||||
#
|
||||
# DIGEST_PROFILE
|
||||
# Notification frequency: always | smart | weekly. (default: weekly)
|
||||
#
|
||||
# DIGEST_DAY
|
||||
# Day name for weekly profile — must match `date +%A` output. (default: Sunday)
|
||||
#
|
||||
# DIGEST_SMART_ON_WATCHDOG
|
||||
# Send smart profile notification if any active watchdog strikes. (default: true)
|
||||
#
|
||||
# DIGEST_SMART_ON_FALLBACK
|
||||
# Send smart profile notification if fallback state is not NORMAL. (default: true)
|
||||
#
|
||||
# DIGEST_SMART_ON_CERT_WARN
|
||||
# Send smart profile notification if any cert is within CERT_WARN_DAYS. (default: true)
|
||||
#
|
||||
# DIGEST_SMART_ON_BANDWIDTH
|
||||
# Send smart profile notification if any transfer exceeded BANDWIDTH_WARN_GB. (default: true)
|
||||
#
|
||||
# CERT_WARN_DAYS / CERT_CRIT_DAYS / CERT_TIMEOUT
|
||||
# Cert check thresholds — shared with cert_monitor.sh.
|
||||
#
|
||||
# BANDWIDTH_WARN_GB
|
||||
# High-transfer threshold — shared with bandwidth_monitor.sh.
|
||||
#
|
||||
# TRANSCODE_DAILY_LOG
|
||||
# Path to the transcode statistics log.
|
||||
#
|
||||
# ==============================================================================================
|
||||
# RUNTIME MODES
|
||||
# ==============================================================================================
|
||||
#
|
||||
# weekly_health_digest.sh
|
||||
# Run digest. DIGEST_PROFILE determines whether a notification is sent.
|
||||
#
|
||||
# weekly_health_digest.sh --dry-run
|
||||
# Generate and display digest output. No notification sent regardless of profile.
|
||||
#
|
||||
# weekly_health_digest.sh --status
|
||||
# Show profile, day, and smart trigger configuration. Then exit.
|
||||
#
|
||||
# weekly_health_digest.sh --log
|
||||
# Verbose per-section output during digest generation.
|
||||
#
|
||||
# ==============================================================================================
|
||||
|
||||
SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
|
||||
|
||||
source "$SCRIPT_DIR/../load_config.sh"
|
||||
|
||||
parse_args "$@"
|
||||
|
||||
# ==============================================================================================
|
||||
# ━━━ Setup ━━━
|
||||
# ==============================================================================================
|
||||
if [[ "$EUID" -ne 0 ]]; then
|
||||
error "Must be run as root"
|
||||
exit 1
|
||||
fi
|
||||
|
||||
platform_require_cmd \
|
||||
"/usr/local/emhttp/plugins/dynamix/scripts/notify" \
|
||||
"" "" \
|
||||
"unRAID notify script" || warn "unRAID notify script not found — native notifications disabled"
|
||||
|
||||
platform_require_cmd \
|
||||
"$(command -v openssl 2>/dev/null || echo /usr/bin/openssl)" \
|
||||
"version" "OpenSSL" \
|
||||
"openssl" || warn "openssl not found — SSL cert checks will be skipped"
|
||||
|
||||
acquire_lock
|
||||
|
||||
# detect_hosts() sets MY_ID and aliases all host-specific vars used in this report
|
||||
detect_hosts
|
||||
|
||||
log "$ICON_GEAR Config: profile=${DIGEST_PROFILE} day=${DIGEST_DAY}"
|
||||
log "$ICON_GEAR Smart triggers: watchdog=${DIGEST_SMART_ON_WATCHDOG} fallback=${DIGEST_SMART_ON_FALLBACK} cert=${DIGEST_SMART_ON_CERT_WARN} bandwidth=${DIGEST_SMART_ON_BANDWIDTH}"
|
||||
|
||||
[[ "$DRY_RUN" == true ]] && warn "DRY RUN — report generated but no notification sent"
|
||||
|
||||
# ==============================================================================================
|
||||
# ━━━ Status ━━━
|
||||
# ==============================================================================================
|
||||
if [[ "$SHOW_STATUS" == true ]]; then
|
||||
echo ""
|
||||
echo "━━━━━ $ICON_SUMMARY STATUS ━━━━━"
|
||||
echo "$ICON_HOST Identity: $MY_ID ($LOCAL_SERVER_NAME)"
|
||||
echo "$ICON_DIGEST Profile: $DIGEST_PROFILE"
|
||||
echo "$ICON_DIGEST Digest day: $DIGEST_DAY"
|
||||
echo "$ICON_DIGEST Smart triggers: watchdog=$DIGEST_SMART_ON_WATCHDOG fallback=$DIGEST_SMART_ON_FALLBACK cert=$DIGEST_SMART_ON_CERT_WARN bandwidth=$DIGEST_SMART_ON_BANDWIDTH"
|
||||
echo "$ICON_CERT Cert domains: ${CERT_MONITOR_DOMAINS[*]:-none}"
|
||||
echo "$ICON_BANDWIDTH Bandwidth warn: ${BANDWIDTH_WARN_GB}GB"
|
||||
echo "$ICON_GEAR Dry Run: $DRY_RUN"
|
||||
echo "━━━━━━━━━━━━━━━━━━━━━━━"
|
||||
exit 0
|
||||
fi
|
||||
|
||||
# ==============================================================================================
|
||||
# ── Profile gate — should we send today? ──────────────────────────────────────────────────────
|
||||
# ==============================================================================================
|
||||
SHOULD_SEND=false
|
||||
|
||||
case "$DIGEST_PROFILE" in
|
||||
always)
|
||||
SHOULD_SEND=true
|
||||
log "Profile: always — will send"
|
||||
;;
|
||||
weekly)
|
||||
TODAY_NAME=$(date '+%A')
|
||||
if [[ "$TODAY_NAME" == "$DIGEST_DAY" ]]; then
|
||||
SHOULD_SEND=true
|
||||
log "Profile: weekly — today is $DIGEST_DAY — will send"
|
||||
else
|
||||
echo "Profile: weekly — today is $TODAY_NAME, digest day is $DIGEST_DAY — no-op"
|
||||
exit 0
|
||||
fi
|
||||
;;
|
||||
smart)
|
||||
log "Profile: smart — evaluating findings before deciding"
|
||||
SHOULD_SEND=false
|
||||
;;
|
||||
*)
|
||||
warn "Unknown DIGEST_PROFILE: $DIGEST_PROFILE — defaulting to weekly"
|
||||
TODAY_NAME=$(date '+%A')
|
||||
[[ "$TODAY_NAME" == "$DIGEST_DAY" ]] && SHOULD_SEND=true
|
||||
;;
|
||||
esac
|
||||
|
||||
# ==============================================================================================
|
||||
# ── Data Gathering ────────────────────────────────────────────────────────────────────────────
|
||||
# ==============================================================================================
|
||||
FINDINGS=() # notable but not critical
|
||||
ISSUES=() # need attention
|
||||
DIGEST_LINES=() # full report lines
|
||||
|
||||
# ── fallback State ────────────────────────────────────────────────────────────────────────────
|
||||
log "Reading: fallback=$FALLBACK_STATE_FILE skip=$DOCKER_WATCHDOG_FAILED_FILE watchdog=$WATCHDOG_STATE_FILE sys=$SYS_WATCHDOG_STATE_FILE"
|
||||
if [[ -f "$FALLBACK_STATE_FILE" ]]; then
|
||||
FALLBACK_STATE=$(grep "^state=" "$FALLBACK_STATE_FILE" 2>/dev/null | cut -d= -f2)
|
||||
if [[ -n "$FALLBACK_STATE" ]]; then
|
||||
DIGEST_LINES+=("$ICON_FALLBACK Fallback: $FALLBACK_STATE")
|
||||
if [[ "$FALLBACK_STATE" != "NORMAL" ]]; then
|
||||
ISSUES+=("Fallback state: $FALLBACK_STATE")
|
||||
[[ "$DIGEST_SMART_ON_FALLBACK" == true ]] && SHOULD_SEND=true
|
||||
fi
|
||||
fi
|
||||
else
|
||||
DIGEST_LINES+=("$ICON_FALLBACK Fallback: state file not found")
|
||||
fi
|
||||
|
||||
# ── Container Skip List ───────────────────────────────────────────────────────────────────────
|
||||
if [[ -f "$DOCKER_WATCHDOG_FAILED_FILE" ]] && [[ -s "$DOCKER_WATCHDOG_FAILED_FILE" ]]; then
|
||||
SKIP_COUNT=$(wc -l < "$DOCKER_WATCHDOG_FAILED_FILE")
|
||||
SKIP_LIST=$(cat "$DOCKER_WATCHDOG_FAILED_FILE" | tr '\n' ' ')
|
||||
DIGEST_LINES+=("$ICON_NOT_RUNNING Skip list: $SKIP_COUNT containers — $SKIP_LIST")
|
||||
ISSUES+=("Containers on skip list (manual intervention needed): $SKIP_LIST")
|
||||
SHOULD_SEND=true
|
||||
else
|
||||
DIGEST_LINES+=("$ICON_RUNNING Skip list: empty ✅")
|
||||
fi
|
||||
|
||||
# ── Container Watchdog Strikes ────────────────────────────────────────────────────────────────
|
||||
if [[ -f "$WATCHDOG_STATE_FILE" ]]; then
|
||||
ACTIVE_STRIKES=$(grep -v ":0$" "$WATCHDOG_STATE_FILE" 2>/dev/null | grep -c ".")
|
||||
if [[ "$ACTIVE_STRIKES" -gt 0 ]]; then
|
||||
STRIKE_LIST=$(grep -v ":0$" "$WATCHDOG_STATE_FILE" 2>/dev/null | tr '\n' ' ')
|
||||
DIGEST_LINES+=("$ICON_WATCHDOG Container strikes: $ACTIVE_STRIKES active — $STRIKE_LIST")
|
||||
FINDINGS+=("Container watchdog: $ACTIVE_STRIKES active strikes")
|
||||
[[ "$DIGEST_SMART_ON_WATCHDOG" == true ]] && SHOULD_SEND=true
|
||||
else
|
||||
DIGEST_LINES+=("$ICON_WATCHDOG Container watchdog: no active strikes ✅")
|
||||
fi
|
||||
fi
|
||||
|
||||
# ── System Watchdog Strikes ───────────────────────────────────────────────────────────────────
|
||||
if [[ -f "$SYS_WATCHDOG_STATE_FILE" ]]; then
|
||||
SYS_STRIKES=$(grep "^[^=]*:" "$SYS_WATCHDOG_STATE_FILE" 2>/dev/null | grep -cv -E "(:0$|:false$)")
|
||||
if [[ "$SYS_STRIKES" -gt 0 ]]; then
|
||||
SYS_STRIKE_LIST=$(grep "^[^=]*:" "$SYS_WATCHDOG_STATE_FILE" 2>/dev/null | grep -v -E "(:0$|:false$)" | tr '\n' ' ')
|
||||
DIGEST_LINES+=("$ICON_REBOOT_SMART System strikes: $SYS_STRIKES active — $SYS_STRIKE_LIST")
|
||||
FINDINGS+=("System watchdog: $SYS_STRIKES active strikes")
|
||||
[[ "$DIGEST_SMART_ON_WATCHDOG" == true ]] && SHOULD_SEND=true
|
||||
else
|
||||
DIGEST_LINES+=("$ICON_REBOOT_SMART System watchdog: no active strikes ✅")
|
||||
fi
|
||||
fi
|
||||
|
||||
# ── Transcode Ramdisk ─────────────────────────────────────────────────────────────────────────
|
||||
if mountpoint -q "$RAMDISK_PATH" 2>/dev/null; then
|
||||
RAMDISK_USED_KB=$(df "$RAMDISK_PATH" --output=used 2>/dev/null | tail -1 | tr -d ' ')
|
||||
RAMDISK_USED_GB=$(awk "BEGIN {printf \"%.2f\", ${RAMDISK_USED_KB:-0} / 1048576}")
|
||||
SYMLINK_TARGET=$(readlink "$TRANSCODE_LINK" 2>/dev/null || echo "unknown")
|
||||
DIGEST_LINES+=("$ICON_RAM Transcodes: ${RAMDISK_USED_GB}GB used → $SYMLINK_TARGET")
|
||||
|
||||
# Weekly transcode stats from TRANSCODE_DAILY_LOG
|
||||
if [[ -f "${TRANSCODE_DAILY_LOG:-}" ]]; then
|
||||
WEEK_CUTOFF=$(date -d "7 days ago" '+%Y-%m-%d')
|
||||
WEEK_PEAK=$(awk -F'|' -v c="$WEEK_CUTOFF" \
|
||||
'$1>=c{if($2>max)max=$2} END{printf "%.2f",max+0}' "$TRANSCODE_DAILY_LOG")
|
||||
WEEK_FLIPS=$(awk -F'|' -v c="$WEEK_CUTOFF" \
|
||||
'$1>=c{sum+=$3} END{print sum+0}' "$TRANSCODE_DAILY_LOG")
|
||||
WEEK_RAM=$(awk -F'|' -v c="$WEEK_CUTOFF" \
|
||||
'$1>=c{sum+=$4} END{print sum+0}' "$TRANSCODE_DAILY_LOG")
|
||||
WEEK_SSD=$(awk -F'|' -v c="$WEEK_CUTOFF" \
|
||||
'$1>=c{sum+=$5} END{print sum+0}' "$TRANSCODE_DAILY_LOG")
|
||||
WEEK_FILES=$(awk -F'|' -v c="$WEEK_CUTOFF" \
|
||||
'$1>=c{sum+=$6} END{print sum+0}' "$TRANSCODE_DAILY_LOG")
|
||||
|
||||
DIGEST_LINES+=("$ICON_RAM Transcode week: peak ${WEEK_PEAK}GB | flips: $WEEK_FLIPS | sessions: ${WEEK_RAM} ramdisk / ${WEEK_SSD} SSD | cleaned: ${WEEK_FILES} files")
|
||||
|
||||
PEAK_INT=$(printf "%.0f" "$WEEK_PEAK")
|
||||
WARN_INT=$(printf "%.0f" "${RAMDISK_WARN_GB:-6.8}")
|
||||
if [[ "$PEAK_INT" -ge "$WARN_INT" ]]; then
|
||||
DIGEST_LINES+=("$ICON_WARN Peak ${WEEK_PEAK}GB near threshold ${RAMDISK_WARN_GB}GB — consider increasing HOST*_RAMDISK_SIZE")
|
||||
FINDINGS+=("Transcode ramdisk near threshold: ${WEEK_PEAK}GB / ${RAMDISK_WARN_GB}GB")
|
||||
fi
|
||||
fi
|
||||
else
|
||||
DIGEST_LINES+=("$ICON_RAM Transcodes: ramdisk not mounted")
|
||||
ISSUES+=("Ramdisk not mounted at $RAMDISK_PATH")
|
||||
SHOULD_SEND=true
|
||||
fi
|
||||
|
||||
# ── Bandwidth ─────────────────────────────────────────────────────────────────────────────────
|
||||
# Updated for new log format: date|time|profile|duration|status|bytes|warn_flag
|
||||
if [[ -f "${BANDWIDTH_LOG:-}" ]] && [[ -s "$BANDWIDTH_LOG" ]]; then
|
||||
YESTERDAY=$(date -d "yesterday" '+%Y-%m-%d')
|
||||
YESTERDAY_BYTES=$(awk -F'|' -v d="$YESTERDAY" '$1==d{sum+=$6} END{print sum+0}' \
|
||||
"$BANDWIDTH_LOG")
|
||||
YESTERDAY_GB=$(awk "BEGIN {printf \"%.2f\", ${YESTERDAY_BYTES:-0} / 1073741824}")
|
||||
YESTERDAY_LARGE=$(awk -F'|' -v d="$YESTERDAY" '$1==d && $7=="LARGE"' \
|
||||
"$BANDWIDTH_LOG" | wc -l)
|
||||
|
||||
if [[ "${YESTERDAY_LARGE:-0}" -gt 0 ]]; then
|
||||
DIGEST_LINES+=("$ICON_BANDWIDTH Yesterday: ${YESTERDAY_GB}GB — $YESTERDAY_LARGE large transfer(s) exceeded ${BANDWIDTH_WARN_GB}GB")
|
||||
FINDINGS+=("High bandwidth day: ${YESTERDAY_GB}GB transferred")
|
||||
[[ "$DIGEST_SMART_ON_BANDWIDTH" == true ]] && SHOULD_SEND=true
|
||||
else
|
||||
DIGEST_LINES+=("$ICON_BANDWIDTH Yesterday: ${YESTERDAY_GB}GB")
|
||||
fi
|
||||
else
|
||||
DIGEST_LINES+=("$ICON_BANDWIDTH Bandwidth: no data yet")
|
||||
fi
|
||||
|
||||
# ── SSL Certificates ──────────────────────────────────────────────────────────────────────────
|
||||
if [[ ${#CERT_MONITOR_DOMAINS[@]} -gt 0 ]] && command -v openssl >/dev/null 2>&1; then
|
||||
CERT_ISSUES=()
|
||||
for domain in "${CERT_MONITOR_DOMAINS[@]}"; do
|
||||
[[ -z "$domain" ]] && continue
|
||||
expiry_str=$(echo | timeout "${CERT_TIMEOUT:-10}" openssl s_client \
|
||||
-connect "${domain}:443" -servername "$domain" \
|
||||
2>/dev/null | openssl x509 -noout -enddate 2>/dev/null | cut -d= -f2)
|
||||
if [[ -n "$expiry_str" ]]; then
|
||||
expiry_epoch=$(date -d "$expiry_str" +%s 2>/dev/null)
|
||||
days_remaining=$(( (expiry_epoch - $(date +%s)) / 86400 ))
|
||||
if [[ "$days_remaining" -le "${CERT_CRIT_DAYS:-7}" ]]; then
|
||||
CERT_ISSUES+=("$domain: ${days_remaining}d CRITICAL")
|
||||
SHOULD_SEND=true
|
||||
elif [[ "$days_remaining" -le "${CERT_WARN_DAYS:-30}" ]]; then
|
||||
CERT_ISSUES+=("$domain: ${days_remaining}d warning")
|
||||
[[ "$DIGEST_SMART_ON_CERT_WARN" == true ]] && SHOULD_SEND=true
|
||||
else
|
||||
log "$ICON_CERT $domain: ${days_remaining}d remaining ✅"
|
||||
fi
|
||||
fi
|
||||
done
|
||||
if [[ ${#CERT_ISSUES[@]} -gt 0 ]]; then
|
||||
DIGEST_LINES+=("$ICON_CERT Certificates: ${CERT_ISSUES[*]}")
|
||||
FINDINGS+=("Cert issues: ${CERT_ISSUES[*]}")
|
||||
else
|
||||
DIGEST_LINES+=("$ICON_CERT Certificates: all healthy ✅")
|
||||
fi
|
||||
fi
|
||||
|
||||
# ==============================================================================================
|
||||
# ── Smart profile — exit silently if nothing to report ────────────────────────────────────────
|
||||
# ==============================================================================================
|
||||
if [[ "$DIGEST_PROFILE" == "smart" && "$SHOULD_SEND" == false ]]; then
|
||||
echo "Profile: smart — no findings worth reporting — silent exit"
|
||||
exit 0
|
||||
fi
|
||||
|
||||
# ==============================================================================================
|
||||
# ━━━ Build and Send Digest ━━━
|
||||
# ==============================================================================================
|
||||
echo ""
|
||||
echo "━━━ $ICON_DIGEST Health Digest — $(date '+%Y-%m-%d %H:%M:%S') ━━━"
|
||||
echo "$ICON_HOST $MY_ID ($LOCAL_SERVER_NAME)"
|
||||
echo ""
|
||||
|
||||
for line in "${DIGEST_LINES[@]}"; do
|
||||
echo " $line"
|
||||
done
|
||||
|
||||
echo ""
|
||||
echo "━━━━━ $ICON_SUMMARY DIGEST SUMMARY ━━━━━"
|
||||
echo "$ICON_HOST Identity: $MY_ID ($LOCAL_SERVER_NAME)"
|
||||
echo "$ICON_DIGEST Profile: $DIGEST_PROFILE"
|
||||
echo "$ICON_ERROR Issues: ${#ISSUES[@]}"
|
||||
echo "$ICON_WARN Findings: ${#FINDINGS[@]}"
|
||||
echo "━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━"
|
||||
|
||||
# Build notification message
|
||||
NOTIFY_MSG="Health Digest — $MY_ID ($LOCAL_SERVER_NAME)"
|
||||
[[ ${#ISSUES[@]} -gt 0 ]] && NOTIFY_MSG+=" | Issues: ${ISSUES[*]}"
|
||||
[[ ${#FINDINGS[@]} -gt 0 ]] && NOTIFY_MSG+=" | Findings: ${FINDINGS[*]}"
|
||||
[[ ${#ISSUES[@]} -eq 0 && ${#FINDINGS[@]} -eq 0 ]] && NOTIFY_MSG+=" | All systems healthy"
|
||||
|
||||
NOTIFY_SEV="normal"
|
||||
[[ ${#ISSUES[@]} -gt 0 ]] && NOTIFY_SEV="warning"
|
||||
|
||||
if [[ "$DRY_RUN" == true ]]; then
|
||||
warn "DRY RUN — digest generated but not sent"
|
||||
elif [[ "$SHOULD_SEND" == true ]]; then
|
||||
notify "$NOTIFY_MSG" "Health Digest" "$NOTIFY_SEV"
|
||||
echo "Digest sent"
|
||||
fi
|
||||
Reference in New Issue
Block a user