feat: slskd reconnect guard in downloaders_reset, mass v2 sync
- downloaders_reset: connection check block before slskd API sections; triggers PUT /api/v0/server reconnect if disconnected, polls 60s, gates Stuck Searches and Dead Transfer Records on SLSKD_CONNECTED - Sync all modified/new/deleted files from v2 refactor across Docker_Essentials, Media, Monitors, Partnership, Rsync, Tools, Transcodes, unRAID_Essentials, common.sh, master confs, and new Manual/README docs
This commit is contained in:
Gmer4Lfe
@@ -0,0 +1,600 @@
|
||||
# ━━━━━ TOOLS — Manual ━━━━━
|
||||
|
||||
Configuration reference, usage procedures, and field guides for every script
|
||||
in `Tools/`. Run any script with `--status` first — it shows current state before
|
||||
making any changes.
|
||||
|
||||
---
|
||||
|
||||
## ━━━ CONTENTS ━━━
|
||||
|
||||
- [failover_state_reset.sh](#failover_state_resetsh)
|
||||
- [watchdog_skip_list_manager.sh](#watchdog_skip_list_managersh)
|
||||
- [bulk_permissions_repair.sh](#bulk_permissions_repairsh)
|
||||
- [container_data_export.sh](#container_data_exportsh)
|
||||
- [emby_database_repair.sh](#emby_database_repairsh)
|
||||
- [zfs_pool_scrub.sh](#zfs_pool_scrubsh)
|
||||
- [recreate_shares.sh](#recreate_sharessh)
|
||||
- [continuous_scripts_status.sh](#continuous_scripts_statussh)
|
||||
- [claude_startup.sh](#claude_startupsh)
|
||||
- [Adding a New Tool](#adding-a-new-tool)
|
||||
|
||||
---
|
||||
|
||||
## failover_state_reset.sh
|
||||
|
||||
Resets the fallback state file to NORMAL and clears all tier flags. State file only —
|
||||
does NOT start or stop any containers.
|
||||
|
||||
### When to Use
|
||||
|
||||
```
|
||||
After failover_test.sh didn't complete cleanly
|
||||
→ state left in FALLBACK but containers are actually back to normal
|
||||
|
||||
After a failed handback
|
||||
→ state shows FALLBACK but remote is back up and containers are split
|
||||
|
||||
After killing fallback.sh directly (not via User Scripts Abort)
|
||||
→ state is unknown, cycle was interrupted mid-operation
|
||||
|
||||
After a dev/debug session
|
||||
→ state left in a non-NORMAL state from testing
|
||||
```
|
||||
|
||||
### Verify Before Resetting
|
||||
|
||||
Run `--status` first and check each of these before writing:
|
||||
|
||||
```bash
|
||||
# Right containers on right server?
|
||||
continuous_scripts_status.sh # shows failover current state
|
||||
|
||||
# DDNS pointing correctly?
|
||||
nslookup Gmer4Lfe.com # confirm it resolves to the right IP
|
||||
|
||||
# fallback.sh not running?
|
||||
pgrep -f "fallback.sh" # empty output = not running
|
||||
|
||||
# Both servers Tailscale connected?
|
||||
tailscale status # both hosts should show active
|
||||
```
|
||||
|
||||
Resetting during an actual failover causes fallback.sh to think everything is normal
|
||||
and stop covering the remote — services go offline until the next detection cycle.
|
||||
|
||||
### Usage
|
||||
|
||||
```bash
|
||||
fallback_state_reset.sh --status # show current state file — always check first
|
||||
fallback_state_reset.sh --dry-run # show what would be written, no write
|
||||
fallback_state_reset.sh # interactive reset — prompts for YES to confirm
|
||||
fallback_state_reset.sh --force # non-interactive — for scripts, no terminal
|
||||
```
|
||||
|
||||
### What Gets Written
|
||||
|
||||
```bash
|
||||
# New state file after reset:
|
||||
state=NORMAL
|
||||
fallback_start=0
|
||||
handback_strikes=0
|
||||
tier2_started=false
|
||||
tier3_started=false
|
||||
tier4_started=false
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## watchdog_skip_list_manager.sh
|
||||
|
||||
View and manage the persistent container skip list used by `docker_watchdog.sh`.
|
||||
|
||||
### When to Use
|
||||
|
||||
```
|
||||
docker_watchdog.sh restarts the same container N times within the rolling window
|
||||
→ container added to skip list on /boot/config/
|
||||
→ critical notification sent
|
||||
→ watchdog stops touching it entirely
|
||||
|
||||
You fix the underlying problem (database, config, dependencies).
|
||||
You need to clear the container from the skip list so monitoring resumes.
|
||||
```
|
||||
|
||||
### Recovery Workflow
|
||||
|
||||
```bash
|
||||
# 1. Understand the situation — always start here:
|
||||
watchdog_skip_list_manager.sh --status
|
||||
# Shows: skip list contents, which are running vs. stopped, restart history
|
||||
|
||||
# 2. Fix the underlying problem first
|
||||
# Check logs: docker logs ContainerName --tail 100
|
||||
# Check disk: df -h /mnt/user
|
||||
# Check db: docker exec ContainerName sqlite3 /path/to.db ".tables"
|
||||
|
||||
# 3. Clear from skip list + restart history:
|
||||
watchdog_skip_list_manager.sh --clear ContainerName
|
||||
|
||||
# 4. Start the container manually — confirm your fix worked:
|
||||
docker start ContainerName
|
||||
|
||||
# 5. Watchdog resumes normal monitoring on next cycle — no further action needed
|
||||
```
|
||||
|
||||
### State Files Managed
|
||||
|
||||
```bash
|
||||
# Both live on /boot/config — survive reboots intentionally.
|
||||
# A container that was skip-listed before a reboot is still broken after it.
|
||||
|
||||
$SYS_WATCHDOG_FAILED_FILE # persistent skip list
|
||||
$WATCHDOG_CONTAINER_RESTART_LOG # restart loop tracking
|
||||
```
|
||||
|
||||
### Configuration (master.conf)
|
||||
|
||||
```bash
|
||||
WATCHDOG_CONTAINER_RESTART_LIMIT=3 # restarts before skip-listing
|
||||
WATCHDOG_CONTAINER_RESTART_WINDOW=1 # rolling window in hours
|
||||
```
|
||||
|
||||
### Usage
|
||||
|
||||
```bash
|
||||
watchdog_skip_list_manager.sh # show status (default)
|
||||
watchdog_skip_list_manager.sh --status # explicit status
|
||||
watchdog_skip_list_manager.sh --clear ContainerName # clear specific + restart history
|
||||
watchdog_skip_list_manager.sh --clear ContainerName --force # no confirmation prompt
|
||||
watchdog_skip_list_manager.sh --clear-all # clear everything
|
||||
watchdog_skip_list_manager.sh --clear-all --force # non-interactive
|
||||
watchdog_skip_list_manager.sh --dry-run # preview any clear action
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## bulk_permissions_repair.sh
|
||||
|
||||
Applies correct ownership and permissions to specific paths. Faster than running
|
||||
`media_shares_permissions.sh` which processes every configured share — use this when
|
||||
you know exactly what needs fixing and don't want to wait for a full library walk.
|
||||
|
||||
### When to Use
|
||||
|
||||
```
|
||||
Admin copy left root:root files — scp, cp, direct file transfer
|
||||
New share needs permissions now — can't wait for nightly run
|
||||
Container wrote as root — before PUID/PGID was fixed
|
||||
Specific directory has wrong perms — targeted fix, not a full library walk
|
||||
```
|
||||
|
||||
Use the full `media_shares_permissions.sh` instead for:
|
||||
- Regular nightly maintenance (already scheduled in daily_sync_maintenance.sh)
|
||||
- After confirming a container's PUID/PGID is now correct
|
||||
- Initial permissions setup on a new server
|
||||
|
||||
### Diagnosing High Wrong-Owner Counts
|
||||
|
||||
The script counts files with wrong ownership before applying the fix. A high count
|
||||
on a share that was recently written means a container has wrong PUID/PGID.
|
||||
|
||||
```bash
|
||||
# Fix: add to the container's Docker template:
|
||||
PUID=99
|
||||
PGID=100
|
||||
|
||||
# Common culprits writing as root:
|
||||
# SABnzbd, qBittorrent, slskd — check each one's Docker env vars
|
||||
```
|
||||
|
||||
### Configuration (master.conf)
|
||||
|
||||
```bash
|
||||
PERMISSIONS_OWNER="nobody:users" # matches PUID=99 PGID=100
|
||||
PERMISSIONS_DIR_MODE="755" # directories — enter, list, no world-write
|
||||
PERMISSIONS_FILE_MODE="664" # files — owner+group rw, others read-only
|
||||
```
|
||||
|
||||
### Usage
|
||||
|
||||
```bash
|
||||
# Single path:
|
||||
bulk_permissions_repair.sh /mnt/user/Movies
|
||||
|
||||
# Multiple paths — all corrected in one run:
|
||||
bulk_permissions_repair.sh /mnt/user/Movies /mnt/user/Tv_Shows /mnt/user/Music
|
||||
|
||||
# Dry run first — shows count of files with wrong ownership per path:
|
||||
bulk_permissions_repair.sh /mnt/user/Movies --dry-run
|
||||
|
||||
# Verbose — show each corrected file:
|
||||
bulk_permissions_repair.sh /mnt/user/Movies --log
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## container_data_export.sh
|
||||
|
||||
Exports a container's appdata directory to a compressed tar archive. Stops the
|
||||
container first for a clean consistent backup, verifies the archive after creation,
|
||||
then restarts the container.
|
||||
|
||||
### When to Use
|
||||
|
||||
```
|
||||
Before major container updates — especially "database migration — no rollback" changelogs
|
||||
Before pool migrations — clean backup before moving appdata to a new pool
|
||||
Before removing a container from the stack — archive its data before deletion
|
||||
Manual point-in-time backup before risky config changes
|
||||
```
|
||||
|
||||
### Export Sequence
|
||||
|
||||
```
|
||||
1. Space check
|
||||
Estimates required space from appdata size × 1.1
|
||||
Aborts if output directory doesn't have enough free space
|
||||
Container is NOT stopped until the space check passes
|
||||
|
||||
2. Stop container cleanly
|
||||
docker stop ContainerName — graceful shutdown
|
||||
|
||||
3. Create archive
|
||||
tar -czf ContainerName_YYYY-MM-DD_HH-MM.tar.gz /path/to/appdata
|
||||
|
||||
4. Verify archive integrity
|
||||
tar --test-file archive.tar.gz — confirms archive is valid and complete
|
||||
If verification fails → restart container anyway, report error
|
||||
|
||||
5. Restart container
|
||||
docker start ContainerName — always happens, even if archiving failed
|
||||
```
|
||||
|
||||
### Usage
|
||||
|
||||
```bash
|
||||
# Syntax: container_data_export.sh ContainerName AppDataPath OutputDir
|
||||
|
||||
# Emby backup:
|
||||
container_data_export.sh \
|
||||
Emby \
|
||||
/mnt/media-servers/Media_Server/Emby \
|
||||
/mnt/user/Backups/
|
||||
|
||||
# Dry run — verify space and paths without stopping anything:
|
||||
container_data_export.sh \
|
||||
Emby \
|
||||
/mnt/media-servers/Media_Server/Emby \
|
||||
/mnt/user/Backups/ \
|
||||
--dry-run
|
||||
|
||||
# Output filename: Emby_2026-05-14_02-30.tar.gz
|
||||
# Timestamped — safe to run multiple times, no overwrite
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## emby_database_repair.sh
|
||||
|
||||
Stops Emby, runs SQLite `PRAGMA integrity_check` on every Emby database, and restarts.
|
||||
Reports per-database — does NOT automatically repair. Recovery requires judgment.
|
||||
|
||||
### When to Use
|
||||
|
||||
```
|
||||
Emby logs show database errors → run this first
|
||||
Emby crashing repeatedly with no clear cause → likely database corruption
|
||||
Playback history or user data behaving strangely → users.db or library.db issue
|
||||
After a hard shutdown or power loss with Emby running → check for WAL corruption
|
||||
```
|
||||
|
||||
### Recovery Guide by Database
|
||||
|
||||
```
|
||||
library.db — media library metadata: titles, seasons, episodes, artwork
|
||||
CORRUPT → safe to delete — Emby fully rebuilds from media files on next start
|
||||
Rebuild takes time (hours on large libraries) but loses nothing permanent
|
||||
|
||||
users.db — user accounts, watch history, playback positions, settings
|
||||
CORRUPT → deleting resets ALL user accounts and watch history
|
||||
Check for a recent backup (weekly_sync_maintenance.sh mirrors Emby/)
|
||||
before deleting — restore from remote if available
|
||||
|
||||
authentication.db — API keys, session tokens
|
||||
CORRUPT → safe to delete — API keys regenerated on restart
|
||||
Any connected clients will need to re-authenticate once
|
||||
|
||||
activity.db — activity/access log
|
||||
CORRUPT → safe to delete — it's a log, losing it is acceptable
|
||||
|
||||
library.db-wal — write-ahead log (uncommitted transactions)
|
||||
PRESENT + CORRUPT → check library.db first; WAL corruption usually means
|
||||
the main library.db is also affected
|
||||
```
|
||||
|
||||
### Configuration (master_host*.conf)
|
||||
|
||||
```bash
|
||||
HOST1_EMBY_CONTAINER="Emby" # aliased by detect_hosts() → EMBY_CONTAINER
|
||||
HOST2_EMBY_CONTAINER="Emby"
|
||||
```
|
||||
|
||||
Emby's config path is detected automatically from Docker volume mounts — no manual
|
||||
path configuration needed.
|
||||
|
||||
### Usage
|
||||
|
||||
```bash
|
||||
emby_database_repair.sh # stop Emby, check all databases, restart
|
||||
emby_database_repair.sh --dry-run # show what would be checked, no Emby stop
|
||||
emby_database_repair.sh --log # verbose — show SQLite output per database
|
||||
emby_database_repair.sh --status # show Emby config path and database locations
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## zfs_pool_scrub.sh
|
||||
|
||||
Triggers ZFS scrub on all pools (or a specific named pool) and waits for completion.
|
||||
Notifies when done with a summary of any errors found.
|
||||
|
||||
### Why Run ZFS Scrub
|
||||
|
||||
ZFS stores a checksum with every block of data. Scrub reads every block and verifies
|
||||
the checksum matches the stored hash. Silent data corruption can sit on disk for months
|
||||
without triggering any error — until you try to read that specific file. By then:
|
||||
- It may already be mirrored to HOST2 in its corrupted state
|
||||
- The original source may no longer exist
|
||||
- ZFS can self-repair during scrub if redundancy exists (RAIDZ or mirrors)
|
||||
|
||||
Run monthly. Also run after any disk replacement or power event.
|
||||
Safe to run while the system is in use — scrub runs at low I/O priority.
|
||||
|
||||
### Configuration (master_host*.conf)
|
||||
|
||||
```bash
|
||||
HOST1_ZFS_REPORT_IGNORE_POOLS=(
|
||||
"disk10" # JBOD member — no redundancy, skipped from default scrub
|
||||
"disk9"
|
||||
"disk8"
|
||||
)
|
||||
|
||||
HOST2_ZFS_REPORT_IGNORE_POOLS=(
|
||||
"cache" # example — single-disk pool excluded from default
|
||||
)
|
||||
```
|
||||
|
||||
To scrub a pool in the ignore list, specify it by name explicitly.
|
||||
|
||||
### Usage
|
||||
|
||||
```bash
|
||||
# Scrub all pools except those in ZFS_REPORT_IGNORE_POOLS:
|
||||
zfs_pool_scrub.sh
|
||||
|
||||
# Scrub a specific pool by name — bypasses the ignore list:
|
||||
zfs_pool_scrub.sh gaming
|
||||
|
||||
# Check current scrub status without starting a new one:
|
||||
zfs_pool_scrub.sh --status
|
||||
|
||||
# Dry run — show which pools would be scrubbed:
|
||||
zfs_pool_scrub.sh --dry-run
|
||||
|
||||
# Verbose — show scrub progress every 60s poll:
|
||||
zfs_pool_scrub.sh --log
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## recreate_shares.sh
|
||||
|
||||
Creates share directories on the correct disks after a fresh unRAID install or disk
|
||||
rebuild. Run once on HOST2 before the first rsync from HOST1.
|
||||
|
||||
### When to Use
|
||||
|
||||
```
|
||||
After a fresh unRAID install where /boot/config/shares/*.cfg were restored:
|
||||
The share definitions exist → UI shows shares → directories are missing on disk
|
||||
rsync.sh tries to write to /mnt/user/Movies → path doesn't exist → aborts
|
||||
|
||||
After a disk replacement or rebuild where share folders were lost:
|
||||
Replacement disk is blank → no share directories on the new disk
|
||||
unRAID won't create them automatically
|
||||
```
|
||||
|
||||
### What It Does
|
||||
|
||||
```
|
||||
For each .cfg file in /boot/config/shares/:
|
||||
1. Read the share name (e.g., Movies)
|
||||
2. Read the shareInclude list (e.g., disk1,disk2,disk5)
|
||||
3. Create /mnt/disk1/Movies, /mnt/disk2/Movies, /mnt/disk5/Movies
|
||||
4. Place a .recovery marker in /mnt/user/Movies/
|
||||
|
||||
The .recovery marker tells rsync.sh this is a fresh share:
|
||||
.recovery present → rsync WITHOUT --delete (safe — new files only, nothing removed)
|
||||
.recovery absent → rsync WITH --delete (normal mirror mode)
|
||||
|
||||
Self-cleaning: after the first successful rsync, the source side has no .recovery file,
|
||||
so the second nightly run deletes it from the mirror and normal --delete resumes.
|
||||
No manual cleanup needed.
|
||||
```
|
||||
|
||||
### Usage
|
||||
|
||||
```bash
|
||||
recreate_shares.sh # create all missing share directories + .recovery markers
|
||||
recreate_shares.sh --dry-run # show what would be created without creating
|
||||
recreate_shares.sh --log # verbose — show each directory created per disk
|
||||
recreate_shares.sh --status # show share configs and current directory state
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## continuous_scripts_status.sh
|
||||
|
||||
Live status dashboard for all continuously running scripts. Read-only — makes no
|
||||
changes to any running process, container, or state file.
|
||||
|
||||
### What It Shows
|
||||
|
||||
```
|
||||
system_watchdog
|
||||
Running state, PID, uptime, approximate cycle count
|
||||
Active strikes, recent restart history
|
||||
Live snapshot: rootfs, RAM, ZFS ARC, load, zombie count, CPU temp
|
||||
|
||||
docker_watchdog
|
||||
Running state, PID, uptime
|
||||
Running / stopped / unhealthy container counts
|
||||
Required containers status
|
||||
Memory-monitored containers
|
||||
Recent restart history + skip list
|
||||
|
||||
failover (fallback.sh)
|
||||
Current state (NORMAL / FALLBACK / HANDBACK)
|
||||
Tier flags and timestamps
|
||||
Remote Tailscale visibility
|
||||
```
|
||||
|
||||
State files are read as-is — if a script is mid-cycle, the display reflects the last
|
||||
completed cycle, not the current in-progress state.
|
||||
|
||||
### Usage
|
||||
|
||||
```bash
|
||||
continuous_scripts_status.sh # show full dashboard
|
||||
continuous_scripts_status.sh --log # verbose output with additional detail per section
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## claude_startup.sh
|
||||
|
||||
Restores Claude Code's persistent data after an unRAID reboot and optionally launches
|
||||
Claude. Standalone script — no common.sh dependency.
|
||||
|
||||
### Why This Exists
|
||||
|
||||
unRAID's root filesystem lives in RAM — `/root/.claude` and `/root/.local` are wiped on
|
||||
every reboot. This script symlinks both directories back to persistent appdata storage
|
||||
at `/mnt/user/appdata/claude-code/` before launching Claude.
|
||||
|
||||
### First Run Migration
|
||||
|
||||
On first run, if persistent storage is empty, the script migrates from current live locations:
|
||||
|
||||
```
|
||||
/root/.claude → /mnt/user/appdata/claude-code/.claude
|
||||
/root/.local/share/claude → /mnt/user/appdata/claude-code/local/share/claude
|
||||
```
|
||||
|
||||
Subsequent runs skip the migration and only create the symlinks.
|
||||
|
||||
### Calling from array_started.sh
|
||||
|
||||
To auto-restore Claude data on every boot without launching an interactive session:
|
||||
|
||||
```bash
|
||||
# In /boot/config/go or array_started.sh:
|
||||
/path/to/Tools/claude_startup.sh --setup
|
||||
```
|
||||
|
||||
### Usage
|
||||
|
||||
```bash
|
||||
claude_startup.sh # set up persistent symlinks and launch Claude
|
||||
claude_startup.sh --setup # set up symlinks only — no launch (for array_started.sh)
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Adding a New Tool
|
||||
|
||||
Write the tool when you solve a problem manually with bash commands. You'll face it again.
|
||||
The cost of writing the tool is 30 minutes. The cost of reconstructing the commands at 2am
|
||||
is much higher.
|
||||
|
||||
### Checklist
|
||||
|
||||
```
|
||||
✓ Header explains the specific situation that requires this tool
|
||||
✓ Root check — most tools need root
|
||||
✓ --dry-run support — always
|
||||
✓ --status support — show current state before acting
|
||||
✓ Confirmation for destructive operations (interactive YES or --force flag)
|
||||
✓ Notify on completion — success and failure
|
||||
✓ Leave system in clean state on any exit — trap for cleanup
|
||||
✓ Add to README-Tools.md scripts table and HOW THE SCRIPTS RELATE diagram
|
||||
```
|
||||
|
||||
### Minimal Skeleton
|
||||
|
||||
```bash
|
||||
#!/bin/bash
|
||||
# ==============================================================================================
|
||||
# ============================= Your Tool Name ================================================
|
||||
# ==============================================================================================
|
||||
#
|
||||
# PURPOSE
|
||||
# ─────────────────────────────────────────────────────────────────────────────
|
||||
# One sentence: what situation this solves and when to use it.
|
||||
#
|
||||
# ==============================================================================================
|
||||
# OPERATIONAL SAFEGUARDS
|
||||
# ==============================================================================================
|
||||
#
|
||||
# Root Required
|
||||
# chown / docker / etc. require root.
|
||||
#
|
||||
# Confirmation Required
|
||||
# Interactive mode prompts for YES. Use --force to bypass in scripts.
|
||||
#
|
||||
# ==============================================================================================
|
||||
# RUNTIME MODES
|
||||
# ==============================================================================================
|
||||
#
|
||||
# your_tool.sh
|
||||
# Normal run.
|
||||
#
|
||||
# your_tool.sh --dry-run
|
||||
# Preview without making changes.
|
||||
#
|
||||
# your_tool.sh --status
|
||||
# Show current state and exit.
|
||||
#
|
||||
# ==============================================================================================
|
||||
|
||||
SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
|
||||
source "$SCRIPT_DIR/../load_config.sh"
|
||||
parse_args "$@"
|
||||
|
||||
if [[ "$EUID" -ne 0 ]]; then error "Must be run as root"; exit 1; fi
|
||||
|
||||
validate_unraid_cmd \
|
||||
"/usr/local/emhttp/plugins/dynamix/scripts/notify" \
|
||||
"" "" "unRAID notify script" || warn "notify not found — notifications disabled"
|
||||
|
||||
acquire_lock
|
||||
detect_hosts
|
||||
|
||||
if [[ "$SHOW_STATUS" == true ]]; then
|
||||
log "Current state: ..."
|
||||
exit 0
|
||||
fi
|
||||
|
||||
[[ "$DRY_RUN" == true ]] && warn "DRY RUN — no changes will be made"
|
||||
|
||||
if [[ "$FORCE" != true ]]; then
|
||||
read -r -p "Type YES to proceed: " CONFIRM
|
||||
[[ "$CONFIRM" != "YES" ]] && { warn "Aborted."; exit 0; }
|
||||
fi
|
||||
|
||||
# Do the work
|
||||
# ...
|
||||
|
||||
notify "Tool completed on $(hostname) ($MY_ID)" "Tool Name" "normal"
|
||||
```
|
||||
+121
-802
@@ -1,834 +1,153 @@
|
||||
# ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
|
||||
# 🔧 TOOLS
|
||||
# ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
|
||||
# ━━━━━ TOOLS ━━━━━
|
||||
|
||||
**Situational utilities — run when something needs fixing, not on a schedule.**
|
||||
Recovery, repair, migration, cleanup, and one-time tasks that don't fit the scheduled
|
||||
maintenance model. These scripts sit ready for the moment you actually need them.
|
||||
|
||||
```
|
||||
unRAID_Essentials/ ← regular system maintenance — scheduled
|
||||
Docker_Essentials/ ← regular container management — scheduled
|
||||
Monitors/ ← regular health reporting — scheduled
|
||||
Orchestrators/ ← regular maintenance windows — scheduled
|
||||
Tools/ ← situational utilities — run when needed
|
||||
```
|
||||
Recovery, repair, migration, and inspection tools for situations that arise outside
|
||||
the scheduled maintenance model. These scripts sit ready for the moment you need them.
|
||||
|
||||
> **None of these scripts run on a schedule.** A script belongs here when it solves
|
||||
> a specific operational situation rather than ongoing maintenance — something you run
|
||||
> in response to a problem, a planned migration, or a one-time task. Having a dedicated
|
||||
> folder keeps the other folders clean and makes it obvious what runs routinely vs what
|
||||
> runs situationally.
|
||||
> a specific operational situation — something you run in response to a problem, before
|
||||
> a risky operation, or during a one-time setup task. Having a dedicated folder keeps
|
||||
> the other folders clean and makes it obvious what runs routinely vs. situationally.
|
||||
|
||||
---
|
||||
|
||||
## ━━━ THE PROBLEM THAT BUILT THIS ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
|
||||
## ━━━ THE PROBLEM THAT BUILT THIS ━━━
|
||||
|
||||
Every tool here exists because a specific situation arose that required bash commands
|
||||
to resolve — and that situation is guaranteed to arise again.
|
||||
**Failover State Stuck After Testing**
|
||||
Run a failover test, something exits uncleanly, state file shows `FALLBACK`.
|
||||
`fallback.sh` resumes and reads FALLBACK — starts containers it shouldn't, makes
|
||||
decisions based on a state that doesn't reflect reality. Manual recovery means
|
||||
knowing the exact file format and every field to reset. At 2am after a failed test.
|
||||
Fix: `fallback_state_reset.sh` — shows current state, prompts for confirmation,
|
||||
resets cleanly to NORMAL.
|
||||
|
||||
**Container Stuck on Watchdog Skip List After Fixing the Problem**
|
||||
Authelia hit the restart loop limit — went on the skip list. Problem fixed. But
|
||||
the watchdog still isn't monitoring it because the skip list persists on `/boot/config`
|
||||
across reboots. Where's the file? What format? How do you clear restart history?
|
||||
Fix: `watchdog_skip_list_manager.sh` — shows the skip list and which containers are
|
||||
running vs. stopped, clears specific containers with confirmation.
|
||||
|
||||
**Emby Crashing With No Clear Cause After a Power Cut**
|
||||
Server lost power with Emby running. Emby comes back, runs for 20 minutes, crashes.
|
||||
Logs show database errors. Which database? library.db? users.db? Each has different
|
||||
recovery implications — deleting the wrong one resets all user watch history.
|
||||
Fix: `emby_database_repair.sh` — stops Emby, runs `PRAGMA integrity_check` on every
|
||||
database, reports per-database with specific guidance on what to do about each one.
|
||||
|
||||
**Files Owned by Root After an Admin Copy**
|
||||
`scp` a file into a media share. File arrives as `root:root`. Radarr fails to import —
|
||||
permission denied. The daily permissions script won't run for another 20 hours. Running
|
||||
`media_shares_permissions.sh` on the whole library takes 30 minutes just to fix one dir.
|
||||
Fix: `bulk_permissions_repair.sh` — takes specific paths, applies correct ownership and
|
||||
permissions in seconds.
|
||||
|
||||
**No Way to Back Up a Container Before a Risky Update**
|
||||
Major version update, changelog says "database migration — no rollback." You want a
|
||||
point-in-time backup. But `cp -r` while the container is running produces an
|
||||
inconsistent backup, and tar without stopping the container is equally unreliable.
|
||||
Fix: `container_data_export.sh` — stops the container cleanly, archives appdata to a
|
||||
timestamped `.tar.gz`, verifies archive integrity, restarts the container.
|
||||
|
||||
**Fresh HOST2 Has Shares Configured But Directories Missing**
|
||||
Fresh install on HOST2. Restored `/boot/config/shares/` from backup. Array starts.
|
||||
Shares show in the UI. But the actual `/mnt/diskN/sharename` directories don't exist —
|
||||
unRAID created the share definitions but not the directories. rsync.sh aborts.
|
||||
Fix: `recreate_shares.sh` — reads every `.cfg` file, creates directories on each
|
||||
included disk, places `.recovery` markers so the first rsync won't delete anything.
|
||||
|
||||
---
|
||||
|
||||
### 🔴 Failover State Stuck After Testing
|
||||
## ━━━ WHAT THIS FOLDER DOES ━━━
|
||||
|
||||
Run `failover_test.sh`, something goes wrong mid-test, script exits uncleanly. State
|
||||
file shows `FAILOVER`. `failover.sh` resumes and reads FAILOVER — starts containers
|
||||
it shouldn't start, makes decisions based on a state that doesn't reflect reality.
|
||||
Or the test completed but handback didn't finish — state is partially reset.
|
||||
One role: hold scripts for situations the scheduled maintenance model can't handle.
|
||||
|
||||
Manual recovery: edit the state file by hand? Know the exact format? Know which
|
||||
fields to reset? At 2am after a failed test, none of that is obvious.
|
||||
Every script here was written because a specific situation arose that required bash
|
||||
commands to resolve — and that situation is guaranteed to arise again. When you encounter
|
||||
something new, write the tool. Store it here. Find it at 2am next time.
|
||||
|
||||
The tool: `failover_state_reset.sh` — one command, shows you the current state before
|
||||
asking for confirmation, resets cleanly to NORMAL, explains exactly what it changed.
|
||||
**Recovery Tools** — Restore known-good state after a failure
|
||||
`fallback_state_reset.sh`, `watchdog_skip_list_manager.sh`
|
||||
|
||||
**Diagnostic Tools** — Inspect and verify before acting
|
||||
`emby_database_repair.sh`, `continuous_scripts_status.sh`
|
||||
|
||||
**Repair Tools** — Fix a specific known problem
|
||||
`bulk_permissions_repair.sh`, `zfs_pool_scrub.sh`
|
||||
|
||||
**Lifecycle Tools** — Backup, setup, and migration support
|
||||
`container_data_export.sh`, `recreate_shares.sh`, `claude_startup.sh`
|
||||
|
||||
---
|
||||
|
||||
### 🔴 Container Stuck on Watchdog Skip List After Fixing the Problem
|
||||
## ━━━ RELATIONSHIP TO OTHER FOLDERS ━━━
|
||||
|
||||
Authelia hit the restart loop limit — three restarts in an hour — went on the skip list.
|
||||
You fixed the underlying database issue. But the watchdog still isn't monitoring it
|
||||
because it's on the skip list and you don't know where that file lives or what format
|
||||
it's in. You restart Authelia manually, it runs fine, but the watchdog has no idea
|
||||
it recovered and still thinks it's broken.
|
||||
```
|
||||
unRAID_Essentials/ ← regular system maintenance — scheduled
|
||||
Docker_Essentials/ ← regular container management — scheduled
|
||||
Monitors/ ← regular health reporting — scheduled
|
||||
Orchestrators/ ← regular maintenance windows — scheduled
|
||||
Fallback/ ← automated failover/handback — event-driven
|
||||
Tools/ ← situational utilities — run when needed
|
||||
```
|
||||
|
||||
The tool: `watchdog_skip_list_manager.sh` — shows the skip list, shows which containers
|
||||
are stopped vs running, clears specific containers with a single command, clears restart
|
||||
history so the loop protection window starts fresh.
|
||||
Some tools interact with state written by other folders:
|
||||
|
||||
```
|
||||
Fallback/
|
||||
fallback.sh ──────── writes FALLBACK_STATE_FILE ──► fallback_state_reset.sh reads/writes it
|
||||
|
||||
Docker_Essentials/
|
||||
docker_watchdog.sh ── writes skip list + history ──► watchdog_skip_list_manager.sh manages them
|
||||
|
||||
Docker_Essentials/ + unRAID_Essentials/ + Fallback/
|
||||
All continuous scripts ──────────────────────────► continuous_scripts_status.sh reads their state
|
||||
```
|
||||
|
||||
Tools never call scripts in other folders. Other folders never call Tools scripts.
|
||||
The relationship is one-way: Tools act on state that other scripts have written.
|
||||
|
||||
---
|
||||
|
||||
### 🔴 Emby Crashing With No Clear Cause After a Power Cut
|
||||
|
||||
Server lost power with Emby running. Emby comes back up, runs for 20 minutes, crashes.
|
||||
Comes back up, crashes again. Logs show database errors. Which database? library.db?
|
||||
users.db? authentication.db? They're all SQLite, they all need a different recovery
|
||||
approach, and the errors aren't always obvious from the log output alone.
|
||||
|
||||
The tool: `emby_database_repair.sh` — stops Emby, runs `PRAGMA integrity_check` on
|
||||
every database, reports per database what's clean and what's corrupted with specific
|
||||
guidance on what to do about each one.
|
||||
|
||||
---
|
||||
|
||||
### 🔴 Files Owned by Root After an Admin Copy
|
||||
|
||||
`scp` a file from another machine directly into a media share. File arrives as
|
||||
`root:root`. Radarr tries to move it and fails — permission denied. The daily
|
||||
permissions script won't run for another 20 hours. Running the full
|
||||
`media_shares_permissions.sh` on the whole share takes 30 minutes on a large library
|
||||
just to fix one directory.
|
||||
|
||||
The tool: `bulk_permissions_repair.sh` — takes a specific path or list of paths,
|
||||
applies correct ownership and permissions in seconds, done.
|
||||
|
||||
---
|
||||
|
||||
### 🔴 No Way to Back Up a Container Before a Risky Update
|
||||
|
||||
Container has a major version update. The changelog says "database migration — no
|
||||
rollback." You want a point-in-time backup before you proceed. But the container's
|
||||
appdata is scattered across dozens of files and `cp -r` while it's running produces
|
||||
an inconsistent backup.
|
||||
|
||||
The tool: `container_data_export.sh` — stops the container cleanly, archives the
|
||||
entire appdata directory to a timestamped `.tar.gz`, verifies the archive integrity,
|
||||
restarts the container. The backup is valid and complete before anything else happens.
|
||||
If the update goes wrong you have a clean restore point.
|
||||
|
||||
---
|
||||
|
||||
### 🔴 Fresh HOST2 Has Shares Configured But Directories Missing
|
||||
|
||||
Fresh install on HOST2. Restored `/boot/config/shares/` cfg files from backup. Array
|
||||
starts. Shares show in the UI. But the actual `/mnt/diskN/sharename` directories don't
|
||||
exist on the individual disks — unRAID created the share definitions but not the
|
||||
directories. rsync.sh tries to write, finds the path doesn't exist, aborts.
|
||||
|
||||
The tool: `recreate_shares.sh` — reads every `.cfg` file from `/boot/config/shares/`,
|
||||
parses the `shareInclude` list, creates the correct directory on each included disk.
|
||||
Run once after fresh setup, directories exist, rsync works.
|
||||
|
||||
---
|
||||
|
||||
## ━━━ SCRIPTS AT A GLANCE ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
|
||||
## ━━━ SCRIPTS IN THIS FOLDER ━━━
|
||||
|
||||
| Script | What It Fixes | When to Run |
|
||||
|--------|--------------|-------------|
|
||||
| `failover_state_reset.sh` | State file stuck in FAILOVER after test or failed handback | After failover testing or manual intervention |
|
||||
| `watchdog_skip_list_manager.sh` | Container stuck on watchdog skip list | After fixing a container that hit restart loop limit |
|
||||
| `bulk_permissions_repair.sh` | Files owned by wrong user after admin copy or bad container | When arr operations fail due to permissions |
|
||||
| `container_data_export.sh` | Need a clean backup before a risky container update | Before major updates, migrations, or removals |
|
||||
| `emby_database_repair.sh` | Emby crashing with database errors after power loss | When Emby logs show corruption or repeated crashes |
|
||||
| `zfs_pool_scrub.sh` | Verify ZFS pool integrity — catch silent corruption | Monthly, or after any disk/power event |
|
||||
| `recreate_shares.sh` | Share directories missing on fresh install or rebuild | After fresh unRAID install or disk replacement |
|
||||
| `rsync_stop.sh` | rsync stuck or needs emergency stop | When rsync is running and must be stopped cleanly |
|
||||
| `user_scripts_stop.sh` | User Scripts running mid-cycle and need stopping | Before planned reboots, emergency stop |
|
||||
| `server_reboot.sh` | Graceful reboot with pre-flight warnings and clean shutdown | Planned maintenance reboots |
|
||||
| `fallback_state_reset.sh` | State file stuck in FALLBACK after test or failed handback | After failover testing or manual intervention |
|
||||
| `watchdog_skip_list_manager.sh` | Container stuck on watchdog skip list after fixing root cause | After fixing a container that hit the restart loop limit |
|
||||
| `bulk_permissions_repair.sh` | Files owned by wrong user after admin copy or bad container config | When arr operations fail due to permissions |
|
||||
| `container_data_export.sh` | Need a clean backup before a risky container update or migration | Before major updates, appdata migrations, or container removals |
|
||||
| `emby_database_repair.sh` | Emby crashing with database errors after power loss or crash | When Emby logs show corruption or repeated crashes |
|
||||
| `zfs_pool_scrub.sh` | Verify ZFS pool integrity — catch silent corruption before it spreads | Monthly, or after any disk or power event |
|
||||
| `recreate_shares.sh` | Share directories missing after fresh install or disk rebuild | After fresh unRAID install or disk replacement on HOST2 |
|
||||
| `continuous_scripts_status.sh` | Need a live view of watchdog and fallback state | Any time — manual dashboard, no schedule |
|
||||
| `claude_startup.sh` | Claude Code session setup after reboot — symlinks persistent storage | After each unRAID reboot, or called by array_started.sh |
|
||||
|
||||
---
|
||||
|
||||
## ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
|
||||
## 🔀 failover_state_reset.sh
|
||||
## ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
|
||||
## ━━━ HOW THE SCRIPTS RELATE ━━━
|
||||
|
||||
Resets the failover state file to NORMAL and clears all tier flags. State file only —
|
||||
does NOT start or stop any containers.
|
||||
|
||||
> Full documentation in `README-Failover.md` — `failover_state_reset.sh` section.
|
||||
> This entry is a quick reference.
|
||||
|
||||
---
|
||||
|
||||
### ── When to Use ──────────────────────────────────────────────────────────────
|
||||
All Tools scripts are independent — none call each other, none are called by other Tools.
|
||||
|
||||
```
|
||||
After failover_test.sh didn't complete cleanly
|
||||
→ state left in FAILOVER but containers are actually back to normal
|
||||
|
||||
After a failed handback
|
||||
→ state shows FAILOVER but remote is back up and containers are split
|
||||
|
||||
After killing failover.sh directly (not via User Scripts Abort)
|
||||
→ state is unknown, cycle was interrupted mid-operation
|
||||
|
||||
After a dev/debug session
|
||||
→ state left in a non-NORMAL state from testing
|
||||
Situation arises
|
||||
│
|
||||
▼
|
||||
┌──────────────────────────────────────────────────────────────────────┐
|
||||
│ Tools/ Run directly when needed │
|
||||
│ │
|
||||
│ fallback_state_reset.sh ◄── after failover test / failed handback│
|
||||
│ watchdog_skip_list_manager ◄── after fixing a crash-looping container│
|
||||
│ bulk_permissions_repair ◄── wrong ownership after copy or rsync │
|
||||
│ container_data_export ◄── before a risky update or migration │
|
||||
│ emby_database_repair ◄── Emby logs show corruption │
|
||||
│ zfs_pool_scrub ◄── monthly integrity check / post-event │
|
||||
│ recreate_shares ◄── fresh HOST2 setup or disk rebuild │
|
||||
│ continuous_scripts_status ◄── manual status check at any time │
|
||||
│ claude_startup ◄── after each unRAID reboot │
|
||||
└──────────────────────────────────────────────────────────────────────┘
|
||||
│
|
||||
▼
|
||||
State files in other folders (Fallback/, Docker_Essentials/) may be read or written.
|
||||
No other scripts call into Tools/.
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
### ── Verify Before Resetting ────────────────────────────────────────────────
|
||||
|
||||
```bash
|
||||
# ─────────────────────────────────────────────────────────────────────────────
|
||||
# Verify reality matches what you're about to declare as NORMAL:
|
||||
#
|
||||
# Right containers on right server?
|
||||
# DDNS pointing correctly? nslookup Gmer4Lfe.com
|
||||
# failover.sh not running? pgrep -f "failover.sh"
|
||||
# Both servers Tailscale connected? tailscale status
|
||||
#
|
||||
# Resetting during an actual failover causes failover.sh to think everything
|
||||
# is normal and stop covering the remote — services go offline until the next
|
||||
# detection cycle catches it again.
|
||||
# ─────────────────────────────────────────────────────────────────────────────
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
### ── Usage ───────────────────────────────────────────────────────────────────
|
||||
|
||||
```bash
|
||||
failover_state_reset.sh --status # show current state file contents — always first
|
||||
failover_state_reset.sh --dry-run # show what would be written, no write
|
||||
failover_state_reset.sh # interactive reset — prompts "YES" to confirm
|
||||
failover_state_reset.sh --force # non-interactive — for scripts, no terminal
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
|
||||
## 🐳 watchdog_skip_list_manager.sh
|
||||
## ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
|
||||
|
||||
View and manage the persistent container skip list used by `docker_watchdog.sh`.
|
||||
|
||||
> Full documentation in `README-Docker_Essentials.md` — `watchdog_skip_list_manager.sh`
|
||||
> section including the full recovery workflow. This entry is a quick reference.
|
||||
|
||||
---
|
||||
|
||||
### ── When to Use ──────────────────────────────────────────────────────────────
|
||||
|
||||
```
|
||||
docker_watchdog.sh restarts the same container N times within the rolling window
|
||||
→ container added to skip list on /boot/config/
|
||||
→ critical notification sent
|
||||
→ watchdog stops touching it entirely
|
||||
|
||||
You fix the underlying problem (database, config, dependencies).
|
||||
You need to clear the container from the skip list so monitoring resumes.
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
### ── Recovery Workflow ────────────────────────────────────────────────────────
|
||||
|
||||
```bash
|
||||
# 1. Understand the situation — always start here:
|
||||
watchdog_skip_list_manager.sh --status
|
||||
# Shows: skip list contents, which are running vs stopped, restart history
|
||||
|
||||
# 2. Fix the underlying problem first
|
||||
# Check logs: docker logs ContainerName --tail 100
|
||||
# Check disk: df -h /mnt/user
|
||||
# Check db: docker exec ContainerName sqlite3 /path/to.db ".tables"
|
||||
|
||||
# 3. Clear from skip list + restart history:
|
||||
watchdog_skip_list_manager.sh --clear ContainerName
|
||||
|
||||
# 4. Start the container manually — confirm your fix worked:
|
||||
docker start ContainerName
|
||||
|
||||
# 5. Watchdog resumes normal monitoring on next cycle — no further action needed
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
### ── Files Managed ────────────────────────────────────────────────────────────
|
||||
|
||||
```bash
|
||||
# ─────────────────────────────────────────────────────────────────────────────
|
||||
/boot/config/system_watchdog_failed.db # persistent skip list
|
||||
/boot/config/container_restart_history.db # restart loop tracking
|
||||
|
||||
# Both live on /boot/config — survive reboots intentionally.
|
||||
# A container that was skip-listed before a reboot is still broken after it.
|
||||
# ─────────────────────────────────────────────────────────────────────────────
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
### ── Usage ───────────────────────────────────────────────────────────────────
|
||||
|
||||
```bash
|
||||
watchdog_skip_list_manager.sh # show status (default)
|
||||
watchdog_skip_list_manager.sh --status # explicit status
|
||||
watchdog_skip_list_manager.sh --clear ContainerName # clear specific + restart history
|
||||
watchdog_skip_list_manager.sh --clear ContainerName --force # no confirmation prompt
|
||||
watchdog_skip_list_manager.sh --clear-all # clear everything
|
||||
watchdog_skip_list_manager.sh --clear-all --force # non-interactive
|
||||
watchdog_skip_list_manager.sh --dry-run # preview any clear action
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
|
||||
## 🔐 bulk_permissions_repair.sh
|
||||
## ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
|
||||
|
||||
Applies correct ownership and permissions to specific paths. Faster than running
|
||||
`media_shares_permissions.sh` which processes every configured share — use this when
|
||||
you know exactly what needs fixing and don't want to wait for a full library walk.
|
||||
|
||||
---
|
||||
|
||||
### ── When to Use ──────────────────────────────────────────────────────────────
|
||||
|
||||
```bash
|
||||
# ─────────────────────────────────────────────────────────────────────────────
|
||||
# Use instead of the full permissions script when:
|
||||
#
|
||||
# Admin copy left root:root files — scp, cp, direct file transfer
|
||||
# New share needs permissions now — can't wait for nightly run
|
||||
# Container wrote as root — before PUID/PGID was fixed
|
||||
# Specific directory has wrong perms — targeted fix, not a full library walk
|
||||
#
|
||||
# The full media_shares_permissions.sh is the right tool for:
|
||||
# Regular nightly maintenance (already scheduled in daily_sync_maintenance.sh)
|
||||
# After confirming a container's PUID/PGID is now correct
|
||||
# Initial permissions setup on a new server
|
||||
# ─────────────────────────────────────────────────────────────────────────────
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
### ── What It Applies ──────────────────────────────────────────────────────────
|
||||
|
||||
```bash
|
||||
# master.conf
|
||||
# ─────────────────────────────────────────────────────────────────────────────
|
||||
# Same values as media_shares_permissions.sh — consistent permissions everywhere
|
||||
PERMISSIONS_DIR_MODE="755" # directories — enter, list, no world-write
|
||||
PERMISSIONS_FILE_MODE="664" # files — owner+group rw, others read-only
|
||||
PERMISSIONS_OWNER="nobody:users" # matches PUID=99 PGID=100 in containers
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
### ── Usage ───────────────────────────────────────────────────────────────────
|
||||
|
||||
```bash
|
||||
# ─────────────────────────────────────────────────────────────────────────────
|
||||
# Single path:
|
||||
bulk_permissions_repair.sh /mnt/user/Movies
|
||||
|
||||
# Multiple paths — all corrected in one run:
|
||||
bulk_permissions_repair.sh /mnt/user/Movies /mnt/user/Tv_Shows /mnt/user/Music
|
||||
|
||||
# Dry run first — shows count of files that would be corrected:
|
||||
bulk_permissions_repair.sh /mnt/user/Movies --dry-run
|
||||
|
||||
# Verbose — show each corrected file:
|
||||
bulk_permissions_repair.sh /mnt/user/Movies --log
|
||||
# ─────────────────────────────────────────────────────────────────────────────
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
|
||||
## 📦 container_data_export.sh
|
||||
## ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
|
||||
|
||||
Exports a container's appdata directory to a compressed tar archive. Stops the
|
||||
container first for a clean consistent backup, verifies the archive after creation,
|
||||
then restarts the container.
|
||||
|
||||
---
|
||||
|
||||
### ── When to Use ──────────────────────────────────────────────────────────────
|
||||
|
||||
```bash
|
||||
# ─────────────────────────────────────────────────────────────────────────────
|
||||
# Before major container updates — especially those with "no rollback" database migrations
|
||||
# Before pool migrations — clean backup before moving appdata to a new pool
|
||||
# Before removing a container from the stack — archive its data before deletion
|
||||
# Manual point-in-time backup before risky config changes
|
||||
# ─────────────────────────────────────────────────────────────────────────────
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
### ── Sequence ─────────────────────────────────────────────────────────────────
|
||||
|
||||
```bash
|
||||
# ─────────────────────────────────────────────────────────────────────────────
|
||||
# 1. Space check
|
||||
# Estimates required space from appdata size × 1.1
|
||||
# Aborts if output directory doesn't have enough free space
|
||||
# Container is NOT stopped until the space check passes
|
||||
#
|
||||
# 2. Stop container cleanly
|
||||
# docker stop ContainerName — graceful shutdown
|
||||
#
|
||||
# 3. Create archive
|
||||
# tar -czf ContainerName_YYYY-MM-DD_HH-MM.tar.gz /path/to/appdata
|
||||
#
|
||||
# 4. Verify archive integrity
|
||||
# tar -tzf archive.tar.gz — confirms archive is valid and complete
|
||||
# If verification fails → restart container anyway, report error
|
||||
#
|
||||
# 5. Restart container
|
||||
# docker start ContainerName — always restarted, even if archiving failed
|
||||
# ─────────────────────────────────────────────────────────────────────────────
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
### ── Usage ───────────────────────────────────────────────────────────────────
|
||||
|
||||
```bash
|
||||
# Syntax: container_data_export.sh ContainerName AppDataPath OutputDir
|
||||
|
||||
# Emby backup example:
|
||||
container_data_export.sh \
|
||||
Emby \
|
||||
/mnt/media-servers/Media_Server/Emby \
|
||||
/mnt/user/Backups/
|
||||
|
||||
# Dry run — verify space and paths without stopping anything:
|
||||
container_data_export.sh \
|
||||
Emby \
|
||||
/mnt/media-servers/Media_Server/Emby \
|
||||
/mnt/user/Backups/ \
|
||||
--dry-run
|
||||
|
||||
# Output filename format: ContainerName_YYYY-MM-DD_HH-MM.tar.gz
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
|
||||
## 🎬 emby_database_repair.sh
|
||||
## ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
|
||||
|
||||
Stops Emby, runs SQLite `PRAGMA integrity_check` on every Emby database, and restarts.
|
||||
Reports per-database with specific guidance on what to do if corruption is found.
|
||||
|
||||
---
|
||||
|
||||
### ── When to Use ──────────────────────────────────────────────────────────────
|
||||
|
||||
```
|
||||
Emby logs show database errors → run this first
|
||||
Emby crashing repeatedly with no clear cause → likely database corruption
|
||||
Playback history or user data behaving strangely → users.db or library.db issue
|
||||
After a hard shutdown or power loss with Emby running → check for WAL corruption
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
### ── Databases Checked ────────────────────────────────────────────────────────
|
||||
|
||||
```bash
|
||||
# ─────────────────────────────────────────────────────────────────────────────
|
||||
# Each database has different recovery implications:
|
||||
#
|
||||
# library.db — media library metadata: titles, seasons, episodes, artwork
|
||||
# CORRUPT → safe to delete — Emby fully rebuilds from media files on next start
|
||||
# Rebuild takes time but loses nothing permanent
|
||||
#
|
||||
# users.db — user accounts, watch history, playback positions, settings
|
||||
# CORRUPT → deleting resets ALL user accounts and watch history
|
||||
# Check if you have a recent backup (weekly_sync_maintenance.sh)
|
||||
# before deleting
|
||||
#
|
||||
# authentication.db — API keys, session tokens
|
||||
# CORRUPT → safe to delete — API keys regenerated on restart
|
||||
# Any connected clients will need to re-authenticate
|
||||
#
|
||||
# activity.db — activity/access log
|
||||
# CORRUPT → safe to delete — it's a log, losing it is acceptable
|
||||
#
|
||||
# ─────────────────────────────────────────────────────────────────────────────
|
||||
# This script checks and reports ONLY. It does NOT automatically delete or repair
|
||||
# corrupted databases. Recovery requires judgment — and potentially a backup restore.
|
||||
# The summary provides specific guidance per database type.
|
||||
# ─────────────────────────────────────────────────────────────────────────────
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
### ── Usage ───────────────────────────────────────────────────────────────────
|
||||
|
||||
```bash
|
||||
# ─────────────────────────────────────────────────────────────────────────────
|
||||
# Normal run — stops Emby, checks all databases, restarts:
|
||||
emby_database_repair.sh
|
||||
|
||||
# Dry run — detect config path and show what would be checked, no Emby stop:
|
||||
emby_database_repair.sh --dry-run
|
||||
|
||||
# Verbose — show SQLite output for each database:
|
||||
emby_database_repair.sh --log
|
||||
|
||||
# Status — show Emby config path and database file locations:
|
||||
emby_database_repair.sh --status
|
||||
# ─────────────────────────────────────────────────────────────────────────────
|
||||
# Config path is detected automatically from Docker volume mounts.
|
||||
# No configuration needed — just run it.
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
|
||||
## 🗄️ zfs_pool_scrub.sh
|
||||
## ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
|
||||
|
||||
Triggers ZFS scrub on all pools (or a specific named pool) and waits for completion.
|
||||
Notifies when done with a summary of any errors found.
|
||||
|
||||
---
|
||||
|
||||
### ── What ZFS Scrub Does ──────────────────────────────────────────────────────
|
||||
|
||||
```bash
|
||||
# ─────────────────────────────────────────────────────────────────────────────
|
||||
# ZFS stores a checksum with every block of data. Scrub reads every block on
|
||||
# every pool and verifies the checksum matches the stored hash.
|
||||
#
|
||||
# Why this matters:
|
||||
# Silent data corruption can sit on disk for months without triggering any
|
||||
# error — until you try to read that specific file. By then:
|
||||
# - It may already be mirrored to HOST2 in its corrupted state
|
||||
# - The original source may no longer exist
|
||||
# - The corruption may have spread if it was a drive issue
|
||||
#
|
||||
# ZFS can self-repair during scrub if redundancy exists — RAIDZ or mirrors.
|
||||
# It cannot repair if you have a single-disk pool (JBOD).
|
||||
# But it will tell you corruption exists before you find out the hard way.
|
||||
#
|
||||
# Recommended: run monthly or after any disk replacement / power event.
|
||||
# Safe to run while the system is in use — scrub runs at low I/O priority.
|
||||
# ─────────────────────────────────────────────────────────────────────────────
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
### ── Pool Filtering ───────────────────────────────────────────────────────────
|
||||
|
||||
```bash
|
||||
# master_host1.conf
|
||||
# ─────────────────────────────────────────────────────────────────────────────
|
||||
# Single-disk JBOD members can be excluded from all-pool scrubs.
|
||||
# To scrub a pool that's in the ignore list: specify it by name explicitly.
|
||||
#
|
||||
HOST1_ZFS_REPORT_IGNORE_POOLS=(
|
||||
"disk10" # JBOD member — no redundancy, scrub still useful but excluded from default
|
||||
"disk9"
|
||||
"disk8"
|
||||
)
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
### ── Usage ───────────────────────────────────────────────────────────────────
|
||||
|
||||
```bash
|
||||
# Scrub all pools except those in ZFS_REPORT_IGNORE_POOLS:
|
||||
zfs_pool_scrub.sh
|
||||
|
||||
# Scrub a specific pool by name — ignores the ignore list:
|
||||
zfs_pool_scrub.sh gaming
|
||||
|
||||
# Check current scrub status without starting a new one:
|
||||
zfs_pool_scrub.sh --status
|
||||
|
||||
# Dry run — show which pools would be scrubbed:
|
||||
zfs_pool_scrub.sh --dry-run
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
|
||||
## 📁 recreate_shares.sh
|
||||
## ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
|
||||
|
||||
Creates share directories on the correct disks after a fresh install or disk rebuild.
|
||||
Reads `.cfg` files from `/boot/config/shares/` and creates the corresponding
|
||||
`/mnt/diskN/sharename` directory on each disk listed in the `shareInclude` setting.
|
||||
|
||||
---
|
||||
|
||||
### ── When to Use ──────────────────────────────────────────────────────────────
|
||||
|
||||
```bash
|
||||
# ─────────────────────────────────────────────────────────────────────────────
|
||||
# After a fresh unRAID install where /boot/config/shares/*.cfg were restored:
|
||||
# The share definitions exist → UI shows shares → directories are missing
|
||||
# rsync.sh tries to write to /mnt/user/Movies → path doesn't exist → aborts
|
||||
#
|
||||
# After a disk replacement or rebuild where share folders were lost:
|
||||
# Replacement disk is blank → no share directories on the new disk
|
||||
# unRAID won't create them automatically
|
||||
#
|
||||
# Run once on HOST2 after fresh setup, before the first rsync from HOST1.
|
||||
# ─────────────────────────────────────────────────────────────────────────────
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
### ── What It Does ─────────────────────────────────────────────────────────────
|
||||
|
||||
```bash
|
||||
# ─────────────────────────────────────────────────────────────────────────────
|
||||
# For each .cfg file in /boot/config/shares/:
|
||||
# 1. Read the share name (e.g. Movies)
|
||||
# 2. Read the shareInclude list (e.g. disk1,disk2,disk5)
|
||||
# 3. Create /mnt/disk1/Movies, /mnt/disk2/Movies, /mnt/disk5/Movies
|
||||
# 4. Set correct ownership: nobody:users
|
||||
#
|
||||
# Does not create content — just the directories.
|
||||
# rsync.sh can then write into them normally.
|
||||
# ─────────────────────────────────────────────────────────────────────────────
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
### ── Usage ───────────────────────────────────────────────────────────────────
|
||||
|
||||
```bash
|
||||
recreate_shares.sh # create all missing share directories
|
||||
recreate_shares.sh --dry-run # show what would be created without creating
|
||||
recreate_shares.sh --log # verbose — show each directory created
|
||||
recreate_shares.sh --status # show share configs and current directory state
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
|
||||
## 🔄 rsync_stop.sh
|
||||
## ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
|
||||
|
||||
Stops rsync intelligently on both local and remote servers. Auto-detects if an
|
||||
orchestrator is running and chooses the safest stop strategy automatically.
|
||||
|
||||
---
|
||||
|
||||
### ── Two Modes ────────────────────────────────────────────────────────────────
|
||||
|
||||
```bash
|
||||
# ─────────────────────────────────────────────────────────────────────────────
|
||||
# Smart (default — auto-detected):
|
||||
# Orchestrator detected → kill rsync subprocess only
|
||||
# Orchestrator sees rsync died → moves to next share or exits cleanly
|
||||
# No orphaned lock files, orchestrator exits naturally
|
||||
#
|
||||
# --full-stop:
|
||||
# Kill orchestrator first, then rsync
|
||||
# Orchestrator will NOT continue to next share
|
||||
# Use when: you need everything stopped immediately
|
||||
#
|
||||
# Why smart is usually correct:
|
||||
# Killing the orchestrator directly (daily_sync_maintenance.sh) leaves it
|
||||
# mid-execution. Containers may be stopped but not restarted. Lock files
|
||||
# may not be released. The smart approach lets the orchestrator clean up
|
||||
# after itself — fewer side effects.
|
||||
# ─────────────────────────────────────────────────────────────────────────────
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
### ── When to Use ──────────────────────────────────────────────────────────────
|
||||
|
||||
```
|
||||
rsync running during a window where it shouldn't be → smart stop
|
||||
rsync stuck with no progress → smart stop
|
||||
Need to start a manual sync that conflicts → smart stop first
|
||||
Everything must stop NOW (emergency) → --full-stop
|
||||
Called by partnership_manage.sh --offboard → --rsync-only flag
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
### ── Usage ───────────────────────────────────────────────────────────────────
|
||||
|
||||
```bash
|
||||
rsync_stop.sh # smart stop — auto-detect orchestrator
|
||||
rsync_stop.sh --full-stop # kill orchestrator + rsync
|
||||
rsync_stop.sh --rsync-only # stop rsync, skip container recovery
|
||||
rsync_stop.sh --dry-run # preview without stopping anything
|
||||
rsync_stop.sh --status # show what's currently running
|
||||
rsync_stop.sh --full-stop --dry-run # preview full stop
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
|
||||
## 🛑 user_scripts_stop.sh
|
||||
## ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
|
||||
|
||||
Stops all running User Script processes spawned by the unRAID User Scripts plugin.
|
||||
Identifies processes by their `/tmp/user.scripts` path signature, shows script names
|
||||
not just PIDs, uses SIGTERM → SIGKILL sequence with verification.
|
||||
|
||||
---
|
||||
|
||||
### ── When to Use ──────────────────────────────────────────────────────────────
|
||||
|
||||
```bash
|
||||
# ─────────────────────────────────────────────────────────────────────────────
|
||||
# A script is stuck and won't respond to the Abort button in the User Scripts UI
|
||||
# → the UI button sends a signal that the script may have trapped or ignored
|
||||
# → user_scripts_stop.sh finds the process by path signature, not by UI state
|
||||
#
|
||||
# Before a planned reboot to ensure scripts exit cleanly
|
||||
# → server_reboot.sh calls this automatically as part of its shutdown sequence
|
||||
#
|
||||
# Emergency stop of all background ecosystem scripts
|
||||
# → stops system_watchdog, docker_watchdog, failover, and any running maintenance
|
||||
# → use when you need to take manual control immediately
|
||||
# ─────────────────────────────────────────────────────────────────────────────
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
### ── Self-Exclusion ────────────────────────────────────────────────────────────
|
||||
|
||||
```bash
|
||||
# ─────────────────────────────────────────────────────────────────────────────
|
||||
# If this script is run via the User Scripts plugin it would find its own PID.
|
||||
# Self-exclusion prevents the script from killing itself mid-execution.
|
||||
# Own PID and parent PID are excluded before any killing begins.
|
||||
# ─────────────────────────────────────────────────────────────────────────────
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
### ── Usage ───────────────────────────────────────────────────────────────────
|
||||
|
||||
```bash
|
||||
user_scripts_stop.sh # stop all — SIGTERM → verify → SIGKILL if needed
|
||||
user_scripts_stop.sh --dry-run # show which scripts would be stopped, by name
|
||||
user_scripts_stop.sh --status # show currently running scripts with PIDs and runtime
|
||||
user_scripts_stop.sh --log # verbose — show each signal and verification step
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
|
||||
## 🔁 server_reboot.sh
|
||||
## ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
|
||||
|
||||
Graceful reboot with pre-flight warnings, wall message, unRAID notification, clean
|
||||
shutdown sequence, and VM graceful shutdown before stopping services.
|
||||
|
||||
---
|
||||
|
||||
### ── When to Use ──────────────────────────────────────────────────────────────
|
||||
|
||||
```bash
|
||||
# ─────────────────────────────────────────────────────────────────────────────
|
||||
# Planned maintenance reboots — gives users notice and shuts down cleanly
|
||||
# After kernel or firmware updates that require a reboot
|
||||
# As an alternative to the unRAID UI reboot — more visibility into state
|
||||
#
|
||||
# NOT needed for: system_watchdog.sh triggered reboots (those use /sbin/reboot
|
||||
# directly after their own shutdown sequence)
|
||||
# ─────────────────────────────────────────────────────────────────────────────
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
### ── Shutdown Sequence ────────────────────────────────────────────────────────
|
||||
|
||||
```bash
|
||||
# ─────────────────────────────────────────────────────────────────────────────
|
||||
# 1. Pre-flight warnings (warn not block):
|
||||
# rsync running → "partial files possible, consider rsync_stop.sh"
|
||||
# mover running → "files may be left mid-move, consider mover_stop.sh"
|
||||
# Emby sessions active → "N streams will be interrupted"
|
||||
#
|
||||
# 2. Wall message → terminal users
|
||||
# 3. unRAID notification → dashboard
|
||||
# 4. Wait REBOOT_SLEEP seconds (default 30) — users can save work
|
||||
# 5. virsh shutdown each running VM → wait REBOOT_VM_WAIT seconds
|
||||
# 6. Stop libvirt (VM Manager)
|
||||
# 7. Stop Docker service
|
||||
# 8. sync — flush filesystem buffers
|
||||
# 9. /sbin/reboot
|
||||
# ─────────────────────────────────────────────────────────────────────────────
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
### ── Usage ───────────────────────────────────────────────────────────────────
|
||||
|
||||
```bash
|
||||
server_reboot.sh # reboot with 30s warning
|
||||
server_reboot.sh --dry-run # walk through sequence without rebooting
|
||||
server_reboot.sh --status # show running processes that would be affected
|
||||
server_reboot.sh --reason="maintenance" # include reason in wall + notification
|
||||
server_reboot.sh --log # verbose output per shutdown step
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## ━━━ ADDING A NEW TOOL ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
|
||||
|
||||
```bash
|
||||
# ─────────────────────────────────────────────────────────────────────────────
|
||||
# When you encounter a situation that required manual bash commands to resolve —
|
||||
# write a tool. You'll face it again. The cost of writing the tool is 30 minutes.
|
||||
# The cost of reconstructing the commands at 2am is much higher.
|
||||
#
|
||||
# Checklist for a new Tools script:
|
||||
#
|
||||
# ✓ Header explains the specific situation that requires this tool
|
||||
# ✓ "When to Use" section — exactly the symptoms that trigger this
|
||||
# ✓ Root check — most tools need root
|
||||
# ✓ --dry-run support — always
|
||||
# ✓ --status support — show current state before acting
|
||||
# ✓ Confirmation for destructive operations (read -p "Type YES:")
|
||||
# ✓ Notify on completion — success and failure
|
||||
# ✓ Leave system in clean state on any exit — trap for cleanup
|
||||
# ─────────────────────────────────────────────────────────────────────────────
|
||||
|
||||
# Minimal skeleton:
|
||||
#!/bin/bash
|
||||
SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
|
||||
source "$SCRIPT_DIR/../load_config.sh"
|
||||
parse_args "$@"
|
||||
|
||||
if [[ "$EUID" -ne 0 ]]; then error "Must be run as root"; exit 1; fi
|
||||
validate_unraid_cmd "/usr/local/emhttp/plugins/dynamix/scripts/notify" "" "" "notify"
|
||||
acquire_lock
|
||||
detect_hosts
|
||||
|
||||
# Show status if requested
|
||||
if [[ "$SHOW_STATUS" == true ]]; then
|
||||
echo "Current state: ..."
|
||||
exit 0
|
||||
fi
|
||||
|
||||
[[ "$DRY_RUN" == true ]] && warn "DRY RUN — no changes will be made"
|
||||
|
||||
# Confirm before destructive operations
|
||||
read -r -p "Type YES to proceed: " CONFIRM
|
||||
[[ "$CONFIRM" != "YES" ]] && { warn "Aborted."; exit 0; }
|
||||
|
||||
# Do the work
|
||||
# ...
|
||||
|
||||
notify "Tool completed on $(hostname) ($MY_ID)" "Tool Name" "normal"
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## ━━━ PHILOSOPHY ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
|
||||
|
||||
```
|
||||
Write the tool when you solve the problem.
|
||||
Store it here.
|
||||
Find it at 2am when you need it again.
|
||||
|
||||
Tools exist because not every problem has a scheduled solution.
|
||||
Some things only need to happen once.
|
||||
Some things only happen after something goes wrong.
|
||||
Having a dedicated folder keeps the other folders clean —
|
||||
everything in Orchestrators, Docker_Essentials, and Monitors
|
||||
has a reason to run regularly.
|
||||
|
||||
Everything here has a reason to exist and wait.
|
||||
```
|
||||
@@ -2,47 +2,84 @@
|
||||
# ==============================================================================================
|
||||
# ============================= Bulk Permissions Repair ========================================
|
||||
# ==============================================================================================
|
||||
#
|
||||
# PURPOSE
|
||||
# ─────────────────────────────────────────────────────────────────────────────
|
||||
# Applies correct ownership and permissions to one or more specific paths.
|
||||
# Faster than running media_shares_permissions.sh which processes all configured shares.
|
||||
# Targeted repair — faster than media_shares_permissions.sh, which processes
|
||||
# every configured share. Use after failed transfers that left root:root ownership,
|
||||
# containers writing as root before PUID/PGID was fixed, manual file copies, or
|
||||
# new shares that need permissions applied before the next nightly run.
|
||||
#
|
||||
# ── WHEN TO USE ───────────────────────────────────────────────────────────────────────────────
|
||||
# Use for targeted repair after:
|
||||
# - A failed transfer that left files owned by wrong user (root:root from rsync)
|
||||
# - A container writing as root instead of nobody:users — before PUID/PGID was fixed
|
||||
# - Manual file copies that bypassed normal permission handling
|
||||
# - A new share that needs permissions applied before the next nightly run
|
||||
# - A large rsync that imported thousands of files before media_shares_permissions.sh ran
|
||||
# Counts files with wrong ownership before fixing. A high count on a recently
|
||||
# written share means a container has wrong PUID/PGID — add PUID=99 PGID=100
|
||||
# to its Docker template. Common culprits: SABnzbd, qBittorrent, slskd.
|
||||
#
|
||||
# ── PERMISSIONS MODEL ─────────────────────────────────────────────────────────────────────────
|
||||
# Directories: PERMISSIONS_DIR_MODE (default 755)
|
||||
# Owner (nobody) — rwx enter, list, create files
|
||||
# Group (users) — r-x enter and list
|
||||
# Others — r-x Samba guests can browse
|
||||
# ==============================================================================================
|
||||
# DESIGN PRINCIPLES
|
||||
# ==============================================================================================
|
||||
#
|
||||
# Files: PERMISSIONS_FILE_MODE (default 664)
|
||||
# Owner (nobody) — rw read + write
|
||||
# Group (users) — rw arrs can import and rename
|
||||
# Others — r Samba guests can read
|
||||
# No execute bit — media files are never executable
|
||||
# Permissions Model
|
||||
# Directories (PERMISSIONS_DIR_MODE, default 755):
|
||||
# Owner (nobody) — rwx enter, list, create files
|
||||
# Group (users) — r-x enter and list
|
||||
# Others — r-x Samba guests can browse
|
||||
# Files (PERMISSIONS_FILE_MODE, default 664):
|
||||
# Owner (nobody) — rw read + write
|
||||
# Group (users) — rw arrs can import and rename
|
||||
# Others — r Samba guests can read
|
||||
# No execute bit — media files are never executable
|
||||
#
|
||||
# ── DIAGNOSTIC — HIGH WRONG OWNER COUNT ───────────────────────────────────────────────────────
|
||||
# This script counts files with wrong ownership before applying the fix.
|
||||
# A high count on a share that was recently written → a container has wrong PUID/PGID.
|
||||
# Fix: add PUID=99 PGID=100 to the container's Docker template.
|
||||
# Common culprits: SABnzbd, qBittorrent, slskd.
|
||||
# Separate Passes
|
||||
# Directories and files are chmod'd in separate find passes. A combined pass
|
||||
# with mode 664 would wrongly strip the execute bit from directories, making
|
||||
# them untraversable.
|
||||
#
|
||||
# ── SAFEGUARDS ────────────────────────────────────────────────────────────────────────────────
|
||||
# Root check — required for chown
|
||||
# Path existence check — skips missing paths with error
|
||||
# Separate passes — directories and files chmod'd separately for correctness
|
||||
# validate_unraid_cmd — notify validated before use
|
||||
# Silent on success — only failures produce visible output
|
||||
# ==============================================================================================
|
||||
# OPERATIONAL SAFEGUARDS
|
||||
# ==============================================================================================
|
||||
#
|
||||
# Root Required
|
||||
# chown requires root — exits immediately if not running as root.
|
||||
#
|
||||
# Path Existence Check
|
||||
# Each path is verified before processing — missing paths log an error and
|
||||
# are skipped rather than silently passing.
|
||||
#
|
||||
# Notification Validated
|
||||
# validate_unraid_cmd confirms the notify script is present before use.
|
||||
#
|
||||
# Silent on Success
|
||||
# Only failures and the wrong-owner diagnostic produce visible output.
|
||||
#
|
||||
# ==============================================================================================
|
||||
# CONFIGURATION
|
||||
# ==============================================================================================
|
||||
#
|
||||
# master.conf
|
||||
#
|
||||
# PERMISSIONS_OWNER
|
||||
# Owner applied to all paths. (default: nobody:users)
|
||||
#
|
||||
# PERMISSIONS_DIR_MODE
|
||||
# chmod mode for directories. (default: 755)
|
||||
#
|
||||
# PERMISSIONS_FILE_MODE
|
||||
# chmod mode for files. (default: 664)
|
||||
#
|
||||
# ==============================================================================================
|
||||
# RUNTIME MODES
|
||||
# ==============================================================================================
|
||||
#
|
||||
# bulk_permissions_repair.sh /path/to/share [/another/path ...]
|
||||
# Apply ownership and permissions to each specified path.
|
||||
#
|
||||
# bulk_permissions_repair.sh /path/to/share --dry-run
|
||||
# Show wrong-owner count per path. No chown or chmod applied.
|
||||
#
|
||||
# bulk_permissions_repair.sh /path/to/share --log
|
||||
# Verbose output including per-path file counts and modes applied.
|
||||
#
|
||||
# ── USAGE ─────────────────────────────────────────────────────────────────────────────────────
|
||||
# bulk_permissions_repair.sh /mnt/user/Movies
|
||||
# bulk_permissions_repair.sh /mnt/user/Movies /mnt/user/Tv_Shows
|
||||
# bulk_permissions_repair.sh /mnt/user/Movies --dry-run
|
||||
# bulk_permissions_repair.sh /mnt/user/Movies --log
|
||||
# ==============================================================================================
|
||||
|
||||
SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
|
||||
|
||||
+24
-11
@@ -2,21 +2,34 @@
|
||||
# ==============================================================================================
|
||||
# ================================= Claude Code Startup ========================================
|
||||
# ==============================================================================================
|
||||
#
|
||||
# PURPOSE
|
||||
# ─────────────────────────────────────────────────────────────────────────────
|
||||
# Restores Claude Code's persistent data after an Unraid reboot and launches Claude.
|
||||
# Unraid's root filesystem lives in RAM — /root/.claude and /root/.local are wiped on
|
||||
# every reboot. This script symlinks both directories back to persistent appdata storage
|
||||
# before launching Claude, so memory, sessions, and settings survive across reboots.
|
||||
#
|
||||
# Unraid's root filesystem lives in RAM — /root/.claude and /root/.local are wiped on every
|
||||
# reboot. This script symlinks both directories back to persistent appdata storage before
|
||||
# launching Claude, so memory, sessions, and settings survive across reboots.
|
||||
# On first run with no existing persistent data, migrates from the current live locations:
|
||||
# /root/.claude → PERSIST_DIR/.claude (memory, sessions, settings)
|
||||
# /root/.local/share/claude → PERSIST_DIR/local/share/claude (installed binaries)
|
||||
# Subsequent runs skip the migration and only create the symlinks.
|
||||
#
|
||||
# ── FIRST RUN ─────────────────────────────────────────────────────────────────────────────────
|
||||
# If persistent storage has no data yet, migrates from the current live locations:
|
||||
# /root/.claude → PERSIST_DIR/.claude (memory, sessions, settings)
|
||||
# /root/.local/share/claude → PERSIST_DIR/local/share/claude (installed binaries)
|
||||
# Subsequent runs skip the migration and just create the symlinks.
|
||||
# Standalone script — no common.sh dependency. Safe to run directly from terminal
|
||||
# or from array_started.sh.
|
||||
#
|
||||
# ==============================================================================================
|
||||
# RUNTIME MODES
|
||||
# ==============================================================================================
|
||||
#
|
||||
# claude_startup.sh
|
||||
# Set up persistent symlinks and launch Claude.
|
||||
#
|
||||
# claude_startup.sh --setup
|
||||
# Set up persistent symlinks only — do not launch Claude.
|
||||
# Used by array_started.sh to prepare the environment on boot without
|
||||
# immediately launching an interactive session.
|
||||
#
|
||||
# ── USAGE ─────────────────────────────────────────────────────────────────────────────────────
|
||||
# claude_startup.sh — set up persistent symlinks and launch Claude
|
||||
# claude_startup.sh --setup — set up only, do not launch (for array_started.sh use)
|
||||
# ==============================================================================================
|
||||
|
||||
PERSIST_DIR="/mnt/user/appdata/claude-code"
|
||||
|
||||
@@ -2,46 +2,67 @@
|
||||
# ==============================================================================================
|
||||
# ============================= Container Data Export ==========================================
|
||||
# ==============================================================================================
|
||||
# Exports a container's appdata directory to a compressed tar archive.
|
||||
# Stops the container before archiving and restarts it after — ensures clean consistent backup.
|
||||
# Verifies the archive after creation — confirms backup is valid before restarting container.
|
||||
#
|
||||
# ── WHEN TO USE ───────────────────────────────────────────────────────────────────────────────
|
||||
# - Before major container updates (roll back if update goes wrong)
|
||||
# - Before pool migrations or disk replacements
|
||||
# - When archiving a container being removed from the stack
|
||||
# - Before destructive operations on appdata (database migrations etc.)
|
||||
# - One-off backup of a specific container without running full backup
|
||||
# PURPOSE
|
||||
# ─────────────────────────────────────────────────────────────────────────────
|
||||
# Exports a container's appdata directory to a compressed tar archive. Stops
|
||||
# the container before archiving and restarts it after — ensures a clean,
|
||||
# consistent backup. Use before major updates, pool migrations, destructive
|
||||
# appdata operations, or when archiving a container being removed from the stack.
|
||||
#
|
||||
# ── OUTPUT FILE NAMING ────────────────────────────────────────────────────────────────────────
|
||||
# ContainerName_YYYY-MM-DD_HH-MM.tar.gz
|
||||
# Timestamp in filename — run multiple times safely, no overwrite ✅
|
||||
# Output: ContainerName_YYYY-MM-DD_HH-MM.tar.gz — timestamped, no overwrite.
|
||||
#
|
||||
# ── SPACE CHECK ───────────────────────────────────────────────────────────────────────────────
|
||||
# Estimates required space as appdata size × 1.1 (10% buffer).
|
||||
# Compressed archive will typically be much smaller — this is a conservative floor.
|
||||
# gzip compression ratio depends heavily on content — database files compress well,
|
||||
# media files do not. If output is on a media share estimate may be pessimistic.
|
||||
# ==============================================================================================
|
||||
# DESIGN PRINCIPLES
|
||||
# ==============================================================================================
|
||||
#
|
||||
# ── ARCHIVE VERIFICATION ──────────────────────────────────────────────────────────────────────
|
||||
# After creation the archive is tested with tar --test-file before restarting the container.
|
||||
# If verification fails the container is still restarted (data unchanged) and an error logged.
|
||||
# A corrupt archive is not a usable backup — do not assume the archive is good without this.
|
||||
# Archive Verification Before Restart
|
||||
# The archive is tested with tar --test-file before the container is restarted.
|
||||
# A corrupt archive is not a usable backup — this catches tar failures, I/O
|
||||
# errors, and truncated writes before declaring success. If verification fails,
|
||||
# the container is still restarted (appdata is unchanged) and an error logged.
|
||||
#
|
||||
# ── SAFEGUARDS ────────────────────────────────────────────────────────────────────────────────
|
||||
# DOCKER_TIMEOUT — docker calls protected against hung daemon
|
||||
# Container restart rule — was running → restart | was stopped → leave stopped ✅
|
||||
# Archive cleanup — partial archive removed on tar failure
|
||||
# Archive verification — tar --test-file after creation
|
||||
# Container restart on — any failure path still restarts container if it was running
|
||||
# validate_unraid_cmd — notify validated before use
|
||||
# Silent on success — only problems produce visible output
|
||||
# Conservative Space Estimate
|
||||
# Required space is estimated as appdata size × 1.1 (10% buffer). The actual
|
||||
# compressed archive will typically be much smaller — database files compress
|
||||
# well, media files do not. The estimate is a conservative floor, not a
|
||||
# prediction.
|
||||
#
|
||||
# ==============================================================================================
|
||||
# OPERATIONAL SAFEGUARDS
|
||||
# ==============================================================================================
|
||||
#
|
||||
# Container Restart Rule
|
||||
# Tracks whether the container was running before the export. Running containers
|
||||
# are restarted after completion; already-stopped containers are left stopped.
|
||||
# The restart happens on every exit path — a failed tar does not leave the
|
||||
# container stuck stopped.
|
||||
#
|
||||
# Partial Archive Cleanup
|
||||
# If tar fails, the incomplete archive is removed. A partial archive is worse
|
||||
# than no archive — it can look valid but restore to an incomplete state.
|
||||
#
|
||||
# Docker Timeout
|
||||
# DOCKER_TIMEOUT (default: 30s) caps all docker calls. Guards against a hung
|
||||
# daemon blocking the script indefinitely.
|
||||
#
|
||||
# Notification Validated
|
||||
# validate_unraid_cmd confirms the notify script is present before use.
|
||||
#
|
||||
# ==============================================================================================
|
||||
# RUNTIME MODES
|
||||
# ==============================================================================================
|
||||
#
|
||||
# container_data_export.sh ContainerName /path/to/appdata /path/to/output/dir
|
||||
# Stop container, create archive, verify, restart container.
|
||||
# Example: container_data_export.sh Emby /mnt/media-servers/.../Emby /mnt/user/Backups/
|
||||
#
|
||||
# container_data_export.sh ContainerName /path/to/appdata /path/to/output/dir --dry-run
|
||||
# Show what would be archived and estimated size. No container stop, no tar.
|
||||
#
|
||||
# container_data_export.sh ContainerName /path/to/appdata /path/to/output/dir --log
|
||||
# Verbose output: space check, tar progress, verification result.
|
||||
#
|
||||
# ── USAGE ─────────────────────────────────────────────────────────────────────────────────────
|
||||
# container_data_export.sh ContainerName /path/to/appdata /path/to/output/dir
|
||||
# container_data_export.sh Emby /mnt/media-servers/Media_Server/Emby /mnt/user/Backups/
|
||||
# container_data_export.sh Emby /mnt/media-servers/Media_Server/Emby /mnt/user/Backups/ --dry-run
|
||||
# container_data_export.sh Emby /mnt/media-servers/Media_Server/Emby /mnt/user/Backups/ --log
|
||||
# ==============================================================================================
|
||||
|
||||
SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
|
||||
|
||||
@@ -2,43 +2,81 @@
|
||||
# ==============================================================================================
|
||||
# ============================= Emby Database Repair ===========================================
|
||||
# ==============================================================================================
|
||||
# Stops Emby, runs SQLite integrity checks on all Emby databases, and restarts.
|
||||
# Use when Emby reports database corruption, unexpected crashes, or playback state issues.
|
||||
#
|
||||
# ── CHECKS PERFORMED ──────────────────────────────────────────────────────────────────────────
|
||||
# PRAGMA integrity_check — full SQLite integrity verification per database
|
||||
# Skips missing databases gracefully — not all files exist on all setups
|
||||
# PURPOSE
|
||||
# ─────────────────────────────────────────────────────────────────────────────
|
||||
# Stops Emby, runs SQLite PRAGMA integrity_check on all Emby databases, and
|
||||
# restarts. Use when Emby reports corruption, unexpected crashes, or playback
|
||||
# state issues.
|
||||
#
|
||||
# ── DATABASES CHECKED ─────────────────────────────────────────────────────────────────────────
|
||||
# Reports which databases are corrupted. Does NOT automatically repair.
|
||||
# Repair requires manual steps — guidance is printed in the summary output.
|
||||
# Always take a backup before deleting any database file.
|
||||
#
|
||||
# ==============================================================================================
|
||||
# OPERATIONAL MODEL
|
||||
# ==============================================================================================
|
||||
#
|
||||
# Databases Checked
|
||||
# library.db — media library metadata (largest, most critical)
|
||||
# library.db-wal — write-ahead log (if exists — uncommitted transactions)
|
||||
# library.db-wal — write-ahead log (if present — uncommitted transactions)
|
||||
# librarydb.db — legacy library database
|
||||
# users.db — user accounts and settings
|
||||
# authentication.db — API keys and sessions
|
||||
# activity.db — activity log (least critical, safe to delete)
|
||||
# activity.db — activity log (least critical, safe to delete if corrupt)
|
||||
#
|
||||
# ── IF CORRUPTION FOUND ───────────────────────────────────────────────────────────────────────
|
||||
# Reports which databases are corrupted. Does NOT automatically repair.
|
||||
# Corruption repair requires manual steps — see guidance in summary output.
|
||||
# Always take a backup before deleting any database file.
|
||||
# Missing databases are skipped gracefully — not all files exist on all setups.
|
||||
# Emby's config path is detected from the Docker mount — no hardcoded paths.
|
||||
#
|
||||
# ── HOST AWARENESS ────────────────────────────────────────────────────────────────────────────
|
||||
# detect_hosts() sets MY_ID and aliases HOST*_EMBY_CONTAINER → EMBY_CONTAINER.
|
||||
# Each server checks its own Emby instance automatically.
|
||||
# ==============================================================================================
|
||||
# OPERATIONAL SAFEGUARDS
|
||||
# ==============================================================================================
|
||||
#
|
||||
# ── SAFEGUARDS ────────────────────────────────────────────────────────────────────────────────
|
||||
# EXIT trap — Emby always restarted even if script crashes mid-check
|
||||
# DOCKER_TIMEOUT — all docker calls protected against hung daemon
|
||||
# jq validation — verifies jq available before config path detection
|
||||
# validate_unraid_cmd — sqlite3 and notify validated before use
|
||||
# Container verify — checks Emby stayed running after restart
|
||||
# Silent healthy — only corruption produces visible output
|
||||
# Guaranteed Restart
|
||||
# EXIT trap ensures Emby is always restarted even if the script crashes
|
||||
# mid-check — Emby is never left stopped due to a script error.
|
||||
#
|
||||
# Docker Timeout
|
||||
# DOCKER_TIMEOUT (30s) protects all docker calls against a hung daemon.
|
||||
# Emby can take time to stop cleanly — 30s is intentionally generous.
|
||||
#
|
||||
# Tool Validation
|
||||
# validate_unraid_cmd confirms sqlite3 and the notify script are present
|
||||
# before use. jq is checked separately — required for config path detection.
|
||||
#
|
||||
# Post-Restart Verify
|
||||
# Checks that Emby is still running after restart — detects cases where
|
||||
# Emby crashes immediately after start (which would indicate deeper trouble).
|
||||
#
|
||||
# Silent When Healthy
|
||||
# Only corruption produces visible output and a notification.
|
||||
#
|
||||
# ==============================================================================================
|
||||
# CONFIGURATION
|
||||
# ==============================================================================================
|
||||
#
|
||||
# master_host*.conf
|
||||
#
|
||||
# HOST*_EMBY_CONTAINER
|
||||
# Name of the Emby Docker container on this host.
|
||||
# Aliased by detect_hosts() → EMBY_CONTAINER.
|
||||
#
|
||||
# ==============================================================================================
|
||||
# RUNTIME MODES
|
||||
# ==============================================================================================
|
||||
#
|
||||
# emby_database_repair.sh
|
||||
# Stop Emby, check all databases with PRAGMA integrity_check, restart.
|
||||
#
|
||||
# emby_database_repair.sh --dry-run
|
||||
# Show which databases would be checked and Emby container name. No stop.
|
||||
#
|
||||
# emby_database_repair.sh --log
|
||||
# Verbose output with per-database check result.
|
||||
#
|
||||
# emby_database_repair.sh --status
|
||||
# Show Emby container name and config path detected from Docker. Then exit.
|
||||
#
|
||||
# ── USAGE ─────────────────────────────────────────────────────────────────────────────────────
|
||||
# emby_database_repair.sh — stop Emby, check all databases, restart
|
||||
# emby_database_repair.sh --dry-run — show what would be checked, no Emby stop
|
||||
# emby_database_repair.sh --log — verbose output per database
|
||||
# emby_database_repair.sh --status — show config and exit
|
||||
# ==============================================================================================
|
||||
|
||||
SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
|
||||
|
||||
@@ -2,46 +2,66 @@
|
||||
# ==============================================================================================
|
||||
# ============================= Failover State Reset ===========================================
|
||||
# ==============================================================================================
|
||||
# Resets the fallback state file to NORMAL and clears all tier flags.
|
||||
# Use when the fallback state file is stuck in a non-NORMAL state after:
|
||||
# - Failover testing that left state as FALLBACK
|
||||
# - A failed handback that did not complete cleanly
|
||||
# - Manual intervention that left state inconsistent
|
||||
# - fallback.sh was killed mid-cycle and state is unknown
|
||||
#
|
||||
# ── WHAT THIS DOES ────────────────────────────────────────────────────────────────────────────
|
||||
# PURPOSE
|
||||
# ─────────────────────────────────────────────────────────────────────────────
|
||||
# Resets the fallback state file to NORMAL and clears all tier flags. Use when
|
||||
# the state file is stuck after failover testing, a failed handback, manual
|
||||
# intervention, or fallback.sh being killed mid-cycle.
|
||||
#
|
||||
# Writes a fresh state file with:
|
||||
# state=NORMAL
|
||||
# fallback_start=0
|
||||
# handback_strikes=0
|
||||
# state=NORMAL / fallback_start=0 / handback_strikes=0
|
||||
# tier2_started=false / tier3_started=false / tier4_started=false
|
||||
#
|
||||
# Does NOT start or stop any containers — state file only.
|
||||
# After reset, fallback.sh will resume from NORMAL on its next cycle.
|
||||
# Does NOT start or stop containers — state file only. After reset, fallback.sh
|
||||
# resumes from NORMAL on its next cycle.
|
||||
#
|
||||
# ── ⚠️ ONLY RUN WHEN SAFE ────────────────────────────────────────────────────────────────────
|
||||
# Verify BEFORE resetting:
|
||||
# ✓ Right containers running on the right server
|
||||
# ✓ DDNS pointing at the correct server
|
||||
# ✓ No active failover actually in progress
|
||||
# ✓ Both servers can see each other
|
||||
# WARNING: Only run when you have verified the stack is actually in a normal
|
||||
# state — right containers on the right server, DDNS correct, no active failover
|
||||
# in progress. Resetting state during a real failover causes fallback.sh to stop
|
||||
# covering the remote server until the next detection cycle.
|
||||
#
|
||||
# Resetting state while a real fallback is happening causes fallback.sh to stop
|
||||
# covering the remote server — services go offline until next detection cycle.
|
||||
# ==============================================================================================
|
||||
# OPERATIONAL SAFEGUARDS
|
||||
# ==============================================================================================
|
||||
#
|
||||
# ── SAFEGUARDS ────────────────────────────────────────────────────────────────────────────────
|
||||
# fallback.sh running check — warns if fallback.sh is active when reset is attempted
|
||||
# acquire_lock — prevents concurrent resets
|
||||
# flock on state write — prevents race with fallback.sh mid-cycle read
|
||||
# Confirmation required — interactive: type YES | non-interactive: --force flag
|
||||
# validate_unraid_cmd — notify validated before use
|
||||
# Active Fallback Detection
|
||||
# Checks whether fallback.sh is currently running and warns if so. A reset
|
||||
# during an active cycle causes fallback.sh to lose its state on the next read.
|
||||
#
|
||||
# Single Instance Lock
|
||||
# acquire_lock prevents concurrent resets.
|
||||
#
|
||||
# flock on State Write
|
||||
# The state file write is protected with flock — prevents a race condition
|
||||
# with fallback.sh reading the file mid-cycle.
|
||||
#
|
||||
# Confirmation Required
|
||||
# Interactive mode prompts for YES before writing. Use --force to bypass in
|
||||
# non-interactive contexts (cron, scripts).
|
||||
#
|
||||
# Notification Validated
|
||||
# validate_unraid_cmd confirms the notify script is present before use.
|
||||
#
|
||||
# ==============================================================================================
|
||||
# RUNTIME MODES
|
||||
# ==============================================================================================
|
||||
#
|
||||
# fallback_state_reset.sh
|
||||
# Show current state and prompt for YES before resetting.
|
||||
#
|
||||
# fallback_state_reset.sh --dry-run
|
||||
# Show current state and what the new state file would contain. No write.
|
||||
#
|
||||
# fallback_state_reset.sh --status
|
||||
# Show current state file contents and exit.
|
||||
#
|
||||
# fallback_state_reset.sh --force
|
||||
# Reset without interactive confirmation. Safe for scripted use.
|
||||
#
|
||||
# fallback_state_reset.sh --force --dry-run
|
||||
# Dry run without the confirmation prompt.
|
||||
#
|
||||
# ── USAGE ─────────────────────────────────────────────────────────────────────────────────────
|
||||
# fallback_state_reset.sh — interactive reset (prompts for YES)
|
||||
# fallback_state_reset.sh --dry-run — show current state, show what would be written
|
||||
# fallback_state_reset.sh --status — show current state file contents and exit
|
||||
# fallback_state_reset.sh --force — non-interactive reset (no prompt, use in scripts)
|
||||
# fallback_state_reset.sh --force --dry-run — dry run without prompt
|
||||
# ==============================================================================================
|
||||
|
||||
SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
|
||||
|
||||
+59
-34
@@ -2,52 +2,77 @@
|
||||
# ==============================================================================================
|
||||
# ============================= Recreate Shares ================================================
|
||||
# ==============================================================================================
|
||||
# Creates share directories on the correct disks after a fresh unRAID install or disk rebuild.
|
||||
# Reads all .cfg files from /boot/config/shares/ and creates the corresponding directories
|
||||
# on each disk listed in the shareInclude setting.
|
||||
#
|
||||
# ── WHEN TO USE ───────────────────────────────────────────────────────────────────────────────
|
||||
# Run directly on HOST2 after array is started following:
|
||||
# - A full disk replacement or rebuild where share folders were lost
|
||||
# - A fresh unRAID install where /boot/config/shares/*.cfg files were restored
|
||||
# - Any situation where the share folder structure exists in config but not on disk
|
||||
# PURPOSE
|
||||
# ─────────────────────────────────────────────────────────────────────────────
|
||||
# Creates share directories on the correct disks after a fresh unRAID install
|
||||
# or disk rebuild. Reads all .cfg files from /boot/config/shares/ and creates
|
||||
# the corresponding directories on each disk listed in the shareInclude setting.
|
||||
# The array must be started before running — /mnt/user must be mounted.
|
||||
#
|
||||
# The array must be started before running this script — /mnt/user must be mounted.
|
||||
# Typically run on HOST2 after a full disk replacement or fresh install where
|
||||
# share folders were lost but /boot/config/shares/*.cfg files were restored.
|
||||
#
|
||||
# ==============================================================================================
|
||||
# OPERATIONAL MODEL
|
||||
# ==============================================================================================
|
||||
#
|
||||
# ── WHAT IT DOES ──────────────────────────────────────────────────────────────────────────────
|
||||
# For each share .cfg file:
|
||||
# 1. Reads shareInclude= to determine which disks own this share
|
||||
# 2. Creates /mnt/diskN/ShareName/ on each listed disk if it doesn't exist
|
||||
# 3. Places a .recovery marker file in /mnt/user/ShareName/ via the union filesystem
|
||||
#
|
||||
# ── .RECOVERY MARKER FILE ─────────────────────────────────────────────────────────────────────
|
||||
# The .recovery marker signals to rsync.sh that this is a fresh share with no existing data.
|
||||
# rsync.sh checks for .recovery before running with --delete:
|
||||
# .recovery present → rsync WITHOUT --delete (safe — new files only, nothing removed)
|
||||
# .recovery absent → rsync WITH --delete (normal — mirror mode)
|
||||
# .recovery Marker
|
||||
# Signals to rsync.sh that this is a fresh share with no existing data.
|
||||
# rsync.sh checks for .recovery before running with --delete:
|
||||
# .recovery present → rsync WITHOUT --delete (new files only, nothing removed)
|
||||
# .recovery absent → rsync WITH --delete (normal mirror mode)
|
||||
#
|
||||
# The marker self-cleans: after the first successful rsync the source side has no .recovery
|
||||
# file so the second nightly run will delete it from the mirror, restoring normal --delete
|
||||
# behaviour automatically. No manual cleanup needed. ✅
|
||||
# Self-cleaning: after the first successful rsync the source side has no .recovery
|
||||
# file, so the second nightly run deletes it from the mirror, restoring normal
|
||||
# --delete behaviour automatically. No manual cleanup needed.
|
||||
#
|
||||
# ── HOST AWARENESS ────────────────────────────────────────────────────────────────────────────
|
||||
# This script runs on the server that needs shares recreated — typically HOST2 during rebuild.
|
||||
# detect_hosts() sets MY_ID for output clarity.
|
||||
# ==============================================================================================
|
||||
# OPERATIONAL SAFEGUARDS
|
||||
# ==============================================================================================
|
||||
#
|
||||
# ── SAFEGUARDS ────────────────────────────────────────────────────────────────────────────────
|
||||
# acquire_lock — prevents duplicate runs placing duplicate markers
|
||||
# Root check — mkdir on /mnt/diskN requires root
|
||||
# Array mount check — exits cleanly if array not started
|
||||
# Empty cfg guard — warns if no share cfg files found
|
||||
# Per-disk guards — skips missing disks with warning, continues others
|
||||
# validate_unraid_cmd — notify validated before use
|
||||
# Silent on success — only failures produce visible output
|
||||
# Single Instance Lock
|
||||
# acquire_lock prevents duplicate runs placing duplicate .recovery markers.
|
||||
#
|
||||
# Root Required
|
||||
# mkdir on /mnt/diskN requires root.
|
||||
#
|
||||
# Array Mount Check
|
||||
# Exits cleanly if the array is not started — /mnt/user not mounted means
|
||||
# all share operations would fail silently.
|
||||
#
|
||||
# Per-Disk Guards
|
||||
# Missing disks are skipped with a warning and the rest continue — a single
|
||||
# offline disk does not abort the full run.
|
||||
#
|
||||
# Empty Config Guard
|
||||
# Warns if no share .cfg files are found — catches the case where
|
||||
# /boot/config/shares/ was not restored.
|
||||
#
|
||||
# Notification Validated
|
||||
# validate_unraid_cmd confirms the notify script is present before use.
|
||||
#
|
||||
# ==============================================================================================
|
||||
# RUNTIME MODES
|
||||
# ==============================================================================================
|
||||
#
|
||||
# recreate_shares.sh
|
||||
# Read all .cfg files, create share directories, place .recovery markers.
|
||||
#
|
||||
# recreate_shares.sh --dry-run
|
||||
# Show what directories and markers would be created. No changes.
|
||||
#
|
||||
# recreate_shares.sh --log
|
||||
# Verbose output per share and per disk.
|
||||
#
|
||||
# recreate_shares.sh --status
|
||||
# Show which shares exist in config and which directories exist on disk.
|
||||
#
|
||||
# ── USAGE ─────────────────────────────────────────────────────────────────────────────────────
|
||||
# recreate_shares.sh — create all shares from .cfg files
|
||||
# recreate_shares.sh --dry-run — preview what would be created, no changes
|
||||
# recreate_shares.sh --log — verbose output per disk
|
||||
# recreate_shares.sh --status — show current share state and exit
|
||||
# ==============================================================================================
|
||||
|
||||
SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
|
||||
|
||||
@@ -2,51 +2,73 @@
|
||||
# ==============================================================================================
|
||||
# =========================== Watchdog Skip List Manager =======================================
|
||||
# ==============================================================================================
|
||||
#
|
||||
# PURPOSE
|
||||
# ─────────────────────────────────────────────────────────────────────────────
|
||||
# View and manage the persistent container skip list used by docker_watchdog.sh.
|
||||
# docker_watchdog.sh adds a container to the skip list when it exceeds
|
||||
# WATCHDOG_CONTAINER_RESTART_LIMIT restarts within WATCHDOG_CONTAINER_RESTART_WINDOW
|
||||
# hours — prevents infinite restart loops on containers that keep crashing.
|
||||
#
|
||||
# ── WHAT THE SKIP LIST IS ─────────────────────────────────────────────────────────────────────
|
||||
# docker_watchdog.sh adds a container to the skip list when it exceeds the restart loop
|
||||
# limit (WATCHDOG_CONTAINER_RESTART_LIMIT in WATCHDOG_CONTAINER_RESTART_WINDOW hours).
|
||||
# Once on the skip list the watchdog stops restarting it — prevents infinite restart loops.
|
||||
# Skip list persists on /boot/config (survives reboots). Auto-clears when
|
||||
# docker_watchdog.sh sees the container running on a later cycle. Use this
|
||||
# script to clear manually after fixing the underlying problem.
|
||||
#
|
||||
# Skip list persists on /boot/config — survives reboots.
|
||||
# Auto-clears when docker_watchdog.sh sees the container running on a cycle.
|
||||
# This script clears it manually when you have fixed the underlying problem.
|
||||
# ==============================================================================================
|
||||
# OPERATIONAL MODEL
|
||||
# ==============================================================================================
|
||||
#
|
||||
# ── ACTIONS ───────────────────────────────────────────────────────────────────────────────────
|
||||
# --status — show skip list, container states, restart history
|
||||
# --clear ContainerName — clear a specific container from skip list + history
|
||||
# --clear-all — clear all skip lists and restart history
|
||||
# Skip List Lifecycle
|
||||
# 1. Container crashes repeatedly → watchdog adds to skip list, notifies
|
||||
# 2. Watchdog stops restarting the container on subsequent cycles
|
||||
# 3a. If container recovers on its own (Docker restart policy), watchdog
|
||||
# sees it running, removes from skip list automatically
|
||||
# 3b. If stuck stopped → fix the root cause, clear via this script, then
|
||||
# docker start ContainerName manually
|
||||
# 4. Watchdog monitors normally on next cycle. If it crashes again → re-added.
|
||||
#
|
||||
# ── AFTER CLEARING ────────────────────────────────────────────────────────────────────────────
|
||||
# 1. Fix whatever was causing the container to fail
|
||||
# 2. Start it manually: docker start ContainerName
|
||||
# 3. docker_watchdog.sh monitors it normally on the next cycle
|
||||
# 4. If it crashes again → watchdog adds it back and notifies
|
||||
# ==============================================================================================
|
||||
# OPERATIONAL SAFEGUARDS
|
||||
# ==============================================================================================
|
||||
#
|
||||
# ── SKIP LIST AUTO-CLEAR ──────────────────────────────────────────────────────────────────────
|
||||
# docker_watchdog.sh auto-clears a container from the skip list when it sees it running.
|
||||
# So if a container recovers on its own (Docker restart policy eventually works),
|
||||
# the watchdog will see it running, remove it from the skip list, and resume monitoring.
|
||||
# Manual clear only needed when container is stuck stopped and needs intervention.
|
||||
# Single Instance Lock
|
||||
# acquire_lock prevents concurrent access with docker_watchdog.sh writing
|
||||
# the same files.
|
||||
#
|
||||
# ── SAFEGUARDS ────────────────────────────────────────────────────────────────────────────────
|
||||
# acquire_lock — prevents concurrent access with docker_watchdog.sh writing files
|
||||
# docker_watchdog check — warns if watchdog is running during clear (could re-add instantly)
|
||||
# DOCKER_TIMEOUT — docker inspect calls protected against daemon hangs
|
||||
# Confirmation required — interactive: YES | non-interactive: --force flag
|
||||
# validate_unraid_cmd — notify validated before use
|
||||
# Active Watchdog Detection
|
||||
# Warns if docker_watchdog.sh is currently running when a clear is attempted
|
||||
# — the watchdog could re-add the container to the skip list within seconds.
|
||||
#
|
||||
# ── FILES MANAGED ─────────────────────────────────────────────────────────────────────────────
|
||||
# SYS_WATCHDOG_FAILED_FILE — persistent container skip list
|
||||
# WATCHDOG_CONTAINER_RESTART_LOG — restart history for loop detection
|
||||
# Docker Timeout
|
||||
# DOCKER_TIMEOUT caps docker inspect calls against a hung daemon.
|
||||
#
|
||||
# Confirmation Required
|
||||
# Interactive mode prompts for YES before clearing. Use --force for scripts.
|
||||
#
|
||||
# ==============================================================================================
|
||||
# STATE FILES
|
||||
# ==============================================================================================
|
||||
#
|
||||
# SYS_WATCHDOG_FAILED_FILE — persistent container skip list (on /boot/config)
|
||||
# WATCHDOG_CONTAINER_RESTART_LOG — restart history used for loop detection
|
||||
#
|
||||
# ==============================================================================================
|
||||
# RUNTIME MODES
|
||||
# ==============================================================================================
|
||||
#
|
||||
# watchdog_skip_list_manager.sh [--status]
|
||||
# Show skip list, container states, and recent restart history.
|
||||
#
|
||||
# watchdog_skip_list_manager.sh --clear ContainerName
|
||||
# Remove a specific container from the skip list and clear its restart history.
|
||||
# Prompts for YES unless --force is passed.
|
||||
#
|
||||
# watchdog_skip_list_manager.sh --clear-all
|
||||
# Clear all skip lists and all restart history.
|
||||
# Prompts for YES unless --force is passed.
|
||||
#
|
||||
# All actions support --dry-run (show what would change) and --force (skip prompt).
|
||||
#
|
||||
# ── USAGE ─────────────────────────────────────────────────────────────────────────────────────
|
||||
# watchdog_skip_list_manager.sh — show status
|
||||
# watchdog_skip_list_manager.sh --status — show status explicitly
|
||||
# watchdog_skip_list_manager.sh --clear ContainerName — clear specific container
|
||||
# watchdog_skip_list_manager.sh --clear-all — clear everything
|
||||
# Any action supports --dry-run and --force
|
||||
# ==============================================================================================
|
||||
|
||||
SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
|
||||
|
||||
+67
-34
@@ -2,47 +2,80 @@
|
||||
# ==============================================================================================
|
||||
# ================================= ZFS Pool Scrub ============================================
|
||||
# ==============================================================================================
|
||||
# Triggers a ZFS scrub on all pools (or a specific pool) and waits for completion.
|
||||
# Sends a notification when scrub completes with a summary of any errors found.
|
||||
#
|
||||
# ── WHAT ZFS SCRUB DOES ───────────────────────────────────────────────────────────────────────
|
||||
# Reads every block on every pool and verifies checksums against the stored hash.
|
||||
# Catches silent data corruption that would otherwise only surface when you read the
|
||||
# corrupted data — by then it may be too late for redundancy to help.
|
||||
# PURPOSE
|
||||
# ─────────────────────────────────────────────────────────────────────────────
|
||||
# Triggers a ZFS scrub on all pools (or a specific pool), waits for completion,
|
||||
# and sends a notification with any errors found. Reads every block on every pool
|
||||
# and verifies checksums — catches silent corruption that would otherwise only
|
||||
# surface when the corrupted data is read (possibly after redundancy can no
|
||||
# longer help). Monthly recommended for all pools; quarterly minimum for large pools.
|
||||
#
|
||||
# Scrub is safe to run while the pool is in use — it does not interrupt normal I/O.
|
||||
# It does consume I/O bandwidth — run during off-peak hours or maintenance windows.
|
||||
# Monthly is recommended for all pools. Quarterly minimum for large pools.
|
||||
# ==============================================================================================
|
||||
# OPERATIONAL MODEL
|
||||
# ==============================================================================================
|
||||
#
|
||||
# ── BEHAVIOUR ─────────────────────────────────────────────────────────────────────────────────
|
||||
# Starts scrub on each pool then polls every 60 seconds until all complete.
|
||||
# Progress shown via warn() every poll (visible) when scrub is running.
|
||||
# Safe to leave running or interrupt — scrub continues even if script is stopped.
|
||||
# On completion reports errors per pool and notifies if any found.
|
||||
# Starts a scrub on each pool, then polls every 60 seconds until all complete.
|
||||
# Progress is shown every poll — safe to leave running or interrupt. ZFS scrub
|
||||
# continues in the kernel even if the script is stopped — it does not depend on
|
||||
# this script remaining alive.
|
||||
#
|
||||
# ── HOST AWARENESS ────────────────────────────────────────────────────────────────────────────
|
||||
# detect_hosts() sets MY_ID and aliases HOST*_ZFS_REPORT_IGNORE_POOLS → ZFS_REPORT_IGNORE_POOLS.
|
||||
# Pools in ZFS_REPORT_IGNORE_POOLS are skipped (single-disk VMs, temp pools etc.)
|
||||
# unless specified explicitly as a positional argument.
|
||||
# Scrub is safe to run while the pool is in use. It does consume I/O bandwidth —
|
||||
# schedule during off-peak hours or maintenance windows.
|
||||
#
|
||||
# ── SAFEGUARDS ────────────────────────────────────────────────────────────────────────────────
|
||||
# acquire_lock — prevents concurrent scrub starts on same server
|
||||
# detect_hosts() — correct pool ignore list per host
|
||||
# validate_unraid_cmd — zpool and notify validated before use
|
||||
# Scrub-in-progress check — skips pools already scrubbing rather than erroring
|
||||
# SIGTERM trap — poll loop exits cleanly on signal
|
||||
# Silent when clean — only errors produce visible output and notification
|
||||
# Pools in HOST*_ZFS_REPORT_IGNORE_POOLS are skipped automatically (single-disk
|
||||
# VM pools, temp pools, etc.). Specifying a pool by name bypasses the ignore list.
|
||||
#
|
||||
# ── CONFIGURATION (master_host*.conf) ─────────────────────────────────────────────────────────
|
||||
# HOST*_ZFS_REPORT_IGNORE_POOLS — pools excluded from automatic scrub
|
||||
# Aliased by detect_hosts() — script uses ZFS_REPORT_IGNORE_POOLS
|
||||
# ==============================================================================================
|
||||
# OPERATIONAL SAFEGUARDS
|
||||
# ==============================================================================================
|
||||
#
|
||||
# Single Instance Lock
|
||||
# acquire_lock prevents concurrent scrub starts on the same server.
|
||||
#
|
||||
# Scrub-in-Progress Check
|
||||
# Detects pools already scrubbing and skips them rather than erroring — safe
|
||||
# to run when a scrub may have been started by another path.
|
||||
#
|
||||
# SIGTERM Trap
|
||||
# The poll loop exits cleanly on signal. The ZFS scrub continues regardless.
|
||||
#
|
||||
# Tool Validation
|
||||
# validate_unraid_cmd confirms zpool and the notify script are present before use.
|
||||
#
|
||||
# Silent When Clean
|
||||
# Only errors produce visible output and a notification.
|
||||
#
|
||||
# ==============================================================================================
|
||||
# CONFIGURATION
|
||||
# ==============================================================================================
|
||||
#
|
||||
# master_host*.conf
|
||||
#
|
||||
# HOST*_ZFS_REPORT_IGNORE_POOLS
|
||||
# Pools to exclude from automatic scrub. Typically single-disk VM pools
|
||||
# or temporary pools that do not need integrity checking.
|
||||
# Aliased by detect_hosts() → ZFS_REPORT_IGNORE_POOLS.
|
||||
#
|
||||
# ==============================================================================================
|
||||
# RUNTIME MODES
|
||||
# ==============================================================================================
|
||||
#
|
||||
# zfs_pool_scrub.sh
|
||||
# Scrub all pools not in ZFS_REPORT_IGNORE_POOLS. Wait for completion.
|
||||
#
|
||||
# zfs_pool_scrub.sh poolname
|
||||
# Scrub a specific pool by name. Bypasses the ignore list.
|
||||
#
|
||||
# zfs_pool_scrub.sh --status
|
||||
# Show current scrub status for all pools and exit.
|
||||
#
|
||||
# zfs_pool_scrub.sh --dry-run
|
||||
# Show which pools would be scrubbed. No scrub started.
|
||||
#
|
||||
# zfs_pool_scrub.sh --log
|
||||
# Verbose progress output every poll cycle.
|
||||
#
|
||||
# ── USAGE ─────────────────────────────────────────────────────────────────────────────────────
|
||||
# zfs_pool_scrub.sh — scrub all non-ignored pools
|
||||
# zfs_pool_scrub.sh poolname — scrub specific pool (bypasses ignore list)
|
||||
# zfs_pool_scrub.sh --status — show scrub status for all pools
|
||||
# zfs_pool_scrub.sh --dry-run — show what would be scrubbed
|
||||
# zfs_pool_scrub.sh --log — verbose progress output
|
||||
# ==============================================================================================
|
||||
|
||||
SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
|
||||
|
||||
Reference in New Issue
Block a user