834 lines
46 KiB
Markdown
834 lines
46 KiB
Markdown
# ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
|
||
# 🔧 TOOLS
|
||
# ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
|
||
|
||
**Situational utilities — run when something needs fixing, not on a schedule.**
|
||
Recovery, repair, migration, cleanup, and one-time tasks that don't fit the scheduled
|
||
maintenance model. These scripts sit ready for the moment you actually need them.
|
||
|
||
```
|
||
unRAID_Essentials/ ← regular system maintenance — scheduled
|
||
Docker_Essentials/ ← regular container management — scheduled
|
||
Monitors/ ← regular health reporting — scheduled
|
||
Orchestrators/ ← regular maintenance windows — scheduled
|
||
Tools/ ← situational utilities — run when needed
|
||
```
|
||
|
||
> **None of these scripts run on a schedule.** A script belongs here when it solves
|
||
> a specific operational situation rather than ongoing maintenance — something you run
|
||
> in response to a problem, a planned migration, or a one-time task. Having a dedicated
|
||
> folder keeps the other folders clean and makes it obvious what runs routinely vs what
|
||
> runs situationally.
|
||
|
||
---
|
||
|
||
## ━━━ THE PROBLEM THAT BUILT THIS ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
|
||
|
||
Every tool here exists because a specific situation arose that required bash commands
|
||
to resolve — and that situation is guaranteed to arise again.
|
||
|
||
---
|
||
|
||
### 🔴 Failover State Stuck After Testing
|
||
|
||
Run `failover_test.sh`, something goes wrong mid-test, script exits uncleanly. State
|
||
file shows `FAILOVER`. `failover.sh` resumes and reads FAILOVER — starts containers
|
||
it shouldn't start, makes decisions based on a state that doesn't reflect reality.
|
||
Or the test completed but handback didn't finish — state is partially reset.
|
||
|
||
Manual recovery: edit the state file by hand? Know the exact format? Know which
|
||
fields to reset? At 2am after a failed test, none of that is obvious.
|
||
|
||
The tool: `failover_state_reset.sh` — one command, shows you the current state before
|
||
asking for confirmation, resets cleanly to NORMAL, explains exactly what it changed.
|
||
|
||
---
|
||
|
||
### 🔴 Container Stuck on Watchdog Skip List After Fixing the Problem
|
||
|
||
Authelia hit the restart loop limit — three restarts in an hour — went on the skip list.
|
||
You fixed the underlying database issue. But the watchdog still isn't monitoring it
|
||
because it's on the skip list and you don't know where that file lives or what format
|
||
it's in. You restart Authelia manually, it runs fine, but the watchdog has no idea
|
||
it recovered and still thinks it's broken.
|
||
|
||
The tool: `watchdog_skip_list_manager.sh` — shows the skip list, shows which containers
|
||
are stopped vs running, clears specific containers with a single command, clears restart
|
||
history so the loop protection window starts fresh.
|
||
|
||
---
|
||
|
||
### 🔴 Emby Crashing With No Clear Cause After a Power Cut
|
||
|
||
Server lost power with Emby running. Emby comes back up, runs for 20 minutes, crashes.
|
||
Comes back up, crashes again. Logs show database errors. Which database? library.db?
|
||
users.db? authentication.db? They're all SQLite, they all need a different recovery
|
||
approach, and the errors aren't always obvious from the log output alone.
|
||
|
||
The tool: `emby_database_repair.sh` — stops Emby, runs `PRAGMA integrity_check` on
|
||
every database, reports per database what's clean and what's corrupted with specific
|
||
guidance on what to do about each one.
|
||
|
||
---
|
||
|
||
### 🔴 Files Owned by Root After an Admin Copy
|
||
|
||
`scp` a file from another machine directly into a media share. File arrives as
|
||
`root:root`. Radarr tries to move it and fails — permission denied. The daily
|
||
permissions script won't run for another 20 hours. Running the full
|
||
`media_shares_permissions.sh` on the whole share takes 30 minutes on a large library
|
||
just to fix one directory.
|
||
|
||
The tool: `bulk_permissions_repair.sh` — takes a specific path or list of paths,
|
||
applies correct ownership and permissions in seconds, done.
|
||
|
||
---
|
||
|
||
### 🔴 No Way to Back Up a Container Before a Risky Update
|
||
|
||
Container has a major version update. The changelog says "database migration — no
|
||
rollback." You want a point-in-time backup before you proceed. But the container's
|
||
appdata is scattered across dozens of files and `cp -r` while it's running produces
|
||
an inconsistent backup.
|
||
|
||
The tool: `container_data_export.sh` — stops the container cleanly, archives the
|
||
entire appdata directory to a timestamped `.tar.gz`, verifies the archive integrity,
|
||
restarts the container. The backup is valid and complete before anything else happens.
|
||
If the update goes wrong you have a clean restore point.
|
||
|
||
---
|
||
|
||
### 🔴 Fresh HOST2 Has Shares Configured But Directories Missing
|
||
|
||
Fresh install on HOST2. Restored `/boot/config/shares/` cfg files from backup. Array
|
||
starts. Shares show in the UI. But the actual `/mnt/diskN/sharename` directories don't
|
||
exist on the individual disks — unRAID created the share definitions but not the
|
||
directories. rsync.sh tries to write, finds the path doesn't exist, aborts.
|
||
|
||
The tool: `recreate_shares.sh` — reads every `.cfg` file from `/boot/config/shares/`,
|
||
parses the `shareInclude` list, creates the correct directory on each included disk.
|
||
Run once after fresh setup, directories exist, rsync works.
|
||
|
||
---
|
||
|
||
## ━━━ SCRIPTS AT A GLANCE ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
|
||
|
||
| Script | What It Fixes | When to Run |
|
||
|--------|--------------|-------------|
|
||
| `failover_state_reset.sh` | State file stuck in FAILOVER after test or failed handback | After failover testing or manual intervention |
|
||
| `watchdog_skip_list_manager.sh` | Container stuck on watchdog skip list | After fixing a container that hit restart loop limit |
|
||
| `bulk_permissions_repair.sh` | Files owned by wrong user after admin copy or bad container | When arr operations fail due to permissions |
|
||
| `container_data_export.sh` | Need a clean backup before a risky container update | Before major updates, migrations, or removals |
|
||
| `emby_database_repair.sh` | Emby crashing with database errors after power loss | When Emby logs show corruption or repeated crashes |
|
||
| `zfs_pool_scrub.sh` | Verify ZFS pool integrity — catch silent corruption | Monthly, or after any disk/power event |
|
||
| `recreate_shares.sh` | Share directories missing on fresh install or rebuild | After fresh unRAID install or disk replacement |
|
||
| `rsync_stop.sh` | rsync stuck or needs emergency stop | When rsync is running and must be stopped cleanly |
|
||
| `user_scripts_stop.sh` | User Scripts running mid-cycle and need stopping | Before planned reboots, emergency stop |
|
||
| `server_reboot.sh` | Graceful reboot with pre-flight warnings and clean shutdown | Planned maintenance reboots |
|
||
|
||
---
|
||
|
||
## ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
|
||
## 🔀 failover_state_reset.sh
|
||
## ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
|
||
|
||
Resets the failover state file to NORMAL and clears all tier flags. State file only —
|
||
does NOT start or stop any containers.
|
||
|
||
> Full documentation in `README-Failover.md` — `failover_state_reset.sh` section.
|
||
> This entry is a quick reference.
|
||
|
||
---
|
||
|
||
### ── When to Use ──────────────────────────────────────────────────────────────
|
||
|
||
```
|
||
After failover_test.sh didn't complete cleanly
|
||
→ state left in FAILOVER but containers are actually back to normal
|
||
|
||
After a failed handback
|
||
→ state shows FAILOVER but remote is back up and containers are split
|
||
|
||
After killing failover.sh directly (not via User Scripts Abort)
|
||
→ state is unknown, cycle was interrupted mid-operation
|
||
|
||
After a dev/debug session
|
||
→ state left in a non-NORMAL state from testing
|
||
```
|
||
|
||
---
|
||
|
||
### ── Verify Before Resetting ────────────────────────────────────────────────
|
||
|
||
```bash
|
||
# ─────────────────────────────────────────────────────────────────────────────
|
||
# Verify reality matches what you're about to declare as NORMAL:
|
||
#
|
||
# Right containers on right server?
|
||
# DDNS pointing correctly? nslookup Gmer4Lfe.com
|
||
# failover.sh not running? pgrep -f "failover.sh"
|
||
# Both servers Tailscale connected? tailscale status
|
||
#
|
||
# Resetting during an actual failover causes failover.sh to think everything
|
||
# is normal and stop covering the remote — services go offline until the next
|
||
# detection cycle catches it again.
|
||
# ─────────────────────────────────────────────────────────────────────────────
|
||
```
|
||
|
||
---
|
||
|
||
### ── Usage ───────────────────────────────────────────────────────────────────
|
||
|
||
```bash
|
||
failover_state_reset.sh --status # show current state file contents — always first
|
||
failover_state_reset.sh --dry-run # show what would be written, no write
|
||
failover_state_reset.sh # interactive reset — prompts "YES" to confirm
|
||
failover_state_reset.sh --force # non-interactive — for scripts, no terminal
|
||
```
|
||
|
||
---
|
||
|
||
## ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
|
||
## 🐳 watchdog_skip_list_manager.sh
|
||
## ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
|
||
|
||
View and manage the persistent container skip list used by `docker_watchdog.sh`.
|
||
|
||
> Full documentation in `README-Docker_Essentials.md` — `watchdog_skip_list_manager.sh`
|
||
> section including the full recovery workflow. This entry is a quick reference.
|
||
|
||
---
|
||
|
||
### ── When to Use ──────────────────────────────────────────────────────────────
|
||
|
||
```
|
||
docker_watchdog.sh restarts the same container N times within the rolling window
|
||
→ container added to skip list on /boot/config/
|
||
→ critical notification sent
|
||
→ watchdog stops touching it entirely
|
||
|
||
You fix the underlying problem (database, config, dependencies).
|
||
You need to clear the container from the skip list so monitoring resumes.
|
||
```
|
||
|
||
---
|
||
|
||
### ── Recovery Workflow ────────────────────────────────────────────────────────
|
||
|
||
```bash
|
||
# 1. Understand the situation — always start here:
|
||
watchdog_skip_list_manager.sh --status
|
||
# Shows: skip list contents, which are running vs stopped, restart history
|
||
|
||
# 2. Fix the underlying problem first
|
||
# Check logs: docker logs ContainerName --tail 100
|
||
# Check disk: df -h /mnt/user
|
||
# Check db: docker exec ContainerName sqlite3 /path/to.db ".tables"
|
||
|
||
# 3. Clear from skip list + restart history:
|
||
watchdog_skip_list_manager.sh --clear ContainerName
|
||
|
||
# 4. Start the container manually — confirm your fix worked:
|
||
docker start ContainerName
|
||
|
||
# 5. Watchdog resumes normal monitoring on next cycle — no further action needed
|
||
```
|
||
|
||
---
|
||
|
||
### ── Files Managed ────────────────────────────────────────────────────────────
|
||
|
||
```bash
|
||
# ─────────────────────────────────────────────────────────────────────────────
|
||
/boot/config/system_watchdog_failed.db # persistent skip list
|
||
/boot/config/container_restart_history.db # restart loop tracking
|
||
|
||
# Both live on /boot/config — survive reboots intentionally.
|
||
# A container that was skip-listed before a reboot is still broken after it.
|
||
# ─────────────────────────────────────────────────────────────────────────────
|
||
```
|
||
|
||
---
|
||
|
||
### ── Usage ───────────────────────────────────────────────────────────────────
|
||
|
||
```bash
|
||
watchdog_skip_list_manager.sh # show status (default)
|
||
watchdog_skip_list_manager.sh --status # explicit status
|
||
watchdog_skip_list_manager.sh --clear ContainerName # clear specific + restart history
|
||
watchdog_skip_list_manager.sh --clear ContainerName --force # no confirmation prompt
|
||
watchdog_skip_list_manager.sh --clear-all # clear everything
|
||
watchdog_skip_list_manager.sh --clear-all --force # non-interactive
|
||
watchdog_skip_list_manager.sh --dry-run # preview any clear action
|
||
```
|
||
|
||
---
|
||
|
||
## ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
|
||
## 🔐 bulk_permissions_repair.sh
|
||
## ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
|
||
|
||
Applies correct ownership and permissions to specific paths. Faster than running
|
||
`media_shares_permissions.sh` which processes every configured share — use this when
|
||
you know exactly what needs fixing and don't want to wait for a full library walk.
|
||
|
||
---
|
||
|
||
### ── When to Use ──────────────────────────────────────────────────────────────
|
||
|
||
```bash
|
||
# ─────────────────────────────────────────────────────────────────────────────
|
||
# Use instead of the full permissions script when:
|
||
#
|
||
# Admin copy left root:root files — scp, cp, direct file transfer
|
||
# New share needs permissions now — can't wait for nightly run
|
||
# Container wrote as root — before PUID/PGID was fixed
|
||
# Specific directory has wrong perms — targeted fix, not a full library walk
|
||
#
|
||
# The full media_shares_permissions.sh is the right tool for:
|
||
# Regular nightly maintenance (already scheduled in daily_sync_maintenance.sh)
|
||
# After confirming a container's PUID/PGID is now correct
|
||
# Initial permissions setup on a new server
|
||
# ─────────────────────────────────────────────────────────────────────────────
|
||
```
|
||
|
||
---
|
||
|
||
### ── What It Applies ──────────────────────────────────────────────────────────
|
||
|
||
```bash
|
||
# master.conf
|
||
# ─────────────────────────────────────────────────────────────────────────────
|
||
# Same values as media_shares_permissions.sh — consistent permissions everywhere
|
||
PERMISSIONS_DIR_MODE="755" # directories — enter, list, no world-write
|
||
PERMISSIONS_FILE_MODE="664" # files — owner+group rw, others read-only
|
||
PERMISSIONS_OWNER="nobody:users" # matches PUID=99 PGID=100 in containers
|
||
```
|
||
|
||
---
|
||
|
||
### ── Usage ───────────────────────────────────────────────────────────────────
|
||
|
||
```bash
|
||
# ─────────────────────────────────────────────────────────────────────────────
|
||
# Single path:
|
||
bulk_permissions_repair.sh /mnt/user/Movies
|
||
|
||
# Multiple paths — all corrected in one run:
|
||
bulk_permissions_repair.sh /mnt/user/Movies /mnt/user/Tv_Shows /mnt/user/Music
|
||
|
||
# Dry run first — shows count of files that would be corrected:
|
||
bulk_permissions_repair.sh /mnt/user/Movies --dry-run
|
||
|
||
# Verbose — show each corrected file:
|
||
bulk_permissions_repair.sh /mnt/user/Movies --log
|
||
# ─────────────────────────────────────────────────────────────────────────────
|
||
```
|
||
|
||
---
|
||
|
||
## ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
|
||
## 📦 container_data_export.sh
|
||
## ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
|
||
|
||
Exports a container's appdata directory to a compressed tar archive. Stops the
|
||
container first for a clean consistent backup, verifies the archive after creation,
|
||
then restarts the container.
|
||
|
||
---
|
||
|
||
### ── When to Use ──────────────────────────────────────────────────────────────
|
||
|
||
```bash
|
||
# ─────────────────────────────────────────────────────────────────────────────
|
||
# Before major container updates — especially those with "no rollback" database migrations
|
||
# Before pool migrations — clean backup before moving appdata to a new pool
|
||
# Before removing a container from the stack — archive its data before deletion
|
||
# Manual point-in-time backup before risky config changes
|
||
# ─────────────────────────────────────────────────────────────────────────────
|
||
```
|
||
|
||
---
|
||
|
||
### ── Sequence ─────────────────────────────────────────────────────────────────
|
||
|
||
```bash
|
||
# ─────────────────────────────────────────────────────────────────────────────
|
||
# 1. Space check
|
||
# Estimates required space from appdata size × 1.1
|
||
# Aborts if output directory doesn't have enough free space
|
||
# Container is NOT stopped until the space check passes
|
||
#
|
||
# 2. Stop container cleanly
|
||
# docker stop ContainerName — graceful shutdown
|
||
#
|
||
# 3. Create archive
|
||
# tar -czf ContainerName_YYYY-MM-DD_HH-MM.tar.gz /path/to/appdata
|
||
#
|
||
# 4. Verify archive integrity
|
||
# tar -tzf archive.tar.gz — confirms archive is valid and complete
|
||
# If verification fails → restart container anyway, report error
|
||
#
|
||
# 5. Restart container
|
||
# docker start ContainerName — always restarted, even if archiving failed
|
||
# ─────────────────────────────────────────────────────────────────────────────
|
||
```
|
||
|
||
---
|
||
|
||
### ── Usage ───────────────────────────────────────────────────────────────────
|
||
|
||
```bash
|
||
# Syntax: container_data_export.sh ContainerName AppDataPath OutputDir
|
||
|
||
# Emby backup example:
|
||
container_data_export.sh \
|
||
Emby \
|
||
/mnt/media-servers/Media_Server/Emby \
|
||
/mnt/user/Backups/
|
||
|
||
# Dry run — verify space and paths without stopping anything:
|
||
container_data_export.sh \
|
||
Emby \
|
||
/mnt/media-servers/Media_Server/Emby \
|
||
/mnt/user/Backups/ \
|
||
--dry-run
|
||
|
||
# Output filename format: ContainerName_YYYY-MM-DD_HH-MM.tar.gz
|
||
```
|
||
|
||
---
|
||
|
||
## ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
|
||
## 🎬 emby_database_repair.sh
|
||
## ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
|
||
|
||
Stops Emby, runs SQLite `PRAGMA integrity_check` on every Emby database, and restarts.
|
||
Reports per-database with specific guidance on what to do if corruption is found.
|
||
|
||
---
|
||
|
||
### ── When to Use ──────────────────────────────────────────────────────────────
|
||
|
||
```
|
||
Emby logs show database errors → run this first
|
||
Emby crashing repeatedly with no clear cause → likely database corruption
|
||
Playback history or user data behaving strangely → users.db or library.db issue
|
||
After a hard shutdown or power loss with Emby running → check for WAL corruption
|
||
```
|
||
|
||
---
|
||
|
||
### ── Databases Checked ────────────────────────────────────────────────────────
|
||
|
||
```bash
|
||
# ─────────────────────────────────────────────────────────────────────────────
|
||
# Each database has different recovery implications:
|
||
#
|
||
# library.db — media library metadata: titles, seasons, episodes, artwork
|
||
# CORRUPT → safe to delete — Emby fully rebuilds from media files on next start
|
||
# Rebuild takes time but loses nothing permanent
|
||
#
|
||
# users.db — user accounts, watch history, playback positions, settings
|
||
# CORRUPT → deleting resets ALL user accounts and watch history
|
||
# Check if you have a recent backup (weekly_sync_maintenance.sh)
|
||
# before deleting
|
||
#
|
||
# authentication.db — API keys, session tokens
|
||
# CORRUPT → safe to delete — API keys regenerated on restart
|
||
# Any connected clients will need to re-authenticate
|
||
#
|
||
# activity.db — activity/access log
|
||
# CORRUPT → safe to delete — it's a log, losing it is acceptable
|
||
#
|
||
# ─────────────────────────────────────────────────────────────────────────────
|
||
# This script checks and reports ONLY. It does NOT automatically delete or repair
|
||
# corrupted databases. Recovery requires judgment — and potentially a backup restore.
|
||
# The summary provides specific guidance per database type.
|
||
# ─────────────────────────────────────────────────────────────────────────────
|
||
```
|
||
|
||
---
|
||
|
||
### ── Usage ───────────────────────────────────────────────────────────────────
|
||
|
||
```bash
|
||
# ─────────────────────────────────────────────────────────────────────────────
|
||
# Normal run — stops Emby, checks all databases, restarts:
|
||
emby_database_repair.sh
|
||
|
||
# Dry run — detect config path and show what would be checked, no Emby stop:
|
||
emby_database_repair.sh --dry-run
|
||
|
||
# Verbose — show SQLite output for each database:
|
||
emby_database_repair.sh --log
|
||
|
||
# Status — show Emby config path and database file locations:
|
||
emby_database_repair.sh --status
|
||
# ─────────────────────────────────────────────────────────────────────────────
|
||
# Config path is detected automatically from Docker volume mounts.
|
||
# No configuration needed — just run it.
|
||
```
|
||
|
||
---
|
||
|
||
## ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
|
||
## 🗄️ zfs_pool_scrub.sh
|
||
## ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
|
||
|
||
Triggers ZFS scrub on all pools (or a specific named pool) and waits for completion.
|
||
Notifies when done with a summary of any errors found.
|
||
|
||
---
|
||
|
||
### ── What ZFS Scrub Does ──────────────────────────────────────────────────────
|
||
|
||
```bash
|
||
# ─────────────────────────────────────────────────────────────────────────────
|
||
# ZFS stores a checksum with every block of data. Scrub reads every block on
|
||
# every pool and verifies the checksum matches the stored hash.
|
||
#
|
||
# Why this matters:
|
||
# Silent data corruption can sit on disk for months without triggering any
|
||
# error — until you try to read that specific file. By then:
|
||
# - It may already be mirrored to HOST2 in its corrupted state
|
||
# - The original source may no longer exist
|
||
# - The corruption may have spread if it was a drive issue
|
||
#
|
||
# ZFS can self-repair during scrub if redundancy exists — RAIDZ or mirrors.
|
||
# It cannot repair if you have a single-disk pool (JBOD).
|
||
# But it will tell you corruption exists before you find out the hard way.
|
||
#
|
||
# Recommended: run monthly or after any disk replacement / power event.
|
||
# Safe to run while the system is in use — scrub runs at low I/O priority.
|
||
# ─────────────────────────────────────────────────────────────────────────────
|
||
```
|
||
|
||
---
|
||
|
||
### ── Pool Filtering ───────────────────────────────────────────────────────────
|
||
|
||
```bash
|
||
# master_host1.conf
|
||
# ─────────────────────────────────────────────────────────────────────────────
|
||
# Single-disk JBOD members can be excluded from all-pool scrubs.
|
||
# To scrub a pool that's in the ignore list: specify it by name explicitly.
|
||
#
|
||
HOST1_ZFS_REPORT_IGNORE_POOLS=(
|
||
"disk10" # JBOD member — no redundancy, scrub still useful but excluded from default
|
||
"disk9"
|
||
"disk8"
|
||
)
|
||
```
|
||
|
||
---
|
||
|
||
### ── Usage ───────────────────────────────────────────────────────────────────
|
||
|
||
```bash
|
||
# Scrub all pools except those in ZFS_REPORT_IGNORE_POOLS:
|
||
zfs_pool_scrub.sh
|
||
|
||
# Scrub a specific pool by name — ignores the ignore list:
|
||
zfs_pool_scrub.sh gaming
|
||
|
||
# Check current scrub status without starting a new one:
|
||
zfs_pool_scrub.sh --status
|
||
|
||
# Dry run — show which pools would be scrubbed:
|
||
zfs_pool_scrub.sh --dry-run
|
||
```
|
||
|
||
---
|
||
|
||
## ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
|
||
## 📁 recreate_shares.sh
|
||
## ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
|
||
|
||
Creates share directories on the correct disks after a fresh install or disk rebuild.
|
||
Reads `.cfg` files from `/boot/config/shares/` and creates the corresponding
|
||
`/mnt/diskN/sharename` directory on each disk listed in the `shareInclude` setting.
|
||
|
||
---
|
||
|
||
### ── When to Use ──────────────────────────────────────────────────────────────
|
||
|
||
```bash
|
||
# ─────────────────────────────────────────────────────────────────────────────
|
||
# After a fresh unRAID install where /boot/config/shares/*.cfg were restored:
|
||
# The share definitions exist → UI shows shares → directories are missing
|
||
# rsync.sh tries to write to /mnt/user/Movies → path doesn't exist → aborts
|
||
#
|
||
# After a disk replacement or rebuild where share folders were lost:
|
||
# Replacement disk is blank → no share directories on the new disk
|
||
# unRAID won't create them automatically
|
||
#
|
||
# Run once on HOST2 after fresh setup, before the first rsync from HOST1.
|
||
# ─────────────────────────────────────────────────────────────────────────────
|
||
```
|
||
|
||
---
|
||
|
||
### ── What It Does ─────────────────────────────────────────────────────────────
|
||
|
||
```bash
|
||
# ─────────────────────────────────────────────────────────────────────────────
|
||
# For each .cfg file in /boot/config/shares/:
|
||
# 1. Read the share name (e.g. Movies)
|
||
# 2. Read the shareInclude list (e.g. disk1,disk2,disk5)
|
||
# 3. Create /mnt/disk1/Movies, /mnt/disk2/Movies, /mnt/disk5/Movies
|
||
# 4. Set correct ownership: nobody:users
|
||
#
|
||
# Does not create content — just the directories.
|
||
# rsync.sh can then write into them normally.
|
||
# ─────────────────────────────────────────────────────────────────────────────
|
||
```
|
||
|
||
---
|
||
|
||
### ── Usage ───────────────────────────────────────────────────────────────────
|
||
|
||
```bash
|
||
recreate_shares.sh # create all missing share directories
|
||
recreate_shares.sh --dry-run # show what would be created without creating
|
||
recreate_shares.sh --log # verbose — show each directory created
|
||
recreate_shares.sh --status # show share configs and current directory state
|
||
```
|
||
|
||
---
|
||
|
||
## ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
|
||
## 🔄 rsync_stop.sh
|
||
## ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
|
||
|
||
Stops rsync intelligently on both local and remote servers. Auto-detects if an
|
||
orchestrator is running and chooses the safest stop strategy automatically.
|
||
|
||
---
|
||
|
||
### ── Two Modes ────────────────────────────────────────────────────────────────
|
||
|
||
```bash
|
||
# ─────────────────────────────────────────────────────────────────────────────
|
||
# Smart (default — auto-detected):
|
||
# Orchestrator detected → kill rsync subprocess only
|
||
# Orchestrator sees rsync died → moves to next share or exits cleanly
|
||
# No orphaned lock files, orchestrator exits naturally
|
||
#
|
||
# --full-stop:
|
||
# Kill orchestrator first, then rsync
|
||
# Orchestrator will NOT continue to next share
|
||
# Use when: you need everything stopped immediately
|
||
#
|
||
# Why smart is usually correct:
|
||
# Killing the orchestrator directly (daily_sync_maintenance.sh) leaves it
|
||
# mid-execution. Containers may be stopped but not restarted. Lock files
|
||
# may not be released. The smart approach lets the orchestrator clean up
|
||
# after itself — fewer side effects.
|
||
# ─────────────────────────────────────────────────────────────────────────────
|
||
```
|
||
|
||
---
|
||
|
||
### ── When to Use ──────────────────────────────────────────────────────────────
|
||
|
||
```
|
||
rsync running during a window where it shouldn't be → smart stop
|
||
rsync stuck with no progress → smart stop
|
||
Need to start a manual sync that conflicts → smart stop first
|
||
Everything must stop NOW (emergency) → --full-stop
|
||
Called by partnership_manage.sh --offboard → --rsync-only flag
|
||
```
|
||
|
||
---
|
||
|
||
### ── Usage ───────────────────────────────────────────────────────────────────
|
||
|
||
```bash
|
||
rsync_stop.sh # smart stop — auto-detect orchestrator
|
||
rsync_stop.sh --full-stop # kill orchestrator + rsync
|
||
rsync_stop.sh --rsync-only # stop rsync, skip container recovery
|
||
rsync_stop.sh --dry-run # preview without stopping anything
|
||
rsync_stop.sh --status # show what's currently running
|
||
rsync_stop.sh --full-stop --dry-run # preview full stop
|
||
```
|
||
|
||
---
|
||
|
||
## ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
|
||
## 🛑 user_scripts_stop.sh
|
||
## ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
|
||
|
||
Stops all running User Script processes spawned by the unRAID User Scripts plugin.
|
||
Identifies processes by their `/tmp/user.scripts` path signature, shows script names
|
||
not just PIDs, uses SIGTERM → SIGKILL sequence with verification.
|
||
|
||
---
|
||
|
||
### ── When to Use ──────────────────────────────────────────────────────────────
|
||
|
||
```bash
|
||
# ─────────────────────────────────────────────────────────────────────────────
|
||
# A script is stuck and won't respond to the Abort button in the User Scripts UI
|
||
# → the UI button sends a signal that the script may have trapped or ignored
|
||
# → user_scripts_stop.sh finds the process by path signature, not by UI state
|
||
#
|
||
# Before a planned reboot to ensure scripts exit cleanly
|
||
# → server_reboot.sh calls this automatically as part of its shutdown sequence
|
||
#
|
||
# Emergency stop of all background ecosystem scripts
|
||
# → stops system_watchdog, docker_watchdog, failover, and any running maintenance
|
||
# → use when you need to take manual control immediately
|
||
# ─────────────────────────────────────────────────────────────────────────────
|
||
```
|
||
|
||
---
|
||
|
||
### ── Self-Exclusion ────────────────────────────────────────────────────────────
|
||
|
||
```bash
|
||
# ─────────────────────────────────────────────────────────────────────────────
|
||
# If this script is run via the User Scripts plugin it would find its own PID.
|
||
# Self-exclusion prevents the script from killing itself mid-execution.
|
||
# Own PID and parent PID are excluded before any killing begins.
|
||
# ─────────────────────────────────────────────────────────────────────────────
|
||
```
|
||
|
||
---
|
||
|
||
### ── Usage ───────────────────────────────────────────────────────────────────
|
||
|
||
```bash
|
||
user_scripts_stop.sh # stop all — SIGTERM → verify → SIGKILL if needed
|
||
user_scripts_stop.sh --dry-run # show which scripts would be stopped, by name
|
||
user_scripts_stop.sh --status # show currently running scripts with PIDs and runtime
|
||
user_scripts_stop.sh --log # verbose — show each signal and verification step
|
||
```
|
||
|
||
---
|
||
|
||
## ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
|
||
## 🔁 server_reboot.sh
|
||
## ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
|
||
|
||
Graceful reboot with pre-flight warnings, wall message, unRAID notification, clean
|
||
shutdown sequence, and VM graceful shutdown before stopping services.
|
||
|
||
---
|
||
|
||
### ── When to Use ──────────────────────────────────────────────────────────────
|
||
|
||
```bash
|
||
# ─────────────────────────────────────────────────────────────────────────────
|
||
# Planned maintenance reboots — gives users notice and shuts down cleanly
|
||
# After kernel or firmware updates that require a reboot
|
||
# As an alternative to the unRAID UI reboot — more visibility into state
|
||
#
|
||
# NOT needed for: system_watchdog.sh triggered reboots (those use /sbin/reboot
|
||
# directly after their own shutdown sequence)
|
||
# ─────────────────────────────────────────────────────────────────────────────
|
||
```
|
||
|
||
---
|
||
|
||
### ── Shutdown Sequence ────────────────────────────────────────────────────────
|
||
|
||
```bash
|
||
# ─────────────────────────────────────────────────────────────────────────────
|
||
# 1. Pre-flight warnings (warn not block):
|
||
# rsync running → "partial files possible, consider rsync_stop.sh"
|
||
# mover running → "files may be left mid-move, consider mover_stop.sh"
|
||
# Emby sessions active → "N streams will be interrupted"
|
||
#
|
||
# 2. Wall message → terminal users
|
||
# 3. unRAID notification → dashboard
|
||
# 4. Wait REBOOT_SLEEP seconds (default 30) — users can save work
|
||
# 5. virsh shutdown each running VM → wait REBOOT_VM_WAIT seconds
|
||
# 6. Stop libvirt (VM Manager)
|
||
# 7. Stop Docker service
|
||
# 8. sync — flush filesystem buffers
|
||
# 9. /sbin/reboot
|
||
# ─────────────────────────────────────────────────────────────────────────────
|
||
```
|
||
|
||
---
|
||
|
||
### ── Usage ───────────────────────────────────────────────────────────────────
|
||
|
||
```bash
|
||
server_reboot.sh # reboot with 30s warning
|
||
server_reboot.sh --dry-run # walk through sequence without rebooting
|
||
server_reboot.sh --status # show running processes that would be affected
|
||
server_reboot.sh --reason="maintenance" # include reason in wall + notification
|
||
server_reboot.sh --log # verbose output per shutdown step
|
||
```
|
||
|
||
---
|
||
|
||
## ━━━ ADDING A NEW TOOL ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
|
||
|
||
```bash
|
||
# ─────────────────────────────────────────────────────────────────────────────
|
||
# When you encounter a situation that required manual bash commands to resolve —
|
||
# write a tool. You'll face it again. The cost of writing the tool is 30 minutes.
|
||
# The cost of reconstructing the commands at 2am is much higher.
|
||
#
|
||
# Checklist for a new Tools script:
|
||
#
|
||
# ✓ Header explains the specific situation that requires this tool
|
||
# ✓ "When to Use" section — exactly the symptoms that trigger this
|
||
# ✓ Root check — most tools need root
|
||
# ✓ --dry-run support — always
|
||
# ✓ --status support — show current state before acting
|
||
# ✓ Confirmation for destructive operations (read -p "Type YES:")
|
||
# ✓ Notify on completion — success and failure
|
||
# ✓ Leave system in clean state on any exit — trap for cleanup
|
||
# ─────────────────────────────────────────────────────────────────────────────
|
||
|
||
# Minimal skeleton:
|
||
#!/bin/bash
|
||
SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
|
||
source "$SCRIPT_DIR/../load_config.sh"
|
||
parse_args "$@"
|
||
|
||
if [[ "$EUID" -ne 0 ]]; then error "Must be run as root"; exit 1; fi
|
||
validate_unraid_cmd "/usr/local/emhttp/plugins/dynamix/scripts/notify" "" "" "notify"
|
||
acquire_lock
|
||
detect_hosts
|
||
|
||
# Show status if requested
|
||
if [[ "$SHOW_STATUS" == true ]]; then
|
||
echo "Current state: ..."
|
||
exit 0
|
||
fi
|
||
|
||
[[ "$DRY_RUN" == true ]] && warn "DRY RUN — no changes will be made"
|
||
|
||
# Confirm before destructive operations
|
||
read -r -p "Type YES to proceed: " CONFIRM
|
||
[[ "$CONFIRM" != "YES" ]] && { warn "Aborted."; exit 0; }
|
||
|
||
# Do the work
|
||
# ...
|
||
|
||
notify "Tool completed on $(hostname) ($MY_ID)" "Tool Name" "normal"
|
||
```
|
||
|
||
---
|
||
|
||
## ━━━ PHILOSOPHY ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
|
||
|
||
```
|
||
Write the tool when you solve the problem.
|
||
Store it here.
|
||
Find it at 2am when you need it again.
|
||
|
||
Tools exist because not every problem has a scheduled solution.
|
||
Some things only need to happen once.
|
||
Some things only happen after something goes wrong.
|
||
Having a dedicated folder keeps the other folders clean —
|
||
everything in Orchestrators, Docker_Essentials, and Monitors
|
||
has a reason to run regularly.
|
||
|
||
Everything here has a reason to exist and wait.
|
||
``` |