Varaverk/Tools/Manual-Tools.md

# ━━━━━ TOOLS — Manual ━━━━━

Configuration reference, usage procedures, and field guides for every script
in `Tools/`. Run any script with `--status` first — it shows current state before
making any changes.

---

## ━━━ CONTENTS ━━━

- [emby_to_lidarr_sync.sh](#emby_to_lidarr_syncsh)
- [emby_to_sonarr_sync.sh](#emby_to_sonarr_syncsh)
- [emby_to_radarr_sync.sh](#emby_to_radarr_syncsh)
- [failover_state_reset.sh](#failover_state_resetsh)
- [watchdog_skip_list_manager.sh](#watchdog_skip_list_managersh)
- [bulk_permissions_repair.sh](#bulk_permissions_repairsh)
- [container_data_export.sh](#container_data_exportsh)
- [emby_database_repair.sh](#emby_database_repairsh)
- [zfs_pool_scrub.sh](#zfs_pool_scrubsh)
- [recreate_shares.sh](#recreate_sharessh)
- [continuous_scripts_status.sh](#continuous_scripts_statussh)
- [claude_startup.sh](#claude_startupsh)
- [Adding a New Tool](#adding-a-new-tool)

---

## emby_to_lidarr_sync.sh

One-shot bootstrap tool. Scans Emby play history, finds artists you've actually
listened to that are not yet tracked in Lidarr, and adds them. No scoring — if
it was played, Lidarr should monitor it. Not scheduled; run manually when you
want to close the gap between what's in your library and what Lidarr watches.

### When to Use

- After initial Lidarr setup — bring it in line with existing listening history
- After a Lidarr database wipe or migration
- Any time you suspect artists you listen to are slipping through unmonitored

### Usage

```bash
# See what would be added (no changes)
bash Tools/emby_to_lidarr_sync.sh --dry-run

# Limit to recent plays only
bash Tools/emby_to_lidarr_sync.sh --dry-run --days 30

# Run for real
bash Tools/emby_to_lidarr_sync.sh
```

### Notes

- Reads completions from the Emby activity log (`has finished playing` events)
- Filters out VA, Various Artists, and other metadata placeholders
- Uses `searchForMissingAlbums: false` — adds monitoring without triggering a
  full album search; run Lidarr's own missing-album search afterwards if desired
- Activity log has a finite history; use `--days N` if the log has been pruned

---

## emby_to_sonarr_sync.sh

One-shot bootstrap tool. Finds TV series present in Emby that are not tracked in
Sonarr and adds them. Uses TVDB ID matching when available (more reliable than
title matching), falling back to case-insensitive title comparison.

### When to Use

- After initial Sonarr setup — bring it in line with your existing library
- After a Sonarr database wipe or migration
- Any time series you own are slipping through unmonitored

### Usage

```bash
# See what would be added (no changes)
bash Tools/emby_to_sonarr_sync.sh --dry-run

# Run for real
bash Tools/emby_to_sonarr_sync.sh
```

### Notes

- Uses `searchForMissingEpisodes: false` — adds monitoring without triggering
  episode searches; Sonarr's library scan will pick up existing files
- Adds to the first accessible root folder in Sonarr (`/tv` by default)
- TVDB ID match preferred over title; title fallback handles edge cases

---

## emby_to_radarr_sync.sh

One-shot bootstrap tool. Finds movies present in Emby that are not tracked in
Radarr and adds them. Uses TMDB ID matching when available, falling back to
case-insensitive title comparison.

### When to Use

- After initial Radarr setup — bring it in line with your existing library
- After a Radarr database wipe or migration
- Any time movies you own are slipping through unmonitored

### Usage

```bash
# See what would be added (no changes)
bash Tools/emby_to_radarr_sync.sh --dry-run

# Run for real
bash Tools/emby_to_radarr_sync.sh
```

### Notes

- Uses `searchForMovie: false` — adds monitoring without triggering movie
  searches; Radarr's library scan will pick up existing files
- Adds to the first accessible root folder in Radarr (`/movies` by default)
- TMDB ID match preferred over title; title fallback handles edge cases

---

## failover_state_reset.sh

Resets the fallback state file to NORMAL and clears all tier flags. State file only —
does NOT start or stop any containers.

### When to Use

```
After failover_test.sh didn't complete cleanly
  → state left in FALLBACK but containers are actually back to normal

After a failed handback
  → state shows FALLBACK but remote is back up and containers are split

After killing fallback.sh directly (not via User Scripts Abort)
  → state is unknown, cycle was interrupted mid-operation

After a dev/debug session
  → state left in a non-NORMAL state from testing
```

### Verify Before Resetting

Run `--status` first and check each of these before writing:

```bash
# Right containers on right server?
continuous_scripts_status.sh        # shows failover current state

# DDNS pointing correctly?
nslookup Gmer4Lfe.com               # confirm it resolves to the right IP

# fallback.sh not running?
pgrep -f "fallback.sh"              # empty output = not running

# Both servers Tailscale connected?
tailscale status                    # both hosts should show active
```

Resetting during an actual failover causes fallback.sh to think everything is normal
and stop covering the remote — services go offline until the next detection cycle.

### Usage

```bash
fallback_state_reset.sh --status     # show current state file — always check first
fallback_state_reset.sh --dry-run    # show what would be written, no write
fallback_state_reset.sh              # interactive reset — prompts for YES to confirm
fallback_state_reset.sh --force      # non-interactive — for scripts, no terminal
```

### What Gets Written

```bash
# New state file after reset:
state=NORMAL
fallback_start=0
handback_strikes=0
tier2_started=false
tier3_started=false
tier4_started=false
```

---

## watchdog_skip_list_manager.sh

View and manage the persistent container skip list used by `docker_watchdog.sh`.

### When to Use

```
docker_watchdog.sh restarts the same container N times within the rolling window
→ container added to skip list on /boot/config/
→ critical notification sent
→ watchdog stops touching it entirely

You fix the underlying problem (database, config, dependencies).
You need to clear the container from the skip list so monitoring resumes.
```

### Recovery Workflow

```bash
# 1. Understand the situation — always start here:
watchdog_skip_list_manager.sh --status
# Shows: skip list contents, which are running vs. stopped, restart history

# 2. Fix the underlying problem first
#    Check logs:    docker logs ContainerName --tail 100
#    Check disk:    df -h /mnt/user
#    Check db:      docker exec ContainerName sqlite3 /path/to.db ".tables"

# 3. Clear from skip list + restart history:
watchdog_skip_list_manager.sh --clear ContainerName

# 4. Start the container manually — confirm your fix worked:
docker start ContainerName

# 5. Watchdog resumes normal monitoring on next cycle — no further action needed
```

### State Files Managed

```bash
# Both live on /boot/config — survive reboots intentionally.
# A container that was skip-listed before a reboot is still broken after it.

$SYS_WATCHDOG_FAILED_FILE          # persistent skip list
$WATCHDOG_CONTAINER_RESTART_LOG    # restart loop tracking
```

### Configuration (master.conf)

```bash
WATCHDOG_CONTAINER_RESTART_LIMIT=3     # restarts before skip-listing
WATCHDOG_CONTAINER_RESTART_WINDOW=1    # rolling window in hours
```

### Usage

```bash
watchdog_skip_list_manager.sh                        # show status (default)
watchdog_skip_list_manager.sh --status               # explicit status
watchdog_skip_list_manager.sh --clear ContainerName  # clear specific + restart history
watchdog_skip_list_manager.sh --clear ContainerName --force   # no confirmation prompt
watchdog_skip_list_manager.sh --clear-all            # clear everything
watchdog_skip_list_manager.sh --clear-all --force    # non-interactive
watchdog_skip_list_manager.sh --dry-run              # preview any clear action
```

---

## bulk_permissions_repair.sh

Applies correct ownership and permissions to specific paths. Faster than running
`media_shares_permissions.sh` which processes every configured share — use this when
you know exactly what needs fixing and don't want to wait for a full library walk.

### When to Use

```
Admin copy left root:root files       — scp, cp, direct file transfer
New share needs permissions now       — can't wait for nightly run
Container wrote as root               — before PUID/PGID was fixed
Specific directory has wrong perms    — targeted fix, not a full library walk
```

Use the full `media_shares_permissions.sh` instead for:
- Regular nightly maintenance (already scheduled in daily_sync_maintenance.sh)
- After confirming a container's PUID/PGID is now correct
- Initial permissions setup on a new server

### Diagnosing High Wrong-Owner Counts

The script counts files with wrong ownership before applying the fix. A high count
on a share that was recently written means a container has wrong PUID/PGID.

```bash
# Fix: add to the container's Docker template:
PUID=99
PGID=100

# Common culprits writing as root:
#   SABnzbd, qBittorrent, slskd — check each one's Docker env vars
```

### Configuration (master.conf)

```bash
PERMISSIONS_OWNER="nobody:users"    # matches PUID=99 PGID=100
PERMISSIONS_DIR_MODE="755"          # directories — enter, list, no world-write
PERMISSIONS_FILE_MODE="664"         # files — owner+group rw, others read-only
```

### Usage

```bash
# Single path:
bulk_permissions_repair.sh /mnt/user/Movies

# Multiple paths — all corrected in one run:
bulk_permissions_repair.sh /mnt/user/Movies /mnt/user/Tv_Shows /mnt/user/Music

# Dry run first — shows count of files with wrong ownership per path:
bulk_permissions_repair.sh /mnt/user/Movies --dry-run

# Verbose — show each corrected file:
bulk_permissions_repair.sh /mnt/user/Movies --log
```

---

## container_data_export.sh

Exports a container's appdata directory to a compressed tar archive. Stops the
container first for a clean consistent backup, verifies the archive after creation,
then restarts the container.

### When to Use

```
Before major container updates — especially "database migration — no rollback" changelogs
Before pool migrations — clean backup before moving appdata to a new pool
Before removing a container from the stack — archive its data before deletion
Manual point-in-time backup before risky config changes
```

### Export Sequence

```
1. Space check
   Estimates required space from appdata size × 1.1
   Aborts if output directory doesn't have enough free space
   Container is NOT stopped until the space check passes

2. Stop container cleanly
   docker stop ContainerName — graceful shutdown

3. Create archive
   tar -czf ContainerName_YYYY-MM-DD_HH-MM.tar.gz /path/to/appdata

4. Verify archive integrity
   tar --test-file archive.tar.gz — confirms archive is valid and complete
   If verification fails → restart container anyway, report error

5. Restart container
   docker start ContainerName — always happens, even if archiving failed
```

### Usage

```bash
# Syntax: container_data_export.sh ContainerName AppDataPath OutputDir

# Emby backup:
container_data_export.sh \
    Emby \
    /mnt/media-servers/Media_Server/Emby \
    /mnt/user/Backups/

# Dry run — verify space and paths without stopping anything:
container_data_export.sh \
    Emby \
    /mnt/media-servers/Media_Server/Emby \
    /mnt/user/Backups/ \
    --dry-run

# Output filename: Emby_2026-05-14_02-30.tar.gz
# Timestamped — safe to run multiple times, no overwrite
```

---

## emby_database_repair.sh

Stops Emby, runs SQLite `PRAGMA integrity_check` on every Emby database, and restarts.
Reports per-database — does NOT automatically repair. Recovery requires judgment.

### When to Use

```
Emby logs show database errors                     → run this first
Emby crashing repeatedly with no clear cause       → likely database corruption
Playback history or user data behaving strangely   → users.db or library.db issue
After a hard shutdown or power loss with Emby running → check for WAL corruption
```

### Recovery Guide by Database

```
library.db          — media library metadata: titles, seasons, episodes, artwork
  CORRUPT → safe to delete — Emby fully rebuilds from media files on next start
            Rebuild takes time (hours on large libraries) but loses nothing permanent

users.db            — user accounts, watch history, playback positions, settings
  CORRUPT → deleting resets ALL user accounts and watch history
            Check for a recent backup (weekly_sync_maintenance.sh mirrors Emby/)
            before deleting — restore from remote if available

authentication.db   — API keys, session tokens
  CORRUPT → safe to delete — API keys regenerated on restart
            Any connected clients will need to re-authenticate once

activity.db         — activity/access log
  CORRUPT → safe to delete — it's a log, losing it is acceptable

library.db-wal      — write-ahead log (uncommitted transactions)
  PRESENT + CORRUPT → check library.db first; WAL corruption usually means
                      the main library.db is also affected
```

### Configuration (master_host*.conf)

```bash
HOST1_EMBY_CONTAINER="Emby"    # aliased by detect_hosts() → EMBY_CONTAINER
HOST2_EMBY_CONTAINER="Emby"
```

Emby's config path is detected automatically from Docker volume mounts — no manual
path configuration needed.

### Usage

```bash
emby_database_repair.sh             # stop Emby, check all databases, restart
emby_database_repair.sh --dry-run   # show what would be checked, no Emby stop
emby_database_repair.sh --log       # verbose — show SQLite output per database
emby_database_repair.sh --status    # show Emby config path and database locations
```

---

## zfs_pool_scrub.sh

Triggers ZFS scrub on all pools (or a specific named pool) and waits for completion.
Notifies when done with a summary of any errors found.

### Why Run ZFS Scrub

ZFS stores a checksum with every block of data. Scrub reads every block and verifies
the checksum matches the stored hash. Silent data corruption can sit on disk for months
without triggering any error — until you try to read that specific file. By then:
- It may already be mirrored to HOST2 in its corrupted state
- The original source may no longer exist
- ZFS can self-repair during scrub if redundancy exists (RAIDZ or mirrors)

Run monthly. Also run after any disk replacement or power event.
Safe to run while the system is in use — scrub runs at low I/O priority.

### Configuration (master_host*.conf)

```bash
HOST1_ZFS_REPORT_IGNORE_POOLS=(
    "disk10"    # JBOD member — no redundancy, skipped from default scrub
    "disk9"
    "disk8"
)

HOST2_ZFS_REPORT_IGNORE_POOLS=(
    "cache"     # example — single-disk pool excluded from default
)
```

To scrub a pool in the ignore list, specify it by name explicitly.

### Usage

```bash
# Scrub all pools except those in ZFS_REPORT_IGNORE_POOLS:
zfs_pool_scrub.sh

# Scrub a specific pool by name — bypasses the ignore list:
zfs_pool_scrub.sh gaming

# Check current scrub status without starting a new one:
zfs_pool_scrub.sh --status

# Dry run — show which pools would be scrubbed:
zfs_pool_scrub.sh --dry-run

# Verbose — show scrub progress every 60s poll:
zfs_pool_scrub.sh --log
```

---

## recreate_shares.sh

Creates share directories on the correct disks after a fresh unRAID install or disk
rebuild. Run once on HOST2 before the first rsync from HOST1.

### When to Use

```
After a fresh unRAID install where /boot/config/shares/*.cfg were restored:
  The share definitions exist → UI shows shares → directories are missing on disk
  rsync.sh tries to write to /mnt/user/Movies → path doesn't exist → aborts

After a disk replacement or rebuild where share folders were lost:
  Replacement disk is blank → no share directories on the new disk
  unRAID won't create them automatically
```

### What It Does

```
For each .cfg file in /boot/config/shares/:
  1. Read the share name (e.g., Movies)
  2. Read the shareInclude list (e.g., disk1,disk2,disk5)
  3. Create /mnt/disk1/Movies, /mnt/disk2/Movies, /mnt/disk5/Movies
  4. Place a .recovery marker in /mnt/user/Movies/

The .recovery marker tells rsync.sh this is a fresh share:
  .recovery present → rsync WITHOUT --delete (safe — new files only, nothing removed)
  .recovery absent  → rsync WITH --delete (normal mirror mode)

Self-cleaning: after the first successful rsync, the source side has no .recovery file,
so the second nightly run deletes it from the mirror and normal --delete resumes.
No manual cleanup needed.
```

### Usage

```bash
recreate_shares.sh              # create all missing share directories + .recovery markers
recreate_shares.sh --dry-run    # show what would be created without creating
recreate_shares.sh --log        # verbose — show each directory created per disk
recreate_shares.sh --status     # show share configs and current directory state
```

---

## continuous_scripts_status.sh

Live status dashboard for all continuously running scripts. Read-only — makes no
changes to any running process, container, or state file.

### What It Shows

```
system_watchdog
  Running state, PID, uptime, approximate cycle count
  Active strikes, recent restart history
  Live snapshot: rootfs, RAM, ZFS ARC, load, zombie count, CPU temp

docker_watchdog
  Running state, PID, uptime
  Running / stopped / unhealthy container counts
  Required containers status
  Memory-monitored containers
  Recent restart history + skip list

failover (fallback.sh)
  Current state (NORMAL / FALLBACK / HANDBACK)
  Tier flags and timestamps
  Remote Tailscale visibility
```

State files are read as-is — if a script is mid-cycle, the display reflects the last
completed cycle, not the current in-progress state.

### Usage

```bash
continuous_scripts_status.sh        # show full dashboard
continuous_scripts_status.sh --log  # verbose output with additional detail per section
```

---

## claude_startup.sh

Restores Claude Code's persistent data after an unRAID reboot and optionally launches
Claude. Standalone script — no common.sh dependency.

### Why This Exists

unRAID's root filesystem lives in RAM — `/root/.claude` and `/root/.local` are wiped on
every reboot. This script symlinks both directories back to persistent appdata storage
at `/mnt/user/appdata/claude-code/` before launching Claude.

### First Run Migration

On first run, if persistent storage is empty, the script migrates from current live locations:

```
/root/.claude              → /mnt/user/appdata/claude-code/.claude
/root/.local/share/claude  → /mnt/user/appdata/claude-code/local/share/claude
```

Subsequent runs skip the migration and only create the symlinks.

### Calling from array_started.sh

To auto-restore Claude data on every boot without launching an interactive session:

```bash
# In /boot/config/go or array_started.sh:
/path/to/Tools/claude_startup.sh --setup
```

### Usage

```bash
claude_startup.sh           # set up persistent symlinks and launch Claude
claude_startup.sh --setup   # set up symlinks only — no launch (for array_started.sh)
```

---

## Adding a New Tool

Write the tool when you solve a problem manually with bash commands. You'll face it again.
The cost of writing the tool is 30 minutes. The cost of reconstructing the commands at 2am
is much higher.

### Checklist

```
✓ Header explains the specific situation that requires this tool
✓ Root check — most tools need root
✓ --dry-run support — always
✓ --status support — show current state before acting
✓ Confirmation for destructive operations (interactive YES or --force flag)
✓ Notify on completion — success and failure
✓ Leave system in clean state on any exit — trap for cleanup
✓ Add to README-Tools.md scripts table and HOW THE SCRIPTS RELATE diagram
```

### Minimal Skeleton

```bash
#!/bin/bash
# ==============================================================================================
# ============================= Your Tool Name ================================================
# ==============================================================================================
#
# PURPOSE
# ─────────────────────────────────────────────────────────────────────────────
# One sentence: what situation this solves and when to use it.
#
# ==============================================================================================
# OPERATIONAL SAFEGUARDS
# ==============================================================================================
#
# Root Required
#   chown / docker / etc. require root.
#
# Confirmation Required
#   Interactive mode prompts for YES. Use --force to bypass in scripts.
#
# ==============================================================================================
# RUNTIME MODES
# ==============================================================================================
#
# your_tool.sh
#     Normal run.
#
# your_tool.sh --dry-run
#     Preview without making changes.
#
# your_tool.sh --status
#     Show current state and exit.
#
# ==============================================================================================

SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
source "$SCRIPT_DIR/../load_config.sh"
parse_args "$@"

if [[ "$EUID" -ne 0 ]]; then error "Must be run as root"; exit 1; fi

validate_unraid_cmd \
    "/usr/local/emhttp/plugins/dynamix/scripts/notify" \
    "" "" "unRAID notify script" || warn "notify not found — notifications disabled"

acquire_lock
detect_hosts

if [[ "$SHOW_STATUS" == true ]]; then
    log "Current state: ..."
    exit 0
fi

[[ "$DRY_RUN" == true ]] && warn "DRY RUN — no changes will be made"

if [[ "$FORCE" != true ]]; then
    read -r -p "Type YES to proceed: " CONFIRM
    [[ "$CONFIRM" != "YES" ]] && { warn "Aborted."; exit 0; }
fi

# Do the work
# ...

notify "Tool completed on $(hostname) ($MY_ID)" "Tool Name" "normal"
```