Varaverk/Tools/Manual-Tools.md

# ━━━━━ TOOLS — Manual ━━━━━

Configuration reference, usage procedures, and field guides for every script
in `Tools/`. Run any script with `--status` first — it shows current state before
making any changes.

---

## ━━━ CONTENTS ━━━

- [emby_to_lidarr_sync.sh](#emby_to_lidarr_syncsh)
- [emby_to_sonarr_sync.sh](#emby_to_sonarr_syncsh)
- [emby_to_radarr_sync.sh](#emby_to_radarr_syncsh)
- [fallback_state_reset.sh](#fallback_state_resetsh) *(not yet built — manual workaround)*
- [watchdog_skip_list_manager.sh](#watchdog_skip_list_managersh)
- [bulk_permissions_repair.sh](#bulk_permissions_repairsh)
- [container_data_export.sh](#container_data_exportsh)
- [emby_database_repair.sh](#emby_database_repairsh)
- [zfs_pool_scrub.sh](#zfs_pool_scrubsh)
- [recreate_shares.sh](#recreate_sharessh)
- [continuous_scripts_status.sh](#continuous_scripts_statussh)
- [claude_startup.sh](#claude_startupsh)
- [ramdisk_stop.sh](#ramdisk_stopsh)
- [Adding a New Tool](#adding-a-new-tool)

---

## Output Tiers

All tools have two output levels controlled by `--log`.

Without `--log`, each script processes and always concludes with a summary block
showing identity, duration, counts, and a status line. Warnings and errors are
always visible. State-display scripts (watchdog_skip_list_manager, zfs_pool_scrub
`--status`, fallback_state_reset current-state section) always show their state
output — `--log` adds configuration detail and per-item resolution within each
section.

With `--log`, per-item detail appears: individual items added/skipped, per-database
check results, per-pool scan lines, per-container stop/start state, per-directory
creation results. Use when debugging unexpected results or confirming a first run.

---

## emby_to_lidarr_sync.sh

One-shot bootstrap tool. Scans Emby play history, finds artists you've actually
listened to that are not yet tracked in Lidarr, and adds them. No scoring — if
it was played, Lidarr should monitor it. Not scheduled; run manually when you
want to close the gap between what's in your library and what Lidarr watches.

### When to Use

- After initial Lidarr setup — bring it in line with existing listening history
- After a Lidarr database wipe or migration
- Any time you suspect artists you listen to are slipping through unmonitored

### Usage

```bash
# See what would be added (no changes)
bash Tools/emby_to_lidarr_sync.sh --dry-run

# Limit to recent plays only
bash Tools/emby_to_lidarr_sync.sh --dry-run --days 30

# Run for real
bash Tools/emby_to_lidarr_sync.sh
```

### Notes

- Reads all MusicAlbum items from Emby and extracts AlbumArtist — primary album
  artists only, not guest features or tag credits
- Filters out VA, Various Artists, and other metadata placeholders
- Triggers `ArtistSearch` immediately after each successful add — no manual search needed
- Activity log has a finite history; use `--days N` if the log has been pruned

---

## emby_to_sonarr_sync.sh

One-shot bootstrap tool. Finds TV series present in Emby that are not tracked in
Sonarr and adds them. Uses TVDB ID matching when available (more reliable than
title matching), falling back to case-insensitive title comparison.

### When to Use

- After initial Sonarr setup — bring it in line with your existing library
- After a Sonarr database wipe or migration
- Any time series you own are slipping through unmonitored

### Usage

```bash
# See what would be added (no changes)
bash Tools/emby_to_sonarr_sync.sh --dry-run

# Run for real
bash Tools/emby_to_sonarr_sync.sh
```

### Notes

- Triggers `SeriesSearch` immediately after each successful add — Sonarr begins
  searching for missing episodes right away
- Adds to the first accessible root folder in Sonarr
- TVDB ID match preferred over title; title fallback handles edge cases
- Requires `SONARR_EMBY_LIBRARIES` configured in `master.conf`

---

## emby_to_radarr_sync.sh

One-shot bootstrap tool. Finds movies present in Emby that are not tracked in
Radarr and adds them. Uses TMDB ID matching when available, falling back to
case-insensitive title comparison.

### When to Use

- After initial Radarr setup — bring it in line with your existing library
- After a Radarr database wipe or migration
- Any time movies you own are slipping through unmonitored

### Usage

```bash
# See what would be added (no changes)
bash Tools/emby_to_radarr_sync.sh --dry-run

# Run for real
bash Tools/emby_to_radarr_sync.sh
```

### Notes

- Triggers `MoviesSearch` immediately after each successful add — Radarr begins
  searching for the movie right away
- Adds to the first accessible root folder in Radarr
- TMDB ID match preferred over title; title fallback handles edge cases

---

## fallback_state_reset.sh

> **Not yet built.** Use the manual workaround below.

Planned: reset the fallback state file to NORMAL and clear all tier flags. State file
only — does NOT start or stop any containers.

### When to Use

```
After fallback_test.sh didn't complete cleanly
  → state left in FALLBACK but containers are actually back to normal

After a failed handback
  → state shows FALLBACK but remote is back up and containers are split

After killing fallback.sh directly (not gracefully via SIGTERM)
  → state is unknown, cycle was interrupted mid-operation

After a dev/debug session
  → state left in a non-NORMAL state from testing
```

### Manual Workaround

```bash
# Verify before resetting:

# Right containers on right server?
continuous_scripts_status.sh        # shows fallback current state

# DDNS pointing correctly?
nslookup Gmer4Lfe.com               # confirm it resolves to the right IP

# fallback.sh not running?
pgrep -f "fallback.sh"              # empty output = not running

# Both servers Tailscale connected?
tailscale status                    # both hosts should show active

# Check current state file:
cat /boot/config/fallback_state.db

# Reset to NORMAL (only after confirming containers and DDNS are correct):
echo "state=NORMAL" > /boot/config/fallback_state.db
```

Resetting during an actual fallback causes fallback.sh to think everything is normal
and stop covering the remote — services go offline until the next detection cycle.

### What the State File Contains

```bash
state=NORMAL
fallback_start=0
handback_strikes=0
tier2_started=false
tier3_started=false
tier4_started=false
```

---

## watchdog_skip_list_manager.sh

View and manage the persistent container skip list used by `docker_watchdog.sh`.

### When to Use

```
docker_watchdog.sh restarts the same container N times within the rolling window
→ container added to skip list on /boot/config/
→ critical notification sent
→ watchdog stops touching it entirely

You fix the underlying problem (database, config, dependencies).
You need to clear the container from the skip list so monitoring resumes.
```

### Recovery Workflow

```bash
# 1. Understand the situation — always start here:
watchdog_skip_list_manager.sh --status
# Shows: skip list contents, which are running vs. stopped, restart history

# 2. Fix the underlying problem first
#    Check logs:    docker logs ContainerName --tail 100
#    Check disk:    df -h /mnt/user
#    Check db:      docker exec ContainerName sqlite3 /path/to.db ".tables"

# 3. Clear from skip list + restart history:
watchdog_skip_list_manager.sh --clear ContainerName

# 4. Start the container manually — confirm your fix worked:
docker start ContainerName

# 5. Watchdog resumes normal monitoring on next cycle — no further action needed
```

### State Files Managed

```bash
# Both live on /boot/config — survive reboots intentionally.
# A container that was skip-listed before a reboot is still broken after it.

$SYS_WATCHDOG_FAILED_FILE          # persistent skip list
$WATCHDOG_CONTAINER_RESTART_LOG    # restart loop tracking
```

### Configuration (master.conf)

```bash
WATCHDOG_CONTAINER_RESTART_LIMIT=3     # restarts before skip-listing
WATCHDOG_CONTAINER_RESTART_WINDOW=1    # rolling window in hours
```

### Usage

```bash
watchdog_skip_list_manager.sh                        # show status (default)
watchdog_skip_list_manager.sh --status               # explicit status
watchdog_skip_list_manager.sh --clear ContainerName  # clear specific + restart history
watchdog_skip_list_manager.sh --clear ContainerName --force   # no confirmation prompt
watchdog_skip_list_manager.sh --clear-all            # clear everything
watchdog_skip_list_manager.sh --clear-all --force    # non-interactive
watchdog_skip_list_manager.sh --dry-run              # preview any clear action
```

---

## bulk_permissions_repair.sh

Applies correct ownership and permissions to specific paths. Faster than running
`media_shares_permissions.sh` which processes every configured share — use this when
you know exactly what needs fixing and don't want to wait for a full library walk.

### When to Use

```
Admin copy left root:root files       — scp, cp, direct file transfer
New share needs permissions now       — can't wait for nightly run
Container wrote as root               — before PUID/PGID was fixed
Specific directory has wrong perms    — targeted fix, not a full library walk
```

Use the full `media_shares_permissions.sh` instead for:
- Regular nightly maintenance (already scheduled in daily_sync_maintenance.sh)
- After confirming a container's PUID/PGID is now correct
- Initial permissions setup on a new server

### Diagnosing High Wrong-Owner Counts

The script counts files with wrong ownership before applying the fix. A high count
on a share that was recently written means a container has wrong PUID/PGID.

```bash
# Fix: add to the container's Docker template:
PUID=99
PGID=100

# Common culprits writing as root:
#   SABnzbd, qBittorrent, slskd — check each one's Docker env vars
```

### Configuration (master.conf)

```bash
PERMISSIONS_OWNER="nobody:users"    # matches PUID=99 PGID=100
PERMISSIONS_DIR_MODE="755"          # directories — enter, list, no world-write
PERMISSIONS_FILE_MODE="664"         # files — owner+group rw, others read-only
```

### Usage

```bash
# Single path:
bulk_permissions_repair.sh /mnt/user/Movies

# Multiple paths — all corrected in one run:
bulk_permissions_repair.sh /mnt/user/Movies /mnt/user/Tv_Shows /mnt/user/Music

# Dry run first — shows count of files with wrong ownership per path:
bulk_permissions_repair.sh /mnt/user/Movies --dry-run

# Verbose — show each corrected file:
bulk_permissions_repair.sh /mnt/user/Movies --log
```

---

## container_data_export.sh

Exports a container's appdata directory to a compressed tar archive. Stops the
container first for a clean consistent backup, verifies the archive after creation,
then restarts the container.

### When to Use

```
Before major container updates — especially "database migration — no rollback" changelogs
Before pool migrations — clean backup before moving appdata to a new pool
Before removing a container from the stack — archive its data before deletion
Manual point-in-time backup before risky config changes
```

### Export Sequence

```
1. Space check
   Estimates required space from appdata size × 1.1
   Aborts if output directory doesn't have enough free space
   Container is NOT stopped until the space check passes

2. Stop container cleanly
   docker stop ContainerName — graceful shutdown

3. Create archive
   tar -czf ContainerName_YYYY-MM-DD_HH-MM.tar.gz /path/to/appdata

4. Verify archive integrity
   tar --test-file archive.tar.gz — confirms archive is valid and complete
   If verification fails → restart container anyway, report error

5. Restart container
   docker start ContainerName — always happens, even if archiving failed
```

### Usage

```bash
# Syntax: container_data_export.sh ContainerName AppDataPath OutputDir

# Emby backup:
container_data_export.sh \
    Emby \
    /mnt/media-servers/Media_Server/Emby \
    /mnt/user/Backups/

# Dry run — verify space and paths without stopping anything:
container_data_export.sh \
    Emby \
    /mnt/media-servers/Media_Server/Emby \
    /mnt/user/Backups/ \
    --dry-run

# Output filename: Emby_2026-05-14_02-30.tar.gz
# Timestamped — safe to run multiple times, no overwrite
```

---

## emby_database_repair.sh

Stops Emby, runs SQLite `PRAGMA integrity_check` on every Emby database, and restarts.
Reports per-database — does NOT automatically repair. Recovery requires judgment.

### When to Use

```
Emby logs show database errors                     → run this first
Emby crashing repeatedly with no clear cause       → likely database corruption
Playback history or user data behaving strangely   → users.db or library.db issue
After a hard shutdown or power loss with Emby running → check for WAL corruption
```

### Recovery Guide by Database

```
library.db          — media library metadata: titles, seasons, episodes, artwork
  CORRUPT → safe to delete — Emby fully rebuilds from media files on next start
            Rebuild takes time (hours on large libraries) but loses nothing permanent

users.db            — user accounts, watch history, playback positions, settings
  CORRUPT → deleting resets ALL user accounts and watch history
            Check for a recent backup (weekly_sync_maintenance.sh mirrors Emby/)
            before deleting — restore from remote if available

authentication.db   — API keys, session tokens
  CORRUPT → safe to delete — API keys regenerated on restart
            Any connected clients will need to re-authenticate once

activity.db         — activity/access log
  CORRUPT → safe to delete — it's a log, losing it is acceptable

library.db-wal      — write-ahead log (uncommitted transactions)
  PRESENT + CORRUPT → check library.db first; WAL corruption usually means
                      the main library.db is also affected
```

### Configuration (host*.conf)

```bash
HOST1_EMBY_CONTAINER="Emby"    # aliased by detect_hosts() → EMBY_CONTAINER
HOST2_EMBY_CONTAINER="Emby"
```

Emby's config path is detected automatically from Docker volume mounts — no manual
path configuration needed.

### Usage

```bash
emby_database_repair.sh             # stop Emby, check all databases, restart
emby_database_repair.sh --dry-run   # show what would be checked, no Emby stop
emby_database_repair.sh --log       # verbose — show SQLite output per database
emby_database_repair.sh --status    # show Emby config path and database locations
```

---

## zfs_pool_scrub.sh

Triggers ZFS scrub on all pools (or a specific named pool) and waits for completion.
Notifies when done with a summary of any errors found.

### Why Run ZFS Scrub

ZFS stores a checksum with every block of data. Scrub reads every block and verifies
the checksum matches the stored hash. Silent data corruption can sit on disk for months
without triggering any error — until you try to read that specific file. By then:
- It may already be mirrored to HOST2 in its corrupted state
- The original source may no longer exist
- ZFS can self-repair during scrub if redundancy exists (RAIDZ or mirrors)

Run monthly. Also run after any disk replacement or power event.
Safe to run while the system is in use — scrub runs at low I/O priority.

### Configuration (host*.conf)

```bash
HOST1_ZFS_REPORT_IGNORE_POOLS=(
    "disk10"    # JBOD member — no redundancy, skipped from default scrub
    "disk9"
    "disk8"
)

HOST2_ZFS_REPORT_IGNORE_POOLS=(
    "cache"     # example — single-disk pool excluded from default
)
```

To scrub a pool in the ignore list, specify it by name explicitly.

### Usage

```bash
# Scrub all pools except those in ZFS_REPORT_IGNORE_POOLS:
zfs_pool_scrub.sh

# Scrub a specific pool by name — bypasses the ignore list:
zfs_pool_scrub.sh gaming

# Check current scrub status without starting a new one:
zfs_pool_scrub.sh --status

# Dry run — show which pools would be scrubbed:
zfs_pool_scrub.sh --dry-run

# Verbose — show scrub progress every 60s poll:
zfs_pool_scrub.sh --log
```

---

## recreate_shares.sh

Creates share directories on the correct disks after a fresh unRAID install or disk
rebuild. Run once on HOST2 before the first rsync from HOST1.

### When to Use

```
After a fresh unRAID install where /boot/config/shares/*.cfg were restored:
  The share definitions exist → UI shows shares → directories are missing on disk
  rsync.sh tries to write to /mnt/user/Movies → path doesn't exist → aborts

After a disk replacement or rebuild where share folders were lost:
  Replacement disk is blank → no share directories on the new disk
  unRAID won't create them automatically
```

### What It Does

```
For each .cfg file in /boot/config/shares/:
  1. Read the share name (e.g., Movies)
  2. Read the shareInclude list (e.g., disk1,disk2,disk5)
  3. Create /mnt/disk1/Movies, /mnt/disk2/Movies, /mnt/disk5/Movies
  4. Place a .recovery marker in /mnt/user/Movies/

The .recovery marker tells rsync.sh this is a fresh share:
  .recovery present → rsync WITHOUT --delete (safe — new files only, nothing removed)
  .recovery absent  → rsync WITH --delete (normal mirror mode)

Self-cleaning: after the first successful rsync, the source side has no .recovery file,
so the second nightly run deletes it from the mirror and normal --delete resumes.
No manual cleanup needed.
```

### Usage

```bash
recreate_shares.sh              # create all missing share directories + .recovery markers
recreate_shares.sh --dry-run    # show what would be created without creating
recreate_shares.sh --log        # verbose — show each directory created per disk
recreate_shares.sh --status     # show share configs and current directory state
```

---

## continuous_scripts_status.sh

Live status dashboard for all continuously running scripts. Read-only — makes no
changes to any running process, container, or state file.

### What It Shows

```
stability_watchdog
  Running state, PID, uptime, approximate cycle count
  Active strikes, recent restart history
  Live snapshot: rootfs, RAM, ZFS ARC, load, zombie count, CPU temp

docker_watchdog
  Running state, PID, uptime
  Running / stopped / unhealthy container counts
  Required containers status
  Memory-monitored containers
  Recent restart history + skip list

failover (fallback.sh)
  Current state (NORMAL / FALLBACK / HANDBACK)
  Tier flags and timestamps
  Remote Tailscale visibility
```

State files are read as-is — if a script is mid-cycle, the display reflects the last
completed cycle, not the current in-progress state.

### Usage

```bash
continuous_scripts_status.sh        # show full dashboard
continuous_scripts_status.sh --log  # verbose output with additional detail per section
```

---

## claude_startup.sh

Restores Claude Code's persistent data after an unRAID reboot and optionally launches
Claude. Standalone script — no common.sh dependency.

### Why This Exists

unRAID's root filesystem lives in RAM — `/root/.claude` and `/root/.local` are wiped on
every reboot. This script symlinks both directories back to persistent appdata storage
at `/mnt/user/appdata/claude-code/` before launching Claude.

### First Run Migration

On first run, if persistent storage is empty, the script migrates from current live locations:

```
/root/.claude              → /mnt/user/appdata/claude-code/.claude
/root/.local/share/claude  → /mnt/user/appdata/claude-code/local/share/claude
```

Subsequent runs skip the migration and only create the symlinks.

### Calling from array_started.sh

To auto-restore Claude data on every boot without launching an interactive session:

```bash
# In /boot/config/go or array_started.sh:
/path/to/Tools/claude_startup.sh --setup
```

### Usage

```bash
claude_startup.sh           # set up persistent symlinks and launch Claude
claude_startup.sh --setup   # set up symlinks only — no launch (for array_started.sh)
```

---

## ramdisk_stop.sh

Safely stops the transcode ramdisk: redirects the transcode symlink to the SSD
fallback first (so Emby continues writing without interruption), then unmounts the
tmpfs and updates the state file. Primary use case is stopping the current ramdisk
before re-running `ramdisk_setup.sh` with new size or threshold values.

### When to Use

```
Bumping RAMDISK_SIZE — setup script is idempotent, skips remount if already mounted
  → stop first, then re-run ramdisk_setup.sh with new HOST*_RAMDISK_SIZE value

Adjusting RAMDISK_WARN_GB / RAMDISK_LOW_GB thresholds
  → no need to stop for threshold changes (transcode_manager reads vars live)
  → only needed if you're also changing the size

Temporarily freeing ramdisk RAM — reclaim tmpfs back to general memory pool
  → stop, restart later with ramdisk_setup.sh
```

### Stop Sequence

```
1. Redirect symlink: TRANSCODE_LINK → TRANSCODE_SSD
   Emby immediately writes to SSD — no broken-path window during unmount

2. Check for active transcode files on ramdisk (warn, don't block)
   Files in progress on the ramdisk are lost on unmount — expected for maintenance

3. Unmount ramdisk
   Regular umount first; if busy (directory handles only, no active writes)
   falls back to lazy unmount automatically

4. Update /tmp/transcode_state.db → current_target=TRANSCODE_SSD
   transcode_manager.sh reads this on its next cycle
```

### transcode_manager Warning

If `transcode_manager.sh` is running, it may flip the symlink back to the ramdisk
on its next cycle (once the ramdisk is unmounted, that flip will fail). Stop
`transcode_manager.sh` first if you need the SSD redirect to hold before remounting.

### After Stopping

```bash
# Update host*.conf with new size values:
# HOST1_RAMDISK_SIZE="10G"
# HOST1_RAMDISK_WARN_GB=8.5
# HOST1_RAMDISK_LOW_GB=7

# Remount at new size:
bash Transcodes/ramdisk_setup.sh
```

### Usage

```bash
ramdisk_stop.sh --status     # show mount state, symlink, active files — always check first
ramdisk_stop.sh --dry-run    # show what would happen without making changes
ramdisk_stop.sh              # stop the ramdisk
ramdisk_stop.sh --log        # verbose — show each step
```

---

## Adding a New Tool

Write the tool when you solve a problem manually with bash commands. You'll face it again.
The cost of writing the tool is 30 minutes. The cost of reconstructing the commands at 2am
is much higher.

### Checklist

```
✓ Header explains the specific situation that requires this tool
✓ Root check — most tools need root
✓ --dry-run support — always
✓ --status support — show current state before acting
✓ Confirmation for destructive operations (interactive YES or --force flag)
✓ Notify on completion — success and failure
✓ Leave system in clean state on any exit — trap for cleanup
✓ Add to README-Tools.md scripts table and HOW THE SCRIPTS RELATE diagram
```

### Minimal Skeleton

```bash
#!/bin/bash
# ==============================================================================================
# ============================= Your Tool Name ================================================
# ==============================================================================================
#
# PURPOSE
# ─────────────────────────────────────────────────────────────────────────────
# One sentence: what situation this solves and when to use it.
#
# ==============================================================================================
# OPERATIONAL SAFEGUARDS
# ==============================================================================================
#
# Root Required
#   chown / docker / etc. require root.
#
# Confirmation Required
#   Interactive mode prompts for YES. Use --force to bypass in scripts.
#
# ==============================================================================================
# RUNTIME MODES
# ==============================================================================================
#
# your_tool.sh
#     Normal run.
#
# your_tool.sh --dry-run
#     Preview without making changes.
#
# your_tool.sh --status
#     Show current state and exit.
#
# ==============================================================================================

SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
source "$SCRIPT_DIR/../load_config.sh"
parse_args "$@"

if [[ "$EUID" -ne 0 ]]; then error "Must be run as root"; exit 1; fi

validate_unraid_cmd \
    "/usr/local/emhttp/plugins/dynamix/scripts/notify" \
    "" "" "unRAID notify script" || warn "notify not found — notifications disabled"

acquire_lock
detect_hosts

if [[ "$SHOW_STATUS" == true ]]; then
    log "Current state: ..."
    exit 0
fi

[[ "$DRY_RUN" == true ]] && warn "DRY RUN — no changes will be made"

if [[ "$FORCE" != true ]]; then
    read -r -p "Type YES to proceed: " CONFIRM
    [[ "$CONFIRM" != "YES" ]] && { warn "Aborted."; exit 0; }
fi

# Do the work
# ...

notify "Tool completed on $(hostname) ($MY_ID)" "Tool Name" "normal"
```