Files
Varaverk/Tools/Manual-Tools.md
T

28 KiB
Raw Blame History

━━━━━ TOOLS — Manual ━━━━━

Configuration reference, usage procedures, and field guides for every script in Tools/. Run any script with --status first — it shows current state before making any changes.


━━━ CONTENTS ━━━


Output Tiers

All tools have two output levels controlled by --log.

Without --log, each script processes and always concludes with a summary block showing identity, duration, counts, and a status line. Warnings and errors are always visible. State-display scripts (watchdog_skip_list_manager, zfs_pool_scrub --status, fallback_state_reset current-state section) always show their state output — --log adds configuration detail and per-item resolution within each section.

With --log, per-item detail appears: individual items added/skipped, per-database check results, per-pool scan lines, per-container stop/start state, per-directory creation results. Use when debugging unexpected results or confirming a first run.


emby_to_lidarr_sync.sh

One-shot bootstrap tool. Scans Emby play history, finds artists you've actually listened to that are not yet tracked in Lidarr, and adds them. No scoring — if it was played, Lidarr should monitor it. Not scheduled; run manually when you want to close the gap between what's in your library and what Lidarr watches.

When to Use

  • After initial Lidarr setup — bring it in line with existing listening history
  • After a Lidarr database wipe or migration
  • Any time you suspect artists you listen to are slipping through unmonitored

Usage

# See what would be added (no changes)
bash Tools/emby_to_lidarr_sync.sh --dry-run

# Limit to recent plays only
bash Tools/emby_to_lidarr_sync.sh --dry-run --days 30

# Run for real
bash Tools/emby_to_lidarr_sync.sh

Notes

  • Reads all MusicAlbum items from Emby and extracts AlbumArtist — primary album artists only, not guest features or tag credits
  • Filters out VA, Various Artists, and other metadata placeholders
  • Triggers ArtistSearch immediately after each successful add — no manual search needed
  • Activity log has a finite history; use --days N if the log has been pruned

emby_to_sonarr_sync.sh

One-shot bootstrap tool. Finds TV series present in Emby that are not tracked in Sonarr and adds them. Uses TVDB ID matching when available (more reliable than title matching), falling back to case-insensitive title comparison.

When to Use

  • After initial Sonarr setup — bring it in line with your existing library
  • After a Sonarr database wipe or migration
  • Any time series you own are slipping through unmonitored

Usage

# See what would be added (no changes)
bash Tools/emby_to_sonarr_sync.sh --dry-run

# Run for real
bash Tools/emby_to_sonarr_sync.sh

Notes

  • Triggers SeriesSearch immediately after each successful add — Sonarr begins searching for missing episodes right away
  • Adds to the first accessible root folder in Sonarr
  • TVDB ID match preferred over title; title fallback handles edge cases
  • Requires SONARR_EMBY_LIBRARIES configured in master.conf

emby_to_radarr_sync.sh

One-shot bootstrap tool. Finds movies present in Emby that are not tracked in Radarr and adds them. Uses TMDB ID matching when available, falling back to case-insensitive title comparison.

When to Use

  • After initial Radarr setup — bring it in line with your existing library
  • After a Radarr database wipe or migration
  • Any time movies you own are slipping through unmonitored

Usage

# See what would be added (no changes)
bash Tools/emby_to_radarr_sync.sh --dry-run

# Run for real
bash Tools/emby_to_radarr_sync.sh

Notes

  • Triggers MoviesSearch immediately after each successful add — Radarr begins searching for the movie right away
  • Adds to the first accessible root folder in Radarr
  • TMDB ID match preferred over title; title fallback handles edge cases

fallback_state_reset.sh

Resets the fallback state file to NORMAL and clears all tier flags. State file only — does NOT start or stop any containers. After reset, fallback.sh resumes from NORMAL on its next cycle.

Only run after verifying the stack is actually in a normal state — right containers on the right server, DDNS correct, no active fallback in progress. Resetting state during a real fallback causes fallback.sh to stop covering the remote until the next detection cycle.

When to Use

After fallback_test.sh didn't complete cleanly
  → state left in FALLBACK but containers are actually back to normal

After a failed handback
  → state shows FALLBACK but remote is back up and containers are split

After killing fallback.sh directly (not gracefully via SIGTERM)
  → state is unknown, cycle was interrupted mid-operation

After a dev/debug session
  → state left in a non-NORMAL state from testing

Usage

fallback_state_reset.sh             # show current state, prompt for YES before resetting
fallback_state_reset.sh --status    # show current state file contents only
fallback_state_reset.sh --dry-run   # show what the new state file would contain, no write
fallback_state_reset.sh --force     # reset without confirmation prompt (for scripted use)

Verify Before Resetting

# Right containers on right server?
cat "$STATE_DIR/fallback_state"     # shows fallback current state

# DDNS pointing correctly?
nslookup Gmer4Lfe.com               # confirm it resolves to the right IP

# fallback.sh not running?
pgrep -f "fallback.sh"              # empty output = not running

# Both servers Tailscale connected?
tailscale status                    # both hosts should show active

What the State File Contains

state=NORMAL
fallback_start=0
handback_strikes=0
tier2_started=false
tier3_started=false
tier4_started=false

watchdog_skip_list_manager.sh

View and manage the persistent container skip list used by docker_watchdog.sh.

When to Use

docker_watchdog.sh restarts the same container N times within the rolling window
→ container added to skip list in $STATE_DIR
→ critical notification sent
→ watchdog stops touching it entirely

You fix the underlying problem (database, config, dependencies).
You need to clear the container from the skip list so monitoring resumes.

Recovery Workflow

# 1. Understand the situation — always start here:
watchdog_skip_list_manager.sh --status
# Shows: skip list contents, which are running vs. stopped, restart history

# 2. Fix the underlying problem first
#    Check logs:    docker logs ContainerName --tail 100
#    Check disk:    df -h /mnt/user
#    Check db:      docker exec ContainerName sqlite3 /path/to.db ".tables"

# 3. Clear from skip list + restart history:
watchdog_skip_list_manager.sh --clear ContainerName

# 4. Start the container manually — confirm your fix worked:
docker start ContainerName

# 5. Watchdog resumes normal monitoring on next cycle — no further action needed

State Files Managed

# Both live on /boot/config — survive reboots intentionally.
# A container that was skip-listed before a reboot is still broken after it.

$SYS_WATCHDOG_FAILED_FILE          # persistent skip list
$WATCHDOG_CONTAINER_RESTART_LOG    # restart loop tracking

Configuration (master.conf)

WATCHDOG_CONTAINER_RESTART_LIMIT=3     # restarts before skip-listing
WATCHDOG_CONTAINER_RESTART_WINDOW=1    # rolling window in hours

Usage

watchdog_skip_list_manager.sh                        # show status (default)
watchdog_skip_list_manager.sh --status               # explicit status
watchdog_skip_list_manager.sh --clear ContainerName  # clear specific + restart history
watchdog_skip_list_manager.sh --clear ContainerName --force   # no confirmation prompt
watchdog_skip_list_manager.sh --clear-all            # clear everything
watchdog_skip_list_manager.sh --clear-all --force    # non-interactive
watchdog_skip_list_manager.sh --dry-run              # preview any clear action

bulk_permissions_repair.sh

Applies correct ownership and permissions to specific paths. Faster than running media_shares_permissions.sh which processes every configured share — use this when you know exactly what needs fixing and don't want to wait for a full library walk.

When to Use

Admin copy left root:root files       — scp, cp, direct file transfer
New share needs permissions now       — can't wait for nightly run
Container wrote as root               — before PUID/PGID was fixed
Specific directory has wrong perms    — targeted fix, not a full library walk

Use the full media_shares_permissions.sh instead for:

  • Regular nightly maintenance (already scheduled in daily_sync_maintenance.sh)
  • After confirming a container's PUID/PGID is now correct
  • Initial permissions setup on a new server

Diagnosing High Wrong-Owner Counts

The script counts files with wrong ownership before applying the fix. A high count on a share that was recently written means a container has wrong PUID/PGID.

# Fix: add to the container's Docker template:
PUID=99
PGID=100

# Common culprits writing as root:
#   SABnzbd, qBittorrent, slskd — check each one's Docker env vars

Configuration (master.conf)

PERMISSIONS_OWNER="nobody:users"    # matches PUID=99 PGID=100
PERMISSIONS_DIR_MODE="755"          # directories — enter, list, no world-write
PERMISSIONS_FILE_MODE="664"         # files — owner+group rw, others read-only

Usage

# Single path:
bulk_permissions_repair.sh /mnt/user/Movies

# Multiple paths — all corrected in one run:
bulk_permissions_repair.sh /mnt/user/Movies /mnt/user/Tv_Shows /mnt/user/Music

# Dry run first — shows count of files with wrong ownership per path:
bulk_permissions_repair.sh /mnt/user/Movies --dry-run

# Verbose — show each corrected file:
bulk_permissions_repair.sh /mnt/user/Movies --log

container_data_export.sh

Exports a container's appdata directory to a compressed tar archive. Stops the container first for a clean consistent backup, verifies the archive after creation, then restarts the container.

When to Use

Before major container updates — especially "database migration — no rollback" changelogs
Before pool migrations — clean backup before moving appdata to a new pool
Before removing a container from the stack — archive its data before deletion
Manual point-in-time backup before risky config changes

Export Sequence

1. Space check
   Estimates required space from appdata size × 1.1
   Aborts if output directory doesn't have enough free space
   Container is NOT stopped until the space check passes

2. Stop container cleanly
   docker stop ContainerName — graceful shutdown

3. Create archive
   tar -czf ContainerName_YYYY-MM-DD_HH-MM.tar.gz /path/to/appdata

4. Verify archive integrity
   tar --test-file archive.tar.gz — confirms archive is valid and complete
   If verification fails → restart container anyway, report error

5. Restart container
   docker start ContainerName — always happens, even if archiving failed

Usage

# Syntax: container_data_export.sh ContainerName AppDataPath OutputDir

# Emby backup:
container_data_export.sh \
    Emby \
    /mnt/media-servers/Media_Server/Emby \
    /mnt/user/Backups/

# Dry run — verify space and paths without stopping anything:
container_data_export.sh \
    Emby \
    /mnt/media-servers/Media_Server/Emby \
    /mnt/user/Backups/ \
    --dry-run

# Output filename: Emby_2026-05-14_02-30.tar.gz
# Timestamped — safe to run multiple times, no overwrite

emby_database_repair.sh

Stops Emby, runs SQLite PRAGMA integrity_check on every Emby database, and restarts. Reports per-database — does NOT automatically repair. Recovery requires judgment.

When to Use

Emby logs show database errors                     → run this first
Emby crashing repeatedly with no clear cause       → likely database corruption
Playback history or user data behaving strangely   → users.db or library.db issue
After a hard shutdown or power loss with Emby running → check for WAL corruption

Recovery Guide by Database

library.db          — media library metadata: titles, seasons, episodes, artwork
  CORRUPT → safe to delete — Emby fully rebuilds from media files on next start
            Rebuild takes time (hours on large libraries) but loses nothing permanent

users.db            — user accounts, watch history, playback positions, settings
  CORRUPT → deleting resets ALL user accounts and watch history
            Check for a recent backup (weekly_sync_maintenance.sh mirrors Emby/)
            before deleting — restore from remote if available

authentication.db   — API keys, session tokens
  CORRUPT → safe to delete — API keys regenerated on restart
            Any connected clients will need to re-authenticate once

activity.db         — activity/access log
  CORRUPT → safe to delete — it's a log, losing it is acceptable

library.db-wal      — write-ahead log (uncommitted transactions)
  PRESENT + CORRUPT → check library.db first; WAL corruption usually means
                      the main library.db is also affected

Configuration (host*.conf)

HOST1_EMBY_CONTAINER="Emby"    # aliased by detect_hosts() → EMBY_CONTAINER
HOST2_EMBY_CONTAINER="Emby"

Emby's config path is detected automatically from Docker volume mounts — no manual path configuration needed.

Usage

emby_database_repair.sh             # stop Emby, check all databases, restart
emby_database_repair.sh --dry-run   # show what would be checked, no Emby stop
emby_database_repair.sh --log       # verbose — show SQLite output per database
emby_database_repair.sh --status    # show Emby config path and database locations

zfs_pool_scrub.sh

Triggers ZFS scrub on all pools (or a specific named pool) and waits for completion. Notifies when done with a summary of any errors found.

Why Run ZFS Scrub

ZFS stores a checksum with every block of data. Scrub reads every block and verifies the checksum matches the stored hash. Silent data corruption can sit on disk for months without triggering any error — until you try to read that specific file. By then:

  • It may already be mirrored to HOST2 in its corrupted state
  • The original source may no longer exist
  • ZFS can self-repair during scrub if redundancy exists (RAIDZ or mirrors)

Run monthly. Also run after any disk replacement or power event. Safe to run while the system is in use — scrub runs at low I/O priority.

Configuration (host*.conf)

HOST1_ZFS_REPORT_IGNORE_POOLS=(
    "disk10"    # JBOD member — no redundancy, skipped from default scrub
    "disk9"
    "disk8"
)

HOST2_ZFS_REPORT_IGNORE_POOLS=(
    "cache"     # example — single-disk pool excluded from default
)

To scrub a pool in the ignore list, specify it by name explicitly.

Usage

# Scrub all pools except those in ZFS_REPORT_IGNORE_POOLS:
zfs_pool_scrub.sh

# Scrub a specific pool by name — bypasses the ignore list:
zfs_pool_scrub.sh gaming

# Check current scrub status without starting a new one:
zfs_pool_scrub.sh --status

# Dry run — show which pools would be scrubbed:
zfs_pool_scrub.sh --dry-run

# Verbose — show scrub progress every 60s poll:
zfs_pool_scrub.sh --log

smart_long_test.sh

Runs a SMART extended (long) self-test on all drives sequentially and reports results. Extended tests read every sector — they catch bad sectors and pre-failure reallocations that the short test skips. Called monthly by monthly_maintenance.sh.

When to Use

Monthly via monthly_maintenance.sh — automatic, no manual trigger needed
After a disk replacement or rebuild — verify the new drive before it enters production
After a power cut or hard shutdown — check for newly reallocated sectors
When a drive shows elevated reallocated sectors in ZFS scrub or system logs

Configuration (host*.conf)

HOST1_SMART_IGNORE_DRIVES=("/dev/sdb")   # boot USB — no useful SMART data
HOST2_SMART_IGNORE_DRIVES=()

Aliased by detect_hosts()SMART_IGNORE_DRIVES.

Usage

# Test all drives (skips SMART_IGNORE_DRIVES):
smart_long_test.sh

# Test a specific drive — bypasses the ignore list:
smart_long_test.sh /dev/sda

# Show last self-test result for all drives without starting a new test:
smart_long_test.sh --status

# Show which drives would be tested, no tests started:
smart_long_test.sh --dry-run

# Verbose — show poll progress every 60s per drive:
smart_long_test.sh --log

What Gets Reported

Each drive reports one of: completed without error, completed with errors, interrupted, in progress. Any result other than completed without error triggers a notification. Drives are tested one at a time — total runtime depends on drive count and size (a full 8TB HDD takes roughly 90180 minutes).


arr_profile_enforcer.sh

Ensures every series in Sonarr and every movie in Radarr is on the correct quality profile based on where it lives on disk. Only touches items with the wrong profile — idempotent, safe to re-run.

Profile Rules

Root folder path contains "kids" or "anime"  →  ARR_KIDS_PROFILE_NAME
All other root folders                        →  ARR_SONARR_DEFAULT_PROFILE (Sonarr)
                                                 ARR_RADARR_DEFAULT_PROFILE (Radarr)

Profile IDs are resolved from the API at runtime by name — no IDs need to be hardcoded and the same script works across hosts.

When to Use

After initial arr setup — library was imported with a generic profile
After adding a new root folder — new imports may use the wrong default
After a profile rename — profile names in master.conf must match arr exactly
Any time arr is reporting downloads to the wrong quality level

Configuration (master.conf)

ARR_KIDS_PROFILE_NAME="Kids shows"     # must exactly match the profile name in Sonarr/Radarr
ARR_SONARR_DEFAULT_PROFILE="Any"
ARR_RADARR_DEFAULT_PROFILE="Any (mine)"

Usage

# Dry run first — shows what would change without touching anything:
arr_profile_enforcer.sh --dry-run

# Fix all Sonarr and Radarr libraries:
arr_profile_enforcer.sh

# Sonarr only:
arr_profile_enforcer.sh --sonarr-only

# Radarr only:
arr_profile_enforcer.sh --radarr-only

webhook_setup.sh

Registers the Varaverk upgrade webhook notification in Sonarr, Radarr, and Lidarr. Idempotent — skips any arr that already has the webhook registered. Runs on both hosts unless --local-only is passed.

When to Use

After initial Varaverk install
After adding a new arr to the stack
After adding a new host — both hosts need the webhook registered locally
After WEBHOOK_SECRET was regenerated — re-register to push the new secret

What It Does

1. Generates WEBHOOK_SECRET in master.conf if empty
   (openssl rand -hex 32 → written to master.conf in place)

2. Registers webhook in local Sonarr, Radarr, and Lidarr
   POST to /api/v3/notification with the correct event triggers

3. SSHes to remote host and runs itself --local-only
   Both hosts register using the same secret

The webhook URL registered is http://<local-ip>:<WEBHOOK_PORT>/webhook?key=<secret> — uses the server's own LAN IP so arrs talk directly to the local listener.

Configuration (master.conf)

WEBHOOK_PORT=7821      # port the listener runs on
WEBHOOK_SECRET=""      # auto-generated on first run if empty

Usage

# Register on both hosts (normal first-time setup):
webhook_setup.sh

# Local host only (used internally when SSH'd from the other host):
webhook_setup.sh --local-only

# Show what would be registered without making changes:
webhook_setup.sh --dry-run

claude_startup.sh

Restores Claude Code's persistent data after an unRAID reboot and optionally launches Claude. Standalone script — no common.sh dependency.

Why This Exists

unRAID's root filesystem lives in RAM — /root/.claude and /root/.local are wiped on every reboot. This script symlinks both directories back to persistent appdata storage at /mnt/user/appdata/claude-code/ before launching Claude.

First Run Migration

On first run, if persistent storage is empty, the script migrates from current live locations:

/root/.claude              → /mnt/user/appdata/claude-code/.claude
/root/.local/share/claude  → /mnt/user/appdata/claude-code/local/share/claude

Subsequent runs skip the migration and only create the symlinks.

Calling from array_started.sh

array_started.sh calls claude_startup.sh directly (no flags). This sets up the symlinks only — no interactive session is launched. That is the default behavior.

Usage

claude_startup.sh           # set up persistent symlinks only (default — used by array_started.sh)
claude_startup.sh --launch  # set up symlinks and launch Claude interactively

ramdisk_stop.sh

Safely stops the transcode ramdisk: redirects the transcode symlink to the SSD fallback first (so Emby continues writing without interruption), then unmounts the tmpfs and updates the state file. Primary use case is stopping the current ramdisk before re-running ramdisk_setup.sh with new size or threshold values.

When to Use

Bumping RAMDISK_SIZE — setup script is idempotent, skips remount if already mounted
  → stop first, then re-run ramdisk_setup.sh with new HOST*_RAMDISK_SIZE value

Adjusting RAMDISK_WARN_GB / RAMDISK_LOW_GB thresholds
  → no need to stop for threshold changes (transcode_manager reads vars live)
  → only needed if you're also changing the size

Temporarily freeing ramdisk RAM — reclaim tmpfs back to general memory pool
  → stop, restart later with ramdisk_setup.sh

Stop Sequence

1. Redirect symlink: TRANSCODE_LINK → TRANSCODE_SSD
   Emby immediately writes to SSD — no broken-path window during unmount

2. Check for active transcode files on ramdisk (warn, don't block)
   Files in progress on the ramdisk are lost on unmount — expected for maintenance

3. Unmount ramdisk
   Regular umount first; if busy (directory handles only, no active writes)
   falls back to lazy unmount automatically

4. Update /tmp/transcode_state.db → current_target=TRANSCODE_SSD
   transcode_manager.sh reads this on its next cycle

transcode_manager Warning

If transcode_manager.sh is running, it may flip the symlink back to the ramdisk on its next cycle (once the ramdisk is unmounted, that flip will fail). Stop transcode_manager.sh first if you need the SSD redirect to hold before remounting.

After Stopping

# Update host*.conf with new size values:
# HOST1_RAMDISK_SIZE="10G"
# HOST1_RAMDISK_WARN_GB=8.5
# HOST1_RAMDISK_LOW_GB=7

# Remount at new size:
bash Transcodes/ramdisk_setup.sh

Usage

ramdisk_stop.sh --status     # show mount state, symlink, active files — always check first
ramdisk_stop.sh --dry-run    # show what would happen without making changes
ramdisk_stop.sh              # stop the ramdisk
ramdisk_stop.sh --log        # verbose — show each step

docker_prune_images.sh

Removes orphaned Docker images that accumulate after container updates. Two modes:

Default (dangling only) — removes untagged images (no name, no container reference). Safe — running containers are never affected. Use routinely after update cycles.

--all (full orphan cleanup) — first removes stopped/exited containers, then removes all images not used by any running container. Use when you've removed apps and want to recover the disk space. CAUTION: also removes intentionally stopped containers.

Usage

docker_prune_images.sh           # remove dangling (untagged) images only
docker_prune_images.sh --all     # remove stopped containers, then all unused images
docker_prune_images.sh --dry-run # show what would be removed without making changes
docker_prune_images.sh --status  # show dangling images and stopped containers

Adding a New Tool

Write the tool when you solve a problem manually with bash commands. You'll face it again. The cost of writing the tool is 30 minutes. The cost of reconstructing the commands at 2am is much higher.

Checklist

✓ Header explains the specific situation that requires this tool
✓ Root check — most tools need root
✓ --dry-run support — always
✓ --status support — show current state before acting
✓ Confirmation for destructive operations (interactive YES or --force flag)
✓ Notify on completion — success and failure
✓ Leave system in clean state on any exit — trap for cleanup
✓ Add to README-Tools.md scripts table and HOW THE SCRIPTS RELATE diagram

Minimal Skeleton

#!/bin/bash
# ==============================================================================================
# ============================= Your Tool Name ================================================
# ==============================================================================================
#
# PURPOSE
# ─────────────────────────────────────────────────────────────────────────────
# One sentence: what situation this solves and when to use it.
#
# ==============================================================================================
# OPERATIONAL SAFEGUARDS
# ==============================================================================================
#
# Root Required
#   chown / docker / etc. require root.
#
# Confirmation Required
#   Interactive mode prompts for YES. Use --force to bypass in scripts.
#
# ==============================================================================================
# RUNTIME MODES
# ==============================================================================================
#
# your_tool.sh
#     Normal run.
#
# your_tool.sh --dry-run
#     Preview without making changes.
#
# your_tool.sh --status
#     Show current state and exit.
#
# ==============================================================================================

SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
source "$SCRIPT_DIR/../load_config.sh"
parse_args "$@"

if [[ "$EUID" -ne 0 ]]; then error "Must be run as root"; exit 1; fi

platform_require_cmd \
    "/usr/local/emhttp/plugins/dynamix/scripts/notify" \
    "" "" "unRAID notify script" || warn "notify not found — notifications disabled"

acquire_lock
detect_hosts

if [[ "$SHOW_STATUS" == true ]]; then
    log "Current state: ..."
    exit 0
fi

[[ "$DRY_RUN" == true ]] && warn "DRY RUN — no changes will be made"

if [[ "$FORCE" != true ]]; then
    read -r -p "Type YES to proceed: " CONFIRM
    [[ "$CONFIRM" != "YES" ]] && { warn "Aborted."; exit 0; }
fi

# Do the work
# ...

notify "Tool completed on $(hostname) ($MY_ID)" "Tool Name" "normal"