Varaverk/Tools/README-Tools.md

━━━━━ TOOLS ━━━━━

Situational utilities — run when something needs fixing, not on a schedule. Recovery, repair, migration, and inspection tools for situations that arise outside the scheduled maintenance model. These scripts sit ready for the moment you need them.

None of these scripts run on a schedule. A script belongs here when it solves a specific operational situation — something you run in response to a problem, before a risky operation, or during a one-time setup task. Having a dedicated folder keeps the other folders clean and makes it obvious what runs routinely vs. situationally.

---

━━━ THE PROBLEM THAT BUILT THIS ━━━

Fallback State Stuck After Testing
Run a fallback test, something exits uncleanly, state file shows FALLBACK. fallback.sh resumes and reads FALLBACK — starts containers it shouldn't, makes decisions based on a state that doesn't reflect reality. Manual recovery means knowing the exact file format and every field to reset. At 2am after a failed test.
Fix: fallback_state_reset.sh — shows current state, prompts for confirmation, resets cleanly to NORMAL.

Container Stuck on Watchdog Skip List After Fixing the Problem
Authelia hit the restart loop limit — went on the skip list. Problem fixed. But the watchdog still isn't monitoring it because the skip list persists on /boot/config across reboots. Where's the file? What format? How do you clear restart history?
Fix: watchdog_skip_list_manager.sh — shows the skip list and which containers are running vs. stopped, clears specific containers with confirmation.

Emby Crashing With No Clear Cause After a Power Cut
Server lost power with Emby running. Emby comes back, runs for 20 minutes, crashes. Logs show database errors. Which database? library.db? users.db? Each has different recovery implications — deleting the wrong one resets all user watch history.
Fix: emby_database_repair.sh — stops Emby, runs PRAGMA integrity_check on every database, reports per-database with specific guidance on what to do about each one.

Files Owned by Root After an Admin Copy
scp a file into a media share. File arrives as root:root. Radarr fails to import — permission denied. The daily permissions script won't run for another 20 hours. Running media_shares_permissions.sh on the whole library takes 30 minutes just to fix one dir.
Fix: bulk_permissions_repair.sh — takes specific paths, applies correct ownership and permissions in seconds.

No Way to Back Up a Container Before a Risky Update
Major version update, changelog says "database migration — no rollback." You want a point-in-time backup. But cp -r while the container is running produces an inconsistent backup, and tar without stopping the container is equally unreliable.
Fix: container_data_export.sh — stops the container cleanly, archives appdata to a timestamped .tar.gz, verifies archive integrity, restarts the container.

Manually Kicked Off a Rescan and the Cache Never Caught Up
Triggered a Lidarr RescanFolders by hand to fix a stats-drift problem. It ran for two hours. arr_cache_write() correctly refuses to write while a rescan's active (2026-07-17 — a mid-scan write once looked exactly like real data loss to every script trusting the cache), but that means nothing writes the real post-scan number through once it finishes, unless something was watching for completion.
Fix: arr_rescan_monitor.sh — waits for the arr's active rescan to finish, then refreshes its cache with the real number.

---

━━━ WHAT THIS FOLDER DOES ━━━

One role: hold scripts for situations the scheduled maintenance model can't handle.

Every script here was written because a specific situation arose that required bash commands to resolve — and that situation is guaranteed to arise again. When you encounter something new, write the tool. Store it here. Find it at 2am next time.

Recovery Tools — Restore known-good state after a failure
fallback_state_reset.sh, watchdog_skip_list_manager.sh

Diagnostic Tools — Inspect and verify before acting
emby_database_repair.sh, zfs_pool_scrub.sh, smart_long_test.sh

Repair Tools — Fix a specific known problem
bulk_permissions_repair.sh, arr_profile_enforcer.sh, arr_rescan_monitor.sh

Lifecycle Tools — Backup, setup, and migration support
container_data_export.sh, ramdisk_stop.sh, webhook_setup.sh

Library Sync Bootstrap — Close the gap between Emby and arr libraries
emby_to_lidarr_sync.sh, emby_to_sonarr_sync.sh, emby_to_radarr_sync.sh

---

━━━ RELATIONSHIP TO OTHER FOLDERS ━━━

System_Essentials/  ← regular system maintenance    — scheduled
Docker_Essentials/  ← regular container management  — scheduled
Monitors/           ← regular health reporting       — scheduled
Orchestrators/      ← regular maintenance windows   — scheduled
Fallback/           ← automated fallback/handback   — event-driven
Tools/              ← situational utilities          — run when needed

Some tools interact with state written by other folders:

Fallback/
  fallback.sh ──────── writes FALLBACK_STATE_FILE ──► fallback_state_reset.sh reads/writes it

Docker_Essentials/
  docker_watchdog.sh ── writes skip list + history ──► watchdog_skip_list_manager.sh manages them

Tools never call scripts in other folders. Other folders never call Tools scripts. The relationship is one-way: Tools act on state that other scripts have written.

---

━━━ SCRIPTS IN THIS FOLDER ━━━

Script	What It Fixes	When to Run
fallback_state_reset.sh	State file stuck in FALLBACK after test or failed handback	After fallback testing or manual intervention
watchdog_skip_list_manager.sh	Container stuck on watchdog skip list after fixing root cause	After fixing a container that hit the restart loop limit
bulk_permissions_repair.sh	Files owned by wrong user after admin copy or bad container config	When arr operations fail due to permissions
container_data_export.sh	Need a clean backup before a risky container update or migration	Before major updates, appdata migrations, or container removals
emby_database_repair.sh	Emby crashing with database errors after power loss or crash	When Emby logs show corruption or repeated crashes
zfs_pool_scrub.sh	Verify ZFS pool integrity — catch silent corruption before it spreads	Monthly, or after any disk or power event
smart_long_test.sh	Run SMART extended self-test on all drives — full sector scan	Monthly via monthly_maintenance.sh, or after any disk event
docker_prune_images.sh	Remove dangling or unused Docker images accumulated after updates	After update cycles, or when disk space is low
ramdisk_stop.sh	Safely stop the transcode ramdisk — redirect symlink to SSD, unmount, update state	Before re-running ramdisk_setup.sh with new size or thresholds
arr_profile_enforcer.sh	Enforce correct quality profiles across all Sonarr/Radarr libraries	After arr setup, profile changes, or when library was imported with wrong profile
webhook_setup.sh	Register the Varaverk upgrade webhook in Sonarr, Radarr, and Lidarr	After initial install or when adding a new arr or host
arr_rescan_monitor.sh	Shared tracked-data cache stuck stale after a manually-triggered rescan	After manually kicking off a RescanFolders/RescanSeries/RescanMovie outside arr_full_rescan.sh
emby_to_lidarr_sync.sh	Add all Emby album artists not yet tracked in Lidarr	After Lidarr setup, database wipe, or when you suspect gaps
emby_to_sonarr_sync.sh	Add all Emby TV series not yet tracked in Sonarr	After Sonarr setup, database wipe, or when you suspect gaps
emby_to_radarr_sync.sh	Add all Emby movies not yet tracked in Radarr	After Radarr setup, database wipe, or when you suspect gaps

---

━━━ HOW THE SCRIPTS RELATE ━━━

All Tools scripts are independent — none call each other, none are called by other Tools.

Situation arises
      │
      ▼
┌──────────────────────────────────────────────────────────────────────┐
│  Tools/                          Run directly when needed             │
│                                                                       │
│  fallback_state_reset.sh     ◄── after fallback test / failed handback│
│  watchdog_skip_list_manager  ◄── after fixing a crash-looping container│
│  bulk_permissions_repair     ◄── wrong ownership after copy or rsync  │
│  container_data_export       ◄── before a risky update or migration   │
│  emby_database_repair        ◄── Emby logs show corruption            │
│  zfs_pool_scrub              ◄── monthly integrity check / post-event │
│  smart_long_test             ◄── monthly SMART scan / post-disk event │
│  arr_profile_enforcer        ◄── wrong profiles after import or setup │
│  webhook_setup               ◄── after install or adding a new arr    │
│  ramdisk_stop                ◄── before ramdisk resize / remount      │
│  arr_rescan_monitor          ◄── after manually triggering a rescan   │
│                                                                       │
│  emby_to_lidarr_sync         ◄── Lidarr setup / database wipe / gap  │
│  emby_to_sonarr_sync         ◄── Sonarr setup / database wipe / gap  │
│  emby_to_radarr_sync         ◄── Radarr setup / database wipe / gap  │
└──────────────────────────────────────────────────────────────────────┘
      │
      ▼
  State files in other folders (Fallback/, Docker_Essentials/) may be read or written.
  No other scripts call into Tools/.