added all ai generated Readme files, added the last of the tools scripts
This commit is contained in:
@@ -0,0 +1,262 @@
|
||||
# Tools
|
||||
|
||||
Utility scripts for specific operational situations — recovery, repair, migration, and one-time tasks that don't fit the scheduled maintenance model of the other folders.
|
||||
|
||||
```
|
||||
unRAID_Essentials/ — regular system maintenance, scheduled
|
||||
Docker_Essentials/ — regular container management, scheduled
|
||||
Monitors/ — regular health reporting, scheduled
|
||||
Tools/ — situational utilities, run when needed
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## What Belongs Here
|
||||
|
||||
A script belongs in Tools when it:
|
||||
|
||||
- Solves a specific operational problem rather than ongoing maintenance
|
||||
- Is run manually in response to a situation rather than on a schedule
|
||||
- Is used rarely — recovery scenarios, repairs, migrations, initial setup
|
||||
- Would be dangerous or meaningless to run routinely
|
||||
- Doesn't fit cleanly into any of the other folders
|
||||
|
||||
Tools scripts are not scheduled. They sit here ready for when you need them.
|
||||
|
||||
---
|
||||
|
||||
## Scripts
|
||||
|
||||
### `failover_state_reset.sh`
|
||||
|
||||
Resets the failover state file to NORMAL manually.
|
||||
|
||||
```bash
|
||||
/mnt/user/appdata/unraid_scripts/Tools/failover_state_reset.sh --status
|
||||
/mnt/user/appdata/unraid_scripts/Tools/failover_state_reset.sh --dry-run
|
||||
/mnt/user/appdata/unraid_scripts/Tools/failover_state_reset.sh
|
||||
```
|
||||
|
||||
**When you need this:**
|
||||
|
||||
After failover testing, a failed handback, or manual intervention that left the state file inconsistent. The `failover.sh` state machine reads this file on every cycle — if it shows `FAILOVER` when the system is actually in `NORMAL` operation, the script will make incorrect decisions.
|
||||
|
||||
**What it does:** Rewrites the state file with `state=NORMAL` and clears all tier flags. Does NOT start or stop any containers — state file only.
|
||||
|
||||
**⚠️ Verify first:** Only run after manually confirming both servers are in their correct states — right containers running on the right server, DDNS pointing correctly. The reset doesn't check any of this — it just trusts you.
|
||||
|
||||
**Confirmation required:** Type `YES` to proceed — prevents accidental runs.
|
||||
|
||||
---
|
||||
|
||||
### `watchdog_skip_list_manager.sh`
|
||||
|
||||
View and manage the persistent container skip list used by `docker_watchdog.sh`.
|
||||
|
||||
```bash
|
||||
# View current skip list and restart history
|
||||
/mnt/user/appdata/unraid_scripts/Tools/watchdog_skip_list_manager.sh --status
|
||||
|
||||
# Clear a specific container
|
||||
/mnt/user/appdata/unraid_scripts/Tools/watchdog_skip_list_manager.sh --clear Authelia
|
||||
|
||||
# Clear everything
|
||||
/mnt/user/appdata/unraid_scripts/Tools/watchdog_skip_list_manager.sh --clear-all
|
||||
```
|
||||
|
||||
**When you need this:**
|
||||
|
||||
When a container hits the restart loop limit and gets added to the skip list — it stops being monitored until manually cleared. The `--status` view shows which containers are on the list and whether they're currently running, so you can see at a glance what needs attention.
|
||||
|
||||
**After clearing a container:**
|
||||
1. Fix whatever caused the failure
|
||||
2. Start it manually: `docker start ContainerName`
|
||||
3. The watchdog monitors it normally on the next cycle
|
||||
|
||||
**Files managed:**
|
||||
```
|
||||
/boot/config/system_watchdog_failed.db — skip list
|
||||
/boot/config/container_restart_history.db — restart loop tracking
|
||||
```
|
||||
|
||||
Both are cleared per-container or together. The restart history is also cleared when clearing a specific container — gives it a fresh slate for the loop protection window.
|
||||
|
||||
---
|
||||
|
||||
### `bulk_permissions_repair.sh`
|
||||
|
||||
Applies correct permissions to a single share or specific path. Faster than running the full `media_shares_permissions.sh` which processes every share.
|
||||
|
||||
```bash
|
||||
# Single share
|
||||
/mnt/user/appdata/unraid_scripts/Tools/bulk_permissions_repair.sh /mnt/user/Movies
|
||||
|
||||
# Multiple shares
|
||||
/mnt/user/appdata/unraid_scripts/Tools/bulk_permissions_repair.sh \
|
||||
/mnt/user/Movies /mnt/user/Tv_Shows
|
||||
|
||||
# Dry run first
|
||||
/mnt/user/appdata/unraid_scripts/Tools/bulk_permissions_repair.sh \
|
||||
/mnt/user/Movies --dry-run
|
||||
```
|
||||
|
||||
**When you need this:**
|
||||
|
||||
- A failed transfer left files owned by wrong user
|
||||
- A container wrote files as root instead of `nobody:users`
|
||||
- Manual file operations bypassed normal permission handling
|
||||
- A new share needs permissions applied before the next nightly run
|
||||
|
||||
Uses `PERMISSIONS_MODE` and `PERMISSIONS_OWNER` from `Master.conf` — same values as the full permissions script. Applies `chown` before `chmod` to ensure correct ownership before mode change.
|
||||
|
||||
---
|
||||
|
||||
### `container_data_export.sh`
|
||||
|
||||
Exports a container's appdata directory to a compressed tar archive. Stops the container before archiving for a clean consistent backup, restarts after.
|
||||
|
||||
```bash
|
||||
/mnt/user/appdata/unraid_scripts/Tools/container_data_export.sh \
|
||||
Emby \
|
||||
/mnt/media-servers/Media_Server/Emby \
|
||||
/mnt/user/Backups/
|
||||
|
||||
# Dry run — verify space and paths without stopping anything
|
||||
/mnt/user/appdata/unraid_scripts/Tools/container_data_export.sh \
|
||||
Emby \
|
||||
/mnt/media-servers/Media_Server/Emby \
|
||||
/mnt/user/Backups/ \
|
||||
--dry-run
|
||||
```
|
||||
|
||||
**Output filename:** `ContainerName_YYYY-MM-DD_HH-MM.tar.gz`
|
||||
|
||||
**When you need this:**
|
||||
|
||||
- Before a major container update you're not sure about
|
||||
- Before migrating appdata to a different pool
|
||||
- Before removing a container from the stack — archive its data first
|
||||
- As a manual point-in-time backup before making significant config changes
|
||||
|
||||
**Space check:** Script estimates required space from appdata size × 1.1 and aborts if the output directory doesn't have enough free space. The container is not stopped until the space check passes.
|
||||
|
||||
**Recovery:** If archiving fails, the container is restarted before the script exits — it tries to leave things clean regardless of outcome.
|
||||
|
||||
---
|
||||
|
||||
### `emby_database_repair.sh`
|
||||
|
||||
Stops Emby, runs SQLite integrity checks on all Emby databases, and restarts.
|
||||
|
||||
```bash
|
||||
# Check and report (restarts Emby after)
|
||||
/mnt/user/appdata/unraid_scripts/Tools/emby_database_repair.sh
|
||||
|
||||
# Dry run — detect config path and show what would be checked
|
||||
/mnt/user/appdata/unraid_scripts/Tools/emby_database_repair.sh --dry-run
|
||||
```
|
||||
|
||||
**When you need this:**
|
||||
|
||||
- Emby reports database errors in logs
|
||||
- Unexpected Emby crashes with no clear cause
|
||||
- Playback history or user data behaving strangely
|
||||
- After a hard shutdown or power loss with Emby running
|
||||
|
||||
**Databases checked:**
|
||||
|
||||
| Database | Contains | If corrupted |
|
||||
|----------|----------|--------------|
|
||||
| `library.db` | Media library metadata | Delete — Emby rebuilds from media files |
|
||||
| `users.db` | User accounts and settings | Delete resets all user accounts |
|
||||
| `authentication.db` | API keys and sessions | Delete — keys regenerated on restart |
|
||||
| `activity.db` | Activity log | Delete safely — log only |
|
||||
|
||||
**Important:** This script checks and reports. It does NOT automatically delete or repair corrupted databases — that requires judgment about which database is corrupted and whether you have a backup. The summary provides specific guidance per database type.
|
||||
|
||||
**Config path detection:** Automatically detects the Emby config path from Docker volume mounts — no configuration needed beyond `TRANSCODE_EMBY_CONTAINER` in `Master.conf`.
|
||||
|
||||
---
|
||||
|
||||
### `zfs_pool_scrub.sh`
|
||||
|
||||
Triggers ZFS scrub on all pools (or a specific pool), waits for completion, and reports results.
|
||||
|
||||
```bash
|
||||
# Scrub all pools (skips ZFS_REPORT_IGNORE_POOLS)
|
||||
/mnt/user/appdata/unraid_scripts/Tools/zfs_pool_scrub.sh
|
||||
|
||||
# Scrub a specific pool
|
||||
/mnt/user/appdata/unraid_scripts/Tools/zfs_pool_scrub.sh gaming
|
||||
|
||||
# Check current scrub status without starting a new one
|
||||
/mnt/user/appdata/unraid_scripts/Tools/zfs_pool_scrub.sh --status
|
||||
|
||||
# Dry run — show which pools would be scrubbed
|
||||
/mnt/user/appdata/unraid_scripts/Tools/zfs_pool_scrub.sh --dry-run
|
||||
```
|
||||
|
||||
**When you need this:**
|
||||
|
||||
ZFS scrub reads every block on every pool and verifies checksums — it catches silent data corruption that would otherwise only surface when you try to read the corrupted data. Running monthly is recommended.
|
||||
|
||||
**Safe to run while in use.** Scrub does not interrupt normal I/O — it runs in the background at low priority. The script polls every 60 seconds until all scrubs complete, then reports errors found.
|
||||
|
||||
**Pool filtering:** Pools in `ZFS_REPORT_IGNORE_POOLS` are skipped during all-pool scrubs. To scrub an ignored pool explicitly, specify it by name.
|
||||
|
||||
**Notifications:**
|
||||
- Clean completion — normal notification with pool count and duration
|
||||
- Errors found — warning notification listing affected pools
|
||||
|
||||
---
|
||||
|
||||
## Scheduled Summary
|
||||
|
||||
None. Tools are not scheduled — they run when needed.
|
||||
|
||||
---
|
||||
|
||||
## Adding a New Tool
|
||||
|
||||
When you encounter a situation that required manual bash commands to resolve — write a tool. You'll face it again.
|
||||
|
||||
The pattern for a Tools script:
|
||||
|
||||
```bash
|
||||
#!/bin/bash
|
||||
# Short description of what situation this solves.
|
||||
# When to run it.
|
||||
# Any warnings about destructive operations.
|
||||
|
||||
SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
|
||||
source "$SCRIPT_DIR/../Master.conf"
|
||||
source "$SCRIPT_DIR/../common.sh"
|
||||
|
||||
parse_args "$@"
|
||||
|
||||
# Always:
|
||||
# 1. Check for root
|
||||
# 2. Support --dry-run
|
||||
# 3. Confirm before destructive operations (read -p "Type YES:")
|
||||
# 4. Notify on completion
|
||||
```
|
||||
|
||||
Good candidates for future tools:
|
||||
```
|
||||
array_migration.sh — move appdata from one pool to another
|
||||
with container stop/start and path updates
|
||||
|
||||
emby_metadata_refresh.sh — trigger full library refresh via API
|
||||
useful after storage changes
|
||||
|
||||
tailscale_rekey.sh — rotate Tailscale keys on both servers
|
||||
with SSH key update on both ends
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Philosophy
|
||||
|
||||
Tools exist because not every problem has a scheduled solution. Some things only need to happen once. Some things only happen after something goes wrong. Having a dedicated folder keeps the other folders clean and makes it obvious what runs routinely vs what runs situationally.
|
||||
|
||||
Write the tool when you solve the problem. Store it here. Find it at 2am when you need it again.
|
||||
@@ -0,0 +1,133 @@
|
||||
#!/bin/bash
|
||||
# -----------------------------------------------------------------------------------------------
|
||||
# --------------------------------- Bulk Permissions Repair ------------------------------------
|
||||
# -----------------------------------------------------------------------------------------------
|
||||
# Applies correct permissions to a single share or specific path.
|
||||
# Faster than running media_shares_permissions.sh which processes all shares.
|
||||
# Use when a specific share has wrong ownership or permissions after:
|
||||
# - A failed transfer that left files owned by wrong user
|
||||
# - A container writing files as root instead of nobody:users
|
||||
# - Manual file operations that bypassed normal permission handling
|
||||
# - A new share that needs permissions applied before the next nightly run
|
||||
#
|
||||
# Usage:
|
||||
# bulk_permissions_repair.sh /mnt/user/Movies
|
||||
# bulk_permissions_repair.sh /mnt/user/Movies --dry-run
|
||||
# bulk_permissions_repair.sh /mnt/user/Movies /mnt/user/Tv_Shows
|
||||
#
|
||||
# Uses PERMISSIONS_MODE and PERMISSIONS_OWNER from Master.conf.
|
||||
# Supports --dry-run to show what would be changed without applying.
|
||||
# -----------------------------------------------------------------------------------------------
|
||||
|
||||
SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
|
||||
|
||||
source "$SCRIPT_DIR/../Master.conf"
|
||||
source "$SCRIPT_DIR/../common.sh"
|
||||
|
||||
parse_args "$@"
|
||||
|
||||
# -----------------------------------------------------------------------------------------------
|
||||
# ━━━ $ICON_GEAR Setup ━━━
|
||||
# -----------------------------------------------------------------------------------------------
|
||||
echo ""
|
||||
echo "━━━ $ICON_GEAR Setup ━━━"
|
||||
|
||||
if [[ "$EUID" -ne 0 ]]; then
|
||||
error "Must be run as root"
|
||||
exit 1
|
||||
fi
|
||||
|
||||
success "Running as root"
|
||||
|
||||
if [[ ${#PARSED_ARGS[@]} -eq 0 ]]; then
|
||||
error "No paths specified"
|
||||
error "Usage: bulk_permissions_repair.sh /path/to/share [/another/path]"
|
||||
error " bulk_permissions_repair.sh /mnt/user/Movies --dry-run"
|
||||
exit 1
|
||||
fi
|
||||
|
||||
info "Mode: $PERMISSIONS_MODE"
|
||||
info "Owner: $PERMISSIONS_OWNER"
|
||||
[[ "$DRY_RUN" == true ]] && warn "DRY RUN — no permissions will be changed"
|
||||
|
||||
# -----------------------------------------------------------------------------------------------
|
||||
# ━━━ $ICON_PERMS Apply Permissions ━━━
|
||||
# -----------------------------------------------------------------------------------------------
|
||||
echo ""
|
||||
echo "━━━ $ICON_PERMS Permissions Repair ━━━"
|
||||
|
||||
START=$(date +%s)
|
||||
PASS=()
|
||||
FAIL=()
|
||||
|
||||
for share_path in "${PARSED_ARGS[@]}"; do
|
||||
[[ -z "$share_path" ]] && continue
|
||||
|
||||
echo ""
|
||||
echo "━━━ $ICON_PERMS $(basename "$share_path") ━━━"
|
||||
|
||||
if [[ ! -d "$share_path" ]]; then
|
||||
error "$share_path — not found"
|
||||
FAIL+=("$share_path")
|
||||
continue
|
||||
fi
|
||||
|
||||
# Count files for progress context
|
||||
FILE_COUNT=$(find "$share_path" -type f 2>/dev/null | wc -l)
|
||||
DIR_COUNT=$(find "$share_path" -type d 2>/dev/null | wc -l)
|
||||
SIZE=$(du -sh "$share_path" 2>/dev/null | cut -f1)
|
||||
|
||||
info "$share_path — $FILE_COUNT files, $DIR_COUNT dirs ($SIZE)"
|
||||
|
||||
if [[ "$DRY_RUN" == true ]]; then
|
||||
warn "DRY RUN — would apply: chmod -R $PERMISSIONS_MODE $share_path"
|
||||
warn "DRY RUN — would apply: chown -R $PERMISSIONS_OWNER $share_path"
|
||||
PASS+=("$(basename "$share_path")")
|
||||
continue
|
||||
fi
|
||||
|
||||
# Apply ownership first — chmod after so files are owned correctly before mode change
|
||||
info "Applying ownership: $PERMISSIONS_OWNER..."
|
||||
chown -R "$PERMISSIONS_OWNER" "$share_path" 2>/dev/null
|
||||
CHOWN_EXIT=$?
|
||||
|
||||
info "Applying permissions: $PERMISSIONS_MODE..."
|
||||
chmod -R "$PERMISSIONS_MODE" "$share_path" 2>/dev/null
|
||||
CHMOD_EXIT=$?
|
||||
|
||||
if [[ "$CHOWN_EXIT" -eq 0 && "$CHMOD_EXIT" -eq 0 ]]; then
|
||||
success "$ICON_UNLOCKED $(basename "$share_path") — permissions applied"
|
||||
PASS+=("$(basename "$share_path")")
|
||||
else
|
||||
error "$(basename "$share_path") — permission repair failed"
|
||||
FAIL+=("$(basename "$share_path")")
|
||||
fi
|
||||
done
|
||||
|
||||
END=$(date +%s)
|
||||
|
||||
# -----------------------------------------------------------------------------------------------
|
||||
# ━━━ $ICON_SUMMARY Summary ━━━
|
||||
# -----------------------------------------------------------------------------------------------
|
||||
echo ""
|
||||
echo "━━━━━ $ICON_SUMMARY PERMISSIONS REPAIR SUMMARY ━━━━━"
|
||||
echo "$ICON_PERMS Mode: $PERMISSIONS_MODE"
|
||||
echo "$ICON_PERMS Owner: $PERMISSIONS_OWNER"
|
||||
echo "$ICON_TIME Duration: $(format_duration $((END - START)))"
|
||||
echo ""
|
||||
echo " $ICON_SUCCESS Pass: ${#PASS[@]} $ICON_ERROR Fail: ${#FAIL[@]}"
|
||||
echo ""
|
||||
[[ ${#PASS[@]} -gt 0 ]] && for p in "${PASS[@]}"; do echo " $ICON_UNLOCKED $p"; done
|
||||
[[ ${#FAIL[@]} -gt 0 ]] && for f in "${FAIL[@]}"; do echo " $ICON_ERROR $f"; done
|
||||
echo ""
|
||||
|
||||
if [[ "$DRY_RUN" == true ]]; then
|
||||
echo "$ICON_WARN Status: DRY RUN — no changes made"
|
||||
elif [[ ${#FAIL[@]} -gt 0 ]]; then
|
||||
echo "$ICON_ERROR Status: SOME REPAIRS FAILED"
|
||||
notify "Permissions repair failed on $(hostname) — failed shares: ${FAIL[*]}" "Permissions Repair" "warning"
|
||||
else
|
||||
echo "$ICON_DONE Status: $ICON_SUCCESS DONE"
|
||||
notify "Permissions repair complete on $(hostname) — ${#PASS[@]} share(s) repaired" "Permissions Repair" "normal"
|
||||
fi
|
||||
echo "━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━"
|
||||
@@ -0,0 +1,173 @@
|
||||
#!/bin/bash
|
||||
# -----------------------------------------------------------------------------------------------
|
||||
# --------------------------------- Container Data Export --------------------------------------
|
||||
# -----------------------------------------------------------------------------------------------
|
||||
# Exports a container's appdata directory to a compressed tar archive.
|
||||
# Stops the container before archiving and restarts it after — ensures clean consistent backup.
|
||||
#
|
||||
# Usage:
|
||||
# container_data_export.sh ContainerName /path/to/appdata /path/to/output/dir
|
||||
# container_data_export.sh Emby /mnt/media-servers/Media_Server/Emby /mnt/user/Backups/
|
||||
#
|
||||
# Output file naming:
|
||||
# ContainerName_YYYY-MM-DD_HH-MM.tar.gz
|
||||
#
|
||||
# Use before major container updates, pool migrations, or when archiving
|
||||
# a container you are removing from the stack.
|
||||
#
|
||||
# Supports --dry-run to show what would be archived without making changes.
|
||||
# -----------------------------------------------------------------------------------------------
|
||||
|
||||
SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
|
||||
|
||||
source "$SCRIPT_DIR/../Master.conf"
|
||||
source "$SCRIPT_DIR/../common.sh"
|
||||
|
||||
parse_args "$@"
|
||||
|
||||
# -----------------------------------------------------------------------------------------------
|
||||
# Args
|
||||
# -----------------------------------------------------------------------------------------------
|
||||
CONTAINER_NAME="${PARSED_ARGS[0]:-}"
|
||||
APPDATA_PATH="${PARSED_ARGS[1]:-}"
|
||||
OUTPUT_DIR="${PARSED_ARGS[2]:-}"
|
||||
|
||||
# -----------------------------------------------------------------------------------------------
|
||||
# ━━━ $ICON_GEAR Setup ━━━
|
||||
# -----------------------------------------------------------------------------------------------
|
||||
echo ""
|
||||
echo "━━━ $ICON_GEAR Setup ━━━"
|
||||
|
||||
if [[ "$EUID" -ne 0 ]]; then
|
||||
error "Must be run as root"
|
||||
exit 1
|
||||
fi
|
||||
|
||||
success "Running as root"
|
||||
|
||||
if [[ -z "$CONTAINER_NAME" || -z "$APPDATA_PATH" || -z "$OUTPUT_DIR" ]]; then
|
||||
error "Usage: container_data_export.sh <ContainerName> <appdata_path> <output_dir>"
|
||||
error "Example: container_data_export.sh Emby /mnt/media-servers/Media_Server/Emby /mnt/user/Backups/"
|
||||
exit 1
|
||||
fi
|
||||
|
||||
if [[ ! -d "$APPDATA_PATH" ]]; then
|
||||
error "Appdata path not found: $APPDATA_PATH"
|
||||
exit 1
|
||||
fi
|
||||
|
||||
if [[ ! -d "$OUTPUT_DIR" ]]; then
|
||||
error "Output directory not found: $OUTPUT_DIR"
|
||||
exit 1
|
||||
fi
|
||||
|
||||
# Check free space — rough estimate: appdata size × 1.1
|
||||
APPDATA_SIZE_KB=$(du -sk "$APPDATA_PATH" 2>/dev/null | cut -f1)
|
||||
OUTPUT_FREE_KB=$(df "$OUTPUT_DIR" --output=avail | tail -1 | tr -d ' ')
|
||||
REQUIRED_KB=$(( APPDATA_SIZE_KB * 11 / 10 ))
|
||||
|
||||
APPDATA_SIZE_H=$(du -sh "$APPDATA_PATH" 2>/dev/null | cut -f1)
|
||||
OUTPUT_FREE_H=$(df -h "$OUTPUT_DIR" --output=avail | tail -1 | tr -d ' ')
|
||||
|
||||
info "Container: $CONTAINER_NAME"
|
||||
info "Appdata: $APPDATA_PATH ($APPDATA_SIZE_H)"
|
||||
info "Output dir: $OUTPUT_DIR ($OUTPUT_FREE_H free)"
|
||||
|
||||
if [[ "$OUTPUT_FREE_KB" -lt "$REQUIRED_KB" ]]; then
|
||||
error "Insufficient space in $OUTPUT_DIR — need ~${APPDATA_SIZE_H}, have ${OUTPUT_FREE_H}"
|
||||
exit 1
|
||||
fi
|
||||
|
||||
success "Space check passed"
|
||||
[[ "$DRY_RUN" == true ]] && warn "DRY RUN — no changes will be made"
|
||||
|
||||
# -----------------------------------------------------------------------------------------------
|
||||
# ━━━ $ICON_STOP Stop Container ━━━
|
||||
# -----------------------------------------------------------------------------------------------
|
||||
echo ""
|
||||
echo "━━━ $ICON_STOP Stop Container ━━━"
|
||||
|
||||
CONTAINER_WAS_RUNNING=false
|
||||
STATUS=$(docker inspect -f '{{.State.Running}}' "$CONTAINER_NAME" 2>/dev/null)
|
||||
|
||||
if [[ "$STATUS" == "true" ]]; then
|
||||
CONTAINER_WAS_RUNNING=true
|
||||
info "$ICON_STOP Stopping $CONTAINER_NAME for clean export..."
|
||||
if [[ "$DRY_RUN" == false ]]; then
|
||||
docker stop "$CONTAINER_NAME" >/dev/null 2>&1 && \
|
||||
success "$ICON_STOPPED $CONTAINER_NAME stopped" || \
|
||||
{ error "Failed to stop $CONTAINER_NAME"; exit 1; }
|
||||
else
|
||||
warn "DRY RUN — would stop $CONTAINER_NAME"
|
||||
fi
|
||||
else
|
||||
info "$CONTAINER_NAME is not running — archiving as-is"
|
||||
fi
|
||||
|
||||
# -----------------------------------------------------------------------------------------------
|
||||
# ━━━ $ICON_SYNC Archive ━━━
|
||||
# -----------------------------------------------------------------------------------------------
|
||||
echo ""
|
||||
echo "━━━ $ICON_SYNC Archive ━━━"
|
||||
|
||||
TIMESTAMP=$(date '+%Y-%m-%d_%H-%M')
|
||||
ARCHIVE_NAME="${CONTAINER_NAME}_${TIMESTAMP}.tar.gz"
|
||||
ARCHIVE_PATH="${OUTPUT_DIR}/${ARCHIVE_NAME}"
|
||||
|
||||
info "Creating: $ARCHIVE_PATH"
|
||||
START=$(date +%s)
|
||||
|
||||
if [[ "$DRY_RUN" == false ]]; then
|
||||
tar -czf "$ARCHIVE_PATH" -C "$(dirname "$APPDATA_PATH")" "$(basename "$APPDATA_PATH")" 2>/dev/null
|
||||
TAR_EXIT=$?
|
||||
|
||||
if [[ "$TAR_EXIT" -ne 0 ]]; then
|
||||
error "Archive failed (exit code $TAR_EXIT)"
|
||||
# Restart container before exiting
|
||||
[[ "$CONTAINER_WAS_RUNNING" == true ]] && docker start "$CONTAINER_NAME" >/dev/null 2>&1
|
||||
exit 1
|
||||
fi
|
||||
|
||||
ARCHIVE_SIZE=$(du -sh "$ARCHIVE_PATH" 2>/dev/null | cut -f1)
|
||||
success "Archive created: $ARCHIVE_NAME ($ARCHIVE_SIZE)"
|
||||
else
|
||||
warn "DRY RUN — would create: $ARCHIVE_PATH"
|
||||
fi
|
||||
|
||||
END=$(date +%s)
|
||||
|
||||
# -----------------------------------------------------------------------------------------------
|
||||
# ━━━ $ICON_START Restart Container ━━━
|
||||
# -----------------------------------------------------------------------------------------------
|
||||
echo ""
|
||||
echo "━━━ $ICON_START Restart Container ━━━"
|
||||
|
||||
if [[ "$CONTAINER_WAS_RUNNING" == true ]]; then
|
||||
info "$ICON_START Restarting $CONTAINER_NAME..."
|
||||
if [[ "$DRY_RUN" == false ]]; then
|
||||
docker start "$CONTAINER_NAME" >/dev/null 2>&1 && \
|
||||
success "$ICON_STARTED $CONTAINER_NAME restarted" || \
|
||||
error "Failed to restart $CONTAINER_NAME — start it manually"
|
||||
else
|
||||
warn "DRY RUN — would restart $CONTAINER_NAME"
|
||||
fi
|
||||
else
|
||||
info "$CONTAINER_NAME was not running — not restarting"
|
||||
fi
|
||||
|
||||
# -----------------------------------------------------------------------------------------------
|
||||
# ━━━ $ICON_SUMMARY Summary ━━━
|
||||
# -----------------------------------------------------------------------------------------------
|
||||
echo ""
|
||||
echo "━━━━━ $ICON_SUMMARY CONTAINER EXPORT SUMMARY ━━━━━"
|
||||
echo "$ICON_CONTAINERS Container: $CONTAINER_NAME"
|
||||
echo "$ICON_DISK Appdata: $APPDATA_PATH ($APPDATA_SIZE_H)"
|
||||
echo "$ICON_SYNC Archive: ${ARCHIVE_NAME:-DRY RUN} ${ARCHIVE_SIZE:+($ARCHIVE_SIZE)}"
|
||||
echo "$ICON_TIME Duration: $(format_duration $((END - START)))"
|
||||
if [[ "$DRY_RUN" == true ]]; then
|
||||
echo "$ICON_WARN Status: DRY RUN — no changes made"
|
||||
else
|
||||
echo "$ICON_DONE Status: $ICON_SUCCESS DONE"
|
||||
notify "Container export complete — $CONTAINER_NAME archived to $ARCHIVE_NAME" "Container Export" "normal"
|
||||
fi
|
||||
echo "━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━"
|
||||
@@ -0,0 +1,214 @@
|
||||
#!/bin/bash
|
||||
# -----------------------------------------------------------------------------------------------
|
||||
# --------------------------------- Emby Database Repair ---------------------------------------
|
||||
# -----------------------------------------------------------------------------------------------
|
||||
# Stops Emby, runs SQLite integrity checks on all Emby databases, and restarts.
|
||||
# Use when Emby reports database corruption, unexpected crashes, or playback state issues.
|
||||
#
|
||||
# Checks performed:
|
||||
# integrity_check — full SQLite integrity verification per database file
|
||||
# quick_check — faster check for common corruption patterns
|
||||
#
|
||||
# If corruption is found:
|
||||
# Reports which database files are corrupted
|
||||
# Does NOT automatically repair — corruption repair requires manual steps
|
||||
# Provides guidance on next steps per database type
|
||||
#
|
||||
# Emby database files checked:
|
||||
# library.db — media library metadata
|
||||
# library.db-wal — write-ahead log (if exists)
|
||||
# librarydb.db — legacy library database
|
||||
# users.db — user accounts and settings
|
||||
# authentication.db — API keys and sessions
|
||||
# activity.db — activity log
|
||||
#
|
||||
# All configuration in Master.conf — uses TRANSCODE_EMBY_CONTAINER and EMBY_URL.
|
||||
# Supports --dry-run to show what would be checked without stopping Emby.
|
||||
# -----------------------------------------------------------------------------------------------
|
||||
|
||||
SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
|
||||
|
||||
source "$SCRIPT_DIR/../Master.conf"
|
||||
source "$SCRIPT_DIR/../common.sh"
|
||||
|
||||
parse_args "$@"
|
||||
|
||||
# Emby config path — inside the container it's /config, map to host path
|
||||
# Detected from Docker inspect at runtime
|
||||
EMBY_CONTAINER="$TRANSCODE_EMBY_CONTAINER"
|
||||
|
||||
# -----------------------------------------------------------------------------------------------
|
||||
# ━━━ $ICON_GEAR Setup ━━━
|
||||
# -----------------------------------------------------------------------------------------------
|
||||
echo ""
|
||||
echo "━━━ $ICON_GEAR Setup ━━━"
|
||||
|
||||
if [[ "$EUID" -ne 0 ]]; then
|
||||
error "Must be run as root"
|
||||
exit 1
|
||||
fi
|
||||
|
||||
success "Running as root"
|
||||
|
||||
if ! command -v sqlite3 >/dev/null 2>&1; then
|
||||
error "sqlite3 not found — install sqlite package"
|
||||
exit 1
|
||||
fi
|
||||
|
||||
success "sqlite3 available"
|
||||
[[ "$DRY_RUN" == true ]] && warn "DRY RUN — Emby will not be stopped"
|
||||
|
||||
# Detect Emby config path from Docker mount
|
||||
EMBY_CONFIG_HOST=$(docker inspect "$EMBY_CONTAINER" 2>/dev/null | \
|
||||
jq -r '.[] | .Mounts[] | select(.Destination == "/config") | .Source' 2>/dev/null)
|
||||
|
||||
if [[ -z "$EMBY_CONFIG_HOST" ]]; then
|
||||
error "Could not detect Emby config path from Docker mounts"
|
||||
error "Make sure $EMBY_CONTAINER is the correct container name in Master.conf"
|
||||
exit 1
|
||||
fi
|
||||
|
||||
success "Emby config path: $EMBY_CONFIG_HOST"
|
||||
|
||||
# -----------------------------------------------------------------------------------------------
|
||||
# ━━━ $ICON_STOP Stop Emby ━━━
|
||||
# -----------------------------------------------------------------------------------------------
|
||||
echo ""
|
||||
echo "━━━ $ICON_STOP Stop Emby ━━━"
|
||||
|
||||
EMBY_WAS_RUNNING=false
|
||||
STATUS=$(docker inspect -f '{{.State.Running}}' "$EMBY_CONTAINER" 2>/dev/null)
|
||||
|
||||
if [[ "$STATUS" == "true" ]]; then
|
||||
EMBY_WAS_RUNNING=true
|
||||
warn "Stopping $EMBY_CONTAINER — active sessions will be interrupted"
|
||||
if [[ "$DRY_RUN" == false ]]; then
|
||||
docker stop "$EMBY_CONTAINER" >/dev/null 2>&1 && \
|
||||
success "$ICON_STOPPED $EMBY_CONTAINER stopped" || \
|
||||
{ error "Failed to stop $EMBY_CONTAINER"; exit 1; }
|
||||
sleep 3 # brief wait for file handles to release
|
||||
else
|
||||
warn "DRY RUN — would stop $EMBY_CONTAINER"
|
||||
fi
|
||||
else
|
||||
info "$EMBY_CONTAINER is not running — proceeding with checks"
|
||||
fi
|
||||
|
||||
# -----------------------------------------------------------------------------------------------
|
||||
# ━━━ $ICON_HEALTH Database Integrity Check ━━━
|
||||
# -----------------------------------------------------------------------------------------------
|
||||
echo ""
|
||||
echo "━━━ $ICON_HEALTH Database Integrity Check ━━━"
|
||||
|
||||
START=$(date +%s)
|
||||
|
||||
# Emby database files to check
|
||||
DB_FILES=(
|
||||
"data/library.db"
|
||||
"data/librarydb.db"
|
||||
"data/users.db"
|
||||
"data/authentication.db"
|
||||
"data/activity.db"
|
||||
)
|
||||
|
||||
PASS_DBS=()
|
||||
FAIL_DBS=()
|
||||
MISSING_DBS=()
|
||||
|
||||
for db_rel in "${DB_FILES[@]}"; do
|
||||
db_path="${EMBY_CONFIG_HOST}/${db_rel}"
|
||||
db_name=$(basename "$db_rel")
|
||||
|
||||
if [[ ! -f "$db_path" ]]; then
|
||||
log "$db_name — not found, skipping"
|
||||
MISSING_DBS+=("$db_name")
|
||||
continue
|
||||
fi
|
||||
|
||||
DB_SIZE=$(du -sh "$db_path" 2>/dev/null | cut -f1)
|
||||
info "$ICON_HEALTH Checking $db_name ($DB_SIZE)..."
|
||||
|
||||
if [[ "$DRY_RUN" == true ]]; then
|
||||
warn "DRY RUN — would check: $db_path"
|
||||
continue
|
||||
fi
|
||||
|
||||
# Run integrity check
|
||||
RESULT=$(sqlite3 "$db_path" "PRAGMA integrity_check;" 2>/dev/null)
|
||||
EXIT_CODE=$?
|
||||
|
||||
if [[ "$EXIT_CODE" -ne 0 ]]; then
|
||||
error "$db_name — sqlite3 could not open database (may be locked or corrupt)"
|
||||
FAIL_DBS+=("$db_name")
|
||||
elif [[ "$RESULT" == "ok" ]]; then
|
||||
success "$db_name — integrity check passed"
|
||||
PASS_DBS+=("$db_name")
|
||||
else
|
||||
error "$db_name — integrity check FAILED"
|
||||
echo "$RESULT" | head -10 | while IFS= read -r line; do
|
||||
error " $line"
|
||||
done
|
||||
FAIL_DBS+=("$db_name")
|
||||
fi
|
||||
done
|
||||
|
||||
END=$(date +%s)
|
||||
|
||||
# -----------------------------------------------------------------------------------------------
|
||||
# ━━━ $ICON_START Restart Emby ━━━
|
||||
# -----------------------------------------------------------------------------------------------
|
||||
echo ""
|
||||
echo "━━━ $ICON_START Restart Emby ━━━"
|
||||
|
||||
if [[ "$EMBY_WAS_RUNNING" == true && "$DRY_RUN" == false ]]; then
|
||||
docker start "$EMBY_CONTAINER" >/dev/null 2>&1 && \
|
||||
success "$ICON_STARTED $EMBY_CONTAINER restarted" || \
|
||||
error "Failed to restart $EMBY_CONTAINER — start it manually"
|
||||
elif [[ "$DRY_RUN" == true && "$EMBY_WAS_RUNNING" == true ]]; then
|
||||
warn "DRY RUN — would restart $EMBY_CONTAINER"
|
||||
else
|
||||
info "$EMBY_CONTAINER was not running — not restarting"
|
||||
fi
|
||||
|
||||
# -----------------------------------------------------------------------------------------------
|
||||
# ━━━ $ICON_SUMMARY Summary ━━━
|
||||
# -----------------------------------------------------------------------------------------------
|
||||
echo ""
|
||||
echo "━━━━━ $ICON_SUMMARY EMBY DATABASE REPAIR SUMMARY ━━━━━"
|
||||
echo "$ICON_HEALTH Container: $EMBY_CONTAINER"
|
||||
echo "$ICON_HEALTH Config path: $EMBY_CONFIG_HOST"
|
||||
echo "$ICON_TIME Duration: $(format_duration $((END - START)))"
|
||||
echo ""
|
||||
echo " $ICON_SUCCESS Passed: ${#PASS_DBS[@]} $ICON_ERROR Failed: ${#FAIL_DBS[@]} $ICON_INFO Missing: ${#MISSING_DBS[@]}"
|
||||
echo ""
|
||||
|
||||
if [[ ${#PASS_DBS[@]} -gt 0 ]]; then
|
||||
for db in "${PASS_DBS[@]}"; do echo " $ICON_SUCCESS $db"; done
|
||||
fi
|
||||
if [[ ${#FAIL_DBS[@]} -gt 0 ]]; then
|
||||
for db in "${FAIL_DBS[@]}"; do echo " $ICON_ERROR $db"; done
|
||||
fi
|
||||
|
||||
echo ""
|
||||
|
||||
if [[ "$DRY_RUN" == true ]]; then
|
||||
echo "$ICON_WARN Status: DRY RUN — no checks performed"
|
||||
elif [[ ${#FAIL_DBS[@]} -gt 0 ]]; then
|
||||
echo "$ICON_ERROR Status: CORRUPTION FOUND"
|
||||
echo ""
|
||||
echo "$ICON_INFO Next steps for corrupted databases:"
|
||||
echo " library.db — Stop Emby, delete library.db, restart"
|
||||
echo " Emby will rebuild from media files (slow first start)"
|
||||
echo " users.db — Stop Emby, restore from backup or delete"
|
||||
echo " Deleting resets all user accounts"
|
||||
echo " authentication.db — Stop Emby, delete, restart"
|
||||
echo " API keys and sessions will be regenerated"
|
||||
echo " activity.db — Stop Emby, delete, restart — activity log only"
|
||||
echo ""
|
||||
echo "$ICON_WARN Always take a backup before deleting any database file"
|
||||
notify "Emby database corruption found on $(hostname) — failed: ${FAIL_DBS[*]} — manual intervention needed" "Emby DB Repair" "warning"
|
||||
else
|
||||
echo "$ICON_DONE Status: $ICON_SUCCESS ALL DATABASES HEALTHY"
|
||||
notify "Emby database integrity check passed on $(hostname) — ${#PASS_DBS[@]} databases healthy" "Emby DB Repair" "normal"
|
||||
fi
|
||||
echo "━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━"
|
||||
@@ -0,0 +1,127 @@
|
||||
#!/bin/bash
|
||||
# -----------------------------------------------------------------------------------------------
|
||||
# --------------------------------- Failover State Reset ---------------------------------------
|
||||
# -----------------------------------------------------------------------------------------------
|
||||
# Resets the failover state file to NORMAL and clears all tier flags.
|
||||
# Use when the failover state file is stuck in a non-NORMAL state after testing,
|
||||
# a failed handback, or manual intervention that left state inconsistent.
|
||||
#
|
||||
# Does NOT start or stop any containers — state file only.
|
||||
# After reset, failover.sh will resume from NORMAL on its next cycle.
|
||||
#
|
||||
# ⚠️ Only run this when you have manually verified both servers are in their
|
||||
# correct states — right containers running on the right server, DDNS correct.
|
||||
# Resetting state without verifying the actual state can cause failover.sh
|
||||
# to make incorrect decisions on its next cycle.
|
||||
#
|
||||
# Supports --dry-run to show what would be reset without changing anything.
|
||||
# Supports --status to show the current state file contents.
|
||||
# -----------------------------------------------------------------------------------------------
|
||||
|
||||
SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
|
||||
|
||||
source "$SCRIPT_DIR/../Master.conf"
|
||||
source "$SCRIPT_DIR/../common.sh"
|
||||
|
||||
parse_args "$@"
|
||||
|
||||
# -----------------------------------------------------------------------------------------------
|
||||
# ━━━ $ICON_GEAR Setup ━━━
|
||||
# -----------------------------------------------------------------------------------------------
|
||||
echo ""
|
||||
echo "━━━ $ICON_GEAR Setup ━━━"
|
||||
|
||||
if [[ "$EUID" -ne 0 ]]; then
|
||||
error "Must be run as root"
|
||||
exit 1
|
||||
fi
|
||||
|
||||
success "Running as root"
|
||||
[[ "$DRY_RUN" == true ]] && warn "DRY RUN — no changes will be made"
|
||||
|
||||
# -----------------------------------------------------------------------------------------------
|
||||
# ━━━ $ICON_SUMMARY Current State ━━━
|
||||
# -----------------------------------------------------------------------------------------------
|
||||
echo ""
|
||||
echo "━━━ $ICON_SUMMARY Current State ━━━"
|
||||
|
||||
if [[ ! -f "$FAILOVER_STATE_FILE" ]]; then
|
||||
warn "State file not found: $FAILOVER_STATE_FILE"
|
||||
warn "Will be created fresh on reset"
|
||||
else
|
||||
info "State file: $FAILOVER_STATE_FILE"
|
||||
echo ""
|
||||
while IFS='=' read -r key value; do
|
||||
[[ -z "$key" ]] && continue
|
||||
echo " $ICON_INFO $key = $value"
|
||||
done < "$FAILOVER_STATE_FILE"
|
||||
fi
|
||||
|
||||
if [[ "$SHOW_STATUS" == true ]]; then
|
||||
exit 0
|
||||
fi
|
||||
|
||||
# -----------------------------------------------------------------------------------------------
|
||||
# ━━━ Confirmation ━━━
|
||||
# -----------------------------------------------------------------------------------------------
|
||||
echo ""
|
||||
warn "$ICON_WARN This will reset the failover state to NORMAL"
|
||||
warn "Only proceed if you have verified both servers are in their correct states"
|
||||
warn " — Right containers running on the right server"
|
||||
warn " — DDNS pointing at the correct server"
|
||||
warn " — No active failover in progress"
|
||||
echo ""
|
||||
|
||||
if [[ "$DRY_RUN" == false ]]; then
|
||||
read -r -p "Type YES to confirm reset: " CONFIRM
|
||||
if [[ "$CONFIRM" != "YES" ]]; then
|
||||
info "Reset cancelled"
|
||||
exit 0
|
||||
fi
|
||||
fi
|
||||
|
||||
# -----------------------------------------------------------------------------------------------
|
||||
# ━━━ $ICON_FAILOVER Reset State File ━━━
|
||||
# -----------------------------------------------------------------------------------------------
|
||||
echo ""
|
||||
echo "━━━ $ICON_FAILOVER Resetting State File ━━━"
|
||||
|
||||
if [[ "$DRY_RUN" == true ]]; then
|
||||
warn "DRY RUN — would write:"
|
||||
echo " state=NORMAL"
|
||||
echo " failover_start=0"
|
||||
echo " handback_strikes=0"
|
||||
echo " tier2_started=false"
|
||||
echo " tier3_started=false"
|
||||
echo " tier4_started=false"
|
||||
echo " last_reset=$(date '+%Y-%m-%d %H:%M:%S')"
|
||||
else
|
||||
mkdir -p "$(dirname "$FAILOVER_STATE_FILE")"
|
||||
cat > "$FAILOVER_STATE_FILE" << EOF
|
||||
state=NORMAL
|
||||
failover_start=0
|
||||
handback_strikes=0
|
||||
tier2_started=false
|
||||
tier3_started=false
|
||||
tier4_started=false
|
||||
last_reset=$(date '+%Y-%m-%d %H:%M:%S')
|
||||
EOF
|
||||
success "State file reset to NORMAL"
|
||||
fi
|
||||
|
||||
# -----------------------------------------------------------------------------------------------
|
||||
# ━━━ $ICON_SUMMARY Summary ━━━
|
||||
# -----------------------------------------------------------------------------------------------
|
||||
echo ""
|
||||
echo "━━━━━ $ICON_SUMMARY FAILOVER STATE RESET SUMMARY ━━━━━"
|
||||
if [[ "$DRY_RUN" == true ]]; then
|
||||
echo "$ICON_WARN Status: DRY RUN — no changes made"
|
||||
else
|
||||
echo "$ICON_DONE Status: $ICON_SUCCESS State reset to NORMAL"
|
||||
echo "$ICON_TIME Reset at: $(date '+%Y-%m-%d %H:%M:%S')"
|
||||
echo ""
|
||||
echo "$ICON_INFO failover.sh will resume from NORMAL on next cycle"
|
||||
echo "$ICON_INFO No containers were started or stopped"
|
||||
notify "Failover state manually reset to NORMAL on $(hostname)" "Failover State Reset" "normal"
|
||||
fi
|
||||
echo "━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━"
|
||||
@@ -0,0 +1,175 @@
|
||||
#!/bin/bash
|
||||
# -----------------------------------------------------------------------------------------------
|
||||
# --------------------------------- Watchdog Skip List Manager ---------------------------------
|
||||
# -----------------------------------------------------------------------------------------------
|
||||
# View and manage the persistent container skip list used by docker_watchdog.sh.
|
||||
# Containers are added to the skip list when they exceed the restart loop limit.
|
||||
# They stay there until manually cleared or until found running again automatically.
|
||||
#
|
||||
# Usage:
|
||||
# watchdog_skip_list_manager.sh --status — show current skip list and restart history
|
||||
# watchdog_skip_list_manager.sh --clear-all — clear all skip lists and restart history
|
||||
# watchdog_skip_list_manager.sh --clear ContainerName — clear specific container
|
||||
#
|
||||
# After clearing a container from the skip list:
|
||||
# 1. Fix whatever was causing the container to fail
|
||||
# 2. Start the container manually: docker start ContainerName
|
||||
# 3. The watchdog will monitor it normally on the next cycle
|
||||
#
|
||||
# Files managed:
|
||||
# SYS_WATCHDOG_FAILED_FILE — persistent container skip list
|
||||
# WATCHDOG_CONTAINER_RESTART_LOG — restart history for loop detection
|
||||
# -----------------------------------------------------------------------------------------------
|
||||
|
||||
SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
|
||||
|
||||
source "$SCRIPT_DIR/../Master.conf"
|
||||
source "$SCRIPT_DIR/../common.sh"
|
||||
|
||||
parse_args "$@"
|
||||
|
||||
# Parse action from args
|
||||
ACTION=""
|
||||
TARGET_CONTAINER=""
|
||||
|
||||
for arg in "${PARSED_ARGS[@]}"; do
|
||||
case "$arg" in
|
||||
--clear-all) ACTION="clear-all" ;;
|
||||
--clear) ACTION="clear" ;;
|
||||
--status) ACTION="status" ;;
|
||||
*)
|
||||
[[ "$ACTION" == "clear" && -z "$TARGET_CONTAINER" ]] && TARGET_CONTAINER="$arg"
|
||||
;;
|
||||
esac
|
||||
done
|
||||
|
||||
[[ -z "$ACTION" ]] && ACTION="status"
|
||||
|
||||
# -----------------------------------------------------------------------------------------------
|
||||
# ━━━ $ICON_GEAR Setup ━━━
|
||||
# -----------------------------------------------------------------------------------------------
|
||||
echo ""
|
||||
echo "━━━ $ICON_GEAR Setup ━━━"
|
||||
|
||||
if [[ "$EUID" -ne 0 ]]; then
|
||||
error "Must be run as root"
|
||||
exit 1
|
||||
fi
|
||||
|
||||
success "Running as root"
|
||||
[[ "$DRY_RUN" == true ]] && warn "DRY RUN — no changes will be made"
|
||||
|
||||
touch "$SYS_WATCHDOG_FAILED_FILE" "$WATCHDOG_CONTAINER_RESTART_LOG" 2>/dev/null
|
||||
|
||||
# -----------------------------------------------------------------------------------------------
|
||||
# ━━━ STATUS ━━━
|
||||
# -----------------------------------------------------------------------------------------------
|
||||
echo ""
|
||||
echo "━━━ $ICON_WATCHDOG Skip List Status ━━━"
|
||||
|
||||
SKIP_COUNT=$(grep -c "." "$SYS_WATCHDOG_FAILED_FILE" 2>/dev/null || echo 0)
|
||||
RESTART_COUNT=$(wc -l < "$WATCHDOG_CONTAINER_RESTART_LOG" 2>/dev/null || echo 0)
|
||||
|
||||
if [[ "$SKIP_COUNT" -eq 0 ]]; then
|
||||
success "Skip list is empty — all containers healthy"
|
||||
else
|
||||
warn "$SKIP_COUNT container(s) on skip list:"
|
||||
while IFS= read -r container; do
|
||||
[[ -z "$container" ]] && continue
|
||||
# Check if container is currently running
|
||||
STATUS=$(docker inspect -f '{{.State.Running}}' "$container" 2>/dev/null || echo "unknown")
|
||||
if [[ "$STATUS" == "true" ]]; then
|
||||
echo " $ICON_RUNNING $container — currently RUNNING (will auto-clear on next watchdog cycle)"
|
||||
elif [[ "$STATUS" == "false" ]]; then
|
||||
echo " $ICON_STOPPED $container — currently STOPPED — fix and start manually"
|
||||
else
|
||||
echo " $ICON_INFO $container — container not found"
|
||||
fi
|
||||
done < "$SYS_WATCHDOG_FAILED_FILE"
|
||||
fi
|
||||
|
||||
echo ""
|
||||
echo "━━━ $ICON_WATCHDOG Restart History ━━━"
|
||||
if [[ "$RESTART_COUNT" -eq 0 ]]; then
|
||||
success "No restart history"
|
||||
else
|
||||
info "$RESTART_COUNT restart entries (window: ${WATCHDOG_CONTAINER_RESTART_WINDOW}h)"
|
||||
echo ""
|
||||
# Show per-container restart counts
|
||||
awk -F'|' '{counts[$1]++} END {for (c in counts) printf " %-30s %d restart(s)\n", c, counts[c]}' \
|
||||
"$WATCHDOG_CONTAINER_RESTART_LOG" 2>/dev/null | sort
|
||||
fi
|
||||
|
||||
[[ "$ACTION" == "status" ]] && exit 0
|
||||
|
||||
# -----------------------------------------------------------------------------------------------
|
||||
# ━━━ CLEAR ALL ━━━
|
||||
# -----------------------------------------------------------------------------------------------
|
||||
if [[ "$ACTION" == "clear-all" ]]; then
|
||||
echo ""
|
||||
echo "━━━ $ICON_TRASH Clear All Skip Lists ━━━"
|
||||
warn "This will clear the skip list and restart history for ALL containers"
|
||||
|
||||
if [[ "$DRY_RUN" == false ]]; then
|
||||
read -r -p "Type YES to confirm: " CONFIRM
|
||||
if [[ "$CONFIRM" != "YES" ]]; then
|
||||
info "Cancelled"
|
||||
exit 0
|
||||
fi
|
||||
|
||||
> "$SYS_WATCHDOG_FAILED_FILE"
|
||||
> "$WATCHDOG_CONTAINER_RESTART_LOG"
|
||||
success "Skip list cleared"
|
||||
success "Restart history cleared"
|
||||
notify "Watchdog skip list manually cleared on $(hostname) — all containers will be monitored normally" "Watchdog Manager" "normal"
|
||||
else
|
||||
warn "DRY RUN — would clear: $SYS_WATCHDOG_FAILED_FILE"
|
||||
warn "DRY RUN — would clear: $WATCHDOG_CONTAINER_RESTART_LOG"
|
||||
fi
|
||||
fi
|
||||
|
||||
# -----------------------------------------------------------------------------------------------
|
||||
# ━━━ CLEAR SPECIFIC CONTAINER ━━━
|
||||
# -----------------------------------------------------------------------------------------------
|
||||
if [[ "$ACTION" == "clear" ]]; then
|
||||
echo ""
|
||||
echo "━━━ $ICON_TRASH Clear Container: $TARGET_CONTAINER ━━━"
|
||||
|
||||
if [[ -z "$TARGET_CONTAINER" ]]; then
|
||||
error "No container specified. Usage: --clear ContainerName"
|
||||
exit 1
|
||||
fi
|
||||
|
||||
if ! grep -q "^${TARGET_CONTAINER}$" "$SYS_WATCHDOG_FAILED_FILE" 2>/dev/null; then
|
||||
warn "$TARGET_CONTAINER is not on the skip list"
|
||||
else
|
||||
if [[ "$DRY_RUN" == false ]]; then
|
||||
sed -i "/^${TARGET_CONTAINER}$/d" "$SYS_WATCHDOG_FAILED_FILE"
|
||||
success "$TARGET_CONTAINER removed from skip list"
|
||||
else
|
||||
warn "DRY RUN — would remove $TARGET_CONTAINER from skip list"
|
||||
fi
|
||||
fi
|
||||
|
||||
# Clear restart history for this container
|
||||
HIST_COUNT=$(grep -c "^${TARGET_CONTAINER}|" "$WATCHDOG_CONTAINER_RESTART_LOG" 2>/dev/null || echo 0)
|
||||
if [[ "$HIST_COUNT" -gt 0 ]]; then
|
||||
if [[ "$DRY_RUN" == false ]]; then
|
||||
sed -i "/^${TARGET_CONTAINER}|/d" "$WATCHDOG_CONTAINER_RESTART_LOG"
|
||||
success "Cleared $HIST_COUNT restart history entries for $TARGET_CONTAINER"
|
||||
else
|
||||
warn "DRY RUN — would clear $HIST_COUNT restart history entries"
|
||||
fi
|
||||
else
|
||||
info "No restart history for $TARGET_CONTAINER"
|
||||
fi
|
||||
|
||||
echo ""
|
||||
echo "$ICON_INFO Next steps:"
|
||||
echo " 1. Fix whatever was causing $TARGET_CONTAINER to fail"
|
||||
echo " 2. Start it manually: docker start $TARGET_CONTAINER"
|
||||
echo " 3. Watchdog will monitor it normally on the next cycle"
|
||||
fi
|
||||
|
||||
echo ""
|
||||
echo "━━━━━ $ICON_SUMMARY DONE ━━━━━"
|
||||
@@ -0,0 +1,193 @@
|
||||
#!/bin/bash
|
||||
# -----------------------------------------------------------------------------------------------
|
||||
# --------------------------------- ZFS Pool Scrub ---------------------------------------------
|
||||
# -----------------------------------------------------------------------------------------------
|
||||
# Triggers a ZFS scrub on all pools (or a specific pool) and waits for completion.
|
||||
# Sends a notification when scrub completes with a summary of any errors found.
|
||||
#
|
||||
# ZFS scrub reads every block on every pool and verifies checksums — it catches
|
||||
# silent data corruption that would otherwise only surface when you try to read
|
||||
# the corrupted data. Running monthly is recommended for all ZFS pools.
|
||||
#
|
||||
# Usage:
|
||||
# zfs_pool_scrub.sh — scrub all pools
|
||||
# zfs_pool_scrub.sh poolname — scrub specific pool only
|
||||
# zfs_pool_scrub.sh --status — show scrub status for all pools
|
||||
# zfs_pool_scrub.sh --dry-run — show what would be scrubbed
|
||||
#
|
||||
# Pools in ZFS_REPORT_IGNORE_POOLS are skipped unless specified explicitly.
|
||||
# Scrub runs in background — script polls until complete then reports.
|
||||
# Safe to run while the pool is in use — scrub does not interrupt normal I/O.
|
||||
# -----------------------------------------------------------------------------------------------
|
||||
|
||||
SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
|
||||
|
||||
source "$SCRIPT_DIR/../Master.conf"
|
||||
source "$SCRIPT_DIR/../common.sh"
|
||||
|
||||
parse_args "$@"
|
||||
|
||||
TARGET_POOL="${PARSED_ARGS[0]:-}"
|
||||
|
||||
# -----------------------------------------------------------------------------------------------
|
||||
# ━━━ $ICON_GEAR Setup ━━━
|
||||
# -----------------------------------------------------------------------------------------------
|
||||
echo ""
|
||||
echo "━━━ $ICON_GEAR Setup ━━━"
|
||||
|
||||
if [[ "$EUID" -ne 0 ]]; then
|
||||
error "Must be run as root"
|
||||
exit 1
|
||||
fi
|
||||
|
||||
success "Running as root"
|
||||
|
||||
if ! command -v zpool >/dev/null 2>&1; then
|
||||
error "ZFS not available on this system"
|
||||
exit 1
|
||||
fi
|
||||
|
||||
success "ZFS available"
|
||||
[[ "$DRY_RUN" == true ]] && warn "DRY RUN — no scrubs will be started"
|
||||
|
||||
# Build ignore map
|
||||
declare -A IGNORE_MAP
|
||||
for pool in "${ZFS_REPORT_IGNORE_POOLS[@]}"; do
|
||||
[[ -n "$pool" ]] && IGNORE_MAP["$pool"]=1
|
||||
done
|
||||
|
||||
# -----------------------------------------------------------------------------------------------
|
||||
# ━━━ $ICON_SUMMARY Status ━━━
|
||||
# -----------------------------------------------------------------------------------------------
|
||||
if [[ "$SHOW_STATUS" == true ]]; then
|
||||
echo ""
|
||||
echo "━━━━━ $ICON_SUMMARY SCRUB STATUS ━━━━━"
|
||||
zpool list -H -o name 2>/dev/null | while read -r pool; do
|
||||
SCAN=$(zpool status "$pool" 2>/dev/null | grep "scan:")
|
||||
echo " $ICON_ZFS $pool — $SCAN"
|
||||
done
|
||||
echo "━━━━━━━━━━━━━━━━━━━━━━━"
|
||||
exit 0
|
||||
fi
|
||||
|
||||
# -----------------------------------------------------------------------------------------------
|
||||
# Build pool list to scrub
|
||||
# -----------------------------------------------------------------------------------------------
|
||||
POOLS_TO_SCRUB=()
|
||||
|
||||
if [[ -n "$TARGET_POOL" ]]; then
|
||||
# Specific pool requested — validate it exists
|
||||
if ! zpool list "$TARGET_POOL" >/dev/null 2>&1; then
|
||||
error "Pool not found: $TARGET_POOL"
|
||||
exit 1
|
||||
fi
|
||||
POOLS_TO_SCRUB=("$TARGET_POOL")
|
||||
else
|
||||
# All pools — skip ignored ones
|
||||
while IFS= read -r pool; do
|
||||
[[ -z "$pool" ]] && continue
|
||||
if [[ -n "${IGNORE_MAP[$pool]:-}" ]]; then
|
||||
info "Skipping $pool (in ZFS_REPORT_IGNORE_POOLS)"
|
||||
continue
|
||||
fi
|
||||
POOLS_TO_SCRUB+=("$pool")
|
||||
done < <(zpool list -H -o name 2>/dev/null)
|
||||
fi
|
||||
|
||||
if [[ ${#POOLS_TO_SCRUB[@]} -eq 0 ]]; then
|
||||
warn "No pools to scrub"
|
||||
exit 0
|
||||
fi
|
||||
|
||||
info "Pools to scrub: ${POOLS_TO_SCRUB[*]}"
|
||||
|
||||
# -----------------------------------------------------------------------------------------------
|
||||
# ━━━ $ICON_ZFS Start Scrubs ━━━
|
||||
# -----------------------------------------------------------------------------------------------
|
||||
echo ""
|
||||
echo "━━━ $ICON_ZFS Starting ZFS Scrubs ━━━"
|
||||
START=$(date +%s)
|
||||
|
||||
for pool in "${POOLS_TO_SCRUB[@]}"; do
|
||||
info "$ICON_ZFS Starting scrub on $pool..."
|
||||
if [[ "$DRY_RUN" == false ]]; then
|
||||
zpool scrub "$pool" 2>/dev/null && \
|
||||
success "$pool scrub started" || \
|
||||
error "Failed to start scrub on $pool"
|
||||
else
|
||||
warn "DRY RUN — would scrub: $pool"
|
||||
fi
|
||||
done
|
||||
|
||||
[[ "$DRY_RUN" == true ]] && {
|
||||
echo ""
|
||||
echo "━━━━━ $ICON_SUMMARY ZFS SCRUB SUMMARY ━━━━━"
|
||||
echo "$ICON_WARN Status: DRY RUN — no scrubs started"
|
||||
echo "━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━"
|
||||
exit 0
|
||||
}
|
||||
|
||||
# -----------------------------------------------------------------------------------------------
|
||||
# ━━━ Poll until complete ━━━
|
||||
# -----------------------------------------------------------------------------------------------
|
||||
echo ""
|
||||
echo "━━━ $ICON_TIME Waiting for scrubs to complete ━━━"
|
||||
info "Polling every 60 seconds — this may take a while on large pools"
|
||||
info "Safe to leave running — scrub continues even if this script is stopped"
|
||||
|
||||
STILL_RUNNING=true
|
||||
while [[ "$STILL_RUNNING" == true ]]; do
|
||||
sleep 60
|
||||
STILL_RUNNING=false
|
||||
for pool in "${POOLS_TO_SCRUB[@]}"; do
|
||||
STATUS=$(zpool status "$pool" 2>/dev/null | grep "scan:" | grep -c "in progress" || true)
|
||||
if [[ "$STATUS" -gt 0 ]]; then
|
||||
STILL_RUNNING=true
|
||||
REPAIRED=$(zpool status "$pool" 2>/dev/null | grep "scan:" | grep -oE "[0-9]+ repaired")
|
||||
log "$pool — scrub in progress ${REPAIRED:+($REPAIRED)}"
|
||||
fi
|
||||
done
|
||||
done
|
||||
|
||||
END=$(date +%s)
|
||||
success "All scrubs complete"
|
||||
|
||||
# -----------------------------------------------------------------------------------------------
|
||||
# ━━━ $ICON_SUMMARY Results ━━━
|
||||
# -----------------------------------------------------------------------------------------------
|
||||
echo ""
|
||||
echo "━━━ $ICON_ZFS Scrub Results ━━━"
|
||||
|
||||
POOLS_OK=()
|
||||
POOLS_ERRORS=()
|
||||
|
||||
for pool in "${POOLS_TO_SCRUB[@]}"; do
|
||||
SCAN_LINE=$(zpool status "$pool" 2>/dev/null | grep "scan:")
|
||||
ERRORS=$(zpool status "$pool" 2>/dev/null | grep "errors:" | grep -v "No known data errors")
|
||||
|
||||
if [[ -n "$ERRORS" ]]; then
|
||||
error "$pool — $SCAN_LINE"
|
||||
error "$pool — $ERRORS"
|
||||
POOLS_ERRORS+=("$pool")
|
||||
else
|
||||
success "$pool — $SCAN_LINE"
|
||||
POOLS_OK+=("$pool")
|
||||
fi
|
||||
done
|
||||
|
||||
echo ""
|
||||
echo "━━━━━ $ICON_SUMMARY ZFS SCRUB SUMMARY ━━━━━"
|
||||
echo "$ICON_ZFS Pools scrubbed: ${#POOLS_TO_SCRUB[@]}"
|
||||
echo "$ICON_SUCCESS Clean: ${#POOLS_OK[@]}"
|
||||
echo "$ICON_ERROR Errors: ${#POOLS_ERRORS[@]}"
|
||||
echo "$ICON_TIME Duration: $(format_duration $((END - START)))"
|
||||
echo ""
|
||||
|
||||
if [[ ${#POOLS_ERRORS[@]} -gt 0 ]]; then
|
||||
echo "$ICON_ERROR Status: ERRORS FOUND — ${POOLS_ERRORS[*]}"
|
||||
notify "ZFS scrub complete on $(hostname) — ERRORS found in pools: ${POOLS_ERRORS[*]}" "ZFS Scrub" "warning"
|
||||
else
|
||||
echo "$ICON_DONE Status: $ICON_SUCCESS ALL POOLS CLEAN"
|
||||
notify "ZFS scrub complete on $(hostname) — ${#POOLS_OK[@]} pools clean in $(format_duration $((END - START)))" "ZFS Scrub" "normal"
|
||||
fi
|
||||
echo "━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━"
|
||||
Reference in New Issue
Block a user