FailedProxy/Varaverk
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Watchdogs/ docs: new README + Manual, update affected folders

Browse Source 
New docs:
  Watchdogs/README-Watchdogs.md  — design, relationships, script table, state file map
  Watchdogs/Manual-Watchdogs.md  — full config reference for all 4 watchdogs

Docker_Essentials/:
  README — remove docker_watchdog, update folder description and diagrams
  Manual  — strip watchdog config sections, add pointer to Watchdogs/Manual

unRAID_Essentials/:
  README — remove system/resource watchdog, update diagrams and script table
  Manual  — strip system/resource watchdog sections, update TOC + config reference

README.md:
  Add Watchdogs/ to folder structure
  Fix "WHAT RUNS WHEN" — watchdogs run via orchestrator every minute, not array start
  Fix daily cycle and monitoring diagrams
This commit is contained in:
Gmer4Lfe
2026-05-22 17:26:21 -04:00
parent ec79a926e8
commit 8a3e22c9b8
7 changed files with 912 additions and 707 deletions
Download Patch File Download Diff File Expand all files Collapse all files
Docker_Essentials/Manual-Docker_Essentials.md
+25 -364
View File
@@ -6,235 +6,9 @@ Configuration reference, setup procedures, and operational workflows.
For folder overview and design philosophy see `README-Docker_Essentials.md`.
For per-script detail see the script headers directly.
---
## ━━━ WATCHDOG CONFIGURATION ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
All watchdog configuration lives in `host*.conf` (per-container lists) and
`master.conf` (shared thresholds and toggles). `detect_hosts()` aliases all
`HOST1_` / `HOST2_` prefixed vars to their unprefixed names at runtime — scripts
always read the right values for the server they're running on.
---
### ── Memory Hard Limits ────────────────────────────────────────────────────────
```bash
# host1.conf
# Format: "ContainerName:LimitInMB"
# Immediate restart when exceeded — no strike system. Memory leaks are not spikes.
#
# Sizing guidance:
# Check normal peak: docker stats ContainerName
# Set limit at ~150-200% of normal peak
# Emby peaks ~12GB under heavy transcode load → 18GB gives headroom
# without triggering on legitimate load spikes
#
HOST1_WATCHDOG_CONTAINERS=(
"Emby:18432" # 18GB — peaks ~12GB under heavy transcode load
"LidaTube:6144" # 6GB — YouTube downloader, grows with large queues
"Tdarr:6144" # 6GB — video transcoder, memory-intensive by nature
"Code-Server:1024" # 1GB — IDE, should be light; 1GB is generous
)
```
A soft warning fires at `SOFT_MEM_THRESHOLD=80` percent of the hard limit — early
visibility into a container approaching its ceiling before a restart is triggered.
---
### ── CPU Thresholds ────────────────────────────────────────────────────────────
```bash
# master.conf
# CPU is normalised against total core count — meaningful regardless of hardware.
#
# Why normalised:
# 85% on one core of a 16-core machine = 5.3% normalised → ignore it
# 85% normalised on a 16-core machine = 13.6 cores worth → runaway process
#
# CPU uses a STRIKE SYSTEM — not immediate restart like memory.
# Brief spikes are normal (Tdarr encoding, Emby transcoding, SABnzbd unpacking).
# The strike system ignores spikes and acts on sustained high usage.
#
# Strike 1: above HARD_CPU_THRESHOLD this cycle → warn, increment strike
# Strike 2: above threshold next cycle → restart, reset counter
# Recovery: drops below threshold any cycle → reset counter to 0
#
SOFT_CPU_THRESHOLD=50 # warn at 50% normalised — informational only
HARD_CPU_THRESHOLD=85 # strike at 85% normalised
CPU_FAIL_LIMIT=2 # consecutive strikes before restart
```
---
### ── HTTP Health Checks ────────────────────────────────────────────────────────
```bash
# host1.conf
# Format: "ContainerName:http://host:port/optional-path"
# Hits the actual service endpoint on every watchdog cycle.
# "Container running" and "service responding" are not the same thing.
#
# Uses a STRIKE SYSTEM — network hiccups and brief restarts happen.
# Two consecutive non-responses before acting prevents false positives.
#
# The path can be a lightweight health endpoint or the root URL.
# Docker's own HEALTHCHECK requires the image to define it — most don't.
# These checks work regardless of what the image defines.
#
HOST1_WATCHDOG_CONTAINER_URLS=(
"Emby:http://localhost:8096" # Emby WebUI root — fast to respond
"NginxProxyManager:http://localhost:81" # NPM admin interface
)
# master.conf
CURL_TIMEOUT=5 # seconds before non-response counts as a failure
RESP_FAIL_LIMIT=2 # consecutive failures before restart
```
---
### ── Required Containers ───────────────────────────────────────────────────────
```bash
# host1.conf
# Containers that must always be running.
# Found stopped → watchdog attempts restart every cycle until running or skip-listed.
# Uses the STRIKE SYSTEM — one miss might be mid-restart.
# Persistent failure → skip list → critical notification.
#
# These are the containers whose absence breaks everything else:
#
HOST1_WATCHDOG_REQUIRED_CONTAINERS=(
"NginxProxyManager" # reverse proxy — all external traffic routes through this
"Authelia" # SSO authentication — all protected services need it
"Mariadb-Authelia" # Authelia database — must be up before Authelia starts
"Redis-Authelia" # Authelia session cache — same startup dependency
)
```
---
### ── Dependency Ordering ───────────────────────────────────────────────────────
```bash
# host1.conf
# Format: "DependentContainer:dependency1 dependency2"
# Multiple dependencies space-separated. All must be running before dependent restarts.
#
# When a container and its dependency are both down:
# → restart the dependency first
# → skip the dependent this cycle
# → next cycle: dependency healthy → dependent restarts cleanly
#
# Without this: Authelia starts, can't connect to MariaDB (still starting),
# exits immediately, strike 1. Next cycle: same, strike 2. Skip list.
# MariaDB was fine the whole time.
#
HOST1_WATCHDOG_DEPENDENCIES=(
"Authelia:Mariadb-Authelia Redis-Authelia"
"Authelia-Secondary:Mariadb-Authelia Redis-Authelia-Secondary"
"NextCloud:Postgres-NextCloud"
)
```
The same dependency configuration is used by `docker_daily_restart.sh` and
`docker_weekly_restart.sh` — configure once, applies everywhere.
---
### ── Startup Grace Period ──────────────────────────────────────────────────────
```bash
# master.conf
# Suppress restart actions for N seconds after array start.
# Checks still run and log — only restart actions are suppressed.
# Clock starts from when the watchdog process itself starts.
#
# Without this: false-positive restarts fire in the first minutes after
# every array start while containers are still initialising.
#
WATCHDOG_STARTUP_GRACE=600 # 10 minutes
```
---
### ── Tier 2 Global Scan ────────────────────────────────────────────────────────
Tier 2 scans every running container when `WATCHDOG_SCAN_ALL=true`. No per-container
configuration required — it's the catch-all for everything not explicitly in Tier 1.
```bash
# master.conf
WATCHDOG_SCAN_ALL=true # enable global scan
WATCHDOG_RESTART_UNHEALTHY=true # act on Docker HEALTHCHECK failures
WATCHDOG_NOTIFY_OOM=true # detect and notify kernel OOM kills
WATCHDOG_NOTIFY_CRASHLOOP=true # detect escalating restart counts
WATCHDOG_RESTART_DEAD=true # recover containers in dead state
WATCHDOG_RESTART_CRASHED=true # restart containers that exited non-zero
WATCHDOG_CRASH_LIMIT=5 # RestartCount above this → restart + skip list
# Containers excluded from Tier 2 entirely.
# Use for containers you intentionally stop/start manually, or containers that
# have benign non-zero exits as part of their normal operation.
WATCHDOG_SCAN_IGNORE=(
"my-one-shot-container" # runs and exits normally — not a crash
)
```
Each toggle is independent — disable any check that produces false positives in your
environment without affecting the others.
---
### ── Restart Loop Protection ──────────────────────────────────────────────────
```bash
# master.conf
# N restarts within a rolling window → skip list + critical notification.
# The watchdog stops touching the container entirely.
# Skip list lives on /boot/config/ — survives reboots intentionally.
# A container bad enough to be skip-listed is still broken after a reboot.
#
WATCHDOG_CONTAINER_RESTART_LIMIT=3 # restarts in the window before skip list
WATCHDOG_CONTAINER_RESTART_WINDOW=1 # rolling window in hours
WATCHDOG_CONTAINER_RESTART_LOG="/boot/config/container_restart_history.db"
SYS_WATCHDOG_FAILED_FILE="/boot/config/system_watchdog_failed.db"
```
**Auto-clear:** The watchdog checks the skip list every cycle and removes any container
it finds running. If the container recovers on its own, monitoring resumes automatically.
Manual clear is only needed when the container is stuck stopped.
---
### ── Notification Batching ────────────────────────────────────────────────────
```bash
# master.conf
# All events from one cycle collected → one notification at end of cycle.
#
# Why: a shared database going down can cascade 10+ containers failing
# simultaneously. Without batching: 10 individual pings. With batching:
# one summary listing all affected containers. Actionable vs overwhelming.
#
WATCHDOG_BATCH_NOTIFY=true
```
---
### ── State Files Reference ────────────────────────────────────────────────────
| File | Config Var | Location | Resets | Purpose |
|------|-----------|----------|--------|---------|
| Strike counts | `WATCHDOG_STATE_FILE` | `/tmp/` | On reboot | Per-container CPU/HTTP strike counters |
| Skip list | `SYS_WATCHDOG_FAILED_FILE` | `/boot/config/` | Never (manual / auto-clear) | Containers that exhausted restart attempts |
| Restart history | `WATCHDOG_CONTAINER_RESTART_LOG` | `/boot/config/` | Auto-purge after window | Restart loop detection data |
| Shared state | `SYS_WATCHDOG_STATE_FILE` | `/tmp/` | On reboot | RAM emergency flag + cycle heartbeat from `system_watchdog.sh` |
`/tmp/` resets on reboot — correct, strike counts before a reboot are meaningless after it.
`/boot/config/` survives reboots — correct, a skip-listed container is still broken after a reboot.
> **Watchdog configuration has moved.** `docker_watchdog.sh` is now in `Watchdogs/`.
> Memory limits, CPU thresholds, HTTP health checks, dependency ordering, skip list
> recovery, and all watchdog config vars are in `Watchdogs/Manual-Watchdogs.md`.
---
@@ -344,18 +118,6 @@ Everything gets updated at least once per week with no explicit configuration.
```bash
# Per-host — varies between HOST1 and HOST2
# Tier 1 — memory limits ("ContainerName:LimitInMB")
HOST1_WATCHDOG_CONTAINERS=()
# Tier 1 — HTTP health check endpoints ("ContainerName:http://host:port")
HOST1_WATCHDOG_CONTAINER_URLS=()
# Tier 1 — must always be running
HOST1_WATCHDOG_REQUIRED_CONTAINERS=()
# Tier 1 + 2 — dependency ordering ("Dependent:dep1 dep2")
HOST1_WATCHDOG_DEPENDENCIES=()
# Daily restart list (also drives docker_update.sh normal mode)
HOST1_DAILY_RESTART_CONTAINERS=()
@@ -367,6 +129,9 @@ HOST1_NETWORK_CONNECT_NETWORKS=()
# Containers to connect to every configured network
HOST1_NETWORK_CONNECT_CONTAINERS=()
# Watchdog config (memory limits, HTTP checks, required, dependencies):
# → see Watchdogs/Manual-Watchdogs.md
```
---
@@ -376,47 +141,16 @@ HOST1_NETWORK_CONNECT_CONTAINERS=()
```bash
# Shared — applies to both servers
# ── Watchdog timing ────────────────────────────────────────────────────────
DOCKER_WATCHDOG_INTERVAL=900 # seconds between cycles (15 minutes)
WATCHDOG_STARTUP_GRACE=600 # seconds before restarts begin after boot
CONTAINER_DELAY=15 # seconds between dependency + dependent restart
# ── Memory ─────────────────────────────────────────────────────────────────
SOFT_MEM_THRESHOLD=80 # warn at % of hard limit (no restart)
# ── CPU ────────────────────────────────────────────────────────────────────
SOFT_CPU_THRESHOLD=50 # warn threshold — normalised %
HARD_CPU_THRESHOLD=85 # strike threshold — normalised %
CPU_FAIL_LIMIT=2 # consecutive strikes before restart
# ── HTTP health check ──────────────────────────────────────────────────────
CURL_TIMEOUT=5 # seconds before curl times out
RESP_FAIL_LIMIT=2 # consecutive failures before restart
# ── Restart loop protection ────────────────────────────────────────────────
WATCHDOG_CONTAINER_RESTART_LIMIT=3 # restarts before skip list
WATCHDOG_CONTAINER_RESTART_WINDOW=1 # rolling window in hours
# ── Tier 2 global scan ─────────────────────────────────────────────────────
WATCHDOG_SCAN_ALL=true
WATCHDOG_SCAN_IGNORE=()
WATCHDOG_RESTART_UNHEALTHY=true
WATCHDOG_NOTIFY_OOM=true
WATCHDOG_NOTIFY_CRASHLOOP=true
WATCHDOG_CRASH_LIMIT=5
WATCHDOG_RESTART_DEAD=true
WATCHDOG_RESTART_CRASHED=true
# ── Notifications ──────────────────────────────────────────────────────────
WATCHDOG_BATCH_NOTIFY=true
# ── Container updates ──────────────────────────────────────────────────────
# ── Container updates ──────────────────────────────────────────────────
DAILY_CONTAINER_UPDATES=true
WEEKLY_REMAINING_UPDATES=true
# ── Retry behaviour (shared by restart scripts) ────────────────────────────
# ── Retry behaviour (shared by restart scripts) ────────────────────────
RETRY_COUNT=3 # retry attempts before marking failed
SLEEP=5 # seconds between retry attempts
# Watchdog thresholds (CPU, memory, HTTP, restart loop):
# → see Watchdogs/Manual-Watchdogs.md
```
---
@@ -425,47 +159,22 @@ SLEEP=5 # seconds between retry attempts
### ── Adding a Container to Monitoring ────────────────────────────────────────
Adding a container is purely additive — add the relevant lines to `host*.conf`.
Adding a container to watchdog monitoring is purely additive — add lines to `host*.conf`.
No script changes. `detect_hosts()` picks up the new config on the next watchdog cycle.
See `Watchdogs/Manual-Watchdogs.md` for the full procedure.
To add a container to **daily restarts** (and daily image updates):
```bash
# host1.conf — example: adding "MyApp" to full Tier 1 + daily restarts
# 1. Memory hard limit — size at ~150-200% of normal peak (check: docker stats MyApp)
HOST1_WATCHDOG_CONTAINERS=(
...existing...
"MyApp:2048" # 2GB ceiling
)
# 2. HTTP health check
HOST1_WATCHDOG_CONTAINER_URLS=(
...existing...
"MyApp:http://localhost:8080/health" # or root URL if no /health endpoint
)
# 3. Required — add only if absence breaks other services
HOST1_WATCHDOG_REQUIRED_CONTAINERS=(
...existing...
"MyApp"
)
# 4. Dependency — add if MyApp needs another container up first
HOST1_WATCHDOG_DEPENDENCIES=(
...existing...
"MyApp:MyApp-Database"
)
# 5. Daily restart — add if MyApp degrades over time
# host1.conf
HOST1_DAILY_RESTART_CONTAINERS=(
...existing...
"MyApp" # also adds it to the daily image update
)
```
Tier 2 picks up MyApp automatically — no configuration needed. It will be included in
the global health scan from the next cycle onward.
To **exclude** a container from Tier 2 global scan (e.g. one-shot that exits normally):
To **exclude** MyApp from Tier 2 (e.g. it's a one-shot container that exits normally):
```bash
# master.conf
WATCHDOG_SCAN_IGNORE=(
@@ -477,45 +186,13 @@ WATCHDOG_SCAN_IGNORE=(
### ── Skip List Recovery ────────────────────────────────────────────────────────
Used when `docker_watchdog.sh` has skip-listed a container after exhausting restart
attempts. The watchdog stops touching it and sends a critical notification. Human
intervention required.
When `docker_watchdog.sh` skip-lists a container, use `Tools/watchdog_skip_list_manager.sh`.
Full procedure in `Watchdogs/Manual-Watchdogs.md`.
```bash
# Step 1 — understand the situation
# Shows skip list contents, container states, restart history
watchdog_skip_list_manager.sh --status
# Step 2 — fix the underlying problem first
# Check logs: docker logs ContainerName --tail 100
# Check disk: df -h /mnt/user
# Check database: docker exec ContainerName sqlite3 /path/to.db ".tables"
# Fix before clearing — clearing without fixing just resets the counter
# Step 3 — clear the container from the skip list + restart history
# Clearing history is important: the counter carries over otherwise and
# the container hits the limit again almost immediately on any startup trouble
watchdog_skip_list_manager.sh --clear ContainerName
# Step 4 — start the container manually
# Confirms your fix worked before handing back to the watchdog
docker start ContainerName
# Step 5 — monitoring resumes automatically
# Next watchdog cycle: container seen running → removed from skip list
# Restart history clean. Back to normal.
```
> ⚠️ **If `docker_watchdog.sh` is currently running when you clear the skip list**, it
> may re-add the container on its very next cycle if the container is still in a bad
> state. The script detects this and warns you. Fix the root cause **before** clearing.
Other skip list actions:
```bash
watchdog_skip_list_manager.sh # status (default)
watchdog_skip_list_manager.sh --clear-all # clear everything
watchdog_skip_list_manager.sh --clear-all --force # non-interactive
watchdog_skip_list_manager.sh --clear-all --dry-run # preview what would clear
Tools/watchdog_skip_list_manager.sh --status # see skip list + container states
Tools/watchdog_skip_list_manager.sh --clear MyApp # clear after fixing root cause
Tools/watchdog_skip_list_manager.sh --clear-all # clear everything
```
---
@@ -530,25 +207,9 @@ All scripts support these standard flags:
| `--status` | Show current config, container states, and relevant runtime info, then exit. |
| `--log` | Verbose mode — adds per-container banners, action lines, pull output, and list details. |
**Output tiers:** Scripts have two output levels. Without `--log`, each script processes
silently and always concludes with a summary block: identity, duration, counts, and
a status line. Per-container detail — individual container names, pull Status lines,
skip reasons — only appears with `--log`. Warnings and errors are always visible
regardless of `--log`.
`docker_watchdog.sh` is the exception — it runs 96 cycles/day as a background daemon
and is silent on clean cycles by design. Its summary block only fires when there are
restarts or warnings. Use `--log` to see per-cycle detail on clean cycles.
### `docker_watchdog.sh --status` shows:
Strike counts for all monitored containers, current skip list contents, whether grace
period is active and how long remains, whether RAM emergency deferral is active, last
cycle timing.
### `docker_watchdog.sh --dry-run` shows:
A full watchdog cycle without restarting anything. Shows what the watchdog would do
based on current container states. Useful for verifying configuration before enabling
automatic restarts.
**Output tiers:** Without `--log`, each script processes silently and concludes with a
summary block: identity, duration, counts, and a status line. Per-container detail only
appears with `--log`. Warnings and errors are always visible regardless of `--log`.
### `docker_update.sh --remainder`
Switches to remainder mode — updates all running containers not in the managed daily/weekly
Docker_Essentials/README-Docker_Essentials.md
+22 -49
View File
@@ -154,24 +154,16 @@ its history after you've fixed the problem. No manual file editing required.
```
Docker_Essentials/ ← acts on containers (this folder)
unRAID_Essentials/ ← acts on the server itself
Watchdogs/ ← reactive monitoring + last-resort stability
Monitors/ ← observes, measures, reports
Rsync/ ← moves data between servers
```
Five distinct responsibilities, each handled by dedicated scripts:
> `docker_watchdog.sh` has moved to `Watchdogs/`. Container healing, memory limits,
> HTTP health checks, and skip list management are documented in
> `Watchdogs/README-Watchdogs.md` and `Watchdogs/Manual-Watchdogs.md`.
---
### 🔁 Reactive Healing — `docker_watchdog.sh`
Continuous two-tier monitoring that catches problems as they happen and acts on them
immediately. Runs as a background process started at array start. Every 15 minutes it
checks the full stack and fixes what it can — silently when everything is fine, visibly
when something needs attention.
Two tiers because different containers need different monitoring strategies:
- **Tier 1** — explicit per-container configuration with specific thresholds
- **Tier 2** — global scan of everything that's running with catch-all health checks
Four distinct responsibilities in this folder, each handled by dedicated scripts:
---
@@ -212,34 +204,22 @@ or in-progress downloads.
---
## ━━━ RELATIONSHIP TO SYSTEM WATCHDOG ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
## ━━━ RELATIONSHIP TO WATCHDOGS ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
Two watchdogs run simultaneously. They are designed to work together, not compete:
`docker_watchdog.sh` has moved to `Watchdogs/` and is now one of four coordinated
single-pass scripts called every minute by `Orchestrators/watchdog_orchestrator.sh`.
```
system_watchdog.sh ← watches the server: RAM, CPU, disk, kernel, daemon health
docker_watchdog.sh ← watches the containers: memory, CPU, HTTP response, crashes
Watchdogs/resource_watchdog.sh ← reduces pressure before healing attempts
Watchdogs/docker_watchdog.sh ← heals containers (reads resource_watchdog state)
Watchdogs/storage_watchdog.sh ← pool growth + runaway log detection
Watchdogs/system_watchdog.sh ← last resort — reboots when healing has failed
```
**The coordination problem:** During a RAM emergency, `system_watchdog.sh` stops
non-essential containers to recover free memory. Without coordination, `docker_watchdog.sh`
would see stopped containers on its next cycle and restart them — directly undoing the
RAM recovery. The two watchdogs would fight indefinitely. RAM would never recover.
The system would eventually hit the reboot threshold anyway, having accomplished nothing.
**The solution:** A shared state file at `SYS_WATCHDOG_STATE_FILE`. When
`system_watchdog.sh` triggers a RAM emergency shutdown it writes
`mem_shutdown_active=true`. `docker_watchdog.sh` reads this flag at the start of every
cycle and defers all container restart logic until it clears. Health URL checks for
excluded containers (DNS, auth, Emby, Dispatcharr — the ones that stayed running) still
run. Everything else stands down.
**The stale state guard:** `system_watchdog.sh` writes `watchdog_cycle=N` to the state
file on every cycle — this keeps the file's modification time current. `docker_watchdog.sh`
checks how long ago the state file was modified. If it's more than 2 hours old while
`mem_shutdown_active=true` is set, `system_watchdog.sh` has likely stopped running.
`docker_watchdog.sh` logs a warning and resumes normal operation — it won't be silenced
indefinitely by a stale flag from a process that's no longer running.
Scripts in this folder (daily restart, updates, network connect) are unaffected —
they run on their own schedules via the maintenance orchestrators and are not part
of the every-minute watchdog cycle. See `Watchdogs/README-Watchdogs.md` for the
full coordination model between all four watchdogs.
---
@@ -247,7 +227,6 @@ indefinitely by a stale flag from a process that's no longer running.
| Script | Role | When It Runs |
|--------|------|-------------|
| `docker_watchdog.sh` | Two-tier self-healing container monitor | Continuous background loop — array start |
| `docker_daily_restart.sh` | Nightly proactive restart of degradation-prone containers | 1am via `daily_sync_maintenance.sh` |
| `docker_weekly_restart.sh` | Weekly restart of less-critical services | 2:30am Sunday via `weekly_sync_maintenance.sh` |
| `docker_update.sh` | Container image updates — daily list + weekly remainder mode | Daily before restart; weekly end of window |
@@ -265,18 +244,12 @@ indefinitely by a stale flag from a process that's no longer running.
```
Array starts
│
├── docker_network_connect.sh ────── run once at start
│ ensure networks + connections exist
│ silent if correct, notify if creating
│
└── docker_watchdog.sh ──────────── continuous background loop (every 15min)
Tier 1: memory, CPU, HTTP, required containers
Tier 2: global unhealthy / OOM / crash / dead scan
reads system_watchdog state (RAM emergency deferral)
│
│ (on skip list event → operator uses)
└── Tools/watchdog_skip_list_manager.sh
inspect state, clear after fixing root cause
└── docker_network_connect.sh ────── run once at start
ensure networks + connections exist
silent if correct, notify if creating
Every minute (Orchestrators/watchdog_orchestrator.sh):
└── Watchdogs/docker_watchdog.sh ── see Watchdogs/README-Watchdogs.md
Daily maintenance window (1am):
daily_sync_maintenance.sh