# ━━━━━ SYSTEM ESSENTIALS ━━━━━

**System-level scripts that act on the server itself — not containers,
not media, not monitoring.** Kernel limits, log hygiene, conf synchronisation
between servers, and graceful shutdowns with proper warning sequences.

> **Platform-specific scripts** (`webgui_watchdog.sh`, `php_fpm_max_children.sh`,
> `mover_stop.sh`, `user_scripts_stop.sh`) live in `Plugin/unraid/System_Essentials/`
> because they call Unraid-specific service commands and paths. This folder
> contains scripts that would run unchanged on any Linux host.

---

## ━━━ THE PROBLEM THAT BUILT THIS ━━━

**50+ Containers Starting and Filling Syslog With Veth Noise**
Array starts. 50+ containers come up simultaneously. Docker creates a virtual
network interface for each one. Each interface generates multiple syslog entries.
In the first minute after array start, syslog is buried under 200–400 lines of
`veth renamed from eth0` and `docker0: port entered forwarding state`. Real events
— a failed mount, a permission error, a service that didn't start — are invisible.

Fix: `docker_syslog_filter.sh` — creates an rsyslog drop rule before any container
starts. Applied at array start. Idempotent — silent when already correct.

**inotify Exhaustion Producing Unexplained Failures**
When inotify limits are exhausted, containers silently stop receiving filesystem
events. Arrs don't detect completed downloads. VSCode shows "unable to watch for
file changes." Code-Server with node_modules alone can consume 100K–200K watches,
and all containers share the same pool.

Fix: `inotify_tuning.sh` — raises all three inotify limits at array start, before
any container-connecting or continuous scripts start.

**Partner Conf Lost Across Reboots When Partner is Down**
Scripts like `fallback.sh` need the partner's conf vars (credentials, container
names, tier delays) to operate. The partner conf lives in a RAM cache at
`/tmp/varaverk/conf/` — wiped every reboot. At array start, `conf_sync.sh` pulls
a fresh copy from the partner. But if the partner is offline at boot time, the
pull fails and fallback has no partner vars to work with.

Fix: `conf_cache_save.sh` + `conf_cache_restore.sh` — snapshot the RAM cache to
`$PERSISTENT_CONF_CACHE` on array stop; reload on next start for any confs the
sync couldn't fetch.

**Mover/Rsync Interruption During Reboot**
Rsync transfers or mover runs are in progress when a reboot is triggered. Stopping
them uncleanly leaves partial files.

Fix: `server_reboot.sh` — orchestrates `array_stopping.sh` which stops rsync, mover,
and containers in the correct order before calling `/sbin/reboot`.

---

## ━━━ WHAT THIS FOLDER DOES ━━━

```
Kernel tuning           inotify_tuning.sh        — file watch limits
Log hygiene             docker_syslog_filter.sh  — suppress veth noise at start
                        clear_logs.sh            — size-threshold log trimming
Conf synchronisation    conf_sync.sh             — pull/push partner confs → RAM cache
                        conf_cache_save.sh       — snapshot RAM cache → persistent at stop
                        conf_cache_restore.sh    — reload from snapshot at start (offline partner)
Graceful operations     rsync_stop.sh            — smart rsync stop (orchestrator-aware)
                        server_reboot.sh         — clean reboot with pre-flight warnings
```

---

## ━━━ RELATIONSHIP TO OTHER FOLDERS ━━━

```
Orchestrators/
  array_started.sh ────────────────────────► conf_sync.sh              (pulls partner confs)
                   ────────────────────────► conf_cache_restore.sh     (fills gaps if partner down)
                   ────────────────────────► docker_syslog_filter.sh   (before containers start)
                   ────────────────────────► inotify_tuning.sh         (before docker_network_connect)
  array_stopping.sh ───────────────────────► conf_cache_save.sh        (first — while cache is fresh)
                    ───────────────────────► rsync_stop.sh --rsync-only (stop transfers)
  weekly_sync_maintenance.sh ──────────────────► clear_logs.sh

Watchdogs/System/
  conf_cache_watchdog.sh ─────────────────► maintains $PERSISTENT_CONF_CACHE while partner offline
  (runs every 15 min via watchdog_orchestrator.sh)

System_Essentials/
  server_reboot.sh ────────────────────────► array_stopping.sh (via Orchestrators/)
                   ────────────────────────► mover_stop.sh, user_scripts_stop.sh
                                             (via Plugin/unraid/System_Essentials/)

Plugin/unraid/System_Essentials/
  php_fpm_max_children.sh  — WebGUI tuning (Unraid-specific PHP paths)
  mover_stop.sh            — clean mover stop (Unraid mover daemon)
  user_scripts_stop.sh     — stop User Scripts plugin processes
  unraid_api_key_renew.sh  — Varaverk plugin API key renewal

Plugin/unraid/Watchdogs/System/
  webgui_watchdog.sh       — nginx → php-fpm → emhttp escalation
```

---

## ━━━ SCRIPTS IN THIS FOLDER ━━━

| Script | Role | When It Runs |
|--------|------|-------------|
| `inotify_tuning.sh` | Raise inotify kernel limits | At array start — before container ops |
| `docker_syslog_filter.sh` | Suppress Docker veth syslog noise | At array start — before containers |
| `conf_sync.sh` | Sync partner confs: pull → RAM cache, push own conf to partners | At array start + every 4 h (--pull-only) |
| `conf_cache_save.sh` | Snapshot partner RAM cache → `$PERSISTENT_CONF_CACHE` | At array stop — first step |
| `conf_cache_restore.sh` | Load missing partner confs from persistent backup into RAM | At array start — after conf_sync |
| `clear_logs.sh` | Size-threshold log cleanup | Weekly via weekly_sync_maintenance.sh |
| `rsync_stop.sh` | Orchestrator-aware rsync stop | Manual / called by array_stopping.sh |
| `server_reboot.sh` | Graceful reboot with pre-flight warnings | Manual |

---

## ━━━ HOW THE SCRIPTS RELATE ━━━

```
Array starts (array_started.sh, ARRAY_START_SCRIPTS):
    │
    ├─ conf_sync.sh              ← SSH/SCP: pull partner confs into /tmp/varaverk/conf/
    │                              push own conf to partner's /tmp/varaverk/conf/
    ├─ conf_cache_restore.sh     ← if partner was down: load last-known-good conf from
    │                              $PERSISTENT_CONF_CACHE into /tmp/varaverk/conf/
    ├─ docker_syslog_filter.sh   ← before any container starts (veth filter must be live)
    └─ inotify_tuning.sh         ← before docker_network_connect.sh and continuous scripts

Array stops (array_stopping.sh, ARRAY_STOP_SCRIPTS):
    │
    ├─ conf_cache_save.sh        ← FIRST: snapshot /tmp/varaverk/conf/ → $PERSISTENT_CONF_CACHE
    │                              while RAM cache is still fresh
    ├─ rsync_stop.sh --rsync-only ← kill active rsync, skip container recovery
    └─ ...other stop scripts...

Every 15 minutes (watchdog_orchestrator.sh):
    → Watchdogs/System/conf_cache_watchdog.sh
         If partner is offline and persistent backup is stale → refresh from last RAM cache

Every 4 hours (intermediate_sync_maintenance.sh):
    └─ conf_sync.sh --pull-only  ← refresh partner conf in RAM without pushing own conf

Weekly (weekly_sync_maintenance.sh):
    └─ clear_logs.sh
         System logs: clear if > LOG_MIN_SIZE_MB
         Docker logs: clear per-container if > LOG_DOCKER_MAX_MB

Manual operations:
    rsync_stop.sh    → detect orchestrator → kill rsync (or orchestrator+rsync)
    server_reboot.sh → pre-flight → array_stopping.sh → reboot
```