From ab169a6f482ddc2852737e7ff49ac673ee0c4c65 Mon Sep 17 00:00:00 2001 From: Gmer4Lfe Date: Sat, 1 Aug 2026 23:12:18 -0400 Subject: [PATCH] Document the Deployment folder MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit It was the only folder without docs, and it holds the only versioned record of what configuration exists — the confs themselves are gitignored. --- Deployment/Manual-Deployment.md | 225 ++++++++++++++++++++++++++++++++ Deployment/README-Deployment.md | 189 +++++++++++++++++++++++++++ 2 files changed, 414 insertions(+) create mode 100644 Deployment/Manual-Deployment.md create mode 100644 Deployment/README-Deployment.md diff --git a/Deployment/Manual-Deployment.md b/Deployment/Manual-Deployment.md new file mode 100644 index 0000000..9705540 --- /dev/null +++ b/Deployment/Manual-Deployment.md @@ -0,0 +1,225 @@ +# ━━━━━ DEPLOYMENT — Manual ━━━━━ + +Procedures and flag reference for the schema layer. +For overview see README-Deployment.md. For per-script detail see the script headers. + +--- + +## ━━━ PROCEDURES ━━━ + +### Adding a New Configuration Variable + +The single most common task here, and the one with the quietest failure mode if done wrong. + +```bash +# 1. Add it to the template — this is the versioned schema +# Shared threshold/toggle → Deployment/master.conf.template +# Per-host credential/path → Deployment/host.conf.template (use the HOSTN_ prefix) + +# 2. Add it to your own live conf so you can test immediately +# Configurations/master.conf or Configurations/host1.conf + +# 3. Use it in the script, with a safe default +# [[ "${NEW_THRESHOLD:-50}" -gt ... ]] + +# 4. Commit the script change and the template change TOGETHER +git add Deployment/master.conf.template Watchdogs/System/storage_watchdog.sh +git commit -m "..." +``` + +**Step 1 is the one that gets skipped.** A script merged without its template entry works on +the machine it was written on and nowhere else — the variable is simply empty on every other +node, and the script takes whatever branch an empty value produces. Nothing errors. + +Placement rule, same as everywhere in the ecosystem: + +| Kind | Goes in | +|------|---------| +| Threshold, toggle, profile, job list | `master.conf.template` | +| Credential, path, container name, per-host identity | `host.conf.template` | + +--- + +### Previewing a Schema Change Before It Lands + +```bash +bash Deployment/conf_upgrade.sh \ + --template Deployment/master.conf.template \ + --target Configurations/master.conf \ + --dry-run +``` + +Prints the full ADDED / REMOVED / KEPT report and writes nothing. No root required — safe to +run as any user, on any target, at any time. + +Read the **REMOVED** list carefully. A key showing as REMOVED means it is in your conf but no +longer in the template — either genuinely deprecated, or someone forgot step 1 above and the +next pull will drop a setting you still rely on. + +--- + +### Applying a Schema Change by Hand + +Normally automatic via `git_pull_execute.sh`. To run it manually: + +```bash +bash Deployment/conf_upgrade.sh \ + --template Deployment/master.conf.template \ + --target Configurations/master.conf \ + --backup +``` + +`--backup` writes `master.conf.bak` first. Use it — see The Recovery Gap in the README for +why the backup matters more here than it looks. + +For a host conf, the template must have its prefix resolved first, exactly as +`git_pull_execute.sh` does it: + +```bash +TMPL=$(mktemp) +sed "s/HOSTN_/HOST1_/g; s/REMOTE_ID/HOST2/g" Deployment/host.conf.template > "$TMPL" +bash Deployment/conf_upgrade.sh --template "$TMPL" --target Configurations/host1.conf --backup +rm -f "$TMPL" +``` + +Merging the raw template without substituting `HOSTN_` would add 140 new `HOSTN_*` keys +alongside your real `HOST1_*` ones, and mark every real key as REMOVED. + +--- + +### Populating Credentials on a New Host + +```bash +bash Deployment/conf_populate.sh --dry-run # always first +bash Deployment/conf_populate.sh +``` + +Reads from the services actually running on this host and fills **empty** fields only. +Existing values are never touched without `--overwrite`. + +What it detects: arr API keys and ports from each `config.xml`, root folders from the arr +rootFolder API, path maps from docker volume mounts, SABnzbd/slskd/qBittorrent credentials +from their own config files, Emby/Jellyfin containers and ports, the boot device transport, +and the default-gateway NIC. + +Then pushes the updated conf to partners via `conf_sync.sh` so they hold the fresh keys +immediately. `--no-push` skips that. + +**If a field stays empty after a run,** check the output for an ambiguity warning: + +``` +WARN: Container prefix 'authelia' is ambiguous — matches: Authelia-Secondary Authelia +WARN: Refusing to guess. Set the container name manually in host*.conf. +``` + +That is working as intended. Set it by hand and re-run. + +--- + +### After Rotating an API Key + +```bash +bash Deployment/conf_populate.sh --overwrite --dry-run +bash Deployment/conf_populate.sh --overwrite +``` + +`--overwrite` replaces detected fields even when already set. This is the intended use for it +— rotating an arr key, rebuilding a container, or repointing at a moved service. + +Note it overwrites **every** detected field, not just the rotated one. Run the dry-run first +and read the list. + +--- + +### Rebuilding a Wiped Node + +The templates are what make this possible without copying another machine's credentials. + +```bash +# 1. Clone the repo — templates come with it, confs do not (gitignored) +# 2. Seed the confs from the templates +cp Deployment/master.conf.template Configurations/master.conf +sed "s/HOSTN_/HOST2_/g; s/REMOTE_ID/HOST1/g" Deployment/host.conf.template > Configurations/host2.conf + +# 3. Fill in what can be detected automatically +bash Deployment/conf_populate.sh + +# 4. Fill in the rest by hand — anything conf_populate cannot see: +# hostnames, DDNS containers, fallback tier lists, sync share lists, +# partner credentials, Discord webhook +``` + +From then on, `git_pull_execute.sh` keeps the conf in step with the template automatically. + +--- + +## ━━━ FLAG REFERENCE ━━━ + +### `conf_upgrade.sh` + +| Flag | Required | What it does | +|------|----------|-------------| +| `--template ` | yes | Source of structure and new keys | +| `--target ` | yes | Existing conf — source of real values, always preserved | +| `--dry-run` | | Print the change report, write nothing. No root needed. | +| `--backup` | | Write `.bak` before installing | + +Takes no other flags. It does not source `load_config.sh`, so `--log` and `--status` do not +exist here — see the script header for why that is deliberate. + +### `conf_populate.sh` + +| Flag | What it does | +|------|-------------| +| `--dry-run` | Show what would be written, truncated. Changes nothing. | +| `--overwrite` | Replace detected fields even when already set | +| `--no-push` | Skip pushing the updated conf to partners | +| `--log` | Verbose per-field output | + +--- + +## ━━━ TROUBLESHOOTING ━━━ + +### A variable is empty on one node but set on another + +The template entry is missing. Confirm: + +```bash +grep -n "MY_VARIABLE" Deployment/master.conf.template Deployment/host.conf.template +``` + +No hit means the variable was added to a conf directly and never to the template, so it has +never reached any other node. Add it to the template; the next pull propagates it. + +### `conf_upgrade` reports a key as REMOVED that is still in use + +Same cause, opposite direction — the key is in your conf and in a script, but not in the +template. Add it to the template before the next pull drops it. + +### The conf came back with different permissions + +It should not — the target's mode and owner are copied onto the staged file before the +rename. If they did change, check that the target existed before the run: `chmod --reference` +silently no-ops against a missing file. + +### A conf edit disappeared after a pull + +Expected if the key is not in the template. `conf_upgrade` keeps values for keys that exist in +both; a key present only in your conf is classified REMOVED and dropped. Add it to the +template. + +### `conf_populate` skipped a field + +Either the service is not running, its config file was unreadable, or the container name +prefix was ambiguous. The last case prints an explicit warning with the full match list — +set that value by hand. + +--- + +## ━━━ WHAT THIS FOLDER DOES NOT DO ━━━ + +- **It does not create `Configurations/`.** `conf_upgrade.sh` aborts if the target does not + exist rather than creating a partial conf. Seeding a new node is the manual step above. +- **It does not sync confs between hosts.** That is `System_Essentials/conf_sync.sh`. +- **It does not validate values.** It reconciles *structure*. A threshold set to nonsense + merges through untouched — the consuming script owns validation. diff --git a/Deployment/README-Deployment.md b/Deployment/README-Deployment.md new file mode 100644 index 0000000..f13f0d8 --- /dev/null +++ b/Deployment/README-Deployment.md @@ -0,0 +1,189 @@ +# ━━━━━ DEPLOYMENT ━━━━━ + +The schema layer. `Configurations/*.conf` holds every value the ecosystem runs on — and is +gitignored, because it holds credentials. This folder holds the **templates** those confs are +built from, and the two scripts that keep the confs in step with them. + +Two files that are versioned, and two scripts that reconcile the unversioned confs against +them: + +``` + Deployment/master.conf.template 349 vars ← the versioned schema + Deployment/host.conf.template 140 vars ← per-host schema, HOSTN_-prefixed + Deployment/conf_upgrade.sh ← template → conf, values preserved + Deployment/conf_populate.sh ← running services → conf, empty fields only +``` + +> **The templates are the only versioned record of what configuration exists.** Nothing else +> in git knows that a variable is supposed to be there. + +--- + +## ━━━ THE PROBLEM THAT BUILT THIS ━━━ + +**Config Holds Secrets, So Config Cannot Be Committed** +`master.conf` and `host*.conf` contain API keys, passwords, SSH key paths and personal +hostnames. They are gitignored, along with their `.bak` files: + +``` +.gitignore:4 Configurations/host*.conf +.gitignore:5 Configurations/master.conf +.gitignore:6 Configurations/*.bak +``` + +That is correct and non-negotiable. But it creates a problem: if the confs are not in git, +then **git has no idea a new setting was ever added.** A script that starts reading +`NEW_THRESHOLD` works on the machine where it was developed and silently fails everywhere +else, because no other node's conf has that key. + +**A New Node Would Start With Nothing** +Without a versioned schema there is no way to stand up a second server, or rebuild a wiped +one, except by hand-copying a conf from a machine that already works — which means copying +its credentials too. + +**Hand-Editing Confs Across Nodes Does Not Scale** +Two servers, ~490 variables between them. Adding a setting by hand means editing it on every +node, in the right section, with the right default, without disturbing the values already +there. Miss one and the failure surfaces days later as a script behaving differently on one +host. + +--- + +## ━━━ WHAT THIS FOLDER DOES ━━━ + +### 🔀 Schema Merge — `conf_upgrade.sh` + +Merges a template into an existing conf while preserving **every value the user has already +set**. Runs automatically from `git_pull_execute.sh` after every single pull. + +``` + Key in template only → ADDED placeholder/default, filled in once + Key in conf only → REMOVED deprecated in this version + Key in both → KEPT the conf's value always wins + Comments, blank lines → from the template — structure follows the new version +``` + +That last rule is what makes it safe to run unattended forever: the template supplies +*structure and new keys*, never settings. Your values cannot be overwritten by a pull. + +The live confs currently match their templates exactly — 349 and 140 variables — which is +what a working merge looks like. + +### 🔎 Credential Discovery — `conf_populate.sh` + +Reads settings out of the services actually running on this host and writes them into the +host conf: arr API keys from each `config.xml`, ports from real docker port bindings, paths +from real volume mounts, SABnzbd/slskd/qBittorrent credentials from their own config files. + +Only fills **empty** fields unless `--overwrite`. Manual — it is not scheduled anywhere. + +--- + +## ━━━ HOW A CHANGE REACHES EVERY NODE ━━━ + +``` +You add a variable + │ + └── edit Deployment/master.conf.template ← the versioned schema + │ + git push + │ + └── every node: git_pull_execute.sh + │ + └── conf_upgrade.sh --template ... --target ... --backup + │ + ADDED → new key appears with the template default + KEPT → every existing value untouched + REMOVED → deprecated keys dropped +``` + +**This is the rule that follows from it, and it is not optional:** + +> Any conf variable change — add, remove, or rename — must update +> `Deployment/master.conf.template` and `Deployment/host.conf.template` **in the same pass** +> as the script change that uses it. + +A script merged without its template entry works only on the machine it was written on. +Nothing errors; the variable is simply empty everywhere else, and the script takes whatever +branch an empty value leads to. + +--- + +## ━━━ THE `HOSTN_` PLACEHOLDER ━━━ + +`host.conf.template` is written with a generic prefix — 149 occurrences of `HOSTN_`: + +```bash +HOSTN_SONARR_URL="" +HOSTN_SONARR_API_KEY="" +``` + +`git_pull_execute.sh` substitutes the real identity before merging, so keys match the target: + +```bash +sed "s/HOSTN_/${MY_ID}_/g; s/REMOTE_ID/${REMOTE_ID}/g" host.conf.template > "$TMPL_RESOLVED" +``` + +One template therefore serves every host. HOST1 merges it as `HOST1_*`, HOST2 as `HOST2_*`, +and a third node would work with no template change at all. + +--- + +## ━━━ SCRIPTS IN THIS FOLDER ━━━ + +| Script | Role | When It Runs | +|--------|------|-------------| +| `conf_upgrade.sh` | Merge template into conf — structure forward, values preserved | Automatically, after every `git pull` | +| `conf_populate.sh` | Detect settings from running services into the host conf | Manually — onboarding, or after a key rotation | + +| Template | Role | +|----------|------| +| `master.conf.template` | Shared schema — thresholds, toggles, profiles, orchestrator job lists | +| `host.conf.template` | Per-host schema — credentials, paths, container names. `HOSTN_`-prefixed | + +--- + +## ━━━ SAFEGUARDS WORTH KNOWING ━━━ + +**The install is atomic.** `conf_upgrade.sh` stages the merged conf beside the target and +installs it with a rename, never a copy. A `cp` truncates the live conf and writes into it — +and every watchdog sources `load_config.sh` on every run, so anything reading during that +window would get a partial conf with empty path variables. The temp file is staged in the +target's own directory deliberately: `/tmp` is rootfs while the confs are on flash, and a +cross-device `mv` degrades to copy-then-unlink, which is the exact torn write being avoided. + +**Dry run needs no privilege, writing does.** `--dry-run` prints the full ADDED / REMOVED / +KEPT report and is useful to anyone. Installing over a conf under `/boot` requires root. + +**`conf_upgrade.sh` sources nothing — deliberately.** No `load_config.sh`, no `common.sh`. +It is the tool that repairs the conf `load_config.sh` depends on, so it has to work when that +conf is broken, partial, or missing keys. That is also why it uses plain `echo` rather than +`log()`, and why it has no `acquire_lock` — concurrency is handled by the atomic rename +instead, and since the merge is idempotent, last-writer-wins is identical to running once. + +**`conf_populate.sh` refuses to guess a container.** An ambiguous name prefix skips the field +rather than picking the first match. Writing the wrong container name is worse than writing +nothing: an empty field is visibly incomplete and gets fixed, a wrong one silently points the +whole stack at the wrong instance. This host has a live example — `authelia` prefix-matches +both `Authelia` (9091) and `Authelia-Secondary` (9092). + +--- + +## ━━━ THE RECOVERY GAP ━━━ + +Confs are gitignored, **and so are their `.bak` files**. There is no versioned history to +revert to, and the single `.bak` slot is overwritten by whoever writes next: + +| file | modified | its `.bak` | +|---|---|---| +| `master.conf` | Jul 28 18:52 | Jul 28 18:52 | +| `host1.conf` | Aug 1 21:00 | **Jul 3 17:46** | + +A bad write to `master.conf` currently falls back to a file that may predate weeks of edits. +Worth knowing before hand-editing a conf, and the reason `--backup` exists on `conf_upgrade.sh` +at all. + +Moving `Configurations/` into a private repo would make `git diff` and `git revert` the +recovery mechanism and give the history for free. That overlaps the existing GitHub-mirror +TODO, which is blocked on the same question — see `Notes_AI-Design.md`, where it also blocks +AI-assisted conf writes.