Document the Deployment folder
It was the only folder without docs, and it holds the only versioned record of what configuration exists — the confs themselves are gitignored.
This commit is contained in:
Gmer4Lfe
@@ -0,0 +1,189 @@
|
||||
# ━━━━━ DEPLOYMENT ━━━━━
|
||||
|
||||
The schema layer. `Configurations/*.conf` holds every value the ecosystem runs on — and is
|
||||
gitignored, because it holds credentials. This folder holds the **templates** those confs are
|
||||
built from, and the two scripts that keep the confs in step with them.
|
||||
|
||||
Two files that are versioned, and two scripts that reconcile the unversioned confs against
|
||||
them:
|
||||
|
||||
```
|
||||
Deployment/master.conf.template 349 vars ← the versioned schema
|
||||
Deployment/host.conf.template 140 vars ← per-host schema, HOSTN_-prefixed
|
||||
Deployment/conf_upgrade.sh ← template → conf, values preserved
|
||||
Deployment/conf_populate.sh ← running services → conf, empty fields only
|
||||
```
|
||||
|
||||
> **The templates are the only versioned record of what configuration exists.** Nothing else
|
||||
> in git knows that a variable is supposed to be there.
|
||||
|
||||
---
|
||||
|
||||
## ━━━ THE PROBLEM THAT BUILT THIS ━━━
|
||||
|
||||
**Config Holds Secrets, So Config Cannot Be Committed**
|
||||
`master.conf` and `host*.conf` contain API keys, passwords, SSH key paths and personal
|
||||
hostnames. They are gitignored, along with their `.bak` files:
|
||||
|
||||
```
|
||||
.gitignore:4 Configurations/host*.conf
|
||||
.gitignore:5 Configurations/master.conf
|
||||
.gitignore:6 Configurations/*.bak
|
||||
```
|
||||
|
||||
That is correct and non-negotiable. But it creates a problem: if the confs are not in git,
|
||||
then **git has no idea a new setting was ever added.** A script that starts reading
|
||||
`NEW_THRESHOLD` works on the machine where it was developed and silently fails everywhere
|
||||
else, because no other node's conf has that key.
|
||||
|
||||
**A New Node Would Start With Nothing**
|
||||
Without a versioned schema there is no way to stand up a second server, or rebuild a wiped
|
||||
one, except by hand-copying a conf from a machine that already works — which means copying
|
||||
its credentials too.
|
||||
|
||||
**Hand-Editing Confs Across Nodes Does Not Scale**
|
||||
Two servers, ~490 variables between them. Adding a setting by hand means editing it on every
|
||||
node, in the right section, with the right default, without disturbing the values already
|
||||
there. Miss one and the failure surfaces days later as a script behaving differently on one
|
||||
host.
|
||||
|
||||
---
|
||||
|
||||
## ━━━ WHAT THIS FOLDER DOES ━━━
|
||||
|
||||
### 🔀 Schema Merge — `conf_upgrade.sh`
|
||||
|
||||
Merges a template into an existing conf while preserving **every value the user has already
|
||||
set**. Runs automatically from `git_pull_execute.sh` after every single pull.
|
||||
|
||||
```
|
||||
Key in template only → ADDED placeholder/default, filled in once
|
||||
Key in conf only → REMOVED deprecated in this version
|
||||
Key in both → KEPT the conf's value always wins
|
||||
Comments, blank lines → from the template — structure follows the new version
|
||||
```
|
||||
|
||||
That last rule is what makes it safe to run unattended forever: the template supplies
|
||||
*structure and new keys*, never settings. Your values cannot be overwritten by a pull.
|
||||
|
||||
The live confs currently match their templates exactly — 349 and 140 variables — which is
|
||||
what a working merge looks like.
|
||||
|
||||
### 🔎 Credential Discovery — `conf_populate.sh`
|
||||
|
||||
Reads settings out of the services actually running on this host and writes them into the
|
||||
host conf: arr API keys from each `config.xml`, ports from real docker port bindings, paths
|
||||
from real volume mounts, SABnzbd/slskd/qBittorrent credentials from their own config files.
|
||||
|
||||
Only fills **empty** fields unless `--overwrite`. Manual — it is not scheduled anywhere.
|
||||
|
||||
---
|
||||
|
||||
## ━━━ HOW A CHANGE REACHES EVERY NODE ━━━
|
||||
|
||||
```
|
||||
You add a variable
|
||||
│
|
||||
└── edit Deployment/master.conf.template ← the versioned schema
|
||||
│
|
||||
git push
|
||||
│
|
||||
└── every node: git_pull_execute.sh
|
||||
│
|
||||
└── conf_upgrade.sh --template ... --target ... --backup
|
||||
│
|
||||
ADDED → new key appears with the template default
|
||||
KEPT → every existing value untouched
|
||||
REMOVED → deprecated keys dropped
|
||||
```
|
||||
|
||||
**This is the rule that follows from it, and it is not optional:**
|
||||
|
||||
> Any conf variable change — add, remove, or rename — must update
|
||||
> `Deployment/master.conf.template` and `Deployment/host.conf.template` **in the same pass**
|
||||
> as the script change that uses it.
|
||||
|
||||
A script merged without its template entry works only on the machine it was written on.
|
||||
Nothing errors; the variable is simply empty everywhere else, and the script takes whatever
|
||||
branch an empty value leads to.
|
||||
|
||||
---
|
||||
|
||||
## ━━━ THE `HOSTN_` PLACEHOLDER ━━━
|
||||
|
||||
`host.conf.template` is written with a generic prefix — 149 occurrences of `HOSTN_`:
|
||||
|
||||
```bash
|
||||
HOSTN_SONARR_URL=""
|
||||
HOSTN_SONARR_API_KEY=""
|
||||
```
|
||||
|
||||
`git_pull_execute.sh` substitutes the real identity before merging, so keys match the target:
|
||||
|
||||
```bash
|
||||
sed "s/HOSTN_/${MY_ID}_/g; s/REMOTE_ID/${REMOTE_ID}/g" host.conf.template > "$TMPL_RESOLVED"
|
||||
```
|
||||
|
||||
One template therefore serves every host. HOST1 merges it as `HOST1_*`, HOST2 as `HOST2_*`,
|
||||
and a third node would work with no template change at all.
|
||||
|
||||
---
|
||||
|
||||
## ━━━ SCRIPTS IN THIS FOLDER ━━━
|
||||
|
||||
| Script | Role | When It Runs |
|
||||
|--------|------|-------------|
|
||||
| `conf_upgrade.sh` | Merge template into conf — structure forward, values preserved | Automatically, after every `git pull` |
|
||||
| `conf_populate.sh` | Detect settings from running services into the host conf | Manually — onboarding, or after a key rotation |
|
||||
|
||||
| Template | Role |
|
||||
|----------|------|
|
||||
| `master.conf.template` | Shared schema — thresholds, toggles, profiles, orchestrator job lists |
|
||||
| `host.conf.template` | Per-host schema — credentials, paths, container names. `HOSTN_`-prefixed |
|
||||
|
||||
---
|
||||
|
||||
## ━━━ SAFEGUARDS WORTH KNOWING ━━━
|
||||
|
||||
**The install is atomic.** `conf_upgrade.sh` stages the merged conf beside the target and
|
||||
installs it with a rename, never a copy. A `cp` truncates the live conf and writes into it —
|
||||
and every watchdog sources `load_config.sh` on every run, so anything reading during that
|
||||
window would get a partial conf with empty path variables. The temp file is staged in the
|
||||
target's own directory deliberately: `/tmp` is rootfs while the confs are on flash, and a
|
||||
cross-device `mv` degrades to copy-then-unlink, which is the exact torn write being avoided.
|
||||
|
||||
**Dry run needs no privilege, writing does.** `--dry-run` prints the full ADDED / REMOVED /
|
||||
KEPT report and is useful to anyone. Installing over a conf under `/boot` requires root.
|
||||
|
||||
**`conf_upgrade.sh` sources nothing — deliberately.** No `load_config.sh`, no `common.sh`.
|
||||
It is the tool that repairs the conf `load_config.sh` depends on, so it has to work when that
|
||||
conf is broken, partial, or missing keys. That is also why it uses plain `echo` rather than
|
||||
`log()`, and why it has no `acquire_lock` — concurrency is handled by the atomic rename
|
||||
instead, and since the merge is idempotent, last-writer-wins is identical to running once.
|
||||
|
||||
**`conf_populate.sh` refuses to guess a container.** An ambiguous name prefix skips the field
|
||||
rather than picking the first match. Writing the wrong container name is worse than writing
|
||||
nothing: an empty field is visibly incomplete and gets fixed, a wrong one silently points the
|
||||
whole stack at the wrong instance. This host has a live example — `authelia` prefix-matches
|
||||
both `Authelia` (9091) and `Authelia-Secondary` (9092).
|
||||
|
||||
---
|
||||
|
||||
## ━━━ THE RECOVERY GAP ━━━
|
||||
|
||||
Confs are gitignored, **and so are their `.bak` files**. There is no versioned history to
|
||||
revert to, and the single `.bak` slot is overwritten by whoever writes next:
|
||||
|
||||
| file | modified | its `.bak` |
|
||||
|---|---|---|
|
||||
| `master.conf` | Jul 28 18:52 | Jul 28 18:52 |
|
||||
| `host1.conf` | Aug 1 21:00 | **Jul 3 17:46** |
|
||||
|
||||
A bad write to `master.conf` currently falls back to a file that may predate weeks of edits.
|
||||
Worth knowing before hand-editing a conf, and the reason `--backup` exists on `conf_upgrade.sh`
|
||||
at all.
|
||||
|
||||
Moving `Configurations/` into a private repo would make `git diff` and `git revert` the
|
||||
recovery mechanism and give the history for free. That overlaps the existing GitHub-mirror
|
||||
TODO, which is blocked on the same question — see `Notes_AI-Design.md`, where it also blocks
|
||||
AI-assisted conf writes.
|
||||
Reference in New Issue
Block a user