# ━━━━━ DEPLOYMENT ━━━━━ The schema layer. `Configurations/*.conf` holds every value the ecosystem runs on — and is gitignored, because it holds credentials. This folder holds the **templates** those confs are built from, and the two scripts that keep the confs in step with them. Two files that are versioned, and two scripts that reconcile the unversioned confs against them: ``` Deployment/master.conf.template 349 vars ← the versioned schema Deployment/host.conf.template 140 vars ← per-host schema, HOSTN_-prefixed Deployment/conf_upgrade.sh ← template → conf, values preserved Deployment/conf_populate.sh ← running services → conf, empty fields only ``` > **The templates are the only versioned record of what configuration exists.** Nothing else > in git knows that a variable is supposed to be there. --- ## ━━━ THE PROBLEM THAT BUILT THIS ━━━ **Config Holds Secrets, So Config Cannot Be Committed** `master.conf` and `host*.conf` contain API keys, passwords, SSH key paths and personal hostnames. They are gitignored, along with their `.bak` files: ``` .gitignore:4 Configurations/host*.conf .gitignore:5 Configurations/master.conf .gitignore:6 Configurations/*.bak ``` That is correct and non-negotiable. But it creates a problem: if the confs are not in git, then **git has no idea a new setting was ever added.** A script that starts reading `NEW_THRESHOLD` works on the machine where it was developed and silently fails everywhere else, because no other node's conf has that key. **A New Node Would Start With Nothing** Without a versioned schema there is no way to stand up a second server, or rebuild a wiped one, except by hand-copying a conf from a machine that already works — which means copying its credentials too. **Hand-Editing Confs Across Nodes Does Not Scale** Two servers, ~490 variables between them. Adding a setting by hand means editing it on every node, in the right section, with the right default, without disturbing the values already there. Miss one and the failure surfaces days later as a script behaving differently on one host. --- ## ━━━ WHAT THIS FOLDER DOES ━━━ ### 🔀 Schema Merge — `conf_upgrade.sh` Merges a template into an existing conf while preserving **every value the user has already set**. Runs automatically from `git_pull_execute.sh` after every single pull. ``` Key in template only → ADDED placeholder/default, filled in once Key in conf only → REMOVED deprecated in this version Key in both → KEPT the conf's value always wins Comments, blank lines → from the template — structure follows the new version ``` That last rule is what makes it safe to run unattended forever: the template supplies *structure and new keys*, never settings. Your values cannot be overwritten by a pull. The live confs currently match their templates exactly — 349 and 140 variables — which is what a working merge looks like. ### 🔎 Credential Discovery — `conf_populate.sh` Reads settings out of the services actually running on this host and writes them into the host conf: arr API keys from each `config.xml`, ports from real docker port bindings, paths from real volume mounts, SABnzbd/slskd/qBittorrent credentials from their own config files. Only fills **empty** fields unless `--overwrite`. Manual — it is not scheduled anywhere. --- ## ━━━ HOW A CHANGE REACHES EVERY NODE ━━━ ``` You add a variable │ └── edit Deployment/master.conf.template ← the versioned schema │ git push │ └── every node: git_pull_execute.sh │ └── conf_upgrade.sh --template ... --target ... --backup │ ADDED → new key appears with the template default KEPT → every existing value untouched REMOVED → deprecated keys dropped ``` **This is the rule that follows from it, and it is not optional:** > Any conf variable change — add, remove, or rename — must update > `Deployment/master.conf.template` and `Deployment/host.conf.template` **in the same pass** > as the script change that uses it. A script merged without its template entry works only on the machine it was written on. Nothing errors; the variable is simply empty everywhere else, and the script takes whatever branch an empty value leads to. --- ## ━━━ THE `HOSTN_` PLACEHOLDER ━━━ `host.conf.template` is written with a generic prefix — 149 occurrences of `HOSTN_`: ```bash HOSTN_SONARR_URL="" HOSTN_SONARR_API_KEY="" ``` `git_pull_execute.sh` substitutes the real identity before merging, so keys match the target: ```bash sed "s/HOSTN_/${MY_ID}_/g; s/REMOTE_ID/${REMOTE_ID}/g" host.conf.template > "$TMPL_RESOLVED" ``` One template therefore serves every host. HOST1 merges it as `HOST1_*`, HOST2 as `HOST2_*`, and a third node would work with no template change at all. --- ## ━━━ SCRIPTS IN THIS FOLDER ━━━ | Script | Role | When It Runs | |--------|------|-------------| | `conf_upgrade.sh` | Merge template into conf — structure forward, values preserved | Automatically, after every `git pull` | | `conf_populate.sh` | Detect settings from running services into the host conf | Manually — onboarding, or after a key rotation | | Template | Role | |----------|------| | `master.conf.template` | Shared schema — thresholds, toggles, profiles, orchestrator job lists | | `host.conf.template` | Per-host schema — credentials, paths, container names. `HOSTN_`-prefixed | --- ## ━━━ SAFEGUARDS WORTH KNOWING ━━━ **The install is atomic.** `conf_upgrade.sh` stages the merged conf beside the target and installs it with a rename, never a copy. A `cp` truncates the live conf and writes into it — and every watchdog sources `load_config.sh` on every run, so anything reading during that window would get a partial conf with empty path variables. The temp file is staged in the target's own directory deliberately: `/tmp` is rootfs while the confs are on flash, and a cross-device `mv` degrades to copy-then-unlink, which is the exact torn write being avoided. **Dry run needs no privilege, writing does.** `--dry-run` prints the full ADDED / REMOVED / KEPT report and is useful to anyone. Installing over a conf under `/boot` requires root. **`conf_upgrade.sh` sources nothing — deliberately.** No `load_config.sh`, no `common.sh`. It is the tool that repairs the conf `load_config.sh` depends on, so it has to work when that conf is broken, partial, or missing keys. That is also why it uses plain `echo` rather than `log()`, and why it has no `acquire_lock` — concurrency is handled by the atomic rename instead, and since the merge is idempotent, last-writer-wins is identical to running once. **`conf_populate.sh` refuses to guess a container.** An ambiguous name prefix skips the field rather than picking the first match. Writing the wrong container name is worse than writing nothing: an empty field is visibly incomplete and gets fixed, a wrong one silently points the whole stack at the wrong instance. This host has a live example — `authelia` prefix-matches both `Authelia` (9091) and `Authelia-Secondary` (9092). --- ## ━━━ THE RECOVERY GAP ━━━ Confs are gitignored, **and so are their `.bak` files**. There is no versioned history to revert to, and the single `.bak` slot is overwritten by whoever writes next: | file | modified | its `.bak` | |---|---|---| | `master.conf` | Jul 28 18:52 | Jul 28 18:52 | | `host1.conf` | Aug 1 21:00 | **Jul 3 17:46** | A bad write to `master.conf` currently falls back to a file that may predate weeks of edits. Worth knowing before hand-editing a conf, and the reason `--backup` exists on `conf_upgrade.sh` at all. Moving `Configurations/` into a private repo would make `git diff` and `git revert` the recovery mechanism and give the history for free. That overlaps the existing GitHub-mirror TODO, which is blocked on the same question — see `Notes_AI-Design.md`, where it also blocks AI-assisted conf writes.