Document the Deployment folder
It was the only folder without docs, and it holds the only versioned record of what configuration exists — the confs themselves are gitignored.
This commit is contained in:
Gmer4Lfe
@@ -0,0 +1,225 @@
|
|||||||
|
# ━━━━━ DEPLOYMENT — Manual ━━━━━
|
||||||
|
|
||||||
|
Procedures and flag reference for the schema layer.
|
||||||
|
For overview see README-Deployment.md. For per-script detail see the script headers.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## ━━━ PROCEDURES ━━━
|
||||||
|
|
||||||
|
### Adding a New Configuration Variable
|
||||||
|
|
||||||
|
The single most common task here, and the one with the quietest failure mode if done wrong.
|
||||||
|
|
||||||
|
```bash
|
||||||
|
# 1. Add it to the template — this is the versioned schema
|
||||||
|
# Shared threshold/toggle → Deployment/master.conf.template
|
||||||
|
# Per-host credential/path → Deployment/host.conf.template (use the HOSTN_ prefix)
|
||||||
|
|
||||||
|
# 2. Add it to your own live conf so you can test immediately
|
||||||
|
# Configurations/master.conf or Configurations/host1.conf
|
||||||
|
|
||||||
|
# 3. Use it in the script, with a safe default
|
||||||
|
# [[ "${NEW_THRESHOLD:-50}" -gt ... ]]
|
||||||
|
|
||||||
|
# 4. Commit the script change and the template change TOGETHER
|
||||||
|
git add Deployment/master.conf.template Watchdogs/System/storage_watchdog.sh
|
||||||
|
git commit -m "..."
|
||||||
|
```
|
||||||
|
|
||||||
|
**Step 1 is the one that gets skipped.** A script merged without its template entry works on
|
||||||
|
the machine it was written on and nowhere else — the variable is simply empty on every other
|
||||||
|
node, and the script takes whatever branch an empty value produces. Nothing errors.
|
||||||
|
|
||||||
|
Placement rule, same as everywhere in the ecosystem:
|
||||||
|
|
||||||
|
| Kind | Goes in |
|
||||||
|
|------|---------|
|
||||||
|
| Threshold, toggle, profile, job list | `master.conf.template` |
|
||||||
|
| Credential, path, container name, per-host identity | `host.conf.template` |
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
### Previewing a Schema Change Before It Lands
|
||||||
|
|
||||||
|
```bash
|
||||||
|
bash Deployment/conf_upgrade.sh \
|
||||||
|
--template Deployment/master.conf.template \
|
||||||
|
--target Configurations/master.conf \
|
||||||
|
--dry-run
|
||||||
|
```
|
||||||
|
|
||||||
|
Prints the full ADDED / REMOVED / KEPT report and writes nothing. No root required — safe to
|
||||||
|
run as any user, on any target, at any time.
|
||||||
|
|
||||||
|
Read the **REMOVED** list carefully. A key showing as REMOVED means it is in your conf but no
|
||||||
|
longer in the template — either genuinely deprecated, or someone forgot step 1 above and the
|
||||||
|
next pull will drop a setting you still rely on.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
### Applying a Schema Change by Hand
|
||||||
|
|
||||||
|
Normally automatic via `git_pull_execute.sh`. To run it manually:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
bash Deployment/conf_upgrade.sh \
|
||||||
|
--template Deployment/master.conf.template \
|
||||||
|
--target Configurations/master.conf \
|
||||||
|
--backup
|
||||||
|
```
|
||||||
|
|
||||||
|
`--backup` writes `master.conf.bak` first. Use it — see The Recovery Gap in the README for
|
||||||
|
why the backup matters more here than it looks.
|
||||||
|
|
||||||
|
For a host conf, the template must have its prefix resolved first, exactly as
|
||||||
|
`git_pull_execute.sh` does it:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
TMPL=$(mktemp)
|
||||||
|
sed "s/HOSTN_/HOST1_/g; s/REMOTE_ID/HOST2/g" Deployment/host.conf.template > "$TMPL"
|
||||||
|
bash Deployment/conf_upgrade.sh --template "$TMPL" --target Configurations/host1.conf --backup
|
||||||
|
rm -f "$TMPL"
|
||||||
|
```
|
||||||
|
|
||||||
|
Merging the raw template without substituting `HOSTN_` would add 140 new `HOSTN_*` keys
|
||||||
|
alongside your real `HOST1_*` ones, and mark every real key as REMOVED.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
### Populating Credentials on a New Host
|
||||||
|
|
||||||
|
```bash
|
||||||
|
bash Deployment/conf_populate.sh --dry-run # always first
|
||||||
|
bash Deployment/conf_populate.sh
|
||||||
|
```
|
||||||
|
|
||||||
|
Reads from the services actually running on this host and fills **empty** fields only.
|
||||||
|
Existing values are never touched without `--overwrite`.
|
||||||
|
|
||||||
|
What it detects: arr API keys and ports from each `config.xml`, root folders from the arr
|
||||||
|
rootFolder API, path maps from docker volume mounts, SABnzbd/slskd/qBittorrent credentials
|
||||||
|
from their own config files, Emby/Jellyfin containers and ports, the boot device transport,
|
||||||
|
and the default-gateway NIC.
|
||||||
|
|
||||||
|
Then pushes the updated conf to partners via `conf_sync.sh` so they hold the fresh keys
|
||||||
|
immediately. `--no-push` skips that.
|
||||||
|
|
||||||
|
**If a field stays empty after a run,** check the output for an ambiguity warning:
|
||||||
|
|
||||||
|
```
|
||||||
|
WARN: Container prefix 'authelia' is ambiguous — matches: Authelia-Secondary Authelia
|
||||||
|
WARN: Refusing to guess. Set the container name manually in host*.conf.
|
||||||
|
```
|
||||||
|
|
||||||
|
That is working as intended. Set it by hand and re-run.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
### After Rotating an API Key
|
||||||
|
|
||||||
|
```bash
|
||||||
|
bash Deployment/conf_populate.sh --overwrite --dry-run
|
||||||
|
bash Deployment/conf_populate.sh --overwrite
|
||||||
|
```
|
||||||
|
|
||||||
|
`--overwrite` replaces detected fields even when already set. This is the intended use for it
|
||||||
|
— rotating an arr key, rebuilding a container, or repointing at a moved service.
|
||||||
|
|
||||||
|
Note it overwrites **every** detected field, not just the rotated one. Run the dry-run first
|
||||||
|
and read the list.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
### Rebuilding a Wiped Node
|
||||||
|
|
||||||
|
The templates are what make this possible without copying another machine's credentials.
|
||||||
|
|
||||||
|
```bash
|
||||||
|
# 1. Clone the repo — templates come with it, confs do not (gitignored)
|
||||||
|
# 2. Seed the confs from the templates
|
||||||
|
cp Deployment/master.conf.template Configurations/master.conf
|
||||||
|
sed "s/HOSTN_/HOST2_/g; s/REMOTE_ID/HOST1/g" Deployment/host.conf.template > Configurations/host2.conf
|
||||||
|
|
||||||
|
# 3. Fill in what can be detected automatically
|
||||||
|
bash Deployment/conf_populate.sh
|
||||||
|
|
||||||
|
# 4. Fill in the rest by hand — anything conf_populate cannot see:
|
||||||
|
# hostnames, DDNS containers, fallback tier lists, sync share lists,
|
||||||
|
# partner credentials, Discord webhook
|
||||||
|
```
|
||||||
|
|
||||||
|
From then on, `git_pull_execute.sh` keeps the conf in step with the template automatically.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## ━━━ FLAG REFERENCE ━━━
|
||||||
|
|
||||||
|
### `conf_upgrade.sh`
|
||||||
|
|
||||||
|
| Flag | Required | What it does |
|
||||||
|
|------|----------|-------------|
|
||||||
|
| `--template <file>` | yes | Source of structure and new keys |
|
||||||
|
| `--target <file>` | yes | Existing conf — source of real values, always preserved |
|
||||||
|
| `--dry-run` | | Print the change report, write nothing. No root needed. |
|
||||||
|
| `--backup` | | Write `<target>.bak` before installing |
|
||||||
|
|
||||||
|
Takes no other flags. It does not source `load_config.sh`, so `--log` and `--status` do not
|
||||||
|
exist here — see the script header for why that is deliberate.
|
||||||
|
|
||||||
|
### `conf_populate.sh`
|
||||||
|
|
||||||
|
| Flag | What it does |
|
||||||
|
|------|-------------|
|
||||||
|
| `--dry-run` | Show what would be written, truncated. Changes nothing. |
|
||||||
|
| `--overwrite` | Replace detected fields even when already set |
|
||||||
|
| `--no-push` | Skip pushing the updated conf to partners |
|
||||||
|
| `--log` | Verbose per-field output |
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## ━━━ TROUBLESHOOTING ━━━
|
||||||
|
|
||||||
|
### A variable is empty on one node but set on another
|
||||||
|
|
||||||
|
The template entry is missing. Confirm:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
grep -n "MY_VARIABLE" Deployment/master.conf.template Deployment/host.conf.template
|
||||||
|
```
|
||||||
|
|
||||||
|
No hit means the variable was added to a conf directly and never to the template, so it has
|
||||||
|
never reached any other node. Add it to the template; the next pull propagates it.
|
||||||
|
|
||||||
|
### `conf_upgrade` reports a key as REMOVED that is still in use
|
||||||
|
|
||||||
|
Same cause, opposite direction — the key is in your conf and in a script, but not in the
|
||||||
|
template. Add it to the template before the next pull drops it.
|
||||||
|
|
||||||
|
### The conf came back with different permissions
|
||||||
|
|
||||||
|
It should not — the target's mode and owner are copied onto the staged file before the
|
||||||
|
rename. If they did change, check that the target existed before the run: `chmod --reference`
|
||||||
|
silently no-ops against a missing file.
|
||||||
|
|
||||||
|
### A conf edit disappeared after a pull
|
||||||
|
|
||||||
|
Expected if the key is not in the template. `conf_upgrade` keeps values for keys that exist in
|
||||||
|
both; a key present only in your conf is classified REMOVED and dropped. Add it to the
|
||||||
|
template.
|
||||||
|
|
||||||
|
### `conf_populate` skipped a field
|
||||||
|
|
||||||
|
Either the service is not running, its config file was unreadable, or the container name
|
||||||
|
prefix was ambiguous. The last case prints an explicit warning with the full match list —
|
||||||
|
set that value by hand.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## ━━━ WHAT THIS FOLDER DOES NOT DO ━━━
|
||||||
|
|
||||||
|
- **It does not create `Configurations/`.** `conf_upgrade.sh` aborts if the target does not
|
||||||
|
exist rather than creating a partial conf. Seeding a new node is the manual step above.
|
||||||
|
- **It does not sync confs between hosts.** That is `System_Essentials/conf_sync.sh`.
|
||||||
|
- **It does not validate values.** It reconciles *structure*. A threshold set to nonsense
|
||||||
|
merges through untouched — the consuming script owns validation.
|
||||||
@@ -0,0 +1,189 @@
|
|||||||
|
# ━━━━━ DEPLOYMENT ━━━━━
|
||||||
|
|
||||||
|
The schema layer. `Configurations/*.conf` holds every value the ecosystem runs on — and is
|
||||||
|
gitignored, because it holds credentials. This folder holds the **templates** those confs are
|
||||||
|
built from, and the two scripts that keep the confs in step with them.
|
||||||
|
|
||||||
|
Two files that are versioned, and two scripts that reconcile the unversioned confs against
|
||||||
|
them:
|
||||||
|
|
||||||
|
```
|
||||||
|
Deployment/master.conf.template 349 vars ← the versioned schema
|
||||||
|
Deployment/host.conf.template 140 vars ← per-host schema, HOSTN_-prefixed
|
||||||
|
Deployment/conf_upgrade.sh ← template → conf, values preserved
|
||||||
|
Deployment/conf_populate.sh ← running services → conf, empty fields only
|
||||||
|
```
|
||||||
|
|
||||||
|
> **The templates are the only versioned record of what configuration exists.** Nothing else
|
||||||
|
> in git knows that a variable is supposed to be there.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## ━━━ THE PROBLEM THAT BUILT THIS ━━━
|
||||||
|
|
||||||
|
**Config Holds Secrets, So Config Cannot Be Committed**
|
||||||
|
`master.conf` and `host*.conf` contain API keys, passwords, SSH key paths and personal
|
||||||
|
hostnames. They are gitignored, along with their `.bak` files:
|
||||||
|
|
||||||
|
```
|
||||||
|
.gitignore:4 Configurations/host*.conf
|
||||||
|
.gitignore:5 Configurations/master.conf
|
||||||
|
.gitignore:6 Configurations/*.bak
|
||||||
|
```
|
||||||
|
|
||||||
|
That is correct and non-negotiable. But it creates a problem: if the confs are not in git,
|
||||||
|
then **git has no idea a new setting was ever added.** A script that starts reading
|
||||||
|
`NEW_THRESHOLD` works on the machine where it was developed and silently fails everywhere
|
||||||
|
else, because no other node's conf has that key.
|
||||||
|
|
||||||
|
**A New Node Would Start With Nothing**
|
||||||
|
Without a versioned schema there is no way to stand up a second server, or rebuild a wiped
|
||||||
|
one, except by hand-copying a conf from a machine that already works — which means copying
|
||||||
|
its credentials too.
|
||||||
|
|
||||||
|
**Hand-Editing Confs Across Nodes Does Not Scale**
|
||||||
|
Two servers, ~490 variables between them. Adding a setting by hand means editing it on every
|
||||||
|
node, in the right section, with the right default, without disturbing the values already
|
||||||
|
there. Miss one and the failure surfaces days later as a script behaving differently on one
|
||||||
|
host.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## ━━━ WHAT THIS FOLDER DOES ━━━
|
||||||
|
|
||||||
|
### 🔀 Schema Merge — `conf_upgrade.sh`
|
||||||
|
|
||||||
|
Merges a template into an existing conf while preserving **every value the user has already
|
||||||
|
set**. Runs automatically from `git_pull_execute.sh` after every single pull.
|
||||||
|
|
||||||
|
```
|
||||||
|
Key in template only → ADDED placeholder/default, filled in once
|
||||||
|
Key in conf only → REMOVED deprecated in this version
|
||||||
|
Key in both → KEPT the conf's value always wins
|
||||||
|
Comments, blank lines → from the template — structure follows the new version
|
||||||
|
```
|
||||||
|
|
||||||
|
That last rule is what makes it safe to run unattended forever: the template supplies
|
||||||
|
*structure and new keys*, never settings. Your values cannot be overwritten by a pull.
|
||||||
|
|
||||||
|
The live confs currently match their templates exactly — 349 and 140 variables — which is
|
||||||
|
what a working merge looks like.
|
||||||
|
|
||||||
|
### 🔎 Credential Discovery — `conf_populate.sh`
|
||||||
|
|
||||||
|
Reads settings out of the services actually running on this host and writes them into the
|
||||||
|
host conf: arr API keys from each `config.xml`, ports from real docker port bindings, paths
|
||||||
|
from real volume mounts, SABnzbd/slskd/qBittorrent credentials from their own config files.
|
||||||
|
|
||||||
|
Only fills **empty** fields unless `--overwrite`. Manual — it is not scheduled anywhere.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## ━━━ HOW A CHANGE REACHES EVERY NODE ━━━
|
||||||
|
|
||||||
|
```
|
||||||
|
You add a variable
|
||||||
|
│
|
||||||
|
└── edit Deployment/master.conf.template ← the versioned schema
|
||||||
|
│
|
||||||
|
git push
|
||||||
|
│
|
||||||
|
└── every node: git_pull_execute.sh
|
||||||
|
│
|
||||||
|
└── conf_upgrade.sh --template ... --target ... --backup
|
||||||
|
│
|
||||||
|
ADDED → new key appears with the template default
|
||||||
|
KEPT → every existing value untouched
|
||||||
|
REMOVED → deprecated keys dropped
|
||||||
|
```
|
||||||
|
|
||||||
|
**This is the rule that follows from it, and it is not optional:**
|
||||||
|
|
||||||
|
> Any conf variable change — add, remove, or rename — must update
|
||||||
|
> `Deployment/master.conf.template` and `Deployment/host.conf.template` **in the same pass**
|
||||||
|
> as the script change that uses it.
|
||||||
|
|
||||||
|
A script merged without its template entry works only on the machine it was written on.
|
||||||
|
Nothing errors; the variable is simply empty everywhere else, and the script takes whatever
|
||||||
|
branch an empty value leads to.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## ━━━ THE `HOSTN_` PLACEHOLDER ━━━
|
||||||
|
|
||||||
|
`host.conf.template` is written with a generic prefix — 149 occurrences of `HOSTN_`:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
HOSTN_SONARR_URL=""
|
||||||
|
HOSTN_SONARR_API_KEY=""
|
||||||
|
```
|
||||||
|
|
||||||
|
`git_pull_execute.sh` substitutes the real identity before merging, so keys match the target:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
sed "s/HOSTN_/${MY_ID}_/g; s/REMOTE_ID/${REMOTE_ID}/g" host.conf.template > "$TMPL_RESOLVED"
|
||||||
|
```
|
||||||
|
|
||||||
|
One template therefore serves every host. HOST1 merges it as `HOST1_*`, HOST2 as `HOST2_*`,
|
||||||
|
and a third node would work with no template change at all.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## ━━━ SCRIPTS IN THIS FOLDER ━━━
|
||||||
|
|
||||||
|
| Script | Role | When It Runs |
|
||||||
|
|--------|------|-------------|
|
||||||
|
| `conf_upgrade.sh` | Merge template into conf — structure forward, values preserved | Automatically, after every `git pull` |
|
||||||
|
| `conf_populate.sh` | Detect settings from running services into the host conf | Manually — onboarding, or after a key rotation |
|
||||||
|
|
||||||
|
| Template | Role |
|
||||||
|
|----------|------|
|
||||||
|
| `master.conf.template` | Shared schema — thresholds, toggles, profiles, orchestrator job lists |
|
||||||
|
| `host.conf.template` | Per-host schema — credentials, paths, container names. `HOSTN_`-prefixed |
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## ━━━ SAFEGUARDS WORTH KNOWING ━━━
|
||||||
|
|
||||||
|
**The install is atomic.** `conf_upgrade.sh` stages the merged conf beside the target and
|
||||||
|
installs it with a rename, never a copy. A `cp` truncates the live conf and writes into it —
|
||||||
|
and every watchdog sources `load_config.sh` on every run, so anything reading during that
|
||||||
|
window would get a partial conf with empty path variables. The temp file is staged in the
|
||||||
|
target's own directory deliberately: `/tmp` is rootfs while the confs are on flash, and a
|
||||||
|
cross-device `mv` degrades to copy-then-unlink, which is the exact torn write being avoided.
|
||||||
|
|
||||||
|
**Dry run needs no privilege, writing does.** `--dry-run` prints the full ADDED / REMOVED /
|
||||||
|
KEPT report and is useful to anyone. Installing over a conf under `/boot` requires root.
|
||||||
|
|
||||||
|
**`conf_upgrade.sh` sources nothing — deliberately.** No `load_config.sh`, no `common.sh`.
|
||||||
|
It is the tool that repairs the conf `load_config.sh` depends on, so it has to work when that
|
||||||
|
conf is broken, partial, or missing keys. That is also why it uses plain `echo` rather than
|
||||||
|
`log()`, and why it has no `acquire_lock` — concurrency is handled by the atomic rename
|
||||||
|
instead, and since the merge is idempotent, last-writer-wins is identical to running once.
|
||||||
|
|
||||||
|
**`conf_populate.sh` refuses to guess a container.** An ambiguous name prefix skips the field
|
||||||
|
rather than picking the first match. Writing the wrong container name is worse than writing
|
||||||
|
nothing: an empty field is visibly incomplete and gets fixed, a wrong one silently points the
|
||||||
|
whole stack at the wrong instance. This host has a live example — `authelia` prefix-matches
|
||||||
|
both `Authelia` (9091) and `Authelia-Secondary` (9092).
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## ━━━ THE RECOVERY GAP ━━━
|
||||||
|
|
||||||
|
Confs are gitignored, **and so are their `.bak` files**. There is no versioned history to
|
||||||
|
revert to, and the single `.bak` slot is overwritten by whoever writes next:
|
||||||
|
|
||||||
|
| file | modified | its `.bak` |
|
||||||
|
|---|---|---|
|
||||||
|
| `master.conf` | Jul 28 18:52 | Jul 28 18:52 |
|
||||||
|
| `host1.conf` | Aug 1 21:00 | **Jul 3 17:46** |
|
||||||
|
|
||||||
|
A bad write to `master.conf` currently falls back to a file that may predate weeks of edits.
|
||||||
|
Worth knowing before hand-editing a conf, and the reason `--backup` exists on `conf_upgrade.sh`
|
||||||
|
at all.
|
||||||
|
|
||||||
|
Moving `Configurations/` into a private repo would make `git diff` and `git revert` the
|
||||||
|
recovery mechanism and give the history for free. That overlaps the existing GitHub-mirror
|
||||||
|
TODO, which is blocked on the same question — see `Notes_AI-Design.md`, where it also blocks
|
||||||
|
AI-assisted conf writes.
|
||||||
Reference in New Issue
Block a user