Varaverk/Plugin/unraid/pages/readme/scheduler-readme.md

# Scheduler

Everything that runs on a timer is decided here. The page is the front end for two files:
`schedule.json` holds when things fire, and `master.conf` holds which steps an orchestrator
actually calls. Every control below writes to one of those two — nothing on this page needs a
conf file opened by hand.

**An orchestrator is a job that calls other scripts in order.** The orchestrator has the cron
entry; its steps do not. That one fact explains most of the behaviour below.

---

## Stopping one step without disabling the whole job

The common case: a nightly job is fine, but one script inside it should stop running.

1. Find the orchestrator's card — for example **Daily Sync Maintenance**.
2. Click **Steps ▸** on the right of its row. The list of scripts it calls expands.
3. Toggle off the one you want stopped.

The orchestrator keeps running on its schedule and simply skips that step. Under the hood the
toggle comments the script out of that orchestrator's array in `master.conf`, which is the same
edit you would make by hand — the toggle is just the safe way to make it.

Turn it back on with the same toggle. The commented line is left in place, so the step keeps its
position in the sequence rather than being appended to the end.

> The button is called **Steps**, not Advanced. There is a separate **Advanced** button at the
> top right of the panel that does something quite different — see *Advanced mode* below.

## Turning an orchestrator off entirely

Toggle the orchestrator itself off. Its cron entry stops firing and **every step inside it is
suppressed** — an enabled step under a disabled orchestrator does not run, because the
orchestrator is what calls it.

If you still want one of those scripts running while its orchestrator is off, give that step its
own cron expression. With the orchestrator off, a step's cron field becomes live and it fires
standalone.

## Running something right now

**Run** fires the script immediately and resets its cron timer, so the next scheduled fire is a
full interval later rather than a few minutes away.

**Dry Run** passes `--dry-run`. Nothing is written to disk, and the full output is shown rather
than a summary — with the cleanup scripts, reading the list of what *would* be removed is the
entire point. Anything destructive offers both, and Run asks for confirmation first.

## Changing when something runs

Type into the cron field. It saves when the field loses focus, shows the schedule back to you in
plain English, and turns red if the expression is invalid.

Two shortcuts worth knowing:

- **Suggested cron** — in the Orch tree on the right, each script shows a suggested cron badge.
  Click it to apply and save that schedule instantly.
- **Cron Calculator** — accepts an expression or plain English, shows a description plus the next
  five fire times, and **Apply** pushes it into the cron field you last used.

For array events, put `array_start` or `array_stop` in the cron field instead of an expression.
A badge appears and the script fires on Unraid's array start or stop. This works for any
orchestrator or custom script.

## Finding out why something failed

**Log** opens that script's log in the right-hand panel. It auto-scrolls to the newest line,
pauses when you scroll up, and resumes when you return to the bottom. **Invert** puts newest
lines at the top. The **Search** box in the toolbar filters and highlights matches while dimming
everything else.

Two blocks on the Scheduler Info panel get you there faster:

- **Recent Activity** — the last 24 runs. Click any row to open that log. Expands itself when
  there are errors.
- **Recent Errors** — the most recent error per script over 7 days, so a failure that happened
  once at 3am is still visible days later.

The status dot in the Orch tree is green, orange or red for the last run's outcome, dim if the
script has never run. Hover for detail.

## Asking the assistant about what is on screen

The panel below the right-hand views is the same chat component the AI tab and the Monitor
dashboard use, at this panel's size. What is particular to this tab is that its subject follows
you: open a log, a conf or a script and the chip states both the profile answering and the thing
it is pointed at. Nothing is inferred silently — if what the chip says is wrong, that costs a
glance rather than a confidently wrong answer.

Every row in **Recent Activity** carries a **why?** (failed) or **recap** (clean) button. It opens
that script's log first and then asks, because the question is only worth asking with the log
attached — the profile that reads it is the one that gets the log tail. The question is posted as
visible text, so what was asked on your behalf is never a mystery.

Moving to a different subject keeps the transcript on screen but stops sending the earlier turns
to the model, and starts a new stored conversation. A troubleshooting thread carrying log excerpts
must not bleed into a question about a conf key; hiding that the earlier exchange happened would
be worse than carrying it visibly.

Reasoning is enabled only for Troubleshoot. Working out what a log means is worth waiting for; a
lookup like "what does this setting do" is not, and an inline answer that stalls reads as broken.

## Recording what actually fixed something

When a diagnosis is followed by that same script running clean, the assistant offers to record it:
*"`<script>` just ran clean. Want me to record what fixed it?"* Answer **Yes** and the next thing
you type is stored as the fix rather than asked as a question.

The trigger is the run record, not the conversation. It fires once, at the moment the trouble is
demonstrably over, rather than after every answer — an offer that appears constantly is one you
learn to skip past.

What gets stored is the symptom you asked about and the fix you describe. The model's diagnosis is
deliberately not stored: it is a reading of evidence, and writing a hypothesis into institutional
memory as settled fact is how a wrong answer outlives the incident it came from. What is
remembered is what actually worked, and it is shown the next time the same thing is diagnosed.

## Stopping a script that is stuck

**Stop** sends SIGTERM, waits 3 seconds, then SIGKILL, and clears any lock files the script left
behind.

Locks are never cleared automatically anywhere in Varaverk, and that is deliberate — a lock that
looks stale may belong to a job that is still working. Stop is the explicit way to say you have
decided otherwise.

## Moving a script between orchestrators

Use **Arrange** to drag scripts between orchestrators. Dropping one onto the right-hand panel
removes it from its orchestrator. Nothing is written until you click **Save Arrangement**, which
commits the new order to `master.conf`.

Order matters — an orchestrator runs its steps in the sequence shown.

## Adding your own script

Custom scripts live in `$CUSTOM_SCRIPTS_DIR`, outside the git repo. Any `*.sh` placed there is
detected and listed automatically; it does not have to be registered anywhere. Being outside the
repo means a `git pull` never touches your scripts and they are never committed by accident.

**+ Folder** creates a collapsible subfolder in Custom Scripts that you can drag scripts into.

While editing, **Cancel** discards your changes and returns to the Scheduler Info panel. The file
on disk is never touched until you press Save.

---

## Reference — the controls on a job row

| Control | What it does |
|---|---|
| **Toggle** | Enable/disable. Saves immediately to `schedule.json` |
| **Cron field** | Schedule. Auto-saves on focus-out; plain-English hint; red border means invalid |
| **⚙ Cog** | Settings panel — the script's own header, matching README/Manual sections, and its config fields |
| **Script name** | Jumps to that script's entry in the Orch tree on the right |
| **Run** | Execute now; resets the cron timer |
| **Dry Run** | Execute with `--dry-run`; writes nothing |
| **Log** | Open this script's log in the right panel |
| **Verbose** | Appends `--log` for per-item detail in the output |
| **Stop** | SIGTERM, 3s, then SIGKILL. Clears any lock the script left behind |
| **Steps ▸** | Expand the scripts this orchestrator calls |

## Reference — controls elsewhere on the page

| Control | Where | What it does |
|---|---|---|
| **Suggested cron** | Orch tree | Click the cron badge to apply and save that schedule instantly |
| **Status dot** | Orch tree | Green / orange / red for the last run; dim if never run. Hover for detail |
| **Arrange** | Orch tree | Drag scripts between orchestrators. **Save Arrangement** commits to `master.conf` |
| **Array events** | Cron field | Type `array_start` or `array_stop` instead of an expression |
| **Log search** | Toolbar | Filters the open log, highlighting matches and dimming the rest |
| **Invert** | Toolbar | Newest log lines at the top |
| **Auto Scroll** | Toolbar | Follows the newest line; pauses if you scroll up |
| **Cancel** | Toolbar, editing | Discards changes and returns here. The file is untouched until Save |
| **+ Folder** | Custom Scripts | A collapsible subfolder you can drag scripts into |
| **why? / recap** | Recent Activity | Opens that run's log and asks the assistant about it in one click |
| **Profile chip** | Assistant | What is answering, and what it is pointed at. Click to change the profile |
| **⤢** | Assistant | Expands the conversation into the panel, and collapses it again |
| **Medium / Large** | Assistant banner | How much of the panel expanding is worth — 20% or 40% |
| **Shortcuts** | Assistant banner | The keyboard list. Taken out of the transcript, so the panel keeps its height |
| **Saved** | Assistant | Saved conversations, shared with the AI tab and the Monitor dashboard |
| **New** | Assistant | Starts a fresh conversation; the old one stays in the store |

## Reference — orchestrator and step behaviour

| State | Behaviour |
|---|---|
| Orchestrator **ON** | Sole trigger. Cron fires the orchestrator, which calls its steps in sequence |
| Orchestrator **OFF** | Never runs automatically. Steps are suppressed; each can take its own cron |
| Step, orchestrator ON | Toggle comments/uncomments the script in `master.conf` |
| Step, orchestrator ON, no conf array | The orchestrator hardcodes the call — the toggle is display-only |
| Step, orchestrator OFF | Enter a cron to run it standalone |
| Rsync badge | Toggle writes `TIER_RSYNC_ENABLED` straight to `master.conf` |
| Rsync, orchestrator OFF | Needs both a location and a standalone cron before it will fire independently |

## Reference — the Scheduler Info panel

The right-hand panel is the default view and everything returns to it. **← Scheduler Info** comes
back here from any log, editor, conf form or script view.

- **Notification board** — how many orchestrators are scheduled, how many scripts carry a cron of
  their own, and what is running now. Pinned, so it stays visible while you read the rest.
- **Next Runs** — what fires next, and when.
- **Recent Activity** / **Recent Errors** — described above.
- **Cron Calculator** — described above.

The board counts **orchestrators**, not scripts. A schedule here is normally one cron on an
orchestrator that then calls its steps in order, so the steps are deliberately cronless — counting
them made a fully-scheduled system read as `10 / 82 scheduled`, which is alarming and wrong. Steps
that *do* carry their own cron are counted separately as standalone, because that is the exception
and the number of exceptions is the figure worth knowing. It shows only when there are any.

## Reference — Advanced mode

The **Advanced** button at the top right turns blue when active. It is a display and editing
mode, unrelated to the **Steps ▸** expander on each orchestrator:

- The ⚙ cog switches to an enriched view — full script source, documentation and config together.
- Raw conf editing unlocks, including direct buttons for each conf file.
- **Git Pull** appears on the notification board, which pulls the latest scripts and runs a conf
  upgrade.

Raw conf editing writes the file directly. The toggles elsewhere on this page are surgical — they
preserve the comment blocks that document every threshold — so prefer them where one exists.

## Reference — what this page writes

| File | Written by |
|---|---|
| `schedule.json` | Every toggle and cron field |
| `master.conf` | Step toggles, Save Arrangement, rsync tier flags, raw conf editing |
| `CUSTOM_SCRIPTS_DIR` | The custom script editor |

Cron changes take effect on the next cron rebuild. Nothing here edits the running crontab
directly — `varaverk.cron` is regenerated from `schedule.json`.