during onboard we need to run the docker network script to ensure the custom netork is there before containers get installed


Run docker network script during onboarding


Let me check for the plugin install wizard or initial setup flow.


Bash

find /boot/config/plugins/varaverk -name "*wizard*" -o -name "*install*" -o -name "*setup*" | grep -v ".git" | grep -v "claude-data" | sort

Find wizard, install, and setup scripts


1	<?php
2	// First-run setup wizard — uniform flow for all hosts.
3	// Step 1: auto-detect environment + server identity form.
4	// Step 2: auto-populate + guide + checklist.
5	// master.conf pull (for partner servers) lives in the checklist, not here.
6
7	$detectedHostname = vv_get_hostname();
8	?>
9	<link rel="stylesheet" href="/plugins/varaverk/css/varaverk.css">
10
78
79

80
81
⬡ Varaverk — First Run

82
Set up this server before the plugin can start.

83
84
85

86
87

Detecting environment…

88
89

90	      Storage mode
91

92

93	          Appdata
USB boot · requires array
94

95

96	          Internal Boot
NVMe/SSD · no array dep
97

98

99

100

101
102

103	      This server's hostname
104	      <input type="text" id="vv-hostname" value="<?= htmlspecialchars($detectedHostname) ?>" autocomplete="off" spellcheck="false">
105
Must match Unraid Settings → Identification exactly (case-sensitive)

106

107
108

109	    Server role
110

111

112	        Primary
HOST1 · first server
113

114

115	        Partner
HOST2+ · joining primary
116

117

118
119

120

121	        Partner's hostname (optional — can fill in later)
122	        <input type="text" id="vv-partner-hostname" value="" placeholder="unRAID-PartnerServer" autocomplete="off" spellcheck="false">
123

124

125
126

127

128	        Primary server's hostname required
129	        <input type="text" id="vv-primary-hostname" value="" placeholder="unRAID-PrimaryServer" autocomplete="off" spellcheck="false">
130

131

132	        Your slot
133	        <select id="vv-partner-slot">
134	          <option value="host2">HOST2</option>
135	          <option value="host3">HOST3</option>
136	          <option value="host4">HOST4</option>
137	        </select>
138

139

140	        SSH key and master.conf pull are handled automatically after save.
141

142

143
144	    <button class="vv-btn" id="vv-main-btn" onclick="vvDoSave()">Save and continue →</button>
145

146

147
148
149

150

151
Step 2 of 2

152
153
⟳ Running auto-populate…

154
155

156	      Quick start
157

158
Create your Unraid API key below — needed for live monitor stats

159
Open Scheduler → Edit host.conf — only three things need manual entry:

160
161	              EMBY_API_KEY — Emby Dashboard → API Keys → + New Key

162	              DISCORD_WEBHOOK — for notifications (optional)

163	              DAILY_SYNC_SHARES — media paths to rsync nightly

164	              Everything else was auto-populated or has working defaults
165

166
If partnering: the checklist below will guide you through pulling HOST1's config and running onboard

167

168

169
170

171	      <button id="vv-key-btn" onclick="vvCreateKey(this)" class="vv-btn" style="flex:1;background:#1a3a1a;border-color:#2e6b2e;color:#6fcf97;">
172	        Create API Key
173	      </button>
174	      Skip →
175

176

177
178

179
Setup checklist

180

Loading…

181
182

183	      Go to Scheduler →
184

185

186
187

188
189
434

<?php
// First-run setup wizard — uniform flow for all hosts.
// Step 1: auto-detect environment + server identity form.
// Step 2: auto-populate + guide + checklist.
// master.conf pull (for partner servers) lives in the checklist, not here.

$detectedHostname = vv_get_hostname();
?>
<link rel="stylesheet" href="/plugins/varaverk/css/varaverk.css">

⬡ Varaverk — First Run

Set up this server before the plugin can start.

Detecting environment…

      Storage mode

          Appdata
USB boot · requires array

          Internal Boot
NVMe/SSD · no array dep

      This server's hostname
      <input type="text" id="vv-hostname" value="<?= htmlspecialchars($detectedHostname) ?>" autocomplete="off" spellcheck="false">

Must match Unraid Settings → Identification exactly (case-sensitive)

    Server role

        Primary
HOST1 · first server

        Partner
HOST2+ · joining primary

        Partner's hostname (optional — can fill in later)
        <input type="text" id="vv-partner-hostname" value="" placeholder="unRAID-PartnerServer" autocomplete="off" spellcheck="false">

        Primary server's hostname required
        <input type="text" id="vv-primary-hostname" value="" placeholder="unRAID-PrimaryServer" autocomplete="off" spellcheck="false">

        Your slot
        <select id="vv-partner-slot">
          <option value="host2">HOST2</option>
          <option value="host3">HOST3</option>
          <option value="host4">HOST4</option>
        </select>

        SSH key and master.conf pull are handled automatically after save.

    <button class="vv-btn" id="vv-main-btn" onclick="vvDoSave()">Save and continue →</button>

Step 2 of 2

⟳ Running auto-populate…

      Quick start

Create your Unraid API key below — needed for live monitor stats

Open Scheduler → Edit host.conf — only three things need manual entry:

              EMBY_API_KEY — Emby Dashboard → API Keys → + New Key

              DISCORD_WEBHOOK — for notifications (optional)

              DAILY_SYNC_SHARES — media paths to rsync nightly

              Everything else was auto-populated or has working defaults

If partnering: the checklist below will guide you through pulling HOST1's config and running onboard

      <button id="vv-key-btn" onclick="vvCreateKey(this)" class="vv-btn" style="flex:1;background:#1a3a1a;border-color:#2e6b2e;color:#6fcf97;">
        Create API Key
      </button>
      Skip →

Setup checklist

Loading…

      Go to Scheduler →


Now let me see what the `save` action in setup.php API does.


1	<?php
2	header('Content-Type: application/json');
3	require_once dirname(__DIR__) . '/include/config.php';
4
5	$action = ($_SERVER['REQUEST_METHOD'] === 'GET')
6	    ? trim($_GET['action'] ?? '')
7	    : trim($_POST['action'] ?? 'save');
8
9	// ── GET: detect environment ────────────────────────────────────────────────────────────────────
10	if ($action === 'detect') {
11	    $bootPart  = trim(shell_exec('findmnt -n -o SOURCE /boot 2>/dev/null') ?: '');
12	    $bootDisk  = $bootPart
13	        ? trim(shell_exec('lsblk -no pkname ' . escapeshellarg($bootPart) . ' 2>/dev/null') ?: '')
14	        : '';
15	    $transport = $bootDisk
16	        ? strtolower(trim(shell_exec('lsblk -dno TRAN /dev/' . escapeshellarg($bootDisk) . ' 2>/dev/null') ?: ''))
17	        : 'unknown';
18
19	    $isUsb = ($transport === 'usb');
20
21	    preg_match('/version="([^"]+)"/', @file_get_contents('/etc/unraid-version') ?: '', $vm);
22
23	    echo json_encode([
24	        'ok'           => true,
25	        'hostname'     => vv_get_hostname(),
26	        'unraid_ver'   => $vm[1] ?? 'unknown',
27	        'transport'    => $transport,
28	        'boot_device'  => $bootDisk ? '/dev/' . $bootDisk : 'unknown',
29	        'mode'         => $isUsb ? 'flash' : 'internal',
30	        'scripts_dir'  => SCRIPTS_DIR,
31	    ]);
32	    exit;
33	}
34
35	// ── GET/POST: generate local SSH keypair ──────────────────────────────────────────────────────
36	if ($action === 'ssh_generate') {
37	    $script = SCRIPTS_DIR . '/Partnership/ssh_setup.sh';
38	    if (!file_exists($script)) {
39	        echo json_encode(['ok' => false, 'error' => 'ssh_setup.sh not found']);
40	        exit;
41	    }
42	    exec('bash ' . escapeshellarg($script) . ' --local-only 2>&1', $out, $rc);
43	    // Derive pubkey path from hostname
44	    $hostname  = vv_get_hostname();
45	    $shortName = strtolower(preg_replace('/^unraid-/i', '', $hostname));
46	    $pubPath   = '/root/.ssh/' . $shortName . '_rsync_automation.pub';
47	    $pubKey    = trim(@file_get_contents($pubPath) ?: '');
48	    echo json_encode([
49	        'ok'     => $rc === 0 && !empty($pubKey),
50	        'pubkey' => $pubKey,
51	        'error'  => ($rc !== 0) ? implode(' ', array_slice(array_filter(array_map('trim', $out)), -3)) : null,
52	    ]);
53	    exit;
54	}
55
56	// ── POST: run conf_populate.sh ─────────────────────────────────────────────────────────────────
57	if ($action === 'populate') {
58	    $script = SCRIPTS_DIR . '/Plugin/unraid/Tools/conf_populate.sh';
59	    if (!file_exists($script)) {
60	        echo json_encode(['ok' => false, 'error' => 'conf_populate.sh not found']);
61	        exit;
62	    }
63	    exec('bash ' . escapeshellarg($script) . ' --no-push 2>&1', $out, $rc);
64	    $lines = array_values(array_filter(array_map('trim', $out)));
65	    echo json_encode(['ok' => $rc === 0, 'lines' => array_slice($lines, 0, 20)]);
66	    exit;
67	}
68
69	if ($_SERVER['REQUEST_METHOD'] !== 'POST') {
70	    echo json_encode(['ok' => false, 'error' => 'Method not allowed']);
71	    exit;
72	}
73
74	$sshScript = SCRIPTS_DIR . '/Partnership/ssh_setup.sh';
75
76	// ── Pull master.conf from HOST1 via SSH (wizard or checklist) ────────────────────────────────
77	if ($action === 'pull') {
78	    $mySlot        = trim($_POST['my_slot']        ?? '') ?: strtolower(vv_detect_host());
79	    $myHostname    = trim($_POST['my_hostname']     ?? '') ?: vv_get_hostname();
80	    $host1Hostname = trim($_POST['host1_hostname']  ?? '');
81	    if (!$host1Hostname) {
82	        $masterRaw = vv_read_conf_raw('master.conf');
83	        preg_match('/^\s*HOST1\s*=\s*"([^"]*)"/m', $masterRaw, $_mh);
84	        $host1Hostname = trim($_mh[1] ?? '');
85	    }
86	    if (!$host1Hostname) {
87	        echo json_encode(['ok' => false, 'error' => 'HOST1 hostname not set — fill in master.conf first']);
88	        exit;
89	    }
90	    if (!preg_match('/^host\d+$/', $mySlot)) {
91	        echo json_encode(['ok' => false, 'error' => 'Invalid slot']);
92	        exit;
93	    }
94
95	    $hostId    = strtoupper($mySlot);
96	    $hostIdLow = strtolower($mySlot);
97
98	    // Derive SSH key path from this server's hostname
99	    $sshOwner = strtolower(preg_replace('/^unraid-/i', '', $myHostname ?: vv_get_hostname()));
100	    $sshKey   = '/root/.ssh/' . $sshOwner . '_rsync_automation';
101
102	    if (!file_exists($sshKey)) {
103	        echo json_encode(['ok' => false, 'error' =>
104	            "SSH key not found at $sshKey — run Partnership/ssh_setup.sh first"]);
105	        exit;
106	    }
107
108	    // Resolve HOST1 Tailscale IP
109	    $ip = trim(shell_exec('tailscale ip -4 ' . escapeshellarg($host1Hostname) . ' 2>/dev/null') ?: '');
110	    if (!$ip) {
111	        echo json_encode(['ok' => false, 'error' =>
112	            "Cannot resolve Tailscale IP for $host1Hostname — is Tailscale running on both servers?"]);
113	        exit;
114	    }
115
116	    // Get HOST1's SCRIPTS_DIR from their varaverk.cfg
117	    $sshBase   = 'ssh -i ' . escapeshellarg($sshKey)
118	               . ' -o ConnectTimeout=10 -o StrictHostKeyChecking=no root@' . $ip;
119	    $remoteCfg = trim(shell_exec($sshBase . ' "grep SCRIPTS_DIR /boot/config/plugins/varaverk/varaverk.cfg 2>/dev/null"') ?: '');
120	    preg_match('/SCRIPTS_DIR\s*=\s*["\']?([^"\']+)["\']?/', $remoteCfg, $sm);
121	    $remoteConf = rtrim($sm[1] ?? '/boot/config/plugins/varaverk', '/') . '/Configurations';
122
123	    // SCP master.conf from HOST1
124	    $localMaster = CONF_DIR . '/master.conf';
125	    $src  = escapeshellarg('root@' . $ip . ':' . $remoteConf . '/master.conf');
126	    $cmd  = 'scp -i ' . escapeshellarg($sshKey)
127	          . ' -o ConnectTimeout=10 -o StrictHostKeyChecking=no'
128	          . ' ' . $src . ' ' . escapeshellarg($localMaster) . ' 2>&1';
129	    exec($cmd, $out, $rc);
130	    if ($rc !== 0) {
131	        echo json_encode(['ok' => false, 'error' =>
132	            'SCP failed: ' . implode('; ', $out) .
133	            ' — ensure your SSH key is authorised on HOST1 (run Partnership/ssh_setup.sh)']);
134	        exit;
135	    }
136
137	    // Create host conf from template if it doesn't exist
138	    $confFile = $hostIdLow . '.conf';
139	    if (!file_exists(CONF_DIR . '/' . $confFile)) {
140	        $template = @file_get_contents(CONF_DIR . '/host.conf.template') ?: '';
141	        if ($template) {
142	            $bootPart2  = trim(shell_exec('findmnt -n -o SOURCE /boot 2>/dev/null') ?: '');
143	            $bootDisk2  = $bootPart2 ? trim(shell_exec('lsblk -no pkname ' . escapeshellarg($bootPart2) . ' 2>/dev/null') ?: '') : '';
144	            $transport2 = $bootDisk2 ? strtolower(trim(shell_exec('lsblk -dno TRAN /dev/' . escapeshellarg($bootDisk2) . ' 2>/dev/null') ?: '')) : '';
145	            $storageInternal2 = ($transport2 !== 'usb') ? 'true' : 'false';
146	            $conf = str_replace('HOSTN', $hostId,    $template);
147	            $conf = str_replace('hostn', $hostIdLow, $conf);
148	            $conf = preg_replace('/^(\s*' . $hostId . '_SSH_KEY\s*=\s*)""/m',
149	                '${1}"' . $sshKey . '"', $conf);
150	            $conf = preg_replace('/^(\s*' . $hostId . '_STORAGE_MODE_INTERNAL\s*=\s*)\S+/m',
151	                '${1}' . $storageInternal2, $conf);
152	            vv_write_conf_raw($confFile, $conf);
153	        }
154	    }
155
156	    if (file_exists($sshScript)) {
157	        exec('bash ' . escapeshellarg($sshScript) . ' --local-only 2>/dev/null');
158	    }
159	    $apiKeyResult = vv_auto_create_api_key($hostId, $confFile);
160
161	    $state = vv_setup_state_read();
162	    $state['master_conf_pulled'] = 'true';
163	    vv_setup_state_write($state);
164
165	    echo json_encode(['ok' => true, 'host_id' => $hostId, 'conf_file' => $confFile,
166	                      'api_key'  => $apiKeyResult,
167	                      'redirect' => '?tab=scheduler&vv_setup=' . $confFile]);
168	    exit;
169	}
170
171	// ── Default action: save (HOST1 first-run wizard) ────────────────────────────────────────────
172	$host1      = trim($_POST['host1']      ?? '');
173	$host2      = trim($_POST['host2']      ?? '');
174	$mySlot     = trim($_POST['my_slot']    ?? 'host1');
175	$myHostname = trim($_POST['my_hostname'] ?? '');
176
177	if (empty($host1)) {
178	    echo json_encode(['ok' => false, 'error' => 'HOST1 hostname is required']);
179	    exit;
180	}
181	if (!preg_match('/^host\d+$/', $mySlot)) {
182	    echo json_encode(['ok' => false, 'error' => 'Invalid slot']);
183	    exit;
184	}
185
186	// Write HOST1 / HOST2 into master.conf
187	$master = vv_read_conf_raw('master.conf');
188	if ($master === '') {
189	    echo json_encode(['ok' => false, 'error' => 'master.conf not found — check SCRIPTS_DIR in varaverk.cfg']);
190	    exit;
191	}
192
193	$master = preg_replace('/^(\s*HOST1\s*=\s*).*$/m', '${1}"' . addslashes($host1) . '"', $master);
194	$master = preg_replace('/^(\s*HOST2\s*=\s*).*$/m', '${1}"' . addslashes($host2) . '"', $master);
195
196	$slotNum = (int) preg_replace('/\D/', '', $mySlot);
197	if ($slotNum > 2 && !empty($myHostname)) {
198	    $hostKey = 'HOST' . $slotNum;
199	    if (!preg_match('/^\s*' . $hostKey . '\s*=/m', $master)) {
200	        $master = preg_replace('/^(\s*HOST2\s*=.*$)/m',
201	            '$1' . "\n    {$hostKey}=\"" . addslashes($myHostname) . '"', $master);
202	    } else {
203	        $master = preg_replace('/^(\s*' . $hostKey . '\s*=\s*).*$/m',
204	            '${1}"' . addslashes($myHostname) . '"', $master);
205	    }
206	}
207
208	if (!vv_write_conf_raw('master.conf', $master)) {
209	    echo json_encode(['ok' => false, 'error' => 'Failed to write master.conf']);
210	    exit;
211	}
212
213	// Create host*.conf from template
214	$hostId    = strtoupper($mySlot);
215	$hostIdLow = strtolower($mySlot);
216	$confFile  = $hostIdLow . '.conf';
217
218	// Storage mode: use wizard selection, fall back to auto-detect from boot transport
219	$smParam = trim($_POST['storage_mode'] ?? '');
220	if ($smParam === 'flash') {
221	    $storageInternal = 'false';
222	} elseif ($smParam === 'internal') {
223	    $storageInternal = 'true';
224	} else {
225	    $bootPart  = trim(shell_exec('findmnt -n -o SOURCE /boot 2>/dev/null') ?: '');
226	    $bootDisk  = $bootPart ? trim(shell_exec('lsblk -no pkname ' . escapeshellarg($bootPart) . ' 2>/dev/null') ?: '') : '';
227	    $transport = $bootDisk ? strtolower(trim(shell_exec('lsblk -dno TRAN /dev/' . escapeshellarg($bootDisk) . ' 2>/dev/null') ?: '')) : '';
228	    $storageInternal = ($transport !== 'usb') ? 'true' : 'false';
229	}
230
231	if (!file_exists(CONF_DIR . '/' . $confFile)) {
232	    $template = @file_get_contents(CONF_DIR . '/host.conf.template') ?: '';
233	    if ($template) {
234	        $sshOwner   = strtolower(preg_replace('/^unraid-/i', '', $myHostname));
235	        $sshKeyPath = '/root/.ssh/' . $sshOwner . '_rsync_automation';
236
237	        $conf = str_replace('HOSTN', $hostId,    $template);
238	        $conf = str_replace('hostn', $hostIdLow, $conf);
239	        $conf = preg_replace('/^(\s*' . $hostId . '_SSH_KEY\s*=\s*)""/m',
240	            '${1}"' . $sshKeyPath . '"', $conf);
241	        $conf = preg_replace('/^(\s*' . $hostId . '_STORAGE_MODE_INTERNAL\s*=\s*)\S+/m',
242	            '${1}' . $storageInternal, $conf);
243	        if (!vv_write_conf_raw($confFile, $conf)) {
244	            echo json_encode(['ok' => false, 'error' => "Failed to write $confFile"]);
245	            exit;
246	        }
247	    }
248	}
249
250	// Write setup state file — lets partner servers know HOST1 is configured
251	vv_setup_state_write(['host1_hostname' => $host1]);
252
253	// Auto-generate SSH keypair (local only — remote copy happens during onboarding)
254	if (file_exists($sshScript)) {
255	    exec('bash ' . escapeshellarg($sshScript) . ' --local-only 2>/dev/null');
256	}
257
258	// Auto-create Unraid API key and write into the fresh conf
259	$apiKeyResult = vv_auto_create_api_key($hostId, $confFile);
260
261	$targetDir      = ($storageInternal === 'true') ? '/boot/config/plugins/varaverk' : '/mnt/user/appdata/Varaverk';
262	$needsMigration = (defined('SCRIPTS_DIR') && SCRIPTS_DIR !== $targetDir);
263
264	echo json_encode([
265	    'ok'             => true,
266	    'host_id'        => $hostId,
267	    'api_key'        => $apiKeyResult,
268	    'needs_migration'=> $needsMigration,
269	    'migrate_to'     => $needsMigration ? ($storageInternal === 'true' ? 'internal' : 'flash') : null,
270	    'redirect'       => '?tab=scheduler&vv_setup=master.conf',
271	]);
272

<?php
header('Content-Type: application/json');
require_once dirname(__DIR__) . '/include/config.php';

$action = ($_SERVER['REQUEST_METHOD'] === 'GET')
    ? trim($_GET['action'] ?? '')
    : trim($_POST['action'] ?? 'save');

// ── GET: detect environment ────────────────────────────────────────────────────────────────────
if ($action === 'detect') {
    $bootPart  = trim(shell_exec('findmnt -n -o SOURCE /boot 2>/dev/null') ?: '');
    $bootDisk  = $bootPart
        ? trim(shell_exec('lsblk -no pkname ' . escapeshellarg($bootPart) . ' 2>/dev/null') ?: '')
        : '';
    $transport = $bootDisk
        ? strtolower(trim(shell_exec('lsblk -dno TRAN /dev/' . escapeshellarg($bootDisk) . ' 2>/dev/null') ?: ''))
        : 'unknown';

    $isUsb = ($transport === 'usb');

    preg_match('/version="([^"]+)"/', @file_get_contents('/etc/unraid-version') ?: '', $vm);

    echo json_encode([
        'ok'           => true,
        'hostname'     => vv_get_hostname(),
        'unraid_ver'   => $vm[1] ?? 'unknown',
        'transport'    => $transport,
        'boot_device'  => $bootDisk ? '/dev/' . $bootDisk : 'unknown',
        'mode'         => $isUsb ? 'flash' : 'internal',
        'scripts_dir'  => SCRIPTS_DIR,
    ]);
    exit;
}

// ── GET/POST: generate local SSH keypair ──────────────────────────────────────────────────────
if ($action === 'ssh_generate') {
    $script = SCRIPTS_DIR . '/Partnership/ssh_setup.sh';
    if (!file_exists($script)) {
        echo json_encode(['ok' => false, 'error' => 'ssh_setup.sh not found']);
        exit;
    }
    exec('bash ' . escapeshellarg($script) . ' --local-only 2>&1', $out, $rc);
    // Derive pubkey path from hostname
    $hostname  = vv_get_hostname();
    $shortName = strtolower(preg_replace('/^unraid-/i', '', $hostname));
    $pubPath   = '/root/.ssh/' . $shortName . '_rsync_automation.pub';
    $pubKey    = trim(@file_get_contents($pubPath) ?: '');
    echo json_encode([
        'ok'     => $rc === 0 && !empty($pubKey),
        'pubkey' => $pubKey,
        'error'  => ($rc !== 0) ? implode(' ', array_slice(array_filter(array_map('trim', $out)), -3)) : null,
    ]);
    exit;
}

// ── POST: run conf_populate.sh ─────────────────────────────────────────────────────────────────
if ($action === 'populate') {
    $script = SCRIPTS_DIR . '/Plugin/unraid/Tools/conf_populate.sh';
    if (!file_exists($script)) {
        echo json_encode(['ok' => false, 'error' => 'conf_populate.sh not found']);
        exit;
    }
    exec('bash ' . escapeshellarg($script) . ' --no-push 2>&1', $out, $rc);
    $lines = array_values(array_filter(array_map('trim', $out)));
    echo json_encode(['ok' => $rc === 0, 'lines' => array_slice($lines, 0, 20)]);
    exit;
}

if ($_SERVER['REQUEST_METHOD'] !== 'POST') {
    echo json_encode(['ok' => false, 'error' => 'Method not allowed']);
    exit;
}

$sshScript = SCRIPTS_DIR . '/Partnership/ssh_setup.sh';

// ── Pull master.conf from HOST1 via SSH (wizard or checklist) ────────────────────────────────
if ($action === 'pull') {
    $mySlot        = trim($_POST['my_slot']        ?? '') ?: strtolower(vv_detect_host());
    $myHostname    = trim($_POST['my_hostname']     ?? '') ?: vv_get_hostname();
    $host1Hostname = trim($_POST['host1_hostname']  ?? '');
    if (!$host1Hostname) {
        $masterRaw = vv_read_conf_raw('master.conf');
        preg_match('/^\s*HOST1\s*=\s*"([^"]*)"/m', $masterRaw, $_mh);
        $host1Hostname = trim($_mh[1] ?? '');
    }
    if (!$host1Hostname) {
        echo json_encode(['ok' => false, 'error' => 'HOST1 hostname not set — fill in master.conf first']);
        exit;
    }
    if (!preg_match('/^host\d+$/', $mySlot)) {
        echo json_encode(['ok' => false, 'error' => 'Invalid slot']);
        exit;
    }

    $hostId    = strtoupper($mySlot);
    $hostIdLow = strtolower($mySlot);

    // Derive SSH key path from this server's hostname
    $sshOwner = strtolower(preg_replace('/^unraid-/i', '', $myHostname ?: vv_get_hostname()));
    $sshKey   = '/root/.ssh/' . $sshOwner . '_rsync_automation';

    if (!file_exists($sshKey)) {
        echo json_encode(['ok' => false, 'error' =>
            "SSH key not found at $sshKey — run Partnership/ssh_setup.sh first"]);
        exit;
    }

    // Resolve HOST1 Tailscale IP
    $ip = trim(shell_exec('tailscale ip -4 ' . escapeshellarg($host1Hostname) . ' 2>/dev/null') ?: '');
    if (!$ip) {
        echo json_encode(['ok' => false, 'error' =>
            "Cannot resolve Tailscale IP for $host1Hostname — is Tailscale running on both servers?"]);
        exit;
    }

    // Get HOST1's SCRIPTS_DIR from their varaverk.cfg
    $sshBase   = 'ssh -i ' . escapeshellarg($sshKey)
               . ' -o ConnectTimeout=10 -o StrictHostKeyChecking=no root@' . $ip;
    $remoteCfg = trim(shell_exec($sshBase . ' "grep SCRIPTS_DIR /boot/config/plugins/varaverk/varaverk.cfg 2>/dev/null"') ?: '');
    preg_match('/SCRIPTS_DIR\s*=\s*["\']?([^"\']+)["\']?/', $remoteCfg, $sm);
    $remoteConf = rtrim($sm[1] ?? '/boot/config/plugins/varaverk', '/') . '/Configurations';

    // SCP master.conf from HOST1
    $localMaster = CONF_DIR . '/master.conf';
    $src  = escapeshellarg('root@' . $ip . ':' . $remoteConf . '/master.conf');
    $cmd  = 'scp -i ' . escapeshellarg($sshKey)
          . ' -o ConnectTimeout=10 -o StrictHostKeyChecking=no'
          . ' ' . $src . ' ' . escapeshellarg($localMaster) . ' 2>&1';
    exec($cmd, $out, $rc);
    if ($rc !== 0) {
        echo json_encode(['ok' => false, 'error' =>
            'SCP failed: ' . implode('; ', $out) .
            ' — ensure your SSH key is authorised on HOST1 (run Partnership/ssh_setup.sh)']);
        exit;
    }

    // Create host conf from template if it doesn't exist
    $confFile = $hostIdLow . '.conf';
    if (!file_exists(CONF_DIR . '/' . $confFile)) {
        $template = @file_get_contents(CONF_DIR . '/host.conf.template') ?: '';
        if ($template) {
            $bootPart2  = trim(shell_exec('findmnt -n -o SOURCE /boot 2>/dev/null') ?: '');
            $bootDisk2  = $bootPart2 ? trim(shell_exec('lsblk -no pkname ' . escapeshellarg($bootPart2) . ' 2>/dev/null') ?: '') : '';
            $transport2 = $bootDisk2 ? strtolower(trim(shell_exec('lsblk -dno TRAN /dev/' . escapeshellarg($bootDisk2) . ' 2>/dev/null') ?: '')) : '';
            $storageInternal2 = ($transport2 !== 'usb') ? 'true' : 'false';
            $conf = str_replace('HOSTN', $hostId,    $template);
            $conf = str_replace('hostn', $hostIdLow, $conf);
            $conf = preg_replace('/^(\s*' . $hostId . '_SSH_KEY\s*=\s*)""/m',
                '${1}"' . $sshKey . '"', $conf);
            $conf = preg_replace('/^(\s*' . $hostId . '_STORAGE_MODE_INTERNAL\s*=\s*)\S+/m',
                '${1}' . $storageInternal2, $conf);
            vv_write_conf_raw($confFile, $conf);
        }
    }

    if (file_exists($sshScript)) {
        exec('bash ' . escapeshellarg($sshScript) . ' --local-only 2>/dev/null');
    }
    $apiKeyResult = vv_auto_create_api_key($hostId, $confFile);

    $state = vv_setup_state_read();
    $state['master_conf_pulled'] = 'true';
    vv_setup_state_write($state);

    echo json_encode(['ok' => true, 'host_id' => $hostId, 'conf_file' => $confFile,
                      'api_key'  => $apiKeyResult,
                      'redirect' => '?tab=scheduler&vv_setup=' . $confFile]);
    exit;
}

// ── Default action: save (HOST1 first-run wizard) ────────────────────────────────────────────
$host1      = trim($_POST['host1']      ?? '');
$host2      = trim($_POST['host2']      ?? '');
$mySlot     = trim($_POST['my_slot']    ?? 'host1');
$myHostname = trim($_POST['my_hostname'] ?? '');

if (empty($host1)) {
    echo json_encode(['ok' => false, 'error' => 'HOST1 hostname is required']);
    exit;
}
if (!preg_match('/^host\d+$/', $mySlot)) {
    echo json_encode(['ok' => false, 'error' => 'Invalid slot']);
    exit;
}

// Write HOST1 / HOST2 into master.conf
$master = vv_read_conf_raw('master.conf');
if ($master === '') {
    echo json_encode(['ok' => false, 'error' => 'master.conf not found — check SCRIPTS_DIR in varaverk.cfg']);
    exit;
}

$master = preg_replace('/^(\s*HOST1\s*=\s*).*$/m', '${1}"' . addslashes($host1) . '"', $master);
$master = preg_replace('/^(\s*HOST2\s*=\s*).*$/m', '${1}"' . addslashes($host2) . '"', $master);

$slotNum = (int) preg_replace('/\D/', '', $mySlot);
if ($slotNum > 2 && !empty($myHostname)) {
    $hostKey = 'HOST' . $slotNum;
    if (!preg_match('/^\s*' . $hostKey . '\s*=/m', $master)) {
        $master = preg_replace('/^(\s*HOST2\s*=.*$)/m',
            '$1' . "\n    {$hostKey}=\"" . addslashes($myHostname) . '"', $master);
    } else {
        $master = preg_replace('/^(\s*' . $hostKey . '\s*=\s*).*$/m',
            '${1}"' . addslashes($myHostname) . '"', $master);
    }
}

if (!vv_write_conf_raw('master.conf', $master)) {
    echo json_encode(['ok' => false, 'error' => 'Failed to write master.conf']);
    exit;
}

// Create host*.conf from template
$hostId    = strtoupper($mySlot);
$hostIdLow = strtolower($mySlot);
$confFile  = $hostIdLow . '.conf';

// Storage mode: use wizard selection, fall back to auto-detect from boot transport
$smParam = trim($_POST['storage_mode'] ?? '');
if ($smParam === 'flash') {
    $storageInternal = 'false';
} elseif ($smParam === 'internal') {
    $storageInternal = 'true';
} else {
    $bootPart  = trim(shell_exec('findmnt -n -o SOURCE /boot 2>/dev/null') ?: '');
    $bootDisk  = $bootPart ? trim(shell_exec('lsblk -no pkname ' . escapeshellarg($bootPart) . ' 2>/dev/null') ?: '') : '';
    $transport = $bootDisk ? strtolower(trim(shell_exec('lsblk -dno TRAN /dev/' . escapeshellarg($bootDisk) . ' 2>/dev/null') ?: '')) : '';
    $storageInternal = ($transport !== 'usb') ? 'true' : 'false';
}

if (!file_exists(CONF_DIR . '/' . $confFile)) {
    $template = @file_get_contents(CONF_DIR . '/host.conf.template') ?: '';
    if ($template) {
        $sshOwner   = strtolower(preg_replace('/^unraid-/i', '', $myHostname));
        $sshKeyPath = '/root/.ssh/' . $sshOwner . '_rsync_automation';

        $conf = str_replace('HOSTN', $hostId,    $template);
        $conf = str_replace('hostn', $hostIdLow, $conf);
        $conf = preg_replace('/^(\s*' . $hostId . '_SSH_KEY\s*=\s*)""/m',
            '${1}"' . $sshKeyPath . '"', $conf);
        $conf = preg_replace('/^(\s*' . $hostId . '_STORAGE_MODE_INTERNAL\s*=\s*)\S+/m',
            '${1}' . $storageInternal, $conf);
        if (!vv_write_conf_raw($confFile, $conf)) {
            echo json_encode(['ok' => false, 'error' => "Failed to write $confFile"]);
            exit;
        }
    }
}

// Write setup state file — lets partner servers know HOST1 is configured
vv_setup_state_write(['host1_hostname' => $host1]);

// Auto-generate SSH keypair (local only — remote copy happens during onboarding)
if (file_exists($sshScript)) {
    exec('bash ' . escapeshellarg($sshScript) . ' --local-only 2>/dev/null');
}

// Auto-create Unraid API key and write into the fresh conf
$apiKeyResult = vv_auto_create_api_key($hostId, $confFile);

$targetDir      = ($storageInternal === 'true') ? '/boot/config/plugins/varaverk' : '/mnt/user/appdata/Varaverk';
$needsMigration = (defined('SCRIPTS_DIR') && SCRIPTS_DIR !== $targetDir);

echo json_encode([
    'ok'             => true,
    'host_id'        => $hostId,
    'api_key'        => $apiKeyResult,
    'needs_migration'=> $needsMigration,
    'migrate_to'     => $needsMigration ? ($storageInternal === 'true' ? 'internal' : 'flash') : null,
    'redirect'       => '?tab=scheduler&vv_setup=master.conf',
]);


I have a clear picture of the flow. The `partnership_onboard.sh` deploys containers to the mirror (Steps 3, 5, 7) via `deploy_container_from_xml`, which uses `--network=` from the XML. If that custom network doesn't exist on the mirror yet, the `docker create` will fail.

The fix is: after Step 1 (SSH confirmed) and before Step 2 (stop/deploy stacks), SSH to the mirror and run `docker_network_connect.sh` there.


Edit

STEP_SSH_OK=false
STEP_STOP_AUTH_OK=true
STEP_AUTH_OK=true
AUTH_DEPLOYED=0
AUTH_FAILED=0
STEP_STOP_ARR_OK=true
STEP_ARR_OK=true
ARR_DEPLOYED=0
ARR_FAILED=0
STEP_STOP_SERVICES_OK=true
STEP_SERVICES_OK=true
SERVICES_DEPLOYED=0
SERVICES_FAILED=0
ONBOARD_OK=false
ARR_SYNC_OK=false
MASTER_PUSH_OK=false

STEP_SSH_OK=false
STEP_NETWORK_OK=false
STEP_STOP_AUTH_OK=true
STEP_AUTH_OK=true
AUTH_DEPLOYED=0
AUTH_FAILED=0
STEP_STOP_ARR_OK=true
STEP_ARR_OK=true
ARR_DEPLOYED=0
ARR_FAILED=0
STEP_STOP_SERVICES_OK=true
STEP_SERVICES_OK=true
SERVICES_DEPLOYED=0
SERVICES_FAILED=0
ONBOARD_OK=false
ARR_SYNC_OK=false
MASTER_PUSH_OK=false


The file /boot/config/plugins/varaverk/Partnership/partnership_onboard.sh has been updated successfully. (file state is current in your context — no need to Read it back)

STEP_SSH_OK=false
STEP_STOP_AUTH_OK=true
STEP_AUTH_OK=true
AUTH_DEPLOYED=0
AUTH_FAILED=0
STEP_STOP_ARR_OK=true
STEP_ARR_OK=true
ARR_DEPLOYED=0
ARR_FAILED=0
STEP_STOP_SERVICES_OK=true
STEP_SERVICES_OK=true
SERVICES_DEPLOYED=0
SERVICES_FAILED=0
ONBOARD_OK=false
ARR_SYNC_OK=false
MASTER_PUSH_OK=false

STEP_SSH_OK=false
STEP_NETWORK_OK=false
STEP_STOP_AUTH_OK=true
STEP_AUTH_OK=true
AUTH_DEPLOYED=0
AUTH_FAILED=0
STEP_STOP_ARR_OK=true
STEP_ARR_OK=true
ARR_DEPLOYED=0
ARR_FAILED=0
STEP_STOP_SERVICES_OK=true
STEP_SERVICES_OK=true
SERVICES_DEPLOYED=0
SERVICES_FAILED=0
ONBOARD_OK=false
ARR_SYNC_OK=false
MASTER_PUSH_OK=false


Now add the network ensure step after the Phase 1 exit point and before Step 2:


# ── Step 2: Stop mirror's existing auth stack ─────────────────────────────────────────────────
echo ""
echo "━━━ Step 2 — Stop Mirror Auth Stack ━━━"

# ── Step 1b: Ensure custom Docker network exists on mirror ────────────────────────────────────
# Must run before any container deploy — docker create fails if the network is missing.
echo ""
echo "━━━ Step 1b — Docker Network (Mirror) ━━━"

_net_script="${SCRIPTS_ROOT}/Docker_Essentials/docker_network_connect.sh"
if [[ "$DRY_RUN" == true ]]; then
    warn "DRY RUN — would run docker_network_connect.sh on $MIRROR"
    STEP_NETWORK_OK=true
elif timeout 60 ssh -i "$MIRROR_SSH_KEY" \
        -o ConnectTimeout="$SSH_TIMEOUT" -o BatchMode=yes root@"$MIRROR_IP" \
        "bash '$_net_script'" 2>/dev/null; then
    log "Docker network ready on $MIRROR ✅"
    STEP_NETWORK_OK=true
else
    warn "docker_network_connect.sh failed on $MIRROR — containers may fail if network is missing"
    warn "Check ${_net_script} on $MIRROR and re-run with --skip-ssh if needed"
fi

# ── Step 2: Stop mirror's existing auth stack ─────────────────────────────────────────────────
echo ""
echo "━━━ Step 2 — Stop Mirror Auth Stack ━━━"


# ── Step 1b: Ensure custom Docker network exists on mirror ────────────────────────────────────
# Must run before any container deploy — docker create fails if the network is missing.
echo ""
echo "━━━ Step 1b — Docker Network (Mirror) ━━━"

_net_script="${SCRIPTS_ROOT}/Docker_Essentials/docker_network_connect.sh"
if [[ "$DRY_RUN" == true ]]; then
    warn "DRY RUN — would run docker_network_connect.sh on $MIRROR"
    STEP_NETWORK_OK=true
elif timeout 60 ssh -i "$MIRROR_SSH_KEY" \
        -o ConnectTimeout="$SSH_TIMEOUT" -o BatchMode=yes root@"$MIRROR_IP" \
        "bash '$_net_script'" 2>/dev/null; then
    log "Docker network ready on $MIRROR ✅"
    STEP_NETWORK_OK=true
else
    warn "docker_network_connect.sh failed on $MIRROR — containers may fail if network is missing"
    warn "Check ${_net_script} on $MIRROR and re-run with --skip-ssh if needed"
fi

# ── Step 2: Stop mirror's existing auth stack ─────────────────────────────────────────────────
echo ""
echo "━━━ Step 2 — Stop Mirror Auth Stack ━━━"

+# ── Step 1b: Ensure custom Docker network exists on mirror ────────────────────────────────────

+# Must run before any container deploy — docker create fails if the network is missing.

+echo "━━━ Step 1b — Docker Network (Mirror) ━━━"

+    warn "DRY RUN — would run docker_network_connect.sh on $MIRROR"

+elif timeout 60 ssh -i "$MIRROR_SSH_KEY" \

+        -o ConnectTimeout="$SSH_TIMEOUT" -o BatchMode=yes root@"$MIRROR_IP" \

+        "bash '$_net_script'" 2>/dev/null; then

+    log "Docker network ready on $MIRROR ✅"

+    warn "docker_network_connect.sh failed on $MIRROR — containers may fail if network is missing"

+    warn "Check ${_net_script} on $MIRROR and re-run with --skip-ssh if needed"


Now add it to the summary:


echo "  Step 1  — SSH keys:         $(_skip "$SKIP_SSH"             "$STEP_SSH_OK")"
echo "  Step 2  — Stop auth:        $(_skip "$SKIP_AUTH_STACK"       "$STEP_STOP_AUTH_OK")"

echo "  Step 1  — SSH keys:         $(_skip "$SKIP_SSH"             "$STEP_SSH_OK")"
echo "  Step 1b — Docker network:   $(_ok "$STEP_NETWORK_OK")"
echo "  Step 2  — Stop auth:        $(_skip "$SKIP_AUTH_STACK"       "$STEP_STOP_AUTH_OK")"


echo "  Step 1  — SSH keys:         $(_skip "$SKIP_SSH"             "$STEP_SSH_OK")"
echo "  Step 2  — Stop auth:        $(_skip "$SKIP_AUTH_STACK"       "$STEP_STOP_AUTH_OK")"

echo "  Step 1  — SSH keys:         $(_skip "$SKIP_SSH"             "$STEP_SSH_OK")"
echo "  Step 1b — Docker network:   $(_ok "$STEP_NETWORK_OK")"
echo "  Step 2  — Stop auth:        $(_skip "$SKIP_AUTH_STACK"       "$STEP_STOP_AUTH_OK")"

+echo "  Step 1b — Docker network:   $(_ok "$STEP_NETWORK_OK")"

echo "  Step 3  — Auth stack:       $( [[ "$SKIP_AUTH_STACK" == true ]] && echo "skipped" || echo "${AUTH_DEPLOYED} deployed, ${AUTH_FAILED} failed" )"

echo "  Step 4  — Stop arr:         $(_skip "$SKIP_ARR_STACK"        "$STEP_STOP_ARR_OK")"


Let me verify the changes look right in context:


500	    echo "  When HOST2 completes its onboard, it will automatically trigger Phase 2 here."
501	    echo "━━━━━━━━━━━━━━━━━━━━━━━"
502	    exit 0
503	fi
504
505	# ── Step 1b: Ensure custom Docker network exists on mirror ────────────────────────────────────
506	# Must run before any container deploy — docker create fails if the network is missing.
507	echo ""
508	echo "━━━ Step 1b — Docker Network (Mirror) ━━━"
509
510	_net_script="${SCRIPTS_ROOT}/Docker_Essentials/docker_network_connect.sh"
511	if [[ "$DRY_RUN" == true ]]; then
512	    warn "DRY RUN — would run docker_network_connect.sh on $MIRROR"
513	    STEP_NETWORK_OK=true
514	elif timeout 60 ssh -i "$MIRROR_SSH_KEY" \
515	        -o ConnectTimeout="$SSH_TIMEOUT" -o BatchMode=yes root@"$MIRROR_IP" \
516	        "bash '$_net_script'" 2>/dev/null; then
517	    log "Docker network ready on $MIRROR ✅"
518	    STEP_NETWORK_OK=true
519	else
520	    warn "docker_network_connect.sh failed on $MIRROR — containers may fail if network is missing"
521	    warn "Check ${_net_script} on $MIRROR and re-run with --skip-ssh if needed"
522	fi
523
524	# ── Step 2: Stop mirror's existing auth stack ─────────────────────────────────────────────────
525	echo ""
526	echo "━━━ Step 2 — Stop Mirror Auth Stack ━━━"
527
528	if [[ "$SKIP_AUTH_STACK" == true ]]; then
529	    warn "Skipping (--skip-auth-stack)"
530	else
531	    stop_mirror_stack "PARTNERSHIP_REPLACE_CONTAINERS" "auth stack"
532	fi
533
534	# ── Step 4: Deploy auth stack on mirror ───────────────────────────────────────────────────────
535	echo ""
536	echo "━━━ Step 3 — Deploy Auth Stack on Mirror ━━━"
537
538	if [[ "$SKIP_AUTH_STACK" == true ]]; then
539	    warn "Skipping (--skip-auth-stack)"
540	elif [[ ${#PARTNERSHIP_AUTH_STACK[@]} -eq 0 ]]; then
541	    warn "PARTNERSHIP_AUTH_STACK not set in ${MY_ID} conf — skipping auth stack deploy"
542	    warn "Add HOST${MY_ID: -1}_PARTNERSHIP_AUTH_STACK to host${MY_ID: -1}.conf"
543	    STEP_AUTH_OK=false
544	else
545	    deploy_xml_stack PARTNERSHIP_AUTH_STACK
546	    AUTH_DEPLOYED=$_STACK_DEPLOYED
547	    AUTH_FAILED=$_STACK_FAILED
548	    echo "Auth stack: $AUTH_DEPLOYED deployed, $AUTH_FAILED failed"
549	    [[ "$AUTH_FAILED" -gt 0 ]] && STEP_AUTH_OK=false
550	fi
551
552	# ── Step 5: Stop mirror's existing arr stack ──────────────────────────────────────────────────
553	echo ""
554	echo "━━━ Step 4 — Stop Mirror Arr Stack ━━━"
555
556	if [[ "$SKIP_ARR_STACK" == true ]]; then
557	    warn "Skipping (--skip-arr-stack)"
558	elif [[ ${#PARTNERSHIP_ARR_STACK[@]} -eq 0 ]]; then
559	    log "PARTNERSHIP_ARR_STACK not configured — skipping arr stack deploy"

    echo "  When HOST2 completes its onboard, it will automatically trigger Phase 2 here."
    echo "━━━━━━━━━━━━━━━━━━━━━━━"
    exit 0
fi

# ── Step 1b: Ensure custom Docker network exists on mirror ────────────────────────────────────
# Must run before any container deploy — docker create fails if the network is missing.
echo ""
echo "━━━ Step 1b — Docker Network (Mirror) ━━━"

_net_script="${SCRIPTS_ROOT}/Docker_Essentials/docker_network_connect.sh"
if [[ "$DRY_RUN" == true ]]; then
    warn "DRY RUN — would run docker_network_connect.sh on $MIRROR"
    STEP_NETWORK_OK=true
elif timeout 60 ssh -i "$MIRROR_SSH_KEY" \
        -o ConnectTimeout="$SSH_TIMEOUT" -o BatchMode=yes root@"$MIRROR_IP" \
        "bash '$_net_script'" 2>/dev/null; then
    log "Docker network ready on $MIRROR ✅"
    STEP_NETWORK_OK=true
else
    warn "docker_network_connect.sh failed on $MIRROR — containers may fail if network is missing"
    warn "Check ${_net_script} on $MIRROR and re-run with --skip-ssh if needed"
fi

# ── Step 2: Stop mirror's existing auth stack ─────────────────────────────────────────────────
echo ""
echo "━━━ Step 2 — Stop Mirror Auth Stack ━━━"

if [[ "$SKIP_AUTH_STACK" == true ]]; then
    warn "Skipping (--skip-auth-stack)"
else
    stop_mirror_stack "PARTNERSHIP_REPLACE_CONTAINERS" "auth stack"
fi

# ── Step 4: Deploy auth stack on mirror ───────────────────────────────────────────────────────
echo ""
echo "━━━ Step 3 — Deploy Auth Stack on Mirror ━━━"

if [[ "$SKIP_AUTH_STACK" == true ]]; then
    warn "Skipping (--skip-auth-stack)"
elif [[ ${#PARTNERSHIP_AUTH_STACK[@]} -eq 0 ]]; then
    warn "PARTNERSHIP_AUTH_STACK not set in ${MY_ID} conf — skipping auth stack deploy"
    warn "Add HOST${MY_ID: -1}_PARTNERSHIP_AUTH_STACK to host${MY_ID: -1}.conf"
    STEP_AUTH_OK=false
else
    deploy_xml_stack PARTNERSHIP_AUTH_STACK
    AUTH_DEPLOYED=$_STACK_DEPLOYED
    AUTH_FAILED=$_STACK_FAILED
    echo "Auth stack: $AUTH_DEPLOYED deployed, $AUTH_FAILED failed"
    [[ "$AUTH_FAILED" -gt 0 ]] && STEP_AUTH_OK=false
fi

# ── Step 5: Stop mirror's existing arr stack ──────────────────────────────────────────────────
echo ""
echo "━━━ Step 4 — Stop Mirror Arr Stack ━━━"

if [[ "$SKIP_ARR_STACK" == true ]]; then
    warn "Skipping (--skip-arr-stack)"
elif [[ ${#PARTNERSHIP_ARR_STACK[@]} -eq 0 ]]; then
    log "PARTNERSHIP_ARR_STACK not configured — skipping arr stack deploy"


668	echo "  Mirror:   $MIRROR ($MIRROR_IP)"
669	[[ "$PHASE2_ONLY" == true ]] && echo "  Mode:     Phase 2 (triggered by HOST2 notification)"
670	echo "  Duration: $(format_duration $(( END - START )))"
671	echo ""
672	_ok()   { [[ "$1" == true ]] && echo "✅" || echo "❌"; }
673	_skip() { [[ "$1" == true ]] && echo "skipped" || echo "$(_ok "$2")"; }
674
675	echo "  Step 1  — SSH keys:         $(_skip "$SKIP_SSH"             "$STEP_SSH_OK")"
676	echo "  Step 1b — Docker network:   $(_ok "$STEP_NETWORK_OK")"
677	echo "  Step 2  — Stop auth:        $(_skip "$SKIP_AUTH_STACK"       "$STEP_STOP_AUTH_OK")"
678	echo "  Step 3  — Auth stack:       $( [[ "$SKIP_AUTH_STACK" == true ]] && echo "skipped" || echo "${AUTH_DEPLOYED} deployed, ${AUTH_FAILED} failed" )"
679	echo "  Step 4  — Stop arr:         $(_skip "$SKIP_ARR_STACK"        "$STEP_STOP_ARR_OK")"
680	echo "  Step 5  — Arr stack:        $( [[ "$SKIP_ARR_STACK" == true ]] && echo "skipped" || echo "${ARR_DEPLOYED} deployed, ${ARR_FAILED} failed" )"
681	echo "  Step 6  — Stop services:    $(_skip "$SKIP_SERVICES_STACK"   "$STEP_STOP_SERVICES_OK")"
682	echo "  Step 7  — Services stack:   $( [[ "$SKIP_SERVICES_STACK" == true ]] && echo "skipped" || echo "${SERVICES_DEPLOYED} deployed, ${SERVICES_FAILED} failed" )"
683	echo "  Step 8  — Onboard:          $(_ok "$ONBOARD_OK")"
684	echo "  Step 9  — Arr bootstrap:    $( [[ "$SKIP_ARR_SYNC" == true || "$ONBOARD_OK" == false ]] && echo "skipped" || echo "$(_ok "$ARR_SYNC_OK")" )"
685	echo "  Step 10 — Conf push:        $( [[ "$ONBOARD_OK" == false ]] && echo "skipped" || echo "$(_ok "$MASTER_PUSH_OK")" )"
686	echo ""
687

echo "  Mirror:   $MIRROR ($MIRROR_IP)"
[[ "$PHASE2_ONLY" == true ]] && echo "  Mode:     Phase 2 (triggered by HOST2 notification)"
echo "  Duration: $(format_duration $(( END - START )))"
echo ""
_ok()   { [[ "$1" == true ]] && echo "✅" || echo "❌"; }
_skip() { [[ "$1" == true ]] && echo "skipped" || echo "$(_ok "$2")"; }

echo "  Step 1  — SSH keys:         $(_skip "$SKIP_SSH"             "$STEP_SSH_OK")"
echo "  Step 1b — Docker network:   $(_ok "$STEP_NETWORK_OK")"
echo "  Step 2  — Stop auth:        $(_skip "$SKIP_AUTH_STACK"       "$STEP_STOP_AUTH_OK")"
echo "  Step 3  — Auth stack:       $( [[ "$SKIP_AUTH_STACK" == true ]] && echo "skipped" || echo "${AUTH_DEPLOYED} deployed, ${AUTH_FAILED} failed" )"
echo "  Step 4  — Stop arr:         $(_skip "$SKIP_ARR_STACK"        "$STEP_STOP_ARR_OK")"
echo "  Step 5  — Arr stack:        $( [[ "$SKIP_ARR_STACK" == true ]] && echo "skipped" || echo "${ARR_DEPLOYED} deployed, ${ARR_FAILED} failed" )"
echo "  Step 6  — Stop services:    $(_skip "$SKIP_SERVICES_STACK"   "$STEP_STOP_SERVICES_OK")"
echo "  Step 7  — Services stack:   $( [[ "$SKIP_SERVICES_STACK" == true ]] && echo "skipped" || echo "${SERVICES_DEPLOYED} deployed, ${SERVICES_FAILED} failed" )"
echo "  Step 8  — Onboard:          $(_ok "$ONBOARD_OK")"
echo "  Step 9  — Arr bootstrap:    $( [[ "$SKIP_ARR_SYNC" == true || "$ONBOARD_OK" == false ]] && echo "skipped" || echo "$(_ok "$ARR_SYNC_OK")" )"
echo "  Step 10 — Conf push:        $( [[ "$ONBOARD_OK" == false ]] && echo "skipped" || echo "$(_ok "$MASTER_PUSH_OK")" )"
echo ""


Looks clean. The changes are:

- **`STEP_NETWORK_OK=false`** added to the tracking var block
- **Step 1b block** inserted after Phase 1 exit and before Step 2 — SSHes to the mirror and runs `docker_network_connect.sh`, soft-fails with a warning (doesn't abort the entire onboard since it's non-fatal if the network happens to already exist via other means)
- **Summary row** for Step 1b added between Step 1 and Step 2

The 60-second timeout on the SSH call is generous enough for the network script to create networks and connect containers if needed, without hanging indefinitely.


ok now daily and weekly ran. no containers were updated


Let me check the daily update script and recent logs.


Bash

find /boot/config/plugins/varaverk/Orchestrators -name "daily*" -o -name "weekly*" | sort

Find daily and weekly orchestrator scripts


/boot/config/plugins/varaverk/Orchestrators/daily_sync_maintenance.sh
/boot/config/plugins/varaverk/Orchestrators/weekly_sync_maintenance.sh


find /boot/config/plugins/varaverk -name "*docker_update*" -o -name "*container_update*" | grep -v ".git" | grep -v "claude-data" | sort

Find docker update scripts


/boot/config/plugins/varaverk/Docker_Essentials/docker_update.sh
/boot/config/plugins/varaverk/Docker_Essentials/docker_update_remaining.sh


1	#!/bin/bash
2	# ==============================================================================================
3	# ============================= Daily Sync Maintenance =========================================
4	# ==============================================================================================
5	# Daily orchestrator — runs the full daily maintenance window in the correct order.
6	# Schedule: 0 1 * * * (1am daily via User Scripts plugin)
7	#
8	# ── EXECUTION ORDER ───────────────────────────────────────────────────────────────────────────
9	#   Pre-sync:
10	#     git_pull_execute.sh              — pull latest scripts first, always
11	#
12	#   Arr Sync (arr_sync.sh):
13	#     Syncs Lidarr/Sonarr/Radarr libraries across all nodes bidirectionally.
14	#     All nodes agree on tracked library before any files are transferred.
15	#     Remote nodes that don't have an arr running are skipped gracefully.
16	#
17	#   Rsync window (DAILY_SYNC_SHARES per host):
18	#     HOST*_DAILY_SYNC_SHARES          — media shares spread to all nodes (no --delete)
19	#     HOST*_PERSONAL_SHARES            — encrypted personal shares
20	#
21	#   Post-sync maintenance (DAILY_MAINTENANCE_SCRIPTS):
22	#     media_shares_permissions.sh      — fix ownership before arr cleanup
23	#     media_cleaner.sh anime           — remove junk from anime shares
24	#     media_cleaner.sh media           — remove junk from media shares
25	#     lidarr_cleanup.sh                — remove orphaned music files (local arr = truth)
26	#     sonarr_cleanup.sh                — remove orphaned TV files (local arr = truth)
27	#     radarr_cleanup.sh                — remove orphaned movie files (local arr = truth)
28	#     docker_daily_restart.sh          — restart containers needing daily restart
29	#
30	# ── WHY ORDER MATTERS ─────────────────────────────────────────────────────────────────────────
31	#   git pull first        — maintenance runs on latest code, not yesterday's
32	#   arr sync before rsync — all nodes track the same library before files are spread;
33	#                           prevents remote arrs from searching for content already owned
34	#   rsync before cleanup  — cleanup sees fully spread state, rsync has no --delete
35	#   permissions before arr cleanup — arrs need correct ownership to delete/rename
36	#   arr cleanup after permissions  — clean ownership = successful orphan deletion
37	#   docker restart last   — containers already processed by cleanup
38	#
39	# ── HOST AWARENESS ────────────────────────────────────────────────────────────────────────────
40	# Bidirectional — same script runs on both servers, correct direction automatic.
41	# detect_hosts() aliases DAILY_SYNC_SHARES and PERSONAL_SHARES from HOST*_ vars.
42	# No manual HOST1/HOST2 comparisons — MY_ID routes correctly on any server.
43	#
44	# ── DRIVE TEMP HANDLING ───────────────────────────────────────────────────────────────────────
45	# rsync.sh returns exit codes for temperature issues:
46	#   exit 1 = temp WARN — skip this share, continue to next
47	#   exit 2 = temp CRITICAL — abort ALL remaining syncs in this window
48	#   All other failures — skip share, continue to next
49	#
50	# ── SILENT WHEN HEALTHY ───────────────────────────────────────────────────────────────────────
51	# Runs daily at 1am — clean run should produce minimal output.
52	# Each job reports log() on success (silent), warn()/error() on failure (visible).
53	# Summary always shown — gives window timing and share/job counts.
54	# Notify only on failure — successful daily maintenance doesn't need notification.
55	#
56	# ── CONFIGURATION (master.conf + host*.conf) ───────────────────────────────────────────
57	#   HOST*_DAILY_SYNC_SHARES      — shares pushed to mirror each day
58	#   HOST*_PERSONAL_SHARES        — encrypted personal shares
59	#   DAILY_MAINTENANCE_SCRIPTS    — maintenance jobs (permissions, cleanup, restart)
60	#   DAILY_RSYNC_ENABLED          — enable/disable rsync section
61	#
62	# ── USAGE ─────────────────────────────────────────────────────────────────────────────────────
63	#   daily_sync_maintenance.sh              — normal run
64	#   daily_sync_maintenance.sh --dry-run    — preview without syncing or changing
65	#   daily_sync_maintenance.sh --log        — verbose per-share/per-job output
66	#   daily_sync_maintenance.sh --status     — show configured shares and jobs
67	# ==============================================================================================
68
69	SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
70
71	source "$SCRIPT_DIR/../load_config.sh"
72
73	RSYNC_SCRIPT="$SCRIPT_DIR/../Rsync/rsync.sh"
74	SCRIPTS_ROOT="$SCRIPT_DIR/.."
75
76	parse_args "$@"
77
78	# ==============================================================================================
79	# ━━━ Setup ━━━
80	# ==============================================================================================
81	if [[ "$EUID" -ne 0 ]]; then
82	    error "Must be run as root"
83	    exit 1
84	fi
85
86
87	if ! command -v docker &>/dev/null; then
88	    error "Docker command not found"
89	    exit 1
90	fi
91
92	detect_hosts
93	resolve_remote_ip
94
95	acquire_lock
96
97	[[ "$DRY_RUN" == true ]] && warn "DRY RUN — no changes will be made"
98
99	# ==============================================================================================
100	# ━━━ Status ━━━
101	# ==============================================================================================
102	if [[ "$SHOW_STATUS" == true ]]; then
103	    echo ""
104	    echo "━━━━━ $ICON_SUMMARY DAILY SYNC STATUS ━━━━━"
105	    echo "$ICON_HOST   Identity:        $MY_ID ($LOCAL_SERVER_NAME)"
106	    echo "$ICON_NET    Remote:          $REMOTE_ID ($REMOTE_SERVER_NAME)"
107	    echo "$ICON_SYNC   Rsync enabled:   ${RSYNC_ENABLED:-true}"
108	    echo "$ICON_SYNC   Daily enabled:   ${DAILY_RSYNC_ENABLED:-false}"
109	    echo ""
110	    echo "━━━ Daily Sync Shares ━━━"
111	    for share in "${DAILY_SYNC_SHARES[@]}"; do
112	        [[ -n "$share" ]] && echo "  $ICON_SYNC $share"
113	    done
114	    for share in "${PERSONAL_SHARES[@]}"; do
115	        [[ -n "$share" ]] && echo "  $ICON_SYNC $share (personal)"
116	    done
117	    echo ""
118	    echo "━━━ Daily Maintenance Scripts ━━━"
119	    for entry in "${DAILY_MAINTENANCE_SCRIPTS[@]}"; do
120	        [[ -z "$entry" ]] && continue
121	        echo "  $ICON_GEAR ${entry##*/}"
122	    done
123	    echo "━━━━━━━━━━━━━━━━━━━━━━━"
124	    exit 0
125	fi
126
127	# ==============================================================================================
128	# ── BUILD SHARE LIST via detect_hosts aliases ─────────────────────────────────────────────────
129	# detect_hosts() sets DAILY_SYNC_SHARES and PERSONAL_SHARES from HOST*_ vars
130	# No manual HOST1/HOST2 comparison needed — aliased automatically per server
131	# ==============================================================================================
132	ALL_SHARES=()
133	for share in "${DAILY_SYNC_SHARES[@]}"; do
134	    [[ -n "$share" ]] && ALL_SHARES+=("$share")
135	done
136	for share in "${PERSONAL_SHARES[@]}"; do
137	    [[ -n "$share" ]] && ALL_SHARES+=("$share")
138	done
139	SHARE_COUNT=${#ALL_SHARES[@]}
140
141	# ── Split maintenance scripts: git pull runs pre-sync, rest run post-sync ─────────────────────
142	PRE_SYNC_SCRIPTS=()
143	POST_SYNC_SCRIPTS=()
144	for script_entry in "${DAILY_MAINTENANCE_SCRIPTS[@]}"; do
145	    [[ -z "$script_entry" ]] && continue
146	    script_name=$(basename "${script_entry%% *}")
147	    if [[ "$script_name" == "git_pull_execute.sh" ]]; then
148	        PRE_SYNC_SCRIPTS+=("$script_entry")
149	    else
150	        POST_SYNC_SCRIPTS+=("$script_entry")
151	    fi
152	done
153
154	# Helper — run a maintenance script, track pass/fail
155	run_job() {
156	    local script_entry="$1"
157	    local extra_dry=""
158	    [[ "$DRY_RUN" == true ]] && extra_dry="--dry-run"
159
160	    read -r -a script_args <<< "$script_entry"
161	    local script_path="$SCRIPTS_ROOT/${script_args[0]}"
162	    local script_name
163	    script_name=$(basename "${script_args[0]}")
164	    local extra_args=("${script_args[@]:1}")
165
166	    if [[ ! -f "$script_path" ]]; then
167	        error "$script_name — not found at $script_path"
168	        JOB_FAIL+=("$script_name")
169	        return 1
170	    fi
171
172	    log "Running: $script_name ${extra_args[*]}"
173	    # shellcheck disable=SC2086
174	    if bash "$script_path" "${extra_args[@]}" $extra_dry; then
175	        log "$script_name — done ✅"
176	        JOB_PASS+=("$script_name ${extra_args[*]}")
177	    else
178	        error "$script_name — failed (exit $?)"
179	        JOB_FAIL+=("$script_name ${extra_args[*]}")
180	    fi
181	}
182
183	WINDOW_START=$(date +%s)
184	JOB_PASS=()
185	JOB_FAIL=()
186	PASS=()
187	FAIL=()
188	SHARE_TIMES=()
189
190	echo ""
191	echo "━━━ $ICON_GEAR Daily Maintenance — $MY_ID — $(date '+%Y-%m-%d %H:%M:%S') ━━━"
192
193	# ==============================================================================================
194	# ━━━ Pre-sync — git pull ━━━
195	# ==============================================================================================
196	if [[ ${#PRE_SYNC_SCRIPTS[@]} -gt 0 ]]; then
197	    echo ""
198	    echo "━━━ $ICON_GIT Pre-sync ━━━"
199	    for script_entry in "${PRE_SYNC_SCRIPTS[@]}"; do
200	        run_job "$script_entry"
201	    done
202	fi
203
204	# ==============================================================================================
205	# ━━━ Arr Sync — library reconciliation before file spreading ━━━
206	# ==============================================================================================
207	echo ""
208	echo "━━━ $ICON_SYNC Arr Sync ━━━"
209
210	ARR_SYNC_SCRIPT="$SCRIPTS_ROOT/Media/arr_sync.sh"
211	if [[ "${ARR_SYNC_ENABLED:-true}" != "true" ]]; then
212	    echo "ARR_SYNC_ENABLED=false — skipping"
213	elif [[ ! -f "$ARR_SYNC_SCRIPT" ]]; then
214	    warn "arr_sync.sh not found at $ARR_SYNC_SCRIPT — skipping"
215	    JOB_FAIL+=("arr_sync.sh")
216	else
217	    _arr_sync_args=()
218	    [[ "$DRY_RUN" == true ]] && _arr_sync_args+=("--dry-run")
219	    if bash "$ARR_SYNC_SCRIPT" "${_arr_sync_args[@]}"; then
220	        echo "Arr sync complete ✅"
221	        JOB_PASS+=("arr_sync.sh")
222	    else
223	        warn "Arr sync completed with errors — continuing to rsync"
224	        JOB_FAIL+=("arr_sync.sh")
225	    fi
226	    unset _arr_sync_args
227	fi
228
229	# ==============================================================================================
230	# ━━━ Media Share Sync ━━━
231	# ==============================================================================================
232	echo ""
233	echo "━━━ $ICON_SYNC Media Share Sync — $SHARE_COUNT share(s) ━━━"
234
235	TOTAL_START=$(date +%s)
236	SHARE_INDEX=0
237	ABORT_ALL_SYNCS=false
238
239	if ! check_rsync_enabled "DAILY"; then
240	    warn "Daily rsync disabled — skipping all $SHARE_COUNT share syncs"
241	    warn "Proceeding to maintenance jobs..."
242	elif [[ "$SHARE_COUNT" -eq 0 ]]; then
243	    warn "No shares configured for $MY_ID — check HOST*_DAILY_SYNC_SHARES in host*.conf"
244	else
245	    # Pre-flight — connectivity then remote rootfs
246	    check_connectivity
247	    check_remote_rootfs
248
249	    RSYNC_DRY=""
250	    [[ "$DRY_RUN" == true ]] && RSYNC_DRY="--dry-run"
251
252	    for SHARE in "${ALL_SHARES[@]}"; do
253	        (( SHARE_INDEX++ ))
254	        SHARE_NAME=$(basename "$SHARE")
255	        SHARE_START=$(date +%s)
256
257	        echo ""
258	        echo "━━━ $ICON_SYNC Share $SHARE_INDEX/$SHARE_COUNT: $SHARE_NAME ━━━"
259
260	        if [[ "$ABORT_ALL_SYNCS" == true ]]; then
261	            warn "$SHARE_NAME — skipped (drive temps CRITICAL earlier in window)"
262	            FAIL+=("$SHARE_NAME:temp-critical")
263	            continue
264	        fi
265
266	        bash "$RSYNC_SCRIPT" "$SHARE" $RSYNC_DRY
267	        RSYNC_EXIT=$?
268
269	        SHARE_TIMES+=("$SHARE_NAME:$(( $(date +%s) - SHARE_START ))")
270
271	        case "$RSYNC_EXIT" in
272	            0)
273	                PASS+=("$SHARE_NAME")
274	                log "$SHARE_NAME — done ✅"
275	                ;;
276	            1)
277	                FAIL+=("$SHARE_NAME:temp-warn")
278	                warn "$SHARE_NAME skipped — drive temps too high"
279	                ;;
280	            2)
281	                FAIL+=("$SHARE_NAME:temp-critical")
282	                ABORT_ALL_SYNCS=true
283	                error "$SHARE_NAME aborted — drive temps CRITICAL, stopping all remaining syncs"
284	                notify "Daily sync aborted on $(hostname) ($MY_ID) — drive temps CRITICAL during $SHARE_NAME" \
285	                    "Daily Sync" "warning"
286	                ;;
287	            *)
288	                FAIL+=("$SHARE_NAME")
289	                error "$SHARE_NAME failed (exit $RSYNC_EXIT) — continuing to next share"
290	                ;;
291	        esac
292
293	    done
294	fi
295
296	TOTAL_END=$(date +%s)
297
298	# ==============================================================================================
299	# ━━━ Post-sync Maintenance Jobs ━━━
300	# ==============================================================================================
301	if [[ ${#POST_SYNC_SCRIPTS[@]} -gt 0 ]]; then
302	    echo ""
303	    echo "━━━ $ICON_CLEAN Post-sync Maintenance ━━━"
304	    for script_entry in "${POST_SYNC_SCRIPTS[@]}"; do
305	        echo ""
306	        run_job "$script_entry"
307	    done
308	fi
309
310	WINDOW_END=$(date +%s)
311
312	# ==============================================================================================
313	# ━━━ Summary ━━━
314	# ==============================================================================================
315	echo ""
316	echo "━━━━━ $ICON_SUMMARY DAILY MAINTENANCE SUMMARY ━━━━━"
317	echo "$ICON_HOST   Identity: $MY_ID ($LOCAL_SERVER_NAME)"
318	echo "$ICON_TIME   Window:   $(date -d @"$WINDOW_START" '+%Y-%m-%d %H:%M:%S') → $(date -d @"$WINDOW_END" '+%H:%M:%S')"
319	echo "$ICON_TIME   Duration: $(format_duration $(( WINDOW_END - WINDOW_START )))"
320	echo ""
321
322	echo "$ICON_SYNC  Shares ($SHARE_COUNT):"
323	for entry in "${SHARE_TIMES[@]}"; do
324	    sname="${entry%%:*}"
325	    sdur="${entry##*:}"
326	    if printf '%s\n' "${FAIL[@]}" | grep -q "^${sname}"; then
327	        echo "  $ICON_ERROR $sname — $(format_duration "$sdur")"
328	    else
329	        echo "  $ICON_DONE  $sname — $(format_duration "$sdur")"
330	    fi
331	done
332	[[ "$SHARE_COUNT" -eq 0 || "${DAILY_RSYNC_ENABLED:-false}" == "false" ]] && \
333	    echo "  (rsync disabled)"
334	echo "  Passed: ${#PASS[@]}/$SHARE_COUNT  Failed: ${#FAIL[@]}/$SHARE_COUNT"
335	echo ""
336
337	if [[ ${#JOB_PASS[@]} -gt 0 || ${#JOB_FAIL[@]} -gt 0 ]]; then
338	    echo "$ICON_GEAR  Jobs:"
339	    for job in "${JOB_PASS[@]}"; do echo "  $ICON_DONE  $job"; done
340	    for job in "${JOB_FAIL[@]}"; do echo "  $ICON_ERROR $job"; done
341	    echo ""
342	fi
343
344	TOTAL_FAIL=$(( ${#FAIL[@]} + ${#JOB_FAIL[@]} ))
345
346	if [[ "$TOTAL_FAIL" -gt 0 ]]; then
347	    warn "Status: $TOTAL_FAIL failure(s)"
348	    notify "Daily maintenance completed with failures on $(hostname) ($MY_ID) — shares: ${#FAIL[@]}/$SHARE_COUNT failed, jobs: ${#JOB_FAIL[@]} failed" \
349	        "Daily Maintenance" "warning"
350	    echo "━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━"
351	    exit 1
352	else
353	    echo "$ICON_DONE Status: all complete — ${#PASS[@]} share(s) synced, ${#JOB_PASS[@]} job(s) run"
354	    echo "━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━"
355	    exit 0
356	fi

#!/bin/bash
# ==============================================================================================
# ============================= Daily Sync Maintenance =========================================
# ==============================================================================================
# Daily orchestrator — runs the full daily maintenance window in the correct order.
# Schedule: 0 1 * * * (1am daily via User Scripts plugin)
#
# ── EXECUTION ORDER ───────────────────────────────────────────────────────────────────────────
#   Pre-sync:
#     git_pull_execute.sh              — pull latest scripts first, always
#
#   Arr Sync (arr_sync.sh):
#     Syncs Lidarr/Sonarr/Radarr libraries across all nodes bidirectionally.
#     All nodes agree on tracked library before any files are transferred.
#     Remote nodes that don't have an arr running are skipped gracefully.
#
#   Rsync window (DAILY_SYNC_SHARES per host):
#     HOST*_DAILY_SYNC_SHARES          — media shares spread to all nodes (no --delete)
#     HOST*_PERSONAL_SHARES            — encrypted personal shares
#
#   Post-sync maintenance (DAILY_MAINTENANCE_SCRIPTS):
#     media_shares_permissions.sh      — fix ownership before arr cleanup
#     media_cleaner.sh anime           — remove junk from anime shares
#     media_cleaner.sh media           — remove junk from media shares
#     lidarr_cleanup.sh                — remove orphaned music files (local arr = truth)
#     sonarr_cleanup.sh                — remove orphaned TV files (local arr = truth)
#     radarr_cleanup.sh                — remove orphaned movie files (local arr = truth)
#     docker_daily_restart.sh          — restart containers needing daily restart
#
# ── WHY ORDER MATTERS ─────────────────────────────────────────────────────────────────────────
#   git pull first        — maintenance runs on latest code, not yesterday's
#   arr sync before rsync — all nodes track the same library before files are spread;
#                           prevents remote arrs from searching for content already owned
#   rsync before cleanup  — cleanup sees fully spread state, rsync has no --delete
#   permissions before arr cleanup — arrs need correct ownership to delete/rename
#   arr cleanup after permissions  — clean ownership = successful orphan deletion
#   docker restart last   — containers already processed by cleanup
#
# ── HOST AWARENESS ────────────────────────────────────────────────────────────────────────────
# Bidirectional — same script runs on both servers, correct direction automatic.
# detect_hosts() aliases DAILY_SYNC_SHARES and PERSONAL_SHARES from HOST*_ vars.
# No manual HOST1/HOST2 comparisons — MY_ID routes correctly on any server.
#
# ── DRIVE TEMP HANDLING ───────────────────────────────────────────────────────────────────────
# rsync.sh returns exit codes for temperature issues:
#   exit 1 = temp WARN — skip this share, continue to next
#   exit 2 = temp CRITICAL — abort ALL remaining syncs in this window
#   All other failures — skip share, continue to next
#
# ── SILENT WHEN HEALTHY ───────────────────────────────────────────────────────────────────────
# Runs daily at 1am — clean run should produce minimal output.
# Each job reports log() on success (silent), warn()/error() on failure (visible).
# Summary always shown — gives window timing and share/job counts.
# Notify only on failure — successful daily maintenance doesn't need notification.
#
# ── CONFIGURATION (master.conf + host*.conf) ───────────────────────────────────────────
#   HOST*_DAILY_SYNC_SHARES      — shares pushed to mirror each day
#   HOST*_PERSONAL_SHARES        — encrypted personal shares
#   DAILY_MAINTENANCE_SCRIPTS    — maintenance jobs (permissions, cleanup, restart)
#   DAILY_RSYNC_ENABLED          — enable/disable rsync section
#
# ── USAGE ─────────────────────────────────────────────────────────────────────────────────────
#   daily_sync_maintenance.sh              — normal run
#   daily_sync_maintenance.sh --dry-run    — preview without syncing or changing
#   daily_sync_maintenance.sh --log        — verbose per-share/per-job output
#   daily_sync_maintenance.sh --status     — show configured shares and jobs
# ==============================================================================================

SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"

source "$SCRIPT_DIR/../load_config.sh"

RSYNC_SCRIPT="$SCRIPT_DIR/../Rsync/rsync.sh"
SCRIPTS_ROOT="$SCRIPT_DIR/.."

parse_args "$@"

# ==============================================================================================
# ━━━ Setup ━━━
# ==============================================================================================
if [[ "$EUID" -ne 0 ]]; then
    error "Must be run as root"
    exit 1
fi

if ! command -v docker &>/dev/null; then
    error "Docker command not found"
    exit 1
fi

detect_hosts
resolve_remote_ip

acquire_lock

[[ "$DRY_RUN" == true ]] && warn "DRY RUN — no changes will be made"

# ==============================================================================================
# ━━━ Status ━━━
# ==============================================================================================
if [[ "$SHOW_STATUS" == true ]]; then
    echo ""
    echo "━━━━━ $ICON_SUMMARY DAILY SYNC STATUS ━━━━━"
    echo "$ICON_HOST   Identity:        $MY_ID ($LOCAL_SERVER_NAME)"
    echo "$ICON_NET    Remote:          $REMOTE_ID ($REMOTE_SERVER_NAME)"
    echo "$ICON_SYNC   Rsync enabled:   ${RSYNC_ENABLED:-true}"
    echo "$ICON_SYNC   Daily enabled:   ${DAILY_RSYNC_ENABLED:-false}"
    echo ""
    echo "━━━ Daily Sync Shares ━━━"
    for share in "${DAILY_SYNC_SHARES[@]}"; do
        [[ -n "$share" ]] && echo "  $ICON_SYNC $share"
    done
    for share in "${PERSONAL_SHARES[@]}"; do
        [[ -n "$share" ]] && echo "  $ICON_SYNC $share (personal)"
    done
    echo ""
    echo "━━━ Daily Maintenance Scripts ━━━"
    for entry in "${DAILY_MAINTENANCE_SCRIPTS[@]}"; do
        [[ -z "$entry" ]] && continue
        echo "  $ICON_GEAR ${entry##*/}"
    done
    echo "━━━━━━━━━━━━━━━━━━━━━━━"
    exit 0
fi

# ==============================================================================================
# ── BUILD SHARE LIST via detect_hosts aliases ─────────────────────────────────────────────────
# detect_hosts() sets DAILY_SYNC_SHARES and PERSONAL_SHARES from HOST*_ vars
# No manual HOST1/HOST2 comparison needed — aliased automatically per server
# ==============================================================================================
ALL_SHARES=()
for share in "${DAILY_SYNC_SHARES[@]}"; do
    [[ -n "$share" ]] && ALL_SHARES+=("$share")
done
for share in "${PERSONAL_SHARES[@]}"; do
    [[ -n "$share" ]] && ALL_SHARES+=("$share")
done
SHARE_COUNT=${#ALL_SHARES[@]}

# ── Split maintenance scripts: git pull runs pre-sync, rest run post-sync ─────────────────────
PRE_SYNC_SCRIPTS=()
POST_SYNC_SCRIPTS=()
for script_entry in "${DAILY_MAINTENANCE_SCRIPTS[@]}"; do
    [[ -z "$script_entry" ]] && continue
    script_name=$(basename "${script_entry%% *}")
    if [[ "$script_name" == "git_pull_execute.sh" ]]; then
        PRE_SYNC_SCRIPTS+=("$script_entry")
    else
        POST_SYNC_SCRIPTS+=("$script_entry")
    fi
done

# Helper — run a maintenance script, track pass/fail
run_job() {
    local script_entry="$1"
    local extra_dry=""
    [[ "$DRY_RUN" == true ]] && extra_dry="--dry-run"

    read -r -a script_args <<< "$script_entry"
    local script_path="$SCRIPTS_ROOT/${script_args[0]}"
    local script_name
    script_name=$(basename "${script_args[0]}")
    local extra_args=("${script_args[@]:1}")

    if [[ ! -f "$script_path" ]]; then
        error "$script_name — not found at $script_path"
        JOB_FAIL+=("$script_name")
        return 1
    fi

    log "Running: $script_name ${extra_args[*]}"
    # shellcheck disable=SC2086
    if bash "$script_path" "${extra_args[@]}" $extra_dry; then
        log "$script_name — done ✅"
        JOB_PASS+=("$script_name ${extra_args[*]}")
    else
        error "$script_name — failed (exit $?)"
        JOB_FAIL+=("$script_name ${extra_args[*]}")
    fi
}

WINDOW_START=$(date +%s)
JOB_PASS=()
JOB_FAIL=()
PASS=()
FAIL=()
SHARE_TIMES=()

echo ""
echo "━━━ $ICON_GEAR Daily Maintenance — $MY_ID — $(date '+%Y-%m-%d %H:%M:%S') ━━━"

# ==============================================================================================
# ━━━ Pre-sync — git pull ━━━
# ==============================================================================================
if [[ ${#PRE_SYNC_SCRIPTS[@]} -gt 0 ]]; then
    echo ""
    echo "━━━ $ICON_GIT Pre-sync ━━━"
    for script_entry in "${PRE_SYNC_SCRIPTS[@]}"; do
        run_job "$script_entry"
    done
fi

# ==============================================================================================
# ━━━ Arr Sync — library reconciliation before file spreading ━━━
# ==============================================================================================
echo ""
echo "━━━ $ICON_SYNC Arr Sync ━━━"

ARR_SYNC_SCRIPT="$SCRIPTS_ROOT/Media/arr_sync.sh"
if [[ "${ARR_SYNC_ENABLED:-true}" != "true" ]]; then
    echo "ARR_SYNC_ENABLED=false — skipping"
elif [[ ! -f "$ARR_SYNC_SCRIPT" ]]; then
    warn "arr_sync.sh not found at $ARR_SYNC_SCRIPT — skipping"
    JOB_FAIL+=("arr_sync.sh")
else
    _arr_sync_args=()
    [[ "$DRY_RUN" == true ]] && _arr_sync_args+=("--dry-run")
    if bash "$ARR_SYNC_SCRIPT" "${_arr_sync_args[@]}"; then
        echo "Arr sync complete ✅"
        JOB_PASS+=("arr_sync.sh")
    else
        warn "Arr sync completed with errors — continuing to rsync"
        JOB_FAIL+=("arr_sync.sh")
    fi
    unset _arr_sync_args
fi

# ==============================================================================================
# ━━━ Media Share Sync ━━━
# ==============================================================================================
echo ""
echo "━━━ $ICON_SYNC Media Share Sync — $SHARE_COUNT share(s) ━━━"

TOTAL_START=$(date +%s)
SHARE_INDEX=0
ABORT_ALL_SYNCS=false

if ! check_rsync_enabled "DAILY"; then
    warn "Daily rsync disabled — skipping all $SHARE_COUNT share syncs"
    warn "Proceeding to maintenance jobs..."
elif [[ "$SHARE_COUNT" -eq 0 ]]; then
    warn "No shares configured for $MY_ID — check HOST*_DAILY_SYNC_SHARES in host*.conf"
else
    # Pre-flight — connectivity then remote rootfs
    check_connectivity
    check_remote_rootfs

    RSYNC_DRY=""
    [[ "$DRY_RUN" == true ]] && RSYNC_DRY="--dry-run"

    for SHARE in "${ALL_SHARES[@]}"; do
        (( SHARE_INDEX++ ))
        SHARE_NAME=$(basename "$SHARE")
        SHARE_START=$(date +%s)

        echo ""
        echo "━━━ $ICON_SYNC Share $SHARE_INDEX/$SHARE_COUNT: $SHARE_NAME ━━━"

        if [[ "$ABORT_ALL_SYNCS" == true ]]; then
            warn "$SHARE_NAME — skipped (drive temps CRITICAL earlier in window)"
            FAIL+=("$SHARE_NAME:temp-critical")
            continue
        fi

        bash "$RSYNC_SCRIPT" "$SHARE" $RSYNC_DRY
        RSYNC_EXIT=$?

        SHARE_TIMES+=("$SHARE_NAME:$(( $(date +%s) - SHARE_START ))")

        case "$RSYNC_EXIT" in
            0)
                PASS+=("$SHARE_NAME")
                log "$SHARE_NAME — done ✅"
                ;;
            1)
                FAIL+=("$SHARE_NAME:temp-warn")
                warn "$SHARE_NAME skipped — drive temps too high"
                ;;
            2)
                FAIL+=("$SHARE_NAME:temp-critical")
                ABORT_ALL_SYNCS=true
                error "$SHARE_NAME aborted — drive temps CRITICAL, stopping all remaining syncs"
                notify "Daily sync aborted on $(hostname) ($MY_ID) — drive temps CRITICAL during $SHARE_NAME" \
                    "Daily Sync" "warning"
                ;;
            *)
                FAIL+=("$SHARE_NAME")
                error "$SHARE_NAME failed (exit $RSYNC_EXIT) — continuing to next share"
                ;;
        esac

    done
fi

TOTAL_END=$(date +%s)

# ==============================================================================================
# ━━━ Post-sync Maintenance Jobs ━━━
# ==============================================================================================
if [[ ${#POST_SYNC_SCRIPTS[@]} -gt 0 ]]; then
    echo ""
    echo "━━━ $ICON_CLEAN Post-sync Maintenance ━━━"
    for script_entry in "${POST_SYNC_SCRIPTS[@]}"; do
        echo ""
        run_job "$script_entry"
    done
fi

WINDOW_END=$(date +%s)

# ==============================================================================================
# ━━━ Summary ━━━
# ==============================================================================================
echo ""
echo "━━━━━ $ICON_SUMMARY DAILY MAINTENANCE SUMMARY ━━━━━"
echo "$ICON_HOST   Identity: $MY_ID ($LOCAL_SERVER_NAME)"
echo "$ICON_TIME   Window:   $(date -d @"$WINDOW_START" '+%Y-%m-%d %H:%M:%S') → $(date -d @"$WINDOW_END" '+%H:%M:%S')"
echo "$ICON_TIME   Duration: $(format_duration $(( WINDOW_END - WINDOW_START )))"
echo ""

echo "$ICON_SYNC  Shares ($SHARE_COUNT):"
for entry in "${SHARE_TIMES[@]}"; do
    sname="${entry%%:*}"
    sdur="${entry##*:}"
    if printf '%s\n' "${FAIL[@]}" | grep -q "^${sname}"; then
        echo "  $ICON_ERROR $sname — $(format_duration "$sdur")"
    else
        echo "  $ICON_DONE  $sname — $(format_duration "$sdur")"
    fi
done
[[ "$SHARE_COUNT" -eq 0 || "${DAILY_RSYNC_ENABLED:-false}" == "false" ]] && \
    echo "  (rsync disabled)"
echo "  Passed: ${#PASS[@]}/$SHARE_COUNT  Failed: ${#FAIL[@]}/$SHARE_COUNT"
echo ""

if [[ ${#JOB_PASS[@]} -gt 0 || ${#JOB_FAIL[@]} -gt 0 ]]; then
    echo "$ICON_GEAR  Jobs:"
    for job in "${JOB_PASS[@]}"; do echo "  $ICON_DONE  $job"; done
    for job in "${JOB_FAIL[@]}"; do echo "  $ICON_ERROR $job"; done
    echo ""
fi

TOTAL_FAIL=$(( ${#FAIL[@]} + ${#JOB_FAIL[@]} ))

if [[ "$TOTAL_FAIL" -gt 0 ]]; then
    warn "Status: $TOTAL_FAIL failure(s)"
    notify "Daily maintenance completed with failures on $(hostname) ($MY_ID) — shares: ${#FAIL[@]}/$SHARE_COUNT failed, jobs: ${#JOB_FAIL[@]} failed" \
        "Daily Maintenance" "warning"
    echo "━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━"
    exit 1
else
    echo "$ICON_DONE Status: all complete — ${#PASS[@]} share(s) synced, ${#JOB_PASS[@]} job(s) run"
    echo "━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━"
    exit 0
fi


ls /boot/config/plugins/varaverk/data/ 2>/dev/null | grep -i "docker_update\|container_update\|daily\|weekly" | tail -20

Find recent logs related to docker updates


transcode_daily.db
zfs-weekly-health.log


Daily doesn't include `docker_update.sh`. Let me check master.conf and the update scripts.


Check master.conf for maintenance script lists and docker update config