# homehub — lean home-drive sync across 3–4 machines via Synology Status: **design, no code yet**. Open decisions in §13. ## 0. Ground truth this design was built against | Fact | Value | Consequence | |---|---|---| | Hub | `nas.lue.dk`, DSM, Linux 4.4.302+ x86_64, `/bin/sh` | rsync-over-ssh, no daemon, no Syncthing package | | Hub target | `/volume1/homes/bl` (4.9 TB free) | **cannot be the sync root — see next row** | | Hub root | `/volume1/homes/bl` already holds **178 G of unrelated legacy data** (`Documents`, `Photos`, `OneDrive`, `CloudStation`, `home_backup`, `x1c_backup`, `wg-bo.conf`, `certs`, `@eaDir`) | sync root is a dedicated subtree `$HUB_BASE/homehub`; `check_hub_stray()` refuses to sync into a root that is not ours | | NAS rsync | **3.1.2 (protocol 31)** | **no `--ignore-older`** → use `--update` both ways | | Client rsync | 3.5.1 | protocol downgrade fine; do not use ≥3.2-only flags | | Hub perms | siblings `admin, cl, fk, ida, …` on same share | `--no-perms --no-owner --no-group --chmod=Du=rwx,Dgo=,Fu=rw,Fgo=` | | Laptop offline | weeks | no `--delete`; NAT-friendly health check (hub-side markers only) | | Client ssh | `id_rsa` has a passphrase; `SSH_AUTH_SOCK` comes from a running agent | a **passphrase-free dedicated key** is mandatory (a user unit has no agent) and rsync gets the same identity via `-e` | | NixOS | not started (`/nix` absent) | design must be identical pre/post migration → §8 | Size reality check: `~` ≈ 250 GB, of which file-sync-worthy personal content is **~0.5 GB** (`Documents` 71 M, `Pictures` 323 M, `bin` 45 M, `journal` 72 K, dotfiles, `~/.pi/agent/sessions` 11 M). `~/git` 139 G + `~/src` 21 G are *repos*, not files, and 7.2 G of `~/src` alone is `build/`+`target/`+`dist/`. ## 1. Four data classes, four mechanisms | Class | What | Mechanism | Direction | |---|---|---|---| | A | personal files (docs, pictures, bin, dotfiles, tool *state*) | rsync via hub, mtime-arbitrated | two-way | | B | git repos (`~/git/*`, repo-like dirs in `~/src/*`) | **manifest + clone** (`repos.tsv`) | two-way, metadata only | | C | Chromium bookmark bar | per-host export + **union-by-URL merge** | many-to-one-to-many | | D | everything else (caches, toolchains, builds, `tmp`, `Downloads`) | **default deny** | never | Class B is the load-bearing idea. "Copy only the directory and the remote; each machine has its own state" is not a file-sync requirement — it is a manifest. A `repos.tsv` line is ~120 bytes where the repo is 4 GB, dirty state and local branches never leave the machine, and 130 GB of vendored upstream clones (`esp-idf`, `poky`, `bitbake`, `zephyr-sdk`, `llama.cpp`, `ollama`) become one line each instead of a 139 GB transfer. ## 2. Hub layout ``` /volume1/homes/bl/homehub/ # dedicated sync root — NOT the Synology home .hub/ meta/ markers/ # epoch of last SUCCESS (hub clock: no client skew) alive/ # epoch of last ATTEMPT (written even on failure) alerted/ # epoch of last alert (alert de-duplication) log.tsv # one line per attempt: epoch, host, rc, note tombstones/ # epoch \t path — paths that host deleted gitman/repos. # class B manifest bookmarks/.json # exported bookmark bar per host prev/YYYY-MM/ # overwritten older versions; prev/reaped/ for reaped # Documents/, Pictures/, bin/, .ssh/, .pi/, ... ``` **The sync root must be a dedicated subtree.** `/volume1/homes/bl` is 178 G of pre-existing, unrelated data. Pointing the tool at it would have (a) merged the NAS's `Documents/` into `~/Documents` on the very first pull, (b) written our dotfiles over the NAS's own `.config`, and (c) let the tombstone reaper move unrelated legacy files into `.hub/prev/` whenever a local delete happened to match a legacy path. `check_hub_stray()` now refuses to `sync --exec` into a hub root containing anything that is not ours. Nothing on the hub is ever edited by hand except `.hub/bookmarks/*` if you must. `.hub/` is excluded from every pull and every push, so it can never leak into a home tree or be deleted by one. ## 3. Sync algorithm (one run, on one machine, ~10 min cadence) ``` 1 touch .hub/alive/ # "I am up" — even if the sync then fails 2 PULL tombstones from all hosts # apply locally: rm -rf within active sets 3 PULL rsync -rltDl -z --update # hub -> local, mtime-arbitrated, additive 4 PUSH rsync --update --backup # local -> hub, older hub files parked in .hub/prev/ 5 write .hub/markers/ # ONLY on rc=0 ``` Ordering note: pull-before-push means a file you edited locally while the hub had a newer version is preserved by `--update` (skipped) *and* the hub copy is kept in `.hub/prev/` — you never silently lose the other machine's work. Command shape (pass 4, local → hub): ``` rsync -rltDl -z --update \ --no-perms --no-owner --no-group \ --chmod=Du=rwx,Dgo=,Fu=rw,Fgo= \ --backup --backup-dir="$HUB/.hub/prev/$(date +%Y-%m)" \ --filter="merge $STATE/effective.filter" \ ~/ nas.lue.dk:/volume1/homes/bl/ ``` Semantics, stated plainly so it is not a surprise later: **edits always win; deletions only win if nobody else touched the file.** ## 4. Which files: sets + host membership (default deny) `sets.d/.include` holds rsync filter rules (`+ /Documents/***`, …). `hosts/` lists which sets that machine participates in. `effective.filter` is generated per host per run: ``` - /.hub/*** - <§9 denylist rules, emitted first so they beat set includes> + //*** # union of this host's sets, in stable order - * # default deny: anything not explicitly included never moves ``` First-match-wins, so the `-` global denylist must be emitted **before** `+` lines that would otherwise match (e.g. `+ /src/***` vs `- /src/**/build/***`). Proposed initial sets (tick/adjust in §13): | Set | Members | Desktops | Laptop | |---|---|---|---| | `core` | `.ssh/`, `.gitconfig`, `.bashrc`, `.profile`, `.p10k.zsh`, `bin/`, `Documents/`, `Pictures/`, `journal/`, `.local/state/homehub/`, `.pi/agent/{sessions,extensions,settings.json,models.json,AGENTS.md}`, `.gtk-bookmarks` | yes | yes | | `repos` | `.hub`-side `gitman/repos.tsv` (Class B metadata only) | yes | yes | | `work` | non-repo remainder of `~/src` (see §13 D2) | yes | opt-in | | `bulk` | non-repo leftovers in `rustlings/`, `rustql/` | yes | no | `~/Downloads` (35 G) and `~/tmp` are out, as decided. `~/.config/**` is **not** in any set — see §8. ## 5. Deletion policy: tombstones + reaper, never `--delete` `--delete` is incompatible with a weeks-offline peer: machine A, back from a 3-week trip, would delete from the hub files that another machine created or edited while it was away, and the offline machine would then pull that deletion. Instead: - Push pass 1 is a `--dry-run --delete --itemize-changes` scan; `deleting` lines are appended to `.hub/tombstones/` (cost: one extra metadata-only scan). - Real push is additive. Tombstones are applied locally on every pull (step 2). - A **reaper** (runs only when a tombstone is older than the grace window, §13) moves the hub copy to `.hub/prev/reaped/` and prunes the tombstone line, so a file deleted in spring does not resurrect on a laptop that returns in autumn. - `homehub forget ` deletes tombstones when you want a file back for good. Why not `restic`? `--backup --backup-dir=.hub/prev/YYYY-MM` gives one prior generation for free on every overwrite, needs no new tool on DSM 7/kernel 4.4, and 4.9 TB free makes "keep the last version" affordable. Check whether Snapshot Replication covers the `homes` share — `/volume1/@snapshots` was not readable as `bl`; if it does, you get coarse history for free and should enable it. ## 6. Class B — git manifest spec `gitman/inventory` walks `$HOME` over `git/` and `src/`, **stops descending at the first `.git`** (so a repo's own submodules/modules are not double-counted), and writes `.hub/gitman/repos..tsv`: ``` relpath remote_url head_branch head_sha dirty unpushed src/rustql https://… main 9f2c1ab 0 3 git/ev_fw git@…:…/ev_fw main 41aa0c2 12 0 ``` - `remote_url` is `git remote get-url origin`; that plus the path is the only thing that crosses machines, exactly as asked. - `dirty` / `unpushed` are **advisory only** — read by `homehub status`, never synced as content. They answer "why does `ev_fw` look different on the laptop" without turning rsync into a bad git. - `homehub git missing` = union(all hosts' tsv) − repos present locally. `homehub git clone-missing` prints commands; `--exec` runs them; `--lazy` adds `--filter=blob:none` for the multi-GB vendored clones. - Never file-sync any path that the inventory recognises as a repo. No `.git` exclusion weirdness, no `build/` 7.2 GB problem: repo trees are cloned, not rsynced. - Caveat to accept: unpushed commits do **not** survive disk death (you declined a bare mirror on the hub — fine for personal use, just know it is the trade). - `~/src/zephyrproject` is a `west` workspace: inventory records the top repo and its sub-repos get recreated by `west update`, not by us. ## 7. Class C — bookmark bar Live file: `~/.config/chromium/Profile 5/Bookmarks` (mtime 2026-10-24 — this is the live profile; `~/.config/google-chrome/Default/Bookmarks` is dead, from 2021). - **Read** the live file, copy to `.hub/bookmarks/.json` (safe: read-only). - **Merge** all hosts' files: union of `bookmark_bar` children keyed by URL, preserving each host's folder nesting, dedupe identical URL+title. Idempotent, so no conflict concept is needed for bookmarks. - Write the merge back to the live path **only when the browser is not running** (Chromium holds the file in memory and re-serialises it constantly; writing while it runs is how bookmarks silently vanish). If it is running, defer to next run. - Keep the last N exports on the hub, so a bad merge is always reversible. - Boring alternative, if you ever stop caring about "only the bar": Chromium Sync. ## 8. Class D and the NixOS migration (the conflict in your answer 5) Stop treating `~/.config` as one thing. Split by ownership, not by path: | Kind | Owner | Where it physically lives | Synced? | |---|---|---|---| | **declarative config** | Nix / home-manager (later: a git repo) | nix store, symlinked into `~/.config` | **no** | | **state** (session history, caches-of-record, per-app data you want everywhere) | you | `~/.local/state/homehub//` | **yes** | | **cache / binaries** | tools | `~/.cache`, `~/.local/share/…`, `.pi/agent/install` | **no** | The sync set is defined **only by paths under the shared state root** plus an explicit top-level list. Two roots, and the difference between them is not cosmetic: | Root | Contents | Synced? | |---|---|---| | `~/.local/state/shared//` | **your** app state, symlinked to wherever each app expects it | **yes** | | `~/.local/state/homehub/` | homehub's own scratch: generated filters, markers, `pull.err`, tombstone cache, `broken.*` | **no** (denied) | Conflating them is a real bug I hit while testing: with the state root itself in the `base` set, g9 pushed `effective.filter`, the laptop overwrote it with its own, and they would have swapped generated filters forever. `~/.local/state/homehub/` is now in the denylist so it cannot come back by accident. On Debian you create the symlink by hand (or stow); on NixOS home-manager's `file.` / `xdg.config."…".source` creates the *same* symlink into the *same* synced path. So the migration rewrites the symlink layer and touches zero sync rules — that is the resolution of "I like dotfiles locally, but I also want pi history synced": **the sync tree owns state, Nix owns config, and neither writes into `~/.config` directly.** Concrete for pi agent, from this box: sync `sessions/` (11 M, 96 `.jsonl` files — append-only, so rsync-safe) and `settings.json`, `models.json`, `AGENTS.md`, `extensions/`, hand-written `skills/*` trees; exclude `install/` (154 M), `bin/` (9 M), `npm/`, and any `node_modules` under `skills/`. No sqlite/`-wal` files exist in `~/.pi`, which is why plain rsync is safe here — if a tool ever stores state in sqlite, it goes in the never-sync list or gets a dump-first hook. ## 9. Never-sync denylist (derived from actual `du`, not guessed) `tmp/` (11 G) · `Downloads/` (35 G, decided) · `git/` and repo trees (Class B) · `.cache/` 12 G · `.steam/` 25 G · `.wine/` 14 G, `.wine_games/` 8.5 G, `.wine_firefly/` 2.2 G · `zephyr-sdk-0.17.4/` 12 G · `go/` 8.5 G · `.espressif/` 6 G · `.npm/` 4.8 G · `.rustup/` 4.5 G · `.cargo/` 982 M · `.local/share/Trash/` · `.vscode/` 1.7 G, `.config/{google-chrome,discord,chromium,Code,sublime-text,VSCodium,Atom}` 11 G total (local by default; only the bookmark file crosses machines, via §7) · `**/{build,target,dist,node_modules,__pycache__,.venv,.pixi,.zig-cache}/` · `.git/` (belt-and-braces alongside §6) · `.bash_history-*.tmp` (four 0-byte junk files from a broken history append — also worth fixing at the source) · `.gksu.lock`, `*.tmp`, `*.bak`, `*.backup`. ## 10. "Only tell me when it has been broken for a day" This is a **freshness** problem, not an error-notification problem, so build a freshness monitor, not an error notifier. - Sync units are silent: `homehub-sync.service` prints nothing on success, `LogLevelMax=warning`. Failures produce no output and no popup; they simply fail to update `markers/`. No popups, no retry storms. - Health unit: `homehub-fresh.timer`, `OnUnitActiveSec=30m`, runs `homehub fresh`, which reads hub markers over one ssh call — **no inbound connectivity needed**, so it works for the NATted laptop. - Alert rule: `alive/` is fresh (machine is on) **and** `markers/` is older than **24 h** **and** the last attempt's rc is *not* a network-class rc. Network-class rc (5, 10, 11, 12, 30, 35, 255 — unreachable/timeout/connection) is the weeks-offline laptop and stays **silent forever**; it only shows in `homehub status`. That distinction is what keeps your requirement #2 (offline for weeks) true without ever popping a notification. - De-duplication: after alerting, stamp `alerted/`; re-alert at most weekly while still broken, clear on success. So one mail per incident, not 48. - Transport: `sendmail` → your existing `mail.lue.dk`. Works with no GUI session, which `notify-send` does not. ## 11. Files to be created (this repo = the policy; ~200 lines of shell total) ``` ~/homehub/ # small git repo, hub-backed via a bare repo on the NAS DESIGN.md README.md bin/homehub # subcommands: sync, status, fresh, forget, git, bookmarks, filter, setup, install lib/merge_bookmarks.py # class C union-merge config # per-machine: hub path, key, thresholds, bookmark source rules/denylist # §9 — DEFAULT DENY, emitted first rules/sets.d/base.list # every machine (laptop included) rules/sets.d/docs.list media.list misc.list rules/hosts/g9 # -> sets: base docs media misc rules/hosts/_laptop-template # -> sets: base docs systemd/homehub-sync.{service,timer} systemd/homehub-fresh.{service,timer} ``` Unit details that matter: `OnUnitActiveSec=10min RandomizedDelaySec=3min Persistent=true` (catches up after suspend/off), `After=network-online.target Conflicts=sleep.target`, `LogLevelMax=warning` + zero stdout on success, and a **passphrase-less dedicated ed25519 deploy key** (`~/.ssh/homehub_ed25519`, `IdentitiesOnly yes`) rather than `id_rsa` — a user unit has no `SSH_AUTH_SOCK` (yours is a socket at `~/.ssh/agent/…` from a running agent), so depending on the agent makes the background sync fail in exactly the way you would notice 26 hours later. Optionally restrict the hub key with `from=` and a forced `command="rsync --server…"`. ## 12. Rollout order 1. Hub: `mkdir -p .hub/{meta/{markers,alive,alerted},tombstones,bookmarks,prev,gitman}`, deploy key. 2. Write rules + `homehub sync --dry` on `g9`; inspect the file list by hand (this is the only real review step). 3. First real `core` sync from `g9`; verify `700/600` perms on the hub and that `.hub/` is absent locally. 4. `gitman/inventory` from `g9`; on machine #2 run `homehub git clone-missing --lazy`. 5. Machine #2 file sync, then watch convergence both ways. 6. Bookmark export+merge. 7. **Health monitor last** — it is only trustworthy once the sync semantics above have survived a week. 8. NixOS migration later: re-point the symlink layer only (§8); rules unchanged. ## 13. Decisions taken (was: open decisions) | # | Decision | Outcome | |---|---|---| | D1 | Secrets in the sync set | **yes** — `.ssh/` incl. private keys, and `auth.json` if you add it. The hub now holds every key you own: it must stay LAN-only, and the hub copy is forced `700/600`. | | D2 | `~/src` | **manifest only.** No file sync of repo trees, so the 7.2 G of `build/`+`target/`+`dist/` and every `.git` ambiguity disappeared. | | D3 | Reaper grace | **30 days.** | | D4 | Laptop membership | **`base` + `docs`.** (`core` was renamed `base`; `Documents`/`Pictures` were split into `docs`/`media` so the question was even well-posed — my original set list had `Documents` inside `core` *and* asked whether to include it.) | | D5 | Alert transport | no local MTA yet → auto-detect `msmtp`/`sendmail`, else spool to `~/.local/state/homehub/alerts.out`. msmtp config for `mail.lue.dk:587` is in README. | ### Found while implementing (each one changed the design) 1. **`/volume1/homes/bl` is not empty** — 178 G of unrelated legacy data. Sync root is now the dedicated subtree `$HUB_BASE/homehub`, plus a stray-content guard (§2). 2. **A live GitLab PAT sat inside a repo remote URL** (`https://user:glpat-…@gitlab.com/…`, 3 repos). The manifest would have published it to the NAS. Remotes are now scrubbed of userinfo and the row is flagged `stripped`, which `git clone` reports. *Separately: rotate that token, it has been sitting in `.git/config` on every machine that cloned those repos.* 3. **`id_rsa` is passphrase-protected** and `SSH_AUTH_SOCK` comes from a live agent, so a systemd user unit cannot authenticate. Fixed by a dedicated passphrase-free ed25519 key — and, more importantly, by splitting *auth failure* (loud, after 24 h) from *network failure* (silent forever) in `hub_probe()`. Before that split, a dead key produced no marker, no heartbeat, and therefore no alert, ever. 4. **Synology writes `@eaDir` into every directory** — now denied, or a pull litters an `@eaDir` next to every synced directory. 5. **`~/debian` is a dpkg work area** (150 M, 3598 files = 86 % of the first dry-run push list). Denied as build output, like `~/tmp` and `~/Downloads`. 6. **rsync was not being told which key to use** (no `-e`), so it would have authenticated as a different identity from the rest of the tool. ### Still open * Rotate the GitLab PAT mentioned above; consider `id_rsa` (2016, 4096-bit RSA) too. * Reconcile the legacy `home_backup` / `x1c_backup` trees on the NAS — they look like earlier attempts at exactly this problem. * Whether `~/bin` should sync at all: it is your scripts *plus* vendored binaries (`protoc` 9 M, five gRPC plugins), 45 M of mostly machine-local bytes. * `~/public_html` symlinks into `~/src`, `~/git` and `/tmp`; they will dangle on a machine that has not cloned the target.