From 78496330f38d8c1e71694509bc5986c5748d0ddb Mon Sep 17 00:00:00 2001 From: Bo Lorentsen Date: Sat, 24 Oct 2026 16:36:40 +0200 Subject: [PATCH] homehub: rsync-based home-drive sync via Synology hub DESIGN.md is the rationale; README.md the operations half. Four data classes: files (mtime-arbitrated two-way), git repos (manifest + clone, never byte-synced), the browser link bar (union merge), and everything else denied by default. --- DESIGN.md | 336 +++++++++++++++++ README.md | 117 ++++++ bin/homehub | 664 ++++++++++++++++++++++++++++++++++ config | 64 ++++ lib/merge_bookmarks.py | 142 ++++++++ rules/denylist | 134 +++++++ rules/hosts/_laptop-template | 5 + rules/hosts/g9 | 6 + rules/sets.d/base.list | 42 +++ rules/sets.d/docs.list | 2 + rules/sets.d/media.list | 2 + rules/sets.d/misc.list | 7 + systemd/homehub-fresh.service | 12 + systemd/homehub-fresh.timer | 12 + systemd/homehub-sync.service | 19 + systemd/homehub-sync.timer | 12 + 16 files changed, 1576 insertions(+) create mode 100644 DESIGN.md create mode 100644 README.md create mode 100755 bin/homehub create mode 100644 config create mode 100755 lib/merge_bookmarks.py create mode 100644 rules/denylist create mode 100644 rules/hosts/_laptop-template create mode 100644 rules/hosts/g9 create mode 100644 rules/sets.d/base.list create mode 100644 rules/sets.d/docs.list create mode 100644 rules/sets.d/media.list create mode 100644 rules/sets.d/misc.list create mode 100644 systemd/homehub-fresh.service create mode 100644 systemd/homehub-fresh.timer create mode 100644 systemd/homehub-sync.service create mode 100644 systemd/homehub-sync.timer diff --git a/DESIGN.md b/DESIGN.md new file mode 100644 index 0000000..d8f52b3 --- /dev/null +++ b/DESIGN.md @@ -0,0 +1,336 @@ +# homehub — lean home-drive sync across 3–4 machines via Synology + +Status: **design, no code yet**. Open decisions in §13. + +## 0. Ground truth this design was built against + +| Fact | Value | Consequence | +|---|---|---| +| Hub | `nas.lue.dk`, DSM, Linux 4.4.302+ x86_64, `/bin/sh` | rsync-over-ssh, no daemon, no Syncthing package | +| Hub target | `/volume1/homes/bl` (4.9 TB free) | **cannot be the sync root — see next row** | +| Hub root | `/volume1/homes/bl` already holds **178 G of unrelated legacy data** (`Documents`, `Photos`, `OneDrive`, `CloudStation`, `home_backup`, `x1c_backup`, `wg-bo.conf`, `certs`, `@eaDir`) | sync root is a dedicated subtree `$HUB_BASE/homehub`; `check_hub_stray()` refuses to sync into a root that is not ours | +| NAS rsync | **3.1.2 (protocol 31)** | **no `--ignore-older`** → use `--update` both ways | +| Client rsync | 3.5.1 | protocol downgrade fine; do not use ≥3.2-only flags | +| Hub perms | siblings `admin, cl, fk, ida, …` on same share | `--no-perms --no-owner --no-group --chmod=Du=rwx,Dgo=,Fu=rw,Fgo=` | +| Laptop offline | weeks | no `--delete`; NAT-friendly health check (hub-side markers only) | +| Client ssh | `id_rsa` has a passphrase; `SSH_AUTH_SOCK` comes from a running agent | a **passphrase-free dedicated key** is mandatory (a user unit has no agent) and rsync gets the same identity via `-e` | +| NixOS | not started (`/nix` absent) | design must be identical pre/post migration → §8 | + +Size reality check: `~` ≈ 250 GB, of which file-sync-worthy personal content is +**~0.5 GB** (`Documents` 71 M, `Pictures` 323 M, `bin` 45 M, `journal` 72 K, +dotfiles, `~/.pi/agent/sessions` 11 M). `~/git` 139 G + `~/src` 21 G are *repos*, +not files, and 7.2 G of `~/src` alone is `build/`+`target/`+`dist/`. + +## 1. Four data classes, four mechanisms + +| Class | What | Mechanism | Direction | +|---|---|---|---| +| A | personal files (docs, pictures, bin, dotfiles, tool *state*) | rsync via hub, mtime-arbitrated | two-way | +| B | git repos (`~/git/*`, repo-like dirs in `~/src/*`) | **manifest + clone** (`repos.tsv`) | two-way, metadata only | +| C | Chromium bookmark bar | per-host export + **union-by-URL merge** | many-to-one-to-many | +| D | everything else (caches, toolchains, builds, `tmp`, `Downloads`) | **default deny** | never | + +Class B is the load-bearing idea. "Copy only the directory and the remote; each +machine has its own state" is not a file-sync requirement — it is a manifest. A +`repos.tsv` line is ~120 bytes where the repo is 4 GB, dirty state and local +branches never leave the machine, and 130 GB of vendored upstream clones +(`esp-idf`, `poky`, `bitbake`, `zephyr-sdk`, `llama.cpp`, `ollama`) become one +line each instead of a 139 GB transfer. + +## 2. Hub layout + +``` +/volume1/homes/bl/homehub/ # dedicated sync root — NOT the Synology home + .hub/ + meta/ + markers/ # epoch of last SUCCESS (hub clock: no client skew) + alive/ # epoch of last ATTEMPT (written even on failure) + alerted/ # epoch of last alert (alert de-duplication) + log.tsv # one line per attempt: epoch, host, rc, note + tombstones/ # epoch \t path — paths that host deleted + gitman/repos. # class B manifest + bookmarks/.json # exported bookmark bar per host + prev/YYYY-MM/ # overwritten older versions; prev/reaped/ for reaped + # Documents/, Pictures/, bin/, .ssh/, .pi/, ... +``` + +**The sync root must be a dedicated subtree.** `/volume1/homes/bl` is 178 G of +pre-existing, unrelated data. Pointing the tool at it would have (a) merged the +NAS's `Documents/` into `~/Documents` on the very first pull, (b) written our +dotfiles over the NAS's own `.config`, and (c) let the tombstone reaper move +unrelated legacy files into `.hub/prev/` whenever a local delete happened to match +a legacy path. `check_hub_stray()` now refuses to `sync --exec` into a hub root +containing anything that is not ours. + +Nothing on the hub is ever edited by hand except `.hub/bookmarks/*` if you must. +`.hub/` is excluded from every pull and every push, so it can never leak into a +home tree or be deleted by one. + +## 3. Sync algorithm (one run, on one machine, ~10 min cadence) + +``` +1 touch .hub/alive/ # "I am up" — even if the sync then fails +2 PULL tombstones from all hosts # apply locally: rm -rf within active sets +3 PULL rsync -rltDl -z --update # hub -> local, mtime-arbitrated, additive +4 PUSH rsync --update --backup # local -> hub, older hub files parked in .hub/prev/ +5 write .hub/markers/ # ONLY on rc=0 +``` + +Ordering note: pull-before-push means a file you edited locally while the hub had +a newer version is preserved by `--update` (skipped) *and* the hub copy is kept in +`.hub/prev/` — you never silently lose the other machine's work. + +Command shape (pass 4, local → hub): + +``` +rsync -rltDl -z --update \ + --no-perms --no-owner --no-group \ + --chmod=Du=rwx,Dgo=,Fu=rw,Fgo= \ + --backup --backup-dir="$HUB/.hub/prev/$(date +%Y-%m)" \ + --filter="merge $STATE/effective.filter" \ + ~/ nas.lue.dk:/volume1/homes/bl/ +``` + +Semantics, stated plainly so it is not a surprise later: +**edits always win; deletions only win if nobody else touched the file.** + +## 4. Which files: sets + host membership (default deny) + +`sets.d/.include` holds rsync filter rules (`+ /Documents/***`, …). +`hosts/` lists which sets that machine participates in. +`effective.filter` is generated per host per run: + +``` +- /.hub/*** +- <§9 denylist rules, emitted first so they beat set includes> ++ //*** # union of this host's sets, in stable order +- * # default deny: anything not explicitly included never moves +``` + +First-match-wins, so the `-` global denylist must be emitted **before** `+` lines +that would otherwise match (e.g. `+ /src/***` vs `- /src/**/build/***`). + +Proposed initial sets (tick/adjust in §13): + +| Set | Members | Desktops | Laptop | +|---|---|---|---| +| `core` | `.ssh/`, `.gitconfig`, `.bashrc`, `.profile`, `.p10k.zsh`, `bin/`, `Documents/`, `Pictures/`, `journal/`, `.local/state/homehub/`, `.pi/agent/{sessions,extensions,settings.json,models.json,AGENTS.md}`, `.gtk-bookmarks` | yes | yes | +| `repos` | `.hub`-side `gitman/repos.tsv` (Class B metadata only) | yes | yes | +| `work` | non-repo remainder of `~/src` (see §13 D2) | yes | opt-in | +| `bulk` | non-repo leftovers in `rustlings/`, `rustql/` | yes | no | + +`~/Downloads` (35 G) and `~/tmp` are out, as decided. `~/.config/**` is **not** in +any set — see §8. + +## 5. Deletion policy: tombstones + reaper, never `--delete` + +`--delete` is incompatible with a weeks-offline peer: machine A, back from a +3-week trip, would delete from the hub files that another machine created or +edited while it was away, and the offline machine would then pull that deletion. + +Instead: + +- Push pass 1 is a `--dry-run --delete --itemize-changes` scan; `deleting` lines + are appended to `.hub/tombstones/` (cost: one extra metadata-only scan). +- Real push is additive. Tombstones are applied locally on every pull (step 2). +- A **reaper** (runs only when a tombstone is older than the grace window, §13) + moves the hub copy to `.hub/prev/reaped/` and prunes the tombstone line, so a + file deleted in spring does not resurrect on a laptop that returns in autumn. +- `homehub forget ` deletes tombstones when you want a file back for good. + +Why not `restic`? `--backup --backup-dir=.hub/prev/YYYY-MM` gives one prior +generation for free on every overwrite, needs no new tool on DSM 7/kernel 4.4, +and 4.9 TB free makes "keep the last version" affordable. Check whether Snapshot +Replication covers the `homes` share — `/volume1/@snapshots` was not readable as +`bl`; if it does, you get coarse history for free and should enable it. + +## 6. Class B — git manifest spec + +`gitman/inventory` walks `$HOME` over `git/` and `src/`, **stops descending at the +first `.git`** (so a repo's own submodules/modules are not double-counted), and +writes `.hub/gitman/repos..tsv`: + +``` +relpath remote_url head_branch head_sha dirty unpushed +src/rustql https://… main 9f2c1ab 0 3 +git/ev_fw git@…:…/ev_fw main 41aa0c2 12 0 +``` + +- `remote_url` is `git remote get-url origin`; that plus the path is the only thing + that crosses machines, exactly as asked. +- `dirty` / `unpushed` are **advisory only** — read by `homehub status`, never + synced as content. They answer "why does `ev_fw` look different on the laptop" + without turning rsync into a bad git. +- `homehub git missing` = union(all hosts' tsv) − repos present locally. + `homehub git clone-missing` prints commands; `--exec` runs them; + `--lazy` adds `--filter=blob:none` for the multi-GB vendored clones. +- Never file-sync any path that the inventory recognises as a repo. No `.git` + exclusion weirdness, no `build/` 7.2 GB problem: repo trees are cloned, not rsynced. +- Caveat to accept: unpushed commits do **not** survive disk death (you declined a + bare mirror on the hub — fine for personal use, just know it is the trade). +- `~/src/zephyrproject` is a `west` workspace: inventory records the top repo and + its sub-repos get recreated by `west update`, not by us. + +## 7. Class C — bookmark bar + +Live file: `~/.config/chromium/Profile 5/Bookmarks` (mtime 2026-10-24 — this is the +live profile; `~/.config/google-chrome/Default/Bookmarks` is dead, from 2021). + +- **Read** the live file, copy to `.hub/bookmarks/.json` (safe: read-only). +- **Merge** all hosts' files: union of `bookmark_bar` children keyed by URL, + preserving each host's folder nesting, dedupe identical URL+title. Idempotent, + so no conflict concept is needed for bookmarks. +- Write the merge back to the live path **only when the browser is not running** + (Chromium holds the file in memory and re-serialises it constantly; writing while + it runs is how bookmarks silently vanish). If it is running, defer to next run. +- Keep the last N exports on the hub, so a bad merge is always reversible. +- Boring alternative, if you ever stop caring about "only the bar": Chromium Sync. + +## 8. Class D and the NixOS migration (the conflict in your answer 5) + +Stop treating `~/.config` as one thing. Split by ownership, not by path: + +| Kind | Owner | Where it physically lives | Synced? | +|---|---|---|---| +| **declarative config** | Nix / home-manager (later: a git repo) | nix store, symlinked into `~/.config` | **no** | +| **state** (session history, caches-of-record, per-app data you want everywhere) | you | `~/.local/state/homehub//` | **yes** | +| **cache / binaries** | tools | `~/.cache`, `~/.local/share/…`, `.pi/agent/install` | **no** | + +The sync set is defined **only by paths under the shared state root** plus an explicit +top-level list. Two roots, and the difference between them is not cosmetic: + +| Root | Contents | Synced? | +|---|---|---| +| `~/.local/state/shared//` | **your** app state, symlinked to wherever each app expects it | **yes** | +| `~/.local/state/homehub/` | homehub's own scratch: generated filters, markers, `pull.err`, tombstone cache, `broken.*` | **no** (denied) | + +Conflating them is a real bug I hit while testing: with the state root itself in the +`base` set, g9 pushed `effective.filter`, the laptop overwrote it with its own, and +they would have swapped generated filters forever. `~/.local/state/homehub/` is now +in the denylist so it cannot come back by accident. + +On Debian you create the symlink by hand (or stow); on NixOS home-manager's `file.` / +`xdg.config."…".source` creates the *same* symlink into the *same* synced path. So +the migration rewrites the symlink layer and touches zero sync rules — that is the +resolution of "I like dotfiles locally, but I also want pi history synced": **the +sync tree owns state, Nix owns config, and neither writes into `~/.config` directly.** + +Concrete for pi agent, from this box: sync `sessions/` (11 M, 96 `.jsonl` files — +append-only, so rsync-safe) and `settings.json`, `models.json`, `AGENTS.md`, +`extensions/`, hand-written `skills/*` trees; exclude `install/` (154 M), +`bin/` (9 M), `npm/`, and any `node_modules` under `skills/`. No sqlite/`-wal` +files exist in `~/.pi`, which is why plain rsync is safe here — if a tool ever +stores state in sqlite, it goes in the never-sync list or gets a dump-first hook. + +## 9. Never-sync denylist (derived from actual `du`, not guessed) + +`tmp/` (11 G) · `Downloads/` (35 G, decided) · `git/` and repo trees (Class B) · +`.cache/` 12 G · `.steam/` 25 G · `.wine/` 14 G, `.wine_games/` 8.5 G, +`.wine_firefly/` 2.2 G · `zephyr-sdk-0.17.4/` 12 G · `go/` 8.5 G · +`.espressif/` 6 G · `.npm/` 4.8 G · `.rustup/` 4.5 G · `.cargo/` 982 M · +`.local/share/Trash/` · `.vscode/` 1.7 G, `.config/{google-chrome,discord,chromium,Code,sublime-text,VSCodium,Atom}` 11 G total (local by +default; only the bookmark file crosses machines, via §7) · `**/{build,target,dist,node_modules,__pycache__,.venv,.pixi,.zig-cache}/` · +`.git/` (belt-and-braces alongside §6) · `.bash_history-*.tmp` (four 0-byte junk +files from a broken history append — also worth fixing at the source) · +`.gksu.lock`, `*.tmp`, `*.bak`, `*.backup`. + +## 10. "Only tell me when it has been broken for a day" + +This is a **freshness** problem, not an error-notification problem, so build a +freshness monitor, not an error notifier. + +- Sync units are silent: `homehub-sync.service` prints nothing on success, + `LogLevelMax=warning`. Failures produce no output and no popup; they simply fail + to update `markers/`. No popups, no retry storms. +- Health unit: `homehub-fresh.timer`, `OnUnitActiveSec=30m`, runs `homehub fresh`, + which reads hub markers over one ssh call — **no inbound connectivity needed**, + so it works for the NATted laptop. +- Alert rule: `alive/` is fresh (machine is on) **and** `markers/` is + older than **24 h** **and** the last attempt's rc is *not* a network-class rc. + Network-class rc (5, 10, 11, 12, 30, 35, 255 — unreachable/timeout/connection) + is the weeks-offline laptop and stays **silent forever**; it only shows in + `homehub status`. That distinction is what keeps your requirement #2 (offline + for weeks) true without ever popping a notification. +- De-duplication: after alerting, stamp `alerted/`; re-alert at most weekly + while still broken, clear on success. So one mail per incident, not 48. +- Transport: `sendmail` → your existing `mail.lue.dk`. Works with no GUI session, + which `notify-send` does not. + +## 11. Files to be created (this repo = the policy; ~200 lines of shell total) + +``` +~/homehub/ # small git repo, hub-backed via a bare repo on the NAS + DESIGN.md README.md + bin/homehub # subcommands: sync, status, fresh, forget, git, bookmarks, filter, setup, install + lib/merge_bookmarks.py # class C union-merge + config # per-machine: hub path, key, thresholds, bookmark source + rules/denylist # §9 — DEFAULT DENY, emitted first + rules/sets.d/base.list # every machine (laptop included) + rules/sets.d/docs.list media.list misc.list + rules/hosts/g9 # -> sets: base docs media misc + rules/hosts/_laptop-template # -> sets: base docs + systemd/homehub-sync.{service,timer} + systemd/homehub-fresh.{service,timer} +``` + +Unit details that matter: `OnUnitActiveSec=10min RandomizedDelaySec=3min +Persistent=true` (catches up after suspend/off), `After=network-online.target +Conflicts=sleep.target`, `LogLevelMax=warning` + zero stdout on success, and a +**passphrase-less dedicated ed25519 deploy key** (`~/.ssh/homehub_ed25519`, +`IdentitiesOnly yes`) rather than `id_rsa` — a user unit has no `SSH_AUTH_SOCK` +(yours is a socket at `~/.ssh/agent/…` from a running agent), so depending on the +agent makes the background sync fail in exactly the way you would notice 26 hours +later. Optionally restrict the hub key with `from=` and a forced +`command="rsync --server…"`. + +## 12. Rollout order + +1. Hub: `mkdir -p .hub/{meta/{markers,alive,alerted},tombstones,bookmarks,prev,gitman}`, deploy key. +2. Write rules + `homehub sync --dry` on `g9`; inspect the file list by hand (this is the only real review step). +3. First real `core` sync from `g9`; verify `700/600` perms on the hub and that `.hub/` is absent locally. +4. `gitman/inventory` from `g9`; on machine #2 run `homehub git clone-missing --lazy`. +5. Machine #2 file sync, then watch convergence both ways. +6. Bookmark export+merge. +7. **Health monitor last** — it is only trustworthy once the sync semantics above have survived a week. +8. NixOS migration later: re-point the symlink layer only (§8); rules unchanged. + +## 13. Decisions taken (was: open decisions) + +| # | Decision | Outcome | +|---|---|---| +| D1 | Secrets in the sync set | **yes** — `.ssh/` incl. private keys, and `auth.json` if you add it. The hub now holds every key you own: it must stay LAN-only, and the hub copy is forced `700/600`. | +| D2 | `~/src` | **manifest only.** No file sync of repo trees, so the 7.2 G of `build/`+`target/`+`dist/` and every `.git` ambiguity disappeared. | +| D3 | Reaper grace | **30 days.** | +| D4 | Laptop membership | **`base` + `docs`.** (`core` was renamed `base`; `Documents`/`Pictures` were split into `docs`/`media` so the question was even well-posed — my original set list had `Documents` inside `core` *and* asked whether to include it.) | +| D5 | Alert transport | no local MTA yet → auto-detect `msmtp`/`sendmail`, else spool to `~/.local/state/homehub/alerts.out`. msmtp config for `mail.lue.dk:587` is in README. | + +### Found while implementing (each one changed the design) + +1. **`/volume1/homes/bl` is not empty** — 178 G of unrelated legacy data. Sync root + is now the dedicated subtree `$HUB_BASE/homehub`, plus a stray-content guard (§2). +2. **A live GitLab PAT sat inside a repo remote URL** (`https://user:glpat-…@gitlab.com/…`, + 3 repos). The manifest would have published it to the NAS. Remotes are now scrubbed + of userinfo and the row is flagged `stripped`, which `git clone` reports. + *Separately: rotate that token, it has been sitting in `.git/config` on every + machine that cloned those repos.* +3. **`id_rsa` is passphrase-protected** and `SSH_AUTH_SOCK` comes from a live agent, so + a systemd user unit cannot authenticate. Fixed by a dedicated passphrase-free + ed25519 key — and, more importantly, by splitting *auth failure* (loud, after 24 h) + from *network failure* (silent forever) in `hub_probe()`. Before that split, a dead + key produced no marker, no heartbeat, and therefore no alert, ever. +4. **Synology writes `@eaDir` into every directory** — now denied, or a pull litters + an `@eaDir` next to every synced directory. +5. **`~/debian` is a dpkg work area** (150 M, 3598 files = 86 % of the first dry-run + push list). Denied as build output, like `~/tmp` and `~/Downloads`. +6. **rsync was not being told which key to use** (no `-e`), so it would have + authenticated as a different identity from the rest of the tool. + +### Still open + +* Rotate the GitLab PAT mentioned above; consider `id_rsa` (2016, 4096-bit RSA) too. +* Reconcile the legacy `home_backup` / `x1c_backup` trees on the NAS — they look like + earlier attempts at exactly this problem. +* Whether `~/bin` should sync at all: it is your scripts *plus* vendored binaries + (`protoc` 9 M, five gRPC plugins), 45 M of mostly machine-local bytes. +* `~/public_html` symlinks into `~/src`, `~/git` and `/tmp`; they will dangle on a + machine that has not cloned the target. diff --git a/README.md b/README.md new file mode 100644 index 0000000..cc4c590 --- /dev/null +++ b/README.md @@ -0,0 +1,117 @@ +# homehub — operator notes + +Operational doc. **Why** each decision exists is in [`DESIGN.md`](DESIGN.md); this file +is only the operational half. + +``` +~/homehub/ + bin/homehub the tool (~450 lines of bash + one python merge) + config machine-specific settings (per-machine, not synced) + rules/denylist DEFAULT DENY, emitted first, wins over everything + rules/sets.d/*.list named path-lists ("sets") + rules/hosts/ which sets this machine joins + lib/merge_bookmarks.py class C union-merge (jq + python3 only) + systemd/homehub-*.{service,timer} + DESIGN.md rationale for everything below +``` + +## Install (per machine) + +```sh +git clone ~/homehub && cd ~/homehub +cp rules/hosts/_laptop-template rules/hosts/$(hostname -s) # pick your sets +./bin/homehub setup-key # once, see warning below +ssh-copy-id -i ~/.ssh/homehub_ed25519.pub nas.lue.dk # YOU run this +./bin/homehub setup +./bin/homehub sync # DRY: prints what would move, writes nothing +homehub sync --exec # first real run +sudo loginctl enable-linger $USER # headless boxes only +~/homehub/bin/homehub install # systemd user timers +``` + +`homehub install` copies the units into `~/.config/systemd/user` and enables +`homehub-sync.timer` (10 min) and `homehub-fresh.timer` (30 min). + +## Commands + +| Command | What | +|---|---| +| `sync [--exec]` | pull → apply deletions → harvest deletions → push. **Dry unless `--exec`.** | +| `status` | last sync per machine, sets, pending dry-run lists, repo manifest, bookmark exports | +| `fresh [--dry]` | the 24 h alert check; only printer of alerts | +| `git inventory\|list\|missing\|clone [--exec] [--lazy]` | class B | +| `bookmarks [--write]` | class C export + union merge | +| `forget ` | drop tombstones so a deleted path can return | +| `filter` | dump the generated rsync filter | +| `setup`, `setup-key`, `install` | one-time plumbing | + +## The rules that matter when editing config + +1. **Default deny.** `rules/denylist` is emitted first, then the host's set includes, + then `- *`. A path in a set that sits under a deny rule can never sync — the tool + warns (`shadowed: ...`) when you write one. +2. **Set syntax.** One home-relative path per line; trailing `/` = subtree. + `/Documents/` → `- /Documents/***`; `build/` (no leading slash) matches at any depth. +3. **Never add a git repo path to a set.** Repos are class B (`homehub git inventory`). +4. **`~/.config` is unreachable by design** (it is denied wholesale). State you want + shared goes under `~/.local/state/shared//`, with the real config path + symlinked to it. `~/.local/state/homehub/` is homehub's *private* scratch and is + denied too — syncing it would make machines overwrite each other's generated + filters forever. +5. **Deletions never propagate as `--delete`.** They become tombstones on the hub, + are reaped after `GRACE_DAYS`, and always lose to a competing edit. + +## Alerting + +`homehub fresh` alerts only when a machine is **on** (heartbeat < 3 h) but has not +synced for > 24 h, and never for network-class rsync/ssh exit codes — so a laptop +away for three weeks is silent indefinitely. Delivery auto-detects `msmtp`, then +`sendmail`, else appends to `~/.local/state/homehub/alerts.out`. To wire it to your +own mail server: + +```sh +# /etc/msmtprc (or ~/.config/msmtp/config) +defaults +tls on +tls_starttls on +account homehub +host mail.lue.dk +port 587 +auth on +user homehub@lue.dk +from homehub@lue.dk +password +``` + +Then `msmtp -a homehub you@lue.dk` is picked up automatically. On Debian: +`apt install msmtp msmtp-mta`. + +A machine whose **own** config is broken (ssh key revoked, hub disk full) records a +local marker in `~/.local/state/homehub/broken.*` and alerts about itself even with +the hub unreachable — otherwise a dead key would produce no heartbeat, no marker, +and therefore no alert, forever. + +## Known limits (deliberate, not bugs) + +* **mtime is the arbiter.** Two machines editing the same file while both offline + resolve by mtime, so clocks must be NTP-synced (`timedatectl` on Debian, `ntpd` on + NixOS). The overwritten copy is kept in `/.hub/prev/YYYY-MM/`. +* **rsync 3.1.2 on DSM 6/7** — no `--ignore-older`; the design uses `--update` only. +* `--checksum` is off, so a file edited to the same size in the same second as a + previous sync may be missed. Re-run `rsync -mc` by hand if you ever suspect it. +* Unpushed commits are advisory in the manifest and never transferred. A dead laptop + disk loses them. +* Empty bookmark folders do not survive the union merge (there is nothing to key on). +* Symlinks travel as symlinks (`-l`, never `-L`); ones pointing into `~/src`, `~/git` + or `/tmp` will dangle on a machine that has not cloned the target. + +## Troubleshooting + +```sh +~/homehub/bin/homehub status # ages, sets, manifest +journalctl --user -u homehub-sync.service -n 50 +cat ~/.local/state/homehub/{pull,push}.err +cat ~/.local/state/homehub/broken.* # self-detected breakage +ssh nas.lue.dk 'cat /volume1/homes/bl/homehub/.hub/log.tsv | tail' +~/homehub/bin/homehub filter | grep -n shadowed +``` diff --git a/bin/homehub b/bin/homehub new file mode 100755 index 0000000..f03aa26 --- /dev/null +++ b/bin/homehub @@ -0,0 +1,664 @@ +#!/usr/bin/env bash +# homehub — lean home-drive sync: rsync against a Synology hub, mtime-arbitrated, +# silent unless something has been broken for a day. +# +# sync [--exec] one pull+push cycle (DRY unless --exec) +# status what would move, and who synced when +# fresh [--dry] the only command allowed to complain (24 h rule) +# git class B: repos.tsv manifest, clone on the far side +# bookmarks [--write] class C: union-merge the link bar +# forget ... drop tombstones so a path can come back +# filter dump the generated rsync filter +# setup | setup-key | install one-time plumbing +# +# Rationale for every odd choice: ../DESIGN.md +set -uo pipefail + +HH_ROOT=$(cd -- "$(dirname -- "${BASH_SOURCE[0]}")/.." && pwd) +[ -r "$HH_ROOT/config" ] || { printf 'homehub: no config at %s\n' "$HH_ROOT/config" >&2; exit 1; } +# shellcheck disable=SC1091 +. "$HH_ROOT/config" + +HOST=$(hostname -s) +HUB_URL="$HUB_HOST:$HUB_PATH" +RULES=$HH_ROOT/rules +STATE=${STATE_ROOT:?config: STATE_ROOT unset} +FILTER=$STATE/effective.filter +mkdir -p "$STATE/tmp" + +# systemd sets HH_QUIET=1: silence on success is a feature, not an omission (§10) +v() { [ "${HH_QUIET:-0}" = 1 ] || printf ' %s\n' "$*"; } +warn() { printf 'homehub: %s\n' "$*" >&2; } +die() { printf 'homehub: %s\n' "$*" >&2; exit 1; } +now() { date +%s; } +hr() { [ "${HH_QUIET:-0}" = 1 ] || printf '\n== %s ==\n' "$*"; } + +# ------------------------------------------------------------- ssh plumbing -- +SSH_OPTS=(-o BatchMode=yes -o ConnectTimeout="${SSH_TIMEOUT:-10}" -o LogLevel=ERROR) +[ -f "$HH_KEY" ] && SSH_OPTS+=(-i "$HH_KEY" -o IdentitiesOnly=yes) +hub() { ssh "${SSH_OPTS[@]}" "$HUB_HOST" "$@"; } +hub_ok() { hub true >/dev/null 2>&1; } + +# Telling "off the LAN" apart from "our own config is broken" is the whole point of +# §10: the first must stay silent for weeks, the second must become loud after a +# day. 0 = usable, 1 = network/unreachable (stays silent), 2 = auth/config (loud). +hub_probe() { + local out + if ! out=$(ssh "${SSH_OPTS[@]}" "$HUB_HOST" true 2>&1); then + case $out in + *"Permission denied"*|*"Host key verification"*|*"no matching host key"*|*"invalid format"*|*"no such identity"*) + return 2 ;; + esac + return 1 + fi + return 0 +} + +# Local breakage markers — readable with the hub down, so a machine whose own key +# stopped working can still alert about itself. First-failure epoch is preserved. +mark_broken() { # kind message + local f=$STATE/broken.$1 e + e=$(awk -F'\t' 'NF{print $1; exit}' "$f" 2>/dev/null) + [ -n "${e:-}" ] || e=$(now) + printf '%s\t%s\n' "$e" "$2" > "$f" +} +clear_broken() { rm -f "$STATE/broken.$1" 2>/dev/null || true; } +# Remote script with positional args, no quoting hell: +# hub_script "'$HUB_PATH' '$HOST'" <<'EOS' ... EOS (POSIX sh on DSM!) +hub_script() { ssh "${SSH_OPTS[@]}" "$HUB_HOST" "sh -s -- $1"; } + +touch_marker() { + hub "mkdir -p '$HUB_PATH/.hub/meta/$1' && touch '$HUB_PATH/.hub/meta/$1/$HOST'" || true +} +log_line() { # rc note + printf '%s\t%s\t%s\t%s\n' "$(now)" "$HOST" "$1" "$2" | + hub "mkdir -p '$HUB_PATH/.hub' && cat >> '$HUB_PATH/.hub/log.tsv'" 2>/dev/null || true +} +hub_setup_dirs() { + hub "mkdir -p '$HUB_PATH/.hub/meta/markers' '$HUB_PATH/.hub/meta/alive' \ + '$HUB_PATH/.hub/meta/alerted' '$HUB_PATH/.hub/tombstones' '$HUB_PATH/.hub/prev' \ + '$HUB_PATH/.hub/gitman' '$HUB_PATH/.hub/bookmarks' && chmod 700 '$HUB_PATH'" || true +} + +# ---------------------------------------------------- filter (default deny) -- +sets_for_host() { + local f=$RULES/hosts/$HOST + [ -r "$f" ] || { warn "no rules/hosts/$HOST — this host syncs nothing (see rules/hosts/_laptop-template)"; return 1; } + awk '!/^[[:space:]]*#/ && NF' "$f" +} +# one path per line -> rsync rule; trailing slash means "this dir and its subtree" +rules_from() { # file sign + awk -v s="$2" '!/^[[:space:]]*#/ && NF { + if ($0 ~ /\/$/) printf "%s %s***\n", s, $0; else printf "%s %s\n", s, $0 }' "$1" +} +# ancestors must be explicitly traversable, otherwise a blanket "- *" can prune +# the branch before rsync ever reaches the include below it +ancestors_of() { # /a/b/c/ -> "+ /a/", "+ /a/b/" + local p=${1%/} + while [ -n "$p" ] && [ "${p#*/}" != "$p" ]; do + p=${p%/*} + [ -n "$p" ] && printf '+ %s/\n' "$p" + done +} +check_shadowing() { # $1 set name + local f=$RULES/sets.d/$1.list d inc nm + [ -r "$f" ] || return 0 + while read -r inc; do + while read -r d; do + case $d in + "$inc") warn "shadowed: include '$inc' is also denied verbatim (set $1)"; continue ;; + esac + case $d in + /*/) case $inc in "$d"*|"${d%/}") warn "shadowed: include '$inc' sits under deny rule '$d' (set $1)" ;; esac ;; + */) nm=${d%/}; case "/$inc" in *"/$nm"/*|*"/$nm") warn "shadowed: include '$inc' matches unanchored deny '$d' (set $1)" ;; esac ;; + esac + done < <(awk '!/^[[:space:]]*#/ && NF' "$RULES/denylist" 2>/dev/null) + done < <(awk '!/^[[:space:]]*#/ && NF' "$f") +} +# base filter: deny first, then includes (+ ancestors), NO trailing default-deny +gen_filter() { + local sets s + sets=$(sets_for_host) || return 1 + { + echo '- /.hub/***' # hub metadata can never leak into a home + [ -r "$RULES/denylist" ] && rules_from "$RULES/denylist" '-' + for s in $sets; do + [ -r "$RULES/sets.d/$s.list" ] || { warn "unknown set '$s' (no rules/sets.d/$s.list)"; continue; } + { rules_from "$RULES/sets.d/$s.list" '+' + awk '!/^[[:space:]]*#/ && NF' "$RULES/sets.d/$s.list" | while read -r l; do ancestors_of "$l"; done; } + check_shadowing "$s" + done + } | awk '!seen[$0]++' > "$FILTER.base.tmp" || return 1 + mv "$FILTER.base.tmp" "$FILTER.base" + # push filter: default deny only + { cat "$FILTER.base"; echo '- *'; } > "$FILTER.tmp" && mv "$FILTER.tmp" "$FILTER" + # pull filter: default deny, PLUS still-tombstoned paths excluded so a deleted + # file is not re-downloaded every cycle for 30 days only to be re-deleted. + # Tombstones never enter the push filter: a file re-created after the delete + # must still be able to travel (edits win — DESIGN.md §3). + { + cat "$FILTER.base" + awk -F'\t' 'NF>=2 { p=$2; sub(/^\//, "", p); if (p != "" && p !~ /^\.hub\// && p !~/\.\./) { + print "- /" p; print "- /" p "/***" } }' "$STATE/tombstones.all" 2>/dev/null + echo '- *' + } | awk '!seen[$0]++' > "$FILTER.pull.tmp" && mv "$FILTER.pull.tmp" "$FILTER.pull" +} + +# --------------------------------------------------------------- rsync bits -- +# rsync must authenticate with the SAME identity as the rest of homehub, or it +# silently picks another key/agent and unreachable comes back from the wrong door. +RSH="ssh -o BatchMode=yes -o ConnectTimeout=${SSH_TIMEOUT:-10} -o LogLevel=ERROR" +[ -f "${HH_KEY:-}" ] && RSH="$RSH -i $HH_KEY -o IdentitiesOnly=yes" +RS=(-rltDz --partial --timeout=90 --out-format='%i|%n' -e "$RSH") +run_rsync() { # dry filter extra args... + local dry=$1 filt=$2; shift 2 + if [ "$dry" = 1 ]; then + "$RSYNC_BIN" "${RS[@]}" --filter="merge $filt" -n "$@" + else + "$RSYNC_BIN" "${RS[@]}" --filter="merge $filt" "$@" + fi +} +pull_rsync() { # dry + run_rsync "$1" "$FILTER.pull" --update "$HUB_URL/" "$HOME/" +} +push_rsync() { # dry + if [ "$1" = 1 ]; then + run_rsync 1 "$FILTER" --update "$HOME/" "$HUB_URL/" + else + run_rsync 0 "$FILTER" --update --backup --backup-dir=".hub/prev/$(date +%Y-%m)" "$HOME/" "$HUB_URL/" + fi +} +count_items() { grep -cE '^[<>]' "$1" 2>/dev/null || true; } +show_head() { grep -E '^[<>]' "$1" 2>/dev/null | head -n "${2:-10}"; } + +# ----------------------------------------------------- deletions (tombstones) -- +# Format on the hub: .hub/tombstones/ lines = epoch \t home-rel-path +hub_tombstones_all() { + hub_script "'$HUB_PATH'" <<'EOS' +root=$1 +cd "$root/.hub/tombstones" 2>/dev/null || exit 0 +for f in *; do + [ -f "$f" ] || continue + while IFS= read -r l; do printf '%s\t%s\n' "${f%%:*}" "$l"; done < "$f" +done +exit 0 +EOS +} +# is this home-relative path inside one of this host's sets? +active_path() { # $1 = path without leading slash + local p="/$1" + while :; do + grep -qxF "+ $p***" "$FILTER" && return 0 # inside an included subtree + grep -qxF "+ $p" "$FILTER" && return 0 # is itself an included file + [ "${p%/*}" = "$p" ] && return 1 + p=${p%/*} + [ -z "$p" ] && return 1 + done +} +apply_tombstones() { # deletions recorded by any machine, edit-wins (§3) + local all=$STATE/tombstones.all n=0 kept=0 out=0 bad=0 + [ -s "$all" ] || { v "tombstones: none recorded"; return 0; } + while IFS=$'\t' read -r src ep path; do + [ -n "${path:-}" ] || continue + path=${path#/} + case $path in + .hub/*|*..*|/*) warn "ignoring suspicious tombstone '$path'"; bad=$((bad+1)); continue ;; + esac + active_path "$path" || { out=$((out+1)); continue; } + [ -e "$HOME/$path" ] || continue + if [ -n "$ep" ] && find "$HOME/$path" -newermt "@$ep" -print -quit 2>/dev/null | grep -q .; then + kept=$((kept+1)); continue # touched after the delete: edits win, keep it + fi + n=$((n+1)) + if [ "$n" -gt "${TOMB_LIMIT:-100}" ] && [ "${HH_FORCE:-0}" != 1 ]; then + die "tombstones ask for more than $TOMB_LIMIT deletions — refusing. Check $HUB_HOST:$HUB_PATH/.hub/tombstones/ then re-run with HH_FORCE=1" + fi + v "deleted (tombstone from $src): $path" + rm -rf --one-file-system -- "$HOME/$path" + done < "$all" + v "tombstones: applied=$n edited-since-so-kept=$kept outside-sets=$out malformed=$bad" +} +harvest_tombstones() { # record what this machine deleted; runs only after a real pull + local new=$STATE/tomb.new merged=$STATE/tomb.merged n ep + run_rsync 1 --delete "$HOME/" "$HUB_URL/" 2>/dev/null | + awk -F'|' '$1 ~ /^\*/ { sub(/^\//, "", $2); print $2 }' | sort -u > "$new" + n=$(grep -c . "$new" 2>/dev/null || true) + [ "${n:-0}" = 0 ] && { v "no local deletions to record"; return 0; } + ep=$(now) + { hub "cat '$HUB_PATH/.hub/tombstones/$HOST' 2>/dev/null" + awk -v e="$ep" '{ print e "\t" $0 }' "$new" + } | awk -F'\t' 'NF>=2 { p=$2; if (!(p in e) || $1+0 < e[p]) e[p]=$1+0 } + END { for (p in e) printf "%d\t%s\n", e[p], p }' | sort -t$'\t' -k2 > "$merged" + hub "mkdir -p '$HUB_PATH/.hub/tombstones' && cat > '$HUB_PATH/.hub/tombstones/$HOST'" < "$merged" || + { warn "could not write tombstones"; return 1; } + v "recorded $n deletion(s) as tombstones on the hub" +} +reap() { # stop resurrecting files deleted more than GRACE_DAYS ago (§5) + hub_script "'$HUB_PATH' '${GRACE_DAYS:-30}'" <<'EOS' || true +root=$1; grace=$2 +cd "$root/.hub/tombstones" 2>/dev/null || exit 0 +cutoff=$(( $(date +%s) - grace * 86400 )) +for f in *; do + [ -f "$f" ] || continue + awk -F'\t' -v c="$cutoff" '$1+0 < c { print }' "$f" > "$f.gone" + awk -F'\t' -v c="$cutoff" '$1+0 >= c { print }' "$f" > "$f.keep" + while IFS=$'\t' read -r e p; do + [ -n "$p" ] || continue + t="$root/$p" + [ -e "$t" ] || continue + d="$root/.hub/prev/reaped/$(date -d "@$e" +%Y-%m 2>/dev/null || echo unknown)" + mkdir -p "$d/$(dirname "$p")" 2>/dev/null && mv "$t" "$d/$p" 2>/dev/null && echo "reaped $p" + done < "$f.gone" + if [ -s "$f.keep" ]; then mv "$f.keep" "$f"; else rm -f "$f.keep" "$f"; fi + rm -f "$f.gone" +done +exit 0 +EOS +} + +# The hub sync root must contain ONLY what homehub put there. Anything else means +# HUB_PATH points somewhere that already has data — which would be merged into +# $HOME on the first pull (and could be reaped by a tombstone). This guard exists +# because /volume1/homes/bl really did hold 178 G of unrelated legacy data. +check_hub_stray() { + [ -r "$FILTER.base" ] || return 0 + local allowed stray + allowed=$(printf '.hub\n@eaDir\n'; grep '^+ /' "$FILTER.base" | sed -E 's#^\+ /([^/]+).*#\1#; s#/$##' | sort -u) + stray=$(hub "cd '$HUB_PATH' 2>/dev/null && ls -A" | grep -vxF -f <(printf '%s\n' "$allowed") || true) + [ -z "$stray" ] && return 1 + warn "unexpected content in hub sync root $HUB_PATH (pull would merge it into $HOME):" + printf ' %s\n' $stray + return 0 +} + +# ------------------------------------------------------------------ sync ---- +cmd_sync() { + local exec=0; [ "${1:-}" = "--exec" ] && exec=1 + local dry=1; [ $exec = 1 ] && dry=0 + local sets rc=0 + + sets=$(sets_for_host) || return 1 + + case "$(hub_probe; echo $?)" in + 1) # Off the LAN. Silent on purpose: the laptop-in-another-country case (§10). + v "hub $HUB_HOST unreachable — nothing done (silent by design; a live machine that stays broken alerts through 'fresh' after ${FRESH_MAX_AGE}s)" + return 0 ;; + 2) warn "cannot authenticate to $HUB_HOST (key: ${HH_KEY:-default agent}). This is a CONFIGURATION failure, not an offline machine." + warn " fix: ssh-copy-id -i ${HH_KEY:-~/.ssh/id_ed25519}.pub $HUB_HOST" + mark_broken auth "ssh auth to $HUB_HOST failed (key ${HH_KEY:-default})" + return 64 ;; + esac + clear_broken auth + hub_setup_dirs + if check_hub_stray; then + [ $exec = 1 ] && [ "${HH_FORCE:-0}" != 1 ] && + die "refusing to sync into a hub root that is not ours. Fix HUB_PATH in config, or move that data away, or re-run with HH_FORCE=1 if you really mean it" + fi + # tombstones must be known BEFORE the filter is built: they shape the pull filter + hub_tombstones_all > "$STATE/tombstones.all" || true + gen_filter || return 1 + hr "homehub sync ($([ $exec = 1 ] && echo EXECUTE || echo 'dry run')) on $HOST" + v "sets: $sets" + v "filter: $(grep -c . "$FILTER" || true) push, $(grep -c . "$FILTER.pull" || true) pull rules, $(grep -c . "$STATE/tombstones.all" 2>/dev/null || true) tombstoned" + [ $exec = 1 ] && touch_marker alive + + hr "1. pull $HUB_URL -> $HOME/" + pull_rsync "$dry" > "$STATE/pull.out" 2> "$STATE/pull.err" || rc=$? + if [ $rc != 0 ]; then + warn "pull failed (rsync rc=$rc): $(tr '\n' ' ' < "$STATE/pull.err" | tail -c 300)" + else + v "$(count_items "$STATE/pull.out") change(s) inbound" + [ $dry = 1 ] && show_head "$STATE/pull.out" + fi + + if [ $exec = 1 ]; then + hr "2. apply deletions recorded by any machine (edit-wins)" + apply_tombstones + hr "3. harvest this machine's deletions" + harvest_tombstones || rc=1 + else + hr "2./3. deletion handling skipped in dry run" + fi + + hr "4. push $HOME/ -> $HUB_URL" + push_rsync "$dry" > "$STATE/push.out" 2> "$STATE/push.err" || rc=$? + if [ $rc != 0 ]; then + warn "push failed (rsync rc=$rc): $(tr '\n' ' ' < "$STATE/push.err" | tail -c 300)" + else + v "$(count_items "$STATE/push.out") change(s) outbound" + [ $dry = 1 ] && show_head "$STATE/push.out" + fi + + if [ $exec = 1 ] && [ $rc = 0 ]; then + reap | sed 's/^/ /' + touch_marker markers + hub "rm -f '$HUB_PATH/.hub/meta/alerted/$HOST'" || true + clear_broken sync + elif [ $exec = 1 ]; then + mark_broken sync "last sync failed with rsync rc=$rc (see $STATE/pull.err, $STATE/push.err)" + fi + [ $exec = 1 ] && log_line "$rc" "sync" + + if [ $dry = 1 ]; then + local tot; tot=$(count_items "$STATE/pull.out"); tot=$((tot + $(count_items "$STATE/push.out"))) + hr "summary" + v "$tot change(s) pending — nothing was written. Run: homehub sync --exec" + [ "${tot:-0}" -gt "${WARN_FILES:-20000}" ] && + warn "that is a lot of files for a first run; review $STATE/{pull,push}.out first" + fi + return $rc +} + +# ---------------------------------------------------------------- status ---- +hub_meta() { + hub_script "'$HUB_PATH'" <<'EOS' +root=$1 +cd "$root/.hub" 2>/dev/null || exit 0 +for d in meta/markers meta/alive meta/alerted; do + for f in "$d"/*; do + [ -f "$f" ] || continue + printf '%s\t%s\t%s\n' "${d##*/}" "$(basename "$f")" "$(date -r "$f" +%s 2>/dev/null || echo 0)" + done +done +if [ -f log.tsv ]; then + tail -n 400 log.tsv | awk -F'\t' '{ r[$2] = $3 "\t" $4 } END { for (h in r) printf "lastlog\t%s\t%s\n", h, r[h] }' +fi +for f in tombstones/*; do [ -f "$f" ] && printf 'tomb\t%s\t%s\n' "$(basename "$f")" "$(grep -c . "$f")"; done +for f in gitman/repos.*; do [ -f "$f" ] && printf 'repos\t%s\t%s\n' "$(basename "$f" 2>/dev/null)" "$(grep -c . "$f")"; done +for f in bookmarks/*; do [ -f "$f" ] && printf 'bmarks\t%s\t%s\n' "$(basename "$f" .json)" "$(date -r "$f" +%s)"; done +exit 0 +EOS +} +age() { local n=$(( $(now) - ${1:-0} )); [ "${1:-0}" = 0 ] && { printf 'never'; return; } + if [ $n -lt 3600 ]; then printf '%dm' $((n/60)) + elif [ $n -lt 172800 ]; then printf '%.1fh' "$(awk -v n=$n 'BEGIN{print n/3600}')" + else printf '%.0fd' "$(awk -v n=$n 'BEGIN{print n/86400}')"; fi; } + +cmd_status() { + hub_ok || die "hub $HUB_HOST unreachable" + sets_for_host >/dev/null || true + gen_filter 2>/dev/null || true + hub_meta > "$STATE/meta.tsv" || die "could not read hub metadata" + + hr "machines" + printf ' %-14s %-9s %-9s %-6s %s\n' HOST synced heartbeat last-rc notes + local h + for h in $(awk -F'\t' '$1=="markers"||$1=="alive"{print $2}' "$STATE/meta.tsv" | sort -u); do + local m a rc note + m=$(awk -F'\t' -v h="$h" '$1=="markers"&&$2==h{print $3}' "$STATE/meta.tsv") + a=$(awk -F'\t' -v h="$h" '$1=="alive"&&$2==h{print $3}' "$STATE/meta.tsv") + rc=$(awk -F'\t' -v h="$h" '$1=="lastlog"{split($0,x,"\t"); if (x[2]==h) print x[3]}' "$STATE/meta.tsv") + note=$(awk -F'\t' -v h="$h" '$1=="lastlog"{split($0,x,"\t"); if (x[2]==h) print x[4]}' "$STATE/meta.tsv") + printf ' %-14s %-9s %-9s %-6s %s\n' "$h" "$(age "$m")" "$(age "$a")" "${rc:-?}" "${note:-}" + done + + hr "this host ($HOST)" + v "sets: $(sets_for_host | tr '\n' ' ')" + v "included: $(grep -c '^+ ' "$FILTER" 2>/dev/null || echo 0) rules, denied: $(grep -c '^- ' "$FILTER" 2>/dev/null || echo 0)" + v "state dir: $STATE" + awk -F'\t' -v h="$HOST" '$1=="tomb"&&$2==h{print " tombstones: " $3 " paths"}' "$STATE/meta.tsv" + v "pending dry-run lists: $STATE/pull.out $STATE/push.out (run 'homehub sync' to refresh)" + + hr "class B — repo manifest" + awk -F'\t' '$1=="repos"{printf " %-14s %s repos\n", $2, $3}' "$STATE/meta.tsv" + cmd_git missing 2>/dev/null | sed 's/^/ /' || true + + hr "class C — bookmark bar" + awk -F'\t' '$1=="bmarks"{printf " %-14s exported %s\n", $2, ""}' "$STATE/meta.tsv" + [ -r "$BOOKMARK_SRC" ] && v "live file: $BOOKMARK_SRC" +} + +# ----------------------------------------------------------------- fresh ---- +alert_cmd() { + if [ -n "${ALERT_CMD:-}" ]; then printf '%s' "$ALERT_CMD"; return; fi + if command -v msmtp >/dev/null 2>&1; then printf 'msmtp -a homehub %s' "$ALERT_TO" + elif command -v sendmail >/dev/null 2>&1; then printf 'sendmail -t' + else printf 'cat >> "%s/alerts.out"' "$STATE"; fi +} +alert_send() { # subject body + { printf 'From: %s\nTo: %s\nSubject: %s\n\n%s\n' "${ALERT_FROM:-homehub}" "${ALERT_TO:-root}" "$1" "$2" | + sh -c "$(alert_cmd)"; } 2>>"$STATE/alert.err" || warn "alert delivery failed (see $STATE/alert.err)" + if [ "${HH_QUIET:-0}" != 1 ] && [ -n "${DISPLAY:-}" ] && command -v notify-send >/dev/null 2>&1; then + notify-send -u critical "$1" "$2" || true + fi +} +cmd_fresh() { # the ONLY place that may complain (§10) + local dry=0; [ "${1:-}" = "--dry" ] && dry=1 + local n=$(now) h msgs="" f + + # 1) Self-checks that work with the hub down. Closes the hole where a dead key + # means no marker and no heartbeat, i.e. a machine that can never alert. + for f in "$STATE"/broken.*; do + [ -e "$f" ] || continue + local kind=${f##*.} ep msg age already=0 + ep=$(awk -F'\t' 'NF{print $1; exit}' "$f"); msg=$(cut -f2- "$f") + age=$(( n - ${ep:-0} )) + [ -e "$STATE/alerted-local.$kind" ] && + [ $(( n - $(date -r "$STATE/alerted-local.$kind" +%s) )) -lt "${REALERT_AGE:-604800}" ] && already=1 + if [ "$age" -gt "${FRESH_MAX_AGE:-86400}" ] && [ $already = 0 ]; then + msgs="$msgs$HOST, local check '$kind': $msg (failing for $(age "$ep"))"$'\n' + touch "$STATE/alerted-local.$kind" + else + v "ok(local): $kind has been failing for $(age "$ep")$([ $already = 1 ] && echo ', already alerted')" + fi + done + + # 2) Hub-side freshness of every machine + if ! hub_ok; then + v "hub unreachable — cannot judge other machines (the local checks above still ran)" + else + hub_meta > "$STATE/meta.tsv" || die "cannot read hub metadata" + for h in $(awk -F'\t' '$1=="markers"||$1=="alive"{print $2}' "$STATE/meta.tsv" | sort -u); do + local m a al rc mka maa + m=$(awk -F'\t' -v h="$h" '$1=="markers"&&$2==h{print $3}' "$STATE/meta.tsv") + a=$(awk -F'\t' -v h="$h" '$1=="alive"&&$2==h{print $3}' "$STATE/meta.tsv") + al=$(awk -F'\t' -v h="$h" '$1=="alerted"&&$2==h{print $3}' "$STATE/meta.tsv") + rc=$(awk -F'\t' -v h="$h" '$1=="lastlog"{split($0,x,"\t"); if (x[2]==h) print x[3]}' "$STATE/meta.tsv") + mka=$(( n - ${m:-0} )); maa=$(( n - ${a:-$n} )) + local alive_now=0; [ "${a:-0}" != 0 ] && [ $maa -lt "${ALIVE_FRESH:-10800}" ] && alive_now=1 + local netrc=0 x + for x in ${NET_RC:-5 255}; do [ "$rc" = "$x" ] && netrc=1; done + local realert_ok=1 + [ "${al:-0}" != 0 ] && [ $(( n - al )) -lt "${REALERT_AGE:-604800}" ] && realert_ok=0 + + if [ $alive_now = 1 ] && [ $mka -gt "${FRESH_MAX_AGE:-86400}" ] && [ $netrc = 0 ] && [ $realert_ok = 1 ]; then + msgs="$msgs$h (last success $(age "$m"), last rc=${rc:-?})"$'\n' + v "STALE: $h — on, but not synced for $(age "$m")" + hub "mkdir -p '$HUB_PATH/.hub/meta/alerted' && touch '$HUB_PATH/.hub/meta/alerted/$h'" || true + else + v "ok: $h (sync $(age "$m"), heartbeat $(age "$a"), rc=${rc:--}$([ $netrc = 1 ] && echo ', off-LAN: silent'))" + fi + done + fi + + [ -n "$msgs" ] || return 0 + if [ $dry = 1 ]; then + printf 'homehub: would alert for:\n%s' "$msgs" >&2 + return 0 + fi + alert_send "homehub: sync stale on $(hostname -f 2>/dev/null || hostname)" \ + "These machines are running but have not completed a homehub sync in over $(( ${FRESH_MAX_AGE:-86400} / 3600 )) h: + +$msgs +Check with: homehub status +Hub: $HUB_HOST:$HUB_PATH/.hub/log.tsv" +} + +# ------------------------------------------------------- class B: the repos -- +git_root_p() { [ -e "$1/.git" ]; } +git_row() { # relpath remote branch sha dirty unpushed (last two are ADVISORY ONLY) + local d=$1 rel=${1#"$HOME"/} url br sha dirty unpush cred=ok + url=$(git -C "$d" remote get-url origin 2>/dev/null || true); [ -n "$url" ] || url=- + # SECURITY: remotes can carry credentials (https://user:TOKEN@host/...). They + # must never be published to the hub. Strip userinfo and mark the row so + # `git clone` can tell you the URL needs re-adding on the far side. + case $url in + *://*:*@*) url=$(printf '%s' "$url" | sed -E 's#(://)[^/@]*@#\1#'); cred=stripped ;; + *://*@*) url=$(printf '%s' "$url" | sed -E 's#(://)[^/@]*@#\1#'); cred=stripped ;; + esac + br=$(git -C "$d" symbolic-ref --short HEAD 2>/dev/null || echo -) + sha=$(git -C "$d" rev-parse --short HEAD 2>/dev/null || echo -) + dirty=$(git -C "$d" status --porcelain 2>/dev/null | wc -l | tr -d ' ') + unpush=$(git -C "$d" rev-list --count '@{u}..HEAD' 2>/dev/null | tail -1); [ -n "$unpush" ] || unpush=0 + printf '%s\t%s\t%s\t%s\t%s\t%s\t%s\n' "$rel" "$url" "$br" "$sha" "${dirty:-0}" "${unpush:-0}" "$cred" +} +git_walk() { # dir depth — stops at the first .git, so submodules are not double counted + local d=$1 depth=$2 c + if git_root_p "$d"; then git_row "$d"; return; fi + [ "$depth" -le 0 ] && return + for c in "$d"/*/; do [ -d "$c" ] || continue; git_walk "${c%/}" $((depth - 1)); done +} +cmd_git() { + local sub=${1:-inventory}; shift || true + local lazy=0 exec=0 a + for a in "$@"; do case $a in --lazy) lazy=1;; --exec) exec=1;; esac; done + case $sub in + inventory) + local out=$STATE/repos.$HOST.tsv + : > "$out" + local r; for r in ${GIT_DIRS:-git src}; do [ -d "$HOME/$r" ] && git_walk "$HOME/$r" 3 >> "$out"; done + v "$(grep -c . "$out") repo(s) under ${GIT_DIRS:-git src}" + awk -F'\t' '$5+0>0 || $6+0>0 { printf " %-34s %s modified, %s unpushed\n", $1, $5, $6 }' "$out" + awk -F'\t' '$7=="stripped" { printf " CREDENTIALS STRIPPED from remote of %s — the far side must supply them\n", $1 }' "$out" + v "advisory only — neither the working tree nor unpushed commits ever leave this machine" + if [ $exec = 1 ] || [ "${HH_PUSH_INVENTORY:-1}" = 1 ]; then + hub "mkdir -p '$HUB_PATH/.hub/gitman' && cat > '$HUB_PATH/.hub/gitman/repos.$HOST'" < "$out" && + v "manifest published to hub" + fi + ;; + list) + hub "cat '$HUB_PATH/.hub/gitman/repos.'* 2>/dev/null" | sort -u -t$'\t' -k1,1 + ;; + missing) + hub "cat '$HUB_PATH/.hub/gitman/repos.'* 2>/dev/null" | sort -u -t$'\t' -k1,1 | + while IFS=$'\t' read -r rel url br sha dirty unpush cred; do + [ -n "${rel:-}" ] || continue + [ -e "$HOME/$rel/.git" ] && continue + printf '%-34s %s (%s%s%s)\n' "$rel" "$url" "$br" \ + "$( [ "${dirty:-0}" != 0 ] && printf ', %s modified elsewhere' "$dirty")" \ + "$( [ "${cred:-ok}" = stripped ] && printf ', creds stripped')" + done + ;; + clone) + local n=0 + hub "cat '$HUB_PATH/.hub/gitman/repos.'* 2>/dev/null" | sort -u -t$'\t' -k1,1 | + while IFS=$'\t' read -r rel url br sha dirty unpush cred; do + [ -n "${rel:-}" ] || continue + [ "$url" = - ] && continue + [ -e "$HOME/$rel/.git" ] && continue + if [ $exec != 1 ]; then + printf ' would clone %s <- %s%s%s\n' "$rel" "$url" \ + "$( [ $lazy = 1 ] && echo ' (blobless)')" "$( [ "${cred:-ok}" = stripped ] && echo ' [creds stripped: set a credential helper or fix the remote]')" + else + printf ' cloning %s <- %s\n' "$rel" "$url" + mkdir -p "$(dirname "$HOME/$rel")" + local opts=(--origin origin) + [ $lazy = 1 ] && opts+=(--filter=blob:none) + [ "$br" != - ] && git clone "${opts[@]}" -b "$br" "$url" "$HOME/$rel" >/dev/null 2>&1 || + git clone "${opts[@]}" "$url" "$HOME/$rel" >/dev/null 2>&1 || + warn "clone failed: $rel ($url)" + fi + n=$((n+1)) + done + ;; + *) die "git: unknown subcommand '$sub' (inventory|list|missing|clone)";; + esac +} + +# ------------------------------------------------- class C: the link bar ----- +cmd_bookmarks() { + local write=0 a + for a in "$@"; do [ "$a" = --write ] && write=1; done + [ -r "$BOOKMARK_SRC" ] || die "no bookmark file at $BOOKMARK_SRC" + cp -- "$BOOKMARK_SRC" "$STATE/bookmarks.$HOST.json" || die "cannot read $BOOKMARK_SRC" + local nu; nu=$(jq '[.. | objects | select(.type=="url")] | length' "$STATE/bookmarks.$HOST.json" 2>/dev/null || echo '?') + hr "class C: bookmark bar ($HOST)" + v "exported $nu url(s) from $BOOKMARK_SRC (the union-merge below covers the bar only)" + if [ $write != 1 ]; then + v "dry: nothing pushed, nothing merged, live file untouched. Use --write." + return 0 + fi + hub "mkdir -p '$HUB_PATH/.hub/bookmarks' && cat > '$HUB_PATH/.hub/bookmarks/$HOST.json'" < "$STATE/bookmarks.$HOST.json" || + die "could not publish bookmark export" + hub "cd '$HUB_PATH/.hub/bookmarks' 2>/dev/null && for f in *.json; do printf '### %s\\n' \"\$f\"; cat \"\$f\"; done" > "$STATE/bookmarks.all.json" || + die "could not fetch exports" + "$PY" "$HH_ROOT/lib/merge_bookmarks.py" "$STATE/bookmarks.all.json" > "$STATE/bookmarks.merged.json" || + die "merge failed" + local mu; mu=$(jq '.urls' "$STATE/bookmarks.merged.json" 2>/dev/null || echo '?') + v "merged union: $mu urls (from $(jq -r '.hosts | join(",")' "$STATE/bookmarks.merged.json" 2>/dev/null))" + v "union result: $STATE/bookmarks.merged.json" + if pgrep -f "$BOOKMARK_PGREP" >/dev/null 2>&1; then + warn "browser is running — merged bar NOT written (it would be overwritten in memory). Close it and re-run." + v "merged result is ready at $STATE/bookmarks.merged.json" + return 0 + fi + warn "writing the merged bar to $BOOKMARK_SRC is not enabled in v1 — inspect $STATE/bookmarks.merged.json first (see README)" +} + +# -------------------------------------------------------------- housekeeping - +cmd_forget() { + [ $# -ge 1 ] || die "usage: homehub forget ..." + hub_ok || die "hub unreachable" + local p + for p in "$@"; do + p=${p#/} + hub_script "'$HUB_PATH' '$p'" <<'EOS' +root=$1; pat=$2 +cd "$root/.hub/tombstones" 2>/dev/null || exit 0 +for f in *; do + [ -f "$f" ] || continue + awk -F'\t' -v p="$pat" '$2 != p && $2 !~ ("^" p "/")' "$f" > "$f.t" && mv "$f.t" "$f" +done +exit 0 +EOS + v "forget: $p" + done +} +cmd_filter() { gen_filter || return 1; cat "$FILTER"; } + +cmd_setup_key() { + [ -f "$HH_KEY" ] && { warn "$HH_KEY already exists"; return 0; } + mkdir -p "$(dirname "$HH_KEY")" + ssh-keygen -q -t ed25519 -N '' -C "homehub@$HOST" -f "$HH_KEY" || die "keygen failed" + hr "deploy key created" + v "$HH_KEY (passphrase-less, by necessity: a systemd user unit has no agent)" + v "Install it on the hub, then make ~/.ssh 700 there:" + printf '\n ssh-copy-id -i %s.pub %s\n\n' "$HH_KEY" "$HUB_HOST" + warn "Until that is done, background sync cannot authenticate — and it fails silently by design." +} +cmd_setup() { + hub_ok || die "cannot reach $HUB_HOST with the current ssh setup" + hub_setup_dirs + mkdir -p "$SHARED_ROOT" + v "shared state root ready: $SHARED_ROOT (put app state here, symlink apps to it)" + hub "chmod 700 '$HUB_PATH' 2>/dev/null; ls -d '$HUB_PATH/.hub'/*" | sed 's/^/ /' + hr "published rules" + git -C "$HH_ROOT" rev-parse --is-inside-work-tree >/dev/null 2>&1 && + v "rules live in git: $HH_ROOT — clone that repo on the other machines" + [ -f "$HH_KEY" ] || cmd_setup_key +} +cmd_install() { + local d=${XDG_CONFIG_HOME:-$HOME/.config}/systemd/user + mkdir -p "$d" + cp -v "$HH_ROOT"/systemd/homehub-*.service "$HH_ROOT"/systemd/homehub-*.timer "$d/" || return 1 + systemctl --user daemon-reload + systemctl --user enable --now homehub-sync.timer homehub-fresh.timer + systemctl --user --no-pager --full status homehub-sync.timer homehub-fresh.timer 2>&1 | grep -E 'Loaded:|Active:|→' || true +} + +usage() { sed -n '2,14p' "${BASH_SOURCE[0]}" | sed 's/^# \{0,1\}//'; } + +case ${1:-help} in + sync) shift; cmd_sync "$@";; + status) shift; cmd_status "$@";; + fresh) shift; cmd_fresh "$@";; + git) shift; cmd_git "$@";; + bookmarks) shift; cmd_bookmarks "$@";; + forget) shift; cmd_forget "$@";; + filter) shift; cmd_filter "$@";; + setup) shift; cmd_setup "$@";; + setup-key) shift; cmd_setup_key "$@";; + install) shift; cmd_install "$@";; + help|-h|--help) usage;; + *) warn "unknown command '${1:-}'; try 'homehub help'"; exit 2;; +esac diff --git a/config b/config new file mode 100644 index 0000000..0c6cd85 --- /dev/null +++ b/config @@ -0,0 +1,64 @@ +# homehub configuration — sourced by bin/homehub (also readable by systemd units) +# All paths are for user `bl` on the Synology hub `nas.lue.dk`. + +HUB_HOST=nas.lue.dk + +# NEVER point this at your Synology home directory. /volume1/homes/bl already +# holds 178 G of unrelated legacy data (Documents, Photos, OneDrive, CloudStation, +# home_backup, x1c_backup, wg-bo.conf, certs, ...) — pulling from it would merge +# years of NAS files into ~/, and a tombstone could reap unrelated files by name +# accident. The sync root is a dedicated subtree; .hub/ metadata lives inside it. +HUB_BASE=/volume1/homes/bl +HUB_PATH=$HUB_BASE/homehub + +# Dedicated, PASSPHRASE-LESS deploy key. Your id_rsa is passphrase-protected and a +# systemd *user* unit has no SSH_AUTH_SOCK, so using ~/.ssh/id_rsa here would make +# every background sync fail with "Permission denied (publickey)" — forever, and +# silently. Run `homehub setup-key` and install the printed pubkey. +HH_KEY=${HH_KEY:-$HOME/.ssh/homehub_ed25519} + +# Where this machine keeps homehub's OWN private state (generated filters, markers +# cache, inventories, rsync logs, alert spool). NEVER SYNCED — hard-blocked in the +# denylist, because syncing it makes machines overwrite each other's filters. +STATE_ROOT=$HOME/.local/state/homehub + +# The SHARED state root: the one tree whose contents are shared verbatim between +# machines, and the anchor that survives the NixOS migration unchanged (§8). +# App state you want everywhere lives in $SHARED_ROOT// and is symlinked +# to wherever the app expects it. Keep it distinct from STATE_ROOT above: sharing +# homehub's own machinery between machines just makes them fight over it. +SHARED_ROOT=$HOME/.local/state/shared + +# --- policy knobs ------------------------------------------------------------- +GRACE_DAYS=30 # tombstone age before the hub copy is reaped (§5) +FRESH_MAX_AGE=86400 # alert if a live machine has not synced in 24 h +ALIVE_FRESH=10800 # "machine is on" = heartbeat younger than 3 h +REALERT_AGE=604800 # at most one alert per host per week +TOMB_LIMIT=100 # refuse to apply more deletions than this in one run +WARN_FILES=20000 # dry run warns above this many pending changes + +# Where to look for git repos for the class B manifest (DESIGN.md §6). +GIT_DIRS="git src" + +# rc values treated as "machine is simply off the LAN" -> silent forever (§10) +NET_RC="5 10 11 12 30 35 255" + +# --- alerting ---------------------------------------------------------------- +ALERT_TO=bl@lue.dk +ALERT_FROM=homehub@lue.dk +# Empty => auto-detect msmtp, then sendmail, else append to $STATE_ROOT/alerts.out +ALERT_CMD= + +# --- class C: the bookmark bar ---------------------------------------------- +# NOTE the quotes: "Profile 5" contains a space, and an unquoted value here +# silently breaks `. config` in every subshell. +BOOKMARK_SRC="$HOME/.config/chromium/Profile 5/Bookmarks" +BOOKMARK_PGREP='/usr/lib/chromium/chromium' + +# Absolute interpreter: PATH python3 here is /home/bl/.venv/bin/python3. +PY=/usr/bin/python3 +[ -x "$PY" ] || PY=$(command -v python3) + +# rsync 3.1.2 on the NAS: do NOT add --ignore-older (needs >=3.2.3 on both ends). +RSYNC_BIN=rsync +SSH_TIMEOUT=10 diff --git a/lib/merge_bookmarks.py b/lib/merge_bookmarks.py new file mode 100755 index 0000000..e4e314f --- /dev/null +++ b/lib/merge_bookmarks.py @@ -0,0 +1,142 @@ +#!/usr/bin/python3 +"""Class C (DESIGN.md §7): union-merge the Chromium bookmark bar across machines. + +Reads a file holding one concatenated JSON bookmark document per machine, flattens +every bar to (folder-path, title, url) tuples, takes the union, and rebuilds one +tree. Union-by-URL is idempotent, so bookmarks need no conflict concept at all — +which is the only reason this file is 60 lines instead of 600. + +Deletions deliberately do NOT propagate, consistent with the tombstone rule for +files: edits win, deletes lose. Escape hatch is documented in README. + +stdout: JSON { urls, hosts, dropped, bookmark_bar } +""" +import json +import sys + + +def load_docs(text): + """Decode the hub's concatenated exports. + + The hub file is built as `### .json` followed by that host's JSON object, + repeated. Returns [(host, doc), ...] so the report can name its sources. + """ + dec = json.JSONDecoder() + out = [] + for chunk in _split(text): + host, body = chunk + obj = _one(dec, body) + if obj is not None: + out.append((host, obj)) + return out + + +def _split(text): + """Split into [(host, json_text), ...] on '### name' marker lines.""" + chunks, host, buf = [], "?", [] + for line in text.splitlines(keepends=True): + if line.startswith("### "): + body = "".join(buf) + if body.strip(): + chunks.append((host, body)) + host = line[4:].strip() + if host.endswith(".json"): + host = host[:-5] + buf = [] + else: + buf.append(line) + body = "".join(buf) + if body.strip(): + chunks.append((host, body)) + return chunks + + +def _one(dec, body): + i, n = 0, len(body) + while i < n and body[i] in " \r\n\t": + i += 1 + if i >= n: + return None + obj, _ = dec.raw_decode(body, i) + return obj + + +def flatten(bar, prefix, out, seen): + for node in bar.get("children") or []: + ntype = node.get("type") + if ntype == "folder": + flatten(node, prefix + [node.get("name") or "?"], out, seen) + elif ntype == "url": + url, title = node.get("url"), node.get("name") or "" + key = (tuple(prefix), url) + if key in seen: # same bar, same folder, same URL -> one copy + continue + seen.add(key) + out.append({"path": prefix, "title": title, "url": url}) + # typed_pair / bookmark_bar root itself: ignored + + +def rebuild(rows): + def nodes(prefix): + leaves = [ + {"type": "url", "name": r["title"], "url": r["url"], "date_added": "0"} + for r in rows if r["path"] == prefix + ] + names, seen = [], set() + for r in rows: + if len(r["path"]) > len(prefix) and r["path"][: len(prefix)] == prefix: + nm = r["path"][len(prefix)] + if nm not in seen: + seen.add(nm) + names.append(nm) + folders = [ + {"type": "folder", "name": nm, "date_added": "0", "children": nodes(prefix + [nm])} + for nm in names + ] + return leaves + folders + + return {"type": "bookmark_bar", "name": "Bookmarks bar", "children": nodes([])} + + +def main(): + if len(sys.argv) != 2: + sys.exit("usage: merge_bookmarks.py ") + with open(sys.argv[1], encoding="utf-8", errors="replace") as fh: + docs = load_docs(fh.read()) + if not docs: + sys.exit("merge_bookmarks: no JSON documents found") + + rows, seen = [], set() + hosts = [] + for host, d in docs: + hosts.append(host) + bar = (d.get("roots") or {}).get("bookmark_bar") + if bar: + flatten(bar, [], rows, seen) + + # collapse identical URLs that sit in different folders, keeping the shallowest + byurl = {} + for r in rows: + cur = byurl.get(r["url"]) + if cur is None or len(r["path"]) < len(cur["path"]): + byurl[r["url"]] = r + dropped = len(rows) - len(byurl) + final = sorted(byurl.values(), key=lambda r: (r["path"], r["title"].lower())) + + json.dump( + { + "urls": len(final), + "hosts": hosts, + "duplicate_urls_collapsed": dropped, + "bookmark_bar": rebuild(final), + "flat": final, + }, + sys.stdout, + indent=1, + sort_keys=False, + ) + sys.stdout.write("\n") + + +if __name__ == "__main__": + main() diff --git a/rules/denylist b/rules/denylist new file mode 100644 index 0000000..8ebebf2 --- /dev/null +++ b/rules/denylist @@ -0,0 +1,134 @@ +# homehub denylist — DEFAULT DENY. Emitted FIRST so it beats every set include +# (rsync filter rules are first-match-wins). +# +# Syntax — plain paths, one per line, whole line is the pattern (comments on +# their own lines only): +# /Foo/ directory at the home root + its whole subtree +# build/ any directory named build at ANY depth + its subtree +# /file a single file +# The trailing slash is what turns it into a subtree rule. +# +# Measured on g9, 2026-10-24. Everything here is machine-local or rebuildable. + +# --- the big trees: class B owns repos; tmp/Downloads are out by decision --- +/git/ +/src/ +/tmp/ +/Downloads/ + +# --- toolchains and package caches (rebuilt by the toolchain, never shared) -- +/.cache/ +/.cargo/ +/.rustup/ +/.npm/ +/.nvm/ +/.bun/ +/.deno/ +/.yarn/ +/.espressif/ +/.platformio/ +/.conan/ +/.gradle/ +/.m2/ +/go/ +/zephyr-sdk-0.17.4/ +/zephyr-sdk-0.17.0/ +/.dotnet/ +/.emscripten/ +/.emscripten_cache/ +/.dartServer/ +/.dartium/ +/.android/ +/.docker/ +/.var/ +/.local/state/homehub/ +/.local/lib/ +/.local/bin/ +/.local/share/virtualenvs/ +/.venv/ +/.pixi/ + +# --- games / wine ---------------------------------------------------------- +/.steam/ +/.steamrt/ +/.local/share/Steam/ +/Steam/ +/.minecraft/ +/.wine/ +/.wine_games/ +/.wine_firefly/ +/.googleearth/ +/.adobe/ +/.freeeciv-placeholder/ + +# --- browser / editor / desktop profile state: local by default (§8) -------- +# The ONLY browser data that crosses machines is the bookmark bar (class C). +/.config/ +/.mozilla/ +/.mozilla/ +/.mozilla/ +/.thunderbird/ +/.icedove/ +/.thunderbird/ +/.thunderbird/ +/.claws-mail/ +/.icedove/ +/.electron/ +/.electron-vite/ +/.dbus/ +/.macromedia/ +/.mozilla/ +/.cache/ +/.nv/ +/.pki/ +/.lesshst/ +/.ssh/agent/ +/.claude/ +/.codex/ +/.cache/ + +# --- build artifacts (defence in depth; class B makes most of this moot) ---- +build/ +target/ +dist/ +out/ +node_modules/ +__pycache__/ +.venv/ +.pixi/ +.zig-cache/ +.ccache/ +.meson-cache/ +_deps/ +CMakeFiles/ +_results/ + +# --- filesystem metadata from other OSes / the NAS itself ------------------- +# @eaDir is Synology's per-directory metadata; without this rule a pull litters +# an @eaDir directory into every synced directory it touches. +@eaDir/ +.DS_Store +Thumbs.db +desktop.ini + +# --- build/packaging output that lives at the home root --------------------- +# ~/debian is a dpkg work area: 150 M of .deb/.dsc/.orig.tar.xz and extracted +# upstream trees. It was 86% of the first dry-run push list — build output, not +# personal data. Anchored so it only means ~/debian, not every debian/ subdir. +/debian/ + +# --- junk, locks, transient ----------------------------------------------- +.gksu.lock +.gconfd.lock +/core +/lost+found/ +*.tmp +*.swp +*.swo +*.bak +*.backup +*~ +*.log +*.lock +*.part +*.crdownload diff --git a/rules/hosts/_laptop-template b/rules/hosts/_laptop-template new file mode 100644 index 0000000..b24cbc9 --- /dev/null +++ b/rules/hosts/_laptop-template @@ -0,0 +1,5 @@ +# host: TEMPLATE for the laptop. Copy to rules/hosts/ on that machine +# (or edit this repo there, since rules/ itself travels in git). +# Deliberately NOT including: media, misc, and everything under the denylist. +base +docs diff --git a/rules/hosts/g9 b/rules/hosts/g9 new file mode 100644 index 0000000..2a396be --- /dev/null +++ b/rules/hosts/g9 @@ -0,0 +1,6 @@ +# host: g9 (desktop, always at home, no disk-space pressure) +# One set name per line; must match a file in rules/sets.d/ +base +docs +media +misc diff --git a/rules/sets.d/base.list b/rules/sets.d/base.list new file mode 100644 index 0000000..d7c518c --- /dev/null +++ b/rules/sets.d/base.list @@ -0,0 +1,42 @@ +# set: base — the minimum every machine (laptop included) syncs both ways. +# Plain home-relative paths; trailing slash = directory subtree. +# Remember: denylist wins, so anything under /.config/ can never appear here (§8). + +# identity + credentials (accepted in decision D1: the hub then holds every key +# you own, which is why the hub copy is forced 700/600 and the NAS must stay LAN-only) +/.ssh/ +/.gitconfig +/.gitignore_global +/.hgrc + +# shell / terminal +/.bashrc +/.bash_profile +/.bash_logout +/.profile +/.inputrc +/.screenrc +/.tmux.conf +/.p10k.zsh +/.vimrc +/.alsoftrc +/.gtk-bookmarks + +# personal scripts +/bin/ + +# pi agent STATE only — never install/ bin/ npm/ (§8: state vs config vs cache) +/.pi/agent/sessions/ +/.pi/agent/settings.json +/.pi/agent/models.json +/.pi/agent/models-store.json +/.pi/agent/AGENTS.md +/.pi/agent/extensions/ +/.pi/agent/prompts/ +/.pi/agent/skills/ + +# homehub's shared state root: the ONE directory whose contents are shared +# verbatim, and the anchor that survives the NixOS migration unchanged (§8). +# It is deliberately NOT ~/.local/state/homehub — that is homehub's own private +# scratch (filters, markers, error logs), and syncing it makes machines fight. +/.local/state/shared/ diff --git a/rules/sets.d/docs.list b/rules/sets.d/docs.list new file mode 100644 index 0000000..21562e8 --- /dev/null +++ b/rules/sets.d/docs.list @@ -0,0 +1,2 @@ +# set: docs — real documents. On the laptop too (decision D4). +/Documents/ diff --git a/rules/sets.d/media.list b/rules/sets.d/media.list new file mode 100644 index 0000000..e5d12f6 --- /dev/null +++ b/rules/sets.d/media.list @@ -0,0 +1,2 @@ +# set: media — desktops only for now; 323 M and grows sideways. +/Pictures/ diff --git a/rules/sets.d/misc.list b/rules/sets.d/misc.list new file mode 100644 index 0000000..60cad04 --- /dev/null +++ b/rules/sets.d/misc.list @@ -0,0 +1,7 @@ +# set: misc — small hand-written trees that are not git repos. +# Anything here that later becomes a repo must be REMOVED from this set and picked +# up by `homehub git inventory` instead (§6). +/journal/ +/projects/ +/mu_code/ +/public_html/ diff --git a/systemd/homehub-fresh.service b/systemd/homehub-fresh.service new file mode 100644 index 0000000..c012394 --- /dev/null +++ b/systemd/homehub-fresh.service @@ -0,0 +1,12 @@ +[Unit] +Description=homehub freshness check: alerts only when a live machine has not synced for 24 h + +[Service] +Type=oneshot +# HH_QUIET is deliberately NOT set: the ok/stale lines belong in the journal, and +# alert_send() uses DISPLAY+notify-send to reach the desktop when there is one. +ExecStart=%h/homehub/bin/homehub fresh +TimeoutStartSec=5min + +[Install] +WantedBy=default.target diff --git a/systemd/homehub-fresh.timer b/systemd/homehub-fresh.timer new file mode 100644 index 0000000..2c35bf7 --- /dev/null +++ b/systemd/homehub-fresh.timer @@ -0,0 +1,12 @@ +[Unit] +Description=homehub freshness check (half-hourly; the only thing that may complain) + +[Timer] +OnBootSec=2min +OnUnitActiveSec=30min +RandomizedDelaySec=2min +AccuracySec=1min +Persistent=true + +[Install] +WantedBy=timers.target diff --git a/systemd/homehub-sync.service b/systemd/homehub-sync.service new file mode 100644 index 0000000..af09430 --- /dev/null +++ b/systemd/homehub-sync.service @@ -0,0 +1,19 @@ +[Unit] +Description=homehub home-drive sync (rsync to NAS hub, mtime-arbitrated) +After=network-online.target +Wants=network-online.target +Conflicts=sleep.target + +[Service] +Type=oneshot +# A background home sync must never make the machine stutter. +Nice=19 +IOSchedulingClass=idle +# Silent on success is a requirement, not an oversight (DESIGN.md §10). +Environment=HH_QUIET=1 +ExecStart=%h/homehub/bin/homehub sync --exec +TimeoutStartSec=3h +# No retry storm: the next timer tick is the retry. + +[Install] +WantedBy=default.target diff --git a/systemd/homehub-sync.timer b/systemd/homehub-sync.timer new file mode 100644 index 0000000..cbe8a36 --- /dev/null +++ b/systemd/homehub-sync.timer @@ -0,0 +1,12 @@ +[Unit] +Description=homehub home-drive sync cycle (every 10 min, catches up after suspend) + +[Timer] +OnBootSec=5min +OnUnitActiveSec=10min +RandomizedDelaySec=3min +AccuracySec=1min +Persistent=true + +[Install] +WantedBy=timers.target