meshbay.git/packages/meshbay-node/tests/test_daemon.py, branch 0.10

feat(node): share the (path,size,mtime)->hash index cache across every group

2026-08-25T22:40:20Z

An operator routinely shares the same physical folder into more than one group (a music library, a Séries drive) — IndexCache used to be opened once per group (data_dir/{group_id}/index_cache.db), so the second group to reference an already-fully-hashed multi-terabyte folder paid the same full content read the first one did. IndexCache itself carried no group_id in its schema; only daemon.py's wiring did. Now one instance, opened once at startup (data_dir/index_cache.db), shared by every group's DirectoryIndexer. Confirmed against a real deployment (2026-08-25/26): a group sharing an already-indexed folder with an existing group indexes it instantly, with zero rehashing. Also fixes a related cross-group correctness gap found during this work: media_cache.db (thumbnails, TMDB/MusicBrainz metadata — already node-wide, untouched by this change) was pruned for a file the moment it left *one* group's index, even if another group's index still held the same content hash — forcing a redundant re-fetch/re-probe/re-thumbnail for a group that never actually lost anything. Prune now runs only once no group's index references the file_id any more. Adds a node admin UI action ("Maintenance" card, prune-index-cache) to drop cache rows that no longer belong to any group's roots — skips anything under a root that is merely temporarily unavailable (indexer.py's "a root that goes away freezes, never empties" rule extends to this cache too, or a reconnected drive would pay a full rehash for no reason). Co-Authored-By: Claude Sonnet 5 Claude-Session: https://claude.ai/code/session_013XSohfUQQiaE77qyFLgSv3

feat(node): persistent index cache, visible scan progress, adaptive reconcile, and delta sync

2026-08-23T19:55:20Z

Indexer performance work, in four parts: - Persistent (path, size, mtime) -> hash cache (indexer/cache.py) so a node restart no longer re-hashes every file — measured at 23 minutes for a 114 GB library on a slow disk before this, near-instant after. Hashing is deliberately kept sequential (max_workers=1): it was never actually concurrent despite the pool size, and two interleaved reads seek-thrash a spinning disk instead of going faster. - Byte-based scan progress (IndexProgress), surfaced via the loopback index-status route, the handshake ack, and a periodic INDEX_PROGRESS push to connected peers — drives a progress bar in the Create Group wizard and "add a directory" in Settings, and an animated presence dot. Guaranteed to settle back to idle via try/finally and a final push on the scanning->false transition. - The reconcile backstop's directory walks now run in the executor instead of blocking the daemon's event loop; its interval defaults to 10 min (was 60s) with adaptive backoff to 2h when nothing changes, reset on a real change or a peer connecting, and is now a per-group operator setting (signed op + group Settings UI). - INDEX_DELTA wired up (protocol support existed, nothing called it): _on_index_change now sends additions/deletions instead of rebuilding the full entries list, coalesced over a short window so a burst of file events produces one push, and the hub swarm registration for public groups only (re-)registers newly added hashes. Also fixes several bugs found while testing the above against real libraries (a 114 GB and a 100+ GB group on a USB HDD): - /api/reload blocked until the reload — including a brand-new group's full initial scan — finished, which the Electron bridge's fixed 30s call timeout turned into a hard failure on any real library. The route now fires the reload without waiting (ops.start_reload), matching add_root/remove_root's existing pattern; the wizard's own step order was fixed to wait for the group to actually appear hosted before the steps that need it (extra roots, GEK), with retries for the residual race between that and the daemon's own bookkeeping. - transport.js's hand-rolled msgpack codec had no case for uint64/int64 (0xcf/0xd3) and crashed decoding any message containing one — hit by IndexProgress.scanned_bytes/total_bytes for any group over ~4.3 GB. Verified against real msgpack-encoded bytes from the Python side. - chat_hist_resp, and this change's own index_progress and set_scan_settings_ack pushes, were not routed by message type and could be handed to an unrelated pending request by the transport's "oldest pending" fallback, stalling it until its own 30s timeout and corrupting whatever received the wrong reply in its place. Co-Authored-By: Claude Sonnet 5 Claude-Session: https://claude.ai/code/session_016SF6RKNBKg9qejmoMJ9ybA

fix: first-run wizard reliability and node startup performance

2026-08-22T14:40:54Z

Node daemon no longer blocks startup on slow directory scans — initial indexing runs in the background so the node reaches "running" immediately after transports are up. Fixes the wizard failing to detect the node when large USB/NAS roots take minutes to scan. Also: wizard key-linking deadlock resolved (main.js links during poll), invite form stays in DOM during reconnects (disabled instead of destroyed), pairing code bridges to renderer, and firewall docs for LAN casting added. Co-Authored-By: Claude Opus 4.6

feat(node): several named roots per group, and one implementation per operation

2026-08-18T00:15:02Z

Stage A — a group's content is a set of named roots --------------------------------------------------- `shared_dir` becomes a list of {name, path, kind}. The name is the directory's basename, derived once at add time and *stored*: recomputing it would re-identify a whole library the day someone renames a folder on disk. Duplicate names are refused case-insensitively and no root may contain another — both compared with NFC folding, because most of these directories live on exFAT or NTFS where `Films` and `films` are one directory. Every index path carries its root name, in a one-root group as much as in a five-root one. One path shape has to be got right once; two have to be kept right for ever. **A root that goes away freezes; it never empties.** Unmounting a volume makes watchdog report every file under it as deleted, or presents an empty directory to the next scan. Acting on either propagates deletions for a whole library to every member, as though the owner had erased it. So a deletion is acted on only once its root is confirmed readable, and availability is tracked per root — one unplugged drive leaves the others serving. 12 tests, verified to fail against an indexer without the check. Events are not trusted to be complete either: ReadDirectoryChangesW drops them under load and inotify on a FUSE mount misses changes made outside it. A periodic reconciliation sweep is the only thing that recovers a missed event. MNP 0.2 → 0.3 (additive). The hub needs no change: SwarmSource carries a content hash, a node id and an endpoint — no paths, no filenames — and private groups register nothing (H7). Stage B — one implementation behind every front door ---------------------------------------------------- C1 and C6 were both "a second path into the node with its own weaker handshake". Two implementations of `revoke` with two authorization checks is that shape one size down. `meshbay_node/ops.py` holds each operation once, takes the daemon state, and knows nothing about HTTP, argv or MNP. The loopback API is one `_op(...)` line per endpoint; the MNP handlers call the same functions. test_ops.py asserts the shape rather than trusting it. Phase 14 is finished on top of it — `group list`, `gek init|rotate`, `reload` (SIGHUP), `denylist show|clear`, `file list|rm`. **No operator action requires a browser any more.** Plus `gek_rotate` and `member_unpin` as operator-signed MNP operations: rotation is the half of revocation that revocation cannot do, since the ex-member holds the current key, and the node generates the replacement with its own CSPRNG — no key material crosses the wire, which is what the C5b rule is actually about. Two bugs found by running it rather than by testing it ------------------------------------------------------ GroupIndex is keyed by **content hash**, so the same bytes at two paths are one entry — which is also why a scan reports ten files and indexes nine. Reconciliation compared paths, so it decided the second path was a missed event every 60 s, rewrote the entry and pushed an index update to every connected peer. Seen in a live node's log. `meshbay-node reload` crashed on first use with `subprocess` unimported: the module compiles fine, which is the "syntax, not names" trap already recorded for the SPA. test_cli_dispatch.py now walks every verb and refuses to let one be added to the parser without an entry there. Also corrected: protocol.py declared a second MNP_VERSION of "0.1" while the wire carried "0.2" — harmless only because nothing imported it. And _do_dir_create/_do_dir_delete referenced an undefined `filename` on their error path. 740 tests pass; QE/deploy/e2e.py passes end to end against the live deployment. Co-Authored-By: Claude Opus 5

test(node): allocate daemon test ports instead of hardcoding 28000

2026-08-13T23:25:10Z

test_daemon.py started the real admin UI on a fixed port, so every test file that also brought up a node collided with it. Each file passed on its own and the full node suite failed with EADDRINUSE on test_daemon_creates_chat_store — which reads as a flaky regression rather than a test-isolation bug. Confirmed against a clean worktree at HEAD before touching anything: the failure predates the invite work. Co-Authored-By: Claude Opus 5

fix: swarm privacy, revocation persistence, keystore KDF, audit integrity

2026-08-13T09:00:39Z

Phase 11.5 hardening batch — H7, H4, M2, M6, M7, L1, L3, L6. H7 — private content hashes leaked to the hub. The daemon registered blake3 hashes for every group it hosted, private ones included, giving the hub a content fingerprint of every private file and letting anyone confirm whether a known file exists in the network. The leak was dormant only because the routes were declared on the groups router with a full path and mounted at /v1/groups/v1/swarm/* — the node's calls 404'd into a swallowed exception. Fixing the path alone would have activated the leak, so both land together: registration is gated on group visibility, the routes moved to a real /v1/swarm router, and the lookup now requires authentication. H4 — revocation was advisory. Group revocations were signed and broadcast by the hub and then dropped by the node, whose handler understood only "user" and "jti", so "suspend a group" enforced nothing. The denylist was also in-memory only, so a restart silently un-revoked everyone. Now persisted to data_dir/denylist.json, group targets honoured on both transports, and live sessions for a revoked group are closed. M2 — the node keystore, which protects the node's Ed25519 and X25519 private keys, was still deriving at 64 MB long after the hub's password verifier moved to 256 MB; the docs recorded the bump as done, true for the hub only. Raising the constant alone would have made every existing keystore permanently undecryptable, so envelopes now record the parameters they were written with and pre-M2 files continue to open under the legacy profile. M6 — registration inserted its audit row with a NULL user_id and then ran UPDATE ip_logs SET user_id= WHERE user_id IS NULL, claiming every unattributed row in the table: failed logins for other usernames, concurrent registrations. In logs retained a year for legal requests, that attributed other people's connections to the wrong account. M7 — X-Forwarded-For was trusted unconditionally at four call sites, so anyone could forge the IP written to the compliance log and evade per-IP rate limits. New netutil.client_ip honours the header only from a trusted proxy and takes the rightmost hop (the one our proxy appended); no direct header reads remain. L1 dead GEK_REQUEST/GEK_RESPONSE constants removed; L3 peer errors no longer echo exception text (paths, internal state); L6 email sanity-checked instead of accepting any string — deliberately not RFC 5322, to avoid a new dependency. test_daemon_index_change_pushes_to_peers asserted that a PRIVATE group's hashes are registered with the hub. Split: private asserts not-called (index push to members still asserted), and a new test proves public groups still register. That is the fourth pre-existing test found asserting a vulnerability as intended behaviour, after gek auto-activation, the transport-wide chat_store and the blind admin challenge. Tests: 116 node, 132 hub+common. Regression suite now 43. Co-Authored-By: Claude Opus 5

fix(node): group isolation, upload confinement, GEK seizure, admin challenge

2026-08-13T08:43:16Z

Phase 11.5 — findings H1, C5a, H2, C5b, H5 (see second-review.md). Batched together because the node-side changes share webrtc_server.py and cannot be separated into working commits. H1 — cross-group chat leak. chat_store, the peer registry and the display-name cache were read from the shared transport context, and daemon.py hoisted the FIRST group's chat store onto it. On a node hosting several groups every group's messages went to one database, chat_history served them back to members of every other group, and chat broadcast reached all peers regardless of group. All three now resolve through _group_ctx(). C5a — upload confinement. Uploads landed in the shared root under a client-chosen name and overwrote whatever was there. Any member could destroy the operator's files, and by becoming the recorded uploader of the replaced file could then delete it through the uploader path, bypassing the Ed25519 admin challenge. Uploads now go to a per-user quarantine (.uploads/{user_id}/), refuse to overwrite, and enforce chunk ordering, a filename allowlist and a size cap. H2 — stored XSS in the node admin UI. Filenames chosen by any group member were interpolated raw into the localhost UI, which has no authentication, so script execution there equals control of the node admin API. Now html.escape() throughout, textContent in the audit table, plus CSP/nosniff/no-referrer. The CSP contains exfiltration but cannot stop injected inline script — escaping is the fix. C5b — group key seizure. gek_bundle_store wrote whatever any member sent and auto-activated bundles addressed to the node operator. The operator's X25519 public key is public (the node publishes it in handshake_ack), so any member could wrap a key of their choosing for it and take over the group, locking every legitimate member out. Storing now requires an operator signature and _try_activate_gek is removed: nothing arriving over MNP can set a live GEK. H5 — unbound signing oracle. The node challenged with 32 raw random bytes and the client signed them blind, so a signature named no operation, subject, node or time. New meshbay_common/adminop.py defines a length-prefixed, domain-separated transcript; both sides build it independently and the client refuses to sign when the announced op/subject do not match its request. BREAKING: a group admin who does not operate the node can no longer store GEK bundles on it. Invites must be performed by the node operator. Adds tests/test_security_regressions.py. Verified against pre-fix source via git stash. Three pre-existing tests asserted the vulnerable behaviour as a feature and were inverted: gek auto-activation, and the transport-wide chat_store in test_daemon. Tests: 109 node, 132 hub+common. Co-Authored-By: Claude Opus 5

fix(node)!: remove unauthenticated HTTP file API and TCP transport

2026-08-13T02:10:14Z

Phase 11.5.A — findings C1 and C6 (see second-review.md). C1: the per-group HTTP file API bound 0.0.0.0 for every configured group, private ones included, and served two endpoints with no authentication at all: GET /index (full Mesh Group Index) and GET /file/{id} (raw plaintext file via FileResponse). Anyone able to reach the port — LAN, forwarded port, permissive IPv6 — read every private file. This bypassed the entire GEK-proof and node sovereignty layer. Deleted rather than patched: it duplicated MNP without any of its controls. C6: the TCP+TLS chunk server accepted a bare JWT with no GEK proof, leaving a second non-compliant handshake path. Deleted; QUIC remains and will be brought to parity with WebRTC by the unified handshake in 11.5.4. Transport decision recorded in transport/__init__.py: WebRTC/ICE is primary for browser and native clients (the only NAT traversal validated here — 2 ISPs, IPv4 STUN + IPv6, 4G CGNAT); QUIC is kept for LAN, port-forwarded and hub-less group:// access. punch_nat() is a direct-connection helper, not a traversal stack. Also removed server_ssl_context()/client_ssl_context() from tls_cert.py (no remaining callers) and a dead import of the former in quic_server.py. generate_self_signed_cert() stays: QUIC uses it, and the certificate hash is the intended channel-binding anchor for 11.5.6, since QUIC has no DTLS fingerprint to bind the GEK proof to. BREAKING CHANGE: node.toml keys `port` and `http_port` are gone. Regenerate config with `meshbay-node init`. Env var MESHBAY_PORT -> MESHBAY_QUIC_PORT. Tests: 198 passed (209 - 7 test_http_server - 4 test_transport). No other test changed status. Net -1300 lines. Co-Authored-By: Claude Opus 5

feat: Phase 12 — P2P crypto material, password split, node Ed25519 auth

2026-08-13T01:56:30Z

Baseline commit capturing in-progress Phase 12 work that was already present in the working tree (uncommitted) before the Phase 11.5 security remediation begins. Committed as-is, without review or modification, so that remediation changes arrive as a separable diff. Contents: BundleStore (P2P GEK + keypair bundles), password split (auth_key / bundle_key), node Ed25519 auth (POST /v1/nodes/auth, node-scoped JWT), GEK-HMAC handshake proof with DTLS channel binding, Ed25519 admin challenge-response, node local admin UI rewrite, browser key persistence. Not authored in this session — captured to establish a baseline. Co-Authored-By: Claude Opus 5

feat(node): index push to WebRTC peers + swarm registration (11.5, 11.9)

2026-08-11T20:16:59Z

When watchdog detects file changes, the daemon now: - Pushes INDEX_SYNC to all connected WebRTC peers in that group - Registers file hashes with hub /v1/swarm/register endpoint Also registers all file hashes on startup for initial discovery. hub_client: add register_swarm() method for bulk hash registration. Co-Authored-By: Claude Opus 4.6