feat(node): persistent index cache, visible scan progress, adaptive reconcile, and delta sync

Indexer performance work, in four parts:

- Persistent (path, size, mtime) -> hash cache (indexer/cache.py) so a node
  restart no longer re-hashes every file — measured at 23 minutes for a
  114 GB library on a slow disk before this, near-instant after. Hashing
  is deliberately kept sequential (max_workers=1): it was never actually
  concurrent despite the pool size, and two interleaved reads seek-thrash
  a spinning disk instead of going faster.

- Byte-based scan progress (IndexProgress), surfaced via the loopback
  index-status route, the handshake ack, and a periodic INDEX_PROGRESS
  push to connected peers — drives a progress bar in the Create Group
  wizard and "add a directory" in Settings, and an animated presence dot.
  Guaranteed to settle back to idle via try/finally and a final push on
  the scanning->false transition.

- The reconcile backstop's directory walks now run in the executor
  instead of blocking the daemon's event loop; its interval defaults to
  10 min (was 60s) with adaptive backoff to 2h when nothing changes,
  reset on a real change or a peer connecting, and is now a per-group
  operator setting (signed op + group Settings UI).

- INDEX_DELTA wired up (protocol support existed, nothing called it):
  _on_index_change now sends additions/deletions instead of rebuilding
  the full entries list, coalesced over a short window so a burst of
  file events produces one push, and the hub swarm registration for
  public groups only (re-)registers newly added hashes.

Also fixes several bugs found while testing the above against real
libraries (a 114 GB and a 100+ GB group on a USB HDD):

- /api/reload blocked until the reload — including a brand-new group's
  full initial scan — finished, which the Electron bridge's fixed 30s
  call timeout turned into a hard failure on any real library. The route
  now fires the reload without waiting (ops.start_reload), matching
  add_root/remove_root's existing pattern; the wizard's own step order
  was fixed to wait for the group to actually appear hosted before the
  steps that need it (extra roots, GEK), with retries for the residual
  race between that and the daemon's own bookkeeping.
- transport.js's hand-rolled msgpack codec had no case for uint64/int64
  (0xcf/0xd3) and crashed decoding any message containing one — hit by
  IndexProgress.scanned_bytes/total_bytes for any group over ~4.3 GB.
  Verified against real msgpack-encoded bytes from the Python side.
- chat_hist_resp, and this change's own index_progress and
  set_scan_settings_ack pushes, were not routed by message type and
  could be handed to an unrelated pending request by the transport's
  "oldest pending" fallback, stalling it until its own 30s timeout and
  corrupting whatever received the wrong reply in its place.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_016SF6RKNBKg9qejmoMJ9ybA

1 files changed, 119 insertions, 0 deletions

diff --git a/packages/meshbay-hub/src/meshbay_hub/static/transport.js b/packages/meshbay-hub/src/meshbay_hub/static/transport.js
index da98253..4f6b656 100644
--- a/packages/meshbay-hub/src/meshbay_hub/static/transport.js
+++ b/packages/meshbay-hub/src/meshbay_hub/static/transport.js
@@ -103,8 +103,10 @@ class MeshBayTransport {
   set onStreamEnd(fn) { this._onStreamEnd = fn; }
   set onStreamError(fn) { this._onStreamError = fn; }
   set onIndexSync(fn) { this._onIndexSync = fn; }
+  set onIndexDelta(fn) { this._onIndexDelta = fn; }
   set onUploadPolicy(fn) { this._onUploadPolicy = fn; }
   set onAppsEnabled(fn) { this._onAppsEnabled = fn; }
+  set onIndexProgress(fn) { this._onIndexProgress = fn; }
 
   get sessionKeys() { return this._sessionKeys; }
 
@@ -634,6 +636,29 @@ class MeshBayTransport {
     return msg;
   }
 
+  /**
+   * How often the node's reconciliation backstop runs, and how long it
+   * waits after a file's last write before hashing it (indexer.py
+   * DirectoryIndexer). Whole seconds only: the node builds the signing
+   * subject with Python's `%g` (drops a trailing ".0"), and the simplest
+   * way to always match it byte-for-byte from JS is to never send a
+   * fractional value in the first place.
+   */
+  async setScanSettings(reconcileIntervalSecs, debounceSecs, signFn) {
+    const reconcile = Math.round(reconcileIntervalSecs);
+    const debounce = Math.round(debounceSecs);
+    const msg = await this._sendAndWait({
+      type: 'set_scan_settings', v: '0.1',
+      reconcile_interval_secs: reconcile, debounce_secs: debounce,
+    });
+    if (msg.type === 'error') throw new Error(msg.detail);
+    if (msg.type === 'admin_challenge') {
+      return this._authorizeAdminOp(
+        msg, 'set_scan_settings', `${reconcile},${debounce}`, signFn);
+    }
+    return msg;
+  }
+
   async revokeMember(userId, signFn) {
     const msg = await this._sendAndWait({
       type: 'member_revoke', v: '0.1', user_id: userId,
@@ -1251,6 +1276,37 @@ class MeshBayTransport {
       this._onAppsEnabled(msg.apps || []);
     }
 
+    // The operator's node is scanning — never the entries themselves, just
+    // enough to animate a presence dot. Pushed periodically while it runs,
+    // plus once more on the transition back to idle (daemon.py
+    // _progress_pusher). UNLIKE member_upload_ack/apps_enabled_ack above,
+    // this is never a reply to anything this browser asked for — nobody
+    // calls _sendAndWait for it — so it MUST return here. Falling through
+    // to the "oldest pending" guess below hands it to whatever unrelated
+    // request happens to be waiting (a handshake, a chat history fetch),
+    // which then waits forever for its real answer while this one already
+    // "arrived" — and every message after that is one slot off too. Found
+    // live: a group mid-scan corrupted its own handshake and chat history
+    // this way, arriving roughly every 2s for as long as scanning ran.
+    if (msg.type === 'index_progress') {
+      if (this._onIndexProgress) {
+        this._onIndexProgress({
+          scanning: Boolean(msg.scanning),
+          scanned_bytes: msg.scanned_bytes || 0,
+          total_bytes: msg.total_bytes || 0,
+        });
+      }
+      return;
+    }
+
+    // Same reasoning as index_progress: nobody awaits this one either, it
+    // is purely informational (group-settings.js does not currently act on
+    // it), so it must not be left to fall through to the oldest pending
+    // request.
+    if (msg.type === 'set_scan_settings_ack') {
+      return;
+    }
+
     if (msg.type === 'index_sync' && msg.entries) {
       if (this._onIndexSync) this._onIndexSync(msg);
       const oldest = this._pending.entries().next();
@@ -1260,6 +1316,16 @@ class MeshBayTransport {
       return;
     }
 
+    // Incremental update — additions/deletions only, never the whole index.
+    // Only ever arrives after the full index this browser already has (the
+    // node's first push to a newly connected peer is always index_sync, see
+    // daemon.py _broadcast_index_change), so there is always a base to
+    // apply it to.
+    if (msg.type === 'index_delta') {
+      if (this._onIndexDelta) this._onIndexDelta(msg);
+      return;
+    }
+
     if (msg.type === 'file_chunk') {
       const key = `chunk:${msg.file_id}:${msg.chunk_index}`;
       for (const [, handler] of this._pending) {
@@ -1298,6 +1364,29 @@ class MeshBayTransport {
       return;
     }
 
+    // chat_hist_resp answers a `chat_hist` request, but under a different
+    // type string — unlike index_sync, which is asked for and answered under
+    // the same name, so the generic fallback below happens to work for it by
+    // accident. Without this check, whenever a chat_hist_resp arrives while
+    // something else this browser asked for (fetchIndex, even the handshake
+    // itself) is still the oldest pending entry, it gets handed to that
+    // instead: the request chat_hist_resp actually belongs to then hangs
+    // until _sendAndWait's own 30s timeout, and whatever it stole from
+    // resolves with the wrong shape entirely — reproduced live as a
+    // consistent ~30s hang immediately after a successful handshake, for one
+    // specific group and not others connected the same way, which is exactly
+    // what depending on response arrival order rather than on request type
+    // predicts: it fires only when the two responses happen to reorder.
+    if (msg.type === 'chat_hist_resp') {
+      const oldest = this._pending.entries().next();
+      if (!oldest.done && oldest.value[1]._reqType === 'chat_hist') {
+        oldest.value[1].resolve(msg);
+      } else {
+        console.warn('[MeshBay] chat_hist_resp with no matching chat_hist pending');
+      }
+      return;
+    }
+
     // Everything above is routed by something in the message. What is left is
     // matched by arrival order, which is only ever a guess — and a wrong guess
     // here hands one request's answer to another, which then waits for a reply
@@ -1349,6 +1438,16 @@ function _encodeValue(val, parts) {
         const b = new Uint8Array(5); b[0] = 0xce;
         new DataView(b.buffer).setUint32(1, val, false);
         parts.push(b);
+      } else if (val >= 0 && val <= Number.MAX_SAFE_INTEGER) {
+        // Same split as the 0xcf decoder case above, in reverse — without
+        // this, a value over 0xffffffff fell to the plain int32 branch
+        // below and silently wrapped to a wrong, unrelated number instead
+        // of failing loudly.
+        const b = new Uint8Array(9); b[0] = 0xcf;
+        const dv = new DataView(b.buffer);
+        dv.setUint32(1, Math.floor(val / 4294967296), false);
+        dv.setUint32(5, val % 4294967296, false);
+        parts.push(b);
       } else if (val >= -32 && val < 0) {
         parts.push(new Uint8Array([val & 0xff]));
       } else if (val >= -128 && val < 0) {
@@ -1460,6 +1559,26 @@ function _decodeValue(buf, view, offset) {
     case 0xcc: return [buf[offset + 1], offset + 2];
     case 0xcd: return [view.getUint16(offset + 1, false), offset + 3];
     case 0xce: return [view.getUint32(offset + 1, false), offset + 5];
+    // uint64/int64 — never emitted by this file's own encoder (a JS number
+    // above 0xffffffff falls to float64 there), but the node's real msgpack
+    // library sends a plain uint64 for any Python int over ~4.3 billion, and
+    // a raw byte count crosses that easily (found live: IndexProgress.
+    // scanned_bytes/total_bytes in the handshake ack, indexer.py, for a
+    // group whose total library size exceeds ~4 GB). Split into two 32-bit
+    // halves rather than DataView's getBigUint64/getBigInt64 — a BigInt
+    // would silently poison every arithmetic use of these fields elsewhere
+    // (percentage math, comparisons) — and every real byte count fits in a
+    // plain JS number well under Number.MAX_SAFE_INTEGER (2^53).
+    case 0xcf: {
+      const hi = view.getUint32(offset + 1, false);
+      const lo = view.getUint32(offset + 5, false);
+      return [hi * 4294967296 + lo, offset + 9];
+    }
+    case 0xd3: {
+      const hi = view.getInt32(offset + 1, false);
+      const lo = view.getUint32(offset + 5, false);
+      return [hi * 4294967296 + lo, offset + 9];
+    }
     case 0xcb: return [view.getFloat64(offset + 1, false), offset + 9];
     case 0xd0: return [view.getInt8(offset + 1), offset + 2];
     case 0xd1: return [view.getInt16(offset + 1, false), offset + 3];