path: root/docs/indexing-v2.md

# Indexing v2 — Partial-read hashing for large files

> **Superseded by `MESHBAY_DESIGN.md`.** This was the partial-read hashing; its design
> content now lives in §6.3.
>
> It is kept because code comments, tests and other documents cite its
> sections and its labels, and because it records reasoning a synthesis
> compresses. **Where it disagrees with `MESHBAY_DESIGN.md`, the design
> document is right; where either disagrees with the code, the code is.**
> `MESHBAY_DESIGN.md` §16 maps every section reference here onto its
> replacement, and §13 defines every label.

> Status: **built.** `hash_version` and partial-read hashing are in the indexer,
> the protocol and the hash cache. Kept as the decision record and for the
> migration notes; this header said "not built" long after it shipped.

---

## 0. Problem

The current indexer reads every file in full to compute its blake3 content hash (`id`).
For a large media library (multi-terabyte, thousands of files), this means:

- **Time**: initial indexing takes tens of minutes to hours.
- **Disk I/O**: every byte of every file is read, which wears SSDs and saturates spinning
  drives for the entire duration. A USB hard drive serving a 4 TB library is pegged for
  over an hour.
- **Blocking**: no connected peer receives a usable index until the full scan finishes.

The hash exists for **content identity** (deduplication, cross-group search, file
requests). A 4 GB film does not need 4 GB of I/O to be identified with overwhelming
probability — 45 MB of well-chosen samples suffice.

---

## 1. Design

### 1.1 Hashing rules

| File size          | Method                                              | `hash_version` |
|--------------------|-----------------------------------------------------|-----------------|
| <= 40 MB           | Full read, blake3 of entire content (unchanged)     | `1`             |
| > 40 MB            | Partial read, blake3 of 45 MB sampled (see below)   | `2`             |

**Partial-read algorithm (hash_version 2):**

Given a file of `S` bytes where `S > 40 MB`:

1. Read the first **20 MB** (bytes `[0, 20 MB)`).
2. Append the last **20 MB** (bytes `[S - 20 MB, S)`).
3. Append **5 MB** starting at **50% of the file** (bytes `[S // 2, S // 2 + 5 MB)`).
4. Compute `blake3(concatenation of the three regions)`.

The three regions may overlap for files just above 40 MB. This is fine — the concatenation
is deterministic for a given file, which is the only property that matters.

**Why these offsets.** Head and tail catch container headers, trailers, and the common case
of files that differ only at one end (re-encoded, re-muxed, appended). The mid-sample
catches files that share a header and trailer but differ in content (same container,
different media stream).

**Why 40 MB threshold.** Below 40 MB the partial read would sample the entire file anyway
(head + tail >= file size), so the full-read path is both simpler and produces the same
result. The boundary is inclusive: a 40 MB file is read in full.

### 1.2 `hash_version` field

A new field on `IndexEntry`:

```
hash_version: int = 1
```

- `1` — the `id` is blake3 of the full file content. This is the only value any existing
  node has ever produced.
- `2` — the `id` is blake3 of the 45 MB partial sample described above.

**For files <= 40 MB on a v2 node, `hash_version` stays `1`.** The hash is identical to
what a v1 node produces, because both read the file in full. This preserves cross-group
search compatibility for small files across v1 and v2 nodes.

**For files > 40 MB on a v2 node, `hash_version` is `2`.** The hash is different from
what a v1 node would produce for the same file. This is the accepted side effect.

### 1.3 Backward compatibility

| Scenario | Behaviour |
|---|---|
| v2 node sends `hash_version` to v1 client | Client ignores unknown field (JS objects are open) |
| v1 node sends entries without `hash_version` | Client/consumer treats it as `1` (dataclass default) |
| v2 `IndexEntry(**e)` where `e` lacks `hash_version` | Uses default `1` — existing serialized indexes deserialize correctly |
| Cross-group search: same file, one node v1, one node v2 | Different `id` for files > 40 MB — not merged. Accepted |
| Cross-group search: same small file, mixed nodes | Same `id` (both `hash_version=1`) — merged correctly |
| Hub tables (`swarm_sources`, `content_blocklist`, `content_reports`) | Store `content_hash` as an opaque string. No change needed |
| `GroupIndex.serialize()` / `deserialize()` | `asdict(e)` includes `hash_version`; `IndexEntry(**e)` with default handles missing field |

**Nothing breaks.** A v1 node's data remains valid. A v2 node produces correct new hashes.
Mixed v1/v2 environments work, with the documented search side effect.

---

## 2. Affected components

### 2.1 `meshbay_common` — `protocol.py`

| Change | Detail |
|---|---|
| `IndexEntry` dataclass | Add `hash_version: int = 1` field |
| `index_entry_wire()` | Add `"hash_version": e.hash_version` to the wire dict |

### 2.2 `meshbay_node` — `indexer/indexer.py`

| Change | Detail |
|---|---|
| Constants | `_PARTIAL_THRESHOLD = 40 * 1024 * 1024`, `_PARTIAL_HEAD = 20 * 1024 * 1024`, `_PARTIAL_TAIL = 20 * 1024 * 1024`, `_PARTIAL_MID = 5 * 1024 * 1024` |
| `_scan_file()` | After stat, if `size > _PARTIAL_THRESHOLD`: use partial-read blake3. Set `hash_version=2` on the returned `IndexEntry`. Otherwise: unchanged (full read, `hash_version=1`) |
| `_hash_or_cached()` | Compute expected `hash_version` from file size. Pass it to cache `lookup()`. Store it in cache `put()`. Set it on the returned `IndexEntry` |

**`_scan_file` partial-read implementation:**

```python
def _partial_hash(file_path: Path, size: int) -> str:
    hasher = blake3.blake3()
    with open(long_path(file_path), "rb") as f:
        _feed(hasher, f, _PARTIAL_HEAD)
        f.seek(size - _PARTIAL_TAIL)
        _feed(hasher, f, _PARTIAL_TAIL)
        f.seek(size // 2)
        _feed(hasher, f, _PARTIAL_MID)
    return hasher.hexdigest()

def _feed(hasher, f, nbytes: int) -> None:
    remaining = nbytes
    while remaining > 0:
        chunk = f.read(min(_HASH_CHUNK, remaining))
        if not chunk:
            break
        hasher.update(chunk)
        remaining -= len(chunk)
```

### 2.3 `meshbay_node` — `indexer/cache.py`

| Change | Detail |
|---|---|
| Schema | Add `hash_version INTEGER NOT NULL DEFAULT 1` column to `files` table |
| `_SCHEMA` | New databases get the column via `CREATE TABLE` |
| `_MIGRATE_V2` | `ALTER TABLE files ADD COLUMN hash_version INTEGER NOT NULL DEFAULT 1` — applied on `open()` if the column does not exist |
| `lookup()` | Add `hash_version` parameter. WHERE clause becomes `path = ? AND size = ? AND mtime = ? AND hash_version = ?` |
| `put()` | Add `hash_version` parameter. INSERT includes `hash_version` |
| `CachedEntry` | Add `hash_version: int` field |

**Auto-migration on open:** `IndexCache.open()` runs the ALTER TABLE inside a try/except
(column already exists → no-op). This way a node upgrade just works — no manual step
needed for the cache.

### 2.4 `meshbay_node` — `indexer/group_index.py`

No code change needed. `serialize()` calls `asdict(e)` which includes `hash_version`.
`deserialize()` calls `IndexEntry(**e)` which uses the default `1` for entries written
before v2.

### 2.5 `meshbay_node` — `transport/wire.py`

No code change needed. `index_sync_message()` and `index_delta_message()` call
`index_entry_wire()` which is updated in §2.1.

### 2.6 Client side (browser / desktop)

**No code changes required.** Entries are JavaScript objects; the extra `hash_version`
field is carried through without needing explicit handling:

- `transport.js` — `_applyIndexMessage()` passes the opened payload through. `hash_version`
  rides along on each entry object.
- `group-page.js` — `applyIndex()` / `applyIndexDelta()` store entries as-is in state
  and in IndexedDB.
- `hub-client.js` — IndexedDB cache stores entry objects verbatim.
- `source-merge.js` — merges by `id`. Different hashes naturally don't merge.
- `search-page.js` — aggregates entries from cached indexes. No change.
- `files-app.js` — renders entries. `hash_version` is ignored.

### 2.7 Hub side

**No code changes required.** `swarm_sources.content_hash`, `content_reports.content_hash`,
`content_blocklist.content_hash` are opaque `String(64)` columns. They store whatever
blake3 hex the node provides. No hub migration needed.

### 2.8 Tests

| Test | What it verifies |
|---|---|
| `test_indexer.py` — new cases | Partial hash for file > 40 MB produces `hash_version=2`. File <= 40 MB produces `hash_version=1`. Partial hash is deterministic. Partial hash differs from full hash for the same large file |
| `test_indexer.py` — existing cases | All existing tests still pass (small files, type detection, enrichment, etc.) |
| `test_index_cache.py` — new cases | Cache lookup with `hash_version` match. Cache miss when `hash_version` differs. Schema migration from v1 cache |
| `test_index_cache.py` — existing cases | Unchanged behaviour for v1 entries |
| `test_indexer.py` — round-trip | `GroupIndex.serialize()` → `deserialize()` preserves `hash_version` on entries |
| `test_index_seal_client.py` | Wire format includes `hash_version`, old entries without it deserialize as v1 |

---

## 3. Migration

### 3.1 What actually needs migrating

The node has two relevant stores:

| Store | Location | Content | Migration |
|---|---|---|---|
| `index_cache.db` | `data_dir/index_cache.db` | `(path, size, mtime) → hash` accelerator | Add `hash_version` column |
| In-memory `GroupIndex` | rebuilt from disk on every startup | Current file listing | No migration — rebuilt on next scan |

**The IndexCache is the only persistent store that needs a schema change.** The GroupIndex
is rebuilt by scanning the filesystem on every daemon start. Once the code uses v2 hashing,
the next startup produces v2 hashes for large files automatically.

**The cache auto-migrates.** `IndexCache.open()` adds the `hash_version` column if missing.
Existing rows get `DEFAULT 1`. When the v2 indexer looks up a large file with
`hash_version=2`, the cached v1 entry won't match (different hash_version in WHERE), so
the file is re-hashed with the partial algorithm and the new entry is written with
`hash_version=2`.

This means: **large files are re-hashed lazily on first scan after upgrade.** The first
scan after upgrading to v2 re-reads 45 MB per large file instead of the full content —
already much faster than v1's full read.

### 3.2 Migration script — `QE/migration/migrate_index_v2.sh`

A standalone bash script (not in git — QE/ is gitignored) that:

1. Detects the node's `data_dir` from `node.toml` (default `~/.local/share/meshbay-node/`)
2. Checks that `index_cache.db` exists
3. Runs `ALTER TABLE files ADD COLUMN hash_version INTEGER NOT NULL DEFAULT 1`
4. Optionally (`--purge-large`) deletes cache entries for files > 40 MB, forcing immediate
   re-hash on next scan instead of lazy migration
5. Reports what it did

**Works on Ubuntu and Fedora** — uses only `sqlite3` (present by default on both) and
standard bash.

**Not strictly required** if the code's auto-migration in `IndexCache.open()` is
implemented. The script exists for operators who want to:
- Verify the schema change before restarting
- Force a clean re-hash of all large files in one pass
- Run the migration on a machine where the node isn't installed yet (preparing a data dir)

### 3.3 What the operator does

1. Update the node package (or `pip install -e` in dev)
2. (Optional) Run `QE/migration/migrate_index_v2.sh` to preview or force the migration
3. Restart the node daemon
4. The first scan re-hashes files > 40 MB with partial reads — much faster than before

No client-side action needed. No hub-side action needed.

---

## 4. What does NOT change

- **Files <= 40 MB** — identical hashing, identical `id`, `hash_version=1`.
- **Enrichment** (thumbnails, duration, metadata) — unchanged, still runs after hashing.
- **File downloads** — `file_req` uses the current `id` from the index. After re-indexing,
  clients get the new index with new hashes and request accordingly.
- **Watchdog / reconciliation** — filesystem events trigger the same code paths. New or
  modified files are hashed with the appropriate method based on size.
- **Index encryption / signing** — `GroupIndex.serialize()` and the GEK-sealed wire
  messages are unchanged. `hash_version` rides inside each entry via `asdict()`.
- **Index delta computation** — `GroupIndex.diff()` compares entries by all fields
  (dataclass `__eq__`). A re-indexed file whose hash changed (v1 → v2) appears as a
  deletion of the old id + addition of the new id, which is correct.
- **Hub tables** — opaque content_hash storage, untouched.
- **MNP version** — this is an additive field on index entries. No protocol version bump
  needed. An older node that doesn't send `hash_version` is handled by the default.

---

## 5. Side effects — documented and accepted

1. **Cross-group search**: a file > 40 MB indexed on a v1 node and a v2 node produces
   different `id`s. The search page will not merge them as the same file. The user sees
   two entries instead of one, each from its own group. This resolves itself when both
   nodes upgrade.

2. **First scan after upgrade**: files > 40 MB are re-hashed. With v2 this reads 45 MB
   per file (not the full content), so the re-index is fast — but it is not instant. A
   4 TB library with 1000 large files reads ~44 GB instead of 4 TB.

3. **`content_hash` drift on hub tables**: if a public group's node upgrades, the hashes
   it registers in `swarm_sources` change for large files. Old entries with v1 hashes
   become stale. The swarm registration mechanism's `last_seen` update handles this — stale
   entries age out. No explicit cleanup needed.

4. **Index version bump**: re-indexing sets `GroupIndex.version = int(time.time())`,
   which triggers a full `index_sync` to all connected peers. This is the normal path for
   any index change — it is not new load.

---

## 6. Indexing trigger points — verified safe

| Trigger | Location | Impact of v2 |
|---|---|---|
| Daemon startup — initial scan | `daemon.py:648-674` | Uses `_hash_or_cached()` which applies v2 rules. Safe |
| Create Group wizard — Step 3 | `create-group-page.js:244-248` polls `index-status` | No change — polls progress, doesn't control hashing |
| Settings — add root | `group-settings.js:909-927` | Triggers `retarget()` → scan → `_hash_or_cached()`. Safe |
| Watchdog — file created/modified | `indexer.py:797-816` | Calls `_update_entry()` → `_hash_or_cached()`. Safe |
| Reconciliation loop | `indexer.py:513-650` | Calls `_sweep_available_roots()` → `_hash_or_cached()`. Safe |
| Hot reload — `_reload_config()` | `daemon.py:790-829` | Creates new `DirectoryIndexer` with v2 code. Safe |

---

## 7. Implementation order

1. **`protocol.py`** — add `hash_version` field to `IndexEntry` and `index_entry_wire()`
2. **`cache.py`** — add `hash_version` to schema, auto-migrate on open, update lookup/put
3. **`indexer.py`** — implement `_partial_hash()`, update `_scan_file()` and
   `_hash_or_cached()`
4. **Tests** — new test cases for partial hashing, cache versioning, wire round-trip
5. **`QE/migration/migrate_index_v2.sh`** — standalone migration script
6. **Manual test** — run a node with a mixed library (small + large files), verify:
   - Small files: same hash as before, `hash_version=1`
   - Large files: different hash, `hash_version=2`, 45 MB read
   - Cross-group search: small files merge, large files don't (across v1/v2 nodes)
   - Cache hit on second scan: no re-read
   - Index sync to connected peers: entries carry `hash_version`