meshbay.git/packages/meshbay-node/tests/test_indexer.py, branch 0.19

perf(node): sample 9 MB with the size above 9 MB, keep known ids

2026-10-07T23:27:58Z

hash_version 3: size + first 4 MB + last 4 MB + 1 MB at the middle, 5.5x faster cold on a USB disk than the 45 MB sample. The cache now serves a hit under whatever version it holds, so no existing id moves. Co-Authored-By: Claude Opus 5.5

refactor: remove the unused GroupIndex.serialize chain

2026-09-28T14:24:12Z

serialize/deserialize had no production caller, and took with them the per-chunk signature, the ChaCha20 cipher variant and the zstandard dependency. Key derivations are unchanged. Docs corrected, including design §4.3's claim that chunks are compressed. Co-Authored-By: Claude Opus 5.5

style: the 98 ruff could not fix, so the linter is a signal again

2026-09-19T12:39:38Z

The pass before this applied ruff's own fixes. These are the ones needing a decision, and the point of doing them is that `ruff check .` now passes: a linter reporting 98 known-acceptable findings reports nothing, because the next real one arrives invisible. **Lines over 100 (70).** Mostly wrapped where they stood. Two exceptions: the aligned trailing comments in `protocol.py`'s message table were shortened rather than wrapped, because wrapping one row of a table breaks the table; and in `models.py` the column comments moved above their columns for the same reason. **Imports below the first statement (14).** `csam.py` kept its FastAPI imports under a section header halfway down the file; two node tests had a constant and a `pytestmark` wedged between two import blocks. Moved, not suppressed. **Bindings nothing reads (4).** Three in tests, where the call stays and only the name goes — `_user(client, "listener")` is there to create the user, not to return one. The fourth was in `revocation.py` and was not a lint finding at all: `_connect_and_listen` opened an httpx stream to the WebSocket URL, did `pass`, and then opened the real connection through the `websockets` library. One pointless request per connect, left over from before that library was used directly. Removed, and `httpx` with it. **`l` as a name (4)**, **semicolons (6)** in the POC spikes, and the rest. 2893 passed, the same count as the two commits before it. `meshbay_node/revocation.py` is worth a decision separately: 154 lines that nothing imports, superseded by `hub_client.maintain_ws`'s `on_revocation`. This commit only stopped it failing the linter. Co-Authored-By: Claude Opus 5

style: ruff's own fixes, mechanically applied

2026-09-19T12:24:13Z

`ruff check .` had gone unrun long enough to report 568 errors, which is the same as having no linter: the next real finding would have been invisible in the noise. This is the 521 it fixes by itself, in 173 files, and nothing else — the 98 it cannot fix are the next commit. What actually changed: import sorting (225), imports nobody used (87, none of them a re-export — no `__init__.py` is touched, which was the one way this could have broken an import elsewhere), `datetime.timezone.utc` to `datetime.UTC` (69) and `asyncio.TimeoutError` to `TimeoutError` (18), both plain aliases on the 3.12 this project requires, `Optional[X]` to `X | None` (24), and f-strings with nothing to interpolate (19). Checked rather than assumed: every module in the three packages still imports, and the suite is 2893 passed — the same count, test for test, as the merge before it. Co-Authored-By: Claude Opus 5

feat(node): indexing v2 — partial-read hashing for files above 40 MB

2026-09-05T23:33:37Z

Files above 40 MB are no longer read in full. Instead, blake3 hashes 45 MB of samples (first 20 MB + last 20 MB + 5 MB at 50% offset). Files at or below 40 MB are unchanged (full read, hash_version 1). A new `hash_version` field on IndexEntry (default 1) travels on the wire and through the cache so both versions coexist without breaking existing nodes or clients. The IndexCache auto-migrates its schema on open (ALTER TABLE), so no manual step is required on upgrade. A standalone migration script is available in QE/migration/ for operators who want to preview or force a full re-hash. Co-Authored-By: Claude Opus 4.6

feat(node): share the (path,size,mtime)->hash index cache across every group

2026-08-25T22:40:20Z

An operator routinely shares the same physical folder into more than one group (a music library, a Séries drive) — IndexCache used to be opened once per group (data_dir/{group_id}/index_cache.db), so the second group to reference an already-fully-hashed multi-terabyte folder paid the same full content read the first one did. IndexCache itself carried no group_id in its schema; only daemon.py's wiring did. Now one instance, opened once at startup (data_dir/index_cache.db), shared by every group's DirectoryIndexer. Confirmed against a real deployment (2026-08-25/26): a group sharing an already-indexed folder with an existing group indexes it instantly, with zero rehashing. Also fixes a related cross-group correctness gap found during this work: media_cache.db (thumbnails, TMDB/MusicBrainz metadata — already node-wide, untouched by this change) was pruned for a file the moment it left *one* group's index, even if another group's index still held the same content hash — forcing a redundant re-fetch/re-probe/re-thumbnail for a group that never actually lost anything. Prune now runs only once no group's index references the file_id any more. Adds a node admin UI action ("Maintenance" card, prune-index-cache) to drop cache rows that no longer belong to any group's roots — skips anything under a root that is merely temporarily unavailable (indexer.py's "a root that goes away freezes, never empties" rule extends to this cache too, or a reconnected drive would pay a full rehash for no reason). Co-Authored-By: Claude Sonnet 5 Claude-Session: https://claude.ai/code/session_013XSohfUQQiaE77qyFLgSv3

fix(hub,node): Create Group wizard silently skipped apps, and lost track of scanning progress

2026-08-25T16:18:48Z

Two real-world bugs found together while testing multi-root group creation: - CreateGroupWizard only sent the enabled-apps PUT when the operator had *unchecked* something, assuming "every box left checked" already matched the node's own default (Roster.DEFAULT_APPS = chat, files). It doesn't — so leaving every app checked, the common case, silently left Videos/Music/Photos disabled on the node. Now sent unconditionally. - The wizard's "add extra roots" step never polled index-status, so once step 3 (which only watches the first/upload root) finished, the progress bar froze while the node kept scanning the remaining roots for minutes, unwatched. Added waitForRootsIndexed (platform.js), mirroring waitForGroupHosted's own race handling. That fix exposed a deeper one: indexer.py's _scan_root() only flipped `progress.scanning` on *after* walking the directory and stat()-ing every file — both off-loop, but slow enough on a large root that a poller's grace period (waitForRootsIndexed's 5s) could expire before ever observing `scanning: true` (confirmed against production logs: a GEK-init step fired 5.058s after a root started scanning, matching the grace period almost exactly). The stat() pass was also a synchronous loop directly on the asyncio event loop — blocking the whole daemon (WebRTC, chat, admin UI) for as long as it took on a root with many files. Both fixed: `scanning` now flips on before the walk starts, and stat()-ing is now off-loop too (_size_files). Co-Authored-By: Claude Sonnet 5 Claude-Session: https://claude.ai/code/session_013XSohfUQQiaE77qyFLgSv3

fix(node): skip indexing audio files under 50KB, likely-corrupt source

2026-08-24T18:55:52Z

The "P.H. Theme" failure investigated earlier turned out to be a genuinely corrupt 1256-byte source file with no audio stream at all, just an ID3 tag — a real, if rare, corruption pattern worth guarding against directly rather than only handling gracefully at playback time. Scoped to audio only, applied wherever a file actually gets hashed/typed (fresh scan and the cache-miss rehash path alike) — a tiny file of any other type is still indexed normally.

feat(node): recognize WMA and Musepack as audio, read their real tag keys

2026-08-24T18:31:04Z

A real-library scan turned up 250 .wma and 23 .mpc files that the indexer was silently classifying as "other" — genuinely lost from the Music app, not a consolidation-rule artifact (checked separately: the grouping logic itself drops nothing). Both are now indexed as audio and tagged properly: - WMA has no mutagen "easy" wrapper, so the generic tag reader was reading nothing from it at all. Reads the real ASF keys directly instead (Title/Author/WM-AlbumTitle/WM-TrackNumber), confirmed against a real sample file before writing the mapping. - Musepack's format auto-detection is unreliable enough (misidentified a real .mpc as MP3 in spot checks) that it now always opens by its own class instead of guessing from content. - Filters out another placeholder value found along the way: a French ripping tool's auto-generated "Album inconnu ()". Neither format decodes natively in a browser's element, so this gets them correctly visible, tagged, and covered — not yet playable in-browser. That would need server-side transcoding, deliberately left out of this change.

fix(node,hub): HEVC transcode fallback, live-add progress, per-group TMDB toggle

2026-08-24T13:57:41Z

Three bugs found live testing the Videos app against a real HEVC/EAC3 show, plus a design change requested afterward: - Streaming always did "-c:v copy", which faithfully reports a source's real hev1 codec string but is unplayable in a browser with no HEVC decoder (most Chrome/Linux builds). The node now transcodes to H264 whenever the probed codec is browser-incompatible (media_probe.py's new BROWSER_INCOMPATIBLE_VIDEO_CODECS), with a `transcode_incompatible_video` node.toml opt-out for operators who know their viewers already decode it. - Dropping a whole season into an already-watched folder gave no scanning indicator and no progress bar: IndexProgress was only ever updated by the two bulk scan paths, never by the real-time per-file watchdog path (_schedule_update/_debounce/_update_entry). That path now accounts a "burst" the same way, without double-counting a file rewritten mid-debounce. - A stray literal "0" rendered in the video detail modal when there was no TMDB match (`meta.confidence` is 0, and `0 && x` renders "0" in JSX/htm, not nothing) — `confident` is now a real boolean. - Whether TMDB is used at all moves from a node-wide setting to per-group (OP_TMDB_ENABLED/tmdb_enabled/tmdb_enabled_ack, scoped like OP_VIDEO_ROOT): an operator running a real media-library group alongside test/demo groups on one node wants outbound TMDB traffic for the one that needs it, not all of them. The custom API token and query language stay node-wide, one shared credential/cache (tmdb_config/OP_TMDB_CONFIG, unchanged reasoning). MNP_VERSION 0.6 -> 0.7, additive. Co-Authored-By: Claude Sonnet 5 Claude-Session: https://claude.ai/code/session_01LAmyXtc6dAADsH23ydXQpY