meshbay.git/packages/meshbay-node/src/meshbay_node/indexer/indexer.py, branch 0.13

fix(node): carry enrichment across a rescan instead of re-deriving it

2026-09-07T01:13:14Z

e1dbdf0 made a replugged root re-enrich, which was correct and not enough: the operator still watched their albums vanish. Measured on the reported library with a cold metadata cache, the node broadcast twice — the first delta stripped every album, the second put them back 14 seconds later. Fourteen seconds of "no music found" is the bug, whatever happens after. An entry's id is its content hash, so an entry that comes back under the same id, name and path is the same bytes in the same place and everything enrichment derived from it still holds. `_rescan_root` now carries those fields across the drop-and-rescan that `reconcile` and `plug_root` share. Re-enrichment stays as the fallback for what genuinely changed: a different id is different content, and a different name or path can change the folder and filename fallbacks that artist, album, display_title and track_no rest on, so those entries are still handed to the daemon through `rescanned_ids`. `uploader_id`/`uploader_pk` ride along. They are the same shape of field — set once on an entry, readable from nowhere on disk — and they decide who may delete the file, so losing them to a replug quietly took a right away. Verified on the running node: one broadcast 550ms after the plug, carrying the albums, and no metadata lookups at all. The tests now assert the field on the entry rather than a call to an enricher. Counting calls is what let the previous version of this file pass while the operator still saw an empty tab. Co-Authored-By: Claude Opus 5 Claude-Session: https://claude.ai/code/session_011pvMdvLBG92jyhvD5pD6us

fix(node): a replugged root came back without its metadata

2026-09-07T00:26:17Z

Reported live: a removable root ejected from Files and plugged back in returned with its files and without its albums. Music showed "no music found" and stayed there through a force reload — the loss was on the node, not in the client. `plug_root` drops the root's entries and rescans, which is right; the drive may have changed while it was away. What comes back is a bare IndexEntry: `_hash_or_cached` fills id/name/path/size/type and nothing else. Every enrichment field goes with the old object, and the Music tag fields are cached nowhere by design (enrich_audio.py re-reads them so a rename can re-derive the filename fallback), so re-enrichment is the only way back. Two gates then made sure it never ran: * enrichment is scheduled for `delta.additions`, and ejecting broadcasts nothing, so `_last_broadcast_snapshot` still held those ids — the rebuilt entries diffed as updates, not additions; * `_enrich_new_*_entries` skips anything in `_enriched_attempted`, which is only discarded for `delta.deletions` — and dropping and rescanning inside one call broadcasts no deletion either. A restart cleared both, since an empty snapshot makes every entry an addition. Nothing short of one did. The indexer now records the ids it rebuilt and the daemon drains them at broadcast time: their "already attempted" mark is discarded and they rejoin the entries offered to the three enrichment passes. Not Music-specific — Videos lost durations and titles and Photos lost thumbnails the same way; Music is just where an untagged file has no album to file itself under, so the app goes empty rather than plain. `reconcile()` does the same drop-and-rescan when a root reappears on its own, so a USB drive that fell off and re-mounted hit this with nobody touching the UI. Covered too. Co-Authored-By: Claude Opus 5 Claude-Session: https://claude.ai/code/session_011pvMdvLBG92jyhvD5pD6us

fix(groups): finish Phase 1 — MNP root management, upload targets, eject state

2026-09-06T15:48:36Z

Review of the Phase 1 commit found the RO/RW model sound but three paths unfinished, each of which broke the flow the phase exists to deliver. Plus 29 test failures it introduced and no coverage for anything it added. Uploads went to the wrong directory. The node read a `root` field on file_upload that no client ever sent, so every upload landed in the first writable root while the Files toolbar offered its button based on the root being browsed — with two writable roots, uploading from one wrote into the other. Files now names the root it is showing; Chat names one chosen in the shell (an operator-configured directory arrives in Phase 2); the node refuses an unknown name rather than falling back, and refuses read-only and ejected roots by code. Shared directories were unreachable on the web. The table read its roots only from the loopback API, which resolves to "not available" in a browser, so the section rendered for nobody there — while the Uploads controls it replaced had worked — and the transport.updateRoot/ejectRoot/plugRoot methods beside it were dead. MNP is now the path, loopback the fallback for a local node with no live connection, and adding a root over MNP takes a typed path since no web page can browse a remote disk. Ejecting updated nobody's screen. transport.js resolves an admin ack against the pending request and returns, which is right for every op whose caller knows the value it chose; the root acks carry state only the node can compute, so the operator who clicked Eject was the one client that never saw it happen. And the ejected flag reached roster.db but was never read back, so a restart undid it and the next scan read an empty mount point as an erased library. Also: the member-upload endpoint answered 200 and did nothing (removed); the wizard ignored the first root's RW switch; reload compared roots on name and path, so editing writable in node.toml did nothing; the table had no path column, which is the only thing separating two libraries sharing a basename; apps_enabled normalisation differed between the two sides of a signed subject. Tests: eject/plug, per-root upload refusal and the node.toml rewrite had no coverage at all. test_member_upload_policy.py is replaced by test_root_writable_policy.py — it tested a removed feature — and every property worth keeping from it moved rather than being dropped. Docs: draft-v6 structural decision 9 is annotated as superseded (the operator can no longer have a directory only they may write to — a real capability removed, flagged rather than hidden), the man page documents the root verb and the RO/RW fields, and refactor-groups.md §7b records what the plan got wrong. Suite: 41 failures before, 13 after — all 13 pre-existing on main. Co-Authored-By: Claude Opus 5 Claude-Session: https://claude.ai/code/session_011pvMdvLBG92jyhvD5pD6us

feat: groups refactor Phase 1 — root RO/RW model + shared directories UI

2026-09-06T14:05:39Z

Replace the upload boolean with per-root writable/removable/ejected flags. Backend: new ops (update_root, eject_root, plug_root), MNP 1.1 protocol messages, live RootSet updates so API always reflects current state, CLI root subcommand (add/remove/set/list/eject/plug). Frontend: SharedDirectoriesTable with optimistic toggle switches, eject/plug in Files and Settings, upload gated on root.writable, ejected-root filtering in all media apps, updated Create Group wizard, 10-locale i18n. Co-Authored-By: Claude Opus 4.6

fix(node): prevent watchdog from overwriting index entries with duplicate content

2026-09-06T00:46:56Z

The real-time watchdog path lacked the duplicate-content guard that reconciliation already had. When a file with identical content appeared (e.g. browser download appending " (2)"), add_entry overwrote the original's index entry — making it vanish from the file list despite still being on disk. Now check get_entry before adding, matching the reconciliation logic. Co-Authored-By: Claude Opus 4.6

feat(node): indexing v2 — partial-read hashing for files above 40 MB

2026-09-05T23:33:37Z

Files above 40 MB are no longer read in full. Instead, blake3 hashes 45 MB of samples (first 20 MB + last 20 MB + 5 MB at 50% offset). Files at or below 40 MB are unchanged (full read, hash_version 1). A new `hash_version` field on IndexEntry (default 1) travels on the wire and through the cache so both versions coexist without breaking existing nodes or clients. The IndexCache auto-migrates its schema on open (ALTER TABLE), so no manual step is required on upgrade. A standalone migration script is available in QE/migration/ for operators who want to preview or force a full re-hash. Co-Authored-By: Claude Opus 4.6

fix(hub,node): Create Group wizard silently skipped apps, and lost track of scanning progress

2026-08-25T16:18:48Z

Two real-world bugs found together while testing multi-root group creation: - CreateGroupWizard only sent the enabled-apps PUT when the operator had *unchecked* something, assuming "every box left checked" already matched the node's own default (Roster.DEFAULT_APPS = chat, files). It doesn't — so leaving every app checked, the common case, silently left Videos/Music/Photos disabled on the node. Now sent unconditionally. - The wizard's "add extra roots" step never polled index-status, so once step 3 (which only watches the first/upload root) finished, the progress bar froze while the node kept scanning the remaining roots for minutes, unwatched. Added waitForRootsIndexed (platform.js), mirroring waitForGroupHosted's own race handling. That fix exposed a deeper one: indexer.py's _scan_root() only flipped `progress.scanning` on *after* walking the directory and stat()-ing every file — both off-loop, but slow enough on a large root that a poller's grace period (waitForRootsIndexed's 5s) could expire before ever observing `scanning: true` (confirmed against production logs: a GEK-init step fired 5.058s after a root started scanning, matching the grace period almost exactly). The stat() pass was also a synchronous loop directly on the asyncio event loop — blocking the whole daemon (WebRTC, chat, admin UI) for as long as it took on a root with many files. Both fixed: `scanning` now flips on before the walk starts, and stat()-ing is now off-loop too (_size_files). Co-Authored-By: Claude Sonnet 5 Claude-Session: https://claude.ai/code/session_013XSohfUQQiaE77qyFLgSv3

fix(node): skip indexing audio files under 50KB, likely-corrupt source

2026-08-24T18:55:52Z

The "P.H. Theme" failure investigated earlier turned out to be a genuinely corrupt 1256-byte source file with no audio stream at all, just an ID3 tag — a real, if rare, corruption pattern worth guarding against directly rather than only handling gracefully at playback time. Scoped to audio only, applied wherever a file actually gets hashed/typed (fresh scan and the cache-miss rehash path alike) — a tiny file of any other type is still indexed normally.

feat(node): recognize WMA and Musepack as audio, read their real tag keys

2026-08-24T18:31:04Z

A real-library scan turned up 250 .wma and 23 .mpc files that the indexer was silently classifying as "other" — genuinely lost from the Music app, not a consolidation-rule artifact (checked separately: the grouping logic itself drops nothing). Both are now indexed as audio and tagged properly: - WMA has no mutagen "easy" wrapper, so the generic tag reader was reading nothing from it at all. Reads the real ASF keys directly instead (Title/Author/WM-AlbumTitle/WM-TrackNumber), confirmed against a real sample file before writing the mapping. - Musepack's format auto-detection is unreliable enough (misidentified a real .mpc as MP3 in spot checks) that it now always opens by its own class instead of guessing from content. - Filters out another placeholder value found along the way: a French ripping tool's auto-generated "Album inconnu ()". Neither format decodes natively in a browser's element, so this gets them correctly visible, tagged, and covered — not yet playable in-browser. That would need server-side transcoding, deliberately left out of this change.

fix(node,hub): HEVC transcode fallback, live-add progress, per-group TMDB toggle

2026-08-24T13:57:41Z

Three bugs found live testing the Videos app against a real HEVC/EAC3 show, plus a design change requested afterward: - Streaming always did "-c:v copy", which faithfully reports a source's real hev1 codec string but is unplayable in a browser with no HEVC decoder (most Chrome/Linux builds). The node now transcodes to H264 whenever the probed codec is browser-incompatible (media_probe.py's new BROWSER_INCOMPATIBLE_VIDEO_CODECS), with a `transcode_incompatible_video` node.toml opt-out for operators who know their viewers already decode it. - Dropping a whole season into an already-watched folder gave no scanning indicator and no progress bar: IndexProgress was only ever updated by the two bulk scan paths, never by the real-time per-file watchdog path (_schedule_update/_debounce/_update_entry). That path now accounts a "burst" the same way, without double-counting a file rewritten mid-debounce. - A stray literal "0" rendered in the video detail modal when there was no TMDB match (`meta.confidence` is 0, and `0 && x` renders "0" in JSX/htm, not nothing) — `confident` is now a real boolean. - Whether TMDB is used at all moves from a node-wide setting to per-group (OP_TMDB_ENABLED/tmdb_enabled/tmdb_enabled_ack, scoped like OP_VIDEO_ROOT): an operator running a real media-library group alongside test/demo groups on one node wants outbound TMDB traffic for the one that needs it, not all of them. The custom API token and query language stay node-wide, one shared credential/cache (tmdb_config/OP_TMDB_CONFIG, unchanged reasoning). MNP_VERSION 0.6 -> 0.7, additive. Co-Authored-By: Claude Sonnet 5 Claude-Session: https://claude.ai/code/session_01LAmyXtc6dAADsH23ydXQpY