meshbay.git/packages/meshbay-node/src/meshbay_node/indexer/indexer.py, branch 0.16

style: ruff's own fixes, mechanically applied

2026-09-19T12:24:13Z

`ruff check .` had gone unrun long enough to report 568 errors, which is the same as having no linter: the next real finding would have been invisible in the noise. This is the 521 it fixes by itself, in 173 files, and nothing else — the 98 it cannot fix are the next commit. What actually changed: import sorting (225), imports nobody used (87, none of them a re-export — no `__init__.py` is touched, which was the one way this could have broken an import elsewhere), `datetime.timezone.utc` to `datetime.UTC` (69) and `asyncio.TimeoutError` to `TimeoutError` (18), both plain aliases on the 3.12 this project requires, `Optional[X]` to `X | None` (24), and f-strings with nothing to interpolate (19). Checked rather than assumed: every module in the three packages still imports, and the suite is 2893 passed — the same count, test for test, as the merge before it. Co-Authored-By: Claude Opus 5

docs: cite MESHBAY_DESIGN.md and a section instead of the merged notes

2026-09-19T00:12:47Z

The per-feature design notes were merged into docs/MESHBAY_DESIGN.md and deleted from the tree on 2026-09-11, but ~230 comments across the three packages still named them — usually written `docs/musicbay.md §3.2`, as though the file were still in docs/. A reader had to know §16 existed to resolve any of them. They now name the section directly. Every mapping comes from §16, the concordance, which already records where each old section landed: musicbay -> §9.8, mediacenter -> §9.7 for the Videos app and §6.5 where the subject is derived data, photos -> §9.9, auth-confirm -> §3.6, refactoring-search -> §9.11, invite-pairing-v1 -> §3.4, per-node-identity-v1 -> §3.2, captcha -> §7.7, chat-sender-keys -> §4.5, apps/refactor-groups -> §9.1–§9.4, desktop-client-v1 per section. Bare citations of the same documents (`draft-v6 §2.11`, `§4.8`, `§3.4`) are retargeted too: those collide with real section numbers in the design document, so leaving them would have been worse than the named form. Four cases the concordance does not cover, each decided rather than guessed: Sub-item references into documents that no longer exist — mediacenter's `§3.3 row 4`, `§3.4b/c`, `apps.md §3b` — name rows and sub-items §9.7 and §9.2 do not reproduce. The module-level citation stays; the sub-item pointer is dropped. The V-findings keep their labels but lose the dead `§10.1/` prefix. §13.8 lists V1–V13 as per-application open items, which is not what the labels mean in these comments, so pointing them at §13.8 would have been a false citation. `apps.md §5`'s virtualization requirement has no counterpart anywhere in the design document. The requirement is stated in the comment itself, so the citation is dropped rather than aimed at a section that does not say it. Comments that attributed a *sentence* to an old note — musicbay's "several thousand files" example, its "what got measured" note, its measured ~11%/~26% cover-art figures, the "original no root, whole shared tree" call — state the fact without attribution now. §9.8 does not contain those sentences and citing it for them would have been wrong. CLAUDE.md's "a reference to a document that no longer exists" row now says the concordance is for git history and out-of-tree material; the code cites sections directly. Verified: 2851 passed, 4 skipped. The 12 errors in the run are the Firefox leg of test_sticky_header.py's browser harness, which is broken at the browser level on this machine — headless Firefox (snap) dies with `[GFX1-]: RenderCompositorSWGL failed mapping default framebuffer`, renders nothing, and the probe exits `{"error": "no measurement"}` after its full 90s wait. Chrome runs the same 12 assertions in 3.2s and passes. Nothing here can affect it: every changed line in style.css is inside a comment. Also checked: ast.parse on every changed .py, `node --check` on every changed .js, the /* */ balance in style.css, and that no changed line exceeds the width its file already used. Co-Authored-By: Claude Opus 5

fix(node): take the availability poll and every upload write off the loop

2026-09-18T13:59:24Z

The rest of AV9's disk half. Serving a file left the loop in the commit before this one; two paths were still on it. **The availability poll.** `RootSet.refresh_availability` stats every root, and eleven call sites reached it from `async def` — the reconcile loop among them, on a timer. On a sleeping disk that is a stall once per tick, and the stat is also what keeps the disk awake, so a node paid spin-up for a library nobody was reading. All eleven now go through `off_disk`, `Root.is_live` included. **The upload write.** `open`/`write`, and the resolve, the stat, the free-name search, the rename and the unlink around it. This one could not simply be awaited: the handler was synchronous, so nothing could come between the `chunk_index != state.next_index` check and the `advance` that answers it, and that is the whole of the chunk-ordering rule. Awaiting the write opens the gap — chunk 1 arriving while chunk 0 is in the disk thread reads a position that has not moved and is refused as out of order, so an upload would fail on a slow disk and nowhere else. Verified, not assumed: without the lock the new ordering test refuses three chunks of four. So the check, the write and the advance are one critical section again, under a lock held **per group**. Not per session: `partial_uploads` lives in the group context so a reconnecting client finds its upload where it left it, which means two sessions of one member share the position of one `.part` file. Arrival order is preserved by construction — the dispatcher creates one task per message as it arrives, tasks start in creation order, and the lock is the first thing each one waits on, so its waiters queue in arrival order too. `_do_file_upload` is a coroutine now, which is why forty-two test call sites gain an `await`. Their outcomes are unchanged, file by file, against the run before the change. `test_ops.py` asked which public coroutines `ops` exposes and got `off_disk`, imported rather than defined there. It now asks for the ones written in the module, which is what its own docstring means; all forty-three operations are still checked. Co-Authored-By: Claude Opus 5

feat(node): progress names the root under way and the roots waiting

2026-09-14T09:08:12Z

`IndexProgress` said "scanning, this many bytes of that many" and nothing more. A group's roots are walked one after another, so a second directory added during a large scan showed as the bar jumping back to 0 %. It now also carries the root being walked and its position in the roots table, the kind of walk (scan, rescan, reconcile, watch), file counts, and the roots waiting for the scan lock in order: queued by the initial scan, by a retarget, and by a plug; dropped when a root is removed. `GET /api/index-status` answers for every group at once, including a group still in its initial scan, so a client can show indexing on any page. It names roots: loopback only, like `current_dir`. `index_progress` and the handshake ack gain the same counters, still naming nothing (decision D3): the root is a position in the roots table the member already opened from the sealed index, and the queue is a count. The pusher keeps speaking while a root only waits for the lock. Co-Authored-By: Claude Opus 5 Claude-Session: https://claude.ai/code/session_01T6jPTeocXA1BePekdsgPya

fix(node): a scan keeps its progress while a burst or a plug runs beside it

2026-09-14T08:56:40Z

A watchdog burst wrote the indexer's single `progress` directly. A file dropped into a folder during a large scan added its size to the scan's total, then cleared `scanning` when its own hash finished, so the progress went blank with hours of hashing left. Bursts now keep their own counters, shown only while no whole-root walk runs. A plug rescan took no scan lock and walked its root beside an added root's scan, both resetting the same counters and reading the drive in turn. It now waits for the lock, and skips the rescan if the root was ejected or removed while it waited, since the rescan drops the entries before it walks. Co-Authored-By: Claude Opus 5 Claude-Session: https://claude.ai/code/session_01T6jPTeocXA1BePekdsgPya

fix(node): a reload and a plug rescan outlive the session that asked

2026-09-14T07:40:23Z

A root added from the client arrives over MNP, and _retarget_indexer started the daemon's reload with the session's own _spawn. When that session closed - a client reconnecting 47 s into the scan of a 900 GB root - shutdown_tasks() cancelled the reload mid-scan, and the reload queued behind it, without a line in the log. The new root was in node.toml and in the indexer's set but never in the group's context; the lock was free and nothing retried, so the node served the old roots table for hours while reconcile hashed the whole drive as missed events. One loopback reload fixed the live node in 9 ms. _reload_config now runs the work in a node-owned task and awaits it through asyncio.shield, so a caller that goes away only stops waiting; a cancelled reload is logged. plug_root does the same for its rescan, which drops the root's entries before walking the disk and so left the root empty when its admin op's session closed. The existing MNP test replaced _spawn with a list and could not cancel anything; the new tests close the session for real and fail without this. Co-Authored-By: Claude Opus 5 Claude-Session: https://claude.ai/code/session_01T6jPTeocXA1BePekdsgPya

fix(node): an added root is served before it is scanned

2026-09-14T00:35:53Z

Adding a large directory to a running group made the reload await the scan of the new root before putting the new RootSet in the group's context, holding _reload_lock the whole time. For the hours a large drive takes to hash, the node served the old set: - a file request under the new root got None from entry_abs_path and the handler died on None.exists() without replying; - a writable/removable toggle answered with the live table, still the old one, so the directory vanished from the operator's settings; - reconcile saw every file the scan had not reached as a missed event and hashed it again on the same executor, rewriting progress under the scan. retarget now applies the set, the roots table and the watcher first, and with wait=False scans the added roots in the background; the daemon swaps ctx["roots"] before calling it. A scan lock shared by the initial scan, added-root scans and reconcile makes the reconcile loop sit out a running scan without backing off. Every transport site that resolves an entry answers ROOT_NOT_SERVED instead of crashing, and a delete keeps the entry. Co-Authored-By: Claude Opus 5 Claude-Session: https://claude.ai/code/session_01T6jPTeocXA1BePekdsgPya

fix: hold every background task, in both codebases

2026-09-13T13:40:50Z

asyncio keeps only a weak reference to a task, so a coroutine started with `asyncio.ensure_future(...)` whose result is discarded can be collected while it is still running: the loop logs "Task was destroyed but it is pending!" and the work simply does not happen. No error reaches the caller, and what is lost is whatever that coroutine was in the middle of. The node already had a guard for this, written after an abandoned stream task lost a transcode slot for good — and it read one file, `webrtc_server.py`, because that is where the defect was found. Outside that file there were nineteen sites: the hub's `chat_notify` (a notification for every member of a group), the indexer's debounce (every real-time index update), eleven in `daemon.py` including the SIGHUP reload and each enrichment pass, two in `ops.py`, and five in the loopback API. `meshbay_common.background.spawn()` is the one door. It holds the task, drops it when it finishes, and logs what it raised under the coroutine's own name — an exception in a task nobody awaits was otherwise reported by asyncio at collection time, out of context or not at all. A peer session's `_spawn` stays as it is: that one can also *cancel* what it holds, which a module-level holder cannot, because a session ends and a process does not. `test_background_tasks.py` walks every package's source and refuses a discarded handle. It parses rather than greps, so an assignment, a comprehension or an await is not mistaken for one, and it was checked against a deliberate reintroduction. A guard that stops at the edge of the file where the bug was found is a guard against that bug, not against its class. Co-Authored-By: Claude Opus 5 Claude-Session: https://claude.ai/code/session_01UMxEQadpzPkYLFf5CYKhpW

fix(node): an uploaded file records who sent it

2026-09-13T13:40:34Z

`_register_uploader` walked the index for the entry it had just written, at a moment when no such entry can exist: the file was a `.part` until the rename on the line above, which is not indexable, and the watchdog that will index it debounces for two seconds and then hashes. The walk matched nothing, silently, so every uploaded file in every group was owned by nobody — and `file_delete` refuses a caller with no admin authority when the entry records no uploader, so a member could not delete what they had just sent. MESHBAY_DESIGN.md §5.4 grants that to any non-revoked device of the uploading account. The record is now written when the last chunk lands (`indexer.record_upload`) and the entry is stamped from it in `_hash_or_cached`, the one funnel every entry passes through — initial scan, watchdog, reconcile and replug alike. It lives in the index cache rather than on the entry alone, because the index is rebuilt from disk at every start and an owner the node forgets on restart is a right quietly taken away. It is validated against a live `stat()`, so whatever later occupies that path inherits nothing; and `_rescan_root`'s carry-over no longer copies over it, or memory would beat the durable record. §5.4 also claimed ownership was *provable* — a transcript the uploader signs, stored with the entry. No such signature has ever existed; `meshbay:upload:v1` in the code is the groupbox purpose that seals the envelope. The section now states what the code does, and the transcript is an open item in §15.3. `test_upload_attribution.py` drives the real handler and a real indexer across that seam. Against the previous source its two positive cases fail on the property, not on a missing method — an upload, then a rebuild from disk, then a different file at the same path inheriting nothing. Co-Authored-By: Claude Opus 5 Claude-Session: https://claude.ai/code/session_01UMxEQadpzPkYLFf5CYKhpW

fix(node): carry enrichment across a rescan instead of re-deriving it

2026-09-07T01:13:14Z

e1dbdf0 made a replugged root re-enrich, which was correct and not enough: the operator still watched their albums vanish. Measured on the reported library with a cold metadata cache, the node broadcast twice — the first delta stripped every album, the second put them back 14 seconds later. Fourteen seconds of "no music found" is the bug, whatever happens after. An entry's id is its content hash, so an entry that comes back under the same id, name and path is the same bytes in the same place and everything enrichment derived from it still holds. `_rescan_root` now carries those fields across the drop-and-rescan that `reconcile` and `plug_root` share. Re-enrichment stays as the fallback for what genuinely changed: a different id is different content, and a different name or path can change the folder and filename fallbacks that artist, album, display_title and track_no rest on, so those entries are still handed to the daemon through `rescanned_ids`. `uploader_id`/`uploader_pk` ride along. They are the same shape of field — set once on an entry, readable from nowhere on disk — and they decide who may delete the file, so losing them to a replug quietly took a right away. Verified on the running node: one broadcast 550ms after the plug, carrying the albums, and no metadata lookups at all. The tests now assert the field on the entry rather than a call to an enricher. Counting calls is what let the previous version of this file pass while the operator still saw an empty tab. Co-Authored-By: Claude Opus 5 Claude-Session: https://claude.ai/code/session_011pvMdvLBG92jyhvD5pD6us