| Commit message (Collapse) | Author | Age | Files | Lines |
|---|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
`_register_uploader` walked the index for the entry it had just written, at a
moment when no such entry can exist: the file was a `.part` until the rename on
the line above, which is not indexable, and the watchdog that will index it
debounces for two seconds and then hashes. The walk matched nothing, silently,
so every uploaded file in every group was owned by nobody — and `file_delete`
refuses a caller with no admin authority when the entry records no uploader, so
a member could not delete what they had just sent. MESHBAY_DESIGN.md §5.4
grants that to any non-revoked device of the uploading account.
The record is now written when the last chunk lands (`indexer.record_upload`)
and the entry is stamped from it in `_hash_or_cached`, the one funnel every
entry passes through — initial scan, watchdog, reconcile and replug alike. It
lives in the index cache rather than on the entry alone, because the index is
rebuilt from disk at every start and an owner the node forgets on restart is a
right quietly taken away. It is validated against a live `stat()`, so whatever
later occupies that path inherits nothing; and `_rescan_root`'s carry-over no
longer copies over it, or memory would beat the durable record.
§5.4 also claimed ownership was *provable* — a transcript the uploader signs,
stored with the entry. No such signature has ever existed; `meshbay:upload:v1`
in the code is the groupbox purpose that seals the envelope. The section now
states what the code does, and the transcript is an open item in §15.3.
`test_upload_attribution.py` drives the real handler and a real indexer across
that seam. Against the previous source its two positive cases fail on the
property, not on a missing method — an upload, then a rebuild from disk, then
a different file at the same path inheriting nothing.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01UMxEQadpzPkYLFf5CYKhpW
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
Files above 40 MB are no longer read in full. Instead, blake3 hashes
45 MB of samples (first 20 MB + last 20 MB + 5 MB at 50% offset).
Files at or below 40 MB are unchanged (full read, hash_version 1).
A new `hash_version` field on IndexEntry (default 1) travels on the
wire and through the cache so both versions coexist without breaking
existing nodes or clients.
The IndexCache auto-migrates its schema on open (ALTER TABLE), so no
manual step is required on upgrade. A standalone migration script is
available in QE/migration/ for operators who want to preview or force
a full re-hash.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
An operator routinely shares the same physical folder into more than one
group (a music library, a Séries drive) — IndexCache used to be opened
once per group (data_dir/{group_id}/index_cache.db), so the second group
to reference an already-fully-hashed multi-terabyte folder paid the same
full content read the first one did. IndexCache itself carried no
group_id in its schema; only daemon.py's wiring did. Now one instance,
opened once at startup (data_dir/index_cache.db), shared by every group's
DirectoryIndexer.
Confirmed against a real deployment (2026-08-25/26): a group sharing an
already-indexed folder with an existing group indexes it instantly, with
zero rehashing.
Also fixes a related cross-group correctness gap found during this work:
media_cache.db (thumbnails, TMDB/MusicBrainz metadata — already node-wide,
untouched by this change) was pruned for a file the moment it left *one*
group's index, even if another group's index still held the same content
hash — forcing a redundant re-fetch/re-probe/re-thumbnail for a group that
never actually lost anything. Prune now runs only once no group's index
references the file_id any more.
Adds a node admin UI action ("Maintenance" card, prune-index-cache) to
drop cache rows that no longer belong to any group's roots — skips
anything under a root that is merely temporarily unavailable (indexer.py's
"a root that goes away freezes, never empties" rule extends to this
cache too, or a reconnected drive would pay a full rehash for no reason).
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_013XSohfUQQiaE77qyFLgSv3
|
|
|
reconcile, and delta sync
Indexer performance work, in four parts:
- Persistent (path, size, mtime) -> hash cache (indexer/cache.py) so a node
restart no longer re-hashes every file — measured at 23 minutes for a
114 GB library on a slow disk before this, near-instant after. Hashing
is deliberately kept sequential (max_workers=1): it was never actually
concurrent despite the pool size, and two interleaved reads seek-thrash
a spinning disk instead of going faster.
- Byte-based scan progress (IndexProgress), surfaced via the loopback
index-status route, the handshake ack, and a periodic INDEX_PROGRESS
push to connected peers — drives a progress bar in the Create Group
wizard and "add a directory" in Settings, and an animated presence dot.
Guaranteed to settle back to idle via try/finally and a final push on
the scanning->false transition.
- The reconcile backstop's directory walks now run in the executor
instead of blocking the daemon's event loop; its interval defaults to
10 min (was 60s) with adaptive backoff to 2h when nothing changes,
reset on a real change or a peer connecting, and is now a per-group
operator setting (signed op + group Settings UI).
- INDEX_DELTA wired up (protocol support existed, nothing called it):
_on_index_change now sends additions/deletions instead of rebuilding
the full entries list, coalesced over a short window so a burst of
file events produces one push, and the hub swarm registration for
public groups only (re-)registers newly added hashes.
Also fixes several bugs found while testing the above against real
libraries (a 114 GB and a 100+ GB group on a USB HDD):
- /api/reload blocked until the reload — including a brand-new group's
full initial scan — finished, which the Electron bridge's fixed 30s
call timeout turned into a hard failure on any real library. The route
now fires the reload without waiting (ops.start_reload), matching
add_root/remove_root's existing pattern; the wizard's own step order
was fixed to wait for the group to actually appear hosted before the
steps that need it (extra roots, GEK), with retries for the residual
race between that and the daemon's own bookkeeping.
- transport.js's hand-rolled msgpack codec had no case for uint64/int64
(0xcf/0xd3) and crashed decoding any message containing one — hit by
IndexProgress.scanned_bytes/total_bytes for any group over ~4.3 GB.
Verified against real msgpack-encoded bytes from the Python side.
- chat_hist_resp, and this change's own index_progress and
set_scan_settings_ack pushes, were not routed by message type and
could be handed to an unrelated pending request by the transport's
"oldest pending" fallback, stalling it until its own 30s timeout and
corrupting whatever received the wrong reply in its place.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_016SF6RKNBKg9qejmoMJ9ybA
|