| Commit message (Collapse) | Author | Age | Files | Lines |
|---|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
The pass before this applied ruff's own fixes. These are the ones needing a
decision, and the point of doing them is that `ruff check .` now passes: a
linter reporting 98 known-acceptable findings reports nothing, because the next
real one arrives invisible.
**Lines over 100 (70).** Mostly wrapped where they stood. Two exceptions: the
aligned trailing comments in `protocol.py`'s message table were shortened rather
than wrapped, because wrapping one row of a table breaks the table; and in
`models.py` the column comments moved above their columns for the same reason.
**Imports below the first statement (14).** `csam.py` kept its FastAPI imports
under a section header halfway down the file; two node tests had a constant and
a `pytestmark` wedged between two import blocks. Moved, not suppressed.
**Bindings nothing reads (4).** Three in tests, where the call stays and only the
name goes — `_user(client, "listener")` is there to create the user, not to
return one. The fourth was in `revocation.py` and was not a lint finding at all:
`_connect_and_listen` opened an httpx stream to the WebSocket URL, did `pass`,
and then opened the real connection through the `websockets` library. One
pointless request per connect, left over from before that library was used
directly. Removed, and `httpx` with it.
**`l` as a name (4)**, **semicolons (6)** in the POC spikes, and the rest.
2893 passed, the same count as the two commits before it.
`meshbay_node/revocation.py` is worth a decision separately: 154 lines that
nothing imports, superseded by `hub_client.maintain_ws`'s `on_revocation`. This
commit only stopped it failing the linter.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
`ruff check .` had gone unrun long enough to report 568 errors, which is the
same as having no linter: the next real finding would have been invisible in the
noise. This is the 521 it fixes by itself, in 173 files, and nothing else — the
98 it cannot fix are the next commit.
What actually changed: import sorting (225), imports nobody used (87, none of
them a re-export — no `__init__.py` is touched, which was the one way this could
have broken an import elsewhere), `datetime.timezone.utc` to `datetime.UTC` (69)
and `asyncio.TimeoutError` to `TimeoutError` (18), both plain aliases on the 3.12
this project requires, `Optional[X]` to `X | None` (24), and f-strings with
nothing to interpolate (19).
Checked rather than assumed: every module in the three packages still imports,
and the suite is 2893 passed — the same count, test for test, as the merge
before it.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
Files above 40 MB are no longer read in full. Instead, blake3 hashes
45 MB of samples (first 20 MB + last 20 MB + 5 MB at 50% offset).
Files at or below 40 MB are unchanged (full read, hash_version 1).
A new `hash_version` field on IndexEntry (default 1) travels on the
wire and through the cache so both versions coexist without breaking
existing nodes or clients.
The IndexCache auto-migrates its schema on open (ALTER TABLE), so no
manual step is required on upgrade. A standalone migration script is
available in QE/migration/ for operators who want to preview or force
a full re-hash.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
An operator routinely shares the same physical folder into more than one
group (a music library, a Séries drive) — IndexCache used to be opened
once per group (data_dir/{group_id}/index_cache.db), so the second group
to reference an already-fully-hashed multi-terabyte folder paid the same
full content read the first one did. IndexCache itself carried no
group_id in its schema; only daemon.py's wiring did. Now one instance,
opened once at startup (data_dir/index_cache.db), shared by every group's
DirectoryIndexer.
Confirmed against a real deployment (2026-08-25/26): a group sharing an
already-indexed folder with an existing group indexes it instantly, with
zero rehashing.
Also fixes a related cross-group correctness gap found during this work:
media_cache.db (thumbnails, TMDB/MusicBrainz metadata — already node-wide,
untouched by this change) was pruned for a file the moment it left *one*
group's index, even if another group's index still held the same content
hash — forcing a redundant re-fetch/re-probe/re-thumbnail for a group that
never actually lost anything. Prune now runs only once no group's index
references the file_id any more.
Adds a node admin UI action ("Maintenance" card, prune-index-cache) to
drop cache rows that no longer belong to any group's roots — skips
anything under a root that is merely temporarily unavailable (indexer.py's
"a root that goes away freezes, never empties" rule extends to this
cache too, or a reconnected drive would pay a full rehash for no reason).
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_013XSohfUQQiaE77qyFLgSv3
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
scanning progress
Two real-world bugs found together while testing multi-root group
creation:
- CreateGroupWizard only sent the enabled-apps PUT when the operator had
*unchecked* something, assuming "every box left checked" already matched
the node's own default (Roster.DEFAULT_APPS = chat, files). It doesn't —
so leaving every app checked, the common case, silently left
Videos/Music/Photos disabled on the node. Now sent unconditionally.
- The wizard's "add extra roots" step never polled index-status, so once
step 3 (which only watches the first/upload root) finished, the
progress bar froze while the node kept scanning the remaining roots for
minutes, unwatched. Added waitForRootsIndexed (platform.js), mirroring
waitForGroupHosted's own race handling.
That fix exposed a deeper one: indexer.py's _scan_root() only flipped
`progress.scanning` on *after* walking the directory and stat()-ing every
file — both off-loop, but slow enough on a large root that a poller's
grace period (waitForRootsIndexed's 5s) could expire before ever
observing `scanning: true` (confirmed against production logs: a GEK-init
step fired 5.058s after a root started scanning, matching the grace
period almost exactly). The stat() pass was also a synchronous loop
directly on the asyncio event loop — blocking the whole daemon (WebRTC,
chat, admin UI) for as long as it took on a root with many files. Both
fixed: `scanning` now flips on before the walk starts, and stat()-ing is
now off-loop too (_size_files).
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_013XSohfUQQiaE77qyFLgSv3
|
| |
|
|
|
|
|
|
|
|
| |
The "P.H. Theme" failure investigated earlier turned out to be a genuinely
corrupt 1256-byte source file with no audio stream at all, just an ID3
tag — a real, if rare, corruption pattern worth guarding against directly
rather than only handling gracefully at playback time. Scoped to audio
only, applied wherever a file actually gets hashed/typed (fresh scan and
the cache-miss rehash path alike) — a tiny file of any other type is still
indexed normally.
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
A real-library scan turned up 250 .wma and 23 .mpc files that the indexer
was silently classifying as "other" — genuinely lost from the Music app,
not a consolidation-rule artifact (checked separately: the grouping logic
itself drops nothing). Both are now indexed as audio and tagged properly:
- WMA has no mutagen "easy" wrapper, so the generic tag reader was reading
nothing from it at all. Reads the real ASF keys directly instead
(Title/Author/WM-AlbumTitle/WM-TrackNumber), confirmed against a real
sample file before writing the mapping.
- Musepack's format auto-detection is unreliable enough (misidentified a
real .mpc as MP3 in spot checks) that it now always opens by its own
class instead of guessing from content.
- Filters out another placeholder value found along the way: a French
ripping tool's auto-generated "Album inconnu (<timestamp>)".
Neither format decodes natively in a browser's <audio> element, so this
gets them correctly visible, tagged, and covered — not yet playable
in-browser. That would need server-side transcoding, deliberately left
out of this change.
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
Three bugs found live testing the Videos app against a real HEVC/EAC3 show,
plus a design change requested afterward:
- Streaming always did "-c:v copy", which faithfully reports a source's real
hev1 codec string but is unplayable in a browser with no HEVC decoder
(most Chrome/Linux builds). The node now transcodes to H264 whenever the
probed codec is browser-incompatible (media_probe.py's new
BROWSER_INCOMPATIBLE_VIDEO_CODECS), with a `transcode_incompatible_video`
node.toml opt-out for operators who know their viewers already decode it.
- Dropping a whole season into an already-watched folder gave no scanning
indicator and no progress bar: IndexProgress was only ever updated by the
two bulk scan paths, never by the real-time per-file watchdog path
(_schedule_update/_debounce/_update_entry). That path now accounts a
"burst" the same way, without double-counting a file rewritten mid-debounce.
- A stray literal "0" rendered in the video detail modal when there was no
TMDB match (`meta.confidence` is 0, and `0 && x` renders "0" in JSX/htm,
not nothing) — `confident` is now a real boolean.
- Whether TMDB is used at all moves from a node-wide setting to per-group
(OP_TMDB_ENABLED/tmdb_enabled/tmdb_enabled_ack, scoped like OP_VIDEO_ROOT):
an operator running a real media-library group alongside test/demo groups
on one node wants outbound TMDB traffic for the one that needs it, not all
of them. The custom API token and query language stay node-wide, one
shared credential/cache (tmdb_config/OP_TMDB_CONFIG, unchanged reasoning).
MNP_VERSION 0.6 -> 0.7, additive.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01LAmyXtc6dAADsH23ydXQpY
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
reconcile, and delta sync
Indexer performance work, in four parts:
- Persistent (path, size, mtime) -> hash cache (indexer/cache.py) so a node
restart no longer re-hashes every file — measured at 23 minutes for a
114 GB library on a slow disk before this, near-instant after. Hashing
is deliberately kept sequential (max_workers=1): it was never actually
concurrent despite the pool size, and two interleaved reads seek-thrash
a spinning disk instead of going faster.
- Byte-based scan progress (IndexProgress), surfaced via the loopback
index-status route, the handshake ack, and a periodic INDEX_PROGRESS
push to connected peers — drives a progress bar in the Create Group
wizard and "add a directory" in Settings, and an animated presence dot.
Guaranteed to settle back to idle via try/finally and a final push on
the scanning->false transition.
- The reconcile backstop's directory walks now run in the executor
instead of blocking the daemon's event loop; its interval defaults to
10 min (was 60s) with adaptive backoff to 2h when nothing changes,
reset on a real change or a peer connecting, and is now a per-group
operator setting (signed op + group Settings UI).
- INDEX_DELTA wired up (protocol support existed, nothing called it):
_on_index_change now sends additions/deletions instead of rebuilding
the full entries list, coalesced over a short window so a burst of
file events produces one push, and the hub swarm registration for
public groups only (re-)registers newly added hashes.
Also fixes several bugs found while testing the above against real
libraries (a 114 GB and a 100+ GB group on a USB HDD):
- /api/reload blocked until the reload — including a brand-new group's
full initial scan — finished, which the Electron bridge's fixed 30s
call timeout turned into a hard failure on any real library. The route
now fires the reload without waiting (ops.start_reload), matching
add_root/remove_root's existing pattern; the wizard's own step order
was fixed to wait for the group to actually appear hosted before the
steps that need it (extra roots, GEK), with retries for the residual
race between that and the daemon's own bookkeeping.
- transport.js's hand-rolled msgpack codec had no case for uint64/int64
(0xcf/0xd3) and crashed decoding any message containing one — hit by
IndexProgress.scanned_bytes/total_bytes for any group over ~4.3 GB.
Verified against real msgpack-encoded bytes from the Python side.
- chat_hist_resp, and this change's own index_progress and
set_scan_settings_ack pushes, were not routed by message type and
could be handed to an unrelated pending request by the transport's
"oldest pending" fallback, stalling it until its own 30s timeout and
corrupting whatever received the wrong reply in its place.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_016SF6RKNBKg9qejmoMJ9ybA
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
Stage A — a group's content is a set of named roots
---------------------------------------------------
`shared_dir` becomes a list of {name, path, kind}. The name is the directory's
basename, derived once at add time and *stored*: recomputing it would
re-identify a whole library the day someone renames a folder on disk. Duplicate
names are refused case-insensitively and no root may contain another — both
compared with NFC folding, because most of these directories live on exFAT or
NTFS where `Films` and `films` are one directory.
Every index path carries its root name, in a one-root group as much as in a
five-root one. One path shape has to be got right once; two have to be kept
right for ever.
**A root that goes away freezes; it never empties.** Unmounting a volume makes
watchdog report every file under it as deleted, or presents an empty directory
to the next scan. Acting on either propagates deletions for a whole library to
every member, as though the owner had erased it. So a deletion is acted on only
once its root is confirmed readable, and availability is tracked per root — one
unplugged drive leaves the others serving. 12 tests, verified to fail against an
indexer without the check.
Events are not trusted to be complete either: ReadDirectoryChangesW drops them
under load and inotify on a FUSE mount misses changes made outside it. A
periodic reconciliation sweep is the only thing that recovers a missed event.
MNP 0.2 → 0.3 (additive). The hub needs no change: SwarmSource carries a content
hash, a node id and an endpoint — no paths, no filenames — and private groups
register nothing (H7).
Stage B — one implementation behind every front door
----------------------------------------------------
C1 and C6 were both "a second path into the node with its own weaker
handshake". Two implementations of `revoke` with two authorization checks is
that shape one size down. `meshbay_node/ops.py` holds each operation once,
takes the daemon state, and knows nothing about HTTP, argv or MNP. The loopback
API is one `_op(...)` line per endpoint; the MNP handlers call the same
functions. test_ops.py asserts the shape rather than trusting it.
Phase 14 is finished on top of it — `group list`, `gek init|rotate`, `reload`
(SIGHUP), `denylist show|clear`, `file list|rm`. **No operator action requires a
browser any more.** Plus `gek_rotate` and `member_unpin` as operator-signed MNP
operations: rotation is the half of revocation that revocation cannot do, since
the ex-member holds the current key, and the node generates the replacement
with its own CSPRNG — no key material crosses the wire, which is what the C5b
rule is actually about.
Two bugs found by running it rather than by testing it
------------------------------------------------------
GroupIndex is keyed by **content hash**, so the same bytes at two paths are one
entry — which is also why a scan reports ten files and indexes nine.
Reconciliation compared paths, so it decided the second path was a missed event
every 60 s, rewrote the entry and pushed an index update to every connected
peer. Seen in a live node's log.
`meshbay-node reload` crashed on first use with `subprocess` unimported: the
module compiles fine, which is the "syntax, not names" trap already recorded for
the SPA. test_cli_dispatch.py now walks every verb and refuses to let one be
added to the parser without an entry there.
Also corrected: protocol.py declared a second MNP_VERSION of "0.1" while the
wire carried "0.2" — harmless only because nothing imported it. And
_do_dir_create/_do_dir_delete referenced an undefined `filename` on their error
path.
740 tests pass; QE/deploy/e2e.py passes end to end against the live deployment.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
Complete browser-based client: Preact SPA with login, group file browser,
encrypted download, video playback, group chat, i18n, and dark/light theme.
Browser connects P2P to nodes behind residential NAT via WebRTC DataChannel
(aiortc). Hub handles signaling only — all data flows E2E.
Performance: pipelined downloads (8-chunk sliding window), binary msgpack
wire format (no base64), redundant I/O elimination. Large file downloads
stream to disk via File System Access API (showSaveFilePicker).
Validated on SFR + Orange residential NATs, Chrome + Firefox, IPv4/IPv6.
132 tests passing. Deployed to meshbay.org + Orange node.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
|
|
|
GroupIndex: msgpack→zstd→GEK-encrypt→sign for private groups,
plaintext+sign for public groups. DirectoryIndexer: watchdog-based
watcher, async initial scan via thread pool, on_change callback.
Delta support (diff between versions). 10/10 tests passing.
Co-Authored-By: Claude Sonnet 4.6 (1M context) <noreply@anthropic.com>
|