aboutsummaryrefslogtreecommitdiffstats
path: root/docs/MESHBAY_DESIGN.md
diff options
context:
space:
mode:
authorChristophe Besson <cbesson@gmail.com>2026-10-14 10:17:49 +0200
committerChristophe Besson <cbesson@gmail.com>2026-10-14 10:17:49 +0200
commit80ac882346b474bb7ea4e8c8afd4694bd13dd9dd (patch)
tree3aebf475a23dc3e44dff1a6c577d8af82418d024 /docs/MESHBAY_DESIGN.md
parentaf542a40aec6ffe0ac06798d99abd00a24eb9c50 (diff)
downloadmeshbay-80ac882346b474bb7ea4e8c8afd4694bd13dd9dd.tar.gz
feat(node): skip a file the disk stops answering for, then pause its root
A Windows tester's scan stuck for good on one directory of a large disk: one read that never returned held the indexer's only thread, and with it every root behind that one. - A disk call is timed by its silence, not its length: each directory entry listed and each megabyte read counts as an answer. After 60 s of silence the call is abandoned on its daemon thread, the file is noted unread and the scan goes on. - Three silent files in a row pause the root (not persisted). Resume skips the files already known to give no answer; a restart, a replug or a change to the file retries them. - The walk is per directory with os.scandir: a subdirectory that cannot be listed costs itself, and a listing that stops answering pauses the root at once. A scan no longer stats anything from the event loop. - The log names the file after 10 s of silence, the indexing band shows it, and Group settings shows the paused root, where it stuck and the unread files with their reason, on the node's machine only. - generate_http_api.py writes LF on Windows too. Verified with real kernel-blocked opens (an unacknowledged oplock) in the installed frozen app. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Diffstat (limited to 'docs/MESHBAY_DESIGN.md')
-rw-r--r--docs/MESHBAY_DESIGN.md50
1 files changed, 50 insertions, 0 deletions
diff --git a/docs/MESHBAY_DESIGN.md b/docs/MESHBAY_DESIGN.md
index 0703993..1f6c96d 100644
--- a/docs/MESHBAY_DESIGN.md
+++ b/docs/MESHBAY_DESIGN.md
@@ -1783,6 +1783,53 @@ root is therefore not watched, not swept by reconcile, and not hashed by any wal
there; the sweep indexes only what the index lacks. The request returns once
the walk is queued, and the progress band shows it.
+**A disk that stops answering costs a file, then a root, never the scan.** A read
+on a failing drive, a share that dropped or a file a sync client has not fetched
+does not fail: it does not return. The hash ran on the indexer's one thread with
+nothing to bound it, so one such file stopped its group's scan where it stood:
+the roots behind it unread, the enrichment that follows a scan never started, a
+pause unable to take effect since it is looked at between two files, and nothing
+in the log naming the file, because a file is named there only when it fails.
+
+- **What is timed is silence, not work.** Every directory entry listed and every
+ megabyte read counts as an answer (`_beat`). A listing of a million entries and
+ a file on a slow link take as long as they take; neither goes sixty seconds
+ (`STALL_SECS`) without one. A call waiting its turn behind another is not
+ timed for the wait.
+- **The thread is left behind, because nothing can interrupt it.** It stays in
+ the kernel for as long as the kernel keeps it, and a new one takes the calls
+ that were waiting (`_DiskThread.abandon`). They are daemon threads: the
+ interpreter joins a pool's threads on its way out, and a node with one read
+ that never returned could not stop.
+- **A root's walk asks the disk nothing from the event loop.** Each file's stat
+ is taken with the listing, on the indexer's thread, and handed down to the
+ hash. Asked again on the loop before each hash, as it was, it is the loop
+ that waits on a silent disk, and with it every connection of the node. The
+ one file a watcher event names is still asked for there.
+- **The file is listed as unread and not asked for again until it changes.**
+ Reconcile meets it at every pass and would wait a minute on it each time. A
+ file the system refuses outright (a permission, a lock) is listed with what the
+ system said and is asked again, since asking costs nothing. A restart, or the
+ root going away and coming back, forgets the list.
+- **Three in a row, with none read in between, and the root is paused.** A minute
+ a file is weeks on a music library, with a thread left behind each time. The
+ root is paused by the node as the operator would pause it (above): the rest of
+ the walk comes from the hash cache, the watcher and reconcile leave it alone,
+ what is indexed stays served, the next root is walked, and Resume starts it
+ again and gets past the files already known to give no answer. A listing that
+ stops answering pauses its root at once, having no file to give up on; a rescan
+ that gets no listing puts back the entries it had dropped.
+- **This pause is not persisted.** The roster holds the operator's pauses. A
+ restart tries the root again, which is what brings back one whose drive or
+ share has returned, at the cost of three minutes when it has not.
+- **The operator is told, on their own machine only.** Ten seconds of silence is
+ logged and shown in the progress band with the file; the group's shared
+ directories list the files given up on and why, and where a root went silent.
+ These are paths, so they are on the loopback API (`index-status`) and in
+ nothing a member receives: a member sees the root paused.
+- **One directory that cannot be listed costs that directory.** The walk used to
+ skip a refusal in silence and to end the whole root's scan on any other error.
+
### 6.4 Uploads
Five protections, and they are the substance:
@@ -4461,6 +4508,9 @@ process runs it — `systemctl --user` on Linux, Task Scheduler on Windows.
| Item | Status |
|---|---|
+| **A disk that stops answering still stalls what is not the scan** | The indexer's own reads are bounded (§6.3). Serving a chunk, the availability check and the directory deletions go through the roots' I/O thread, which has no such limit, and the tag and photo readers leave a thread in the shared pool at each timeout |
+| **A file that gave no answer is asked again at every start** | The list of unread files is memory. With three or more such files in a row the root is paused again three minutes into each start, until the operator resumes it or moves them |
+| **Paths longer than 260 characters are not indexed on Windows** | With the system's default setting the walk cannot list them; it now says so once per directory instead of nothing. The reads already use the long form (`paths.long_path`), the walk does not |
| **The desktop cast relay drops whole segments** | `cast-relay.js` drops a fragment once 8 MB wait for a receiver, and a fragment is a segment, 5 to 10 MB at a film's bitrate: the picture freezes for its length. Its backlog is bounded in fragments only. The Android relay spools a receiver's lead to disk and bounds its backlog in bytes (§11.4); the desktop has neither |
| **The desktop cast relay takes the first chunk for the whole header** | The node's first chunk can be the ftyp alone, the moov in the next; the receiver then gives up. Fixed in the Android relay (§11.4), not in `cast-relay.js` |
| **A cast restart may pull far ahead** | Seen once on an emulator with a synthetic film: after the restart's reinit the node reported `duration=None`, and the page pulled most of the film at network speed. Not reproduced on a real film; the suspicion is a read-ahead budget computed without a duration |