aboutsummaryrefslogtreecommitdiffstats
path: root/docs/MESHBAY_DESIGN.md
diff options
context:
space:
mode:
Diffstat (limited to 'docs/MESHBAY_DESIGN.md')
-rw-r--r--docs/MESHBAY_DESIGN.md51
1 files changed, 50 insertions, 1 deletions
diff --git a/docs/MESHBAY_DESIGN.md b/docs/MESHBAY_DESIGN.md
index 0703993..f9ceaa2 100644
--- a/docs/MESHBAY_DESIGN.md
+++ b/docs/MESHBAY_DESIGN.md
@@ -1783,6 +1783,53 @@ root is therefore not watched, not swept by reconcile, and not hashed by any wal
there; the sweep indexes only what the index lacks. The request returns once
the walk is queued, and the progress band shows it.
+**A disk that stops answering costs a file, then a root, never the scan.** A read
+on a failing drive, a share that dropped or a file a sync client has not fetched
+does not fail: it does not return. The hash ran on the indexer's one thread with
+nothing to bound it, so one such file stopped its group's scan where it stood:
+the roots behind it unread, the enrichment that follows a scan never started, a
+pause unable to take effect since it is looked at between two files, and nothing
+in the log naming the file, because a file is named there only when it fails.
+
+- **What is timed is silence, not work.** Every directory entry listed and every
+ megabyte read counts as an answer (`_beat`). A listing of a million entries and
+ a file on a slow link take as long as they take; neither goes sixty seconds
+ (`STALL_SECS`) without one. A call waiting its turn behind another is not
+ timed for the wait.
+- **The thread is left behind, because nothing can interrupt it.** It stays in
+ the kernel for as long as the kernel keeps it, and a new one takes the calls
+ that were waiting (`_DiskThread.abandon`). They are daemon threads: the
+ interpreter joins a pool's threads on its way out, and a node with one read
+ that never returned could not stop.
+- **A root's walk asks the disk nothing from the event loop.** Each file's stat
+ is taken with the listing, on the indexer's thread, and handed down to the
+ hash. Asked again on the loop before each hash, as it was, it is the loop
+ that waits on a silent disk, and with it every connection of the node. The
+ one file a watcher event names is still asked for there.
+- **The file is listed as unread and not asked for again until it changes.**
+ Reconcile meets it at every pass and would wait a minute on it each time. A
+ file the system refuses outright (a permission, a lock) is listed with what the
+ system said and is asked again, since asking costs nothing. A restart, or the
+ root going away and coming back, forgets the list.
+- **Three in a row, with none read in between, and the root is paused.** A minute
+ a file is weeks on a music library, with a thread left behind each time. The
+ root is paused by the node as the operator would pause it (above): the rest of
+ the walk comes from the hash cache, the watcher and reconcile leave it alone,
+ what is indexed stays served, the next root is walked, and Resume starts it
+ again and gets past the files already known to give no answer. A listing that
+ stops answering pauses its root at once, having no file to give up on; a rescan
+ that gets no listing puts back the entries it had dropped.
+- **This pause is not persisted.** The roster holds the operator's pauses. A
+ restart tries the root again, which is what brings back one whose drive or
+ share has returned, at the cost of three minutes when it has not.
+- **The operator is told, on their own machine only.** Ten seconds of silence is
+ logged and shown in the progress band with the file; the group's shared
+ directories list the files given up on and why, and where a root went silent.
+ These are paths, so they are on the loopback API (`index-status`) and in
+ nothing a member receives: a member sees the root paused.
+- **One directory that cannot be listed costs that directory.** The walk used to
+ skip a refusal in silence and to end the whole root's scan on any other error.
+
### 6.4 Uploads
Five protections, and they are the substance:
@@ -3705,7 +3752,7 @@ requirements, not compatibility notes.
| **Case-insensitive, case-preserving** | The index needs a canonical identity and a **case-folding collision check** at scan time, reported to the operator rather than resolved silently. One directory indexed as two roots is the same problem one level up |
| **Unicode normalization** | A name written on one platform in decomposed form and on another in composed form are different byte strings. **Normalize to NFC for identity and comparison; preserve the original bytes for display and for opening the file** |
| **Reserved names and characters** | A group indexed on one platform can hold names another cannot create. The client sanitises on save **and says so**; the upload allowlist is the intersection across platforms, or some files are simply undownloadable |
-| **Path length limits** | Use extended-length paths on Windows, in the node and the client alike |
+| **Path length limits** | Windows refuses a path of 260 characters or more unless `LongPathsEnabled` is set, which it is not by default and which takes administrator rights to change. **The node hands every call that touches a group's disk the extended form** (`paths.long_path`, `\\?\`) and keeps the plain path in the index, the hash cache and the log: a directory is listed under its long form and its entries named from the plain one (`indexer.walk_files`, `wire._walk_dirs`). ffmpeg extends a long path itself and is given the plain one. `test_long_paths.py` holds indexing, serving, uploads, folders and enrichment with the setting off. What a client saves is not measured |
| **Coarse timestamps, local time** | mtime alone is not a change detector. Size plus mtime with tolerance, and rehash when in doubt |
| **Watcher reliability** | Change notification drops events under load on Windows, and inotify is unreliable on a FUSE mount. **Periodic reconciliation is mandatory on both platforms** |
| **No symlinks, no POSIX permissions** | Simplifications: nothing to defend against, and the node runs as the user anyway |
@@ -4461,6 +4508,8 @@ process runs it — `systemctl --user` on Linux, Task Scheduler on Windows.
| Item | Status |
|---|---|
+| **A disk that stops answering still stalls what is not the scan** | The indexer's own reads are bounded (§6.3). Serving a chunk, the availability check and the directory deletions go through the roots' I/O thread, which has no such limit, and the tag and photo readers leave a thread in the shared pool at each timeout |
+| **A file that gave no answer is asked again at every start** | The list of unread files is memory. With three or more such files in a row the root is paused again three minutes into each start, until the operator resumes it or moves them |
| **The desktop cast relay drops whole segments** | `cast-relay.js` drops a fragment once 8 MB wait for a receiver, and a fragment is a segment, 5 to 10 MB at a film's bitrate: the picture freezes for its length. Its backlog is bounded in fragments only. The Android relay spools a receiver's lead to disk and bounds its backlog in bytes (§11.4); the desktop has neither |
| **The desktop cast relay takes the first chunk for the whole header** | The node's first chunk can be the ftyp alone, the moov in the next; the receiver then gives up. Fixed in the Android relay (§11.4), not in `cast-relay.js` |
| **A cast restart may pull far ahead** | Seen once on an emulator with a synthetic film: after the restart's reinit the node reported `duration=None`, and the page pulled most of the film at network speed. Not reproduced on a real film; the suspicion is a read-ahead budget computed without a duration |