diff options
| -rw-r--r-- | CLAUDE.md | 41 | ||||
| -rw-r--r-- | docs/MESHBAY_DESIGN.md | 37 | ||||
| -rw-r--r-- | docs/USERGUIDE.md | 13 |
3 files changed, 81 insertions, 10 deletions
@@ -761,6 +761,47 @@ do. Read them before writing anything that touches the same mechanism. socket buffer gave 19.5 MB/s (`udp_buffer.py`); it needs `net.core.rmem_max` raised on a default machine +- **A transfer capped with a core busy: profile that process before tuning + the network.** The command-line client downloaded at 19 MB/s on loopback. + Its 4 MB receive buffer removed 3245 kernel drops and changed nothing else; + `top` then showed the client, not the node, at 100 %, and cProfile (a + wrapper script as `MESHBAY_CLI_E2E_BIN`, the e2e harness does the rest) put + two thirds of its time in aiortc's `InboundStream.pop_messages`: 153 million + steps for 400 MB, the reassembly walked from its start after every fragment. + Next, a SACK per packet cost the node more than sending. Both fixed in + `sctp_fix.py`: 39 MB/s, the node at 49. No amount of reading our code finds + a quadratic loop in a dependency; one profile does + +- **A default of "off" ships a feature nobody has.** `connection_processes` + defaulted to 1, which starts no process: the multi-core path was measured at + 131 MB/s and reached by no node whose operator had not found the setting, so + on Windows a download over eight lanes landed on the node's one core and ran + at the speed of one connection. Worse, the Node page saved **every** setting + on each Save, so any operator who changed one thing had pinned all the others + at that day's defaults, in the roster and in `node.toml`. It sends only what + changed now, and `init` leaves the line commented. A test node that saved + its settings before this keeps `connection_processes = 1` until it is set + again on the Node page + +- **A reconnect let every message through, not just its handshake.** + `_inReconnectAttempt` opened `_sendAndWait` to all callers so the handshake + would not wait on itself; a chunk retry landing between the new channel's + open and the proof got "Handshake required", which no retry treats as a + dropped connection, and the download died with that text in the transfers + panel. Only the opening messages pass now (`SESSION_OPENING`, held to + `dispatch.py` by `test_reconnect_race.py`), and a replaced channel's late + `close` no longer clobbers the new one. `harness/reconnect_race_harness.mjs` + runs the real `connect()`, `_reconnectLoop` and pipeline against a node that + refuses before the proof — the first harness here that does; the earlier + ones stub `connect`, which is exactly where this lived + +- **A push handler that re-fetches the world.** The interactive CLI answered + every `index_delta` by fetching and re-parsing the whole index, on the event + loop that was receiving the download: 67 full indexes in 80 s while a node + enriched a library (one delta a second). It applies the delta in place + (`Tree.apply_delta`), and only a delta that adds or removes files asks again, + once per burst + ### More of the same, from four rounds of live testing - **A protocol harness cannot test the SPA.** Any second implementation of the diff --git a/docs/MESHBAY_DESIGN.md b/docs/MESHBAY_DESIGN.md index 4e99994..e10209a 100644 --- a/docs/MESHBAY_DESIGN.md +++ b/docs/MESHBAY_DESIGN.md @@ -1486,9 +1486,12 @@ download land on different processes without the client doing anything. | 8 | 150 MB/s, the receiving browser being the limit | 21 % | Measured on loopback, a 1.5 GB file through the interface in a headless browser, -byte for byte. A process costs about 85 MB of memory. The default is 1, which -starts none and leaves the node exactly as it was; the operator raises it. A -change applies to new connections, and a connection already open stays where it +byte for byte. A process costs about 85 MB of memory. The default is one per +core, up to four: past four a single download gains nothing, and a process is +memory held for the life of the node. One starts none, and leaves every +connection, every lane of a download included, on the node's own core — the +case where Python's per-packet cost is the whole ceiling, and highest on Windows. +A change applies to new connections, and a connection already open stays where it is until it closes, in a process asked to leave or in the node. A process that dies takes its own connections with it, which their clients see as a dropped connection and reopen, and is replaced at the next offer. The loopback socket is @@ -1512,6 +1515,32 @@ machine nothing changes until the operator raises it; the node says so at startu The limit that remains is the node's own receive path, about 5 % of a core per MB/s. +**A receiver on aiortc reassembles in linear time and acknowledges every second +packet.** That is the command-line client in a download and the node in an +upload. aiortc 1.15 walks the reassembly from its start after every DATA chunk, +so a message of n fragments costs n²/2 steps — a 1 MiB `file_chunk` is about 870 +— and it acknowledges every packet, where RFC 4960 §6.2 asks for one SACK every +second packet; each SACK costs the sender a DTLS decryption and a parse on the +core that also sends. `sctp_fix.py` (the node's, and the client's copy) skips the +walk until a last fragment is in, and acknowledges every second packet of data, +at once on a gap, a duplicate or a packet carrying anything else, and within +200 ms otherwise. The client asks for a 4 MB receive buffer, as the node does. +Measured on loopback, 400 MB to the command-line client over one connection: 19 +MB/s with 3245 datagrams dropped by its kernel; none dropped with the buffer, and +still 20, the client at a full core; 28 with the reassembly fixed; 39 with the +acknowledgements, 49 by the node's own count of the transfer. An upload from the +client went from 25 to 29. + +**A session is open from the end of its handshake, and a reconnect does not +shorten that.** Until connect() returns, a client sends only what the node takes +before the proof — `handshake`, `handshake_response`, `keypair_bundle_fetch`, +`gek_bundle_fetch`, `join_request`, `device_add_request` — and `device_hello`; +anything else waits for the session, or fails as a dropped connection, which a +download waits out. The node answers anything else with "Handshake required", a +refusal and not a dropped connection, so a chunk let through while a reconnect's +handshake runs ends the download the reconnect is for. The close event of a +channel a reconnect has replaced concerns nothing on the new one. + The lease module is free of asyncio and of the transport: it decides, and the caller does the I/O. A queue that reveals itself only through a DataChannel is a queue nobody can prove things about. @@ -2163,7 +2192,7 @@ Node page: | `max_concurrent_downloads` | 8 | node-wide download leases (§5.5) | | `max_concurrent_uploads` | 8 | node-wide upload leases. A separate pool from downloads, because the two cost different things and one queue for both makes each cap meaningless | | `max_upload_gb` | 8 | the largest single upload, §6.4. Fractions are allowed, and it is read per chunk so a change reaches an upload already running | -| `connection_processes` | 1 | how many processes carry the peer connections, §5.5. One is the node itself; above it, that many cores serve transfers. Bounded by the cores of the machine, which the Node page shows beside the field | +| `connection_processes` | one per core, up to 4 | how many processes carry the peer connections, §5.5. One is the node itself; above it, that many cores serve transfers. Bounded by the cores of the machine, which the Node page shows beside the field. `node.toml` as `init` writes it leaves the line commented, so the default reaches every node that has not chosen | | `transcode_incompatible_video` | true | whether browser-incompatible video is transcoded during streaming. Unlike remuxing this costs real CPU per viewer | `hardware_video_encode` (default true) governs whether that transcode may use the diff --git a/docs/USERGUIDE.md b/docs/USERGUIDE.md index 1808598..ab5ee8c 100644 --- a/docs/USERGUIDE.md +++ b/docs/USERGUIDE.md @@ -852,15 +852,16 @@ and running `meshbay-node reload`: | `max_concurrent_downloads` | 8 | node-wide download slots | | `max_concurrent_uploads` | 8 | node-wide upload slots | | `max_upload_gb` | 8 | the largest single file a member may send you | -| `connection_processes` | 1 | how many CPU cores carry members' connections | +| `connection_processes` | one per core, up to 4 | how many CPU cores carry members' connections | | `transcode_incompatible_video` | true | re-encode video no browser can play | | `hardware_video_encode` | true | use the GPU for that, if one actually works | -`connection_processes` is the one for speed. With 1, everything a node sends -goes through one core, about 35 MB/s shared between everyone downloading at -that moment. Raise it when several people download at once and the machine has -cores to spare: each one adds a core's worth, and about 85 MB of memory. It -applies to connections opened after the change. +`connection_processes` is the one for speed. Each process is a core's worth +of sending, shared between everyone downloading at that moment, and about 85 MB +of memory. With 1, everything a node sends goes through one core: about 35 MB/s +on Linux, much less on Windows. Lower it on a machine short of memory; raise it +above 4 only when many people download at once. It applies to connections +opened after the change. `max_concurrent_streams` is the one to think about: a stream costs a video process for as long as the film lasts, so this counts **viewers**, not |