diff options
Diffstat (limited to 'docs')
| -rw-r--r-- | docs/MESHBAY_DESIGN.md | 57 | ||||
| -rw-r--r-- | docs/USERGUIDE.md | 13 |
2 files changed, 60 insertions, 10 deletions
diff --git a/docs/MESHBAY_DESIGN.md b/docs/MESHBAY_DESIGN.md index 4e99994..4db06c4 100644 --- a/docs/MESHBAY_DESIGN.md +++ b/docs/MESHBAY_DESIGN.md @@ -1486,9 +1486,12 @@ download land on different processes without the client doing anything. | 8 | 150 MB/s, the receiving browser being the limit | 21 % | Measured on loopback, a 1.5 GB file through the interface in a headless browser, -byte for byte. A process costs about 85 MB of memory. The default is 1, which -starts none and leaves the node exactly as it was; the operator raises it. A -change applies to new connections, and a connection already open stays where it +byte for byte. A process costs about 85 MB of memory. The default is one per +core, up to four: past four a single download gains nothing, and a process is +memory held for the life of the node. One starts none, and leaves every +connection, every lane of a download included, on the node's own core — the +case where Python's per-packet cost is the whole ceiling, and highest on Windows. +A change applies to new connections, and a connection already open stays where it is until it closes, in a process asked to leave or in the node. A process that dies takes its own connections with it, which their clients see as a dropped connection and reopen, and is replaced at the next offer. The loopback socket is @@ -1512,6 +1515,52 @@ machine nothing changes until the operator raises it; the node says so at startu The limit that remains is the node's own receive path, about 5 % of a core per MB/s. +**A receiver on aiortc reassembles in linear time and acknowledges every second +packet.** That is the command-line client in a download and the node in an +upload. aiortc 1.15 walks the reassembly from its start after every DATA chunk, +so a message of n fragments costs n²/2 steps — a 1 MiB `file_chunk` is about 870 +— and it acknowledges every packet, where RFC 4960 §6.2 asks for one SACK every +second packet; each SACK costs the sender a DTLS decryption and a parse on the +core that also sends. `sctp_fix.py` (the node's, and the client's copy) skips the +walk until a last fragment is in, and acknowledges every second packet of data, +at once on a gap, a duplicate or a packet carrying anything else, and within +200 ms otherwise. The client asks for a 4 MB receive buffer, as the node does. +Measured on loopback, 400 MB to the command-line client over one connection: 19 +MB/s with 3245 datagrams dropped by its kernel; none dropped with the buffer, and +still 20, the client at a full core; 28 with the reassembly fixed; 39 with the +acknowledgements, 49 by the node's own count of the transfer. An upload from the +client went from 25 to 29. + +**On Windows a datagram goes straight through the socket, in both directions.** +The node and its connection processes run on the ProactorEventLoop, the one that +can start ffmpeg there, and its datagram transport keeps one overlapped operation +per direction, starting the next from the completion callback of the one before, +on a later turn of the loop. A socket therefore sent one datagram per turn and +received one, and a turn lasts as long as everything else the loop does: the +other connections, the file, the hub. That is a ceiling per connection that +falls as the node gets busier, with cores to spare. With the loop otherwise busy +0.5 ms a turn: 1,928 datagrams sent a second, and 1,954 received with 87 % of +what arrived dropped by the kernel. In a node holding its connections itself, a +browser's download over eight of them ran at 9 MB/s with the node at 45 % of a +core and an SCTP round trip of 745 ms on loopback, and one of the eight sessions +was lost on the way. `proactor_udp.py` sends with a non-blocking `sendto` when +nothing is queued, and after a completed read takes whatever else the socket +holds before posting the next one; refusals and errors take the proactor's own +path. Measured on loopback on a two-core machine, 400 MB, with it: the +command-line client's download 27 to 33 MB/s, its upload 18 to 22 (23 with two +connection processes), the browser's download over eight connections 9 to 14 +before and 17 to 19 after, where the browser takes the other core. + +**A session is open from the end of its handshake, and a reconnect does not +shorten that.** Until connect() returns, a client sends only what the node takes +before the proof — `handshake`, `handshake_response`, `keypair_bundle_fetch`, +`gek_bundle_fetch`, `join_request`, `device_add_request` — and `device_hello`; +anything else waits for the session, or fails as a dropped connection, which a +download waits out. The node answers anything else with "Handshake required", a +refusal and not a dropped connection, so a chunk let through while a reconnect's +handshake runs ends the download the reconnect is for. The close event of a +channel a reconnect has replaced concerns nothing on the new one. + The lease module is free of asyncio and of the transport: it decides, and the caller does the I/O. A queue that reveals itself only through a DataChannel is a queue nobody can prove things about. @@ -2163,7 +2212,7 @@ Node page: | `max_concurrent_downloads` | 8 | node-wide download leases (§5.5) | | `max_concurrent_uploads` | 8 | node-wide upload leases. A separate pool from downloads, because the two cost different things and one queue for both makes each cap meaningless | | `max_upload_gb` | 8 | the largest single upload, §6.4. Fractions are allowed, and it is read per chunk so a change reaches an upload already running | -| `connection_processes` | 1 | how many processes carry the peer connections, §5.5. One is the node itself; above it, that many cores serve transfers. Bounded by the cores of the machine, which the Node page shows beside the field | +| `connection_processes` | one per core, up to 4 | how many processes carry the peer connections, §5.5. One is the node itself; above it, that many cores serve transfers. Bounded by the cores of the machine, which the Node page shows beside the field. `node.toml` as `init` writes it leaves the line commented, so the default reaches every node that has not chosen | | `transcode_incompatible_video` | true | whether browser-incompatible video is transcoded during streaming. Unlike remuxing this costs real CPU per viewer | `hardware_video_encode` (default true) governs whether that transcode may use the diff --git a/docs/USERGUIDE.md b/docs/USERGUIDE.md index 1808598..ab5ee8c 100644 --- a/docs/USERGUIDE.md +++ b/docs/USERGUIDE.md @@ -852,15 +852,16 @@ and running `meshbay-node reload`: | `max_concurrent_downloads` | 8 | node-wide download slots | | `max_concurrent_uploads` | 8 | node-wide upload slots | | `max_upload_gb` | 8 | the largest single file a member may send you | -| `connection_processes` | 1 | how many CPU cores carry members' connections | +| `connection_processes` | one per core, up to 4 | how many CPU cores carry members' connections | | `transcode_incompatible_video` | true | re-encode video no browser can play | | `hardware_video_encode` | true | use the GPU for that, if one actually works | -`connection_processes` is the one for speed. With 1, everything a node sends -goes through one core, about 35 MB/s shared between everyone downloading at -that moment. Raise it when several people download at once and the machine has -cores to spare: each one adds a core's worth, and about 85 MB of memory. It -applies to connections opened after the change. +`connection_processes` is the one for speed. Each process is a core's worth +of sending, shared between everyone downloading at that moment, and about 85 MB +of memory. With 1, everything a node sends goes through one core: about 35 MB/s +on Linux, much less on Windows. Lower it on a machine short of memory; raise it +above 4 only when many people download at once. It applies to connections +opened after the change. `max_concurrent_streams` is the one to think about: a stream costs a video process for as long as the film lasts, so this counts **viewers**, not |