docs: receiver-side SCTP rules, session gating on reconnect, processes default

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

3 files changed, 81 insertions, 10 deletions

diff --git a/CLAUDE.md b/CLAUDE.md
index 5048488..dfcb0ed 100644
--- a/CLAUDE.md
+++ b/CLAUDE.md
@@ -761,6 +761,47 @@ do. Read them before writing anything that touches the same mechanism.
   socket buffer gave 19.5 MB/s (`udp_buffer.py`); it needs
   `net.core.rmem_max` raised on a default machine
 
+- **A transfer capped with a core busy: profile that process before tuning
+  the network.** The command-line client downloaded at 19 MB/s on loopback.
+  Its 4 MB receive buffer removed 3245 kernel drops and changed nothing else;
+  `top` then showed the client, not the node, at 100 %, and cProfile (a
+  wrapper script as `MESHBAY_CLI_E2E_BIN`, the e2e harness does the rest) put
+  two thirds of its time in aiortc's `InboundStream.pop_messages`: 153 million
+  steps for 400 MB, the reassembly walked from its start after every fragment.
+  Next, a SACK per packet cost the node more than sending. Both fixed in
+  `sctp_fix.py`: 39 MB/s, the node at 49. No amount of reading our code finds
+  a quadratic loop in a dependency; one profile does
+
+- **A default of "off" ships a feature nobody has.** `connection_processes`
+  defaulted to 1, which starts no process: the multi-core path was measured at
+  131 MB/s and reached by no node whose operator had not found the setting, so
+  on Windows a download over eight lanes landed on the node's one core and ran
+  at the speed of one connection. Worse, the Node page saved **every** setting
+  on each Save, so any operator who changed one thing had pinned all the others
+  at that day's defaults, in the roster and in `node.toml`. It sends only what
+  changed now, and `init` leaves the line commented. A test node that saved
+  its settings before this keeps `connection_processes = 1` until it is set
+  again on the Node page
+
+- **A reconnect let every message through, not just its handshake.**
+  `_inReconnectAttempt` opened `_sendAndWait` to all callers so the handshake
+  would not wait on itself; a chunk retry landing between the new channel's
+  open and the proof got "Handshake required", which no retry treats as a
+  dropped connection, and the download died with that text in the transfers
+  panel. Only the opening messages pass now (`SESSION_OPENING`, held to
+  `dispatch.py` by `test_reconnect_race.py`), and a replaced channel's late
+  `close` no longer clobbers the new one. `harness/reconnect_race_harness.mjs`
+  runs the real `connect()`, `_reconnectLoop` and pipeline against a node that
+  refuses before the proof — the first harness here that does; the earlier
+  ones stub `connect`, which is exactly where this lived
+
+- **A push handler that re-fetches the world.** The interactive CLI answered
+  every `index_delta` by fetching and re-parsing the whole index, on the event
+  loop that was receiving the download: 67 full indexes in 80 s while a node
+  enriched a library (one delta a second). It applies the delta in place
+  (`Tree.apply_delta`), and only a delta that adds or removes files asks again,
+  once per burst
+
 ### More of the same, from four rounds of live testing
 
 - **A protocol harness cannot test the SPA.** Any second implementation of the
diff --git a/docs/MESHBAY_DESIGN.md b/docs/MESHBAY_DESIGN.md
index 4e99994..e10209a 100644
--- a/docs/MESHBAY_DESIGN.md
+++ b/docs/MESHBAY_DESIGN.md
@@ -1486,9 +1486,12 @@ download land on different processes without the client doing anything.
 | 8 | 150 MB/s, the receiving browser being the limit | 21 % |
 
 Measured on loopback, a 1.5 GB file through the interface in a headless browser,
-byte for byte. A process costs about 85 MB of memory. The default is 1, which
-starts none and leaves the node exactly as it was; the operator raises it. A
-change applies to new connections, and a connection already open stays where it
+byte for byte. A process costs about 85 MB of memory. The default is one per
+core, up to four: past four a single download gains nothing, and a process is
+memory held for the life of the node. One starts none, and leaves every
+connection, every lane of a download included, on the node's own core — the
+case where Python's per-packet cost is the whole ceiling, and highest on Windows.
+A change applies to new connections, and a connection already open stays where it
 is until it closes, in a process asked to leave or in the node. A process that
 dies takes its own connections with it, which their clients see as a dropped
 connection and reopen, and is replaced at the next offer. The loopback socket is
@@ -1512,6 +1515,32 @@ machine nothing changes until the operator raises it; the node says so at startu
 The limit that remains is the node's own receive path, about 5 % of a core per
 MB/s.
 
+**A receiver on aiortc reassembles in linear time and acknowledges every second
+packet.** That is the command-line client in a download and the node in an
+upload. aiortc 1.15 walks the reassembly from its start after every DATA chunk,
+so a message of n fragments costs n²/2 steps — a 1 MiB `file_chunk` is about 870
+— and it acknowledges every packet, where RFC 4960 §6.2 asks for one SACK every
+second packet; each SACK costs the sender a DTLS decryption and a parse on the
+core that also sends. `sctp_fix.py` (the node's, and the client's copy) skips the
+walk until a last fragment is in, and acknowledges every second packet of data,
+at once on a gap, a duplicate or a packet carrying anything else, and within
+200 ms otherwise. The client asks for a 4 MB receive buffer, as the node does.
+Measured on loopback, 400 MB to the command-line client over one connection: 19
+MB/s with 3245 datagrams dropped by its kernel; none dropped with the buffer, and
+still 20, the client at a full core; 28 with the reassembly fixed; 39 with the
+acknowledgements, 49 by the node's own count of the transfer. An upload from the
+client went from 25 to 29.
+
+**A session is open from the end of its handshake, and a reconnect does not
+shorten that.** Until connect() returns, a client sends only what the node takes
+before the proof — `handshake`, `handshake_response`, `keypair_bundle_fetch`,
+`gek_bundle_fetch`, `join_request`, `device_add_request` — and `device_hello`;
+anything else waits for the session, or fails as a dropped connection, which a
+download waits out. The node answers anything else with "Handshake required", a
+refusal and not a dropped connection, so a chunk let through while a reconnect's
+handshake runs ends the download the reconnect is for. The close event of a
+channel a reconnect has replaced concerns nothing on the new one.
+
 The lease module is free of asyncio and of the transport: it decides, and the
 caller does the I/O. A queue that reveals itself only through a DataChannel is a
 queue nobody can prove things about.
@@ -2163,7 +2192,7 @@ Node page:
 | `max_concurrent_downloads` | 8 | node-wide download leases (§5.5) |
 | `max_concurrent_uploads` | 8 | node-wide upload leases. A separate pool from downloads, because the two cost different things and one queue for both makes each cap meaningless |
 | `max_upload_gb` | 8 | the largest single upload, §6.4. Fractions are allowed, and it is read per chunk so a change reaches an upload already running |
-| `connection_processes` | 1 | how many processes carry the peer connections, §5.5. One is the node itself; above it, that many cores serve transfers. Bounded by the cores of the machine, which the Node page shows beside the field |
+| `connection_processes` | one per core, up to 4 | how many processes carry the peer connections, §5.5. One is the node itself; above it, that many cores serve transfers. Bounded by the cores of the machine, which the Node page shows beside the field. `node.toml` as `init` writes it leaves the line commented, so the default reaches every node that has not chosen |
 | `transcode_incompatible_video` | true | whether browser-incompatible video is transcoded during streaming. Unlike remuxing this costs real CPU per viewer |
 
 `hardware_video_encode` (default true) governs whether that transcode may use the
diff --git a/docs/USERGUIDE.md b/docs/USERGUIDE.md
index 1808598..ab5ee8c 100644
--- a/docs/USERGUIDE.md
+++ b/docs/USERGUIDE.md
@@ -852,15 +852,16 @@ and running `meshbay-node reload`:
 | `max_concurrent_downloads` | 8 | node-wide download slots |
 | `max_concurrent_uploads` | 8 | node-wide upload slots |
 | `max_upload_gb` | 8 | the largest single file a member may send you |
-| `connection_processes` | 1 | how many CPU cores carry members' connections |
+| `connection_processes` | one per core, up to 4 | how many CPU cores carry members' connections |
 | `transcode_incompatible_video` | true | re-encode video no browser can play |
 | `hardware_video_encode` | true | use the GPU for that, if one actually works |
 
-`connection_processes` is the one for speed. With 1, everything a node sends
-goes through one core, about 35 MB/s shared between everyone downloading at
-that moment. Raise it when several people download at once and the machine has
-cores to spare: each one adds a core's worth, and about 85 MB of memory. It
-applies to connections opened after the change.
+`connection_processes` is the one for speed. Each process is a core's worth
+of sending, shared between everyone downloading at that moment, and about 85 MB
+of memory. With 1, everything a node sends goes through one core: about 35 MB/s
+on Linux, much less on Windows. Lower it on a machine short of memory; raise it
+above 4 only when many people download at once. It applies to connections
+opened after the change.
 
 `max_concurrent_streams` is the one to think about: a stream costs a video
 process for as long as the film lasts, so this counts **viewers**, not