meshbay.git/packages/meshbay-common/src/meshbay_common/protocol.py, branch 0.13

refactor(mnp)!: one operation for an app's folders, not one per app

2026-09-10T15:49:58Z

`video_root`, `audio_root` and `photo_roots` are gone — the messages, the signed operations, the handlers, the `ops` wrappers, the three scalars on the handshake ack, and the client's handlers for their acks. `app_directories` does the same thing for every application, keyed by the app's own registry name, and it is what the SPA has been sending. The three were the same instruction three times, differing only in the key they wrote and whether they carried a string or a list. That shape is what made adding an application mean adding a message type, an op, a handler and a widget; it also meant three validation paths, and the older ones validated nothing — a typo was stored and then quietly matched no entry, an app showing an empty tab with no way to tell "misconfigured" from "no files yet". **What stays, and why.** `Roster.LEGACY_DIR_KEYS` still reads `video_root` and friends out of `group_settings`: that is a key on an operator's disk, not on the wire, and a node upgraded into this must find its own configuration. The Search page still reads its own older cache keys, for the same reason — the cache outlives a deploy. `CTX_ALIASES` keeps only `chat`, which is the one app whose second name something still reads. The two per-app policy test files go with the messages. What only they held — the real challenge/response path from message to database, which no other test exercises — is retargeted at `app_directories` in `test_app_directories_signed.py`, and the handler's own refusals (unknown app, malformed `directories`, nobody to authorize it) join `test_app_directories.py`. Node and common suites 1368 passed. Co-Authored-By: Claude Opus 5 Claude-Session: https://claude.ai/code/session_01AsoWC3GmhNdwVFomW3QjH3

refactor(mnp)!: one answer to "may this member write", and it is the root

2026-09-10T15:23:40Z

The group-wide `member_upload` switch is gone: the message, the signed operation, the field on the handshake ack, the `upload` alias on every root in the index payload, and the client's fallback path to it. Whether a member may write has been a property of each root for a while, and that is the model that survives: a single flag over the group cannot express "this library is published read-only and that folder is a drop box", which is the ordinary arrangement. What was left of the switch was a handler that logged a deprecation and acted on nothing, and a client that read `ack.member_upload` whenever the roots carried no `writable` — a second source for one question, with whichever the code consulted first deciding it. `roots.describe()` drops `upload` for the same reason: it was `writable` under an older name, and two names for one boolean is one too many. The paperclip now says "nowhere to write" rather than picking a root, in a group that has none writable. That is the honest answer; the fallback picked whatever came first and failed at send time. Node suite 1215 passed. Co-Authored-By: Claude Opus 5 Claude-Session: https://claude.ai/code/session_01AsoWC3GmhNdwVFomW3QjH3

feat: resume an interrupted upload, and pause one

2026-09-09T10:01:06Z

Stage 8 of ~/next/improve-downloads.md, second half, plus the gap it exposed in stage 7. **Asking where to resume.** The node identifies an upload by (member, directory, filename), so a client resuming one has to name the file — and `transfer_open`, the obvious place to ask, travels in clear. Naming it there would undo exactly what sealing this path bought in MNP 2.0: before it, the same file was ciphertext leaving a node and plaintext arriving at one. So the question is asked inside the seal that already exists, as an ordinary `file_upload` with no bytes and `chunk_index: -1`. The node writes nothing, creates no state, reserves no name, and answers with `resume_from` in the sealed ack. A node that predates it refuses the index, which the client reads as "start from the beginning" — the behaviour it had anyway — and the wait is bounded so one that answers neither does not strand an upload. The probe is answered after every check the write path makes, so it cannot ask questions about a directory the caller may not write to, and it answers only about the member who asks: otherwise one member could measure another's progress on a file they never sent, and worse, resume it. **Pausing an upload.** Reported: no pause button on an upload, even in the desktop app. Stage 7 built pause around the download path — a target declares whether it can be stopped — and an upload has no local target to ask. It was also refused by design, since a transfer handed a lease it cannot re-create must not be offered a button that would drop its slot for good. Uploads now ask for their slot rather than being handed one, and say they are pausable outright: a File is seekable and the node keeps the position. Resuming re-probes rather than trusting the client's own memory, so it works across a reconnect too. **And the slot they hold.** `_do_file_upload` never called `slots.touch(tr)`. Chunks are not gated by the lease, so the file arrived — but the node reclaimed a grant nobody appeared to be using after thirty seconds, twice, then abandoned it, and the widget follows the lease. Measured from the journal: a 3.5 GB upload read "waiting, 0 ahead" for a minute and a half while it was transferring. The download twin of this was fixed on 2026-09-08; the same omission was still here, invisible until uploads took a real lease. `test_the_upload_itself_is_sealed` now checks every message `uploadFile` sends rather than the first. Adding the probe put a second one in front of the one it was written for, and it would have kept passing while guarding nothing. Node suite 1202 passed, hub suite 850 passed. Co-Authored-By: Claude Opus 5 Claude-Session: https://claude.ai/code/session_01HCGdheDLxGReuKHga3BtST

feat(node): make the transfer caps settable, node-wide and per group

2026-09-08T12:53:03Z

Step 3 of ~/next/improve-downloads.md. Step 2 built the pools with constants; this gives them to the operator, in the two scopes they belong to. **The pools are the machine's.** `[node] max_concurrent_downloads` and `max_concurrent_uploads`, default 8, on the §2.11 pattern: node.toml for a fresh install, a roster.db override for immediate effect, editable from the Node page and from `meshbay-node transfers show|set`, applied live through the one `set_capacity` step 1 fixed. **The per-member cap is a group's.** How many transfers one member may run at once here — on the node like every other group setting (not the hub, which would have authority over someone else's disk; not node.toml, which is hand-written and needs a restart), changed by a signed operator instruction (`OP_TRANSFER_LIMITS`, subject "d=2,u=2" so what is signed names the outcome), broadcast to the group, and read live by the pools. That was the one thing step 2's shape could not express: `per_member` was a single node-wide number. `group_limits` and `member_cap(kind, member)` make it a lookup — the group's own value if it has one, the node's default otherwise — and it is deliberately the only dimension that is not node-wide. Three refusals, each with a test: - **absent means the default (2), never "unlimited".** A group that predates the setting coming back unlimited would leave the node-wide pool as the only control, which is the situation slots exist to end; - **zero is not "unlimited"**, and is not "this member may not transfer" either: the floor is one everywhere, and the CLI says to revoke the member instead; - **an unreadable row reads as unset**, not as zero — the same discipline the sealed messages follow, where a payload that does not open must never become a default state on its own. `handshake_ack` carries this member's own caps for this group, so the interface can say "2 of your 2 slots are busy" instead of drawing a bare spinner. Absent reads as "no limit known" and the hint is not drawn — never as "unlimited", which would have the interface contradicting the node. 1164 node, 793 hub, 0 failed. Co-Authored-By: Claude Opus 5 Claude-Session: https://claude.ai/code/session_01HCGdheDLxGReuKHga3BtST

feat(node): transfer leases, pools and a queue for downloads and uploads

2026-09-08T12:20:57Z

Step 2 of ~/next/improve-downloads.md. A download is invisible to the node: it is a series of independent `file_req` messages, with nothing saying one started or ended, so there is nothing to count and nothing to cap. The lease is that missing object. `meshbay_node/transfers.py` holds the decisions and has no asyncio and no transport in it, on purpose. The failure modes this has to survive — a slot the node never gets back, a client waiting on a grant the node has forgotten — are races through a DataChannel and unprovable there; here the clock is a parameter and every method returns what changed, so the caller does the I/O and the tests drive the worst case directly. What it decides: - two pools, downloads and uploads, separate from the stream pool: different resources with different costs, and merging them makes both caps meaningless; - per-member cap checked *before* the node-wide one, so a member at their own limit queues behind their own transfers rather than holding a slot a second member has none of. Per account across their devices, or the cap becomes a function of how many tabs somebody opens; - a queue that skips a member at their cap instead of waiting for them — granting strictly in arrival order lets one member's limit stall everyone; - `tr` drawn by the client and idempotent, which is what makes a reconnect safe; - bounded per member, because unbounded queues are how a node runs out of memory politely. Every way a slot comes back, with the session teardown as the one that matters (a closed tab, a quit browser and a dead network all arrive at `shutdown_tasks`, and none of them needs a timer): explicit close, session gone, a grant nobody took up in 30 s passed to the next in line, and a granted transfer silent for 120 s reclaimed with its peer told, so a widget can offer a resume rather than sit on a lie. `GET /api/transfers` is the operator's window: when somebody reports a transfer stuck at waiting, it is the only thing that says whether the node ever had them in a queue — a log cannot, when the symptom is that nothing is happening. It carries no filename and no path, which a test pins, because this is exactly where one would be tempting. Three things found while writing it, two of them mine: - the randomised property test rejected `in_use <= cap` at once, and it was right to: lowering a cap never interrupts a running transfer, so the count legitimately sits above the new value. The invariant is that a *new* grant never happens past the cap; - the sweeper was started with `self._spawn`, which ties a task to one session's set. It died with whichever peer opened the first transfer, and every other peer's abandoned lease then stopped being reclaimed — a node that fills up over days with nothing in the log. It belongs to the node now, with its strong reference on the transport context; - the pools are node-wide while `_peer_registry` is per group (finding H1), so a slot freed in one group can grant one in another and the peer to notify is not in the notifier's registry. Silently wrong in the first version. Nothing enforces a lease yet: `file_req` is untouched, no client asks, and the node grants everything. That is step 4's flag day, and this lands alone. 1148 node, 793 hub, 0 failed. Co-Authored-By: Claude Opus 5 Claude-Session: https://claude.ai/code/session_01HCGdheDLxGReuKHga3BtST

feat(chat): Tier 2 — a member verifies another member's device itself

2026-09-07T19:04:56Z

Chat messages have been signed by the sending device since MNP 2.0, but a reader had no way to know that the device belonged to the account the node named: the signature proved *a device*, and `sender_id` was still the node's word. This closes that for any account a client has already seen. **What was blocking it was not effort — the evidence was not being kept.** `_do_device_add` verified the countersignature that admits a second device and stored only `added_by_pk`: *which* key approved, never the proof. And `device_add_transcript` binds `nonce_node`, the approving connection's handshake nonce, so even a stored signature was unverifiable by anyone who had not been on that connection. `identities` gains `add_sig`, `add_nonce` and `add_ts`, added before the migration's early return — which fires on every roster widened since 2026-08-18, i.e. all of them, so putting them inside it would have meant they never arrived. `group_roster_req`/`resp` relays, sealed under a new groupbox purpose and answered to **any member of the group**, every live device of every active member with the evidence that admitted it. The node decides nothing: it hands over evidence and the client walks the chain from each account's root outwards (`_verifyRoster`). That is deliberate — the node is the party the property holds against, so it is not asked to assert trust. Two holes the tests caught while this was being built: - "no signature" was being treated as a trust root, so a node that writes the roster could put any key in an account's row and have it laundered straight into the verified set. A root is a device that names **no** countersigner. - pinning only the verified subset at first sight raised "key changed" on legitimate second devices whose countersignature predates this change. First sight pins everything the node says, because that is what trust-on-first-use means and an alarm that fires on normal events stops being read. The property, and it must not be rounded up: **once a client has seen an account, a node that later substitutes a key for it is detected. Nothing is gained at first sight**, where there is nothing to compare against — the same boundary `per-node-identity-v1.md` draws, unmoved. The cost, stated because it is real: the roster is member-visible, so every member learns how many devices the others hold and their public keys. It stays inside the group, the hub is not involved, and it is scoped per group. A member who cannot see the keys cannot check them. User-visible surface: one notice, "this account is using a key you have not seen before", in ten languages. Nothing else. 16 tests — 7 on the node (the evidence is stored, it verifies from the roster alone, a fabricated device carries none, another group's members are not disclosed), 9 running the shipped `_verifyRoster` under node against rosters built by the shipped Python: a chain of three in any order, a signature by the wrong key, one for another node, one for another account, and two fabricated devices signing each other admitting nothing. Tier 3 (operator-signed roster attestation) stays deferred, with nothing depending on it. Co-Authored-By: Claude Opus 5 Claude-Session: https://claude.ai/code/session_01TZZxYjz8YeWRz13xDi8LJr

Merge origin/main into the chat encryption work

2026-09-07T16:03:52Z

Both sides landed a breaking MNP change and both called it 2.0, which is right: the sealed upload, the removal of `stream_seg` and mandatory chat encryption share one flag day. They are recorded as one version in `__init__.py` rather than as a race between two. The resolutions that were decisions rather than mechanics: * **`MNP_MIN_SUPPORTED` moves to "2.0".** The sealed upload alone was a *confined* break — a 1.x peer could still connect, browse, download, stream and chat, with only its uploads refused by `upload_not_sealed` — so the floor deliberately stayed at "1.0". Mandatory chat encryption ends that confinement: a 1.x peer can neither produce a sealed chat message nor read one, so it would connect, look fine, and be unable to say anything. Refusing it at the handshake is the honest form. The per-message `upload_not_sealed` path is untouched and still right if the floor is ever lowered. * **`sendChat` throws on an `error` reply**, from origin, applied to the sealed send. It matters more after this change, not less: the node now refuses a stale epoch, a malformed envelope and a device claim that is not the connection's own, so there are three new ways for a message to be rejected and none of them may look like a message that was sent. * **`req_id` supersedes the per-type routing** this branch added for `chat_keys_resp` and `device_hello_ack`. Both blocks are kept beside the existing `chat_hist_resp` one, for the same stated reason — a node too old to stamp — and their comments no longer claim to be the mechanism that closes the class. `req_id` is. * **`chat_send_probe.py` is rebuilt on origin's structure**, not beside it: two scenarios, a stub that stamps `req_id`, `music_meta_req` as the older pending request. The encrypted path is layered on — a real Ed25519 device key generated in the page, and a `chat_keys_resp` sealed by the shipped Python, because a payload the page built itself would prove only that the page agrees with the page. * **`test_reply_correlation.py` now sends a sealed message.** Its subject is which of the two messages leaving that handler carries the id; plaintext chat was only the fixture, and the node refuses one now. * `groupbox` keeps both new purposes (`upload`, `chat_keys`); `protocol.py` keeps origin's removal of `STREAM_SEGMENT` and this branch's correction of the "Double Ratchet message" comment on `CHAT_MESSAGE`, which was wrong when it was written and is wrong differently now. Full suite on the merged tree: 1993 passed, 11 failed — the same 11 that fail on a pristine checkout (2 Windows service tests, 1 apps-enabled policy, 7 transcode tests that pass in isolation, and the WebRTC invite test that hangs on its own). Co-Authored-By: Claude Opus 5 Claude-Session: https://claude.ai/code/session_01TZZxYjz8YeWRz13xDi8LJr

feat(chat): encrypt group chat under per-device epoch keys (MNP 2.0)

2026-09-07T15:50:28Z

Chat messages are sealed with AES-256-GCM under a key derived per group, per epoch, per *device*, and signed over the ciphertext with the device key the node pinned. The node relays and archives; it cannot read a message. There is no switch. MNP goes to 2.0 and MNP_MIN_SUPPORTED moves with it, so a 1.x peer is refused at the handshake with `version_too_old` rather than admitted and then unable to speak. An opt-in flag was designed and rejected: every node is a test node, so it would have bought nothing and left a plaintext branch reachable — C6's lesson one feature later. A test reads the source and refuses any code that consults a `chat_encrypted` setting. Not Sender Keys, and `senderkeys.py` is now documented as unused. With distribution under the group key and a node that serves history to devices which were not present, the node must retain each chain's earliest key, and a chain key at iteration i yields every message key from i on by pure HKDF — forward secrecy is zero either way. What the ratchet was left buying was stateful client code with silent failure modes, three of them reproduced: any member could sign as any other, a second device dropped the first's chain, and the skipped-key cache grew without bound. The reasoning is in docs/chat-sender-keys.md, which is the specification and the decision record. Epochs, not rotation: the epoch key is wrapped under the group key at delivery and never stored under it, so `gek_rotate` is a re-wrap. A group-key-derived archive key would have made every message ever sent unreadable on the first `member unpin`, which is the documented step after removing a member. A new epoch opens on member revoke/unpin, device revoke and `gek_rotate`; old epochs are kept and still delivered, so history stays readable to everyone who could already read it, and nothing anywhere deletes one. Three prerequisites this needed, each a live defect on its own: * The peer registry was keyed by user_id, so one account's second device evicted the first and the broadcast skipped recipients by account — a person's phone never saw what they typed on their laptop. * The handshake authenticated an account, never a device. `device_hello` (additive, signed, refused unless the key is a live device of this account in the node's own roster) is what lets the node refuse a member claiming somebody else's key. * `_admin_exec_file_delete` authorized against the exact uploading key, so device linking had already broken deleting your own file from your other device. It now authorizes against any non-revoked device of `uploader_id`. Found by driving the real panel over the real transport, not by reading source: `chat_keys_resp` was routed by arrival order and handed to an unanswered `media_meta_req` — the original frozen-tab defect in a message type that did not exist when that probe was written. And `_asText` had been deleted with an unrelated helper beside it; its only caller sits inside a promise the panel catches, so every conversation rendered empty with nothing in the console. Existing node data is migrated by QE/migration/migrate_chat_encryption.py (not versioned, per the QE rule), run with the node stopped. Co-Authored-By: Claude Opus 5 Claude-Session: https://claude.ai/code/session_01TZZxYjz8YeWRz13xDi8LJr

feat(mnp)!: seal the upload under the group key

2026-09-07T15:46:33Z

Downloads have been encrypted under a GEK-derived key since the beginning: `file_chunk` and `stream_data` both go through `chunk_ciphertext`. Uploads never were. `file_upload` carried the filename and the raw bytes in plain msgpack, and `file_upload_ack` carried the name the node stored them under — so the same file was ciphertext leaving a node and plaintext arriving at one. There was no threat model behind that asymmetry. Both halves now travel sealed under a third groupbox purpose, HKDF(GEK, info="meshbay:upload:v1"). The filename, the destination folder and the bytes are all inside the seal; only `upload_id` and `chunk_index` stay in clear, because the node routes and orders on them before it can decrypt. This direction seals *towards* the node — it holds the GEK for its own group — and it opens the payload before it picks a destination or touches the disk. What that forced, and why none of it is optional: - `filename` was the correlation key on both sides. It cannot be: matching an ack to its request by name would hand back exactly what the seal hides. `upload_id` replaces it — client-drawn, opaque to the node, unique within a connection, never an authorization input. The property it guarded (one refusal fails one upload, not every upload in flight) is unchanged. - Refusals can no longer quote what they refused. `No directory named 'X'` becomes `No such directory in this group` plus the `code` that was already there; the client knows what it sent. - No plaintext fallback. A path that still accepts plaintext is not a sealed path, so an unsealed `file_upload` is refused with `upload_not_sealed`. Hardened while here, because what comes out of a seal is authenticated but not validated — a member can seal anything: `filename` and `data` have their types checked before any upload state is created, and `chunk_index`/`total_chunks`, which are outside the seal by necessity, can no longer raise where a refusal was meant. Tests. `test_upload_sealed.py` pins the node half: nothing identifying on the wire, tamper/wrong-key/wrong-group all refused with nothing written, and multi-chunk reassembly unchanged. `test_upload_seal_client.py` drives the shipped `uploadFile` over the shipped `crypto.js` under node and feeds its real frames to the real `_do_file_upload` — the file lands intact, and the ack the node actually produced comes back with the name it chose for a collision, which is the half a source-reading test cannot see. Both upload purposes join the JS/Python groupbox parity vectors. BREAKING CHANGE: MNP 2.0. `file_upload`/`file_upload_ack` change shape on the wire every deployed client speaks, which is MAJOR by the same rule 1.0 was — but the break is confined to uploads. `MNP_MIN_SUPPORTED` stays at "1.0", so a 1.x peer still connects, browses, downloads, streams and chats; only its uploads are refused, with a message saying which side is old. The client checks the node's version before sending a chunk, so neither side meets this as a timeout. This is the version negotiation shipped in 1.0 earning its keep: 1.0 cost a flag day, 2.0 costs a refusal code. Co-Authored-By: Claude Opus 5 Claude-Session: https://claude.ai/code/session_01AsoWC3GmhNdwVFomW3QjH3

refactor(mnp)!: remove stream_seg, the last unencrypted content message

2026-09-07T15:45:54Z

`stream_seg` answered with an MPEG-TS segment as base64 with no encryption at all — the one message on the content plane that never went through a GEK-derived key. Live on both transports, answering any authenticated member. It predates `stream_data`, which does the same job properly (`chunk_ciphertext`, keyed per segment, AES-256-GCM) and has since Phase 12. Its only browser caller, `fetchStreamSegment`, was defined and never once invoked — a plaintext media endpoint with no client. Removed rather than repaired. Gone with it: `_extract_segment` and the ffmpeg semaphore in quic_server, the `fetch_stream_segment` QUIC client method, and `_b64decode` in transport.js, which had no other caller. The H6 regression test lived on this handler — it pinned `_do_stream_segment_async` to a coroutine so `subprocess.run` could not stall the event loop for thirty seconds per request. It is replaced by the property that outlives the handler: no transport carries media outside an AEAD, asserted on `stream_seg` and `data_b64` across all three transport modules. The half of H6 that survives — the live streaming path still spawns ffmpeg — keeps its own test. BREAKING CHANGE: `stream_seg` is no longer answered on either transport. No shipping client sends it. Recorded as part of MNP 2.0, whose other half — the sealed upload — carries the version bump. Co-Authored-By: Claude Opus 5 Claude-Session: https://claude.ai/code/session_01AsoWC3GmhNdwVFomW3QjH3