diff options
| author | Christophe Besson <cbesson@gmail.com> | 2026-09-07 18:03:52 +0200 |
|---|---|---|
| committer | Christophe Besson <cbesson@gmail.com> | 2026-09-07 18:03:52 +0200 |
| commit | e1383e1d545b994f4ad61694f868339defb0bdef (patch) | |
| tree | 67a3b1933f2a9c5caf107d01d9ff91c44a975df6 /packages/meshbay-hub | |
| parent | 36cebf25d0e0f24cf63be4380ccb5d03da726a74 (diff) | |
| parent | 8980a8e42d94ab7c0bc9739283d39f938f8402b0 (diff) | |
| download | meshbay-e1383e1d545b994f4ad61694f868339defb0bdef.tar.gz | |
Merge origin/main into the chat encryption work
Both sides landed a breaking MNP change and both called it 2.0, which is right:
the sealed upload, the removal of `stream_seg` and mandatory chat encryption
share one flag day. They are recorded as one version in `__init__.py` rather
than as a race between two.
The resolutions that were decisions rather than mechanics:
* **`MNP_MIN_SUPPORTED` moves to "2.0".** The sealed upload alone was a
*confined* break — a 1.x peer could still connect, browse, download, stream
and chat, with only its uploads refused by `upload_not_sealed` — so the floor
deliberately stayed at "1.0". Mandatory chat encryption ends that
confinement: a 1.x peer can neither produce a sealed chat message nor read
one, so it would connect, look fine, and be unable to say anything. Refusing
it at the handshake is the honest form. The per-message `upload_not_sealed`
path is untouched and still right if the floor is ever lowered.
* **`sendChat` throws on an `error` reply**, from origin, applied to the sealed
send. It matters more after this change, not less: the node now refuses a
stale epoch, a malformed envelope and a device claim that is not the
connection's own, so there are three new ways for a message to be rejected
and none of them may look like a message that was sent.
* **`req_id` supersedes the per-type routing** this branch added for
`chat_keys_resp` and `device_hello_ack`. Both blocks are kept beside the
existing `chat_hist_resp` one, for the same stated reason — a node too old to
stamp — and their comments no longer claim to be the mechanism that closes
the class. `req_id` is.
* **`chat_send_probe.py` is rebuilt on origin's structure**, not beside it: two
scenarios, a stub that stamps `req_id`, `music_meta_req` as the older pending
request. The encrypted path is layered on — a real Ed25519 device key
generated in the page, and a `chat_keys_resp` sealed by the shipped Python,
because a payload the page built itself would prove only that the page agrees
with the page.
* **`test_reply_correlation.py` now sends a sealed message.** Its subject is
which of the two messages leaving that handler carries the id; plaintext chat
was only the fixture, and the node refuses one now.
* `groupbox` keeps both new purposes (`upload`, `chat_keys`); `protocol.py`
keeps origin's removal of `STREAM_SEGMENT` and this branch's correction of
the "Double Ratchet message" comment on `CHAT_MESSAGE`, which was wrong when
it was written and is wrong differently now.
Full suite on the merged tree: 1993 passed, 11 failed — the same 11 that fail
on a pristine checkout (2 Windows service tests, 1 apps-enabled policy, 7
transcode tests that pass in isolation, and the WebRTC invite test that hangs
on its own).
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01TZZxYjz8YeWRz13xDi8LJr
Diffstat (limited to 'packages/meshbay-hub')
| -rw-r--r-- | packages/meshbay-hub/src/meshbay_hub/static/crypto.js | 11 | ||||
| -rw-r--r-- | packages/meshbay-hub/src/meshbay_hub/static/transport.js | 253 | ||||
| -rw-r--r-- | packages/meshbay-hub/tests/harness/chat_send_probe.py | 242 | ||||
| -rw-r--r-- | packages/meshbay-hub/tests/harness/upload_seal_probe.mjs | 102 | ||||
| -rw-r--r-- | packages/meshbay-hub/tests/test_chat_send.py | 158 | ||||
| -rw-r--r-- | packages/meshbay-hub/tests/test_index_seal_client.py | 30 | ||||
| -rw-r--r-- | packages/meshbay-hub/tests/test_transport_contracts.py | 50 | ||||
| -rw-r--r-- | packages/meshbay-hub/tests/test_upload_seal_client.py | 169 |
8 files changed, 770 insertions, 245 deletions
diff --git a/packages/meshbay-hub/src/meshbay_hub/static/crypto.js b/packages/meshbay-hub/src/meshbay_hub/static/crypto.js index f9cff08..24d1399 100644 --- a/packages/meshbay-hub/src/meshbay_hub/static/crypto.js +++ b/packages/meshbay-hub/src/meshbay_hub/static/crypto.js @@ -111,8 +111,9 @@ async function decryptChunkBin(gek, fileHashHex, chunkIndex, nonce, ct) { // ── Sealing a payload under the group key ──────────────────────────────────── // -// Mirrors meshbay_common/groupbox.py. `index_sync`, `index_delta` and the -// `handshake_ack` config payload travel sealed under a GEK-derived subkey; the +// Mirrors meshbay_common/groupbox.py. `index_sync`, `index_delta`, the +// `handshake_ack` config payload and both halves of an upload travel sealed +// under a GEK-derived subkey; the // routing fields (type, v, group_id) and the ack's own authentication (node_pk, // proof, sig) stay in clear, because a receiver must route, version-check and // *authenticate* before it would trust a decryption. @@ -126,6 +127,12 @@ async function decryptChunkBin(gek, fileHashHex, chunkIndex, nonce, ct) { const GROUPBOX_INFO = { index: new TextEncoder().encode('meshbay:index:v1'), ack: new TextEncoder().encode('meshbay:ack:v1'), + // MNP 2.0: `file_upload` and `file_upload_ack`. This is the one purpose that + // seals *towards* the node — it holds the GEK for its own group — and the one + // with real message volume, one per 48 KB chunk. groupbox.py carries the + // nonce-collision arithmetic that makes a random 96-bit nonce fine at that rate. + upload: new TextEncoder().encode('meshbay:upload:v1'), + // The group's chat epoch keys, on their way to a member. chat_keys: new TextEncoder().encode('meshbay:chat_keys:v1'), }; diff --git a/packages/meshbay-hub/src/meshbay_hub/static/transport.js b/packages/meshbay-hub/src/meshbay_hub/static/transport.js index 927956d..1f6dd8f 100644 --- a/packages/meshbay-hub/src/meshbay_hub/static/transport.js +++ b/packages/meshbay-hub/src/meshbay_hub/static/transport.js @@ -251,7 +251,7 @@ window.addEventListener('hashchange', () => { // The `v: '0.1'` on every other message in this file is the historical value // and is read by nothing; it is left alone deliberately. The range is // negotiated once, at the start, not restated per message. -const MNP_V = '1.1'; +const MNP_V = '2.0'; const MNP_V_MIN = '1.0'; // Codes a NODE sends us, in its own vocabulary (meshbay_common/handshake.py's @@ -290,6 +290,11 @@ class MeshBayTransport { this._pc = null; this._channel = null; this._pending = new Map(); + // Set the first time this connection sees a reply that names the request + // it answers (see _dispatch). A node either stamps every reply or none, + // so one is proof for the connection — and once there is proof, the + // arrival-order fallback at the bottom of _dispatch is never right again. + this._correlates = false; this._seqId = 0; this._recvBuf = new Uint8Array(0); this._connected = false; @@ -299,10 +304,17 @@ class MeshBayTransport { this._onStreamEnd = null; this._onStreamError = null; this._onIndexSync = null; - // filename → the uploader waiting on it. Keyed rather than FIFO because - // several uploads may be in flight at once and their acks interleave; the - // node names the file in every one. + // upload_id → the uploader waiting on it. Keyed rather than FIFO because + // several uploads may be in flight at once and their acks interleave. + // + // It was keyed by filename until MNP 2.0, which is no longer possible: the + // name is sealed under the group key, and echoing it in clear so the two + // sides could match on it would give back precisely what the seal is for. + // `upload_id` is drawn per upload here and is opaque to the node. this._uploaders = new Map(); + // Names, not ids: the "already being uploaded" guard is about the file the + // caller passed, and two `uploadFile` calls for one file draw two ids. + this._inFlightUploads = new Set(); // Set once close() runs — stops the automatic reconnect from firing on a // connection the caller tore down on purpose (leaving the group, page // unload), which would otherwise race back in right as everything else @@ -379,6 +391,19 @@ class MeshBayTransport { if (!m) return false; return (Number(m[1]) > 1) || (Number(m[1]) === 1 && Number(m[2]) >= 1); } + /** + * Whether the node opens a sealed upload (MNP 2.0). + * + * A 1.x node reads `filename` and `data` off the message itself, finds + * neither — they are inside the seal — and answers "Missing filename or + * data", an error about the wrong thing that names no upload_id and so fails + * every upload in flight. Asked before sending rather than discovered after, + * for the same reason `supportsAppOps` is. + */ + get supportsSealedUpload() { + const m = /^(\d+)\.(\d+)$/.exec(this._nodeVersion || ''); + return !!m && Number(m[1]) >= 2; + } set onAppsEnabled(fn) { this._onAppsEnabled = fn; } set onAppDirectories(fn) { this._onAppDirectories = fn; } set onChatDirectory(fn) { this._onChatDirectory = fn; } @@ -1456,18 +1481,6 @@ class MeshBayTransport { return msg; } - async fetchStreamSegment(fileId, segmentIndex, segmentDuration) { - const msg = await this._sendAndWait({ - type: 'stream_seg', - v: '0.1', - file_id: fileId, - segment_index: segmentIndex, - segment_duration: segmentDuration || 4, - }); - if (msg.type === 'error') throw new Error(msg.detail); - return _b64decode(msg.data_b64); - } - /** * A page of chat history, newest first by default. * @@ -1583,7 +1596,7 @@ class MeshBayTransport { this._sessionKeys.skEdB64, C.chatSigningTranscript(gid, epoch, device, nonce, ct))); - return this._sendAndWait({ + const msg = await this._sendAndWait({ type: 'chat_msg', v: '2.0', format: 1, @@ -1596,6 +1609,15 @@ class MeshBayTransport { // Deliberately absent: the display name is inside the envelope now. sender_name: null, }); + // Every other request in this file refuses an `error` reply; this one + // returned it as though the node had accepted the message. It never + // mattered while a refusal reached the wrong caller anyway — now that a + // reply finds the request that made it, a message the node rejected would + // otherwise appear in the conversation as sent. It rejects more of them + // than it used to: a stale epoch, an envelope the node dislikes, or a + // device claim that is not this connection's all come back as `error`. + if (msg.type === 'error') throw new Error(msg.detail || 'chat send refused'); + return msg; } /** @@ -2080,10 +2102,21 @@ class MeshBayTransport { */ async uploadFile(file, { chunkSize, onProgress, signal, root, dir } = {}) { // The same file twice at once would confuse the node, which keys its own - // upload state by name — and would race for the same destination. - if (this._uploaders.has(file.name)) { + // upload state by name — and would race for the same destination. The guard + // is by name for that reason, even though the map below is keyed by id. + if (this._inFlightUploads.has(file.name)) { throw new Error(`${file.name} is already being uploaded`); } + if (!this._gekRaw) throw new Error('This group has no key on this device'); + if (!this.supportsSealedUpload) { + throw new Error( + 'This node is running an older MeshBay and cannot accept an upload ' + + 'from this page. Its operator has to update it.'); + } + const C = window.MeshBayCrypto; + const groupId = (this._connectArgs && this._connectArgs.groupId) || ''; + this._inFlightUploads.add(file.name); + const uploadId = _hex(crypto.getRandomValues(new Uint8Array(16))); const size = chunkSize || UPLOAD_CHUNK_SIZE; const total = Math.max(1, Math.ceil(file.size / size)); let acked = 0; @@ -2091,16 +2124,32 @@ class MeshBayTransport { let failure = null; const acks = []; - this._uploaders.set(file.name, (msg) => { - if (msg.type === 'error') { - failure = new Error(msg.detail || 'Upload refused'); - } else if (msg.stored_as) { - stored = msg; - } + const wake = () => { acked += 1; if (onProgress) onProgress(Math.min(file.size, acked * size), file.size); const waiter = acks.shift(); if (waiter) waiter(); + }; + this._uploaders.set(uploadId, (msg) => { + if (msg.type === 'error') { + failure = new Error(msg.detail || 'Upload refused'); + wake(); + return; + } + // The ack is sealed too — `stored_as` and the folder it landed in name + // the operator's content. Opening it is what makes the result usable, so + // a failure here fails the upload rather than being swallowed: a chat + // attachment that cannot learn its stored name would point at nothing. + C.openGroup(this._gekRaw, 'upload', 'file_upload_ack', groupId, msg) + .then((plain) => { + const payload = msgpack_decode(plain); + if (payload.stored_as) stored = payload; + }) + .catch((e) => { + failure = new Error( + `The node's upload reply did not open under the group key (${e.message})`); + }) + .finally(wake); }); const nextAck = () => new Promise(r => acks.push(r)); @@ -2122,15 +2171,20 @@ class MeshBayTransport { const buf = new Uint8Array( await file.slice(i * size, (i + 1) * size).arrayBuffer()); + // The name, the destination and the bytes go inside the seal together. + // Mirrors `file_upload_wire` in meshbay_common/protocol.py; only the + // fields the node routes on stay outside it. + const sealed = await C.sealGroup( + this._gekRaw, 'upload', 'file_upload', groupId, + msgpack_encode({ filename: file.name, data: buf, + dir: dir || '', root: root || '' })); this._send({ type: 'file_upload', v: '0.1', - filename: file.name, + upload_id: uploadId, chunk_index: i, total_chunks: total, - data: buf, - ...(root ? { root } : {}), - ...(dir ? { dir } : {}), + ...sealed, }); } while (acked < total) { @@ -2138,7 +2192,8 @@ class MeshBayTransport { if (failure) throw failure; } } finally { - this._uploaders.delete(file.name); + this._uploaders.delete(uploadId); + this._inFlightUploads.delete(file.name); } return stored || {}; } @@ -2458,12 +2513,15 @@ class MeshBayTransport { if (msg.type === 'index_sync') { if (this._onIndexSync) this._onIndexSync(opened); - for (const [, handler] of this._pending) { - if (handler._reqType === 'index_sync') { - handler.resolve(opened); - break; - } - } + // The node's first push to a newly connected peer is an index_sync + // nobody asked for, so there is not always a request to resolve. When + // there is, `req_id` says which one — the type match below is what a + // node too old to stamp one leaves us, and it is why two fetches in + // flight at once used to resolve the wrong one. + const handler = opened.req_id !== undefined && opened.req_id !== null + ? this._pending.get(opened.req_id) + : [...this._pending.values()].find(h => h._reqType === 'index_sync'); + if (handler) handler.resolve(opened); return; } if (this._onIndexDelta) this._onIndexDelta(opened); @@ -2550,7 +2608,13 @@ class MeshBayTransport { resolve: (msg) => { clearTimeout(timeout); this._pending.delete(id); resolve(msg); }, reject: (err) => { clearTimeout(timeout); this._pending.delete(id); reject(err); }, }); - this._send(obj); + // The id goes on the wire (MNP 1.1+): a node that understands it stamps + // the reply with it, and _dispatch matches on that alone. It used to be + // local to this map, which is why every reply had to be recognised by + // some field of its own — and why the ones that carry no such field + // reached their caller by luck. An older node ignores the extra key and + // is routed by the per-type fallbacks below, exactly as before. + this._send({ ...obj, req_id: id }); }); } @@ -2590,6 +2654,45 @@ class MeshBayTransport { } _dispatch(msg) { + // A reply that names the request it answers. Nothing below this needs to + // recognise it, and nothing below this may see it: every remaining branch + // exists to identify a reply by some field of its own, which is the job + // this makes unnecessary. + // + // What is left underneath is genuinely unsolicited — a broadcast to every + // connected client, a push, a challenge — or a reply from a node too old + // to stamp one, which is what the per-type keys are for now. + if (msg.req_id !== undefined && msg.req_id !== null) { + this._correlates = true; + // The one exception, and the only one: an index message is sealed under + // the GEK and cannot be handed to its caller until it is opened, which + // is not something this synchronous function can do. Resolving it here + // would give `fetchIndex` the envelope — nonce and ciphertext, no + // entries — and skip `_onIndexSync` entirely. `_queueIndexMessage` + // opens it and then resolves, by this same id. + const sealed = msg.type === 'index_sync' || msg.type === 'index_delta'; + if (!sealed) { + const handler = this._pending.get(msg.req_id); + if (handler) { + handler.resolve(msg); + // The acks whose *broadcast* half their own requester also needs: + // every other client learns the change from the broadcast, and the + // one that asked for it is the only one that would not, because its + // own request swallowed its copy. Same call the keyed `_ack` branch + // below makes, for the same reason. + if (BROADCAST_ACK_TYPES.has(msg.type)) _replayBroadcast(this, msg); + return; + } + // Answers a request that is no longer waiting: it gave up at its own + // timeout, or a reconnect rejected everything in flight. It belongs to + // nobody, and the whole point of this change is that it is not offered + // to somebody else instead. + console.warn('[MeshBay] late reply to req', msg.req_id, '(', msg.type, + ') — nothing waiting'); + return; + } + } + // Two-step admin-op flow (_authorizeAdminOp, ADMIN_OP_TYPES) — resolve // by (op) key before anything below gets a chance to steal it via the // generic "oldest pending" fallback further down. Returns as soon as a @@ -2647,21 +2750,21 @@ class MeshBayTransport { // While an upload is in flight the acks are its own, and there are many of // them: they must not be handed to whatever request happens to be oldest in // the pending map. - if (msg.type === 'file_upload_ack' && this._uploaders.has(msg.filename)) { - this._uploaders.get(msg.filename)(msg); + if (msg.type === 'file_upload_ack' && this._uploaders.has(msg.upload_id)) { + this._uploaders.get(msg.upload_id)(msg); return; } - // An upload refusal names the file it is about, so only that upload fails. - // It did not use to, and there was no way to tell whose error it was, so - // every upload in flight was failed together — send a second file whose - // name the node dislikes and both died. The broadcast is kept for a node - // that does not name it, where guessing wrong is worse than stopping. + // An upload refusal names the upload it is about, so only that upload + // fails. It did not use to, and there was no way to tell whose error it + // was, so every upload in flight was failed together — send a second file + // whose name the node dislikes and both died. The broadcast is kept for a + // refusal that names none, where guessing wrong is worse than stopping. if (msg.type === 'error' && this._uploaders.size) { - if (msg.filename && this._uploaders.has(msg.filename)) { - this._uploaders.get(msg.filename)(msg); + if (msg.upload_id && this._uploaders.has(msg.upload_id)) { + this._uploaders.get(msg.upload_id)(msg); return; } - if (!msg.filename) { + if (!msg.upload_id) { for (const handler of [...this._uploaders.values()]) handler(msg); return; } @@ -2955,10 +3058,10 @@ class MeshBayTransport { return; } - // device_hello_ack ends in `_ack` but is not an admin op, so the branch - // above looks it up under `admin:device_hello`, finds nothing, and drops it - // through to the arrival-order guess. Routed by request type instead: a - // request type deserves a key, and a reply deserves something to key it by. + // device_hello_ack ends in `_ack` but is not an admin op, so the admin + // branch looks it up under `admin:device_hello` and finds nothing. A 2.0 + // node stamps `req_id` and this is never reached; it is the per-type key + // for a node that does not, alongside chat_hist_resp above. if (msg.type === 'device_hello_ack') { for (const [, handler] of this._pending) { if (handler._reqType === 'device_hello') { handler.resolve(msg); return; } @@ -2968,14 +3071,13 @@ class MeshBayTransport { } // Same shape as chat_hist_resp above, and found the same way — by driving - // the panel rather than by reading this file. `chat_keys_resp` answers a - // `chat_keys_req` under a different type string, so without this it fell - // to the arrival-order guess at the end and was handed to whatever was - // oldest in `_pending`. `chat_send_probe.py` caught it on its first run: - // the Videos tab's unanswered `media_meta_req` swallowed the chat keys, - // and the send then waited out its own 30s timeout with the composer - // disabled — which is a frozen Chat tab, the exact defect that harness - // exists for, reappearing one feature later. + // the panel rather than by reading this file. Before `req_id` existed, + // `chat_keys_resp` fell to the arrival-order guess and was handed to + // whatever was oldest in `_pending`; `chat_send_probe.py` caught it on its + // first run, with the Videos tab's unanswered `media_meta_req` swallowing + // the chat keys and the send then waiting out its own 30s timeout with the + // composer disabled. `req_id` is what closes that class now, and this is + // the per-type key for a node that does not stamp one. if (msg.type === 'chat_keys_resp') { for (const [, handler] of this._pending) { if (handler._reqType === 'chat_keys_req') { handler.resolve(msg); return; } @@ -3020,10 +3122,28 @@ class MeshBayTransport { return; } - // Everything above is routed by something in the message. What is left is - // matched by arrival order, which is only ever a guess — and a wrong guess - // here hands one request's answer to another, which then waits for a reply - // that already came. Logged so that guess is visible. + // Everything above is routed by something in the message. What is left + // used to be matched by arrival order — a guess, and a wrong guess hands + // one request's answer to another, which then waits out its own 30s + // timeout for a reply that already came and went. That is how the Chat + // composer, disabled while a send is in flight, could stay disabled for + // thirty seconds on a message the node had already stored. + // + // A node that stamps its replies (`req_id`, handled at the top) has taken + // every one of its answers out of this path, so anything arriving here is + // unsolicited and the guess can only ever be wrong. Dropping it loses + // nothing and stops the theft. + if (this._correlates) { + console.warn('[MeshBay] unsolicited', msg.type, '— dropped (pending:', + this._pending.size, ')'); + return; + } + + // Only a node too old to stamp anything reaches here, where arrival order + // is still the only thing there is. Kept deliberately, and no wider than + // it was: the alternative for such a node is that half the protocol + // (device_list_result, join_result, the handshake's own replies) reaches + // nobody at all. const oldest = this._pending.entries().next(); if (!oldest.done) { const [, handler] = oldest.value; @@ -3276,11 +3396,8 @@ function _decodeMap(buf, view, offset, count) { return [obj, offset]; } -function _b64decode(b64) { - const binary = atob(b64); - const bytes = new Uint8Array(binary.length); - for (let i = 0; i < binary.length; i++) bytes[i] = binary.charCodeAt(i); - return bytes; +function _hex(bytes) { + return [...bytes].map(b => b.toString(16).padStart(2, '0')).join(''); } function _extractDtlsFingerprint(sdp) { diff --git a/packages/meshbay-hub/tests/harness/chat_send_probe.py b/packages/meshbay-hub/tests/harness/chat_send_probe.py index 1f0557b..5f99beb 100644 --- a/packages/meshbay-hub/tests/harness/chat_send_probe.py +++ b/packages/meshbay-hub/tests/harness/chat_send_probe.py @@ -20,8 +20,40 @@ never appeared, while the node had stored it all along. chat_send_probe.py -Prints JSON: `steps`, the state of the panel at each stage, and `log`, what the -transport sent and how the deliberately-unanswered request ended up. +It now drives two shapes of reply, because there were two ways for one to go +astray and only the first was ever fixed: + + `ack` the node accepts the message and answers `{"type": "ack"}`, which + names no request. Routed by request type since 2026-08-30. + `error` the node *refuses* it — every failure in `_dispatch_message` ends at + one catch-all sending `{"type": "error", "detail": "Request failed"}`, + and 238 of this module's 240 error sends name nothing either. That + reply reached no caller at all: it went to whatever request happened + to be waiting, and the send sat out its own 30s timeout with the + composer disabled. + +Both are run with an older request already pending — the condition that turns +"guess by arrival order" from usually-right into wrong — and both must come +back inside a second and a half. + +Since MNP 2.0 a send also has to **seal and sign for real** before it goes +anywhere, so this drives `chatKeys()`, `openGroup`, `sealChat` and a genuine +Ed25519 signature rather than a model of any of them. The device key is +generated in the page — `signBytes` imports a pkcs8 key and WebCrypto will not +be fooled by a stand-in — and the `chat_keys_resp` the stub answers with is +sealed **by the shipped Python**, because a payload the page built itself would +prove only that the page agrees with the page. + +That path found two defects the moment it first ran, neither visible in any +source file: `chat_keys_resp` was routed by arrival order and handed to the +older pending request (this defect, in a message type that did not exist when +the probe was written), and `_asText` had been deleted along with an unrelated +helper beside it — its only caller sits inside a promise the panel catches, so +every conversation rendered empty with nothing in the console. + +Prints JSON: `scenarios`, the state of the panel at each stage of each, and +`log`, what the transport sent and how the deliberately-unanswered request +ended up. """ import http.server import json @@ -49,9 +81,9 @@ def _page() -> str: Sealed here, by the shipped Python, rather than assembled in the browser: msgpack is private to transport.js and exported to nothing, and a payload - the test built itself would prove only that the page agrees with the page. + the page built itself would prove only that the page agrees with the page. """ - import msgpack + import msgpack # noqa: F401 (imported for the failure it gives if absent) from meshbay_common.groupbox import PURPOSE_CHAT_KEYS, seal @@ -73,9 +105,7 @@ PAGE_TEMPLATE = r"""<!doctype html><html><head><meta charset=utf-8> </div></div> <!-- The two the real page loads and the transport reaches for by global: `sealChat`/`openGroup` live in crypto.js, `signBytes` in keyderive.js. - Without them a send fails with "cannot read properties of undefined", - which is what this probe reported the first time it exercised the - encrypted path. --> + Without them a send fails with "cannot read properties of undefined". --> <script src="/crypto.js"></script> <script src="/keyderive.js"></script> <script src="/transport.js"></script> @@ -87,37 +117,16 @@ const log = []; window.addEventListener('error', e => log.push('error: ' + e.message)); window.addEventListener('unhandledrejection', e => log.push('rejected: ' + (e.reason && e.reason.message || e.reason))); -const _warn = console.warn, _err = console.error; -console.warn = (...a) => { log.push('warn: ' + a.join(' ')); _warn(...a); }; -console.error = (...a) => { log.push('console error: ' + a.join(' ')); _err(...a); }; const hex = (s) => Uint8Array.from(s.match(/../g) || [], b => parseInt(b, 16)); +const b64 = (buf) => btoa(String.fromCharCode(...new Uint8Array(buf))); -// The real transport, with only the channel replaced: _send takes the plain -// object _sendAndWait built, so the framing and msgpack are the only things -// skipped — every pending entry, key and dispatch path below is the shipped one. -const tp = new window.MeshBayTransport('', 'token'); -tp._connected = true; -tp._channel = { readyState: 'open', send() {} }; - -// Chat is encrypted (MNP 2.0), so a send that is going to come back has to -// seal and sign for real. The device key is generated here rather than stubbed -// — `signBytes` imports a pkcs8 key and WebCrypto will not be fooled — and the -// group key and epoch keys come from Python, which sealed the `chat_keys_resp` -// below exactly as the node does. So this exercises `chatKeys()`, `openGroup`, -// `sealChat` and the real signature, not a model of any of them. -tp._groupId = '__GROUP_ID__'; -tp._gekRaw = hex('__GEK_HEX__'); -tp.chatEpoch = 1; - +// One device key for both scenarios. Real, not stubbed: `signBytes` imports a +// pkcs8 key, so nothing else gets a signature past `verifyChatSignature`. const kp = await crypto.subtle.generateKey({ name: 'Ed25519' }, true, ['sign', 'verify']); -const b64 = (buf) => btoa(String.fromCharCode(...new Uint8Array(buf))); -tp._sessionKeys = { - skEdB64: b64(await crypto.subtle.exportKey('pkcs8', kp.privateKey)), -}; -// What `device_hello` sets on a live connection. -tp.devicePk = b64(await crypto.subtle.exportKey('raw', kp.publicKey)); +const SK_ED_B64 = b64(await crypto.subtle.exportKey('pkcs8', kp.privateKey)); +const DEVICE_PK_B64 = b64(await crypto.subtle.exportKey('raw', kp.publicKey)); const now = Date.now() / 1000; const history = []; @@ -126,75 +135,72 @@ for (let i = 0; i < 5; i++) { payload: 'message ' + i, timestamp: now - (5 - i) * 60 }); } -// Stands in for the node, answering exactly what webrtc_server.py answers. -// media_meta_req is answered with nothing at all, which is what a refusal -// amounts to for the request that asked: `_do_media_meta_request` sends a bare -// `error` for a file_id the index does not have, and a bare error names no -// request, so it reaches none. -tp._send = (obj) => { - log.push('sent ' + obj.type); - if (obj.type === 'chat_hist') { - setTimeout(() => { - log.push('answering chat_hist'); - tp._dispatch({ type: 'chat_hist_resp', v: '0.2', messages: history, - has_more: false }); - }, 10); - } else if (obj.type === 'chat_keys_req') { - // Sealed under the group key, as `_do_chat_keys_req` sends it. - setTimeout(() => tp._dispatch({ - type: 'chat_keys_resp', v: '2.0', group_id: tp._groupId, - nonce: hex('__KEYS_NONCE_HEX__'), ct: hex('__KEYS_CT_HEX__'), - }), 10); - } else if (obj.type === 'chat_msg') { - // Recorded so the test can assert the message really was sealed and - // signed rather than sent in clear past a composer that let it through. - log.push('chat_msg format=' + obj.format + ' epoch=' + obj.epoch - + ' ct=' + (obj.ct ? obj.ct.length : 0) - + ' sig=' + (obj.sig ? obj.sig.length : 0) - + ' plaintextLeak=' + JSON.stringify(obj).includes('hello')); - setTimeout(() => tp._dispatch({ type: 'ack', v: '2.0' }), 10); - } -}; +const wait = ms => new Promise(r => setTimeout(r, ms)); -// The panel swallows a send failure into `setInput(text)`, which is right for -// a person and useless for a probe: the symptom is the message not appearing, -// with no reason anywhere. Surfaced here so a failure names itself. -const _sendChat = tp.sendChat.bind(tp); -tp.sendChat = (...a) => _sendChat(...a).catch((e) => { - log.push('sendChat failed: ' + (e && e.message || e)); - throw e; -}); +// Stands in for the node, answering what webrtc_server.py answers — including +// stamping the reply with the id of the request it is answering, which is what +// `_send` does there. `chatReply` is the only difference between the two runs. +function makeTransport(name, chatReply) { + // The real transport, with only the channel replaced: _send takes the plain + // object _sendAndWait built, so the framing and msgpack are the only things + // skipped — every pending entry, key and dispatch path below is the shipped + // one. + const tp = new window.MeshBayTransport('', 'token'); + tp._connected = true; + tp._channel = { readyState: 'open', send() {} }; + // What a completed MNP 2.0 handshake leaves behind: the group and its key + // from `connect`, the current epoch from the sealed ack, and the device this + // connection identified itself as with `device_hello`. + tp._groupId = '__GROUP_ID__'; + tp._gekRaw = hex('__GEK_HEX__'); + tp.chatEpoch = 1; + tp._sessionKeys = { skEdB64: SK_ED_B64 }; + tp.devicePk = DEVICE_PK_B64; + tp._send = (obj) => { + log.push('sent ' + obj.type); + const answer = (reply) => setTimeout( + () => tp._dispatch({ ...reply, req_id: obj.req_id }), 10); + if (obj.type === 'chat_hist') { + answer({ type: 'chat_hist_resp', v: '0.2', messages: history, has_more: false }); + } else if (obj.type === 'chat_keys_req') { + // Sealed under the group key, as `_do_chat_keys_req` sends it. + answer({ type: 'chat_keys_resp', v: '2.0', group_id: tp._groupId, + nonce: hex('__KEYS_NONCE_HEX__'), ct: hex('__KEYS_CT_HEX__') }); + } else if (obj.type === 'chat_msg') { + // Recorded so the test can assert what actually left the browser, rather + // than trusting that a composer which accepted the text sealed it. + log.push(name + ': chat_msg format=' + obj.format + ' epoch=' + obj.epoch + + ' ct=' + (obj.ct ? obj.ct.length : 0) + + ' sig=' + (obj.sig ? obj.sig.length : 0) + + ' plaintextLeak=' + JSON.stringify(obj).includes('hello')); + answer(chatReply); + } + // music_meta_req is answered by nothing at all, on purpose: the node holds + // one open for as long as the third-party lookup behind it takes, which + // was measured at over 100 seconds with that service failing. It is the + // older pending request every scenario here needs. + }; + // The panel swallows a send failure into `setInput(text)`, which is right for + // a person and useless for a probe: the symptom is a message not appearing, + // with no reason anywhere. Surfaced here so a failure names itself — the + // `error` scenario is *expected* to reach this. + const _sendChat = tp.sendChat.bind(tp); + tp.sendChat = (...a) => _sendChat(...a).catch((e) => { + log.push(name + ': sendChat failed: ' + (e && e.message || e)); + throw e; + }); + return tp; +} -function Host() { +function Host({ tp }) { const transportRef = useRef(tp); const gekRef = useRef(null); return html`<${ChatPanel} transportRef=${transportRef} gekRef=${gekRef} username="me" userId="user-me" entries=${[]} status="connected" />`; } -render(html`<${Host} />`, document.getElementById('root')); -const out = { steps: [], log }; -const composer = () => document.querySelector('.chat-input'); - -function snap(label) { - const c = composer(); - out.steps.push({ - label, - bubbles: document.querySelectorAll('.chat-bubble').length, - msgs: document.querySelectorAll('.chat-msg').length, - lastText: [...document.querySelectorAll('.chat-text')].pop()?.textContent ?? null, - // What a frozen tab actually is: the composer is disabled for as long as - // a send is in flight. - composerDisabled: c ? c.disabled : null, - composerValue: c ? c.value : null, - pending: tp._pending.size, - }); -} - -const wait = ms => new Promise(r => setTimeout(r, ms)); - -function typeInto(text) { - const c = composer(); +function typeInto(root, text) { + const c = root.querySelector('.chat-input'); c.focus(); // Preact reads e.target.value on input, so the native setter has to run. Object.getOwnPropertyDescriptor(window.HTMLTextAreaElement.prototype, 'value') @@ -202,22 +208,42 @@ function typeInto(text) { c.dispatchEvent(new Event('input', { bubbles: true })); } -(async () => { +async function runScenario(name, chatReply) { + const root = document.createElement('div'); + document.getElementById('root').appendChild(root); + const tp = makeTransport(name, chatReply); + render(html`<${Host} tp=${tp} />`, root); + + const steps = []; + const snap = (label) => { + const c = root.querySelector('.chat-input'); + steps.push({ + label, + bubbles: root.querySelectorAll('.chat-bubble').length, + lastText: [...root.querySelectorAll('.chat-text')].pop()?.textContent ?? null, + // What a frozen tab actually is: the composer is disabled for as long as + // a send is in flight. + composerDisabled: c ? c.disabled : null, + composerValue: c ? c.value : null, + pending: tp._pending.size, + }); + }; + await wait(500); snap('arrived'); - // The Videos tab asked about a file a moment ago and is still waiting. Any - // unanswered request will do; this is the one that was live when the defect - // was found. - tp.fetchMediaMeta('a-file-the-node-refused') - .then(m => log.push('media_meta resolved with ' + m.type), - e => log.push('media_meta rejected: ' + e.message)); + // The Music tab asked about a track and is still waiting on the node, which + // is waiting on something else. Any older unanswered request will do; this + // is the one that was live when the defect was found. + tp.fetchMusicMeta('a-track-the-node-is-slow-about') + .then(m => log.push(name + ': music_meta resolved with ' + m.type), + e => log.push(name + ': music_meta rejected: ' + e.message)); await wait(100); - snap('stale request pending'); + snap('older request pending'); - typeInto('hello'); + typeInto(root, 'hello'); await wait(100); - composer().dispatchEvent(new KeyboardEvent('keydown', + root.querySelector('.chat-input').dispatchEvent(new KeyboardEvent('keydown', { key: 'Enter', bubbles: true, cancelable: true })); // Far short of _sendAndWait's 30s timeout: a send that has not come back by @@ -225,6 +251,14 @@ function typeInto(text) { await wait(1500); snap('after send'); + return { name, steps }; +} + +(async () => { + const out = { scenarios: [], log }; + out.scenarios.push(await runScenario('ack', { type: 'ack', v: '0.14' })); + out.scenarios.push(await runScenario( + 'error', { type: 'error', detail: 'Request failed' })); fetch('/log', { method: 'POST', body: JSON.stringify(out) }); })(); </script></body></html>""" diff --git a/packages/meshbay-hub/tests/harness/upload_seal_probe.mjs b/packages/meshbay-hub/tests/harness/upload_seal_probe.mjs new file mode 100644 index 0000000..0b77e42 --- /dev/null +++ b/packages/meshbay-hub/tests/harness/upload_seal_probe.mjs @@ -0,0 +1,102 @@ +/** + * Does the browser's real `uploadFile` produce frames a real node can open — + * and does it read back what that node actually answered? + * + * Drives **the real `MeshBayTransport` over the real `crypto.js`**. Only the DOM + * and the DataChannel are stand-ins; the msgpack encode, the HKDF, the AES-GCM + * and the whole upload loop are the shipped code. + * + * It exists because a source-reading test cannot show either half of MNP 2.0's + * upload. `test_transport_contracts` can see that `sealGroup` is called; only + * this can show that what comes out opens under `meshbay_common.groupbox` — and + * that the caller of `uploadFile` is told the name the *node* chose, which now + * arrives sealed and would otherwise be `undefined` with nothing to notice it. + * + * node upload_seal_probe.mjs <static-dir> <input.json> + * + * Two modes, because the node half runs in Python between them: + * "send" — run the upload, print every frame it emits, answer nothing + * "receive" — run it again, answering with acks Python built. Their + * `upload_id` is retargeted to this run's; it is outside the + * seal, so the ciphertext stays exactly the one Python produced. + */ +import fs from 'fs'; + +const STATIC = process.argv[2]; +const input = JSON.parse(fs.readFileSync(process.argv[3], 'utf8')); + +for (const level of ['log', 'warn', 'error', 'info', 'debug']) { + console[level] = (...args) => process.stderr.write(args.join(' ') + '\n'); +} + +globalThis.window = globalThis; +globalThis.addEventListener = () => {}; +globalThis.removeEventListener = () => {}; +globalThis.location = { hash: '' }; +globalThis.document = { + addEventListener() {}, removeEventListener() {}, visibilityState: 'visible', +}; + +new Function(fs.readFileSync(`${STATIC}/crypto.js`, 'utf8'))(); +const transportSrc = fs.readFileSync(`${STATIC}/transport.js`, 'utf8'); +new Function(transportSrc)(); +// The msgpack codec is private to transport.js — pulled out the same way the +// groupbox parity harness pulls out sealGroup, so this probe encodes and +// decodes with the codec that is actually shipped rather than a second one. +const { msgpack_encode, msgpack_decode } = + new Function(transportSrc + '\nreturn { msgpack_encode, msgpack_decode };')(); + +const hex = (s) => Uint8Array.from(s.match(/../g).map((b) => parseInt(b, 16))); +const toHex = (u8) => + Array.from(u8).map((b) => b.toString(16).padStart(2, '0')).join(''); + +// Just enough of a File: a name, a size, and slices that yield ArrayBuffers. +const bytes = hex(input.file.data); +const file = { + name: input.file.name, + size: bytes.length, + slice(a, b) { + const part = bytes.slice(a, b); + return { arrayBuffer: async () => part.buffer.slice( + part.byteOffset, part.byteOffset + part.byteLength) }; + }, +}; + +const tp = new window.MeshBayTransport('', 'token'); +tp._connected = true; +tp._channel = { readyState: 'open', bufferedAmount: 0, send() {}, close() {} }; +tp._pc = { close() {} }; +tp._gekRaw = hex(input.gek); +tp._connectArgs = { groupId: input.group_id }; +tp._nodeVersion = input.node_version; + +const frames = []; +let uploadId = null; +tp._send = (msg) => { + frames.push(toHex(msgpack_encode(msg))); + if (msg.upload_id) uploadId = msg.upload_id; + if (input.mode !== 'receive') return; + // Answer as the node did, on the next turn of the loop so the send path + // finishes first — which is also how a real ack arrives. + const ack = msgpack_decode(hex(input.acks[msg.chunk_index])); + ack.upload_id = uploadId; + // Through the real `_dispatch`, so the routing under test — matching an + // ack to its uploader by `upload_id` — is the shipped one. + setImmediate(() => tp._dispatch(ack)); +}; + +const out = { frames, mode: input.mode }; +const done = tp.uploadFile(file, { chunkSize: input.chunk_size, + dir: input.dir, root: input.root }); + +if (input.mode === 'receive') { + done.then((stored) => { out.state = 'resolved'; out.stored = stored; }) + .catch((e) => { out.state = 'rejected'; out.message = e.message; }) + .finally(() => { out.upload_id = uploadId; + process.stdout.write(JSON.stringify(out)); }); +} else { + // Nothing will answer, so let the send loop run itself out and report. + done.catch((e) => { out.state = 'rejected'; out.message = e.message; }); + setTimeout(() => { out.upload_id = uploadId; + process.stdout.write(JSON.stringify(out)); }, 250); +} diff --git a/packages/meshbay-hub/tests/test_chat_send.py b/packages/meshbay-hub/tests/test_chat_send.py index 250c9a3..5db8f26 100644 --- a/packages/meshbay-hub/tests/test_chat_send.py +++ b/packages/meshbay-hub/tests/test_chat_send.py @@ -1,38 +1,29 @@ """ -Sending a chat message must come back — and must go out encrypted. +Sending a chat message must come back — accepted or refused. -The node answers a chat message with a bare `{"type": "ack"}` — no request id, -no type of its own — so `_dispatch` had nothing to match it on and left it to -the arrival-order guess at the end of the function. That guess is wrong as soon -as anything else this browser asked for is still waiting: the ack was handed to -*that* request, and the send waited out `_sendAndWait`'s 30s timeout. Since the -composer is disabled while a send is in flight, the Chat tab stopped taking -clicks and keys, the message never appeared — and it was there on the next -visit, because the node had stored it and answered. +The node's replies to a chat message name no request. The acceptance is a bare +`{"type": "ack"}`; the refusal is a bare `{"type": "error"}`, and it is not a +special case — `_dispatch_message`'s catch-all answers *every* failure that +way, and 238 of webrtc_server.py's 240 error sends name nothing either. So +`_dispatch` had nothing to match either reply on and left both to the +arrival-order guess at the end of the function. -An outstanding request is the ordinary case, not a rare one: the node refuses -an unknown file_id with a bare `error`, which names no request either, so a -Videos tab that asked about a file the index no longer has leaves a -`media_meta_req` in `_pending` for a full 30s. +That guess is wrong as soon as anything else this browser asked for is still +waiting, which is the ordinary case rather than a rare one: a `music_meta_req` +sits in `_pending` for as long as the third-party lookup behind it takes, and +that was measured live at over 100 seconds with the service failing. The reply +went to *that* request, and the send waited out `_sendAndWait`'s 30s timeout. +Since the composer is disabled while a send is in flight, the Chat tab stopped +taking clicks and keys, and the message never appeared. -None of that is visible in `chat-app.js`, where every line is correct, so this +The ack half was fixed by matching on request type. The refusal half could not +be: an `error` has no type of its own to match on. `req_id` is what closed it — +the caller's id, stamped on the reply by the node — so this now drives both +shapes of answer. + +None of it is visible in `chat-app.js`, where every line is correct, so this drives the real panel over the real transport in a browser rather than reading either source. - -Extended for MNP 2.0, where a send seals and signs before it goes anywhere. -That turned out to matter twice on its first run: - - * `chat_keys_resp` answers a `chat_keys_req` under a different type string, - so it fell through to the arrival-order guess and was handed to the very - `media_meta_req` this probe leaves outstanding — the original defect, one - feature later, in a message type that did not exist when it was written. - * `_asText` had been deleted along with an unrelated helper beside it. Its - only caller is inside `_openChatMessage`, whose rejection the panel - swallows, so the whole conversation rendered empty with nothing in the - console and the node answering perfectly. - -Neither is visible in any source file, and neither would have been caught by a -test that reads one. """ import json import shutil @@ -52,7 +43,7 @@ pytestmark = pytest.mark.skipif( @pytest.fixture(scope="module") def probe(): - # `sys.executable`, not a bare "python3": the harness now imports + # `sys.executable`, not a bare "python3": the harness imports # `meshbay_common` to seal the chat keys the way the node does, and the # system interpreter has neither that nor msgpack. The other probes get # away with "python3" because they import nothing from this project. @@ -60,51 +51,81 @@ def probe(): capture_output=True, timeout=180) assert run.returncode == 0, run.stderr.decode()[-2000:] data = json.loads(run.stdout.decode()) - return data, {s["label"]: s for s in data["steps"]} + steps = {sc["name"]: {s["label"]: s for s in sc["steps"]} + for sc in data["scenarios"]} + return data, steps + +@pytest.mark.parametrize("reply", ["ack", "error"]) +def test_the_composer_comes_back(probe, reply): + """The one thing a person sees: the tab is usable again. -def test_the_composer_comes_back(probe): - """The one thing a person sees: the tab is usable again.""" + Both answers have to release it. A refusal that reaches nobody leaves the + composer disabled exactly as long as an acceptance that reaches nobody — + the composer is not waiting for good news, it is waiting for an answer. + """ _, steps = probe - assert steps["stale request pending"]["composerDisabled"] is False, ( + assert steps[reply]["older request pending"]["composerDisabled"] is False, ( "the composer was already unusable before the send") - assert steps["after send"]["composerDisabled"] is False, ( + assert steps[reply]["after send"]["composerDisabled"] is False, ( "the composer is still disabled well inside the 30s request timeout -- " "the send never came back, which is what reads as a frozen Chat tab") -def test_the_message_is_displayed(probe): +def test_an_accepted_message_is_displayed(probe): """A sent message appears at once, not on the next visit to the tab.""" _, steps = probe - before = steps["stale request pending"]["bubbles"] - assert steps["after send"]["bubbles"] == before + 1, ( + before = steps["ack"]["older request pending"]["bubbles"] + assert steps["ack"]["after send"]["bubbles"] == before + 1, ( "the message was not added to the conversation") - assert steps["after send"]["lastText"] == "hello" - assert steps["after send"]["composerValue"] == "", ( + assert steps["ack"]["after send"]["lastText"] == "hello" + assert steps["ack"]["after send"]["composerValue"] == "", ( "the text came back into the composer, so the send was treated as failed") -def test_the_ack_is_not_handed_to_another_request(probe): - """The other half of the same defect: whatever was waiting got the ack and - carried on with a reply to a question it never asked.""" +def test_a_refused_message_is_not_displayed_as_sent(probe): + """The other direction, and the one routing this correctly makes possible. + + While a refusal reached the wrong caller it did not matter what `sendChat` + would have done with it. Now that it arrives, a message the node rejected + must not appear in the conversation as though it had been stored — it must + come back into the composer, where a person can see it did not go. + """ + _, steps = probe + before = steps["error"]["older request pending"]["bubbles"] + assert steps["error"]["after send"]["bubbles"] == before, ( + "a refused message was added to the conversation anyway") + assert steps["error"]["after send"]["composerValue"] == "hello", ( + "the refused text was dropped instead of being handed back") + + +@pytest.mark.parametrize("reply", ["ack", "error"]) +def test_the_reply_is_not_handed_to_another_request(probe, reply): + """The other half of the same defect: whatever was waiting got the reply + and carried on with an answer to a question it never asked.""" data, _ = probe - assert "media_meta resolved with ack" not in data["log"], ( - "the chat ack was routed to the pending media_meta_req -- that request " - "now believes it has an answer, and the chat send is waiting for a " - "reply that already arrived") + stolen = [line for line in data["log"] if line.startswith(f"{reply}: music_meta")] + assert not stolen, ( + f"the chat {reply} was routed to the pending music_meta_req ({stolen}) -- " + "that request now believes it has an answer, and the chat send is " + "waiting for a reply that already arrived") -def test_the_message_goes_out_sealed_and_signed(probe): +@pytest.mark.parametrize("reply", ["ack", "error"]) +def test_the_message_goes_out_sealed_and_signed(probe, reply): """ - What actually left the browser. A composer that let a plaintext message - through would be refused by the node, but the refusal arrives after the - fact and reads as "the message did not send" — so assert the shape here, - where the reason is visible. + What actually left the browser (MNP 2.0). + + Asserted for both scenarios because the composer is what decides to send: + a client that fell back to plaintext when something went wrong would be + refused by the node, but the refusal arrives after the fact and reads as + "the message did not send". There is no plaintext form on the wire. """ data, _ = probe - sent = [line for line in data["log"] if line.startswith("chat_msg ")] - assert sent, ("no chat_msg reached the stand-in node — the send did not " - f"complete. log: {data['log']}") + sent = [line for line in data["log"] + if line.startswith(f"{reply}: chat_msg ")] + assert sent, (f"no chat_msg reached the stand-in node in the {reply} " + f"scenario — the send did not complete. log: {data['log']}") assert "format=1" in sent[0], "the message was not sealed" assert "sig=64" in sent[0], "the message was not signed" assert "ct=" in sent[0] and "ct=0" not in sent[0], "there was no ciphertext" @@ -115,12 +136,25 @@ def test_the_message_goes_out_sealed_and_signed(probe): def test_the_chat_keys_answer_is_not_handed_to_another_request(probe): """ - The original defect's shape, in the message type that carries the group's - chat keys. An unanswered request is the ordinary case, not a rare one, and - the one this probe leaves outstanding swallowed the keys on the first run. + The same defect this file exists for, in the message type that carries the + group's chat keys — which did not exist when it was written, and which a + send now depends on. It went astray on the probe's first encrypted run. """ data, _ = probe - assert not any("media_meta resolved with chat_keys_resp" in line - for line in data["log"]), ( - "chat_keys_resp was routed by arrival order and handed to the stale " - "media_meta_req — the send then waits out its own 30s timeout") + stolen = [line for line in data["log"] if "music_meta resolved with" in line] + assert not stolen, ( + f"a reply was routed to the pending music_meta_req ({stolen}) — the " + "send then waits out its own 30s timeout with the composer disabled") + + +def test_history_still_renders(probe): + """ + Not about sending at all, and here because it broke without a sound: + `_asText` was deleted with an unrelated helper beside it, its only caller + sits inside a promise the panel catches, and every conversation rendered + empty with the node answering perfectly. + """ + _, steps = probe + assert steps["ack"]["older request pending"]["bubbles"] == 5, ( + "the five history messages did not render — the panel swallows a " + "failure in the transport's message reader, so this is silent") diff --git a/packages/meshbay-hub/tests/test_index_seal_client.py b/packages/meshbay-hub/tests/test_index_seal_client.py index ca2c7a2..e1135da 100644 --- a/packages/meshbay-hub/tests/test_index_seal_client.py +++ b/packages/meshbay-hub/tests/test_index_seal_client.py @@ -45,10 +45,15 @@ def _frame(msg: dict) -> str: return (struct.pack(">I", len(body)) + body).hex() -def _sync_frame(gek: bytes, *names: str, version: int = 3) -> str: +def _sync_frame(gek: bytes, *names: str, version: int = 3, + req_id: int | None = None) -> str: payload = {"version": version, "entries": [_entry(n) for n in names], "dirs": ["library"], "roots": [{"name": "library"}]} + # `req_id` is what a current node stamps on a *reply*; the push it sends a + # newly connected peer answers no request and carries none. Both shapes + # arrive here, and only one of them may resolve a waiting fetchIndex. return _frame({"type": "index_sync", "v": "1.0", "group_id": GROUP, + **({"req_id": req_id} if req_id is not None else {}), **seal(gek, PURPOSE_INDEX, "index_sync", GROUP, payload)}) @@ -141,3 +146,26 @@ def test_deltas_are_applied_in_arrival_order(): assert [e["additions"][0] for e in out["events"][1:]] == [ "added-0.mkv", "added-1.mkv", "added-2.mkv"] assert [e["base_version"] for e in out["events"][1:]] == [3, 4, 5] + + +def test_a_sealed_reply_is_opened_before_it_reaches_its_caller(): + """A stamped index_sync must not be short-circuited by its `req_id`. + + Every other reply a node stamps is resolved straight out of the pending + map, which is the whole point of the id. An index message cannot be: it is + sealed, opening it is asynchronous, and `_dispatch` is not. Handing it over + on the strength of the id alone gives `fetchIndex` the envelope — nonce and + ciphertext, no entries — and never calls `onIndexSync` at all. + + `req_id` is 0 here because it is the transport's first request, and a + falsy id is exactly the one a presence check gets wrong. + """ + out = _run([_sync_frame(GEK, "a-film.mkv", req_id=0)]) + + assert [e["event"] for e in out["events"]] == ["index_sync"], ( + "the consumer was never told about an index that arrived as a reply") + assert out["events"][0]["entries"] == ["a-film.mkv"] + assert out["events"][0]["hasCiphertext"] is False + assert out["fetchIndex"]["state"] == "resolved" + assert out["fetchIndex"]["entries"] == ["a-film.mkv"], ( + "the caller was handed the sealed envelope instead of the index") diff --git a/packages/meshbay-hub/tests/test_transport_contracts.py b/packages/meshbay-hub/tests/test_transport_contracts.py index b462242..879062b 100644 --- a/packages/meshbay-hub/tests/test_transport_contracts.py +++ b/packages/meshbay-hub/tests/test_transport_contracts.py @@ -307,26 +307,60 @@ def test_no_setter_survives_the_state_it_belonged_to(): # ── Parallel uploads ────────────────────────────────────────────────────────── -def test_an_upload_refusal_names_the_file_it_is_about(transport): +def test_an_upload_refusal_names_the_upload_it_is_about(transport): """Reported 2026-08-16: a second upload started in parallel killed both. - An error used to carry no filename, so the client could not tell whose it - was and failed every upload in flight — one name the node disliked took the - other file with it. The node names the file now, and only that upload stops. + An error used to carry nothing identifying, so the client could not tell + whose it was and failed every upload in flight — one name the node disliked + took the other file with it. + + The node named the *file* until MNP 2.0 and names the `upload_id` now: the + filename moved inside the seal, and echoing it in clear so the two sides + could match on it would give back precisely what sealing the upload is for. + The property is unchanged — one refusal, one failed upload. """ body = transport[transport.index("if (msg.type === 'error' && this._uploaders.size)"):] body = body[:body.index("\n if (msg.type === 'chat_msg'")] - assert "this._uploaders.has(msg.filename)" in body, ( + assert "this._uploaders.has(msg.upload_id)" in body, ( "a named refusal must reach one uploader, not all of them") - assert "if (!msg.filename)" in body, ( + assert "if (!msg.upload_id)" in body, ( "an unnamed error from an older node must still stop everything — " "guessing which upload it belongs to would be worse") -def test_uploads_are_tracked_per_file(transport): +def test_uploads_are_tracked_per_upload(transport): """Acks interleave when two files are in flight.""" assert "this._uploaders = new Map()" in transport - assert "this._uploaders.set(file.name" in transport + assert "this._uploaders.set(uploadId" in transport + # And the "already being uploaded" guard still speaks in filenames, because + # that is what the caller passed and what it would recognise in the error. + assert "this._inFlightUploads.has(file.name)" in transport + + +def test_the_upload_itself_is_sealed(transport): + """ + MNP 2.0. The filename, the destination and the bytes go inside the seal + together — sealing the content and announcing the name beside it would be + theatre — and only what the node routes on stays outside. + """ + start = transport.index(" async uploadFile(file,") + body = transport[start:transport.index("\n /** Create a directory", start)] + assert "sealGroup(" in body and "'file_upload'" in body, ( + "the upload must be sealed under the group key") + assert "openGroup(" in body and "'file_upload_ack'" in body, ( + "the ack carries the stored name and must be opened, not read") + # The message the node actually receives: everything between `this._send({` + # and its close. Read on its own, because the same field names appear a few + # lines above inside `msgpack_encode({...})`, which is the sealed half. + sent = body[body.index("this._send({"):] + sent = sent[:sent.index("});")] + assert "filename" not in sent, "the filename is on the message in clear" + assert "data" not in sent, "the bytes are on the message in clear" + assert "dir" not in sent and "root" not in sent, ( + "the destination is on the message in clear") + assert "...sealed," in sent, "the message must carry the sealed pair" + assert "supportsSealedUpload" in body, ( + "an older node must be refused before a chunk is sent, not after") # ── MNP 1.0: the sealed handshake ack ──────────────────────────────────────── diff --git a/packages/meshbay-hub/tests/test_upload_seal_client.py b/packages/meshbay-hub/tests/test_upload_seal_client.py new file mode 100644 index 0000000..d6f9156 --- /dev/null +++ b/packages/meshbay-hub/tests/test_upload_seal_client.py @@ -0,0 +1,169 @@ +""" +The browser half of MNP 2.0's sealed upload, measured rather than read. + +`test_upload_sealed.py` (node side) proves the node opens what the shared +encoder produces and refuses everything else. This proves the *shipped browser +code* produces it — and, the part that matters more, that a caller of +`uploadFile` is still told the name the node stored the file under, which now +arrives sealed and would otherwise be `undefined` with nothing on screen to say +so: a chat attachment would point at a file that is not there. + +Driven through `harness/upload_seal_probe.mjs`, which runs the shipped +`transport.js` over the shipped `crypto.js`. The node half in between is the +real `_do_file_upload`, writing to a real directory. + +A source-reading test can see that `sealGroup` is called. Only this can see +whether what comes out of it opens. +""" + +import json +import shutil +import subprocess +import tempfile +from pathlib import Path + +import msgpack +import pytest +from cryptography.hazmat.primitives.asymmetric.ed25519 import Ed25519PrivateKey +from meshbay_common.crypto import generate_gek +from meshbay_common.protocol import MNP +from meshbay_node.indexer.group_index import GroupIndex +from meshbay_node.roots import RootSet +from meshbay_node.transport.webrtc_server import WebRTCPeerSession + +STATIC = Path(__file__).resolve().parents[1] / "src" / "meshbay_hub" / "static" +PROBE = Path(__file__).resolve().parent / "harness" / "upload_seal_probe.mjs" + +pytestmark = pytest.mark.skipif( + shutil.which("node") is None or not PROBE.exists(), + reason="node unavailable — the client half cannot be measured", +) + +GROUP = "g-upload-probe" +CHUNK = 32 +BODY = bytes(range(256)) * 3 # 768 bytes → 24 chunks of 32 + + +def _run_probe(payload: dict) -> dict: + with tempfile.TemporaryDirectory() as d: + f = Path(d) / "input.json" + f.write_text(json.dumps(payload)) + proc = subprocess.run( + ["node", str(PROBE), str(STATIC), str(f)], + capture_output=True, text=True, timeout=60, + ) + if proc.returncode != 0 or not proc.stdout: + pytest.fail(f"upload probe failed:\n{proc.stderr}") + return json.loads(proc.stdout) + + +def _node_session(tmp_path: Path, gek: bytes) -> WebRTCPeerSession: + root = tmp_path / "library" + root.mkdir() + index = GroupIndex(group_id=GROUP, sk_node=Ed25519PrivateKey.generate()) + session = WebRTCPeerSession.__new__(WebRTCPeerSession) + session._ctx = { + "roots": RootSet.build([{"path": str(root), "name": "library", + "writable": True}]), + "index": index, "sk_node": index.sk_node, "gek": gek, + } + session._group_id = GROUP + session._user_id = "prober" + session._pk_user = "" + session._uploads = {} + session.sent = [] + session._send = session.sent.append + session._audit = lambda *a, **k: None + return session + + +def _probe_input(gek: bytes, mode: str, **extra) -> dict: + return { + "mode": mode, "gek": gek.hex(), "group_id": GROUP, + "node_version": "2.0", "chunk_size": CHUNK, "dir": "library", + "root": "library", + "file": {"name": "holiday.jpg", "data": BODY.hex()}, + **extra, + } + + +@pytest.fixture(scope="module") +def _gek(): + return generate_gek() + + +@pytest.fixture(scope="module") +def _sent(_gek): + """Every frame the shipped `uploadFile` puts on the wire, unanswered.""" + return _run_probe(_probe_input(_gek, "send")) + + +def test_the_browser_puts_no_filename_and_no_content_on_the_wire(_sent): + """The whole point, measured on the bytes rather than read off the source.""" + assert _sent["frames"], "the client sent nothing" + for hexframe in _sent["frames"]: + raw = bytes.fromhex(hexframe) + assert b"holiday.jpg" not in raw, "the filename is on the wire in clear" + msg = msgpack.unpackb(raw, raw=False) + assert set(msg) == {"type", "v", "upload_id", "chunk_index", + "total_chunks", "nonce", "ct"} + assert msg["type"] == MNP.FILE_UPLOAD + + +def test_a_real_node_opens_what_the_real_browser_sealed(tmp_path, _gek, _sent): + """ + End to end: the shipped browser encoder into the shipped node handler, with + the file that lands on disk as the assertion. A mismatch in the HKDF salt, + the AAD encoding or the payload shape shows up here as "did not open" — and + nowhere else until somebody tries to upload something. + """ + session = _node_session(tmp_path, _gek) + for hexframe in _sent["frames"]: + session._do_file_upload(msgpack.unpackb(bytes.fromhex(hexframe), raw=False)) + + errors = [m for m in session.sent if m.get("type") == "error"] + assert not errors, f"the node refused a frame the browser built: {errors[:1]}" + + root = session._ctx["roots"].roots[0].path + assert (root / "holiday.jpg").read_bytes() == BODY + assert not list(root.glob("*.part")), "a temp file was left behind" + + +def test_the_caller_is_told_the_name_the_node_chose(tmp_path, _gek, _sent): + """ + `stored_as` is sealed now, so reading it takes a decrypt that can fail + silently. It must not: the node finds a free name rather than replacing + anything, and a chat attachment that never learns which name points at + nothing. + + The acks below are the ones the node really produced — only their + `upload_id`, which is outside the seal, is retargeted to the second probe + run's own upload. + """ + session = _node_session(tmp_path, _gek) + # A file of that name is already there, so the node has to choose another. + (session._ctx["roots"].roots[0].path / "holiday.jpg").write_bytes(b"someone else's") + + for hexframe in _sent["frames"]: + session._do_file_upload(msgpack.unpackb(bytes.fromhex(hexframe), raw=False)) + acks = [msgpack.packb(m, use_bin_type=True).hex() + for m in session.sent if m.get("type") == MNP.FILE_UPLOAD_ACK] + assert len(acks) == len(_sent["frames"]) + + result = _run_probe(_probe_input(_gek, "receive", acks=acks)) + assert result["state"] == "resolved", result.get("message") + assert result["stored"]["stored_as"] == "holiday (2).jpg" + assert result["stored"]["dir"] == "library" + + +def test_the_client_refuses_an_older_node_before_sending_a_chunk(_gek): + """ + A 1.x node would answer "Missing filename or data" — an error about the + wrong thing, naming no upload, which fails every upload in flight. Asked + first instead, and nothing goes on the wire. + """ + result = _run_probe(_probe_input(_gek, "receive", acks=[], + node_version="1.1")) + assert result["state"] == "rejected" + assert "older MeshBay" in result["message"] + assert result["frames"] == [], "a chunk was sent to a node that cannot open it" |