diff options
| author | Christophe Besson <cbesson@gmail.com> | 2026-09-07 17:46:33 +0200 |
|---|---|---|
| committer | Christophe Besson <cbesson@gmail.com> | 2026-09-07 17:46:33 +0200 |
| commit | 8980a8e42d94ab7c0bc9739283d39f938f8402b0 (patch) | |
| tree | bbb830b48162ebfdcf443f300c82495f452ad1e0 /packages/meshbay-hub/tests | |
| parent | 77dd077491aea50e71e21e0d17555a2f91cf818b (diff) | |
| download | meshbay-8980a8e42d94ab7c0bc9739283d39f938f8402b0.tar.gz | |
feat(mnp)!: seal the upload under the group key
Downloads have been encrypted under a GEK-derived key since the beginning:
`file_chunk` and `stream_data` both go through `chunk_ciphertext`. Uploads
never were. `file_upload` carried the filename and the raw bytes in plain
msgpack, and `file_upload_ack` carried the name the node stored them under —
so the same file was ciphertext leaving a node and plaintext arriving at one.
There was no threat model behind that asymmetry.
Both halves now travel sealed under a third groupbox purpose,
HKDF(GEK, info="meshbay:upload:v1"). The filename, the destination folder and
the bytes are all inside the seal; only `upload_id` and `chunk_index` stay in
clear, because the node routes and orders on them before it can decrypt. This
direction seals *towards* the node — it holds the GEK for its own group — and
it opens the payload before it picks a destination or touches the disk.
What that forced, and why none of it is optional:
- `filename` was the correlation key on both sides. It cannot be: matching an
ack to its request by name would hand back exactly what the seal hides.
`upload_id` replaces it — client-drawn, opaque to the node, unique within a
connection, never an authorization input. The property it guarded (one
refusal fails one upload, not every upload in flight) is unchanged.
- Refusals can no longer quote what they refused. `No directory named 'X'`
becomes `No such directory in this group` plus the `code` that was already
there; the client knows what it sent.
- No plaintext fallback. A path that still accepts plaintext is not a sealed
path, so an unsealed `file_upload` is refused with `upload_not_sealed`.
Hardened while here, because what comes out of a seal is authenticated but not
validated — a member can seal anything: `filename` and `data` have their types
checked before any upload state is created, and `chunk_index`/`total_chunks`,
which are outside the seal by necessity, can no longer raise where a refusal
was meant.
Tests. `test_upload_sealed.py` pins the node half: nothing identifying on the
wire, tamper/wrong-key/wrong-group all refused with nothing written, and
multi-chunk reassembly unchanged. `test_upload_seal_client.py` drives the
shipped `uploadFile` over the shipped `crypto.js` under node and feeds its real
frames to the real `_do_file_upload` — the file lands intact, and the ack the
node actually produced comes back with the name it chose for a collision, which
is the half a source-reading test cannot see. Both upload purposes join the
JS/Python groupbox parity vectors.
BREAKING CHANGE: MNP 2.0. `file_upload`/`file_upload_ack` change shape on the
wire every deployed client speaks, which is MAJOR by the same rule 1.0 was —
but the break is confined to uploads. `MNP_MIN_SUPPORTED` stays at "1.0", so a
1.x peer still connects, browses, downloads, streams and chats; only its
uploads are refused, with a message saying which side is old. The client checks
the node's version before sending a chunk, so neither side meets this as a
timeout. This is the version negotiation shipped in 1.0 earning its keep: 1.0
cost a flag day, 2.0 costs a refusal code.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01AsoWC3GmhNdwVFomW3QjH3
Diffstat (limited to 'packages/meshbay-hub/tests')
3 files changed, 313 insertions, 8 deletions
diff --git a/packages/meshbay-hub/tests/harness/upload_seal_probe.mjs b/packages/meshbay-hub/tests/harness/upload_seal_probe.mjs new file mode 100644 index 0000000..0b77e42 --- /dev/null +++ b/packages/meshbay-hub/tests/harness/upload_seal_probe.mjs @@ -0,0 +1,102 @@ +/** + * Does the browser's real `uploadFile` produce frames a real node can open — + * and does it read back what that node actually answered? + * + * Drives **the real `MeshBayTransport` over the real `crypto.js`**. Only the DOM + * and the DataChannel are stand-ins; the msgpack encode, the HKDF, the AES-GCM + * and the whole upload loop are the shipped code. + * + * It exists because a source-reading test cannot show either half of MNP 2.0's + * upload. `test_transport_contracts` can see that `sealGroup` is called; only + * this can show that what comes out opens under `meshbay_common.groupbox` — and + * that the caller of `uploadFile` is told the name the *node* chose, which now + * arrives sealed and would otherwise be `undefined` with nothing to notice it. + * + * node upload_seal_probe.mjs <static-dir> <input.json> + * + * Two modes, because the node half runs in Python between them: + * "send" — run the upload, print every frame it emits, answer nothing + * "receive" — run it again, answering with acks Python built. Their + * `upload_id` is retargeted to this run's; it is outside the + * seal, so the ciphertext stays exactly the one Python produced. + */ +import fs from 'fs'; + +const STATIC = process.argv[2]; +const input = JSON.parse(fs.readFileSync(process.argv[3], 'utf8')); + +for (const level of ['log', 'warn', 'error', 'info', 'debug']) { + console[level] = (...args) => process.stderr.write(args.join(' ') + '\n'); +} + +globalThis.window = globalThis; +globalThis.addEventListener = () => {}; +globalThis.removeEventListener = () => {}; +globalThis.location = { hash: '' }; +globalThis.document = { + addEventListener() {}, removeEventListener() {}, visibilityState: 'visible', +}; + +new Function(fs.readFileSync(`${STATIC}/crypto.js`, 'utf8'))(); +const transportSrc = fs.readFileSync(`${STATIC}/transport.js`, 'utf8'); +new Function(transportSrc)(); +// The msgpack codec is private to transport.js — pulled out the same way the +// groupbox parity harness pulls out sealGroup, so this probe encodes and +// decodes with the codec that is actually shipped rather than a second one. +const { msgpack_encode, msgpack_decode } = + new Function(transportSrc + '\nreturn { msgpack_encode, msgpack_decode };')(); + +const hex = (s) => Uint8Array.from(s.match(/../g).map((b) => parseInt(b, 16))); +const toHex = (u8) => + Array.from(u8).map((b) => b.toString(16).padStart(2, '0')).join(''); + +// Just enough of a File: a name, a size, and slices that yield ArrayBuffers. +const bytes = hex(input.file.data); +const file = { + name: input.file.name, + size: bytes.length, + slice(a, b) { + const part = bytes.slice(a, b); + return { arrayBuffer: async () => part.buffer.slice( + part.byteOffset, part.byteOffset + part.byteLength) }; + }, +}; + +const tp = new window.MeshBayTransport('', 'token'); +tp._connected = true; +tp._channel = { readyState: 'open', bufferedAmount: 0, send() {}, close() {} }; +tp._pc = { close() {} }; +tp._gekRaw = hex(input.gek); +tp._connectArgs = { groupId: input.group_id }; +tp._nodeVersion = input.node_version; + +const frames = []; +let uploadId = null; +tp._send = (msg) => { + frames.push(toHex(msgpack_encode(msg))); + if (msg.upload_id) uploadId = msg.upload_id; + if (input.mode !== 'receive') return; + // Answer as the node did, on the next turn of the loop so the send path + // finishes first — which is also how a real ack arrives. + const ack = msgpack_decode(hex(input.acks[msg.chunk_index])); + ack.upload_id = uploadId; + // Through the real `_dispatch`, so the routing under test — matching an + // ack to its uploader by `upload_id` — is the shipped one. + setImmediate(() => tp._dispatch(ack)); +}; + +const out = { frames, mode: input.mode }; +const done = tp.uploadFile(file, { chunkSize: input.chunk_size, + dir: input.dir, root: input.root }); + +if (input.mode === 'receive') { + done.then((stored) => { out.state = 'resolved'; out.stored = stored; }) + .catch((e) => { out.state = 'rejected'; out.message = e.message; }) + .finally(() => { out.upload_id = uploadId; + process.stdout.write(JSON.stringify(out)); }); +} else { + // Nothing will answer, so let the send loop run itself out and report. + done.catch((e) => { out.state = 'rejected'; out.message = e.message; }); + setTimeout(() => { out.upload_id = uploadId; + process.stdout.write(JSON.stringify(out)); }, 250); +} diff --git a/packages/meshbay-hub/tests/test_transport_contracts.py b/packages/meshbay-hub/tests/test_transport_contracts.py index b462242..879062b 100644 --- a/packages/meshbay-hub/tests/test_transport_contracts.py +++ b/packages/meshbay-hub/tests/test_transport_contracts.py @@ -307,26 +307,60 @@ def test_no_setter_survives_the_state_it_belonged_to(): # ── Parallel uploads ────────────────────────────────────────────────────────── -def test_an_upload_refusal_names_the_file_it_is_about(transport): +def test_an_upload_refusal_names_the_upload_it_is_about(transport): """Reported 2026-08-16: a second upload started in parallel killed both. - An error used to carry no filename, so the client could not tell whose it - was and failed every upload in flight — one name the node disliked took the - other file with it. The node names the file now, and only that upload stops. + An error used to carry nothing identifying, so the client could not tell + whose it was and failed every upload in flight — one name the node disliked + took the other file with it. + + The node named the *file* until MNP 2.0 and names the `upload_id` now: the + filename moved inside the seal, and echoing it in clear so the two sides + could match on it would give back precisely what sealing the upload is for. + The property is unchanged — one refusal, one failed upload. """ body = transport[transport.index("if (msg.type === 'error' && this._uploaders.size)"):] body = body[:body.index("\n if (msg.type === 'chat_msg'")] - assert "this._uploaders.has(msg.filename)" in body, ( + assert "this._uploaders.has(msg.upload_id)" in body, ( "a named refusal must reach one uploader, not all of them") - assert "if (!msg.filename)" in body, ( + assert "if (!msg.upload_id)" in body, ( "an unnamed error from an older node must still stop everything — " "guessing which upload it belongs to would be worse") -def test_uploads_are_tracked_per_file(transport): +def test_uploads_are_tracked_per_upload(transport): """Acks interleave when two files are in flight.""" assert "this._uploaders = new Map()" in transport - assert "this._uploaders.set(file.name" in transport + assert "this._uploaders.set(uploadId" in transport + # And the "already being uploaded" guard still speaks in filenames, because + # that is what the caller passed and what it would recognise in the error. + assert "this._inFlightUploads.has(file.name)" in transport + + +def test_the_upload_itself_is_sealed(transport): + """ + MNP 2.0. The filename, the destination and the bytes go inside the seal + together — sealing the content and announcing the name beside it would be + theatre — and only what the node routes on stays outside. + """ + start = transport.index(" async uploadFile(file,") + body = transport[start:transport.index("\n /** Create a directory", start)] + assert "sealGroup(" in body and "'file_upload'" in body, ( + "the upload must be sealed under the group key") + assert "openGroup(" in body and "'file_upload_ack'" in body, ( + "the ack carries the stored name and must be opened, not read") + # The message the node actually receives: everything between `this._send({` + # and its close. Read on its own, because the same field names appear a few + # lines above inside `msgpack_encode({...})`, which is the sealed half. + sent = body[body.index("this._send({"):] + sent = sent[:sent.index("});")] + assert "filename" not in sent, "the filename is on the message in clear" + assert "data" not in sent, "the bytes are on the message in clear" + assert "dir" not in sent and "root" not in sent, ( + "the destination is on the message in clear") + assert "...sealed," in sent, "the message must carry the sealed pair" + assert "supportsSealedUpload" in body, ( + "an older node must be refused before a chunk is sent, not after") # ── MNP 1.0: the sealed handshake ack ──────────────────────────────────────── diff --git a/packages/meshbay-hub/tests/test_upload_seal_client.py b/packages/meshbay-hub/tests/test_upload_seal_client.py new file mode 100644 index 0000000..d6f9156 --- /dev/null +++ b/packages/meshbay-hub/tests/test_upload_seal_client.py @@ -0,0 +1,169 @@ +""" +The browser half of MNP 2.0's sealed upload, measured rather than read. + +`test_upload_sealed.py` (node side) proves the node opens what the shared +encoder produces and refuses everything else. This proves the *shipped browser +code* produces it — and, the part that matters more, that a caller of +`uploadFile` is still told the name the node stored the file under, which now +arrives sealed and would otherwise be `undefined` with nothing on screen to say +so: a chat attachment would point at a file that is not there. + +Driven through `harness/upload_seal_probe.mjs`, which runs the shipped +`transport.js` over the shipped `crypto.js`. The node half in between is the +real `_do_file_upload`, writing to a real directory. + +A source-reading test can see that `sealGroup` is called. Only this can see +whether what comes out of it opens. +""" + +import json +import shutil +import subprocess +import tempfile +from pathlib import Path + +import msgpack +import pytest +from cryptography.hazmat.primitives.asymmetric.ed25519 import Ed25519PrivateKey +from meshbay_common.crypto import generate_gek +from meshbay_common.protocol import MNP +from meshbay_node.indexer.group_index import GroupIndex +from meshbay_node.roots import RootSet +from meshbay_node.transport.webrtc_server import WebRTCPeerSession + +STATIC = Path(__file__).resolve().parents[1] / "src" / "meshbay_hub" / "static" +PROBE = Path(__file__).resolve().parent / "harness" / "upload_seal_probe.mjs" + +pytestmark = pytest.mark.skipif( + shutil.which("node") is None or not PROBE.exists(), + reason="node unavailable — the client half cannot be measured", +) + +GROUP = "g-upload-probe" +CHUNK = 32 +BODY = bytes(range(256)) * 3 # 768 bytes → 24 chunks of 32 + + +def _run_probe(payload: dict) -> dict: + with tempfile.TemporaryDirectory() as d: + f = Path(d) / "input.json" + f.write_text(json.dumps(payload)) + proc = subprocess.run( + ["node", str(PROBE), str(STATIC), str(f)], + capture_output=True, text=True, timeout=60, + ) + if proc.returncode != 0 or not proc.stdout: + pytest.fail(f"upload probe failed:\n{proc.stderr}") + return json.loads(proc.stdout) + + +def _node_session(tmp_path: Path, gek: bytes) -> WebRTCPeerSession: + root = tmp_path / "library" + root.mkdir() + index = GroupIndex(group_id=GROUP, sk_node=Ed25519PrivateKey.generate()) + session = WebRTCPeerSession.__new__(WebRTCPeerSession) + session._ctx = { + "roots": RootSet.build([{"path": str(root), "name": "library", + "writable": True}]), + "index": index, "sk_node": index.sk_node, "gek": gek, + } + session._group_id = GROUP + session._user_id = "prober" + session._pk_user = "" + session._uploads = {} + session.sent = [] + session._send = session.sent.append + session._audit = lambda *a, **k: None + return session + + +def _probe_input(gek: bytes, mode: str, **extra) -> dict: + return { + "mode": mode, "gek": gek.hex(), "group_id": GROUP, + "node_version": "2.0", "chunk_size": CHUNK, "dir": "library", + "root": "library", + "file": {"name": "holiday.jpg", "data": BODY.hex()}, + **extra, + } + + +@pytest.fixture(scope="module") +def _gek(): + return generate_gek() + + +@pytest.fixture(scope="module") +def _sent(_gek): + """Every frame the shipped `uploadFile` puts on the wire, unanswered.""" + return _run_probe(_probe_input(_gek, "send")) + + +def test_the_browser_puts_no_filename_and_no_content_on_the_wire(_sent): + """The whole point, measured on the bytes rather than read off the source.""" + assert _sent["frames"], "the client sent nothing" + for hexframe in _sent["frames"]: + raw = bytes.fromhex(hexframe) + assert b"holiday.jpg" not in raw, "the filename is on the wire in clear" + msg = msgpack.unpackb(raw, raw=False) + assert set(msg) == {"type", "v", "upload_id", "chunk_index", + "total_chunks", "nonce", "ct"} + assert msg["type"] == MNP.FILE_UPLOAD + + +def test_a_real_node_opens_what_the_real_browser_sealed(tmp_path, _gek, _sent): + """ + End to end: the shipped browser encoder into the shipped node handler, with + the file that lands on disk as the assertion. A mismatch in the HKDF salt, + the AAD encoding or the payload shape shows up here as "did not open" — and + nowhere else until somebody tries to upload something. + """ + session = _node_session(tmp_path, _gek) + for hexframe in _sent["frames"]: + session._do_file_upload(msgpack.unpackb(bytes.fromhex(hexframe), raw=False)) + + errors = [m for m in session.sent if m.get("type") == "error"] + assert not errors, f"the node refused a frame the browser built: {errors[:1]}" + + root = session._ctx["roots"].roots[0].path + assert (root / "holiday.jpg").read_bytes() == BODY + assert not list(root.glob("*.part")), "a temp file was left behind" + + +def test_the_caller_is_told_the_name_the_node_chose(tmp_path, _gek, _sent): + """ + `stored_as` is sealed now, so reading it takes a decrypt that can fail + silently. It must not: the node finds a free name rather than replacing + anything, and a chat attachment that never learns which name points at + nothing. + + The acks below are the ones the node really produced — only their + `upload_id`, which is outside the seal, is retargeted to the second probe + run's own upload. + """ + session = _node_session(tmp_path, _gek) + # A file of that name is already there, so the node has to choose another. + (session._ctx["roots"].roots[0].path / "holiday.jpg").write_bytes(b"someone else's") + + for hexframe in _sent["frames"]: + session._do_file_upload(msgpack.unpackb(bytes.fromhex(hexframe), raw=False)) + acks = [msgpack.packb(m, use_bin_type=True).hex() + for m in session.sent if m.get("type") == MNP.FILE_UPLOAD_ACK] + assert len(acks) == len(_sent["frames"]) + + result = _run_probe(_probe_input(_gek, "receive", acks=acks)) + assert result["state"] == "resolved", result.get("message") + assert result["stored"]["stored_as"] == "holiday (2).jpg" + assert result["stored"]["dir"] == "library" + + +def test_the_client_refuses_an_older_node_before_sending_a_chunk(_gek): + """ + A 1.x node would answer "Missing filename or data" — an error about the + wrong thing, naming no upload, which fails every upload in flight. Asked + first instead, and nothing goes on the wire. + """ + result = _run_probe(_probe_input(_gek, "receive", acks=[], + node_version="1.1")) + assert result["state"] == "rejected" + assert "older MeshBay" in result["message"] + assert result["frames"] == [], "a chunk was sent to a node that cannot open it" |