Merge origin/main into the chat encryption work

Both sides landed a breaking MNP change and both called it 2.0, which is right:
the sealed upload, the removal of `stream_seg` and mandatory chat encryption
share one flag day. They are recorded as one version in `__init__.py` rather
than as a race between two.

The resolutions that were decisions rather than mechanics:

* **`MNP_MIN_SUPPORTED` moves to "2.0".** The sealed upload alone was a
  *confined* break — a 1.x peer could still connect, browse, download, stream
  and chat, with only its uploads refused by `upload_not_sealed` — so the floor
  deliberately stayed at "1.0". Mandatory chat encryption ends that
  confinement: a 1.x peer can neither produce a sealed chat message nor read
  one, so it would connect, look fine, and be unable to say anything. Refusing
  it at the handshake is the honest form. The per-message `upload_not_sealed`
  path is untouched and still right if the floor is ever lowered.
* **`sendChat` throws on an `error` reply**, from origin, applied to the sealed
  send. It matters more after this change, not less: the node now refuses a
  stale epoch, a malformed envelope and a device claim that is not the
  connection's own, so there are three new ways for a message to be rejected
  and none of them may look like a message that was sent.
* **`req_id` supersedes the per-type routing** this branch added for
  `chat_keys_resp` and `device_hello_ack`. Both blocks are kept beside the
  existing `chat_hist_resp` one, for the same stated reason — a node too old to
  stamp — and their comments no longer claim to be the mechanism that closes
  the class. `req_id` is.
* **`chat_send_probe.py` is rebuilt on origin's structure**, not beside it: two
  scenarios, a stub that stamps `req_id`, `music_meta_req` as the older pending
  request. The encrypted path is layered on — a real Ed25519 device key
  generated in the page, and a `chat_keys_resp` sealed by the shipped Python,
  because a payload the page built itself would prove only that the page agrees
  with the page.
* **`test_reply_correlation.py` now sends a sealed message.** Its subject is
  which of the two messages leaving that handler carries the id; plaintext chat
  was only the fixture, and the node refuses one now.
* `groupbox` keeps both new purposes (`upload`, `chat_keys`); `protocol.py`
  keeps origin's removal of `STREAM_SEGMENT` and this branch's correction of
  the "Double Ratchet message" comment on `CHAT_MESSAGE`, which was wrong when
  it was written and is wrong differently now.

Full suite on the merged tree: 1993 passed, 11 failed — the same 11 that fail
on a pristine checkout (2 Windows service tests, 1 apps-enabled policy, 7
transcode tests that pass in isolation, and the WebRTC invite test that hangs
on its own).

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01TZZxYjz8YeWRz13xDi8LJr

1 files changed, 145 insertions, 6 deletions

diff --git a/packages/meshbay-common/src/meshbay_common/protocol.py b/packages/meshbay-common/src/meshbay_common/protocol.py
index dfb5d56..cc5324a 100644
--- a/packages/meshbay-common/src/meshbay_common/protocol.py
+++ b/packages/meshbay-common/src/meshbay_common/protocol.py
@@ -6,8 +6,31 @@ MHP (Mesh Bay Hub Protocol) — v0.1
 
 All wire messages are length-prefixed msgpack (4-byte big-endian length header).
 Every message carries a "v" field for protocol version.
+
+**`req_id` — the correlation id (added 2026-09-07).** A request may carry one;
+the reply to it carries the same value back, and nothing else on the wire does.
+It is the caller's own key for its pending request, opaque to the node, and
+unique only within one connection.
+
+There was none for a long time, and its absence was not neutral. A reply named
+its own type and nothing else, so a caller with more than one request in flight
+had to work out which one a message answered from the message itself — and the
+replies that name nothing (a bare `ack`, and `{"type": "error"}`, which
+webrtc_server.py sends from 240 places while two of them say what they are
+about) could only be matched by arrival order. That is a guess, wrong whenever
+two replies reorder, and it does not fail quietly: one request is resolved with
+another's answer while the request that answer belonged to waits out its own
+timeout. Live symptom (2026-09-06): a chat send whose reply went astray left
+the composer disabled for thirty seconds, and the Chat tab read as frozen.
+
+Both halves are optional and degrade to what came before: a request without one
+is answered without one, and a client that gets no id back falls back to
+matching by type. Neither side may treat it as authentication or as a sequence
+number — it is a label chosen by the peer, and the only thing it decides is
+which local promise a reply belongs to.
 """
 
+import os
 from dataclasses import dataclass, field
 from typing import Any
 
@@ -15,6 +38,7 @@ from typing import Any
 # second copy here said "0.1" while every message on the wire carried "0.2".
 # Nothing imported it, which is the only reason it was harmless.
 from meshbay_common import MNP_VERSION, MHP_VERSION   # noqa: F401  (re-export)
+from meshbay_common.groupbox import PURPOSE_UPLOAD, seal, unseal
 from meshbay_common.webcrypto import (
     chunk_key_aes,
     decrypt_chunk_aes,
@@ -37,13 +61,21 @@ class MNP:
     INDEX_PROGRESS  = "index_progress"
     FILE_REQUEST    = "file_req"         # request chunk(s)
     FILE_CHUNK      = "file_chunk"       # encrypted chunk response
-    STREAM_SEGMENT  = "stream_seg"       # HLS/DASH segment
+    # STREAM_SEGMENT ("stream_seg") was removed in MNP 2.0. It served an
+    # MPEG-TS segment as base64 **with no encryption at all** — the one message
+    # on the content plane that never was under a GEK-derived key. It predates
+    # STREAM_DATA, which does the same job properly (`chunk_ciphertext`, keyed
+    # per segment), and its browser caller `fetchStreamSegment` was defined and
+    # never once invoked. A live handler on both transports, plaintext media,
+    # and no client: removed rather than repaired.
+    #
     # Not a Double Ratchet message, and never was — `first-review.md` C1
-    # rejected exactly that for groups. Plaintext until a group turns
-    # encryption on, then AES-256-GCM under a per-device subkey of the group's
-    # chat epoch key, signed with the sending device's pinned Ed25519 key
-    # (`chatbox.py`, docs/chat-sender-keys.md).
-    CHAT_MESSAGE    = "chat_msg"         # one chat message, plain or sealed
+    # rejected exactly that for groups. Since MNP 2.0 it is AES-256-GCM under a
+    # per-device subkey of the group's chat epoch key, signed over the
+    # ciphertext with the sending device's pinned Ed25519 key. There is no
+    # plaintext form on the wire (`chatbox.py`, docs/chat-sender-keys.md);
+    # `format` distinguishes a *stored* pre-2.0 row, which is still served.
+    CHAT_MESSAGE    = "chat_msg"         # one chat message, sealed and signed
     CHAT_ATTACHMENT = "chat_attach"      # attachment metadata
     CHAT_HISTORY    = "chat_hist"       # request message history (newest, or before a cursor)
     CHAT_HISTORY_RESPONSE = "chat_hist_resp"  # history response with messages
@@ -369,3 +401,110 @@ def file_chunk_plaintext(
         file_hash = bytes.fromhex(msg["file_id"])
     ckey = chunk_key_aes(gek, file_hash, msg["chunk_index"])
     return decrypt_chunk_aes(ckey, msg["nonce"], msg["ct"])
+
+
+# ── Uploads (MNP 2.0) ─────────────────────────────────────────────────────────
+#
+# The write path, sealed under the group key the way the read path always was.
+# One encoder for both directions, here rather than in the client, for the reason
+# `file_chunk` has one: two copies of a wire shape with a single consumer each is
+# how `index_sync` and `file_chunk` forked (finding C6), and nothing noticed
+# until someone went looking.
+#
+# What stays in clear, and why each has to:
+#   `type`, `v`     — routed and version-checked before anything can be decrypted
+#   `upload_id`     — the correlation key. It replaces `filename`, which used to
+#                     play that role and cannot any more: naming the file in clear
+#                     to match an ack against a request would give back exactly
+#                     what the seal is for. Client-chosen, opaque to the node,
+#                     unique within one connection; never an authorization input.
+#   `chunk_index`   — ordering, which the node enforces before it opens anything
+#   `total_chunks`  — how many to expect
+#
+# `group_id` is *not* on the message: the session already decided which group it
+# is on, and the node uses that as the AAD. A client naming its own group here
+# would be choosing which key its bytes are checked against.
+
+UPLOAD_ID_LEN = 16   # 128 bits of client-chosen correlation, hex on the wire
+
+
+def new_upload_id() -> str:
+    """A fresh correlation id for one upload."""
+    return os.urandom(UPLOAD_ID_LEN).hex()
+
+
+def file_upload_wire(
+    gek: bytes,
+    group_id: str,
+    *,
+    upload_id: str,
+    chunk_index: int,
+    total_chunks: int,
+    filename: str,
+    data: bytes,
+    dir: str = "",
+    root: str = "",
+) -> dict:
+    """
+    One sealed `file_upload` chunk.
+
+    `filename`, `dir` and `root` ride inside the seal with the bytes: sealing the
+    content and announcing the name beside it would be theatre. They are repeated
+    on every chunk rather than sent once — a hundred bytes against a 48 KiB chunk
+    — because a header that arrives once is state the node has to carry, and
+    upload state that can disagree with the chunk in hand is what `_free_name` and
+    the chunk-ordering rule exist to prevent.
+    """
+    payload = {"filename": filename, "data": data, "dir": dir, "root": root}
+    return {
+        "type": MNP.FILE_UPLOAD,
+        "v": MNP_VERSION,
+        "upload_id": upload_id,
+        "chunk_index": chunk_index,
+        "total_chunks": total_chunks,
+        **seal(gek, PURPOSE_UPLOAD, MNP.FILE_UPLOAD, group_id, payload),
+    }
+
+
+def file_upload_payload(gek: bytes, group_id: str, msg: dict) -> dict:
+    """
+    Open a `file_upload`. Raises on anything that does not open.
+
+    Never a partial result and never a default: a chunk that does not open is not
+    an empty file with an empty name, it is a peer we cannot talk to. `unseal`
+    says why at length.
+    """
+    return unseal(gek, PURPOSE_UPLOAD, MNP.FILE_UPLOAD, group_id, msg)
+
+
+def file_upload_ack_wire(
+    gek: bytes,
+    group_id: str,
+    *,
+    upload_id: str,
+    chunk_index: int,
+    filename: str,
+    stored_as: str,
+    dir: str = "",
+) -> dict:
+    """
+    The node's answer to one chunk, sealed the same way.
+
+    `stored_as` is the name the node settled on — it finds a free one rather than
+    replacing anything — and `dir` is where it landed. Both name the operator's
+    content, so both belong inside the seal; only `upload_id` and `chunk_index`
+    stay out, because the client matches on them.
+    """
+    payload = {"filename": filename, "stored_as": stored_as, "dir": dir}
+    return {
+        "type": MNP.FILE_UPLOAD_ACK,
+        "v": MNP_VERSION,
+        "upload_id": upload_id,
+        "chunk_index": chunk_index,
+        **seal(gek, PURPOSE_UPLOAD, MNP.FILE_UPLOAD_ACK, group_id, payload),
+    }
+
+
+def file_upload_ack_payload(gek: bytes, group_id: str, msg: dict) -> dict:
+    """Open a `file_upload_ack`. Raises on anything that does not open."""
+    return unseal(gek, PURPOSE_UPLOAD, MNP.FILE_UPLOAD_ACK, group_id, msg)