Merge origin/main into the chat encryption work

Both sides landed a breaking MNP change and both called it 2.0, which is right:
the sealed upload, the removal of `stream_seg` and mandatory chat encryption
share one flag day. They are recorded as one version in `__init__.py` rather
than as a race between two.

The resolutions that were decisions rather than mechanics:

* **`MNP_MIN_SUPPORTED` moves to "2.0".** The sealed upload alone was a
  *confined* break — a 1.x peer could still connect, browse, download, stream
  and chat, with only its uploads refused by `upload_not_sealed` — so the floor
  deliberately stayed at "1.0". Mandatory chat encryption ends that
  confinement: a 1.x peer can neither produce a sealed chat message nor read
  one, so it would connect, look fine, and be unable to say anything. Refusing
  it at the handshake is the honest form. The per-message `upload_not_sealed`
  path is untouched and still right if the floor is ever lowered.
* **`sendChat` throws on an `error` reply**, from origin, applied to the sealed
  send. It matters more after this change, not less: the node now refuses a
  stale epoch, a malformed envelope and a device claim that is not the
  connection's own, so there are three new ways for a message to be rejected
  and none of them may look like a message that was sent.
* **`req_id` supersedes the per-type routing** this branch added for
  `chat_keys_resp` and `device_hello_ack`. Both blocks are kept beside the
  existing `chat_hist_resp` one, for the same stated reason — a node too old to
  stamp — and their comments no longer claim to be the mechanism that closes
  the class. `req_id` is.
* **`chat_send_probe.py` is rebuilt on origin's structure**, not beside it: two
  scenarios, a stub that stamps `req_id`, `music_meta_req` as the older pending
  request. The encrypted path is layered on — a real Ed25519 device key
  generated in the page, and a `chat_keys_resp` sealed by the shipped Python,
  because a payload the page built itself would prove only that the page agrees
  with the page.
* **`test_reply_correlation.py` now sends a sealed message.** Its subject is
  which of the two messages leaving that handler carries the id; plaintext chat
  was only the fixture, and the node refuses one now.
* `groupbox` keeps both new purposes (`upload`, `chat_keys`); `protocol.py`
  keeps origin's removal of `STREAM_SEGMENT` and this branch's correction of
  the "Double Ratchet message" comment on `CHAT_MESSAGE`, which was wrong when
  it was written and is wrong differently now.

Full suite on the merged tree: 1993 passed, 11 failed — the same 11 that fail
on a pristine checkout (2 Windows service tests, 1 apps-enabled policy, 7
transcode tests that pass in isolation, and the WebRTC invite test that hangs
on its own).

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01TZZxYjz8YeWRz13xDi8LJr

2 files changed, 240 insertions, 104 deletions

diff --git a/packages/meshbay-hub/tests/harness/chat_send_probe.py b/packages/meshbay-hub/tests/harness/chat_send_probe.py
index 1f0557b..5f99beb 100644
--- a/packages/meshbay-hub/tests/harness/chat_send_probe.py
+++ b/packages/meshbay-hub/tests/harness/chat_send_probe.py
@@ -20,8 +20,40 @@ never appeared, while the node had stored it all along.
 
     chat_send_probe.py
 
-Prints JSON: `steps`, the state of the panel at each stage, and `log`, what the
-transport sent and how the deliberately-unanswered request ended up.
+It now drives two shapes of reply, because there were two ways for one to go
+astray and only the first was ever fixed:
+
+  `ack`    the node accepts the message and answers `{"type": "ack"}`, which
+           names no request. Routed by request type since 2026-08-30.
+  `error`  the node *refuses* it — every failure in `_dispatch_message` ends at
+           one catch-all sending `{"type": "error", "detail": "Request failed"}`,
+           and 238 of this module's 240 error sends name nothing either. That
+           reply reached no caller at all: it went to whatever request happened
+           to be waiting, and the send sat out its own 30s timeout with the
+           composer disabled.
+
+Both are run with an older request already pending — the condition that turns
+"guess by arrival order" from usually-right into wrong — and both must come
+back inside a second and a half.
+
+Since MNP 2.0 a send also has to **seal and sign for real** before it goes
+anywhere, so this drives `chatKeys()`, `openGroup`, `sealChat` and a genuine
+Ed25519 signature rather than a model of any of them. The device key is
+generated in the page — `signBytes` imports a pkcs8 key and WebCrypto will not
+be fooled by a stand-in — and the `chat_keys_resp` the stub answers with is
+sealed **by the shipped Python**, because a payload the page built itself would
+prove only that the page agrees with the page.
+
+That path found two defects the moment it first ran, neither visible in any
+source file: `chat_keys_resp` was routed by arrival order and handed to the
+older pending request (this defect, in a message type that did not exist when
+the probe was written), and `_asText` had been deleted along with an unrelated
+helper beside it — its only caller sits inside a promise the panel catches, so
+every conversation rendered empty with nothing in the console.
+
+Prints JSON: `scenarios`, the state of the panel at each stage of each, and
+`log`, what the transport sent and how the deliberately-unanswered request
+ended up.
 """
 import http.server
 import json
@@ -49,9 +81,9 @@ def _page() -> str:
 
     Sealed here, by the shipped Python, rather than assembled in the browser:
     msgpack is private to transport.js and exported to nothing, and a payload
-    the test built itself would prove only that the page agrees with the page.
+    the page built itself would prove only that the page agrees with the page.
     """
-    import msgpack
+    import msgpack  # noqa: F401  (imported for the failure it gives if absent)
 
     from meshbay_common.groupbox import PURPOSE_CHAT_KEYS, seal
 
@@ -73,9 +105,7 @@ PAGE_TEMPLATE = r"""<!doctype html><html><head><meta charset=utf-8>
 </div></div>
 <!-- The two the real page loads and the transport reaches for by global:
      `sealChat`/`openGroup` live in crypto.js, `signBytes` in keyderive.js.
-     Without them a send fails with "cannot read properties of undefined",
-     which is what this probe reported the first time it exercised the
-     encrypted path. -->
+     Without them a send fails with "cannot read properties of undefined". -->
 <script src="/crypto.js"></script>
 <script src="/keyderive.js"></script>
 <script src="/transport.js"></script>
@@ -87,37 +117,16 @@ const log = [];
 window.addEventListener('error', e => log.push('error: ' + e.message));
 window.addEventListener('unhandledrejection',
   e => log.push('rejected: ' + (e.reason && e.reason.message || e.reason)));
-const _warn = console.warn, _err = console.error;
-console.warn = (...a) => { log.push('warn: ' + a.join(' ')); _warn(...a); };
-console.error = (...a) => { log.push('console error: ' + a.join(' ')); _err(...a); };
 
 const hex = (s) => Uint8Array.from(s.match(/../g) || [], b => parseInt(b, 16));
+const b64 = (buf) => btoa(String.fromCharCode(...new Uint8Array(buf)));
 
-// The real transport, with only the channel replaced: _send takes the plain
-// object _sendAndWait built, so the framing and msgpack are the only things
-// skipped — every pending entry, key and dispatch path below is the shipped one.
-const tp = new window.MeshBayTransport('', 'token');
-tp._connected = true;
-tp._channel = { readyState: 'open', send() {} };
-
-// Chat is encrypted (MNP 2.0), so a send that is going to come back has to
-// seal and sign for real. The device key is generated here rather than stubbed
-// — `signBytes` imports a pkcs8 key and WebCrypto will not be fooled — and the
-// group key and epoch keys come from Python, which sealed the `chat_keys_resp`
-// below exactly as the node does. So this exercises `chatKeys()`, `openGroup`,
-// `sealChat` and the real signature, not a model of any of them.
-tp._groupId = '__GROUP_ID__';
-tp._gekRaw = hex('__GEK_HEX__');
-tp.chatEpoch = 1;
-
+// One device key for both scenarios. Real, not stubbed: `signBytes` imports a
+// pkcs8 key, so nothing else gets a signature past `verifyChatSignature`.
 const kp = await crypto.subtle.generateKey({ name: 'Ed25519' }, true,
                                            ['sign', 'verify']);
-const b64 = (buf) => btoa(String.fromCharCode(...new Uint8Array(buf)));
-tp._sessionKeys = {
-  skEdB64: b64(await crypto.subtle.exportKey('pkcs8', kp.privateKey)),
-};
-// What `device_hello` sets on a live connection.
-tp.devicePk = b64(await crypto.subtle.exportKey('raw', kp.publicKey));
+const SK_ED_B64 = b64(await crypto.subtle.exportKey('pkcs8', kp.privateKey));
+const DEVICE_PK_B64 = b64(await crypto.subtle.exportKey('raw', kp.publicKey));
 
 const now = Date.now() / 1000;
 const history = [];
@@ -126,75 +135,72 @@ for (let i = 0; i < 5; i++) {
                  payload: 'message ' + i, timestamp: now - (5 - i) * 60 });
 }
 
-// Stands in for the node, answering exactly what webrtc_server.py answers.
-// media_meta_req is answered with nothing at all, which is what a refusal
-// amounts to for the request that asked: `_do_media_meta_request` sends a bare
-// `error` for a file_id the index does not have, and a bare error names no
-// request, so it reaches none.
-tp._send = (obj) => {
-  log.push('sent ' + obj.type);
-  if (obj.type === 'chat_hist') {
-    setTimeout(() => {
-      log.push('answering chat_hist');
-      tp._dispatch({ type: 'chat_hist_resp', v: '0.2', messages: history,
-                     has_more: false });
-    }, 10);
-  } else if (obj.type === 'chat_keys_req') {
-    // Sealed under the group key, as `_do_chat_keys_req` sends it.
-    setTimeout(() => tp._dispatch({
-      type: 'chat_keys_resp', v: '2.0', group_id: tp._groupId,
-      nonce: hex('__KEYS_NONCE_HEX__'), ct: hex('__KEYS_CT_HEX__'),
-    }), 10);
-  } else if (obj.type === 'chat_msg') {
-    // Recorded so the test can assert the message really was sealed and
-    // signed rather than sent in clear past a composer that let it through.
-    log.push('chat_msg format=' + obj.format + ' epoch=' + obj.epoch
-             + ' ct=' + (obj.ct ? obj.ct.length : 0)
-             + ' sig=' + (obj.sig ? obj.sig.length : 0)
-             + ' plaintextLeak=' + JSON.stringify(obj).includes('hello'));
-    setTimeout(() => tp._dispatch({ type: 'ack', v: '2.0' }), 10);
-  }
-};
+const wait = ms => new Promise(r => setTimeout(r, ms));
 
-// The panel swallows a send failure into `setInput(text)`, which is right for
-// a person and useless for a probe: the symptom is the message not appearing,
-// with no reason anywhere. Surfaced here so a failure names itself.
-const _sendChat = tp.sendChat.bind(tp);
-tp.sendChat = (...a) => _sendChat(...a).catch((e) => {
-  log.push('sendChat failed: ' + (e && e.message || e));
-  throw e;
-});
+// Stands in for the node, answering what webrtc_server.py answers — including
+// stamping the reply with the id of the request it is answering, which is what
+// `_send` does there. `chatReply` is the only difference between the two runs.
+function makeTransport(name, chatReply) {
+  // The real transport, with only the channel replaced: _send takes the plain
+  // object _sendAndWait built, so the framing and msgpack are the only things
+  // skipped — every pending entry, key and dispatch path below is the shipped
+  // one.
+  const tp = new window.MeshBayTransport('', 'token');
+  tp._connected = true;
+  tp._channel = { readyState: 'open', send() {} };
+  // What a completed MNP 2.0 handshake leaves behind: the group and its key
+  // from `connect`, the current epoch from the sealed ack, and the device this
+  // connection identified itself as with `device_hello`.
+  tp._groupId = '__GROUP_ID__';
+  tp._gekRaw = hex('__GEK_HEX__');
+  tp.chatEpoch = 1;
+  tp._sessionKeys = { skEdB64: SK_ED_B64 };
+  tp.devicePk = DEVICE_PK_B64;
+  tp._send = (obj) => {
+    log.push('sent ' + obj.type);
+    const answer = (reply) => setTimeout(
+      () => tp._dispatch({ ...reply, req_id: obj.req_id }), 10);
+    if (obj.type === 'chat_hist') {
+      answer({ type: 'chat_hist_resp', v: '0.2', messages: history, has_more: false });
+    } else if (obj.type === 'chat_keys_req') {
+      // Sealed under the group key, as `_do_chat_keys_req` sends it.
+      answer({ type: 'chat_keys_resp', v: '2.0', group_id: tp._groupId,
+               nonce: hex('__KEYS_NONCE_HEX__'), ct: hex('__KEYS_CT_HEX__') });
+    } else if (obj.type === 'chat_msg') {
+      // Recorded so the test can assert what actually left the browser, rather
+      // than trusting that a composer which accepted the text sealed it.
+      log.push(name + ': chat_msg format=' + obj.format + ' epoch=' + obj.epoch
+               + ' ct=' + (obj.ct ? obj.ct.length : 0)
+               + ' sig=' + (obj.sig ? obj.sig.length : 0)
+               + ' plaintextLeak=' + JSON.stringify(obj).includes('hello'));
+      answer(chatReply);
+    }
+    // music_meta_req is answered by nothing at all, on purpose: the node holds
+    // one open for as long as the third-party lookup behind it takes, which
+    // was measured at over 100 seconds with that service failing. It is the
+    // older pending request every scenario here needs.
+  };
+  // The panel swallows a send failure into `setInput(text)`, which is right for
+  // a person and useless for a probe: the symptom is a message not appearing,
+  // with no reason anywhere. Surfaced here so a failure names itself — the
+  // `error` scenario is *expected* to reach this.
+  const _sendChat = tp.sendChat.bind(tp);
+  tp.sendChat = (...a) => _sendChat(...a).catch((e) => {
+    log.push(name + ': sendChat failed: ' + (e && e.message || e));
+    throw e;
+  });
+  return tp;
+}
 
-function Host() {
+function Host({ tp }) {
   const transportRef = useRef(tp);
   const gekRef = useRef(null);
   return html`<${ChatPanel} transportRef=${transportRef} gekRef=${gekRef}
     username="me" userId="user-me" entries=${[]} status="connected" />`;
 }
-render(html`<${Host} />`, document.getElementById('root'));
 
-const out = { steps: [], log };
-const composer = () => document.querySelector('.chat-input');
-
-function snap(label) {
-  const c = composer();
-  out.steps.push({
-    label,
-    bubbles: document.querySelectorAll('.chat-bubble').length,
-    msgs: document.querySelectorAll('.chat-msg').length,
-    lastText: [...document.querySelectorAll('.chat-text')].pop()?.textContent ?? null,
-    // What a frozen tab actually is: the composer is disabled for as long as
-    // a send is in flight.
-    composerDisabled: c ? c.disabled : null,
-    composerValue: c ? c.value : null,
-    pending: tp._pending.size,
-  });
-}
-
-const wait = ms => new Promise(r => setTimeout(r, ms));
-
-function typeInto(text) {
-  const c = composer();
+function typeInto(root, text) {
+  const c = root.querySelector('.chat-input');
   c.focus();
   // Preact reads e.target.value on input, so the native setter has to run.
   Object.getOwnPropertyDescriptor(window.HTMLTextAreaElement.prototype, 'value')
@@ -202,22 +208,42 @@ function typeInto(text) {
   c.dispatchEvent(new Event('input', { bubbles: true }));
 }
 
-(async () => {
+async function runScenario(name, chatReply) {
+  const root = document.createElement('div');
+  document.getElementById('root').appendChild(root);
+  const tp = makeTransport(name, chatReply);
+  render(html`<${Host} tp=${tp} />`, root);
+
+  const steps = [];
+  const snap = (label) => {
+    const c = root.querySelector('.chat-input');
+    steps.push({
+      label,
+      bubbles: root.querySelectorAll('.chat-bubble').length,
+      lastText: [...root.querySelectorAll('.chat-text')].pop()?.textContent ?? null,
+      // What a frozen tab actually is: the composer is disabled for as long as
+      // a send is in flight.
+      composerDisabled: c ? c.disabled : null,
+      composerValue: c ? c.value : null,
+      pending: tp._pending.size,
+    });
+  };
+
   await wait(500);
   snap('arrived');
 
-  // The Videos tab asked about a file a moment ago and is still waiting. Any
-  // unanswered request will do; this is the one that was live when the defect
-  // was found.
-  tp.fetchMediaMeta('a-file-the-node-refused')
-    .then(m => log.push('media_meta resolved with ' + m.type),
-          e => log.push('media_meta rejected: ' + e.message));
+  // The Music tab asked about a track and is still waiting on the node, which
+  // is waiting on something else. Any older unanswered request will do; this
+  // is the one that was live when the defect was found.
+  tp.fetchMusicMeta('a-track-the-node-is-slow-about')
+    .then(m => log.push(name + ': music_meta resolved with ' + m.type),
+          e => log.push(name + ': music_meta rejected: ' + e.message));
   await wait(100);
-  snap('stale request pending');
+  snap('older request pending');
 
-  typeInto('hello');
+  typeInto(root, 'hello');
   await wait(100);
-  composer().dispatchEvent(new KeyboardEvent('keydown',
+  root.querySelector('.chat-input').dispatchEvent(new KeyboardEvent('keydown',
     { key: 'Enter', bubbles: true, cancelable: true }));
 
   // Far short of _sendAndWait's 30s timeout: a send that has not come back by
@@ -225,6 +251,14 @@ function typeInto(text) {
   await wait(1500);
   snap('after send');
 
+  return { name, steps };
+}
+
+(async () => {
+  const out = { scenarios: [], log };
+  out.scenarios.push(await runScenario('ack', { type: 'ack', v: '0.14' }));
+  out.scenarios.push(await runScenario(
+    'error', { type: 'error', detail: 'Request failed' }));
   fetch('/log', { method: 'POST', body: JSON.stringify(out) });
 })();
 </script></body></html>"""
diff --git a/packages/meshbay-hub/tests/harness/upload_seal_probe.mjs b/packages/meshbay-hub/tests/harness/upload_seal_probe.mjs
new file mode 100644
index 0000000..0b77e42
--- /dev/null
+++ b/packages/meshbay-hub/tests/harness/upload_seal_probe.mjs
@@ -0,0 +1,102 @@
+/**
+ * Does the browser's real `uploadFile` produce frames a real node can open —
+ * and does it read back what that node actually answered?
+ *
+ * Drives **the real `MeshBayTransport` over the real `crypto.js`**. Only the DOM
+ * and the DataChannel are stand-ins; the msgpack encode, the HKDF, the AES-GCM
+ * and the whole upload loop are the shipped code.
+ *
+ * It exists because a source-reading test cannot show either half of MNP 2.0's
+ * upload. `test_transport_contracts` can see that `sealGroup` is called; only
+ * this can show that what comes out opens under `meshbay_common.groupbox` — and
+ * that the caller of `uploadFile` is told the name the *node* chose, which now
+ * arrives sealed and would otherwise be `undefined` with nothing to notice it.
+ *
+ *   node upload_seal_probe.mjs <static-dir> <input.json>
+ *
+ * Two modes, because the node half runs in Python between them:
+ *   "send"    — run the upload, print every frame it emits, answer nothing
+ *   "receive" — run it again, answering with acks Python built. Their
+ *               `upload_id` is retargeted to this run's; it is outside the
+ *               seal, so the ciphertext stays exactly the one Python produced.
+ */
+import fs from 'fs';
+
+const STATIC = process.argv[2];
+const input = JSON.parse(fs.readFileSync(process.argv[3], 'utf8'));
+
+for (const level of ['log', 'warn', 'error', 'info', 'debug']) {
+  console[level] = (...args) => process.stderr.write(args.join(' ') + '\n');
+}
+
+globalThis.window = globalThis;
+globalThis.addEventListener = () => {};
+globalThis.removeEventListener = () => {};
+globalThis.location = { hash: '' };
+globalThis.document = {
+  addEventListener() {}, removeEventListener() {}, visibilityState: 'visible',
+};
+
+new Function(fs.readFileSync(`${STATIC}/crypto.js`, 'utf8'))();
+const transportSrc = fs.readFileSync(`${STATIC}/transport.js`, 'utf8');
+new Function(transportSrc)();
+// The msgpack codec is private to transport.js — pulled out the same way the
+// groupbox parity harness pulls out sealGroup, so this probe encodes and
+// decodes with the codec that is actually shipped rather than a second one.
+const { msgpack_encode, msgpack_decode } =
+  new Function(transportSrc + '\nreturn { msgpack_encode, msgpack_decode };')();
+
+const hex = (s) => Uint8Array.from(s.match(/../g).map((b) => parseInt(b, 16)));
+const toHex = (u8) =>
+  Array.from(u8).map((b) => b.toString(16).padStart(2, '0')).join('');
+
+// Just enough of a File: a name, a size, and slices that yield ArrayBuffers.
+const bytes = hex(input.file.data);
+const file = {
+  name: input.file.name,
+  size: bytes.length,
+  slice(a, b) {
+    const part = bytes.slice(a, b);
+    return { arrayBuffer: async () => part.buffer.slice(
+      part.byteOffset, part.byteOffset + part.byteLength) };
+  },
+};
+
+const tp = new window.MeshBayTransport('', 'token');
+tp._connected = true;
+tp._channel = { readyState: 'open', bufferedAmount: 0, send() {}, close() {} };
+tp._pc = { close() {} };
+tp._gekRaw = hex(input.gek);
+tp._connectArgs = { groupId: input.group_id };
+tp._nodeVersion = input.node_version;
+
+const frames = [];
+let uploadId = null;
+tp._send = (msg) => {
+  frames.push(toHex(msgpack_encode(msg)));
+  if (msg.upload_id) uploadId = msg.upload_id;
+  if (input.mode !== 'receive') return;
+  // Answer as the node did, on the next turn of the loop so the send path
+  // finishes first — which is also how a real ack arrives.
+  const ack = msgpack_decode(hex(input.acks[msg.chunk_index]));
+  ack.upload_id = uploadId;
+  // Through the real `_dispatch`, so the routing under test — matching an
+  // ack to its uploader by `upload_id` — is the shipped one.
+  setImmediate(() => tp._dispatch(ack));
+};
+
+const out = { frames, mode: input.mode };
+const done = tp.uploadFile(file, { chunkSize: input.chunk_size,
+                                   dir: input.dir, root: input.root });
+
+if (input.mode === 'receive') {
+  done.then((stored) => { out.state = 'resolved'; out.stored = stored; })
+      .catch((e) => { out.state = 'rejected'; out.message = e.message; })
+      .finally(() => { out.upload_id = uploadId;
+                       process.stdout.write(JSON.stringify(out)); });
+} else {
+  // Nothing will answer, so let the send loop run itself out and report.
+  done.catch((e) => { out.state = 'rejected'; out.message = e.message; });
+  setTimeout(() => { out.upload_id = uploadId;
+                     process.stdout.write(JSON.stringify(out)); }, 250);
+}